PyPi building and uploading ARM7 versions of packages and making a cross platform project

I am wondering how I go about rebuilding or uploading the existing ARMv7l versions of a number of existing standard Python packages I have built rather than creating my own local repository for them.

To fill you in with the whole of what I am doing…

I have an existing project I picked up called MailInABox (GitHub - mail-in-a-box/mailinabox: Mail-in-a-Box helps individuals take back control of their email by defining a one-click, easy-to-deploy SMTP+everything else server: a mail server in a box.) that does not even have a requirements.txt and uses virtualenv. I have successfully built a modded version of a fork called power-mailinabox on Armbian Debian (GitHub - AaronNGray/armbian-mailinabox) but the build builds existing PyPi packages for ARMv7l and it takes ages, literally hours.
I want to be able to upload packages to PyPi, this might require rebuilding ?
Are there any instructions on how to do this for ARM anywhere or can someone give me guidance on how to go about doing this.
I am not really a Python Programmer but know the language well enough to get by but I dont really know the development environment very well. I want to be able to build and upload its ARM7 packages to PyPi. Then be able to turn MailInABox into a normal Python project with a requirements.txt that will work cross platform and load either on an x86 or an ARM architecture.

Here is the list of packages :-

bcrypt
b2sdk
boto3
typer<0.5.0,>=0.4.1
tomli<3.0.0,>=2.0.1
rich<13.0.0,>=12.4.4
httpx<0.24.0,>=0.23.0
anyio<4.0.0,>=3.6.1
nala
duplicity
rtyaml
email_validator>=1.0.0
exclusiveprocess
flask
dnspython
python-dateutil
expiringdict
gunicorn
qrcode[pil]
pyotp
pyopenssl
idna
cryptography
boto
psutil
postfix-mta-sts-resolver

Many thanks in advance,

Aaron

Reading PEP 599 – The manylinux2014 Platform Tag | peps.python.org it looks like I will have to create my own repository and wheels as I have dependencies on many other Linux librairies !

If someone could give me the appropriate guidance to creating my own dual armv7l and x86_64 repo for this purpose please ? I gather that I have to be compliant with PEP 600 – Future ‘manylinux’ Platform Tags for Portable Linux Built Distributions | peps.python.org ?

I have discovered pypi-server and twine and howto’s.

I am still confused as to how I will go about getting my existing built code structure I have saved from the armv7l build either into my own repository or ideally up to the standard PyPi repository.

I have the following :-

aaron@aaron-System-Product-Name:~/backup$ ls usr/local/lib/mailinabox/mailinabox/env/lib/python3.9/site-packages/
aiodns
aiodns-3.2.0.dist-info
aiohttp
aiohttp-3.9.5.dist-info
aiosignal
aiosignal-1.3.1.dist-info
annotated_types
annotated_types-0.7.0.dist-info
asn1crypto
asn1crypto-1.5.1.dist-info
async_timeout
async_timeout-4.0.3.dist-info
attr
attrs
attrs-23.2.0.dist-info
b2sdk
b2sdk-2.4.1.dist-info
blinker
blinker-1.8.2.dist-info
boto
boto-2.49.0.dist-info
boto3
boto3-1.34.131.dist-info
botocore
botocore-1.34.131.dist-info
certifi
certifi-2024.6.2.dist-info
cffi
cffi-1.16.0.dist-info
_cffi_backend.cpython-39-arm-linux-gnueabihf.so
charset_normalizer
charset_normalizer-3.3.2.dist-info
click
click-8.1.7.dist-info
cryptography
cryptography-2.2.2.dist-info
dateutil
dns
dnspython-2.6.1.dist-info
easy_install.py
email_validator
email_validator-2.2.0.dist-info
exclusiveprocess
exclusiveprocess-0.9.4.dist-info
expiringdict
expiringdict-1.2.2.dist-info
flask
flask-3.0.3.dist-info
frozenlist
frozenlist-1.4.1.dist-info
gunicorn
gunicorn-22.0.0.dist-info
idna
idna-3.7.dist-info
importlib_metadata
importlib_metadata-7.2.0.dist-info
itsdangerous
itsdangerous-2.2.0.dist-info
jinja2
jinja2-3.1.4.dist-info
jmespath
jmespath-1.0.1.dist-info
logfury
logfury-1.0.1.dist-info
markupsafe
MarkupSafe-2.1.5.dist-info
multidict
multidict-6.0.5.dist-info
packaging
packaging-24.1.dist-info
PIL
pillow-10.3.0.dist-info
pip
pip-24.0.dist-info
pkg_resources
pkg_resources-0.0.0.dist-info
pkg_resources-0.0.0.virtualenv
png.py
postfix_mta_sts_resolver
postfix_mta_sts_resolver-1.4.0.dist-info
psutil
psutil-6.0.0.dist-info
__pycache__
pycares
pycares-4.4.0.dist-info
pycparser
pycparser-2.22.dist-info
pyotp
pyotp-2.9.0.dist-info
pypng-0.20220715.0.dist-info
python_dateutil-2.9.0.post0.dist-info
PyYAML-6.0.1.dist-info
qrcode
qrcode-7.4.2.dist-info
requests
requests-2.32.3.dist-info
rtyaml
rtyaml-1.0.0.dist-info
s3transfer
s3transfer-0.10.1.dist-info
setuptools
setuptools-44.1.1.dist-info
setuptools-44.1.1.virtualenv
six-1.16.0.dist-info
six.py
typing_extensions-4.12.2.dist-info
typing_extensions.py
urllib3
urllib3-1.26.19.dist-info
_virtualenv.pth
_virtualenv.py
werkzeug
werkzeug-3.0.3.dist-info
wheel
wheel-0.34.2.dist-info
wheel-0.34.2.virtualenv
_yaml
yaml
yarl
yarl-1.9.4.dist-info
zipp
zipp-3.19.2.dist-info
aaron@aaron-System-Product-Name:~/backup$ ls usr/local/lib/mailinabox/mailinabox/env/lib/python3.9/site-packages/cryptography
__about__.py   fernet.py  __init__.py  utils.py
exceptions.py  hazmat     __pycache__  x509
aaron@aaron-System-Product-Name:~/backup$ ls usr/local/lib/mailinabox/mailinabox/env/lib/python3.9/site-packages/cryptography/__pycache__
__about__.cpython-39.pyc   fernet.cpython-39.pyc    utils.cpython-39.pyc
exceptions.cpython-39.pyc  __init__.cpython-39.pyc
aaron@aaron-System-Product-Name:~/backup$ ls usr/local/lib/mailinabox/mailinabox/env/lib/python3.9/site-packages/cryptography-2.2.2.dist-info
AUTHORS.rst  LICENSE         LICENSE.BSD  RECORD     top_level.txt
INSTALLER    LICENSE.APACHE  METADATA     REQUESTED  WHEEL
aaron@aaron-System-Product-Name:~/backup$ cat usr/local/lib/mailinabox/mailinabox/env/lib/python3.9/site-packages/cryptography-2.2.2.dist-info/WHEEL
Wheel-Version: 1.0
Generator: bdist_wheel (0.34.2)
Root-Is-Purelib: false
Tag: cp39-cp39-linux_armv7l

aaron@aaron-System-Product-Name:~/backup$ 

Many thanks in advance,

Aaron

You aren’t going to be able to upload wheels to PyPI for any
packages you’re not a collaborator on. That’s simply not its model.

You can create your own repository fairly easily though:

  1. pip install all the packages on a representative of your target
    platform; pip will cache the wheels that end up being built for
    this

  2. copy the wheels from pip’s cache to a webserver with some
    autoindexing (or generate your own “simple API” index that links
    them all)

  3. set the URL to where you’re serving those wheels as an
    –extra-index-url on future pip install commands, and it see
    those wheels as available

A project I work on does this for platform-specific wheels
supporting a variety of systems we use in our CI/CD environment and
it functions just fine.

https://packaging.python.org/en/latest/guides/hosting-your-own-index/

1 Like

okay I have discovered I have to use PyPi/Warehouse instead of pypi-server as I cannot get futures to build as its based on an egg so obviously totally out of date.

I wanted to try a local repository and how to convert what I have file wise to .whl files.

What exactly do you have so far, and how did you produce it?

I listed it in a post above ^^^ they are basically the contents of the .whl zip files, so I can write a bash script to generate .whl files if there is not a standard way of doing this ?

It seems like you’re overcomplicating this. The
build · PyPI tool should be able to take care of
creating wheels from an sdist you download from PyPI or a source
tree for the software in question. But as I said earlier, pip itself
can also work out what’s needed, just install your project or a
requirements.txt file into a venv with pip and then pull the
resulting whl files for all the dependencies from your pip cache.

1 Like

I am using nginx so am following this without the docker level :-

I am wondering how I go about supporting different architectures both on the same server and also on my project that uses them so its cross platfrom ?

You should be able to just dump all the different per-platform
wheels for a given project name into the same directory and simple
API index file, installers like pip figure out which ones to use by
pattern matching various fields in their filenames. You can look at
PyPI itself for examples:

https://pypi.org/simple/cryptography/

I think I can do 'pip wheel <binary-package-dir> -w <dst-dir>` to generate the wheel files ?