I ran into an issue trying to install a Python package for a Python platform that’s not supported on PyPI. We’ve been going back and forth, and have a solution (fake support for said Python platform using the none-any tag). However, the maintainer has asked me some questions on what would be the actual best way to distribute wheels and I’m not sure how to answer (I’m posting this on their behalf).
wasmtime is a Web Assembly Runtime written in Rust. It has a Python package that will call into the Rust code to actually execute WASM. It uses ctypes for everything, is compatible with all Python 3 versions, and essentially the only difference between wheels for different platforms is “the wasmtime blob included with each wheel”.
The maintainer doesn’t want all blobs to be available in each wheel. Consequently, right now, wasmtime uses setup.py and uses bdist_wheel --plat-name to set each wheel from none-any to “the relevant supported platform”. The none-any wheel is a fallback meant for “any Python platforms not supported by PyPI, including all the relevant WASM blobs for those platforms alone” (currently MinGW); sdists haven’t been set up yet, but in the future they should serve those platforms just fine also.
My question is: As comment in the link above, bdist_wheel --plat-name is discouraged. Is there an alternate way to cross-compile using setuptools as a backend? I don’t think build as a frontend provides enough knobs to do this, but not sure. I think we’d like to avoid invasively changing the setup.py or switching to a different backend. Although at this point I think adding a pyproject.toml such as below is unavoidable:
You might get some hints from how cibuildwheel sets up cross-compiling on Windows. The --plat-name option can be set through a configuration file, which can be specified by environment variable (as shown in the linked code), and the variable to override the extension module suffix explicitly should work on all platforms.
VSCMD_ARG_TGT_ARCH probably isn’t relevant, though it may work if you need the distutils.get_platform() function (or its equivalents) to report something different.
So it’s a bit complicated, but the only real alternative is going to be invasive. (Adding that pyproject.toml - if none currently exists - shouldn’t be invasive at all. The build-backend specification is the only one you need, there’s no requirement to fill out any of the rest in order to use setuptools (or other build backends that don’t require it) - if there is, it’s because setuptools chose to make it mandatory, even though they didn’t have to.)
You can use pypa/build’s --config-settings argument to pass arguments to the build backend. In your case you can use this to pass --plat-name to setuptools. You want something like python -m build -w -C "--global-option=--plat-name" -C "--global-option=<tags>". The syntax is tricky as you need to pass each part of setuptool’s --global-option through pypa/build’s --config-settings.
The other thing you could do is use the wheel tags CLI to change the tags you need. At work we use wheel tags to re-tag wheels.
Interesting. I wonder why you can’t do -C "--global-option=--plat-name=<tags>"? Presumably, setuptools way of handling multiple --global-options is create dict entries from pairs ._.?
Okay, both of your replies are useful information. To summarize, and based on what I’m reading to make changes non-invasive:
Add a stub pyproject.toml for forward-compat.
Create pass the --plat-name argument via --config-setting to the setuptools build backend.
Alternatively, skip the --config-setting dance and use wheel tags after the fact. Both versions of the solution alleviate the need to call setup.py directly.
By doing my own reading, coupled w/ this thread, I have to be aware:
By and large, --plat-name, --global-option, --build-option, etc, are not documented.
Additionally, setuptoolshasn’t not really standardized how to use PEP 517 config_settings. In fact, it doesn’t always work as expected with --build-option at least!
--global-option/--build-option are going away (See 11859’s recent comments) in pip, which may force setuptool’s hand.
But the PEP-517 hook works, so “at least I’m not directly calling setup.py anymore”.
Yep! I would definitely recommend just using wheel tags right after running build on the project directory. Then you only have to add the pyproject.toml file and run the wheel command, which should work regardless of how setuptools/pip/build change.
Indeed, using the wheel command makes sense, I was just curious how to do it the long way for completeness. Now I have an idea of how to go about fixing the original package non-invasively, until there is bandwidth to switch to a different backend (probably maturin) :D!