As Brett asked on https://github.com/pypa/packaging/issues/363, I am opening this thread to document and discuss what is missing to make PEP 517 usable for Linux (and other) distributions.
Pre PEP 517 we had two workflows I am aware of:
- Install via setuptools (eggs)
- Build a wheel via setuptools and install it with a wheel installer
In a PEP 517 world, we have just one:
- Build a wheel via PEP 517 or by invoking the backend directly, if it supports that, and install it with a wheel installer.
Currently, the only usable wheel installer is pip.
The problem is that some distros can’t use pip, AFAIK there are two problems they may have:
- pip is very hard to convince to behave like a simple and reproducible wheel installer. Distro packagers need to be able to tell pip to install to a custom location, ignore dependencies and to not try to connect to the internet. pip was simply not designed for this.
(the biggest one)
- pip vendors its dependencies. Some distros have rules against vendoring of dependencies, so they need to devendor pip. Usually this would be annoying, but should be fine, but it is very problematic in this case as pip would be a required tool for the workflow. This forces distros to manually bootstrap pip and all its dependencies, which I think is around 30 packages. This is simply not a feasible thing to do, especially considering it needs to be done every time Python updates.
This last issue has already been discussed at length, I don’t want to rehash the discussion as I believe that would not be productive. I think PyPA maintainers should just take it as pip not being a suitable wheel installer for bootstrapping a Python environment.
And as so, we also don’t need to have more discussions about the first issue as the second one is a blocker, it is simply stated here to document it. It is not a very big blocker but it is an issue.
So, distributions that can use a vendored pip do not have an issue, they can just bootstrap pip and use it to build the rest of the packages. But distros that cannot, currently are forced to use the old approach and install directly via setuptools (eggs) as that is the only other alternative to get a package installed.
It is worth noting that building via PEP 517 also used to be one of the blocker issues. We need a way to build the wheels. pypa/build now solves that.
Currently, distros that have the issue described above are forced to install via setuptools directly (eggs). We want them to move to PEP 517, which only provides installation via wheels.
For this, we need two components:
- PEP 517 builder
- Wheel installer
The first one has recently been solved by pypa/build.
The last one still needs to be resolved. There is an initial implementation in pradyunsg/installer but progress has been slow and is still a WIP.
So, the current blocker of distributions adopting a PEP 517 workflow is a suitable wheel installer not existing.
The provisional solution
Since there is no path for the distributions described here to adopt PEP 517, they are forced to use the previous workflow, installing via setuptools (eggs).
Some distros have a heavy preference from building for the original source, rather than the sdist. AFAIK this is not required, but important to note. As a last resort, I think distro packagers could compromise and install via sdist if it provided a
The best option, naturally, would be to keep using setuptools until a wheel installer exists and distro can finally embrace PEP 517.
setuptools supports PEP 517 so I don’t think arguments like “we want to update the tooling to PEP 517” make sense.
I do understand some core packaging tools maintainers might want to move away from setuptools to other backends but I ask if you think the benefit you get there is worth breaking the workflow for this many people?
Distributions will still have to use the old workflow. So for you saving a few lines in your metadata definition or not having to keep some metadata fields in sync, distro packagers will have to craft a
setup.py to use to build your package and will have to respond to possible bugs coming from there.
So, I would like to ask, please, keep using setuptools until there is a wheel installer suitable for distros to use. If you really need to move away from it, at least make sure your backend generates a
setup.py to put in the sdist.
Improving the workflow
Okay, moving past the workflow problem. The current state of the packaging utilities is still hostile against distribution packagers and everyone bootstraping a Python environment from scratch. PyPA solved this issue by vendoring pip’s dependencies but this is not something everyone can do. I believe the PyPA, given its position, should make sure this workflow is as easy as possible, even if it is not directly affected.
There is no escaping from some manual package bootstraping, but this issue can be heavily improved by reducing the amount of packages required to bootstrap to the reasonable minimum. One way this can be easily improved is by making sure all packages use the same backend. When bootstraping the packages we would have to also bootstrap their backend, in order to build them, if all packages use the same backend we reduce the number of packages required, which even just as one, is a huge improvement.
The target packages would be:
- pradyunsg/installer + its dependencies
Currently, two backends are used, setuptools and flit. I would really like if we could agree on one backend to switch to once the wheel installer is in place. The only foreign package is toml, but I think the maintainer could be persuaded by presenting this reasoning. We still have to wait on pradyunsg/installer to see what else it introduces.
TLDR: Currently, the only thing blocking distributions from adopting PEP 517 is there not being a wheel installer (pip is not a solution here).