Two releases worth of deprecation period is plenty of time to migrate any additional APIs that folks feel are sufficiently hard to replicate correctly that if makes sense for them to stay somewhere in the standard library rather than the projects using them making a copy in their own utility libraries, so I also think it would be fine to stick with Steve’s initial plan, where the suggestion for some APIs is “copy the code you’re using”.
3 Likes
Anyone want to add or reinforce any complaints that they feel haven’t been addressed sufficiently? If not, I’ll ask the SC for a delegate (you can still raise concerns after that).
2 Likes
Hello. We have met with @jaraco, @pradyunsg, @encukou and Matthias Klose and discussed the issues with the move of distutils from the standard library to setuptools (or any other pip-installable project) affecting Fedora and Debian.
Status quo
Both distributions currently patch distutils.command.install.INSTALL_SCHEMES and have custom logic for selecting the appropriate scheme.
- The Fedora patch is minimal yet “hacked”.
- The Debian patch is more complex yet more systematic.
Both patches are designed for the same purpose: When the Python packages are installed by users (e.g. via sudo), they are installed to a location that is not managed by the system package manager (i.e. /usr/local/lib(64)?/pythonX.Y/(site|dist)-packages). When the Python packages are installed to a fake --root by the system package manager build scripts (e.g. rpmbuild), they are installed to the system location (i.e. /path/to/buildroot/usr/lib64/pythonX.Y/(site|dist)-packages, so the system packages (.deb or .rpm) are installed to /usr/lib(64)?/pythonX.Y/(site|dist)-packages). The site module is also patched to ensure proper sys.path.
I might not like it, but users often run sudo pip install, sudo pip install --upgrade or even sudo python setup.py install. The patches exist to prevent systems being bricked by this behavior (e.g. by replacing a dependency for the package manager with an incompatible version).
The problem
If the code moves to setuptools (or pip or pypa/distutils or any other pip-installable project), users will eventually override our patches with the pip-installed version of distutils. This will then let them override system packages.
Hence, if distutils needs to be removed from stdlib, the code that we patch needs to move to another part of the standard library.
The plan
At the meeting, we have come up with the following plan that suits both PyPA representatives and distros representatives. We present the plan to CPython/PEP 632 representatives and would like to see if it works for you as well.
To avoid confusion, I use the term pypa/distutils for the distutils fork that is to be bundled in setuptools. I use the term cpython/distutils for the distutils in Python’s standard library.
The plan is presented first as is, rationales to follow.
- pypa/distutils keeps
INSTALL_SCHEMESdefined for compatibility with Python < 3.10, it also gains aget_default_scheme()function. - cpython/distutils in Python 3.10+ moves
INSTALL_SCHEMEStosysconfig.INSTALL_SCHEMESand imports them for backwards compatibility, usessysconfig._get_default_scheme()to determine the scheme. - distributors backport (2) to older Python versions and patch sysconfig instead of patching cpython/distutils.
- pypa/distutils uses
INSTALL_SCHEMESand_get_default_scheme()from sysconfig if found, fallbacks to its own definitions if not. - pypa/distutils fallbacks will honor distributors’ intentions to the greatest extent possible.
Rationales
-
Setuptools would like to evolve its pypa/distutils. To do so, it cannot use cpython/distutils on older Python versions and pypa/distutils on newer; it needs to always use pypa/distutils. Hence pypa/distutils need to be able to work with Pythons that don’t yet have
sysconfig.INSTALL_SCHEMESandsysconfig._get_default_scheme(). Hence, it needs to contain a reasonable set of install schemes (as it does today). Nothing needs to change. -
sysconfig was selected based on our common understanding of the module’s purpose. We don’t necessarily insist it is sysconfig, as long as the module exists in currently existing Python versions. sysconfig already contains some installation schemes that are not 1:1 copy of the installation schemes from distutils. The idea is to unify those as much as possible and provide a functionality that would allow the deprecated cpython/distutils module to present the installation schemes from sysconfig in a backward-compatible way. (In the worst-case scenario, we namespace the different sets of installation schemes.) I am prepared to work on a proof of concept of this if we agree on the plan. If the proof of concept works, the Python Maintenance team at Red Hat can dedicate a long-term commitment.
-
The distributions already patch both old and new Python versions. We are prepared to change our patches to move the patched definitions to sysconfig and use read them form there in cpython/distutils. That way, the behavior remains the same, except the source of the information shifts aside to allow the next points to happen.
There are these edge cases where users might get a broken system:- users of end-of-life Fedora/Debian versions who decide to install/update setuptools from PyPI
- users who don’t update their system and decide to install/update setuptools from PyPI
There’s not much we can do about that.
-
pypa/distutils support old and new Python versions. It needs to function if the installation schemes exist in sysconfig as well as if it does not. Preferring feature-detection over version-detection, pypa/distutils fully supports the distributors who backported the changes and respect their patches. Setuptools maintainers are prepared to work on this.
-
Setuptools maintainers would like to be able to adapt pypa/distutils defaults in a way that would make the patches minimal (ideally unnecessary, however, that might not be possible). If useful, such defaults might propagate to Python’s sysconfig later. This step is not necessarily needed right now.
I’ve emphasized some parts to illustrate that we have the resources to make this happen.
3 Likes
The above all make perfect sense. Thank you all so much for taking the effort to come up with a solution!
I have a few questions, probably cosmetic and not really related to the problem faced by distros:
- Since
sysconfigalready has_INSTALL_SCHEMES, would it make sense to reuse it instead of having a new global variable named similarly but for different purposes? - Can other packaging tools make use of
_get_default_scheme()andINSTALLED_SCHEMES, or should they be considered a private interface between distros and pypa/distutils? - Following the above, would it make sense to make
_get_default_scheme()a guarenteed (public?) API in CPython? (And_INSTALL_SCHEMESif that’s used instead of a new standaloneINSTALLED_SCHEMES, see first point.)
Thanks for the write-up!
I’m more than happy to include migrating install schemes into sysconfig as part of the PEP, though I would like to hear from the pip maintainers how it would impact them.
I assume that pip installs wheels according to the install schemes, though @pf_moore listed distutils.command.install.SCHEME_KEYS above, and since IIUC setuptools also wants to get out of the installation business, pip is probably more important in terms of future proofing.
(Also, I assume you considered patching an additional directory into the sys.path, rather than trying to patch out installing into a distro-controlled directory? That shouldn’t require modifying install schemes at all.)
1 Like
Unless I missed something in my quick read of your proposal, I think it would be important to spell out exactly which version(s) of sysconfig you are talking about. Recall that we are in the unfortunate situation of having two almost identical sysconfigs in the standard library: one in Lib/sysconfig.py and one in Lib/distutils/sysconfig.py. Perhaps this is an opportunity (or requirement) to get rid of one of them?
1 Like
My understanding is that the top-level sysconfig was added when distutils was (infomally) deprecated years ago, but has apparently gone out of sync. Hard to say which is more up to date, but if people have been patching the distutils one with an impact, it’s clearly still in use.
Obviously the distutils version will be going away with the rest of distutils.
1 Like
That is the idea, if at all possible.
I’d like to make it “public”, i.e. renaming to get_default_scheme() or get_default_installation_scheme().
2 Likes
Could you please explain this in more detail?
1 Like
It’s been awhile but IIRC Lib/sysconfig.py was added when work on packaging/distutils2 was underway and it appeared that, for at least some number of releases, both distutils and packaging/distutils2 would need to co-exist in the standard library. Eventually it was decided to not go ahead with that plan and most of the changes to distutils were reverted but the new sysconfig remained. (By the way, that’s also why Lib/_osx_support.py exists: to have in one place the macOS customizations needed for both distutils and packaging/distutils2.) I’m not sure the two versions were ever totally in-sync; I recall that some ones (@jaraco and @barry ?) spent some time a few years back trying to get them more into sync but I don’t see in changelogs where or when that happened so I might be dreaming. Anyway, it’s been a problem for a long time (there are a number of open bpo issues about it) so, rather than muddying the waters further, we should try to resolve this in conjunction with or as part of PEP 632.
1 Like
I don’t think distutils2 included its own copy of sysconfig, as Tarek quite rightly pointed out that sysconfig needs to be in the standard library as it provides information about the interpreter installation and shouldn’t change independently of updates to the interpreter and standard library.
When the migration to distutils2 was judged unsuccessful, the deprecation of distutils.sysconfig still held, but until PEP 632 we didn’t have a new plan for how to follow through on it.
1 Like
On re-reading, the Fedora patch is essentially what I was suggesting. Yes, it’s a little bit hacked, but basically unless you’re building your own packages, you always want /usr/local/lib... to be the default library location, provided /usr/lib... is also on sys.path.
What you described sounds like “default to /usr/lib... and also import from /usr/local/lib... and try to install to /usr/local/lib...”, which strikes me as more complicated logic than “default to /usr/local/lib... and also import from /usr/lib...”.
Yes, both would require patching, but I’m pretty sure that the latter would only add an extra sys.path.append() somewhere (site.py? getpath.c?) and a build-time configure option, compared to the more complex patching you’re doing now.
Just an idea. I don’t understand your constraints as well as you do, so if there are reasons why this doesn’t work then just ignore me 
1 Like
The current text says (approx) “if you’re using distutils.sysconfig, switch to sysconfig”. Is that clearing the waters enough? Or is something more necessary?
1 Like
One constraint is that if site packages are disabled ($PYTHONNOUSERSITE, python -I), you should get /usr/lib but not /usr/local/lib.
2 Likes
Perhaps we could just improve support in upstream for this scenario (not in this PEP)? I might have another go at streamlining the startup process at the sprints and could factor this in. The aim being to remove the need for direct patching.
Or maybe deprecating the legacy install tools and working with the modern install tools would be enough?
Either way, my earlier questions stands:
1 Like
I’m 100% on board, but I also don’t maintain anything that uses legacy install tools, soooo…
I think that works.
Assuming this PEP does come through, I think we’ll likely change pip to use sysconfig.INSTALL_SCHEMES instead of distutils.command.install.SCHEME_KEYS on Python >=3.10. Looking at the existing implementation, I don’t think it’ll be too difficult (famous last words).
2 Likes
Yes, I think it’s enough for the PEP.
2 Likes
I’ve started to work on distutils.sysconfig → sysconfig merge and it seems to be ready from the code point of view. If you want to follow my progress, see: https://github.com/frenzymadness/cpython/pull/2
I’ll open a PR to cpython once I finish documentation.
1 Like
Thanks so much for the work, @frenzy! Your progress looks great so far.
Feel free to open the PR whenever you’re ready. It won’t be merged until the PEP is formally accepted, but it will help show people what changes will happen when it is.
1 Like