I wouldn’t judge Fedora’s “hacked” patch too harshly — there’s something to be said for the simpler approach. Debian’s patch is far more comprehensive, yes, but that’s exactly what gets it into trouble.
When third-party install scripts/tools go to install Python modules, many use the path returned by distutils.sysconfig.get_python_lib(prefix="/usr/local/") as their target path (if building for the default /usr/local install prefix).
On Fedora, that call returns /usr/local/lib/python3.x/site-packages, which will be in the default sys.path if the directory exists. All is well.
Debian’s patched distutils will instead return /usr/local/lib/python3/dist-packages, a path that is not added to sys.path under any circumstances (except manually). That path is also incorrect based on their own documentation. (As you say, the path should be /usr/local/lib/python3.x/dist-packages.)
(setuptools on Debian-based systems uses correct paths, though. So in a roundabout way this is just another point in favor of deprecating broken distutils, and instead using the better-maintained setuptools wherever possible.)
FWIW, setuptools currently gets installed by ensurepip, but that’s considered an implementation detail and subject to removal at any time. It was added because at the time you had to have setuptools already installed to install from a sdist, so it was the path of least resistance for having a “fully working” pip.
At this point, I don’t believe pip requires it anymore by default, so I think we could just remove it now. Though it will likely break some people who are making an assumption that setuptools will always be there.
Actually I take that back, --use-pep517 still isn’t the default, which means we still need setuptools preinstalled unless the project has opted in to PEP 517 via the presence of a pyproject.toml file.
For what it’s worth, I’ve removed setuptools from my default system installation, and that’s led to occasionally having to pass --use-pep517 to pip, but in no circumstance has that workaround failed in general. I’ll take this opportunity to set the relevant environment variable to make that the default.
More generally, where are we (@encukou and @hroncok) with progress on our efforts? I have some time next week I can invest in this effort and would like to push some things forward in distutils/setuptools if possible. I plan to get back up-to-speed with the implementation details this weekend and maybe draft some changes. Feel free to reach out to me wherever to sync/pair.
This caused me to wonder what runtime-stuff like from distutils.version import LooseVersion should be replaced with (because certainly, projects shouldn’t run-depend on build tools). The PEP says
For these modules or types, use the standards-defined Python Packaging Authority packages specified:
distutils.version - use packaging
However packaging doesn’t support this yet. Should I open an issue there? Not sure how much other API surface might be affected, thought I’d ask here first.
packaging supports the agreed standards (so for versions, PEP 440), so it has a Version class that covers that. But LooseVersion covers non-standard versions, and packaging won’t handle that (there is a LegacyVersion class in packaging, but this is deprecated, so is probably going to disappear sooner than LooseVersion will!)
As far as I know, there is no library that covers non-PEP 440 versions, so you’ll likely have to move to using and supporting standard versions only, or depend on setuptools (or the external distutils package) for the distutils class.
I think nearly all users of distutils.version would want to be using packaging.version.Version pretty soon (whatever the exact spelling is).
This would be a migration from an unstandardised “loose” approach to handling versions to a stricter, standards backed approach to handling them. That said, I do think most packages’ versions do comply with the stricter requirements of PEP 440, since PyPI has been validating versions against PEP 440 for quite a while now.
If someone really really needs to have loose version handling, they’re welcome to copy-paste the distutils code and make it available as a reusable PyPI package that they maintain.
There should be a migration path (and documentation of the PEP 440 limitations vs. LooseVersion) as part of the PEP 632 effort - not least because there are about 130k hits on github for from distutils.version import LooseVersion, so it’s going to cause substantial churn anyway.
For all those packages/maintainers, at least there should be a clear guide on how to do that, resp. the trade-offs of each (and IMO, run-depending on setuptools is a bad answer to give, at least if it’s the exclusive one, because that just encourages the proliferation of an unnecessarily complex dependency landscape, as well as other problems like people implicitly relying on setuptools without documenting it, because it’s always there, effectively)
Agreed, and now that it’s confirmed we need one, we’ve got two years to figure it out. And everyone affected can legitimately participate, rather than only those directly authoring the PEP
As an aside, only one of the first page of those search results is actual Python code. The rest are command line invocations or patches removing the use of LooseVersion. No doubt there are some legitimate cases further down the list, but I think 130k is a drastic overestimate (especially when accounting for duplicates and archived code). GitHub search result count is not a good way to get an estimate of the impact of any code change, unfortunately. You’ve got to dig much deeper into the actual results.
distlib has some support for “legacy” version formats. As these aren’t properly standardised, depending on the packages you’re working with, it may work for your needs.
It’s slightly different than the plan but we can still extend it later. The reason for the difference is that distutils contains a lot of logic to determine the right install schema. With this patch, we can simply redefine install schemas downstream. Later, we can come with a more sophisticated solution.
Update: pip has started to investigate the possibility of replacing distutils with sysconfig and hit a roadblock on selecting a scheme for --target and --user etc. I’ve proposed some changes to the sysconfig module that might be of interest to several people involved in this thread.
