This is good territory for a competing PEP though. I think it’s a bad proposal, but at least writing it up will let the steering council (or their nominee) make the decision.
I think we should probably be very clear here what all of the component parts involved do. I suspect many core Python developers don’t really have a good understanding of the packaging landscape, and nor should they. So please take this as some background on the whole “pip/setuptools/packaging standards” area as it relates to this PEP.
Distutils basically has 3 functions - a “build backend” (of sorts), a command line tool (setup.py install etc) and a library containing various “utilities” related to packaging.
Build Backend
It is not mandatory to install projects via pip. Pretty much all of the standards work we have been doing in packaging is to enable people to not use pip if they don’t want to. But at the present time, there aren’t many installers other than pip (and many things that look like they install stuff use pip behind the scenes). That’s not an intention, though, it’s mostly just a lack of anyone being willing to do the work involved (a common story in the world of packaging 
)
PEP 517 allows people to write “build backends” - tools that do the job of building a project - in a way that means they can be used by any installer. Distutils is not, and never will be, a build backend (because its maintainers don’t want to make it one, by virtue of the fact that there are no maintainers…) Setuptools is a PEP 517 build backend, and it strives to be compatible with distutils. It is extremely compatible, due to basically using distutils code and monkeypatching it, but it is not distutils. Switching to setuptools from distutils as a build backend should be as simple as replacing imports. If it’s not, then the setuptools maintainers may be willing to help improve compatibility. Or they might not - distutils never documented its API, so any replacement like setuptools has to draw the line somewhere. But in general, setuptools is sufficiently compatible for almost all uses of distutils as a build backend (yes, I’m repeating myself - it’s an important point!)
Command line interface
The command line interface of distutils (setup.py install etc) is not a standard. Build tools do not have to follow it (and flit, for example, doesn’t). Setuptools does use the same command line, but the setuptools developers have indicated they would like to deprecate it (I don’t have a link, sorry, but I doubt this is immiment, it’s more of a long term aspiration, I believe). And pip (for example) is moving away from relying on the setup.py interface in favour of PEP 517.
If you want to build wheels without using setup.py bdist_wheel, the new pybuild tool will be available soon to let you do so. (Mind you, distutils doesn’t support building wheels, so that’s somewhat irrelevant). If you want to install a project, distlib is an alternative wheel installer if you don’t like to use pip. And others are being developed. There is no standards-backed way to install a project without building an intermediate wheel, that’s true, but given that switching a project to use setuptools as a build backend should be straightforward, that’s not particularly relevant.
Alternatively, individual tools may choose to provide a tool-specific command line interface (flit has flit install I believe).
Library of useful functions
This is likely the biggest area where people are concerned. There are (were) a lot of pieces of functionality in distutils, related to tasks that are relevant to building projects. None of these were ever well-documented - many were not documented at all - but people used them nevertheless. Over time, equivalents for a lot of these APIs ended up in other parts of the stdlib (sysconfig, subprocess, shutil, importlib.metadata, …) and others ended up in 3rd party libraries like packaging. But not all did, and not everyone knew that the functions had moved.
Most of the “blame” for this could be put on the lack of anyone willing to maintain distlib. Ideally, functions with better alternatives could have been properly deprecated and documented as “if you are using X, you should look at moving to Y, which is a supported alternative”. But that never happened, and instead a broad “you shouldn’t use any of this” note was plastered over everything - which unfortunately looked like it mostly meant the “build backend” part of distutils, so didn’t actually catch the intended audience very well.
I don’t think anyone knows how much code still exists that uses distutils functions which have better equivalents - let alone code using functions that have no viable alternative right now. The deprecation warning will presumably flush them out. But at the moment, it’s a huge unknown, although we’re already hearing from people flagging up that they have issues here.
IMO, at the moment, the PEP is largely ignoring the potential for problems that may exist in this area. Maybe there’s nothing we can do to address those problems. But acknowledging them (“the numpy.distutils maintainers have estimated the impact of this change at 2 years worth of effort, second only to the Python 2/3 migration”) would at least put the whole proposal into some context (and may give the SC pause before approving the PEP…)
3 Likes
Thanks, Paul, it’s useful to have the three faces of distutils enumerated like this. I think the PEP would be improved by including this information, in a “Background” section before “Specification”, and that the “Backwards Compatibility” section would be improved by being updated to indicate how impact on each of the three areas would be addressed, in more detail than there is currently.
The current installation docs refer only to pip, but for a long time even after pip appeared on the scene, people were told to use python setup.py install. There are probably scads of shell scripts around that do that - I know I’ve written several in the past that did just this. The approach was part of the official documentation - note the section title “Distutils: the new standard” 
So it could be argued that though not standardised by any PEP, python setup.py install was the officially recommended approach at one time, and we should at least responsibly consider the impact of breakage of this way of doing things which might be in scripts written long ago that have just kept on working, so were never revisited/revised.
I think you mean distutils here rather than distlib.
I agree with the first statement. As for the second, I don’t see what has been tried even as a thought-experiment to address those problems, so it seems premature to say there’s nothing we can do (anyone please feel free to point to prior discussions of specifically this area). For example, I suggested that there may be mileage in deprecating specific submodules of distutils.command, the distutils compiler classes etc. Specific classes in this area could raise suitable exceptions on instantiation, and functions likewise on call. This seems on the face of it to be an approach that would address the concerns, because it would leave the “library of useful functions” untouched except where they were problematic.
We’ve learned from the Python 2/3 migration that potential breakage needs to be treated sensitively and carefully. As this is an area with comparable impact, I think we should descend into detail below the arm-waving level to make sure that what we do minimises any adverse impact for users.
I’m willing to have a crack at identifying the detailed areas in distutils that could be deprecated, without the nuclear option of deprecating and removing everything in distutils wholesale.
2 Likes
I agree, and if @pf_moore gives the okay, I’ll gladly copy and adapt his post into the PEP (and give coauthorship, if he wants).
Thank you. You’ll no doubt find some hints in the open issues associated with distutils on the bug tracker.
The setup.py interface was always tool specific, it’s just that until another tool (flit) came on the scene, it made no difference. PEP 517 was the way we subsequently standardised the interface. But it’s certainly a very commonly used interface. It’s mostly covered by “switch to setuptools”, though - tools that are consumers of the setup.py interface already don’t work with build backends other than setuptools/distutils, so they are either OK with that or need to make plans to change to PEP 517.
I do. Sincere apologies for that typo.
Absolutely. That’s why I said “maybe” but also why I want the PEP to do more in this area.
You’re welcome to add this (in any form you want) to the PEP - I meant to say so but pushed “send” too soon. I’m not sure I want to be a co-author, though, it looks like the PEP author gets a lot of pushback to deal with 
These assertions about the distutils and setuptools documentation were not true, last time I needed to read them (about six months ago, but I just spot-checked and it doesn’t look like anything has changed since).
The distutils documentation is outdated in places, and has holes when it comes to the “library of useful functions” it provides, but it does at least teach you to package simple modules without assuming you’ve already read some other document on the subject. The setuptools documentation does not come anywhere near this bar. The most significant problem with it is that it is written for an audience of people who have already read and understood the distutils documentation.
For example, compare the “how to write a setup script” section of the distutils documentation with the matching section of the setuptools documentation. When reading both, try as hard as you can to put yourself in the shoes of someone who doesn’t already know how to do packaging. Notice not only how much more detailed the distutils documentation is, but how many times the setuptools documentation assumes you already know something that is only explained in the distutils documentation.
I feel that comprehensive, self-contained documentation for setuptools should be considered a hard requirement for doing anything that would have the side-effect of removing the distutils documentation from docs.python.org or making it harder to find.
2 Likes
@jaraco - thoughts? I don’t want to dictate how you run your project here, but it seems it might be helpful if you wrote up an integration plan that would essentially partner with this PEP.
It’s not dictating how the setuptools project is run to make it a requirement that before we remove the code/docs from the stdlib, a suitable replacement (in our eyes, not in the view of setuptools or any other external project) exists.
Other than the docs (which I’ve never actually read myself, as I’ve always had to resort to the source code anyway), I think setuptools is a suitable replacement. Hence the PEP.
I’m still waiting for someone to point out the APIs that are only available in distutils and that need to be preserved.
All the rest of the pushback has been about having to change imported names, which is kind of the point of deprecating something.
I fully support halting support for distutils as a build system, and sympathize with @pganssle’s comments that “just freezing it” (which would also be my preference as a consumer of distutils) isn’t as simple as it sounds from a maintainer’s perspective. I do think the timeline for removal could be gentler.
I think some of it has been less the fact of it, and more the lack of a destination for what to migrate to.
Adding to the “Library of useful functions” anecdote pile, the main thing I still use distutils for these days is distutils.version since it’s the only version-comparison implementation in the standard library. In fact, I just had a PR held up because of this PEP (it was also easily worked around). If e.g. semver or pep440 or similar landed in the standard library by 3.11, I’d be thrilled. I only today noticed that the contents of distutils.version are undocumented, so I can’t argue too hard against its removal.
I imagine there are other utilities that happen to reside in distutils that could likely find a good home outside distutils, but still inside the stdlib. To me, version string comparison is one of them. So I support expanding the migration plan to cover library functionality, not just
as a build system, and taking specific care to see if any such functionality still belongs in the stdlib, but in a new home. That migration plan doesn’t need to be in this PEP, but I do think it should be a criterion for distutils removal. For my case, you could say pip install semver, but I’d be sad to actually lose version comparison from the Python standard library.
2 Likes
I don’t disagree. But my point is that we should say “we need these docs to have a new home before we remove them”. Just because we think that setuptools is the right replacement, doesn’t mean we can avoid that responsibility by saying “it’s up to setuptools whether they want to”.
And I think you’re being a bit evasive with “I’m still waiting”. I’ve already mentioned distutils’ spawn and find_executable functions, the introspection functions replicated in sysconfig, and the metadata discovery stuff that’s gone to importlib.metadata. Sure, maybe there are others, but starting with these doesn’t seem unreasonable.
(Later) OK, I’ve done some digging into what’s documented and what is not, and was surprised to find just how much of distutils actually is documented.
The distutils API is documented here. The docs don’t say “these functions should not be used”, they just say “we’re only keeping this here until it’s moved to the setuptools docs”. I’ve just read through the sections that you go through on the way to the API docs, and there’s nothing that says these are going away, or that projects shouldn’t use them.
So what I’d suggest is the following:
- The PEP notes that the distutils API (as documented at the above link) has been unmaintained for years, and in many cases better alternatives exist for the functions in the API. The proposal is that the whole API is dropped with the deprecation process already in the PEP. This isn’t saying anything new, but it does explicitly define the scope of the removal.
- The PEP notes explicitly that the intention is that setuptools will provide access to at least a subset of the documented API, with at most import renames being needed. We commit to documenting the migration process (the import renames) - if setuptools don’t deliver a document that we can link to, we will document it in the PEP before the deprecation process begins. (If it’s as simple as renaming imports, that shouldn’t be an onerous requirement, and if it’s not, we should make sure to find out, and rethink the PEP!)
- The PEP clearly states that we don’t require that setuptools support the APIs indefinitely, but that any deprecations/removals after the transition will be subject to the setuptools project’s processes, and will be their responsibility.
- The PEP puts a hard requirement on setuptools to state clearly which, if any, of the documented distutils API functions will not be supported at the point of transition. Given that the process seems to be that setuptools will initially just take a copy of distutils, I assume the answer will be “initially, everything will be available”. But if they want to limit what they make available, we need to know that so we can explicitly document in the PEP that certain APIs will be dropped.
- There is a proper transition plan for the existing documentation. That means the PEP confirms a location in the setuptools documentation that’s clearly intended for the API docs, even if it is initially just “This is a work in progress, but will ultimately contain the information currently at XXX” (with a link to the existing stdlib documentation). The PEP should also note that the API docs will remain available (1) on docs.python.org under Python 3.11, even if it gets dropped from later versions, and (2) in the CPython repo under
Doc\distutilsvia the 3.11 tag. If setuptools aren’t willing to copy over the API documentation, the PEP should be clear that on deprecation, the whole distutils API will become undocumented, and users should raise any concerns about that with the setuptools project.
As a final note, I would acknowledge that this does mean that some projects which use distutils at runtime will need to take a new runtime dependency on setuptools. We are aware that this isn’t necessarily simple, but it’s fundamental to what removing functionality from the stdlib means, and we don’t believe the requirement is unreasonable.
As a final note, if you want to incorporate all of the above into the PEP, then I probably should count as a co-author, so feel free to add me at that point if you want 
Edit: I just checked the setuptools migration documention. Apologies, I should have done that before. If you look at that, there’s a huge number of documented distutils APIs that don’t have replacements. So I think the statement “it’s just renaming imports” is demonstrably false - at least at the moment
2 Likes
That’s a fair point. The canonical library for “packaging related stuff” like version comparison is the packaging library (in the sense that it’s the one that implements the agreed standards). Having that in the stdlib would seem like a reasonable thing, although the library (and standards) is moving a bit too fast in practice for that to work
2 Likes
The following question coincidentally just came up on a pip issue. “Is there a guide for upgrading from distutils to setuptools?”
At the moment I had to say “I don’t know of one”. I don’t think we want to be in that state when (if) this PEP gets approved.
Thanks for the rest of your posts, I want to take the time to absorb them properly before responding.
Long term we have to defer to the setuptools team, but currently the guide is “add import setuptools to the top of your existing script” (and apparently enable the environment variable mentioned in the migration guide, though I had the feeling that only mattered if distutils was imported before setuptools - @jaraco ?).
Edit
That doc is incomplete. I posted it as a suitable home for the list of migrations, rather than a PEP that should be frozen once it is accepted.
Sigh. It’s very hard to discuss a moving target. Let’s stick to my original point in that case, the migration path for all the documented APIs in the existing (legacy) distutils docs needs to be clear.
(Second edit: I’m really messing this up. I think I’ll stop posting for a while now).
That seems like it would be hard to test before distutils is removed, how are users expected to confirm that their change works without distutils in that case? Yes, I know I’m complaining that the migration is “too simple”, and I’m aware of the irony. But leaving people with no way to test that they migrated correctly has its own issues.
I’m absolutely being evasive, because I didn’t sign up to do all that work 
You were even on the thread where I did sign up, so you know this.
Once the deprecation is official, there will be blogs and conference talks about migration (and not just from the “anything-but-setuptools” crowd). Perhaps even just having posted this will start people working on those, but many will wait until it’s official.
Run with warnings enabled during their 3.10 testing.
Again, as you well know, we aren’t deleting the files from everyone’s existing installs tomorrow. This is a staged deprecation following our normal procedures, with the exception that it will have lasted for at least 8 years rather than the usual two. And I think enough time has been wasted during that period trying to adhere to distutil’s interface, quirks and release schedule that could have been spent on building a better packaging ecosystem, so I’m not really excited about extending it further.
1 Like
Re what I said earlier, even if https://docs.python.org/3.x/distributing/index.html continues to exist for many values of x, removal of the “Installing Python Modules” and/or “Distributing Python Modules” top-level headings from https://docs.python.org/3 would qualify as “making the distutils documentation harder to find” in my opinion, and should therefore not happen until someone has written a complete, self-contained replacement. (Ideally, once that happens, https://docs.python.org/3 would link to that documentation.)
I absolutely expect that setuptools would adopt the distutils documentation, probably wholesale to start and then incrementally consolidated. Agreed CPython shouldn’t be hosting this documentation and should be blocked on something adopting the documentation as well.
1 Like
As part of the adoption, Setuptools is aiming to maintain as much compatibility as can reasonably be had for existing packages. New packages should import everything from setuptools and soon (after the Debian/Numpy monkeypatching issues can be resolved) import distutils* should trigger a DeprecationWarning. Setuptools aims to own the build tool functionality of distutils. The ‘utilities’ on the other hand are something that I would encourage or sponsor to be in a new package (possibly ‘packaging’ or ‘distlib’ or similar). Setuptools can commit to supporting this functionality (possibly under the distutils name) until a suitable replacement has been developed.
I’d recommend that the CPython docs link to the Python Packaging User’s Guide and not a specific tool.
1 Like