PEP 632 - Deprecate distutils module

The point is not to make it sound like people should try and switch to it. Unlike setuptools, which has always explicit been trying to be a substitute for distutils, numpy has not and is not.

Already I’ve been asked to “list more projects” and also “oh, but not mine please”, so it’s really not clear what people expect to see here. What I do know is that basically any specifics in the PEP itself will be outdated almost immediately, and certainly by the time distutils is actually removed from the stdlib.

Nobody in the discussion has suggested that setup.py files will break, or that the PEP is responsible for those. Only that an unspecified set of helper functions do not have immediately obvious alternatives.

Luckily the deprecation period will last approximately three years, so thank you for confirming that it’s long enough :slight_smile: (don’t worry, I won’t quote you in the PEP on that, since I know that’s not what you were trying to suggest).

Ultimately, I volunteered to write a PEP that essentially formalises the status quo for most of the last decade, which “everyone” knew was necessary, but nobody had done officially. I know a lot of the trouble you’ve already dealt with in numpy trying to play nice with distutils, and this is apparently the only way for us to officially say “it’s okay, you don’t have to use us anymore” (in a way that’s also convenient for you to point your own users at). The same applies for setuptools. Your two projects know better than most how hard it is to work against the “standard”, so hopefully this clarifies that you’re welcome to do whatever you need to enable your users and ignore “what upstream thinks” here.

Well, in that case: setup.py files will break, in multiple ways and with obscure errors.

I don’t care much about the specifics, I’m happy to work out the plan together with setuptools and other projects as we go. I just want the PEP to be somewhat reflective of reality, because this will be the largest backwards compat break in a long time.

Well, any project that doesn’t use setuptools but does use distutils will have code in its setup.py like

from distutils.core import setup
# other stuff
setup(...)

Does it need explicit spelling out that if the import fails because distutils has been removed, any such setup.py file will break?

When invoked via pip, it will be subject to their injection of setuptools (until the project specifies a build backend, which can’t be distutils anyway) and so will work for as long as setuptools provides a shim. But the PEP cannot dictate how setuptools behaves.

It’s already spelled out that direct imports of distutils will fail. That’s normal for API changes and it’s what the deprecation period is for.

That’s what happens today anyway, and was the motivation for PEP 517. This change will make them break in a single consistent way (ModuleNotFoundError) from a known Python version (3.12) in a known and well advertised schedule (approx 3 years). That seems like an improvement.

But that’s not the only way of installing, even if it is the most common. What about python setup.py install?

Things look OK now, but in the past it’s been a wild ride :wink:, and there’s no telling what will happen in the future. It seems reasonable that there should be some minimalist way of installing simple Python packages without mandating the usage of a third party tool, however ubiquitous.

That’s build functionality again (I’m not arguing about the brokenness of distutils build functionality, or the fact that its original design was problematic - just trying to avoid unnecessary breakage due to throwing the baby out with the bathwater).

Other approaches are possible to at least consider. For example, fail consistently only when certain problematic constructs within distutils are accessed, such as parts of Lib/distutils/command and distutils/XXXcompiler.py. Perhaps via a NotImplementedError with a message indicating workarounds.

They’ve been considered but would require more maintenance work than we have available for distutils.

As I said earlier, you’re welcome to take it on. Given a dedicated maintainer I’m happy to withdraw the PEP (until that maintainer disappears on us).

By whom? Is there discussion somewhere which goes into detail?

Is the suggestion that if someone doesn’t step up and take ownership of all of distutils, it has to all be removed, potentially breaking lots of third-party code? No ifs, no buts? Bit of a Hobson’s choice. Did something happen recently to precipitate such a drastic move?

I’m just trying to get clarity about the next steps here. The PEP has been drafted, and concerns have been raised about unnecessary breakage of third-party code if it’s implemented as is. Is it not primarily the responsibility of the PEP author, as the one proposing the change, to take ownership of addressing those concerns?

At least by the PEP author, since I seem to have the responsibility for these things :slight_smile: But in reality, they’ve been discussed frequently over the last decade in various conversations, both online and in person.

I wasn’t around for most of them, but the summary was captured in the official distutils documentation in 2014 or earlier, which says please don’t use it and use setuptools instead.

Basically, yes. As long as the import succeeds, people will expect every part of it to work.

Deprecations/removals always break third-party code that is using the deprecated functionality, and there are plenty of options for that code, ranging from not updating to Python 3.12 through to migrating to alternate functionality.

Which next steps? If you’re using distutils for packaging, switch to setuptools (for closest compatibility) or another backend.

If you’re using distutils for something else, at least tell me what so I can give you a specific answer. Otherwise my answer remains “follow setuptools’s guide for migrating to things that setuptools does not offer”.

This is good territory for a competing PEP though. I think it’s a bad proposal, but at least writing it up will let the steering council (or their nominee) make the decision.

I think we should probably be very clear here what all of the component parts involved do. I suspect many core Python developers don’t really have a good understanding of the packaging landscape, and nor should they. So please take this as some background on the whole “pip/setuptools/packaging standards” area as it relates to this PEP.

Distutils basically has 3 functions - a “build backend” (of sorts), a command line tool (setup.py install etc) and a library containing various “utilities” related to packaging.

Build Backend

It is not mandatory to install projects via pip. Pretty much all of the standards work we have been doing in packaging is to enable people to not use pip if they don’t want to. But at the present time, there aren’t many installers other than pip (and many things that look like they install stuff use pip behind the scenes). That’s not an intention, though, it’s mostly just a lack of anyone being willing to do the work involved (a common story in the world of packaging :slightly_frowning_face:)

PEP 517 allows people to write “build backends” - tools that do the job of building a project - in a way that means they can be used by any installer. Distutils is not, and never will be, a build backend (because its maintainers don’t want to make it one, by virtue of the fact that there are no maintainers…) Setuptools is a PEP 517 build backend, and it strives to be compatible with distutils. It is extremely compatible, due to basically using distutils code and monkeypatching it, but it is not distutils. Switching to setuptools from distutils as a build backend should be as simple as replacing imports. If it’s not, then the setuptools maintainers may be willing to help improve compatibility. Or they might not - distutils never documented its API, so any replacement like setuptools has to draw the line somewhere. But in general, setuptools is sufficiently compatible for almost all uses of distutils as a build backend (yes, I’m repeating myself - it’s an important point!)

Command line interface

The command line interface of distutils (setup.py install etc) is not a standard. Build tools do not have to follow it (and flit, for example, doesn’t). Setuptools does use the same command line, but the setuptools developers have indicated they would like to deprecate it (I don’t have a link, sorry, but I doubt this is immiment, it’s more of a long term aspiration, I believe). And pip (for example) is moving away from relying on the setup.py interface in favour of PEP 517.

If you want to build wheels without using setup.py bdist_wheel, the new pybuild tool will be available soon to let you do so. (Mind you, distutils doesn’t support building wheels, so that’s somewhat irrelevant). If you want to install a project, distlib is an alternative wheel installer if you don’t like to use pip. And others are being developed. There is no standards-backed way to install a project without building an intermediate wheel, that’s true, but given that switching a project to use setuptools as a build backend should be straightforward, that’s not particularly relevant.

Alternatively, individual tools may choose to provide a tool-specific command line interface (flit has flit install I believe).

Library of useful functions

This is likely the biggest area where people are concerned. There are (were) a lot of pieces of functionality in distutils, related to tasks that are relevant to building projects. None of these were ever well-documented - many were not documented at all - but people used them nevertheless. Over time, equivalents for a lot of these APIs ended up in other parts of the stdlib (sysconfig, subprocess, shutil, importlib.metadata, …) and others ended up in 3rd party libraries like packaging. But not all did, and not everyone knew that the functions had moved.

Most of the “blame” for this could be put on the lack of anyone willing to maintain distlib. Ideally, functions with better alternatives could have been properly deprecated and documented as “if you are using X, you should look at moving to Y, which is a supported alternative”. But that never happened, and instead a broad “you shouldn’t use any of this” note was plastered over everything - which unfortunately looked like it mostly meant the “build backend” part of distutils, so didn’t actually catch the intended audience very well.

I don’t think anyone knows how much code still exists that uses distutils functions which have better equivalents - let alone code using functions that have no viable alternative right now. The deprecation warning will presumably flush them out. But at the moment, it’s a huge unknown, although we’re already hearing from people flagging up that they have issues here.

IMO, at the moment, the PEP is largely ignoring the potential for problems that may exist in this area. Maybe there’s nothing we can do to address those problems. But acknowledging them (“the numpy.distutils maintainers have estimated the impact of this change at 2 years worth of effort, second only to the Python 2/3 migration”) would at least put the whole proposal into some context (and may give the SC pause before approving the PEP…)

3 Likes

Thanks, Paul, it’s useful to have the three faces of distutils enumerated like this. I think the PEP would be improved by including this information, in a “Background” section before “Specification”, and that the “Backwards Compatibility” section would be improved by being updated to indicate how impact on each of the three areas would be addressed, in more detail than there is currently.

The current installation docs refer only to pip, but for a long time even after pip appeared on the scene, people were told to use python setup.py install. There are probably scads of shell scripts around that do that - I know I’ve written several in the past that did just this. The approach was part of the official documentation - note the section title “Distutils: the new standard” :laughing:

So it could be argued that though not standardised by any PEP, python setup.py install was the officially recommended approach at one time, and we should at least responsibly consider the impact of breakage of this way of doing things which might be in scripts written long ago that have just kept on working, so were never revisited/revised.

I think you mean distutils here rather than distlib.

I agree with the first statement. As for the second, I don’t see what has been tried even as a thought-experiment to address those problems, so it seems premature to say there’s nothing we can do (anyone please feel free to point to prior discussions of specifically this area). For example, I suggested that there may be mileage in deprecating specific submodules of distutils.command, the distutils compiler classes etc. Specific classes in this area could raise suitable exceptions on instantiation, and functions likewise on call. This seems on the face of it to be an approach that would address the concerns, because it would leave the “library of useful functions” untouched except where they were problematic.

We’ve learned from the Python 2/3 migration that potential breakage needs to be treated sensitively and carefully. As this is an area with comparable impact, I think we should descend into detail below the arm-waving level to make sure that what we do minimises any adverse impact for users.

I’m willing to have a crack at identifying the detailed areas in distutils that could be deprecated, without the nuclear option of deprecating and removing everything in distutils wholesale.

2 Likes

I agree, and if @pf_moore gives the okay, I’ll gladly copy and adapt his post into the PEP (and give coauthorship, if he wants).

Thank you. You’ll no doubt find some hints in the open issues associated with distutils on the bug tracker.

The setup.py interface was always tool specific, it’s just that until another tool (flit) came on the scene, it made no difference. PEP 517 was the way we subsequently standardised the interface. But it’s certainly a very commonly used interface. It’s mostly covered by “switch to setuptools”, though - tools that are consumers of the setup.py interface already don’t work with build backends other than setuptools/distutils, so they are either OK with that or need to make plans to change to PEP 517.

I do. Sincere apologies for that typo.

Absolutely. That’s why I said “maybe” but also why I want the PEP to do more in this area.

You’re welcome to add this (in any form you want) to the PEP - I meant to say so but pushed “send” too soon. I’m not sure I want to be a co-author, though, it looks like the PEP author gets a lot of pushback to deal with :wink:

These assertions about the distutils and setuptools documentation were not true, last time I needed to read them (about six months ago, but I just spot-checked and it doesn’t look like anything has changed since).

The distutils documentation is outdated in places, and has holes when it comes to the “library of useful functions” it provides, but it does at least teach you to package simple modules without assuming you’ve already read some other document on the subject. The setuptools documentation does not come anywhere near this bar. The most significant problem with it is that it is written for an audience of people who have already read and understood the distutils documentation.

For example, compare the “how to write a setup script” section of the distutils documentation with the matching section of the setuptools documentation. When reading both, try as hard as you can to put yourself in the shoes of someone who doesn’t already know how to do packaging. Notice not only how much more detailed the distutils documentation is, but how many times the setuptools documentation assumes you already know something that is only explained in the distutils documentation.

I feel that comprehensive, self-contained documentation for setuptools should be considered a hard requirement for doing anything that would have the side-effect of removing the distutils documentation from docs.python.org or making it harder to find.

2 Likes

@jaraco - thoughts? I don’t want to dictate how you run your project here, but it seems it might be helpful if you wrote up an integration plan that would essentially partner with this PEP.

It’s not dictating how the setuptools project is run to make it a requirement that before we remove the code/docs from the stdlib, a suitable replacement (in our eyes, not in the view of setuptools or any other external project) exists.

Other than the docs (which I’ve never actually read myself, as I’ve always had to resort to the source code anyway), I think setuptools is a suitable replacement. Hence the PEP.

I’m still waiting for someone to point out the APIs that are only available in distutils and that need to be preserved.

All the rest of the pushback has been about having to change imported names, which is kind of the point of deprecating something.

I fully support halting support for distutils as a build system, and sympathize with @pganssle’s comments that “just freezing it” (which would also be my preference as a consumer of distutils) isn’t as simple as it sounds from a maintainer’s perspective. I do think the timeline for removal could be gentler.

I think some of it has been less the fact of it, and more the lack of a destination for what to migrate to.

Adding to the “Library of useful functions” anecdote pile, the main thing I still use distutils for these days is distutils.version since it’s the only version-comparison implementation in the standard library. In fact, I just had a PR held up because of this PEP (it was also easily worked around). If e.g. semver or pep440 or similar landed in the standard library by 3.11, I’d be thrilled. I only today noticed that the contents of distutils.version are undocumented, so I can’t argue too hard against its removal.

I imagine there are other utilities that happen to reside in distutils that could likely find a good home outside distutils, but still inside the stdlib. To me, version string comparison is one of them. So I support expanding the migration plan to cover library functionality, not just
as a build system, and taking specific care to see if any such functionality still belongs in the stdlib, but in a new home. That migration plan doesn’t need to be in this PEP, but I do think it should be a criterion for distutils removal. For my case, you could say pip install semver, but I’d be sad to actually lose version comparison from the Python standard library.

2 Likes

I don’t disagree. But my point is that we should say “we need these docs to have a new home before we remove them”. Just because we think that setuptools is the right replacement, doesn’t mean we can avoid that responsibility by saying “it’s up to setuptools whether they want to”.

And I think you’re being a bit evasive with “I’m still waiting”. I’ve already mentioned distutils’ spawn and find_executable functions, the introspection functions replicated in sysconfig, and the metadata discovery stuff that’s gone to importlib.metadata. Sure, maybe there are others, but starting with these doesn’t seem unreasonable.

(Later) OK, I’ve done some digging into what’s documented and what is not, and was surprised to find just how much of distutils actually is documented.

The distutils API is documented here. The docs don’t say “these functions should not be used”, they just say “we’re only keeping this here until it’s moved to the setuptools docs”. I’ve just read through the sections that you go through on the way to the API docs, and there’s nothing that says these are going away, or that projects shouldn’t use them.

So what I’d suggest is the following:

  1. The PEP notes that the distutils API (as documented at the above link) has been unmaintained for years, and in many cases better alternatives exist for the functions in the API. The proposal is that the whole API is dropped with the deprecation process already in the PEP. This isn’t saying anything new, but it does explicitly define the scope of the removal.
  2. The PEP notes explicitly that the intention is that setuptools will provide access to at least a subset of the documented API, with at most import renames being needed. We commit to documenting the migration process (the import renames) - if setuptools don’t deliver a document that we can link to, we will document it in the PEP before the deprecation process begins. (If it’s as simple as renaming imports, that shouldn’t be an onerous requirement, and if it’s not, we should make sure to find out, and rethink the PEP!)
  3. The PEP clearly states that we don’t require that setuptools support the APIs indefinitely, but that any deprecations/removals after the transition will be subject to the setuptools project’s processes, and will be their responsibility.
  4. The PEP puts a hard requirement on setuptools to state clearly which, if any, of the documented distutils API functions will not be supported at the point of transition. Given that the process seems to be that setuptools will initially just take a copy of distutils, I assume the answer will be “initially, everything will be available”. But if they want to limit what they make available, we need to know that so we can explicitly document in the PEP that certain APIs will be dropped.
  5. There is a proper transition plan for the existing documentation. That means the PEP confirms a location in the setuptools documentation that’s clearly intended for the API docs, even if it is initially just “This is a work in progress, but will ultimately contain the information currently at XXX” (with a link to the existing stdlib documentation). The PEP should also note that the API docs will remain available (1) on docs.python.org under Python 3.11, even if it gets dropped from later versions, and (2) in the CPython repo under Doc\distutils via the 3.11 tag. If setuptools aren’t willing to copy over the API documentation, the PEP should be clear that on deprecation, the whole distutils API will become undocumented, and users should raise any concerns about that with the setuptools project.

As a final note, I would acknowledge that this does mean that some projects which use distutils at runtime will need to take a new runtime dependency on setuptools. We are aware that this isn’t necessarily simple, but it’s fundamental to what removing functionality from the stdlib means, and we don’t believe the requirement is unreasonable.

As a final note, if you want to incorporate all of the above into the PEP, then I probably should count as a co-author, so feel free to add me at that point if you want :slightly_smiling_face:

Edit: I just checked the setuptools migration documention. Apologies, I should have done that before. If you look at that, there’s a huge number of documented distutils APIs that don’t have replacements. So I think the statement “it’s just renaming imports” is demonstrably false - at least at the moment :slightly_frowning_face:

2 Likes

That’s a fair point. The canonical library for “packaging related stuff” like version comparison is the packaging library (in the sense that it’s the one that implements the agreed standards). Having that in the stdlib would seem like a reasonable thing, although the library (and standards) is moving a bit too fast in practice for that to work :slightly_frowning_face:

2 Likes