PEP 632 - Deprecate distutils module

PEPs
21

It does, and adding that link to the PEP would seem like an easy way to alleviate at least some of the “where’s the migration advice?” questions.

1 Like
22

I’ve also just been reminded of this issue, filed by Jason, asking for as quick a deprecation as we can: https://bugs.python.org/issue41282

23

I’m in favour of this PEP, as you write setuptools has replaced distutils for a significant majority of users (in particular anyone using pip, or even it predecessors). Furthermore development of distutils has stalled, even for bug fixes. Removing distutils is more honest to our users.

I don’t agree with the proposed end state. I’d prefer to remove distutils entirely, not keep it as a private library. This will require more work on the *nix build to replace the logic in setup.py (mostly to the configure script). The Windows port proves this is possible in principle.

24

I have some reservations/questions/comments about this proposal:

  1. Does this mean everyone is forced to either use setuptools or else vendor distutils? I maintain distlib, which grew out of the aborted packaging effort in Python 3.3 and when distribute and setuptools were still separate, and distribute was the officially blessed installer tool. At the time of setting up distlib, it was meant to provide an alternative, PEP-based path (as opposed to the de facto implementation-based setuptools/distribute, which had no PEPs behind them) to support features relating to package search and installation. Using setuptools in distlib wouldn’t make sense, if it’s meant to be an alternative. While you may want to deprecate distutils for its build-related functionality, distlib still uses code such as distutils.core.Distribution and distutils.config.PyPIRCCommand to base its functionality on. If you deprecate distutils, that’s one thing - but if you removed it entirely, distlib would stop working and there’d be extra work needed to avoid this.

  2. “Historically, setuptools has extended distutils using subclassing and monkeypatching, but has now taken a copy of the underlying code. As a result, the second last major dependency on distutils is gone and there is no need to keep it in the standard library.” That suggests that distutils was only really around to support setuptools, and now that setuptools subsumes it, nobody else needs it. As per point 1 above, that isn’t the case. If you remove it because setuptools doesn’t need it any more, and “most people use setuptools”, that sounds like the tyranny of the majority to me. Are you saying that all third-party tools that may have used distutils for needs beyond just actually installing packages (that “grab-bag of functionality”) don’t matter, and any extra work they need to do just has to be sucked up? This seems to contravene Python’s general stance on backward compatibility.

  3. At the time distlib was set up, the prevailing idea seemed to be “Let’s define how things should work using PEPs. Then there could be multiple implementations which follow the PEPs, and people will choose what works for them, and there can be innovation while preserving interoperability”. Did I misunderstand that, or are we walking that back now? The message now seems to be more like “Let’s just use pip, wheel and setuptools for everything, it’s more convenient that way” - at least for the package installation side of things. As a maintainer who at least tried to create new things in this space, I find this message more than a little disheartening.

1 Like
25

This is a fun bootstrapping problem. To build CPython setuptools is needed, and to build setuptools Python is needed. Will the CPython source vendor a copy of setuptools for this purpose? setuptools vendors its dependencies, but at one point also chose not to do that. In case setuptools won’t be vendored, can it be guaranteed setuptools won’t choose to unvendor its dependencies again?

Talking about bootstrapping, Python is nowadays used quite early in bootstrapping the build of Linux distributions from source since it’s a build-time dependency of glibc. In Nixpkgs we currently build a minimal Python 3 to deal with this, which does not include a lot of the extension modules. If modules that are included don’t use distutils it is less of a problem.

Without further investigation, unless CPython would embed a copy of setuptools, I consider this a prerequisite for removing distutils from private use in CPython.

26

My (personal, and interop standards PEP-delegate) view is that we are very much still taking the approach that defining standards and building tools based on those standards is crucial. And very definitely, we should be pushing for innovation rather than a “one tool to rule them all” approach. One frustration, though, is that users don’t really want that - it’s next-to impossible, for example, to get people to accept that using an additional tool is a viable alternative to asking for functionality to be added to pip :slightly_frowning_face:. So for all that we’re trying to create an open platform for tools to innovate, people are voting with their feet.

To bring the comment back to the point, the problem is that distutils doesn’t implement any of those standards. And the lack of maintenance means it probably never will. So I don’t think standards-based interop is an argument against removing distutils. (It may be an argument against assuming that people can use setuptools instead, though).

To address your examples:

  • Standardising an equivalent of the distutils.core.Distribution would allow other tools to implement it. Or maybe the stdlib interface importlib.metadata.Distribution is the new standard (the stdlib doesn’t have to define standards - it is the standard, by definition :slightly_smiling_face:).
  • On the other hand, the .pypirc configuration file has never been standardised, so code using distlib to process it are by default using an implementation defined approach. Unlike importlib.metadata, the distutils interface was never documented, so we can’t say “defined by the stdlib means standard” for this one…

In my view, the transition work needed here would have been to pick out those parts of distutils that could usefully be standardised and do so. Then, new tools could have been written to allow code to migrate away from distutils. Sadly, in spite of years of warnings that distutils isn’t maintained, no-one has done much of that. So in that sense, I support deprecating distutils, because it pushes people to make that effort - split out code into independent libraries, push for standardisation, etc.

Where I disagree with the PEP is in the presumption that the core devs can just “pull the plug” without looking at the impact on the ecosystem. Yes, removing distutils from the stdlib is hard. Yes, putting it on PyPI in some form helps. But I think we owe it to the community to do better than that.

Heck, PEP 594 had a really hard time, to the point where it’s still not agreed. And that was talking about removing libraries which are far less critical to the Python community…

27

Well, to some extent they’re being pushed in certain directions. This very PEP is pushing people to use setuptools.

I’m not making that argument against removing distutils - it’s more about breaking third-party code such as distlib that uses distutils to build on. I have no problem with deprecating distutils, by which I mean “don’t use distutils for new code unless you know what you’re doing and are prepared to work around bugs which surface, because distutils won’t get any maintenance love”. That’s altogether different to removing code and causing breakage, for no good reason that I can see.

Yes, exactly. Deprecate, by all means, but don’t break existing third-party code for no good reason by removing distutils. Since it isn’t being maintained, it’s not really a burden, other than the disk space it takes up (which is not a big deal, really).

1 Like
28

As I mentioned on python-dev, if you want to say that the only cost to “leaving distutils in the standard library” is disk space, you need to disable the tests for it so that it can’t break the build or the buildbots, you need to make it so people can’t file issues for distutils in CPython (triage takes time!), and make it so people can’t submit PRs modifying it.

The difference between that “zero cost” scenario and the “deprecate and remove” scenario is that in the “deprecate and remove scenario”, there’s a clear timeline for when distutils breaks. In the “stop touching it or testing it at at all” scenario, different parts of it will break randomly at different times as bitrot sets in (can’t build Windows C extensions anymore because the compiler moved, can’t upload to PyPI anymore because the metadata is not standards-compliant, etc). Eventually Python will change such that import distutils breaks because something distutils is importing has moved or something, and then distutils will just be broken. This will happen at a random, unpredictable time.

It is much better for us to break distutils up and distribute it among the projects that want to maintain the relevant functionality. Already the build backend functionality of distutils is basically maintained in setuptools via monkey patches (many people don’t realize that their code using import distutils would break without running import setuptools, because pip injects it silently and they always use pip). distutils.version is moved to packaging.

I"m -1 on this PEP, but only because it doesn’t address the key issue: a clear path forward that would not be horribly disruptive to the ecosystem. But I’m definitely +1 on a migration out of the standard library.

2 Likes
29

No, CPython doesn’t currently depend on setuptools to build itself, and it seems unlikely that will even happen. As with the rest of its build, it’ll use custom build scripts, and at worst, will take a private copy of distutils for this purpose.

Agreed. Luckily, this PEP is not proposing to remove it from private use (though a number of other people have proposed migrating CPython’s extension modules to another build system, such as @ronaldoussoren above). What CPython uses for its own build is completely indepenent from this PEP, and it’s only mentioned because people will rightly insist on pointing it out.

You can also use another build backend, such as Flit, Poetry, pymsbuild, enscons, or anything else that is invented. The standards have settled on PEP 517, which is a standard way to declare which build tool you require - distutils is not (and will never be) updated for this standard.

My apologies for missing distlib. I’m happy to give you a mention in the PEP along with numpy.distutils if that will satisfy you?

Also, I suggest looking up the distutils docs, which for the last decade have said that they’re only around for the sake of setuptools. This happened before my time, so it’s the best I have to go on :man_shrugging:

The message is now “use PEP 517 compatible backends”. Sorry you didn’t hear about it, we talked about it quite a lot on distutils-sig and here on Discourse.

setuptools is the most compatible migration target for current users of distutils for packaging purposes. They also publish documentation (currently not very complete) for people who need to migrate to other libraries such as shutil. And as I said above, distutils has been pushing users towards setuptools for years, so this is just the continuation of that.

30

You seem to be treating distutils as only a build system, but it’s not. The usage of distutils by distlib doesn’t relate to building extensions, and so your reference to alternative build backends doesn’t apply to it. Is it that you haven’t considered the other uses of distutils? It appears not, from your suggested migration path:

Code that imports distutils will no longer work from Python 3.12.

The suggested migration path is to use the equivalent (though not
identical) imports from setuptools (see [5]), or to migrate to an
alternative build backend (see PEP 517 [4]).

If you haven’t considered these other usages of distutils, I respectfully suggest that this PEP is incomplete, and perhaps premature because not enough thought has gone into downsides of deprecation/removal.

But by saying you want to deprecate and remove all of distutils, aren’t you the one doing the conflating, by treating the two things as a unit?

Why is it just a matter of mentioning it in the PEP? Is it OK to take the view that “as long as the breakage to come is documented somewhere, that’s all we need to do”? That seems to be what you’re saying, but tell me if I’m misinterpreting. I care about unnecessary breakage of existing code, not in distlib getting a mention in the PEP (though it could be mentioned, by virtue of being a library which uses distutils that’s vendored by pip and virtualenv, and not just to satisfy me :slight_smile:).

When you say

I’m not personally interested in drawing the line

That’s fair enough as far as stating what you care and don’t care about, but that doesn’t mean that non-buildsystem usages of distutils should be ridden-over roughshod just because you, and no doubt some other people, don’t care.

I appreciate that I was a bit simplistic in saying that disk space was the only cost of keeping distutils around - thanks @pganssle for the reminder that there’s more to it. :flushed: :grimacing:

I agree - I too think the PEP needs more fleshing out about how disruption would be minimized, about separating the non-build from the build-parts in the discussion, etc. Surely the onus is on those who want to change the status quo to make a detailed proposal showing how disruption could be minimized for all stakeholders?

2 Likes
31

Guess I was confused by the docs:

The distutils package provides support for building and installing additional modules into a Python installation.

If it truly is a package of useful other functionality, it’s a shame it’s not documented. Hopefully now that there is some enthusiasm for this functionality, someone will step up to document, support and maintain it?

32

Hi, thanks for working on this PEP Steve. I’m supportive of the goal of this PEP - it has to happen at some point. The text seems to gloss over some not-so-minor details though, so I hope the PEP can be updated with a more accurate description of the needed effort and backwards compatibility impact.

You do give numpy.distutils a mention, but I’d like to point out that it is a major dependency on distutils:

And it is in turn depended on by SciPy, scikit-learn, scikit-image, statsmodels, and many other projects.

The discussion so far has probably already shown that this isn’t the case. From my perspective as a NumPy and SciPy maintainer, the amount of work this will cause is huge - second only to the Python 2->3 migration.

It’s an estimated 6-9 months of work for setuptools to complete incorporating distutils and exposing its functionality as setuptools APIs. Then it will take at least that same amount of time for numpy.distutils functionality to be updated for that and split off as a separate package or merged into setuptools as well. And after that the downstream users of numpy.distutils need updating. So that’s probably ~2 years of work before this whole exercise is over.

2 Likes
33

The point is not to make it sound like people should try and switch to it. Unlike setuptools, which has always explicit been trying to be a substitute for distutils, numpy has not and is not.

Already I’ve been asked to “list more projects” and also “oh, but not mine please”, so it’s really not clear what people expect to see here. What I do know is that basically any specifics in the PEP itself will be outdated almost immediately, and certainly by the time distutils is actually removed from the stdlib.

Nobody in the discussion has suggested that setup.py files will break, or that the PEP is responsible for those. Only that an unspecified set of helper functions do not have immediately obvious alternatives.

Luckily the deprecation period will last approximately three years, so thank you for confirming that it’s long enough :smiling_face: (don’t worry, I won’t quote you in the PEP on that, since I know that’s not what you were trying to suggest).

Ultimately, I volunteered to write a PEP that essentially formalises the status quo for most of the last decade, which “everyone” knew was necessary, but nobody had done officially. I know a lot of the trouble you’ve already dealt with in numpy trying to play nice with distutils, and this is apparently the only way for us to officially say “it’s okay, you don’t have to use us anymore” (in a way that’s also convenient for you to point your own users at). The same applies for setuptools. Your two projects know better than most how hard it is to work against the “standard”, so hopefully this clarifies that you’re welcome to do whatever you need to enable your users and ignore “what upstream thinks” here.

34

Well, in that case: setup.py files will break, in multiple ways and with obscure errors.

I don’t care much about the specifics, I’m happy to work out the plan together with setuptools and other projects as we go. I just want the PEP to be somewhat reflective of reality, because this will be the largest backwards compat break in a long time.

35

Well, any project that doesn’t use setuptools but does use distutils will have code in its setup.py like

from distutils.core import setup
# other stuff
setup(...)

Does it need explicit spelling out that if the import fails because distutils has been removed, any such setup.py file will break?

36

When invoked via pip, it will be subject to their injection of setuptools (until the project specifies a build backend, which can’t be distutils anyway) and so will work for as long as setuptools provides a shim. But the PEP cannot dictate how setuptools behaves.

It’s already spelled out that direct imports of distutils will fail. That’s normal for API changes and it’s what the deprecation period is for.

That’s what happens today anyway, and was the motivation for PEP 517. This change will make them break in a single consistent way (ModuleNotFoundError) from a known Python version (3.12) in a known and well advertised schedule (approx 3 years). That seems like an improvement.

37

But that’s not the only way of installing, even if it is the most common. What about python setup.py install?

Things look OK now, but in the past it’s been a wild ride :wink:, and there’s no telling what will happen in the future. It seems reasonable that there should be some minimalist way of installing simple Python packages without mandating the usage of a third party tool, however ubiquitous.

That’s build functionality again (I’m not arguing about the brokenness of distutils build functionality, or the fact that its original design was problematic - just trying to avoid unnecessary breakage due to throwing the baby out with the bathwater).

Other approaches are possible to at least consider. For example, fail consistently only when certain problematic constructs within distutils are accessed, such as parts of Lib/distutils/command and distutils/XXXcompiler.py. Perhaps via a NotImplementedError with a message indicating workarounds.

38

They’ve been considered but would require more maintenance work than we have available for distutils.

As I said earlier, you’re welcome to take it on. Given a dedicated maintainer I’m happy to withdraw the PEP (until that maintainer disappears on us).

39

By whom? Is there discussion somewhere which goes into detail?

Is the suggestion that if someone doesn’t step up and take ownership of all of distutils, it has to all be removed, potentially breaking lots of third-party code? No ifs, no buts? Bit of a Hobson’s choice. Did something happen recently to precipitate such a drastic move?

I’m just trying to get clarity about the next steps here. The PEP has been drafted, and concerns have been raised about unnecessary breakage of third-party code if it’s implemented as is. Is it not primarily the responsibility of the PEP author, as the one proposing the change, to take ownership of addressing those concerns?

40

At least by the PEP author, since I seem to have the responsibility for these things :slight_smile: But in reality, they’ve been discussed frequently over the last decade in various conversations, both online and in person.

I wasn’t around for most of them, but the summary was captured in the official distutils documentation in 2014 or earlier, which says please don’t use it and use setuptools instead.

Basically, yes. As long as the import succeeds, people will expect every part of it to work.

Deprecations/removals always break third-party code that is using the deprecated functionality, and there are plenty of options for that code, ranging from not updating to Python 3.12 through to migrating to alternate functionality.

Which next steps? If you’re using distutils for packaging, switch to setuptools (for closest compatibility) or another backend.

If you’re using distutils for something else, at least tell me what so I can give you a specific answer. Otherwise my answer remains “follow setuptools’s guide for migrating to things that setuptools does not offer”.

1 Like