PEP 632 - Deprecate distutils module

Okay, that’s all I was implying by “compatibility shim”. I just wanted to use a generic term (though possibly it’s a Microsoft/Windows-only term) rather than describe the specific mechanism.


Edit: on rereading the whole paragraph, I don’t really see the need to clarify much. Except maybe swap out “Compatibility shims already exist” for “Code already exists”? Thoughts @pf_moore?

Compatibility shims already exist in setuptools and pip to
transparently migrate old code, however, these are controlled by
external projects and are in no way bound by this PEP. Consult their
latest documentation for migration advice.

Thanks for taking this on Steve. Now for the criticism :stuck_out_tongue:

The biggest issue I have with this PEP is that because it’s tightly scoped to the CPython side, it doesn’t cover almost any of the important details that will be relevant to end users, namely:

  1. When distutils is deprecated, what are users supposed to migrate to?
  2. What is the full roll-out plan for the migration targets?
  3. How should one declare a dependency on distutils?

For example:

setuptools actually only has direct equivalents of a few commands. It doesn’t have anything like distutils.util, distutils.version or distutils.spawn, and adding those things should be out of scope for setuptools.

Right now, what happens is that whenever setuptools is installed in the normal way and the distutils adoption is enabled (done by default in a few versions but apparently reverted in the latest version), it forces import distutils to import the version of distutils vendored into setuptools. Right now, that’s an unmodified version that setuptools then monkey patches (as was the case when it was in the standard library), but I believe the next steps there were to start migrating code from distutils directly into setuptools, so that setuptools no longer actually relies on distutils (and in fact the opposite — distutils would import from setuptools). Once we have migration targets for everything, we’d start raising deprecation warnings whenever import distutils is used, and preferably point to a documentation page about how to migrate away from distutils.

Note: Everything I said above is a huge amount of work, and is much more likely to go smoothly if we can get funding for someone to do it.

Migration path

I think the biggest open question is the migration path. I do not think it would be good if, going forward, everyone started adding setuptools (which is now a catch-all name for "setuptools+ pkg_resources + distutils) as a dependency for runtime stuff like distutils.util.strtobool or distutils.dir_util.create_tree. A few options:

  1. Migrate or identify replacements for everything not going directly into setuptools ahead of time, and aggressively deprecate them.
  2. Create a distutils package on PyPI rather than bundling setuptools and distutils together like that, and have people declare a dependency on distutils if they explicitly need stuff from distutils. We can “upstream” the existing setuptools monkey-patches into this version of distutils, and then fork the desired build code into setuptools (so that setuptools doesn’t have a dependency on the PyPI distutils) — the PyPI version would be deprecated and only maintained for a relatively short period of time, and setuptools would never expose a distutils namespace.
  3. Option 2, but instead of having a single distutils package, we migrate distutils to a namespace package, and have people declare their dependencies more granularly. This would make it easier to identify which distutils submodules need migration and which ones are mainly going unused.

I think Option 3 is a bit overboard, and Option 2 is much better than the current mechanism used for setuptools adopting distutils, and accomplishes basically all the same goals.

1 Like

I’d rather completely omit pip from this. There’s no documentation in pip for users to consult, I would not want people to report any issues to pip, and there’s no user-facing mechanism in pip that users can use to mitigate any issue. Mentioning pip IMO obscures the key point here which is that setuptools is the single point of contact for any issues with using distutils post-removal from the stdlib.

2 Likes

Their choice of build backend.

Out of scope for this PEP, as I think you’ll find that the range of existing build backends prefer to set their own schedules.

[build-system]
requires = ["setuptools"]

:slight_smile:

Thanks for linking to the documentation for these modules… oh wait :slight_smile: They’re not documented (except in a legacy page that says it’s all moving into setuptools). No matter why people are using them, they are not guaranteed public API, and despite that, we’re still offering two deprecated releases before they go away.

We’ve been neglecting/refusing to fix issues in those modules for at least half a decade, and unless someone is stepping up to start taking that job seriously, we’re better off having people migrate.

I’ll also add here that a PEP is not a migration guide. Some of these functions certainly exist elsewhere, such as sys, subprocess, packaging, but it’s not the job of a deprecation PEP to provide every single function with an alternative. The primary use case for distutils as stated in the docs is installing and distributing Python packages, and the recommendation of pip for the former and setuptools for the most-compatible option for the latter is enough.

Or we can normally deprecate them and aggressively find alternatives. I can’t dictate what setuptools should or shouldn’t include, so this sounds like a discussion to take over there.

I’ll point out again that distutils has been not-very-aggressively deprecated for over five years, and likely closer to ten (but that’s before my time).

It’s open source, sure, anyone can do that. If anyone from the core team was willing to maintain distutils in its place, or anyone from the community wanted to join the core team and take ownership of distutils in its place, then we wouldn’t need to remove it. But since nobody wants to do that, I think you’ll be lucky to find someone willing to distribute it separately.

Oh, apart from setuptools :slight_smile:

That’s something to take up with the setuptools team, preferably on a thread in the Packaging category or their issue tracker.

Updates to the PEP text are in https://github.com/python/peps/pull/1582 (for now, will probably merge it by the end of the day)

distutils has a bunch of stuff in it that is not related to your build backend.

I realize that all of this is out of the specified scope of the PEP, but that I’m a strong -1 on this PEP unless the scope is expanded to include a plan for what to do.

Alternatively, I recommend modifying the PEP to no longer mention setuptools, since it’s irrelevant. The extent of the PEP can thus be, “We will deprecate distutils in Python 3.10 and remove it in Python 3.12. All current users must find an alternative.” Personally, I don’t think that’s very satisfying.

I don’t think it’s fair to say, “They’re not documented except for a legacy page saying they’re moving to setuptools” when my point is they are not moving to setuptools. “Setuptools provides the distutils module” is an intermediate step to a better future and I’d consider it a worse and more confusing outcome than the current situation if it happened indefinitely, TBH.

I think by “aggressively deprecate” I meant that we would make it very clear that you should never declare a dependency on setuptools to get access to these functions — they’ll be removed there before they get removed from CPython.

I’m talking about how, structurally, distutils gets maintained. In all the currently-proposed scenarios, we’re looking at distutils coming in via a 3rd party PyPI package. I believe the plan (though it’s not laid out here), is to then deprecate distutils both in CPython and in the 3rd party package, with the goal that in X years, nothing will support import distutils.

And no, random people can’t add a distutils package to PyPI because it’s “Open Source” — or at least they shouldn’t be able to, since standard library module names are reserved to prevent squatting.

In any case, I’m a strong -1 on this PEP as it stands. I think the concerns that led people to say that we should write a PEP for this are completely un-addressed, notably this, which echoes other things I’ve seen:

PEP or not, what we need is an overall migration plan so that we can explain to people why what we’re doing is not some horribly destructive thing for the ecosystem and to give them the ability to plan for future changes. “We’ll be removing distutils using the normal deprecation process, setuptools might support some parts of it I guess, but that’s up to them” is not that. It’s a huge migration likely to impact large swathes of the ecosystem, I think it deserves a design review.

1 Like

So I think two things are being conflated here. The first that distutils is (the core of) a build backend, and the second is that distutils is a grab bag of random functionality. They deserve to be treated separately.

I’m not personally interested in drawing the line, so if someone would like to define which parts of distutils are part of the build backend, which parts are the grab bag, and which parts don’t matter, I’m happy to include that information in the PEP (and make whoever wrote it a co-author).

But I disagree that it has to exist before formally deprecating the module. The motivation to deprecate is strong enough on its own, and three years is a long time.

+1

Either the PEP shouldn’t be mentioning 3rd party tools at all, as they are out of scope, or it should be covering (at least to some extent) how projects should migrate, and what (necessarily 3rd party) tools exist to help with the migration.

If the PEP’s stance is that we’re finally removing distutils and it’s not our problem if people are still using it in spite of years of being told it’s frozen, and we don’t have any recommendations on how people should move off distutils, then it should be up front and say so.

I’d be -1 on a PEP that said that, but at least it would be clear about what it was proposing.

Isn’t it doing the second? And at the request of those third party tools, deferring the details to their documentation rather than trying to capture it directly in the PEP.

Can you provide some (false, made-up) examples of what kind of guidance you want to see in the PEP and why it shouldn’t be in a living document instead?

I was thinking of the “grab bag of random functionality” you mentioned. My understanding is that the setuptools maintainers have said they don’t intend to support that long-term, so telling people how to migrate off that seems within the scope of the PEP.

distutils.spawn.find_executable → shutil.which
distutils.spawn.spawn → subprocess.run

I’m happy for it to be in a “living document” instead. But what document is that? The PEP doesn’t say. Just directing users vaguely at “setuptools” isn’t much help, in practice.

As regards the “build backend functionality”, most of the documentation for how to use that is in the legacy distutils docs - which say

This document is being retained solely until the setuptools documentation at Building and Distributing Packages with Setuptools - setuptools 69.0.3.post20231223 documentation independently covers all of the relevant information currently included here.

But that hasn’t happened yet, so presumably the deprecation would result in users having no documentation for that essential functionality - or is the proposal to keep those legacy docs? How would that work if we remove distutils from the stdlib docs (and so remove the link to the legacy pages)?

Sure, you can say it’s setuptools’ problem that they haven’t migrated the docs yet. But conversely, you can’t hold setuptools up as where people should go once distutils has been dropped if they manifestly are not. A proper transition plan for this documentation should be part of the PEP.

Can you link to where that request was made? My questions may be better addressed to the setuptools maintainers if they really did ask CPython to drop distutils and direct future questions to them. The link should probably be in the PEP, too, as it’s fairly important background. As I’ve already said, I’m much less concerned if the setuptools maintainers have accepted this responsibility (although it doesn’t sound like @pganssle for one is happy with the way the PEP interprets whatever has been said…)

1 Like

This seems like a good place to put (more) of those mappings: https://setuptools.readthedocs.io/en/latest/distutils-legacy.html

I definitely had in-person chats with @jaraco about this, though I’m sure some happened online as well because we’ve been talking about this for years now (since the first PyCaribbean, IIRC?). I can try and dig up some written discussions, but it’s probably easier to get a fresh acknowledgement here in light of the new concerns being raised.

It does, and adding that link to the PEP would seem like an easy way to alleviate at least some of the “where’s the migration advice?” questions.

1 Like

I’ve also just been reminded of this issue, filed by Jason, asking for as quick a deprecation as we can: https://bugs.python.org/issue41282

I’m in favour of this PEP, as you write setuptools has replaced distutils for a significant majority of users (in particular anyone using pip, or even it predecessors). Furthermore development of distutils has stalled, even for bug fixes. Removing distutils is more honest to our users.

I don’t agree with the proposed end state. I’d prefer to remove distutils entirely, not keep it as a private library. This will require more work on the *nix build to replace the logic in setup.py (mostly to the configure script). The Windows port proves this is possible in principle.

I have some reservations/questions/comments about this proposal:

  1. Does this mean everyone is forced to either use setuptools or else vendor distutils? I maintain distlib, which grew out of the aborted packaging effort in Python 3.3 and when distribute and setuptools were still separate, and distribute was the officially blessed installer tool. At the time of setting up distlib, it was meant to provide an alternative, PEP-based path (as opposed to the de facto implementation-based setuptools/distribute, which had no PEPs behind them) to support features relating to package search and installation. Using setuptools in distlib wouldn’t make sense, if it’s meant to be an alternative. While you may want to deprecate distutils for its build-related functionality, distlib still uses code such as distutils.core.Distribution and distutils.config.PyPIRCCommand to base its functionality on. If you deprecate distutils, that’s one thing - but if you removed it entirely, distlib would stop working and there’d be extra work needed to avoid this.

  2. “Historically, setuptools has extended distutils using subclassing and monkeypatching, but has now taken a copy of the underlying code. As a result, the second last major dependency on distutils is gone and there is no need to keep it in the standard library.” That suggests that distutils was only really around to support setuptools, and now that setuptools subsumes it, nobody else needs it. As per point 1 above, that isn’t the case. If you remove it because setuptools doesn’t need it any more, and “most people use setuptools”, that sounds like the tyranny of the majority to me. Are you saying that all third-party tools that may have used distutils for needs beyond just actually installing packages (that “grab-bag of functionality”) don’t matter, and any extra work they need to do just has to be sucked up? This seems to contravene Python’s general stance on backward compatibility.

  3. At the time distlib was set up, the prevailing idea seemed to be “Let’s define how things should work using PEPs. Then there could be multiple implementations which follow the PEPs, and people will choose what works for them, and there can be innovation while preserving interoperability”. Did I misunderstand that, or are we walking that back now? The message now seems to be more like “Let’s just use pip, wheel and setuptools for everything, it’s more convenient that way” - at least for the package installation side of things. As a maintainer who at least tried to create new things in this space, I find this message more than a little disheartening.

1 Like

This is a fun bootstrapping problem. To build CPython setuptools is needed, and to build setuptools Python is needed. Will the CPython source vendor a copy of setuptools for this purpose? setuptools vendors its dependencies, but at one point also chose not to do that. In case setuptools won’t be vendored, can it be guaranteed setuptools won’t choose to unvendor its dependencies again?

Talking about bootstrapping, Python is nowadays used quite early in bootstrapping the build of Linux distributions from source since it’s a build-time dependency of glibc. In Nixpkgs we currently build a minimal Python 3 to deal with this, which does not include a lot of the extension modules. If modules that are included don’t use distutils it is less of a problem.

Without further investigation, unless CPython would embed a copy of setuptools, I consider this a prerequisite for removing distutils from private use in CPython.

My (personal, and interop standards PEP-delegate) view is that we are very much still taking the approach that defining standards and building tools based on those standards is crucial. And very definitely, we should be pushing for innovation rather than a “one tool to rule them all” approach. One frustration, though, is that users don’t really want that - it’s next-to impossible, for example, to get people to accept that using an additional tool is a viable alternative to asking for functionality to be added to pip :slightly_frowning_face:. So for all that we’re trying to create an open platform for tools to innovate, people are voting with their feet.

To bring the comment back to the point, the problem is that distutils doesn’t implement any of those standards. And the lack of maintenance means it probably never will. So I don’t think standards-based interop is an argument against removing distutils. (It may be an argument against assuming that people can use setuptools instead, though).

To address your examples:

  • Standardising an equivalent of the distutils.core.Distribution would allow other tools to implement it. Or maybe the stdlib interface importlib.metadata.Distribution is the new standard (the stdlib doesn’t have to define standards - it is the standard, by definition :slightly_smiling_face:).
  • On the other hand, the .pypirc configuration file has never been standardised, so code using distlib to process it are by default using an implementation defined approach. Unlike importlib.metadata, the distutils interface was never documented, so we can’t say “defined by the stdlib means standard” for this one…

In my view, the transition work needed here would have been to pick out those parts of distutils that could usefully be standardised and do so. Then, new tools could have been written to allow code to migrate away from distutils. Sadly, in spite of years of warnings that distutils isn’t maintained, no-one has done much of that. So in that sense, I support deprecating distutils, because it pushes people to make that effort - split out code into independent libraries, push for standardisation, etc.

Where I disagree with the PEP is in the presumption that the core devs can just “pull the plug” without looking at the impact on the ecosystem. Yes, removing distutils from the stdlib is hard. Yes, putting it on PyPI in some form helps. But I think we owe it to the community to do better than that.

Heck, PEP 594 had a really hard time, to the point where it’s still not agreed. And that was talking about removing libraries which are far less critical to the Python community…

Well, to some extent they’re being pushed in certain directions. This very PEP is pushing people to use setuptools.

I’m not making that argument against removing distutils - it’s more about breaking third-party code such as distlib that uses distutils to build on. I have no problem with deprecating distutils, by which I mean “don’t use distutils for new code unless you know what you’re doing and are prepared to work around bugs which surface, because distutils won’t get any maintenance love”. That’s altogether different to removing code and causing breakage, for no good reason that I can see.

Yes, exactly. Deprecate, by all means, but don’t break existing third-party code for no good reason by removing distutils. Since it isn’t being maintained, it’s not really a burden, other than the disk space it takes up (which is not a big deal, really).

1 Like

As I mentioned on python-dev, if you want to say that the only cost to “leaving distutils in the standard library” is disk space, you need to disable the tests for it so that it can’t break the build or the buildbots, you need to make it so people can’t file issues for distutils in CPython (triage takes time!), and make it so people can’t submit PRs modifying it.

The difference between that “zero cost” scenario and the “deprecate and remove” scenario is that in the “deprecate and remove scenario”, there’s a clear timeline for when distutils breaks. In the “stop touching it or testing it at at all” scenario, different parts of it will break randomly at different times as bitrot sets in (can’t build Windows C extensions anymore because the compiler moved, can’t upload to PyPI anymore because the metadata is not standards-compliant, etc). Eventually Python will change such that import distutils breaks because something distutils is importing has moved or something, and then distutils will just be broken. This will happen at a random, unpredictable time.

It is much better for us to break distutils up and distribute it among the projects that want to maintain the relevant functionality. Already the build backend functionality of distutils is basically maintained in setuptools via monkey patches (many people don’t realize that their code using import distutils would break without running import setuptools, because pip injects it silently and they always use pip). distutils.version is moved to packaging.

I"m -1 on this PEP, but only because it doesn’t address the key issue: a clear path forward that would not be horribly disruptive to the ecosystem. But I’m definitely +1 on a migration out of the standard library.

2 Likes

No, CPython doesn’t currently depend on setuptools to build itself, and it seems unlikely that will even happen. As with the rest of its build, it’ll use custom build scripts, and at worst, will take a private copy of distutils for this purpose.

Agreed. Luckily, this PEP is not proposing to remove it from private use (though a number of other people have proposed migrating CPython’s extension modules to another build system, such as @ronaldoussoren above). What CPython uses for its own build is completely indepenent from this PEP, and it’s only mentioned because people will rightly insist on pointing it out.

You can also use another build backend, such as Flit, Poetry, pymsbuild, enscons, or anything else that is invented. The standards have settled on PEP 517, which is a standard way to declare which build tool you require - distutils is not (and will never be) updated for this standard.

My apologies for missing distlib. I’m happy to give you a mention in the PEP along with numpy.distutils if that will satisfy you?

Also, I suggest looking up the distutils docs, which for the last decade have said that they’re only around for the sake of setuptools. This happened before my time, so it’s the best I have to go on :man_shrugging:

The message is now “use PEP 517 compatible backends”. Sorry you didn’t hear about it, we talked about it quite a lot on distutils-sig and here on Discourse.

setuptools is the most compatible migration target for current users of distutils for packaging purposes. They also publish documentation (currently not very complete) for people who need to migrate to other libraries such as shutil. And as I said above, distutils has been pushing users towards setuptools for years, so this is just the continuation of that.