PEP 632 - Deprecate distutils module

PEPs
62

Why not keep shipping setuptools “pre-installed” with CPython indefinitely, or at least split off the "import distutils fails" step for a separate future PEP, once setuptools has taken over maintenance and we’ve seen how it goes? I’m not a fan of the distutils API, but making import distutils fail is going to be a difficult and disruptive transition for the ecosystem, and it’s not necessary to get most of the benefits here.

63

Do you have a proposed timeline? And why is it longer than three years?

So far, setuptools has had a rough time of the transition, mostly because distutils is still there and they don’t have the legitimacy to tell people that the transition is deliberate (rather than just them trying to steal users from the standard library). Even just by existing, this proposal will help, but it’ll help even more if there’s a definite timeline attached to it.

64

I’m suggesting you tweak the target for this PEP to:

  • distutils becomes part of the setuptools project, and is removed from the stdlib
  • BUT setuptools is pre-installed as part of the default CPython install, so import distutils continues to work, it just gives you the version maintained by the setuptools project
  • if specific users want to slim down their installs by removing distutils/setuptools, they can do a regular pip uninstall
  • pip install -U setuptools gives you the latest version of distutils
  • packages can now declare a build-requires on a specific version of setuptools+distutils which gets installed into their isolated build env, and temporarily overrides the version that shipped with CPython
1 Like
65

I’m worried that setuptools’ CI covers much less than CPython’s. If third-party code is pre-installed with CPython, it should also be tested as part of CPython – on all the buildbots – and bugs in it should be release blockers.

66

How do we handle that for pip right now?

67

Right now it gets tested very lightly in test_ensurepip and test_venv, and because it has bugs it is currently marked as a release blocker. (Discussion to https://bugs.python.org/issue41490 please.)

The test coverage isn’t great though. We basically rely on the pip team to have it covered, and assume/hope that their testing includes the important cases for setuptools (which is only included as a “practically essential dependency” of pip, though as PEP 517 use increases that dependency is reducing in importance).

68

Does it make sense to adopt it now? That way what you’ve got is in-sync with the copy of distutils.

If all your setuptools time is going towards making sure it all plays nicely, then keep doing that please :slight_smile: But I assume it would help your case somewhat to have setuptools be the “official” home of distutils?

69

My concern with this approach is that it’s far too much about providing user guidance and not documenting the rationale for the change.

I agree that all of this should appear somewhere, but I don’t think the PEP is the right place. And (quite rightly), nobody wants to preemptively write up migration guides when there appears to be such doubt over whether the migration will even occur :slight_smile:

70

I’m not suggesting a migration guide – the extra bullet points were intended to provide rationale / help visualize the technical change I was suggesting.

The core suggestion is that you focus on moving distutils from being developed as part of the stdlib -> being developed as part of setuptools, and not deprecate distutils or make breaking changes. (And if there are no breaking changes, there’s no need for migration guides!)

71

I think it’s exactly the right place, because it is an amended approach to achieving the goals set out in the PEP. It shows that the deprecation goal could potentially be achieved without breakage of existing third-party code. The PEP as it stands covers the first bullet point in the list provided by @njs - but aren’t all the other bullet points positives to aim for, too? Given that the setuptools team have already decided to adopt distutils, it doesn’t seem to involve any real additional work for the core team beyond what’s in the first bullet point. I’m not sure how much additional work would be required by the setuptools team to do this, but they already monkey-patch distutils and the change to make import distutils work by pointing to their internal copy seems to be at the same level of work (no doubt a setuptools maintainer will pitch in if I’m wrong about that).

72

This was already the plan for the distutils adoption, and it already works if you set an environment variable. It tends to cause widespread breakage whenever we actually turn it on, but parts of that could have been fixed with better testing in the first place.

This superficially sounds good, and it does solve the biggest problem — the fact that we need to be able to develop distutils / setuptools independent of the CPython release cadence for a variety of reasons — but another of the big motivators for doing this is so that we can make backwards incompatible changes.

To the extent that there is a plan (and my big problem with this PEP is that it doesn’t have a clearly outlined plan to be discussed), the plan was to move all the distutils stuff that setuptools relies on or wants to offer into the setuptools namespace, and then deprecate and remove the distutils namespace. Having setuptools live in the standard library would not buy us any backwards compatibility.

Additionally, setuptools itself is a mess of a package, partially because of bitrot and partially because of a bunch of interdependent backwards compatibility hacks. Right now import setuptools provides pkg_resources, so we’d need to adopt pkg_resources, setuptools and distutils into the standard library as default-included packages all together, which is not great.

Another thing to note here is that we want fewer people running around with setuptools installed, not more. Ideally approximately no one other than build tools would have a runtime dependency on setuptools at some point in the future, and setuptools would just be something you include in your pyproject.toml.

For the “distutils as build tool” deprecation, I think the best course forward here is actually different than the current plan:

  1. setuptools adds everything it plans to support from distutils into its own namespace so that everyone can comfortably migrate.
  2. distutils is moved to a standalone PyPI package that provides the distutils namespace with all the setuptools monkey patches already included, and displaying a large deprecation warning whenever it is imported.
  3. setuptools fully forks everything it needs from distutils and initially maintains some compatibility support (like throwing errors derived from distutils when distutils is present), but after a certain amount of time the compatibility shims will be removed.
1 Like
73

There are 1.5 million files named setup.py on github right now. You’re never going to be able to make meaningfully backwards-incompatible changes to the distutils/setuptools namespaces. It sucks, I hate it, but that’s the reality we’re stuck with. Responsible stewardship means helping them gracefully transition into legacy projects receiving the minimal fixes needed to keep up with packaging changes, and to provide exit routes for users, not planning to rewrite them in place.

If you’re making it an alias for setuptools, then why bother printing a deprecation warning? An alias isn’t a significant maintenance burden.

Writing a PEP that makes it possible and supported to pip uninstall setuptools/distutils would be a pretty substantial step towards that goal. And shipping that for a few versions will make it way easier to later write a PEP proposing that we remove them from the default-install-set. Doing things one step at a time makes them way easier to get through.

(And let’s be honest, any schedule we set for 3 releases from now is mostly wishful thinking anyway; none of us have crystal balls.)

1 Like
74

One does not follow from the other, and we do make backwards-incompatible changes to setuptools, regularly (note that the major version number is 50 at this point ­— and we only bump that when there’s a backwards-incompatible change).

In any case, it doesn’t really matter for what I’m proposing here. I’m proposing that distutils gets spun out into a separate PyPI package that is forever frozen at basically the version of distutils that you get when you do pip install today, but with deprecation warnings saying, “This is unmaintained and may break at some unspecified point in the future, you should migrate to something else — setuptools is the easiest thing”.

Considering that the current plan is to do exactly that, except that you’ll depend on setuptools to get you distutils, I see this as a significant improvement.

As mentioned above, in my proposal distutils would remain in its frozen, post-monkeypatching state, so it’s not a simple alias.

That said, there is a cognitive burden associated with maintaining an alias, because people are very confused about when to use distutils and setuptools.

Strongly disagree. Anything that involves setuptools and/or pkg_resources installed by default would be a problem. setuptools exposes entry points, it vendors many packages, it breaks backwards compatibility frequently, and its presence in a runtime (not build time) environment should be a clue that you have something fishy going on. I’ve been meaning to lobby virtualenv to stop including setuptools and wheel in new virtualenvs by default for a while now, frankly.

The issue is that with PEP 518, you should basically never need to do pip install setuptools at this point. The fact that you’d be able to do pip uninstall setuptools would not help.

Moving distutils itself to a PyPI package that is default-installed might be workable, but your whole argument seems to be premised on the idea that there are people who want to continue maintaining distutils in some form and feel restricted by it being in the standard library. I’m fairly certain that is not the case. The “setuptools becomes the reference implementation for distutils’ build functionality” plan was, as far as I know, always a plan to retire the distutils name and provide the relevant functionality in setuptools directly (to avoid the monkey patching and legacy burden). It was not a plan to have setuptools maintainers take over maintenance of distutils as a public-facing endpoint.

1 Like
75

Just on this point, I’d really worry that if you could pip install distutils we’d start seeing it show up as a build backend in pyproject.toml files. While it’d technically be fine as a build dependency, it wouldn’t solve the problem people think it would. An attractive nuisance that is easily avoided.

76

I didn’t think that was the premise, from what @njs said. I thought the premise was more about avoiding the breakage that would result from distutils disappearing altogether. Obviously no-one is stepping up to maintain distutils, which is partly why we’re in the state we’re in.

Yes, but distutils is more than just the build functionality. The PEP starts off by saying

while completely glossing over the other key functionality - that of installing packages. Sure,

but then you have people saying one shouldn’t be forced to use pip. Apart from pip, flit and any other tools like that, there will definitely be cases where people have shell scripts that do
python setup.py install. Searching for that phrase throws up over 500K results, and while those might all be instructions on “how to install Python packages”, it would be quite likely that there are scripts out there that do exactly that to install, given the ubiquity of those instructions, and some fraction of those setup.py files will be importing distutils rather than setuptools. What is the migration path for those? I think it should be addressed in the PEP.

Maybe less relevant, what happens with old .exe and .msi based installers on WIndows? Will they continue to work, if they contain pure-Python code which is 3.x compatible? Even if the PEP’s answer is “don’t care”, it’s worth spelling out.

1 Like
77

None of them use distutils to do installation, so they’ll be fine (as fine as they currently are, which is “barely hanging on by a thread” :wink: ) (Edit I assume you meant the output of bdist_wininst and bdist_msi, yes?)

78

The migration path is “use setuptools, and use their documentation to migrate your specific case”, as mentioned in the PEP.

This is precisely the kind of advice I don’t want to codify in a PEP until the end of time. User guides have to live separately so they can live, while the PEP is capturing the rationale behind the decision.

79

Good to know. Yes, I meant the outputs of those bdist_XXX commands.

80

For argument’s sake, let me posit the following scenario:

Say I’m a end-user that uses a script that (as part of its functionality) takes an archive that contains a Python package I need, extracts it to a directory, cds to that directory and does python setup.py install. Perhaps the script will keep running during the deprecation period, as deprecation will only warn and not fail. Then one day the script breaks (because my system got upgraded to a Python where distutils is gone), and I get a to-me-cryptic “ModuleNotFound” error message. Where would I have seen the “use setuptools” message, and how would I even make sense of it?

Is this an impossible scenario? If it’s not, do we care about such scenarios and their users?

1 Like
81

Honestly, this argument for leaving distutils around for the sake of allowing people to invoke setup.py install is very weak. As it stands today neither setuptools nor distutils are actively supporting the use of that command. Installing things with setup.py install frequently breaks users’ Python installations, and often they cannot be uninstalled, because they don’t contain the metadata necessary for pip to uninstall them, and distutils doesn’t come with an uninstaller.

The sooner invoking setup.py install breaks the better, IMO, because many people are completely unaware of this until they find that their Python installation is all kinds of broken.

I don’t think we’re on the hook for “everything keeps working indefinitely for everyone in every situation”. To me, the important thing here is to lay out what the plan is, precisely what functionality is being deprecated entirely and what, if anything, is being recommended as migration targets.

I understand why Steve doesn’t want PEP 632 to be a migration guide, but to me “PEP 632 as a migration guide” is far less important than PEP 632 outlining the transition plan. What happens with the distutils namespace? Which parts of it have transition targets elsewhere in the standard library? Is the plan that if you add a dependency on setuptools that import distutils will work forever or is the idea that setuptools will manage the transition?