PEP 632 - Deprecate distutils module

PEPs
1

Raw text below, pull request at https://github.com/python/peps/pull/1581, I’m planning to be asleep by the time CI is done, so please nobody steal my number in the meantime :slight_smile:

~Once a rendered version is up I’ll add a link.~

Update: Rendered version is at https://www.python.org/dev/peps/pep-0632/ and text in this post has been updated

PEP: 632
Title: Deprecate distutils module
Author: Steve Dower steve.dower@python.org
Discussions-To: PEP 632 - Deprecate distutils module
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 03-Sep-2020
Python-Version: 3.10
Post-History: 03-Sep-2020

Abstract

The distutils module [1]_ has for a long time recommended using the
setuptools package [2]_ instead. Setuptools has recently integrated a
complete copy of distutils and is no longer dependent on the standard
library [3]_. Pip has been silently replacing distutils with
setuptools when installing packages for a long time already, and the
distutils documentation has stated that it is being phased out since
2014 (or earlier). It is time to remove it from the standard library.

Motivation

distutils [1]_ is a largely undocumented and unmaintained collection
of utilities for packaging and distributing Python packages, including
compilation of native extension modules. It defines a configuration
format that describes a Python distribution and provides the tools to
convert a directory of source code into a source distribution, and
some forms of binary distribution. Because of its place in the
standard library, many updates can only be released with a major
release, and users cannot rely on particular fixes being available.

setuptools [2]_ is a better documented and well maintained enhancement
based on distutils. While it provides very similar functionality, it
is much better able to support users on earlier Python releases, and
can respond to bug reports more quickly. A number of platform-specific
enhancements already exist in setuptools that have not been added to
distutils, and there is been a long-standing recommendation in the
distutils documentation to prefer setuptools.

Historically, setuptools has extended distutils using subclassing and
monkeypatching, but has now taken a copy of the underlying code. [3]_
As a result, the second last major dependency on distutils is gone and
there is no need to keep it in the standard library.

The final dependency on distutils is CPython itself, which uses it to
build native extension modules in the standard library (except on
Windows). Because this is a CPython build-time dependency, it is
possible to continue to use distutils for this specific case without
it being part of the standard library.

Deprecation and removal will make it obvious that issues should be
fixed in the setuptools project, and will reduce a source of bug
reports and unnecessary test maintenance. It will also help promote
the development of alternative build backends, which can now be
supported more easily thanks to PEP 517. [4]_

Specification

In Python 3.10 and 3.11, distutils will be formally marked as
deprecated. All known issues will be closed at this time.
import distutils will raise a deprecation warning.

During Python 3.10 and 3.11, uses of distutils within the standard
library may change to use alternative APIs.

In Python 3.12, distutils will no longer be installed by make install or any of the first-party distribution. Third-party
redistributors should no longer include distutils in their bundles or
repositories.

This PEP makes no specification on migrating the parts of the CPython
build process that currently use distutils. Depending on
contributions, this migration may occur at any time.

After Python 3.12 is started and when the CPython build process no
longer depends on distutils being in the standard library, the entire
Lib/distutils directory and Lib/test/test_distutils.py file
will be removed from the repository.

Other references to distutils will be cleaned up. As of Python 3.9’s
initial release, the following modules have references in code or
comments:

  • Lib/ctypes/util.py
  • Lib/site.py
  • Lib/sysconfig.py
  • Lib/_aix_support.py
  • Lib/_bootsubprocess.py
  • Lib/_osx_support.py
  • Modules/_decimal/tests/formathelper.py

The following Tools in CPython also refer to distutils. Note that none
of these are installed with CPython:

  • PC/layout (references will be removed)
  • Tools/msi (references will be removed)
  • Tools/peg_generator (will be adapted to a different build tool)
  • Tools/test2to3 (example project will be removed)

As the distutils code is already included in setuptools, there is no
need to republish it in any other form. Those who require access to
the functionality should use setuptools or an alternative build
backend.

Backwards Compatibility

Code that imports distutils will no longer work from Python 3.12.

The suggested migration path is to use the equivalent (though not
identical) imports from setuptools (see [5]), or to migrate to an
alternative build backend (see PEP 517 [4]).

Code already exists in setuptools to transparently switch setup.py
files using distutils onto their equivalents, and so most working
build scripts are already known to work with setuptools. Such scripts
may need to update their import statements. Consult the setuptools
documentation for specific migration advice. [5]_

Some projects use alternate sets of patches over distutils, notably,
numpy.distutils. [6]_ Projects that we know are doing this have been
informed.

Many build scripts use custom commands or narrowly scoped patches. As
these packages are already subject to setuptools overriding distutils,
we expect minimal disruption as a result of distutils being removed.
Scripts may still need to be updated to avoid importing distutils.

Reference Implementation

setuptools version 48 includes the complete copy of distutils, and as
such is no longer dependent on the standard library’s copy. Most
implementation issues they have faced are due to the continuing
existence of distutils in the standard library, and so removal will
improve the stability of their implementation.

There is not yet a reference implementation for the removal of
distutils from the standard library, nor is there an implementation
for CPython’s native module builds without relying on the standard
library copy of distutils.

References

… [1] distutils - Building and installing Python modules
(https://docs.python.org/3.9/library/distutils.html)

… [2] setuptools - PyPI
(https://pypi.org/project/setuptools/)

… [3] setuptools Issue #417 - Adopt distutils
(https://github.com/pypa/setuptools/issues/417)

… [4] PEP 517 - A build-system independent format for source trees
(https://www.python.org/dev/peps/pep-0517/)

… [5] Porting from Distutils
(https://setuptools.readthedocs.io/en/latest/distutils-legacy.html)

… [6] Packaging (numpy.distutils)
(https://numpy.org/doc/stable/reference/distutils.html)

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

7 Likes
2

We should add that the test suite for the PEG parser uses distutils to compile extensions to test the C code generator (Tools/peg_generator/pegen/build.py).

2 Likes
3

Thanks, will do. That’s also only for during build, right? So if we move distutils under Tools it’ll just need a path update and will be fine?

1 Like
4

Yeah, I think that moving distutils under Tools and changing paths will do the trick!

1 Like
5

Allowing CPython to use distutils while requiring everyone else to find a work-around seems unfair. If CPython cannot use setuptools.distutils during its build, then how could anyone else? The peg_generator has a requirements.txt file, why can’t it add setuptools as well?

2 Likes
6

Correction - there are no compatibility shims in pip, as pip has no programmatic API. The wrapper pip uses to redirect to setuptools relies totally on setuptools monkeypatching distutils.

Do the recent issues around setuptools 50.0 affect this statement? It looks like setuptools 49.1 and 50.1 both reverted the use of the bundled distutils, implying that the migration is much less complete than this statement suggests.

7

Thanks, I’ll clarify. Though given the bit below it might be useful for pip to inject at least an import of setuptools for legacy scenarios.

My understanding of those issues were that they changed how aggressively they’d try and replace an already imported distutils. Now that the patching isn’t happening to the stdlib version, people who did “import distutils” before “import setuptools” and thought they were only getting setuptools are actually getting confusing behaviour.

The best thing we can do for setuptools here is remove distutils from the stdlib :smiling_face: Until then, we’re in the weird transition period where we have to manage compatibility while also introducing change. More aggressively overriding distutils in cases where we know setuptools will be used should help.

From the POV of this PEP though, the adoption has been done and we don’t have to delay deprecation because of the dependency.

1 Like
8

CPython is building itself, so there isn’t yet a complete runtime environment. If setuptools is willing to restrict itself to only using builtin modules (under all configurations) and no other third party dependencies, I’m sure it would be fine.

More likely we’ll migrate CPython’s extension modules to build through make. I know some people were keen to do this anyway. If that happens, the core build won’t use distutils either.

2 Likes
9

That’s what pip does - inject import setuptools and assume that everything else is done by magic in setuptools.

10

Okay, that’s all I was implying by “compatibility shim”. I just wanted to use a generic term (though possibly it’s a Microsoft/Windows-only term) rather than describe the specific mechanism.

Edit: on rereading the whole paragraph, I don’t really see the need to clarify much. Except maybe swap out “Compatibility shims already exist” for “Code already exists”? Thoughts @pf_moore?

Compatibility shims already exist in setuptools and pip to
transparently migrate old code, however, these are controlled by
external projects and are in no way bound by this PEP. Consult their
latest documentation for migration advice.

11

Thanks for taking this on Steve. Now for the criticism :stuck_out_tongue:

The biggest issue I have with this PEP is that because it’s tightly scoped to the CPython side, it doesn’t cover almost any of the important details that will be relevant to end users, namely:

  1. When distutils is deprecated, what are users supposed to migrate to?
  2. What is the full roll-out plan for the migration targets?
  3. How should one declare a dependency on distutils?

For example:

setuptools actually only has direct equivalents of a few commands. It doesn’t have anything like distutils.util, distutils.version or distutils.spawn, and adding those things should be out of scope for setuptools.

Right now, what happens is that whenever setuptools is installed in the normal way and the distutils adoption is enabled (done by default in a few versions but apparently reverted in the latest version), it forces import distutils to import the version of distutils vendored into setuptools. Right now, that’s an unmodified version that setuptools then monkey patches (as was the case when it was in the standard library), but I believe the next steps there were to start migrating code from distutils directly into setuptools, so that setuptools no longer actually relies on distutils (and in fact the opposite — distutils would import from setuptools). Once we have migration targets for everything, we’d start raising deprecation warnings whenever import distutils is used, and preferably point to a documentation page about how to migrate away from distutils.

Note: Everything I said above is a huge amount of work, and is much more likely to go smoothly if we can get funding for someone to do it.

Migration path

I think the biggest open question is the migration path. I do not think it would be good if, going forward, everyone started adding setuptools (which is now a catch-all name for "setuptools+ pkg_resources + distutils) as a dependency for runtime stuff like distutils.util.strtobool or distutils.dir_util.create_tree. A few options:

  1. Migrate or identify replacements for everything not going directly into setuptools ahead of time, and aggressively deprecate them.
  2. Create a distutils package on PyPI rather than bundling setuptools and distutils together like that, and have people declare a dependency on distutils if they explicitly need stuff from distutils. We can “upstream” the existing setuptools monkey-patches into this version of distutils, and then fork the desired build code into setuptools (so that setuptools doesn’t have a dependency on the PyPI distutils) — the PyPI version would be deprecated and only maintained for a relatively short period of time, and setuptools would never expose a distutils namespace.
  3. Option 2, but instead of having a single distutils package, we migrate distutils to a namespace package, and have people declare their dependencies more granularly. This would make it easier to identify which distutils submodules need migration and which ones are mainly going unused.

I think Option 3 is a bit overboard, and Option 2 is much better than the current mechanism used for setuptools adopting distutils, and accomplishes basically all the same goals.

1 Like
12

I’d rather completely omit pip from this. There’s no documentation in pip for users to consult, I would not want people to report any issues to pip, and there’s no user-facing mechanism in pip that users can use to mitigate any issue. Mentioning pip IMO obscures the key point here which is that setuptools is the single point of contact for any issues with using distutils post-removal from the stdlib.

2 Likes
13

Their choice of build backend.

Out of scope for this PEP, as I think you’ll find that the range of existing build backends prefer to set their own schedules.

[build-system]
requires = ["setuptools"]

:slight_smile:

Thanks for linking to the documentation for these modules… oh wait :slight_smile: They’re not documented (except in a legacy page that says it’s all moving into setuptools). No matter why people are using them, they are not guaranteed public API, and despite that, we’re still offering two deprecated releases before they go away.

We’ve been neglecting/refusing to fix issues in those modules for at least half a decade, and unless someone is stepping up to start taking that job seriously, we’re better off having people migrate.

I’ll also add here that a PEP is not a migration guide. Some of these functions certainly exist elsewhere, such as sys, subprocess, packaging, but it’s not the job of a deprecation PEP to provide every single function with an alternative. The primary use case for distutils as stated in the docs is installing and distributing Python packages, and the recommendation of pip for the former and setuptools for the most-compatible option for the latter is enough.

Or we can normally deprecate them and aggressively find alternatives. I can’t dictate what setuptools should or shouldn’t include, so this sounds like a discussion to take over there.

I’ll point out again that distutils has been not-very-aggressively deprecated for over five years, and likely closer to ten (but that’s before my time).

It’s open source, sure, anyone can do that. If anyone from the core team was willing to maintain distutils in its place, or anyone from the community wanted to join the core team and take ownership of distutils in its place, then we wouldn’t need to remove it. But since nobody wants to do that, I think you’ll be lucky to find someone willing to distribute it separately.

Oh, apart from setuptools :slight_smile:

That’s something to take up with the setuptools team, preferably on a thread in the Packaging category or their issue tracker.

14

Updates to the PEP text are in https://github.com/python/peps/pull/1582 (for now, will probably merge it by the end of the day)

15

distutils has a bunch of stuff in it that is not related to your build backend.

I realize that all of this is out of the specified scope of the PEP, but that I’m a strong -1 on this PEP unless the scope is expanded to include a plan for what to do.

Alternatively, I recommend modifying the PEP to no longer mention setuptools, since it’s irrelevant. The extent of the PEP can thus be, “We will deprecate distutils in Python 3.10 and remove it in Python 3.12. All current users must find an alternative.” Personally, I don’t think that’s very satisfying.

I don’t think it’s fair to say, “They’re not documented except for a legacy page saying they’re moving to setuptools” when my point is they are not moving to setuptools. “Setuptools provides the distutils module” is an intermediate step to a better future and I’d consider it a worse and more confusing outcome than the current situation if it happened indefinitely, TBH.

I think by “aggressively deprecate” I meant that we would make it very clear that you should never declare a dependency on setuptools to get access to these functions — they’ll be removed there before they get removed from CPython.

I’m talking about how, structurally, distutils gets maintained. In all the currently-proposed scenarios, we’re looking at distutils coming in via a 3rd party PyPI package. I believe the plan (though it’s not laid out here), is to then deprecate distutils both in CPython and in the 3rd party package, with the goal that in X years, nothing will support import distutils.

And no, random people can’t add a distutils package to PyPI because it’s “Open Source” — or at least they shouldn’t be able to, since standard library module names are reserved to prevent squatting.

In any case, I’m a strong -1 on this PEP as it stands. I think the concerns that led people to say that we should write a PEP for this are completely un-addressed, notably this, which echoes other things I’ve seen:

PEP or not, what we need is an overall migration plan so that we can explain to people why what we’re doing is not some horribly destructive thing for the ecosystem and to give them the ability to plan for future changes. “We’ll be removing distutils using the normal deprecation process, setuptools might support some parts of it I guess, but that’s up to them” is not that. It’s a huge migration likely to impact large swathes of the ecosystem, I think it deserves a design review.

1 Like
16

So I think two things are being conflated here. The first that distutils is (the core of) a build backend, and the second is that distutils is a grab bag of random functionality. They deserve to be treated separately.

I’m not personally interested in drawing the line, so if someone would like to define which parts of distutils are part of the build backend, which parts are the grab bag, and which parts don’t matter, I’m happy to include that information in the PEP (and make whoever wrote it a co-author).

But I disagree that it has to exist before formally deprecating the module. The motivation to deprecate is strong enough on its own, and three years is a long time.

17

+1

Either the PEP shouldn’t be mentioning 3rd party tools at all, as they are out of scope, or it should be covering (at least to some extent) how projects should migrate, and what (necessarily 3rd party) tools exist to help with the migration.

If the PEP’s stance is that we’re finally removing distutils and it’s not our problem if people are still using it in spite of years of being told it’s frozen, and we don’t have any recommendations on how people should move off distutils, then it should be up front and say so.

I’d be -1 on a PEP that said that, but at least it would be clear about what it was proposing.

18

Isn’t it doing the second? And at the request of those third party tools, deferring the details to their documentation rather than trying to capture it directly in the PEP.

Can you provide some (false, made-up) examples of what kind of guidance you want to see in the PEP and why it shouldn’t be in a living document instead?

19

I was thinking of the “grab bag of random functionality” you mentioned. My understanding is that the setuptools maintainers have said they don’t intend to support that long-term, so telling people how to migrate off that seems within the scope of the PEP.

distutils.spawn.find_executable → shutil.which
distutils.spawn.spawn → subprocess.run

I’m happy for it to be in a “living document” instead. But what document is that? The PEP doesn’t say. Just directing users vaguely at “setuptools” isn’t much help, in practice.

As regards the “build backend functionality”, most of the documentation for how to use that is in the legacy distutils docs - which say

This document is being retained solely until the setuptools documentation at Building and Distributing Packages with Setuptools - setuptools 69.0.3.post20231223 documentation independently covers all of the relevant information currently included here.

But that hasn’t happened yet, so presumably the deprecation would result in users having no documentation for that essential functionality - or is the proposal to keep those legacy docs? How would that work if we remove distutils from the stdlib docs (and so remove the link to the legacy pages)?

Sure, you can say it’s setuptools’ problem that they haven’t migrated the docs yet. But conversely, you can’t hold setuptools up as where people should go once distutils has been dropped if they manifestly are not. A proper transition plan for this documentation should be part of the PEP.

Can you link to where that request was made? My questions may be better addressed to the setuptools maintainers if they really did ask CPython to drop distutils and direct future questions to them. The link should probably be in the PEP, too, as it’s fairly important background. As I’ve already said, I’m much less concerned if the setuptools maintainers have accepted this responsibility (although it doesn’t sound like @pganssle for one is happy with the way the PEP interprets whatever has been said…)

1 Like
20

This seems like a good place to put (more) of those mappings: https://setuptools.readthedocs.io/en/latest/distutils-legacy.html

I definitely had in-person chats with @jaraco about this, though I’m sure some happened online as well because we’ve been talking about this for years now (since the first PyCaribbean, IIRC?). I can try and dig up some written discussions, but it’s probably easier to get a fresh acknowledgement here in light of the new concerns being raised.