I think this argument is going a bit in the wrong direction. “setup.py
install” is no longer needed nowadays, since pip, conda and setuptools
take care of running the package’s setup.py commands.
As long as existing packages can still be installed with pip, conda
and setuptools, we should be fine.
That said, many packages provide custom extensions to distutils via its
class based extension mechanism, e.g. to compile extensions and
find / configure the necessary dependencies for this. This will need
to continue working even without distutils coming from the Python
stdlib. Shouldn’t be a major problem either, if distutils is placed
into setuptools and this also registers itself under the distutils
namespace.
BTW: To test drive all this, you could get in touch with Ben Nuttall
who runs the piwheels.org site. I’m sure he could help figure out
what the consequences of such a change would be by using the site
to run a Python version which has distutils removed and a setuptools
package with a version that does all the needed tricks. A PSF
grant would most likely help put this into reality
(Background: piwheels builds binary packages for a huge number of
packages available on PyPI and thus is a great benchmark for any
such changes.)
That’s not what I’m arguing. I’m arguing for the PEP to explicitly mention such scenarios and say what will happen in such cases. If someone is going to vote on a change with such a wide-ranging impact, then relevant information about adverse impacts should be spelled out, for voters who are not close to the details of particular packaging modules.
Perhaps not everyone has that opinion, though I can’t say I blame people who have to work on distutils expressing their frustrations at its limitations. Why shouldn’t such adverse outcomes for end users be spelled out explicitly as the outcome of implementing this PEP? Breakage will also occur in scenarios where python setup.py install had no adverse impact on someone’s Python installation (which might be a venv in the scenario I mentioned).
Sure, but is it OK to say “some things that have worked since forever will break even when they don’t need to, because we removed functionality that didn’t need to be removed, along with some functionality that did need to be removed”? Isn’t that against Python’s general stance on backwards compatibility, no matter how hateful distutils is to work on/work with?
It’s not a migration guide to say that python setup.py install will stop working for everyone who isn’t using setuptools - just an impact statement, which IMO properly belongs in the PEP.
I agree with your other points about the importance of a transition plan.
Yes, let’s be specific about the impact of the change - which includes expected breakages.
That seems reasonable. What’s most important to me is that a beginner can read one document and come away knowing how to package something, as long as the something is not too complicated. Secondarily, the reference-level documentation for each specific tool should be complete in itself and up to the (high) standard set by the stdlib in general.
Obviously not, but wording it that way is clearly intended to get that reaction.
“Distutils has been unmaintained for many years, and is being removed from the standard library in line with Python’s deprecation policies. People using distutils have the usual 2 releases to migrate their code to use alternative APIs. In general, packaging advice has for many years been to move to setuptools. Such a move should be straightforward for the majority of users and will involve minimal disruption. As with any significant change, though, users are advised to test the migration thoroughly, and not to upgrade Python in their production environment until they have confirmed that they have migrated successfully”.
That’s much more reasonable.
I’m still not in favour of the hardline approach the current version of the PEP advocates, but it doesn’t help the arguments to describe the proposals in the worst way you can…
It’s also not true - assuming you follow the advice that has been given many times and add import setuptools at the top of your setup.py script. If you do that, nothing should stop working, and you’ll then have the benefit of using a maintained tool.
Yes, I know there are other things - you need setuptools installed, the setuptools maintainers have indicated that they don’t plan on supporting setup.py indefinitely, etc. But “setup.py install will stop working in Python 3.12” is an exaggeration that undermines the point you’re trying to make.
This might be veering off-topic, but I have a bunch of deployment automation that manually does pip install --upgrade pip setuptools wheel in each new virtualenv it creates, immediately after creation. This is because, as far as I can tell, you need all three of those packages installed, with current versions, or you can’t reliably do pip install <any other package>. It will do things like download a wheel, fail to unpack it because wheel isn’t available, fall back to building from the source tarball, and then that will fail because setuptools isn’t available either (or because the stripped-down VM where this is happening doesn’t have a C compiler, or something else like that). How is pip install supposed to work in the absence of these packages?
Moreover, python3 -m venvdoesn’t always install setuptoolsorwheelor a current version of pip — of this I am quite certain — which makes me feel like you’re posting from Bizarro World. (Perhaps just a world where you don’t need to deploy under older versions of CPython? 3.8 does seem to behave as you describe, but I still regularly need to work with 3.5.)
I just created a virtualenv and uninstalled everything except pip. pip install worked for winrt, and requests - both of which either provide wheels for everything or I had cached wheels. I tried pip install bs4 and it failed because setuptools wasn’t present and bs4 is a sdist only. But pip install --use-pep517 bs4 worked fine.
If you want to remove setuptools and wheel, you should probably use --use-pep517 to disable pip’s legacy install route, but that should be all you need. At some point pip will make --use-pep517 the default, but it isn’t at the moment.
You can also use --only-binary :all: to block anything being built from source. That will avoid the “no C compiler” issue, too - but only if you have pre-built wheels for everything, of course. (--prefer-binary will use wheels when available, and only sdists if there’s no other choice, which might also be useful).
I agree it’s extreme wording, but only in counterpoint to the wording of the bit I responded to. But is it untrue in terms of the outcome it’s describing?
As a message to package developers, sure. My scenario was about end users, though.
I did say “for argument’s sake”, and to some extent I’m playing devil’s advocate, but not just to be difficult for the sake of it. I think there’s a genuine issue here about breakage for end-users, but perhaps I’m just not expressing it very well.
I’m not saying my scripts will be adversely affected - AFAIK all my public projects can be installed with pip, so they shouldn’t be affected. But I put myself in the shoes of an end-user (like people I’ve worked with) and asked how they would be affected, and your comment above doesn’t address that end-user, only a developer like me who generally doesn’t use python setup.py install.
I’ve already qualified that by limiting that assertion to existing setup.py files that use import distutils, and I don’t see why that’s an exaggeration, given the PEP as it stands.
This is indeed off topic, so this is the last post I’ll make discussing it, but to be clear I was talking about virtualenv, which bundles setuptools and wheel automatically in all new virtual environments (presumably in an attempt to be helpful) unless you explicitly ask it not to. venv doesn’t have this behavior, but it’s also a stripped-down version of virtualenv.
The key point I wanted to get across here is that in general we’re moving to a world where the build-time environment is entirely separate from the runtime environment (whether that’s managed by a redistributor or by pip), and it no longer makes sense to default to having setuptools and wheel in your Python environment — and in fact it is in many ways harmful to do so.
In order to remove distutils from the standard library, more people will need to explicitly declare their dependency on it — this is true and probably will be mildly painful (but really not that bad, in the scheme of things — it’s incredibly easy to fix compared to some of the other changes we’re describing).
Debian’s system venv package doesn’t bootstrap pip properly if you don’t explicitly install python3-venv.
Responding to the thread more generally:
Thanks for taking on this mammoth task, Steve
To help clarify the notes in the documentation, they were mostly written by me not long after a PyCon Q&A panel that we called “‘./setup.py install’ must die”.
As such, those notes ignore the “grab bag of random functionality” aspect of distutils not because that aspect isn’t a concern (it is), but because we most urgently wanted people to stop using distutils as an installer due to the huge problems it has in that role, and doing anything about the other features it offers was a distant future concern. Your PEP means that previously distant future now has a draft timeline
I think it may be worth splitting the PEP into two proposals:
a full removal PEP targeting 3.12
a formally-deprecate-without-committing-to-removal PEP for 3.10 that asks users encountering the new runtime deprecation warning to review & provide feedback on the full removal PEP
I know we don’t normally do that, but in this case, I think getting agreement on the programmatic deprecation will be vastly easier than getting up front agreement on a concrete removal plan, while the latter should be easier to assess after an initial release with the deprecation warning in place.
What makes you think I’m trying to make life easier for myself?
I really don’t like proposals that let people read into them whatever they want. In this case, yes, we’d get agreement, because the pro-removal people would assume that we’ll remove it and the anti-removal people would assume the opposite. And without a scheduled removal to motivate people to update to supported libraries, I worry that this approach basically guarantees we’ll be stuck carrying the code forever, while it rots.
The best precedent we have for this case is probably tp_print, where we decided at the last minute that removal would be too harmful for the ecosystem (despite a decade-long deprecation period) and deferred it by a release. I believe that’s “allowed” by our existing policy, though of course we can do whatever we like when we need to, but I’ve no intention of tricking people into withdrawing their arguments by offering such an escape route.
The general concerns have been made clear enough, and I think once they’re made specific enough we’ll be able to make progress.
I pulled together the list of APIs that people have mentioned that need to be migrated (both here and on python-dev). What else?
distutils.ccompiler
distutils.config.PyPIRCCommand
distutils.core.Distribution
distutils.dir_util.create_tree
distutils.spawn.find_executable
distutils.spawn.spawn
distutils.sysconfig
distutils.util.get_platform
distutils.util.strtobool
distutils.version
(I’ve deliberately not listed existing alternatives yet. We can figure that out once we’ve agreed on the scope of what constitutes the public/expected API.)
For the three mentioned functions without replacements, how about we move them rather than dropping them entirely?
Move to shutil (Tarek moved the archive utils over to shutil.archive years ago, so it seems like a good home for any dir and file manipulation functions we want to keep):
distutils.dir_util.create_tree
Move to os.path:
distutils.util.change_root
Move to configparser:
distutils.util.strtobool
For the last one, configparser already offers similar functionality for config values, so this would mean breaking that functionality out to a standalone helper function and adding ‘y’ and ‘n’ to the set of understood strings:
create_tree is weird in that it operates on a set of paths, and we don’t have anywhere that does that. Replacing it more direct code is likely to be better in almost every case:
for p in {Path(f).parent for f in filenames}:
p.mkdir()
change_root seems like a generally bad idea, but that’s just based on reading the code. If someone can specify its behaviour in a way that makes it seem at all useful, I might consider it.
Adding a static strtobool or getboolean to configparser seems fine. I still don’t think it’s worth blocking the distutils deprecation over, and it’s probably easier to get the new function through the python-ideas discussion by pointing out the existing one has been deprecated.
It might need some improvements, and people may want to argue against configparser being used as a dumping ground for unsupported functionality. Whoever maintains that module should certainly get a say.
