Pip without setuptools, could the experience be improved?

Hello.

In Fedora, our pip package Recommends setuptools. Practically that means:

  1. Majority of users who install pip will get setuptools by default.
  2. Users can explicitly uninstall setuptools after installing pip or exclude setuptools when installing pip.
  3. Users might op-out from installing Recommended packages by default to save space (I suppose the majority of such users do it in containers or a similar environment), and such users won’t get setuptools with pip.

I like that our users have a choice not to install setuptools with pip. I also like that by default, they’ll get it. What worries me is the experience for users from (3) who might get pip without setuptools while not explicitly asking for this setup. When pip attempts to install a package from sdist without a PEP 517/518 build backend specified, it’ll fail with:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named 'setuptools'

This traceback is displayed multiple times, as backtracking causes lots of failed attempts.

Originally, I thought: Right, it’s because the setup.py script imports it, but I’ve realized it is actually pip that does that:

Even packages that don’t use setuptools in their setup.py script will fail with the same problem. I’ve successfully tested the assumption with this example:

from distutils.core import setup
setup()

After the recent discussions on the Python-Dev mailing list – Mailman 3 Does ensurepip still have to include a copy of setuptools? - Python-Dev - python.org, and after the recent discussion with @jaraco in Declare dependencies in metadata by jaraco · Pull Request #2764 · pypa/setuptools · GitHub, I have mixed expectations whether pip should always Require (rather than Recommend) setuptools. In the RPM world, that means:

  1. All users who install pip will get setuptools by default.
  2. Users cannot explicitly uninstall setuptools after installing pip nor exclude setuptools when installing pip.
  3. Users might op-out from installing Recommended packages by default to save space, and such users will still get setuptools with pip.

This feels like a quite inflexible setup. I would rather allow not installing setuptools with pip when users know what they are doing, but users from (3) might not understand this problem, as it might not be from their domain at all. The Traceback might be interpreted as a problem in pip.

Would it make sense to handle missing setuptools from pip’s code in a more explicit way? Ideas:

  1. Instead of displaying a Traceback, display an error message explaining that setuptools is not installed and installing <package> requires it. Suggest installing setuptools before installing <package> or using --use-pep517 to install <package>.
  2. Default to --use-pep517 when setuptools is not installed (see also Default to --use-pep517 · Issue #9175 · pypa/pip · GitHub).

Pip already handles the presence or absence of wheel differently, so there is a precedent.

What do you think?

CC’ing also @pradyunsg and @encukou who might be interested in this topic.

2 Likes

The direct use of setuptools should only be on the “legacy path” for pip. Unfortunately, there’s still a lot of legacy code out there :slightly_frowning_face:

The plan is that we switch to PEP 517 behaviour by default, with build isolation and an implied use of setuptools if there’s no pyproject.toml. I’m not 100% sure on precisely where we are with that migration, but it’s definitely in progress. (Editable installs is the biggest problem here, as we can’t drop the legacy route until the setuptools PEP 517 backend adds editable install support).

But otherwise, I see no reason (other than “this is going away real soon, so why bother?”) why we wouldn’t be happy to improve the error reporting for the case when setuptools isn’t available by default.

See pypa/setuptools#2816 to track Setuptools’ support.

In my workflows, I’ve left Setuptools uninstalled and I rely on --use-pep517 to force pip to install any package without Setuptools being present. Additionally, I just verified that editable installs work in the same environment, at least with the latest pip:

draft $ python -m venv env
draft $ env/bin/pip uninstall -y setuptools
Found existing installation: setuptools 57.4.0
Uninstalling setuptools-57.4.0:
  Successfully uninstalled setuptools-57.4.0
draft $ cat > setup.py
__import__('setuptools').setup()
draft $ env/bin/pip install --use-pep517 -e .
Obtaining file:///Users/jaraco/draft
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: UNKNOWN
  Running setup.py develop for UNKNOWN
Successfully installed UNKNOWN-0.0.0

That is, the lack of a pyproject.toml implies “use setuptools” and pip makes it available, meaning that once pip makes --use-pep517 the default, Fedora can move from “recommends” to entirely optional, even without PEP 660 support and still supporting legacy code.

Am I mistaken?

1 Like

Oh, that’s cool! Sorry, I should have checked before posting. As I say, I’ve not been keeping up with the state of this work.

Improving these error messages is something was working on last Friday.

:slight_smile:

As for --use-pep517 by default, I think we’d need to do quite a few band aid removals, before we can get to that. We need to rip out at least the band aids of the legacy versions and legacy setup.py install, for a switch over to PEP 517 by default. It would likely make sense to do so throughout the codebase, rather than just in the PEP 517 code path.

1 Like

Is there a PR or a draft I can play with? Or is there a way to contribute to help this effort?

So here is the question: what do you like about it? Is it just aesthetical concerns? Would your users lose anything if pip had setuptools as a hard dependency?

I’ll put up a draft PR over the weekend for this. It won’t change the ImportError, but I guess we can improve that as well separately.

It’s a size and attack surface reduction consideration (mostly the former).

In particular, embedded and container use cases often only need wheel installs, where “pip without setuptools” is the most viable current option for both wheel installation and easy environment introspection.

1 Like

What @ncoghlan says + the ability to uninstall setuptools which gives the opportunity for package maintainers to discover packaging bugs (something needs setuptools, but doesn’t declare the dependency).

1 Like