Build system and undeclared dependency on it during runtime

It is not uncommon for packages to depend on pkg_resources which is provided by setuptools. Typically, packages don’t add setuptools to install_requires when they need pkg_resources. I think they should, because even though setuptools may be a build-time dependency as its often the build system, that says nothing about whether it will be available during runtime. What do you think? Where can we clarify this? In Nixpkgs we are now patching our recipes to add setuptools in case they have a runtime dependency.

This is somewhat related to https://discuss.python.org/t/support-for-build-and-run-time-dependencies.

Setuptools is not a build-time dependency under PEP 517, and wheels don’t even need to be built, so the fact that Setuptools is implicitly required at build time should not factor in whether it should be included in install_requires. It should, period.

3 Likes

I think packages relying to use pkg_resources without depending on it explicitly are bad and should be fixed. Packages should instead look for alternatives to pkg_resources and define those explicitly either in their install requires or build requires.

2 Likes

We ran into something similar in Fedora bug 1739776, where Miro said:

I’ve always assumed the following upstream:

If you pip install packages, you have setuptools. setuptools and pip are not part of pip freeze output etc. and are simply assumed to always be present.

If packagers leave setuptools out of install_requires, they never notice. This is a problem for non-pip installers that use wheel metadata. The workaround would be to always add a setuptools dependency.

Is that official? We’ll be happy to remove the workaround, and let Fedora be a place to find these bugs. But we’d want to link to this as an some kind of “official pronouncement” of PyPA or pip maintainers, so it doesn’t appear as a Fedora bug/quirk.


The bug I linked above is about another issue: setuptool-generated entrypoints require setuptools to run, but setuptools doesn’t put itself in install_requires when it creates an entrypoint.
I haven’t gotten around to asking this yet, but it would be nice to get clarification on this:

I’m not sure if pip’s devs see setuptools as a guaranteed dependency or an implementation detail.

As a pip developer, but not speaking in any official capacity (this is just my personal opinion) I believe that projects should declare any dependencies they have. Relying on anything that you don’t declare is incorrect (I carefully didn’t say “bug” there, I’m not sure if I’d put “lying in your metadata” in the same category as an actual software bug).

Packages that rely on pkg_resources at runtime need to declare that dependency. Unfortunately (and I consider this a misfeature of setuptools) pkg_resources isn’t distributed as an independent package, but is rather a part of the full setuptools package, which is otherwise a build time only requirement. That sucks, IMO, and so I don’t really have a good solution to suggest. Luckily for me, it’s not a practical issue in the environments I work with, so I mostly just ignore the issue :slightly_smiling_face:

1 Like

Bug, issue, incorrectness, annoyance,… the category sounds irrelevant. Whatever it is, as an integrator I ask where it should be fixed. Whom do I talk to, which side do I patch, where do I send pull requests? :‍)

Personally, I’d flag it to the project. My point is, though, that there’s no “official pronouncement”, so going back to the OP’s point, I’d phrase such a report as “because relying on an undeclared dependency breaks some tools, for example the way Nix packages projects” rather than a blunt “because it’s wrong”.

Under PEP 517 and build isolation, there is no guarantee that just because setuptools is present at the build stage, it’s definitely present in the target environment.

To offer an alternative perspective, I often experience push backs from maintainers when I phrase a report this way, e.g. when advising projects to adopt environment markers instead of if-based syntax in setup.py. For a significant number of maintainers (that I have encountered), whatever works for them is good enough, and downstream projects are responsible for their own problems. A more “official” stance would help a lot to push projects to the right track IMHO.

1 Like

I’m somewhat amazed that anyone would both know that pkg_resources is provided by setuptools and assume that it exists and that you don’t have to declare a runtime dependency on it if you depend on it at runtime.

Do we really need an official pronouncement that you should declare a dependency on your runtime dependencies? Neither setuptools nor pkg_resources are part of the Python standard library (which is generally the only place where dependencies can be implicit), and despite the fact that many people will already have them installed for various reason (though probably decreasingly so in a post-PEP 518 world), there’s no reason to believe they are special in the sense that you can assume they are installed at runtime.

I think that there were some brief attempts to separate the two packages, but in the end most of the energy that would be put into updating pkg_resources is going into improving importlib and its backports, with the eventual goal of removing pkg_resources entirely. From my understanding great progress has been made on this front so far.

1 Like

Exactly. I occasionally experience the same thing when proposing packaging-related changes.

In this case simply having this thread and seeing agreement on the actual issue is sufficient IMO. I can simply reference it in issues/PR’s then. Thanks everyone!

1 Like

Thanks for sharing what seems obvious! Let’s do more of that.
I filed an issue on setuptools not declaring a dependency it injects automatically: https://github.com/pypa/setuptools/issues/1872

1 Like

https://github.com/ninjaaron/fast-entry_points provides wheel-like wrappers for setup.py installations. It seems to work quite well in my experience. I’ve planned to do a full OS rebuild with it injected into the setup.py of all packages, to see what breaks. c.f. https://github.com/ninjaaron/fast-entry_points/issues/11

I dont know all the reasons for setuptools relying on pkg_resources in the wrappers, and I am sure they are well reasoned, but it seems like there are many usages/scenarios when it is unnecessary, at least on Python 3.7 and more-so on Python 3.8 due to improvements in importlib. IMO the pkg_resources based wrappers should be a fallback only if really needed, or explicitly requested by the setup.py when extra compatibility is desired.

See https://github.com/pypa/setuptools/issues/1872 , pkg_resources is not used when a wheel is built and installed.