Why would any of these more experienced users, that still want to
build from an sdist, for their platform or for other
aforementioned reasons, who know how to compile C extensions,
object to setting an environment variable beforehand (or reading
the docs)?
In my experience, I doubt they’d object, it’s just reaching them to
let them know that’s hard. Breaking existing workflows causes pain,
all the worse when the user doesn’t see it coming. These days, it
seems like it’s about a 50/50 chance at each minor pip upgrade that
people will come running to me complaining their CI/CD jobs began
cryptically failing even though they “didn’t change anything.” At
least I keep an eye on pip beta and release announcements,
discussions about upcoming changes, et cetera. I imagine it’s far
worse in communities/organizations who don’t pay closer attention to
these things.
While reserving my opinion on whether this is the right approach or not, I just want to point out that --build-sdist-for should be additive. One strategy may not work for all listed packages, so you might have different buckets of packages for which different strategies apply. If you’ve been thinking that already, it’s not explicit in your proposal.
This seems like a reasonable fallback, but I still question whether asking during interactive pip usage will just be too painful to enable. If you’ve got a huge cascade of transitive dependencies that need building (let’s say because they are pure Python sdists with no wheels), then I know I’d probably get pretty frustrated after a half dozen of them or so, then quit out and trying something different.
I don’t know that batching the build-asks is feasible though, given the way dependency resolution works.
I did think precisely that. Sorry, I thought it was clear.
In particular, aside from the prompting, I would be adding the ability to prefer source (but allow binary) for some packages, and the opposite for others.
Although it looks quite challenging, given the existing design of the internals.
Does Pip ever actually need to build or install a package in order to inform the resolution process further? Is it not entirely driven by the separate metadata files yet?
I agree that a large number of such prompts could be frustrating. Unfortunately I can’t think of anything better to offer than the “cancel the whole thing and figure out something else” option. (Well, aside from advertising --build-sdists=when-needed etc.)
No it is not. If you have an sdist, dependency resolution requires building today still. It is possible for same sdist to have multiple build behaviors that lead to wheels with different dependencies (any logic inside setup.py that inspects environment/cuda for example). Pytorch would theoretically be one although pytorch avoids uploading sdists at all given most people will not be able to build it.
If pip is planning on using a sdist, the only place it can get metadata from is by asking the backend (this may or may not be a full build, depending on the backend). There is no separate metadata file for a sdist, just for wheels. And even if there were a metadata file, sdists are allowed to flag metadata as “dynamic”, which would mean the installer would still need to ask the backend.
Mm. In that case it would make sense to warn about that possibility, unconditionally, in the pre-build prompt. The cases where Pip could verify an sdist’s requirements before building are ones where there is no setup.py (and an explicit listing in pyproject.toml), and those likely have no good reason not to have wheels anyway.
That’s not accurate. Assuming PEP 643, it is possible for an installer to read the PKG-INFO file from the sdist, confirm that it’s metadata 2.2 or later, confirm that Requires-Dist is not included in the Dynamic metadata field, and then reliably use those dependencies without doing a build.
But if any of those conditions isn’t true, you have to ask the backend. The setup.py file isn’t relevant, nor is pyproject.toml.
Hold on - are you really saying that the build backend is allowed to just override an explicit dependency list in pyproject.toml, according to its own logic and without needing a setup.py?
yes. The requirements are for the build backend to decide what to do with. currently, there are a few interesting backends solving real problems using that, see: nvidia-stub · PyPI
No, PEP 621 states that data in pyproject.toml is canonical. So a standards compliant build backend must put the same data in the sdist. But:
If a field is marked as dynamic, the backend can calculate the value how it likes.
If the backend isn’t standards-compliant (and you have no way of knowing if it is) then anything could happen.
Add to that the fact that PKG-INFO is just as easy to read from the sdist, and is definitive in more cases than pyproject.toml is, and why would you use pyproject.toml except as a possible fallback if PKG-INFO declares that it uses a metadata version older than 2.2?
Not true, actually - PEP 621 states
When specifying project metadata, tools MUST adhere and honour the metadata as specified in this PEP. If metadata is improperly specified then tools MUST raise an error to notify the user about their mistake.
I’m not sure what the link to nvidia-stub is supposed to demonstrate, as the pachage doesn’t provide a sdist, and there’s no link to the project source where I can look to see how the wheel is built…
It’s a clever hack, but a hack nonetheless, and is practically speaking the only way to support the use case today. Even short of solving all the other problems with variants once and for all, it might still be useful to define such “trampoline” or “stub” mechanisms in a more principled way.
I think the idea is that nvidia-stub is itself the backend, and the Python source therein demonstrates an interesting thing that Nvidia is doing with metadata. But I couldn’t easily assess how the “interesting thing” works from a quick read through the source, anyway.
While Non-compliant backends can do a lot, compliant ones can still add additional dependencies (Both buildtime and runtime), I don’t think pip or uv have a means of detecting a non-compliant backend outright replaced one package with a drop-in replacement that used native code, and even within compliant backends, that doesn’t mean that there aren’t other dependencies than what’s listed there that the backend could use, which would still lead to a differing list of actual dependencies, just in that case, that differing list is an extension of the original and dynamic, not discoverable without the backend.
nvidia-stub is a clever hack. It’s a backend, and you can unzip the wheel if you want to look at the specifics of it, I actually thought more people would be aware of what it is and does.
It results in modifying metadata during build, and downloading from https://pypi.nvidia.com during install
Ah. I’d avoided mentioning non-compliant backends because I thought the response would be “it’s OK to ignore things that don’t follow the rules” (which I’m not sure I agree with but I didn’t want to derail the conversation by arguing about). But if there are important non-compliant backends already in existence, then supporting them becomes an important part of the transition plan for any new approach.
And I hadn’t thought of the interpretation of PEP 621 that adding dependencies was acceptable. I’m not sure I agree with the interpretation, and it’s probably something that ought to be clarified at some point, but again if it’s something that’s happening in the wild right now, it needs to be considered in the proposal.
I was aware of it, but wasn’t clear how it was intended to be used. And I couldn’t see the source from the PyPI page, and wasn’t interested enough to download and unpack the wheel to see how it related to your point.
If it’s standards-compliant, I’m willing to call it a “clever hack” (and I’d expect this proposal to support it somehow). If it violates standards, though, I’d call it an abuse that we’re free to break if needed. As I’m not going to bother checking the details, I’ll just describe it as “a bloody nuisance that we have to live with” for the moment
My reading of it allows it (it doesn’t appear to be disallowed, all of the required behaviors still work with it), but I think that both this, and the potentially non-compliant uses that people would be tempted to use to solve problems the ecosystem isn’t solving more directly for them both might be an indication of the kinds of things people feel the ecosystem’s specifications fall short on. which…
I don’t think nvidia-stub specifically is compliant [1] With that said, I’d avoid looking at this as something which should be broken given it’s currently serving a need, but that’s a personal view, and I’d understand the perspective of telling people to use --index-url correctly instead here too.
I also don’t think there’s really a way to detect what a backend is doing or mandate that other tools do without imposing some much more restrictive requirements, like backends needing to be run in an isolated context managed and observed by the installer/built tool.
So speaking practically, I think that in the case you outlined here:
If all of that is true, using those dependencies is fine, but otherwise, I think we should lean that it’s possible that a build backend has other dependencies it uses, even if only at build time[2]
I also don’t want to rules lawyer if it is right now, it essentially grabs the expected package from an unexpected source. ↩︎
If a backend outright lies in metadata, I think we can ignore that pathological case and tell users to stop using broken backends ↩︎
I don’t think it technically violates any standards, although in practice it could because the wheel that gets installed as a result of “building” the sdist isn’t produced by the sdist, so it’s theoretically possible that metadata won’t match (I haven’t check to see if that’s in fact the case or not).
However, I don’t think build_wheel() was meant for the purpose that nvidia-stub is using it for, which is to do some extra platform checks, and download from a known external repository, the wheel that satisfies the dependency. Until we have a workable variant solution, both to resolve fully compatible wheels and to reduce the size of wheels to within PyPI limits, we need to continue to allow this mechanism to work. It’s one of the reasons why I emphasize that sdist “builds” must be allowlisted for individual packages, as opposed to one binary-vs-sdist strategy to rule them all.
Why not? From the POV of the caller, it’s no different from build_wheel invoking a compiler that looks for system installed libraries. The resulting wheel is just as likely to be platform-specific, regardless of what platform it claims to be.
If nvidia-stub downloaded the files from individual sites, assembled them into a wheel layout locally, and then zipped that up into a wheel, would it make a difference? Clearly not.
The biggest “problem” with their build backend is that there isn’t also a reliable way to create your own wheelhouse (without reverse engineering the backend to figure out what criteria matters for choosing the wheel’s contents). But it isn’t that different from having to match libc version or some other native library’s location between where you build and where you install.
To an extent, I’d disagree. The problem is that you can’t set up a local (isolated) build environment. To work, the build environment needs access to the internet, specifically to the online copies of the opaque binaries that will be assembled into the final wheel. To me, at least, that feels contrary to the idea of “building from source”.
You can easily create a wheelhouse, just by using pip wheel (for example). What you won’t have, by doing so, is a portable wheelhouse - but that’s inherent in the idea that the wheel is built to match the GPU setup on the build system.
But it’s a grey area. I don’t think anyone would argue against the claim that what the nvidia-stub backend does doesn’t match what people intuitively expect from the phrase “building a binary from the project source code”. And there’s clearly a need for some solution here, so we have to decide whether we prefer to support this approach, or offer a different solution (or both, but that seems unnecessary - the use case is important, but not common).
That’s fair, but it’s also pretty orthogonal to the problem they’re solving (people just pip installing from PyPI and not getting the right thing). Internet access is already presumed for these users.
Also, you’d still have to set up your isolated build environment using things from the internet. Now it just includes some pre-built wheels, rather than whichever build tools would’ve been needed. A problem it may be, but it’s not in any way a new problem.
Sorry, there’s a lot of different threads ongoing, all discussing slightly different aspects of this issue. I was thinking in terms of the question of distributors trying to set up a “build everything from source” workflow. That is an issue with nvidia-stub, but it’s not relevant to this particular discussion.
