I don’t think adding a bootstrap mechanism makes it easier to implement a backend compared to blessing the idea that backends can rely on pre-existing wheels. Maintaining a self-bootstrapping backend is complicated, and you have to worry about accidentally getting cycles if you take on any dependencies. By comparison, if you can rely on pre-existing wheels, all you have to do is build a wheel once, then you can rely on the last wheel you built to do your current build.
The only reason flit needs to use intreehooks (and thus setuptools) is because we’ve decided at the moment that pip install --no-binary :all:as it currently works, should build everything, including build tools, from an sdist, and that we want --no-binary :all: to continue working. It’s also the reason setuptools can’t use PEP 517 at the moment.
If we either decide that breaking --no-binary :all: is not a big deal or we modify the behavior of --no-binary :all: so that it will build from a wheel to satisfy self-referential build dependencies or some other solution that involves in “blessing” the fact that backends can rely on a wheel of a previous version existing, it actually becomes much easier to build a backend from source, even if the backend builds itself. The only requirement is that you either use an existing backend to build your first wheel or you do it manually, neither of which is terribly onerous.
I think it does, because I think the reality is that the second case is going to break in lots of interesting and weird ways for random contributors and will effectively make it harder to develop and contribute to a backend. It’s a deceptive kind of simplicity. It trades obvious up front work for non-obvious work that’s going to break in random situations that users and contributors alike are not going to expect. It’s the kind of misfeature that generates random bug reports that seemingly only happen in edge cases and are difficult to reproduce.
I also think that making --no-binary :all: mean “no binary, except for cases where we thought it might be some extra work” is a real poor UX for end users.
I’d like to (slightly) disagree here. --no-binary :all: is also sometimes used as a debugging tool/workaround (hmm, my distributed binary wheel don’t work for you—how about you try to install from source and see what happens, and use that before there’s a fix). Now of course listing packages explicitly might be a better solution on paper, but it gets extremely unwieldy very quickly when you have a reasonable amount of dependencies, and difficult for package users to follow that long option line. :all: is therefore commonly used when you don’t know where the problem is, at least in my circle.
I believe it is useful and better UX to have a wildcard meaning “all to-be-installed packages, not build tools” (most users don’t care, or even know about the latter). There is likely a place for “all packages, including build tools” as well, so it might be worthwhile to just have two wildcard names covering both cases.
I don’t think there’s any disagreement here. I think the option you’re suggesting makes sense. I think @dstufft is just saying he thinks the option you’re describing (or related variants) shouldn’t be called :all: because it’s not the maximal case. Otherwise, what would the option with build tools be called? :all_for_real:?
Let’s assume that the stdlib doesn’t have a TOML parser, since even if does eventually get one we won’t be able to count on it for some years yet.
Vendoring seems not too terrible; it’s not like the toml spec is constantly changing or anything. But this does raise an issue for bootstrapping: I’m guessing you might want to split up your source tree like src/flit for the actual package, and vendored-for-bootstrap/pytoml, which is present in the sdist but doesn’t go into the wheel. So, you’d want to somehow end up with both src/ and vendored-for-bootstrap/ on sys.path. How do you get there? One way would be to support multiple locations in backend-bootstrap-location. Or, if we only support one location, then flit’s backend could somehow detect that it’s in bootstrap mode and manually add vendored-for-bootstrap… but how would it detect this? I guess it could consult __file__ to determine that it’s being imported directly out of the source tree?
I’m not saying flit should be “the” root package. It’s just a useful thought experiment: if we can figure out how to solve the bootstrap problem for flit without using setuptools, then that would mean that it’s at least possible in principle to solve it for lots of packages (b/c flit is itself adequate to handle most pure-python packages), and without having to special-case setuptools.
Which is why I think we should be ignoring it, and focusing on actual use cases. Of which all we have so far are:
Distributions wanting to build from source (who have basically said what we do here won’t affect them).
Setuptools and flit wanting to self-host.
So IMO, at the moment any solution that satisfies flit and setuptools is sufficient. The converse question, though, is whether we need any solution, which is equivalent to asking whether “because setuptools and flit want it” (without any clear reason why) is enough justification.
If setuptools and/or flit can explain why they want to self-host, in terms other than “because it enables people to build everything from source”, that would be useful. If anyone with an actual use case for building from source, that is broken by PEP 517 as it stands, can speak up and explain their issue, that would also be useful.
But just arguing about what people might want to use --no-binary :all: for seems to be going round in circles. Certainly, if PEP 517 makes --no-binary :all: unusable in pip, that’s a bit embarrassing, but if we don’t have any clear use cases for it, we can deal with that (of course, saying we’re not planning on fixing the issue could draw out some actual use cases, but that would be a good thing!)
For Flit, I’d like the ability to self-host without a previous wheel, but it doesn’t particularly matter to me that I can do that using the PEP 517 interface; I don’t mind if bootstrapping involves a few manual steps.
If we do decide to say build dependencies can be satisfied using pre-built wheels even when --no-binary :all: is set, I propose that we say the --no-binary flag doesn’t apply to fetching any build dependencies. Restricting this to self-referencing build dependencies seems like an unnecessary complication, because:
It may not always be straightforward for the frontend to know the name of the project it’s trying to build (e.g. when you pip install .)
Installing wheels only for self-referential build dependencies still leaves you with problems if you have a dependency cycle.
If someone really needs to ensure everything is built from source, a flag that can use pre-built wheels in some corner cases isn’t really going to satisfy them anyway.
Unfortunately I think this does run up against at least one --no-binary use case we do know about, which is that some packages cannot be built as wheels. My understanding is that --no-binary is doing double duty - it’s saying “don’t use a wheel if it exists” but it’s also saying “don’t build a wheel as part of the install procedure”.
If you have build dependencies that cannot be built as wheels, having the --no-binary flag not apply to build dependencies would break your build. If the proposal is that the :all: selector simply doesn’t match build dependencies, but any explicitly-listed selections do, then I’m much more sympathetic. You could add another selector like --no-binary :all-build:, but I think it would be a damp squib, since it would be broken for all packages that ultimately depend on setuptools, which at the moment is all packages.
Presumably this is solved by the fact that you immediately recurse into a state where you do know the name of the project you are building, because you’re trying to satisfy a build dependency. If I do pip install . for setuptools and it tries to install setuptools from source, it may not recognize that the name is the same, but on the next recursion it will notice the name is the same.
That said, thinking about it more, we may not be able to get away with the “only have to break cycles of length 1” problem, at least not at the moment. I notice that if I add setuptools to setuptools’s build-system.requires, it fails building wheel, not due to some infinite regress or in building setuptools, because both setuptools and wheel depend on ['setuptools', 'wheel'] (wheel depends on these implicitly because it has no pyproject.toml). I think there are pretty simple solutions to this, but unless I’m way off the mark, it’s not as simple as I thought.
One more thing to note is that there’s a difference between PEP 517 saying you must resolve source build dependency cycles with wheels and pip deciding to resolve source build dependency cycles with wheels. If we specify it in PEP 517, all compliant frontends must do it that way. If we leave PEP 517 silent on the matter, frontends can do whatever they want (including not supporting non-wheel installs of build dependencies), though to be fair everyone’s going to optimize for however pip does it, so it will just become a de facto part of the standard at that point.
I don’t think this is a big problem for this purpose. This was a concern only for a few packages when pip first supported installing wheels, and none of us are so far aware of any specific packages for which it’s still an issue in 2019. So I think we can say that your build dependencies have to be installable via wheels.
(As with other questions, there may be cases we’re not aware of, but if we start by making such a restriction, we’ll discover those cases and then we can solve a concrete problem rather than an abstract one)
I probably don’t understand all the implications here, so I’ll just make a plea that a build-from-source option is always available. There are lots of downstreams that just cannot consume externally built wheels. In things like the compiler case, the bootstrap comes from the OS vendor, a trusted source.
What is the concrete use case for this? A corporation that for whatever reason wants to “build” the pure python wheels from source? As @encukou mentioned, if you’re using Fedora, you can use the system packages which they will build from source. I imagine most other OS vendors will do the same if there’s a demand for it.
Yes, exactly. We cannot use OS packages (for Python packages) and we cannot consume wheels from PyPI. The former is for technical reasons, the latter for legal reasons.
I think the agreement upthread is that this is an additional burden to place on the greater Python ecosystem that is best handled by corporations that can afford the additional infrastructure.
If you’re not consuming anything from PyPI, this is super simple to work around - either you are not using pip as your front-end, in which case your front-end can do whatever it wants to handle the bootstrapping, or you are using pip and you have a mirror (possibly a caching mirror) of PyPI, in which case you just need to manually build your own wheel once for bootstrapping purposes, and then you can build new wheels with pip using your bootstrapped wheel.
Alternatively, you can download a wheel from pip, unzip it and verify that it contains all the same files from the repo you would have built from, and then upload that.
I think the concrete use case we’re looking for is someone who 1. has a legitimate technical reason not to use wheels, 2. does not already have a their own infrastructure and 3. still needs to use pip. Preferably a volunteer project.
The key here is a solid reason why creating a local index, populating it with wheels of selected packages (minimally, only setuptools and wheel, presumably built manually from their sources) and then building using only sources plus that index, is not a feasible option.
It’s not that we’re against providing a solution, but knowing what’s being asked of us, and why, is critical, and we’re having real trouble understanding what’s wrong with the approach described above. (The argument “that’s too complicated” is difficult to sustain when you’re asking for non-trivial amounts of effort from volunteer projects to allow you to avoid that effort - so hopefully there’s something more concrete than that).
That’s probably us. I’d like to know more about the “manually build your own wheel once for bootstrapping purposes”. I’m sure we can figure that part out. It’s a little extra burden on our build system (since it’s something we’d have to implement – not always fun with our build system!), but with clear instructions from upstream, we can very likely adjust.
I don’t think the extra complication will be insurmountable, although we won’t start by populating our internal mirror with wheels. I think as long as the instructions for bootstrapping are clear – and well tested – we’ll be able to handle the change.
So where I work we’ve got some prior art on this and I’d like to argue in favor of a solution that builds the concept of in-tree hooks into the PEP. While the bootstrapping problem creates a hard requirement for it, I’ve seem a lot of other cases where having the ability to customize the behaviour of a build backend is the only reasonable way to achieve the build of a source tree.
Our build system looks a surprisingly large amount like PEP 517, but we’ve had the advantage of 15 or so years running it, and currently we have a massive amount of prior art supporting source-tree level provision of build logic. We achieve this in our case because ours are invoked over a process boundary, so our build system requirements are simply binaries placed on PATH and our in-tree build tool is a well-known path that any source tree can contain that is prepended to PATH before handing off build work to the designated backend.
We do this in part to isolate our entire build process from the system, as much as possible. We rely on the presence of a POSIX shell, because we’re pretty Linux-ey, but that’s very close to it. In the Python world presumably you’d be able to rely on the least-common-denominator of the stdlib to make this work.
The benefits of this are pretty clear, from my perspective. No individual backend author needs to solve the self-hosting problem; it’s something built into the spec, and every build frontend needs to understand it to be a complete one. It neatly solves the bottom-of-the-graph build backend problem, for an arbitrary number of backends. In addition, it provides package developers with a clean place to experiment with their own variations on build backends. I’ve lost count of the cases where all I wanted to do with a build tool here was to mix it with another one and combine them in a slightly different order than “Run everything in this one first, and then that one” or “write my own and ship it as an entire separate package”
Lastly, I want to add my voice to the “please do not make a solution that relies on a binary blob to bootstrap”. For similar reasons to @barry above, we can’t do that. The right thing to do, IMO, is to have the PEP517 spec include an in-tree backend model. How you do that is up to you, of course, but I think it’s pretty damn useful.
(for what it’s worth, I rather like the idea of a well-known module that can only be imported from the source tree, but that’s me)
