For the specific case of Red Hat, there’s already a multi-step bootstrap for new Python stacks in order to get initial bootstrapping binary RPMs built without docs (to break the circular dependency on sphinx et al), without tests (to break circular dependencies on pytest et al), and without ensurepip (to break CPython’s own circular dependency on setuptools and wheel). Only once that core library set has been constructed does it then go back and rebuild them all properly, using their limited-but-good-enough-to-self-host bootstrap variants to build the fully functional versions. (For the curious: rpm-list-builder/python37.yaml at python37 · hroncok/rpm-list-builder · GitHub )
I’d expect any organisation with a “we build everything from source” policy that extends all the way down to core language ecosystem tooling to have a comparable technical capability, and if they adopt the policy without investing in the capability, I agree that’s their problem, not ours.
That means I’m a fan of the idea of making the enhancement here a way for projects to indicate that build requirements must be resolved from the wheel archive, and not built from source, even if a frontend has been asked to build everything from source.
That then makes the key remaining PEP 517 question on the bootstrapping topic the following: should we specify exact conditions under which a frontend MUST consider a build dependency unresolvable and fall back to installing from an available wheel archive?
In addition to promoting consistency across frontends, the benefit I see to our doing that is that projects like pip that need to modify the behaviour of options like --no-binary to still allow binary build dependencies will be able to point to the relevant section in PEP 517 as the rationale, rather than having to defend themselves to their users on a case-by-case basis.
The first clause I think we should add is something like:
Frontends MAY be implemented such that all declared build dependencies are installed from binary wheel archives and never implicitly built from source.
That lowers the barrier to building compliant frontends (those frontends just won’t be able to build some projects, for the same reason pip 18 couldn’t build them).
And then the second clause would be something like:
Frontends that allow for declared build dependencies to be implicitly built from source MUST fall back to instead installing from a binary wheel archive if the project’s own name is listed in
build-system.requires(regardless of any other settings that have been given to the frontend).
Given such regression terminating behaviour in pip, both setuptools and wheel would terminate the current infinite regress, since they’re implicitly added to build-system.requires if there’s no build-backend set.
[Reordered this to put it after the wheel related discussion, as I now doubt we’re going to go in the direction of explicit self-bootstrapping, and instead rely on binary archives to terminate regressions (the same way RPM et al already do).]
I still don’t understand why you view this approach that way. The only code that runs in the backend process before the backend’s own code is frontend wrapper code, and even a modified PEP 517 would still tell frontends to keep in-tree paths out of sys.path unless bootstrap-backend-location had been set. Thus even without any help from the front end, backends would be able to infer the use of the setting (or a functional equivalent like the intreehooks meta-backend) from the presence of in-tree paths in sys.path.
It would also be possible to make the PYTHON_BOOTSTRAP_BACKEND_LOCATION environment variable a defined part of the PEP rather than a pep517 library implementation detail.
That said, I think we can put aside that entire digression, as I agree with you that having the ability to force the use of wheels for build dependencies is a more promising direction to resolve bootstrapping issues.