PEP 517 Backend bootstrapping

pganssle · January 30, 2019, 1:19am

Per this comment on this pip issue, and tracked in setuptools in issue 1644, the new PEP 517 roll-out has caused the problem that it seems that no provision was made in PEP 517 for custom backends to be provided by the project itself.

This is primarily a problem because it means that PEP 517 backends cannot bootstrap their own builds. The biggest problem at the moment is that there is nowhere for setuptools to put its build code that works with PEP 517. We could declare a dependency on setuptools in pyproject.toml, but this only works when you are able to get a setuptools wheel from PyPI. If you specify --no-binary :all:, you will try to install setuptools from an sdist, and if PEP 517 is enabled that will fail.

As far as I can tell, once we ship a release of setuptools that contains pyproject.toml (assuming it adds setuptools to the build-requires), it will become impossible to use pip install --no-binary :all: for any project that uses PEP 517 and uses setuptools as a backend. Regardless of the advisability of using --no-binary :all: I think this is a major problem generally, and specifically it’s a total pain for setuptools, which is one project that really needs to be able to build itself from source.

I propose that we add some mechanism in pyproject.toml for a project to specify that it wants to use a pre-built PEP 517 backend that it is supplying. Probably this would be used only by setuptools or other PEP 517 backends that want to bootstrap rather than using setuptools. A few options for what this could look like:

Add the ability to specify build backends relative to the project root, so for example,
setuptools could specify .setuptools.build_meta instead of setuptools.build_meta.
A dedicated “bootstrap backend” module to invoke, e.g. __build_hooks__.py
An option to specify the current project root as a dependency in build-requires (which would basically have the effect of adding the CWD to the Python path).

I personally like option 1. Option 3 is basically the same thing as option 1, except with sloppier semantics.

I put this here rather than on the pip tracker because as far as I can tell, this needs either a new PEP or a modification to PEP 517. If we can do it without that, I’d be perfectly happy with that as well.

dstufft · January 30, 2019, 1:45am

FWIW I don’t think the --no-binary :all: thing is directly related here. It is my understanding that as of right now, pip simply doesn’t support PEP 517 unless it is installing the backends from wheels. Effectively sidestepping the entire bootstrapping process by saying you have to figure it out yourself… which generally works but does break down in the case of --no-binary :all:.

Given that --no-binary :all: is an installer specific flag, I think the first question we need to decide is whether or not we want to handle the bootstrap case, or if we’re generally happy with the idea of saying that to bootstrap a build backend you have to either have another way to produce wheels, or you need an existing version of that build backend (or another build backend) to produce a wheel first.

This isn’t an unheard of solution in general, most compilers basically work this way where they either have a mini compiler that can bootstrap the compiler just enough for the compiler to build itself, or they have an old version of the compiler around to build the new version of the compiler with.

Ultimately I think that any of the soluutions we pick are ultimately going to boil down to the same rough idea, some way to get a “phase 0” build backend available, so I think I am personally OK with the idea of making it something that can be automated.

That being said, part of me kind of favors choice (2) from above.

My problems with option 1 is basically that I’m not a huge fan of the fact that a one character difference can drastically alter the behavior of the build backends (given we’re going to have to add CWD to sys.path in this example).

My problem with option 3 (and really this applies to option 1 as well) is that it feels like a general case solution to a specific problem. Most projects are not going to need support for this (or shouldn’t be using it) and I think we should try to pave the cowpath away from it instead of towards it.

So with all of that, I think I prefer option 2 where the build backend string is some sigil that indicates to the installer it needs to use a bootstrap build backend. Maybe something like:

[build-system]
requires = []
build-backend = "<bootstrap backend>"

The nice thing about that, is it is an invalid import key, so we know there can’t be any collision with a backend available elsewhere. It’s also something that looks special enough that I think most people will shy away from it unless they really need it.

Of course, nothing we do here can solve a “from source” dependency cycle (e.g. if we’re installing setuptools from source, and it build-depends on packaging, and we install it from source, and packaging build-depends on setuptools, we’re stuck) but that’s something that authors of build backends are generally just going to have to cope with I think.

njs · January 30, 2019, 3:16am

I still think the simplest solution is 4., add support for

[build-system]
# source tree-relative paths prepended to sys.path before invoking build backend
pythonpath = ["."]

It’s trivial to spec/implement/document, and will come in handy for non-bootstrap use cases too (e.g. it would have provided a workaround for at least some of the folks broken by the latest pip release).

We discussed this back when putting PEP 517 together in the first place, and it got cut for simplicity. Which is fine with me. But if we change our minds and decide that we do need something like this after all, then this seems like a simpler solution than 1-3 above. (And that point about folks who want to avoid pre-built binaries is kinda compelling, I don’t know if anyone brought that up before. I’ve heard about multiple corporate users who are picky about that.)

pf_moore · January 30, 2019, 10:54am

That’s not precisely true. Pip supports installing from source when setting up build isolation (there was a lot of work that went into handling that, avoiding infinite regression causing fork bombs, etc). I honestly didn’t fully understand the final solution here (I assume that if pip detects an infinite regression, then at some point it simply aborts, but I haven’t looked at the code to see how it does that). Installation of backends builds on that, so installing backends from source should be fine.

The problem is that backends that build themselves naturally generate that “infinite regression” case that can’t ever work. It’s a straightforward and well-known problem - self-bootstrapping compilers have been around for ages.

The fundamental question here is how we halt that infinite regression, in the specific case of a PEP 517 build backend.

It’s worth pointing out here that this is an extremely specialised issue. None of the mechanisms we define here should ever be needed by anyone who isn’t writing a backend. The backend mechanism is extremely flexible, and should cover basically all of the use cases for non-backend projects. If you want to ship a project-specific backend, use a backend (from PyPI) that loads custom hook code from a project-specific location. If you want the project directory on sys.path, use a backend that modifies sys.path and then redirects to your real backend. Etc. It’s going to take time for projects to realise that this is an option, and adopt it, but we should promote that approach.

OK, having said all of that, we need to decide how to write a backend that doesn’t need any other backend to build from source. At the end of the day, that’s the only way to terminate the infinite regression of backends in the --no-binary worst-case scenario. And the mechanism should be clearly targeted at backend writers, not at general projects.

So how about this? It’s essentially @pganssle’s option (2), just described in terms of the above background.

Projects may declare in pyproject.toml that they are a self-hosting backend, by setting build-backend to <self hosting>.
Self-hosting backends MUST supply a file _self_hosting.py in the project root directory that defines a set of build backend hooks.

The choice of the “self hosting” terminology, and the definition of a fixed filename for the hooks, is deliberate, to emphasise that this is not intended as a general mechanism.

Typically, I’d expect _self_hosting.py to import the actual backend being defined, probably by explicitly adding os.path.dirname(__file__) to sys.path and then importing from the project source. But the details are up to the backend.

IMO, that’s too general. Maybe it’s a good idea, maybe it’s better handled in the backend (is sys.path even going to be relevant to any ultimate backend other than setuptools?) or via a wrapper backend as I suggested above. But I’m wary of treating this issue as anything other than a highly-specialised problem specific to backend authors. Sure, the sys.path solution addresses this problem, but it also opens up a lot of other, more complicated questions which are irrelevant to the case of self-hosting backends. (Do we allow absolute paths? What about ..? How about allowing zipfiles? Or some other path entry that needs a custom importer? I’ve long ago given up assuming that an idea is too daft for anybody to consider using it, I’m afraid ) I don’t want the process of finding a solution to the specific issue of self-hosted backends to get sucked into those sorts of digression.

takluyver · January 30, 2019, 1:34pm

For Flit, I was able to solve this (or work around it, depending on your perspective), with the aid of a tiny package called intreehooks, which lets you load a backend from inside the source tree.

Intreehooks itself is packaged with setuptools, to avoid a circular build-dependency. But of course there would be a circularity if setuptools itself relied on it to build.

So I guess we can either:

Say that either setuptools or intreehooks is a special case which needs to be installed to start bootstrapping a build environment.
Specify some additional general mechanism for allow loading a backend from the source tree, without needing a separate helper package. Of the proposals so far, @njs’ option 4 is the one that seems clearest to me, but maybe there are other use cases I haven’t thought through.

pganssle · January 30, 2019, 2:19pm

To clarify option 1, I was not suggesting that we add the project root to sys.path. Instead I was suggesting something more limited. I don’t know the import machinery well enough to know how easy it would be to do something like that, but the idea would be that we would resolve the import of the build hooks as if CWD is in sys.path, but when the hooks are actually imported and run, sys.path would not include the project root. In that case, option 1 is the same as option 2, except the name is not specified in the spec. I don’t think it makes much difference, but I tend to prefer not to add special names.

I think the infinite regress problem goes away if you specify that the PEP 517 backend bootstrapper must work in-tree - whether you build it ahead of time and ship a binary artifact in your sdist or (much more likely), you ship an importable script as part of your source distribution. The main difference between this and the “PEP 517 backends can’t use PEP 517 to build themselves” situation is that at some point, all source distributions will need to be built with PEP 517, so there will be no way to build a PEP 517 backend entirely from source. With a bootstrap script, you are able to provide a PEP 517 backend invoked from the source distribution, thus terminating the infinite loop.

In fact, the current situation is the one that has an infinite regress. If setuptools specifies in its PEP 518 build requirements that it depends on setuptools to build itself and you specify that the build must happen from source, you’ll infinitely regress, creating deeper and deeper layers of build isolation until you run out of disk space or something.

That said, I think the rest of your post basically agrees with what I said here, but I wanted to explain it to make sure we’re on the same page.

Yes, this was one of my first thoughts on how we could do this, but as you mention, there has to be a PEP 517 backend at the bottom of the stack, so it always comes back to “how do you build setuptools from source as part of PEP 517”, just with one level of indirection.

I think making either setuptools or intreehooks “special” is a valid solution, but it does go against the spirit of PEP 517/518, which is that there shouldn’t be a “special” build backend anymore.

What we’re discussing here - making it so that you can specify in pyproject.toml that your build hooks exist in-tree is essentially the same thing as making the intreehooks backend special, and if we’re doing that then we might as well just vendor intreehooks in pip and give ourselves a special syntax for invoking it.

steve.dower · January 30, 2019, 2:22pm

Could pip provide a backend for this scenario?

takluyver · January 30, 2019, 2:25pm

You mean something like intreehooks, which allows a project to load a local backend? It could, but if we’re adding it to pip, I think it’s worth standardising an interface for local backends so that other frontends aren’t dependent on some pip-specific functionality.

ncoghlan · January 30, 2019, 2:43pm

(Note: PEP 517 is still provisional precisely to allow us to fix this kind of previously unforeseen problem without needing a complete new PEP. I do agree the PEP as it currently stands doesn’t let us resolve it, though)

Like @pf_moore, adding pythonpath=["."] seems overly general to me, as we’d have to then add a bunch of additional rules around what kinds of paths were permitted (it might be enough to restrict it to paths relative to the repository root with no .. entries, but still, ugh, dealing with anything path related in a cross-platform spec is spectacularly evil).

I quite like the semantics of @pganssle’s Option 1, but would resolve the “Semantically significant syntax should not look like grit on Uncle Tim’s monitor” question by making the marker a separate build-system entry:

[build-system]
requires = ['setuptools']
build-backend = "setuptools.build_meta"
self-bootstrapping-backend = True

When self-bootstrapping-backend is True then the frontend would need to make two changes:

Ensure the source directory is on sys.path when looking for the build backend
Ignore any reference to the project’s own name in build-system.requires

This is pretty similar to what @pf_moore suggested, but there are a couple of key advantages to leaving the build-backend field alone and keeping the project’s own name in the requires field:

The above build-system spec should still work on pip 19 (and any other PEP 517 installer), as long as there is a setuptools wheel file available. It’s only the “no wheel” case where the installer will need to understand the new flag in order to break the infinite regress.
With the build-backend field available, then that can be used as normal to tell the frontend where to find the hooks within the repo, rather than having to define an entire separate protocol solely for bootstrapping.

I’ll note that if we do go down this path, then it’s going to require a long-lived frontend capability to inject the source directory as sys.path[0], which means some of the temporary local hacks I made in https://github.com/pypa/pip/pull/6210 would need to be re-designed to be backed by fully supported API features in the pep517 package (probably a bootstrap flag of some kind).

pganssle · January 30, 2019, 2:54pm

I chose .setuptools.build_meta to mirror the way relative imports work in Python now. We could easily pick something like @.setuptools or $PWD/setuptools or or {BOOTSTRAP: setuptools.build_meta} or something of that sort. This is really only a problem for projects that want to be at the absolute bottom of the install stack. I don’t want people using self-bootstrapping-backend=True as a way to hack around the fact that setuptools.build_meta doesn’t have CWD in sys.path.

Option 2 is the absolute most minimum thing we can do: add a specific file name that will supply a bootstrap PEP 517. Option 1 is basically a different flavor of Option 2 that allows you to specify a single module that will be imported and provide your bootstrap backend. We could use any kind of syntax for it, but I don’t like any ideas that involve injecting CWD into sys.path as a matter of course.

dstufft · January 30, 2019, 2:58pm

I don’t think you can reasonably get away without injecting CWD into sys.path for this kind of in tree backend. The import system doesn’t really support that use case natively (though you could maybe hack it with a custom import hook?).

takluyver · January 30, 2019, 3:04pm

Is it necessarily clear to a build frontend what the project’s name is at this point? If you do pip install ., there’s no package name given, and the PEP 517 interface doesn’t include any ‘who are you?’ hook to ask the package for its name. As neither setuptools nor Flit currently list themselves in build-system.requires, maybe this is solving a non-problem?

I see that, but I’d avoid that syntax because it’s not (I imagine) doing a relative import. It’s an absolute import from the CWD. A relative import would have some subtle differences, e.g. the __name__ inside the loaded module.

I think it’s feasible to look up an import without modifying sys.path if you need to. But the recommendation is to load and run the hooks in a temporary subprocess. So I don’t think modifying sys.path is a big problem.

pganssle · January 30, 2019, 3:10pm

I was thinking a custom import hook is likely to have the least “surprising” behavior, but one example of how this could work is that a frontend would copy the relevant files into a temporary directory and put that on the PYTHONPATH. That’s easiest to do if the hook is a single file, but it’s not impossible to do with even a submodule in a package.

I think one relevant design consideration is that this does not need to be easy, and we probably don’t want people using it because of its side effects. If we have something with incredibly limited semantics, setuptools can do an implementation and everyone who wants their backends found in-tree can rely on intreehooks.

Yeah, totally fair. I am fine with any syntax, but I would really prefer it if the specified module and only the specified module were available to be imported from the hooks. The bootstrap hooks might add things to the path as necessary, but it shouldn’t be the default.

Does the subprocess not inherit the sys.path from the parent process? The main thing I’m worried about with respect to “find the hooks in the CWD” is that it will be used as a workaround to make the path work correctly for setuptools.build_meta. Admittedly this is less of a concern if it the only way to activate “CWD mode” is to write an in-tree PEP 517 build backend.

ncoghlan · January 30, 2019, 3:18pm

It’s late enough here that my brain cross-wired “What’s in flit.ini” with “What’s in pyproject.toml”, so I’m going to go get some sleep

But yeah, you’re right - ignore that part of my comment.

takluyver · January 30, 2019, 3:21pm

I don’t like that idea. I think the module specified should be allowed to import other local modules if needed, and figuring out which modules need to be copied is a tricky job. If there’s a way to hook into executing local Python code, that code should have the full capabilities you’d expect, not be limited to one .py file.

Yes, but modifications in the subprocess don’t affect the parent. I thought the concern was about modifying global state (sys.path), but it seems your focus was elsewhere.

Ah, I see what you mean. In that case, requiring a module with a specific name might be reasonable. Or making sure that the specified backend can only be imported from the CWD.

But if we think people will want the CWD on sys.path for other projects, so that e.g. the setup.py script can import their module, maybe we should allow it, rather than trying to ban it? Then again, a setup.py script can easily do sys.path.insert(0, '.').

pf_moore · January 30, 2019, 3:38pm

We need a solution here that isn’t pip-specific. The problem is frontend-agnostic so the solution needs to be too. One option I thought of was that the PEP could mandate that the frontend had to provide some sort of minimal backend, but in practical terms that would mean the PEP mandating a reference implementation that every frontend would have to copy and paste into their codebase somewhere. Which sucks as a solution.

pganssle · January 30, 2019, 4:02pm

I also don’t like the idea of copying to a temporary directory, it’s just an existence proof that you don’t need to play around with the PYTHONPATH if you don’t want to. If we go down this route we should explore how easy it is to import a module from a specific location without adding that location to the path by other means.

I think it’s fine to import other code in the local directory, but it should be done explicitly by the backends if they want to do it. In the original syntax, if you want to import other files, the easiest way to do that is to put them in a package and use relative imports. If we enforce that it’s a single file, then people can append or prepend CWD as desired.

Indeed I don’t think we should try to ban it, I just want it to be done explicitly. If someone discovers that their stuff works when they do bootstrap_backend=True, we may end up with a bunch of files that are improperly relying on some implementation detail of what should reasonably be an arcane bootstrapping mechanism for PEP 517 backends.

It’s probably an overwrought concern, and mostly obviated by going with Option 1 (modulo syntax), regardless of whether Option 1 ends up adding CWD to the path.

steve.dower · January 30, 2019, 4:36pm

I’m just a little concerned about the suggestions that this doesn’t have to be easy, as part of the point of PEP 517 is to let other build systems play in setuptools’s traditional space. One of my future projects will be to build a backend I’m sure (unless I can get away with just doing the extension build part and get another backend to ship it).

Friction in building and testing a backend is only going to deter people who are ultimately trying to provide something for free. There’s no fame in packaging

dstufft · January 30, 2019, 4:59pm

I don’t think we want it so much to be hard, but to be something that is very obviously a special case for a subset of projects that most people do not need to deal with. So while we don’t want it to be overly hard to do (after all, we’re the folks most likely to need to use it), we want it to be less obvious than the preferred mechanism for folks.

pf_moore · January 30, 2019, 5:17pm

Precisely. I view this as a very specific niche case, for developers of backends, and I want it to be easy for them, while not being something that people will try to use for building “normal” projects.

Personally, I’d like to see a thriving ecosystem of backends. For that to happen, I’m strongly in favour of making it easy to build backends - but not so easy for projects to bundle a custom backend with their code. The norm should be that if you have a need for special build processing, create and publish a backend, so that others can benefit.

Bundling a backend with your project is just another way of running arbitrary code in your build process, and we’re trying to move away from that towards more declarative build processes.