Proposal: Adding a persistent cache directory to PEP 517 hooks

So I started out writing the PEP only to realize that we don’t necessarily need this. The backends already can manage their own cache that is out of source tree and this can be non-ephemeral. PEP-517 even seems to address this, see PEP 517 – A build-system independent format for source trees | peps.python.org

The source directory may be read-only. Backends should therefore be prepared to build without creating or modifying any files in the source directory, but they may opt not to handle this case, in which case failures will be visible to the user. Frontends are not responsible for any special handling of read-only source directories.

The backend may store intermediate artifacts in cache locations or temporary directories. The presence or absence of any caches should not make a material difference to the final result of the build.

Would backends threat the source tree directory as read-only, and instead use platform-specific cache locations/temporary directories this issue would not manifest. Build backends currently go for building stuff inline, but they could switch to e.g. using appdirs · PyPI user_cache_dir instead.

Someone needs to manage the lifecycle of this cache/temporary directory, and now I’m not sure it needs to be the frontend (where this PEP would land). E.g. what would be pip strategy @pf_moore to when cleanup these cache folders?

This has become much more relevant as pip has switched to building in-place (and not via sdist, as I expected in my previous comment) and we are seeing people affected by read-only source directories.

I think the comment in PEP 517, and the long discussion when the PEP was being debated that can be summarised as “front ends should trust the backend to do the right thing” is relevant here. My concern is that the evidence is now suggesting that backends aren’t prepared to handle in-place builds, and we were being over-optimistic when the PEP was written.

I’m reluctant to restart that debate, as it was fairly difficult at the time, but I’d like to see some investigation done into how setuptools can address this without front end support (e.g., by working out how to support read-only build directories and using that approach for any other cases where “out of build tree” artifacts are needed). I’m not against front ends being asked to offer a cache location (pip is already caching stuff, so we can allocate an area in our existing cache for backends) but I do not want to see frontends being required to do cache management - a dumb frontend should be permitted to not provide a cache and the backend would use a temporary location, for example. (To give a concrete example, that’s what I expect pep517.build would do).

Specifically, pip’s strategy would likely be “do nothing to clean up caches, the user would have to do that manually” - which is mostly what we do with our current caches.

I’m saying “backends” here, but I suspect the reality is that I mean “setuptools”, as that’s the backend with a load of historical baggage that makes this a hard problem for them. It might be helpful to see if other backends (flit, enscons) have similar issues, though.

What about mandating the frontends to pass in a write-able working directory, however not mandating that this is persistent. E.g would be valid for a dumb frontend to pass in a temporary folder that gets cleaned up post-build? This would allow frontends to always have a safe path to write files to, but not mandates frontend to maintain it. Smarter frontends then can tackle the issue of cache management
to offers performance benefits. Some cache management strategies could be:

  • no management, always start with fresh folder;
  • create if not exists, never delete - leave it up to the user to handle that (maybe the frontend can have a --clean-cache flag),
  • tox could put stuff under .tox and etcetera.

Would that be a viable road ahead?

I’d be ok with that. Although I don’t see a lot of benefit in not having “not set” be equivalent to “fresh folder that the backend creates for itself”…

This probably doesn’t seem too hard for the case of read-only systems, if we’re already retooling to put the build artifacts in an arbitrary directory supplied by the front-end. That said, I haven’t really tried implementing anything like this.

To me the issue isn’t that front-ends must manage the cache in a specific way, just that backends cannot manage the cache. How would they? In an ideal world with perfect separation of backend and frontend, the backend isn’t even installed in the user’s python environment, and it only takes input from the user via PEP 517. If there’s going to be a mechanism for clearing the build cache or something of that nature, it’s got to be the front-end that provides it. “Cache is invalidated on every build” is a valid cache management strategy.

This feels dangerous to me if the cache isn’t stored in-tree. pip’s wheel cache is pretty simple to invalidate, because wheels coming from PyPI are immutable – if you got a file with a specific filename from PyPI once, you should always get the same file in the future. Maybe there are other caches that have a more difficult cache invalidation strategy.

For build backends, I’ve seen many situations where git clean -idx fixed some problem or gave me a reproducible build environment. If there’s no pip clean or similar, then I’m not confident that I’d be able to reliably know where backends are storing cached build artifacts. Even with a pip clean, maybe “cache goes in-tree if it’s there at all” is the only reasonable strategy, in which case the original issue is basically solved by the fact that pip does in-tree builds now.

Unlike with many other things, if setuptools is the only one affected right now, I don’t think it’s because of legacy reasons (except in a very minor way). Anything with support for compiled extensions is going to need a way to handle caching if it’s going to provide a reasonable developer experience. I think flit doesn’t support C extensions and AFAICT poetry punts on it by having you define a build.py that basically calls setuptools.

Agreed, and that’s essentially my point. I don’t think pip’s cache strategy is much use here, so I’d be inclined to have pip pass a temporary directory and in effect not cache at all. Or more easily for pip, pass None to tell the backend they can cache in a temporary location, because pip isn’t going to do anything else.

What I’m less comfortable about (because I’m not clear on the intended semantics of persisting this information) is whether the longer-term impact would be for front ends to get sucked into the question of how to handle incremental builds - which I very much don’t want to happen, as the backends are the experts here. So I don’t want to get into a state where backends say they can’t do incremental builds because frontends keep throwing away build artifacts, but frontends say they can’t do any better because only the backend understands how to invalidate the cache…

The whole point of this proposal was because the pip was building in a temporary directory and throwing it away, so it would very much defeat the purpose for pip to pass a temporary directory. We need the opposite – we need a place to put files to support incremental builds.

The point of the original proposal was to make it possible for backends to do incremental builds, not to get frontends involved. To the extent that front-ends are managing the cache directories, it’s in providing the user manual controls for overriding the backend’s default invalidation strategy.

tox is a good example for this – it has caching of virtual environments that are invalidated automatically under certain conditions (which would be the backend’s role), but you can manually override either by deleting .tox or by using one of the invalidation commands like --force-recreate or whatever (front-end’s role).

I think this may still be relevant if there are any front-ends that want to allow for incremental builds but want the source tree to remain intact. For example, I could very much imagine that tox would want per-environment build caches. Right now I don’t think it’s possible to do that, because setuptools can only ever cache anything in the source tree (anywhere else would be very undiscoverable and could lead to some seriously hair-pulling bugs).

1 Like

A valid caching strategy for a frontend would be to delegete it to the user. The issue at hand is that the backend does not communicate with the user. The frontend does. So sure, pip might decide not to implement providing a cache folder, but then it needs to allow a user to pass through a cache folder to the backend.

1 Like

I’m personally of the opinion that we don’t need to have pip support incremental builds for incremental builds to be a useful, supported part of the PEP 517 interface, The tox example is a good one, tox currently builds a sdist and then relies on pip to turn that into a wheel and install it. Maybe the better answer is for tox to invoke the PEP517 wheel build hook itself with a cache directory pointing to the .tox directory and then pip install the resulting wheel.

IOW, we don’t need pip to natively support caching or incremental builds. We can continue to build the entire thing from scratch whereas more specialized tooling can make a different choice. Remember part of the reason PEP 517 specifies frontends and not pip is the idea that we can have more than one front end, and a really good reason for a new frontend would be because the use cases between a general purpose tool like pip and a a more specialized tool like a tox wheel builder don’t overlap perfectly and need different options or behaviors.

3 Likes

I don’t say pip must have cache management, but pep 517 should offer the option. Currently it doesn’t. tox cannot build a wheel where the build backend is not in source because the PEP 517 interface doesn’t allow to specify the cache directory. This is the scope of this topic. We want to extend pep 517 with an optional cache pass in on the front end side, which is mandated to be respected by the backend if specified by the front-end (whoever that might be, pip, tox or anything else :man_shrugging::thinking:

I think we may not want to mandate that any cached artifacts should go in there, because that might be hard to pull off in some cases (like if your backend is running some other build tool like a compiler and you can’t necessarily control where it puts stuff), though maybe we can have language like “if a cache directory is passed, backends SHOULD store any build artifacts that would be useful to cache for incremental builds in the location.”

Are we not at this point back to the “trust the backend” dilemma that we hit with PEP 517 itself? (Sorry if people find that phrase confrontational - I don’t mean it as such, it just encapsulates well for me the idea that was discussed at the time that the PEP shouldn’t try to mandate behaviour from the backend other than that it produces the required wheel).

I understand that backends might not be able to control all the tools they run, but I do think the PEP should be clear that if the wheel doesn’t get built correctly because the backend can’t respect the cache, it’s the backend’s problem to deal with that.

C/C++ standards have an “as if” concept that I think is useful here - the PEP can say “backends must act as if all generated artefacts get stored there”, meaning that they don’t have to do it, but they have to hide that fact from front ends. Maybe we should use that idea here? I’d certainly prefer it over wording that meant that front ends could supply a cache, but were then expected to deal with the possibility that the backend ignored it (which is basically what “SHOULD” says…)

4 Likes

I’m happy with making it as if :smile: But there should be some mechanism to allow backends that can build out of source tree to do so, and allow the frontend to communicate this to the backend.

Yea sorry I was speaking more towards further up where there was discussions about pip somehow working around a mandated hook by passing in a temporary directory each time or trying to shoehorn pip’s own cache for this.

Just to clarify:

  1. I don’t see any problem in general with PEP 517 adding a “cache directory” option.
  2. I don’t think pip will be interested in using it (but that’s a discussion that should be had among the pip developers later).
  3. I think callers should be allowed to not specify a cache directory.
  4. I think backends should be required to behave as if everything is stored in the cache directory, if provided. I.e., front ends should be allowed to assume that if they supply a cache directory, they don’t have to write code to cover for it being ignored.

I’m not clear what the precise semantics of a cache directory are. I don’t really need to know the semantics from the backend’s perspective, but from the front end’s perspective, I would like to assume that if there’s a cache directory, then repeated builds using different cache directories would not interfere with each other. This would mean that front ends could safely do “in-place” builds. If that isn’t the case, I think it should be explicitly noted in the PEP, as I’m pretty sure people other than me will make that assumption.

Apart from the question of “in-place” builds, my assumption of what the cache means for front ends is “if you pass the same cache for two builds, the second build may be faster - but it’s your responsibility to ensure that exactly the same build is being requested; using the same cache for different builds is not permitted”. Is that accurate?

I’d like it if the semantics of not specifying a cache directory were defined as being the same as if the front end passed a temporary directory that was immediately deleted after the build. That makes “in-place” builds safe by default. If setting a build cache doesn’t mean that in-place builds are safe, I don’t care about this point, though.

I’d the impression Solving issues related to out-of-tree builds · Issue #7555 · pypa/pip · GitHub could use this.

Agreed.

Yes.

Wouldn’t this contradict how setuptools works now? It stores the cache in-line, so secondary builds would not behave as if they would be deleted after the first.

I was meaning a persistent cache. In-tree builds (the issue linked) could pass a temporary directory and delete it immediately afterwards, certainly. But that’s why I’d prefer to say that if the frontend doesn’t specify a cache directory, the backend should do this anyway - I see no real use case for not isolating build artifacts unless the frontend is aware of it (and will therefore help manage the cache).

Sorry for not being more explicit.

Yes it would. What I’m not clear about is who actually finds setuptools’ current behaviour useful. Given that most (?) builds go via pip, and setuptools’ current behaviour is a problem for pip, what I’m trying to find out is who actually wants setuptools’ current behaviour.

But it’s not a huge deal. If we want to say that backends can store artifacts in-place when the frontend doesn’t specify a cache directory, then pip will just specify a throwaway directory and we’re fine.

Be that might as be so, can we really introduce a breaking change for PEP 517, given we did not mandate any such behaviour until now?:thinking:

I don’t see this as “breaking” for consumers, as it’s just defining something that was left undefined previously. For backends, it’s a new feature, and so “breaking” in the sense that the default for the new argument doesn’t match the behaviour that currently occurs with the version of the hook that has no cache directory argument. But generally I think of “breaking changes” as being from the consumer’s point of view, so I don’t feel this is a major issue.

I thought this was mandating a change to setuptools anyway, as there’s currently no means of setting a build cache.at all. And of course, setuptools can continue behaving as it does now when not called via PEP 517, it’s just that PEP 517 builds will no longer pollute the source directory.

As I said, if anyone has a use for PEP 517 builds putting build artifacts in the project root, I’m happy to drop the idea. I’m not trying to argue strongly for it - I just think that “safe by default” is a better approach for frontends in general, and specifically for pip. (But even if we do add this, someone who really wants the existing behaviour could, of course, specify build_directory=project_root.)

Depending on exactly what you mean by setuptools’ current behavior and how pip plans to change, I think I have some examples where it is useful.

Taking as assumptions that the change to setuptools is that instead of putting build artifacts in the build root, setuptools will always put them in the cache directory, which if unspecified will be a setuptools-generated temporary directory, the options for pip are:

  1. Always pass None, in which case no build artifacts persist after the build is complete.
  2. Pass a build directory in the source root.
  3. Create a persistent out-of-tree cache directory somewhere, to be managed by pip.

I think that if you do the first one, you’ll have the same problems that this thread started with — incremental builds are impossible and no C extensions of any size will choose to use PEP 517. The third one is more workable, because the artifacts continue to exist, but it’s way less discoverable that any sort of build cache exists, which could cause some serious bugs. The second one, an in-tree cache, is basically what setuptools does now. It’s annoying for reproducibility (as is the third option), but at least git clean will remove it.

One thing I’ll note is that changes to this may affect coverage in C extensions. gcov has some assumptions baked in about the locations of .o files, in that they need to exist and they need to be in the tree. The current advice I see out there is to use setup.py -i build_ext or to use an editable install (neither of which is going to be viable in a PEP 517-only world). I decided to be a guinea pig for this in the reference implementation for zoneinfo, and found that the only reasonable thing to do in the world of out-of-tree builds was to copy the build files out of the temporary directory in my setup.py. If tox has a persistent build cache this won’t be a problem (because those files only need to live as long as it takes to run gcov anyway), but it’s suggestive that there may be other issues at play here. For example — how will this affect debugging with gdb if pip always passes None?

1 Like