Currently, the PEP 517 can specify a wheel_directory, and I believe it is guaranteed to be executed from the repository root (though I think pip first copies your directory into a temporary directory so the “repository root” is ephemeral anyway), but there is no standard way for front-ends to pass a non-ephemeral location other than the repo root to the backends.
One problem this causes is that setuptools will generate a bunch of build detritus directly into your repo root like build/ and/or lib/, which is not a great practice. I think this also makes it so that it is difficult for pip to cleanly implement any sort of incremental builds, because the only available location for the build detritus is the repo root.
There is open issue in setuptools to allow moving the locations of these folders, but I am worried that this will end up with tox or other front-ends writing setuptools-specific code to pass these options in order to avoid polluting the local development environment. I think we can solve both this problem and the “allow incremental builds” problem by adding a new “persistent cache directory” which is created by the front-end and passed to the back-end. The idea would be that the front-end should create a directory where the backend can store expensive-to-create objects that should persist between builds. It is up to the front-end when this cache is cleared.
I think there are two options:
We modify the hooks in the PEP 517 build interface to add an optional cache_directory parameter, like so:
For backwards compatibility, I think backends that support this feature would maybe have to specify something like backend.__SUPPORTS_CACHE_DIRECTORY___ = True.
We add a top-level configuration function that can be called prior to calling a build hook that would configure this or other global options, possibly like this:
In this case frontends would just check for the existence of set_backend_configuration and pass options if and only if it exists. This version also has some built-in forwards compatibility by taking arbitrary keyword arguments that will be ignored if unsupported (possibly with a warning). We could also add a get_backend_configuration_options function so that frontends could have different behavior based on what the backend supports.
We don’t have to specify what backends should interpret a missing cache_directory as, though I imagine setuptools would default to the repository root. I’m thinking that this will allow tox to specify a cache manager in some per-env directory under .tox and pip could easily grow a flag like --incremental-builds that turns off the isolated build behavior of copying the repository into a temporary directory.
I am OK with writing a new PEP since this may be beyond the scope of what we want included in PEP 517 itself, but I’d like to hear thoughts and criticisms before moving to the “draft a PEP” stage.
As the initiator for this change I’m very much plus one on this. We’ve talked this with @pganssle in person at the core dev sprint. I think, I personally, prefer adding the cache_directory=None to all PEP-517 interface end-points. All functions in there for now seem to not depend on another function being called beforehand, and I think we should keep that.
Sounds like a reasonable idea. There was a fairly long discussion during the PEP 517 debate about incremental builds, which might be worth hunting out (it would be in the distutils-sig archives).
From what I recall, part of that thread was linked to pip’s copying of the source directory, which makes it impossible for backends to persist build artifacts in the build directory - but not copying risks the possibility of inconsistent builds if stale build artifacts are picked up. The general view was (from what I recall) that front ends should trust backends to handle the staleness issue themselves, but it didn’t really matter that much, because pip wasn’t changed to build in place - and longer term, pip is likely to build via sdist rather than in place anyway.
This proposal offers an interesting alternative, and probably expands the options available to front ends. But I’m not that sure how many options we need - it’s not like there are many front ends, for better or worse. You mention tox - is tox doing direct PEP 517 builds these days? (If so, I’m encouraged in some ways, it’s not healthy if pip’s the only frontend around). I’m not sure how pip would expose something like this - pip’s UI is already very cluttered with options that are only needed in edge cases, and this seems like it could be another one.
Regarding PEPs, I’d say this would be best done as a standalone PEP extending PEP 517 (“Adding persistent build cache support to PEP 517 backends” seems like a good title) that would then result in an update to the build system interface page that should be linked from the packaging specifications reference (but isn’t yet - hmm, PEP 592 should probably also be mentioned in the repository API page…)
tox definitely has it’s own PEP-517 build system, and tox 4 will keep this. We really only need one option, a cache directory that the backend can write additional content too. I would envision three folders for now in there in case of setuptools: dist, build and the x.egg-info.
We already have a config settings dict that gets passed into every backend hook. Maybe we just need to standardize a "cache_dir" key? That might simplify the backwards compatibility issues.
It’s not just about having the key, but writing it down and stating that backends must respect it (e.g. it’s fine to have the source tree read only as long as the cache dir is writable, a backend build should still succeeds). Can be under the config setting but that potentially raises backwards compatibility questions.
I did consider this, and it’s a possibility, but I think it would be best to reserve the config_settings namespace for the backends themselves and not have it be a mix of standard “reserved” keys and arbitrary backend-dependent keys. It also means that we can’t continue to add things there as necessary, since the more PEP 517 backends there exist, the less we can be sure that our new standard keyword wouldn’t conflict with an existing setting in the backend.
I also think that at least for this case, we should make it possible for backends to signify which post-PEP 517 features they support. For example, I could imagine that pipmight want to use the “copy to a temporary directory” behavior for isolated builds with backends that don’t support a persistent cache directory, but simply pass the cache directory to backends that do support one.
For forward compatibility reasons, how about changing it like this:
(You could also spell this with an explicit cache_directory=None in the signature and it won’t make any difference). We’ll document in the spec that kwargs are reserved for future specification-standardized keywords, which will all come with an accompanying feature flag.
I think that this “no way to persist build artifacts” thing will be a real problem for adoption of PEP 517 by some of the bigger extension-based projects like scipy, matplotlib, numpy, pandas, etc. Some of these take a very long time to build, and slowing down a developer’s update-build-test cycle would be a huge drag on development.
I agree with you about the complicated UI and options and whatnot, but it’s a pretty complex tool. It might be worth talking to a UX designer about this sort of thing (though I think the bigger issue here is the combinatorial complexity of the test matrix - with a dozen binary flags you’re looking at 4096 different possible run states). I really do think pip needs this option, though, because the solution right now is “stop using pyproject.toml”, which is not sustainable in the long term.
As the author of a packaging tool for native wheels (maturin), I’d love to see such an extension.
Even though it’s relatively small compared the big popular packages, I’m already experiencing this with pyo3 where tox is effectively unusable for development and currently takes up most of the ci time.
For the implementation, I’d prefer having **kwargs with an opt-in like backend.SUPPORTS_KWARGS=True.
The source directory may be read-only. Backends should therefore be prepared to build without creating or modifying any files in the source directory, but they may opt not to handle this case, in which case failures will be visible to the user. Frontends are not responsible for any special handling of read-only source directories.
The backend may store intermediate artifacts in cache locations or temporary directories. The presence or absence of any caches should not make a material difference to the final result of the build.
Would backends threat the source tree directory as read-only, and instead use platform-specific cache locations/temporary directories this issue would not manifest. Build backends currently go for building stuff inline, but they could switch to e.g. using appdirs · PyPI user_cache_dir instead.
Someone needs to manage the lifecycle of this cache/temporary directory, and now I’m not sure it needs to be the frontend (where this PEP would land). E.g. what would be pip strategy @pf_moore to when cleanup these cache folders?
This has become much more relevant as pip has switched to building in-place (and not via sdist, as I expected in my previous comment) and we are seeing people affected by read-only source directories.
I think the comment in PEP 517, and the long discussion when the PEP was being debated that can be summarised as “front ends should trust the backend to do the right thing” is relevant here. My concern is that the evidence is now suggesting that backends aren’t prepared to handle in-place builds, and we were being over-optimistic when the PEP was written.
I’m reluctant to restart that debate, as it was fairly difficult at the time, but I’d like to see some investigation done into how setuptools can address this without front end support (e.g., by working out how to support read-only build directories and using that approach for any other cases where “out of build tree” artifacts are needed). I’m not against front ends being asked to offer a cache location (pip is already caching stuff, so we can allocate an area in our existing cache for backends) but I do not want to see frontends being required to do cache management - a dumb frontend should be permitted to not provide a cache and the backend would use a temporary location, for example. (To give a concrete example, that’s what I expect pep517.build would do).
Specifically, pip’s strategy would likely be “do nothing to clean up caches, the user would have to do that manually” - which is mostly what we do with our current caches.
I’m saying “backends” here, but I suspect the reality is that I mean “setuptools”, as that’s the backend with a load of historical baggage that makes this a hard problem for them. It might be helpful to see if other backends (flit, enscons) have similar issues, though.
What about mandating the frontends to pass in a write-able working directory, however not mandating that this is persistent. E.g would be valid for a dumb frontend to pass in a temporary folder that gets cleaned up post-build? This would allow frontends to always have a safe path to write files to, but not mandates frontend to maintain it. Smarter frontends then can tackle the issue of cache management
to offers performance benefits. Some cache management strategies could be:
no management, always start with fresh folder;
create if not exists, never delete - leave it up to the user to handle that (maybe the frontend can have a --clean-cache flag),
I’d be ok with that. Although I don’t see a lot of benefit in not having “not set” be equivalent to “fresh folder that the backend creates for itself”…
This probably doesn’t seem too hard for the case of read-only systems, if we’re already retooling to put the build artifacts in an arbitrary directory supplied by the front-end. That said, I haven’t really tried implementing anything like this.
To me the issue isn’t that front-ends must manage the cache in a specific way, just that backends cannot manage the cache. How would they? In an ideal world with perfect separation of backend and frontend, the backend isn’t even installed in the user’s python environment, and it only takes input from the user via PEP 517. If there’s going to be a mechanism for clearing the build cache or something of that nature, it’s got to be the front-end that provides it. “Cache is invalidated on every build” is a valid cache management strategy.
This feels dangerous to me if the cache isn’t stored in-tree. pip’s wheel cache is pretty simple to invalidate, because wheels coming from PyPI are immutable – if you got a file with a specific filename from PyPI once, you should always get the same file in the future. Maybe there are other caches that have a more difficult cache invalidation strategy.
For build backends, I’ve seen many situations where git clean -idx fixed some problem or gave me a reproducible build environment. If there’s no pip clean or similar, then I’m not confident that I’d be able to reliably know where backends are storing cached build artifacts. Even with a pip clean, maybe “cache goes in-tree if it’s there at all” is the only reasonable strategy, in which case the original issue is basically solved by the fact that pip does in-tree builds now.
Unlike with many other things, if setuptools is the only one affected right now, I don’t think it’s because of legacy reasons (except in a very minor way). Anything with support for compiled extensions is going to need a way to handle caching if it’s going to provide a reasonable developer experience. I think flit doesn’t support C extensions and AFAICT poetry punts on it by having you define a build.py that basically calls setuptools.
Agreed, and that’s essentially my point. I don’t think pip’s cache strategy is much use here, so I’d be inclined to have pip pass a temporary directory and in effect not cache at all. Or more easily for pip, pass None to tell the backend they can cache in a temporary location, because pip isn’t going to do anything else.
What I’m less comfortable about (because I’m not clear on the intended semantics of persisting this information) is whether the longer-term impact would be for front ends to get sucked into the question of how to handle incremental builds - which I very much don’t want to happen, as the backends are the experts here. So I don’t want to get into a state where backends say they can’t do incremental builds because frontends keep throwing away build artifacts, but frontends say they can’t do any better because only the backend understands how to invalidate the cache…
The whole point of this proposal was because the pip was building in a temporary directory and throwing it away, so it would very much defeat the purpose for pip to pass a temporary directory. We need the opposite – we need a place to put files to support incremental builds.
The point of the original proposal was to make it possible for backends to do incremental builds, not to get frontends involved. To the extent that front-ends are managing the cache directories, it’s in providing the user manual controls for overriding the backend’s default invalidation strategy.
tox is a good example for this – it has caching of virtual environments that are invalidated automatically under certain conditions (which would be the backend’s role), but you can manually override either by deleting .tox or by using one of the invalidation commands like --force-recreate or whatever (front-end’s role).
I think this may still be relevant if there are any front-ends that want to allow for incremental builds but want the source tree to remain intact. For example, I could very much imagine that tox would want per-environment build caches. Right now I don’t think it’s possible to do that, because setuptools can only ever cache anything in the source tree (anywhere else would be very undiscoverable and could lead to some seriously hair-pulling bugs).
A valid caching strategy for a frontend would be to delegete it to the user. The issue at hand is that the backend does not communicate with the user. The frontend does. So sure, pip might decide not to implement providing a cache folder, but then it needs to allow a user to pass through a cache folder to the backend.
I’m personally of the opinion that we don’t need to have pip support incremental builds for incremental builds to be a useful, supported part of the PEP 517 interface, The tox example is a good one, tox currently builds a sdist and then relies on pip to turn that into a wheel and install it. Maybe the better answer is for tox to invoke the PEP517 wheel build hook itself with a cache directory pointing to the .tox directory and then pip install the resulting wheel.
IOW, we don’t need pip to natively support caching or incremental builds. We can continue to build the entire thing from scratch whereas more specialized tooling can make a different choice. Remember part of the reason PEP 517 specifies frontends and not pip is the idea that we can have more than one front end, and a really good reason for a new frontend would be because the use cases between a general purpose tool like pip and a a more specialized tool like a tox wheel builder don’t overlap perfectly and need different options or behaviors.
I don’t say pip must have cache management, but pep 517 should offer the option. Currently it doesn’t. tox cannot build a wheel where the build backend is not in source because the PEP 517 interface doesn’t allow to specify the cache directory. This is the scope of this topic. We want to extend pep 517 with an optional cache pass in on the front end side, which is mandated to be respected by the backend if specified by the front-end (whoever that might be, pip, tox or anything else
I think we may not want to mandate that any cached artifacts should go in there, because that might be hard to pull off in some cases (like if your backend is running some other build tool like a compiler and you can’t necessarily control where it puts stuff), though maybe we can have language like “if a cache directory is passed, backends SHOULD store any build artifacts that would be useful to cache for incremental builds in the location.”
- hmm, 
