Hi, I created a PEP draft. See here TBD: Persistent cache for packaging frontends by gaborbernat · Pull Request #1976 · python/peps · GitHub (also looking for someone who is willing to sponsor it and believes in the need and solution for this).
Some thoughts.
- This is not backward compatible. If a frontend supplies this argument, and the backend is an older one that isn’t aware of this new requirement, it will fail. The frontend has no way to know if this is the case. The original proposal suggested having an attribute on the backend, but we should probably look more closely at how we evolve PEP 517 - adding a whole bunch of feature attributes over time doesn’t seem very scalable.
- It’s not clear to me what the frontend requirements mean - “The build frontend must persist the content within the cache directory, such as subsequent calls can recall data saved in prior calls” vs “The folder must be empty during the first backend call”. How long must the frontend persist the data for, and based on that answer what counts as the “first” backend call?
- I think the “motivation” section needs improving. I’m not convinced that “the backend pollutes the source directory” is a compelling argument (for pip’s use cases, at least) - there’s plenty of tools that add directories to the user’s source directory - tox, mypy, and coverage, for example. Why are build backends so special?
For what it’s worth, from pip’s point of view I don’t see any clear benefits, and a reasonable amount of complexity in implementing the feature, so I’d expect that we’ll ignore this option.
1 Like
Because pip copies the entire source directory to a temp folder this is less important for pip. The idea here a UX benefit of doing a package build does not pollute the current working directory with build backend cache files, but instead allows the frontends to move it to a more hidden location that’s managed by the frontend (for example tox would move the build and .dist-info folder of setuptools into the .tox folder).
We won’t be copying the source in future, but I still don’t see this as being useful for us. But 
if other frontends want to clean up the source directory, who am I to argue 
One thought I do have, what if someone uses tox to do their testing, and pip to install their project? If tox is using a different build cache, does that mean there’s a risk that behaviour could be different because pip and tox supply different cache directories? For example, testing with tox using a clean cache everything looks fine, but then pip wheel builds an invalid wheel because there’s out of date stuff in the cache pip uses - which will likely be the default of the source directory?
I’d imagine it’s the responsibility of the build backend to ensure the cache it starts with is still up to date and valid. So a such use case in my books would be classified as a build backend bug.
1 Like
I also think it definitely won’t be useful for pip. But it’s very useful to frontends that build a project repeatedly, e.g. for development purposes. For example, a dev tool can be run in the background listening for source code changes, and incrementally rebuilding (+ reinstalling) the project into an environment for testing, similar to how npm-watch and sphinx-autobuild. This will work better for native modules than editables (because the extensions still need to be rebuilt), and extra powerful when combined with other modification-watching tools like pytest-watch.
I might see a world where pip might not set this by default, but could allow the user to set it via a flag; for all the automation workflows that use pip under the hood to build and install their packages 
I would hope those tools stop doing that altogether instead. pip is really not a good tool for that (way too slow to launch repeatedly, for one).
3 Likes