Thanks for working on bringing PEP 660 support to setuptools.
If I understand correctly there’s three editable install modes - strict, lax and compat - but lax, which is the default, does not appear to be documented? Is the “redirecting” hook used for lax installs, or how does lax work? What does compat do?
@abravalheri can correct me, but as I understand based on what has been said previously, lax uses a custom import finder hook (at least for implicit layout projects, as implied above, not sure what is used for src ones), compat uses .pth (like the legacy setup.py develop) and strict uses symlinks/hardlinks.
Hi @layday thank you very much for the kind words.
The way the installation works it similar to what @CAM-Gerlach have described (thanks Christopher). I will try to summarize it bellow:
The default behaviour when the user simply runs pip install -e . will depend on the project structure.
I tried to implement a trade-off between allowing the users to edit files freely (including renaming, deleting and adding files[1]) and preventing sys.path from being polluted with auxiliary scripts/folders the user might have added to the project repository.
Currently this means:
src-like layout that does not mess with the package_dir config: a static .pth file is used (since src layouts are relatively safe in terms of sys.path pollution).Users can opt into a “stricter” behaviour with pip install -e . --config-settings editable_mode=strict, which means that setuptools will try to emulate as close as possible a regular wheel installation. Currently this is implemented using a link farm in an auxiliary folder which is added to sys.path.
Temporarily users can opt into a behaviour that is compatible with what the develop command would do. When you run pip install -e . --config-settings editable_mode=compat, setuptools will always use a static .pth file (even for flat-layouts, no trade-off is considered).
However compat is only meant to help during a transition period when will eventually go away (I am choosing the end of the year, happy to revise this date if necessary).
Although I briefly mention all the possibilities of how setuptools achieves an editable installation in this section of the docs, I am deliberately avoiding being categorical in the text, because this is meant to be an internal implementation detail and I don’t want users to rely on the specific method so it can change in the future[2].
Which I am lately calling in the code “lenient” as a direct opposite to “strict”, but the exact name should not matter, since I prefer users to think of it as simply “the default behaviour”. ↩︎
For example, we might decide to revise the implementation once we have more information from Clarification about how to implement namespace packages (as in PEP 420) via import hooks for PEP 660 use case · Issue #92054 · python/cpython · GitHub or when we start getting feedback from real world usage. ↩︎
The reason why I chose to go with a trade-off instead of simply using static .pth files all the time is to try to achieve some compromise between the two sides involved in the previous discussions about setuptools and PEP 660 (the ones advocating for a strict behaviour by default and the ones advocating for a lenient behaviour by default).
The reason why I did not choose to unify all the different modes into custom finders with complex import hooks, is that (in my opinion) they present several limitations. At least I noticed the following:
python/cpython#92054)pkgutil/pkg_resources namespacesstrict installs).In the end of the day it just seems that a static .pth file or a link farm will match more closely the users’ expectations… So I try to use them whenever possible.
Oh, I didn’t realise that this was temporary. Is there any specific reason that we want to go through churn of changing things for users immediately after introducing this functionality to them?
Hi Pradyun, thank you very much for the feedback.
I see things from a slightly different perspective:
We are changing the way setuptools work because of a few motivations: to catch up with PEP 660, to avoid making internal scripts/packages/modues available via sys.path, and to address a series of accumulated issues because of the limitations of the static .pth approach (e.g.pypa/setuptools#230, pypa/setuptools#1801, pypa/setuptools#2662, pypa/setuptools#3399, …).
Between the default editable exerience and the strict mode, setuptools should be able to support all kinds of project layouts. The cost of adding these improvements will be (justifiably) some level of churn until the users understand how to deal with this new dynamic[1].
The only use case identified so far that is not solved by either the default behaviour or the strict mode is the one presented by Stéphane: a flat-layout package using pkgutil namespaces that try to reach for files outside of the package directory with Path(__file__).parent.parent.parent.
If I understood correctly, this use case does not really fit the mindset of PEP 660 (I think that no PyPA standard offers any guarantee about accessing files outside the package directory using the value of __file__).
Therefore, the compat mode is not really a functionality that we are introducing or that we want the users to learn about. Instead, it is a temporary escape hatch (I hope that the documentation makes this clear). This way users have some time to implement any required changes or report use cases that are not covered by the other “blessed” editable installation modes.
I am more than happy to reconsider compat if we identify other use cases requiring it.
But since I personally don’t plan to keep providing support for compat[2], I believe the best is to let it go after the end of the year.
Any sufficiently ambitious/complex change or PEP unfortunately will have this side effect in setuptools. ↩︎
Given the limitations of the static .pth approach, fixing issues or adding features would likely require using file links or path hooks, which is already what the default behaviour / “strict” mode do. ↩︎
I realize I am a bit late to this thread, but I did only just notice this:
Non-Python files, external (data) files, executable script files, binary extensions, headers and metadata may be exposed as a snapshot of the version they were at the moment of the installation.
Is this permanent? I think anyone using watch processes to build/install other components into a source tree will find this to be a big step backwards.
This part is just in keep with PEP 660, right? The editable hooks focus on the ability of editing only the Python files.
This does not have to be permanent, but for the time being I don’t think there are any plans on implementing something with stronger guarantees.
Anyone interested in contributing a different solution to setuptools is very welcome to start a discussion about an alternative implementation (e.g. in the setuptools issue track as a feature request).
Editing the python files is the minimum required by PEP 660. There’s nothing stopping tools providing more. Installing a .pth file gives a result that’s 100% compatible with the old-style setup.py develop. The problem is that it gives all the problems as well as the benefits.
Setuptools has chosen to develop a new, safer, solution as part of its PEP 660 support. That’s exactly what PEP 660 provides for - backends can choose the trade-offs they want. In fact, setuptools has implemented two different options, with different trade-offs, to give the user a choice.
Thank you very much @pf_moore for the clarification.
When you say “providing more” do you mean only binary extensions and non-Python package files[1], or are you also refering to the other file categories of files in the wheel (headers, scripts, metadata, data…) that are also part of Bryan’s question?
Maybe I missed the discussion somewhere but is there a way supported by the mechanism introduced in PEP 660 to dynamically recreate files outside of the platlib and purelib categories?
I understand that by adding entries to sys.path or creating import hooks, we can achieve “editability” regarding both pure Python modules and compiled extensions[2], and that since console/gui_scripts are wrappers they would probably simply work once the modules they refer to are updated (as long as no entry-point is added or removed). But it is still not clear to me how to achieve “editability” on the other categories. Maybe I am missing something…
When using src layout (in default non-strict mode) or flat layout with the compat mode, there should be no “step backwards”, since setuptools will behave essentially as before with setup.py develop. Is that right @abravalheri ?
All I meant was “up to and including everything that the old .pth file approach produced” (if only by just installing a .pth file and accepting all the downsides 
).
As far as I’m aware, no-one has tried to provide more than the .pth file approach offered. But of course, if they manage to, then that’s fine - PEP 660 is just a mechanism for people to do what they want with.
I agree. In general there should be no steps back.
Personally I consider that this implementation brings improvements for every kind of layout (including flat layout), as it fixes the limitations/issues I mentioned in my previous comments. Of course, there maybe be some hiccups if the project uses practices that are no longer recommended/preferred/advised and are difficult to integrate[1], but between the default editable installation and the strict mode all projects should be handled.
That said, users are invited to provide feedback and report issues that might not been foreseen yet. We are also very happy to discuss feature requests that align with the long term objectives of setuptools [2].
For example, setuptools has expressed its intent to retire pkg_resources, and some Python developers also pointed out the general desire in eventually remove pkgutil. This impacts directly the long term viability of old-style namespace packages, and therefore we can say in practice that implicit namespaces are preferred/recommended/advised. ↩︎
A small reminder: feature requests have more chances of being accepted/added to setuptools if the proponent is willing to work in the implementation. ↩︎
I am still uncomfortable with the fact that flat layout project cannot be configured to use the simple .pth approach and will be forced into the importlib wizardry approach without escape hatch.
I can totally understand it becomes the default for flat layout projects, and it will work for many. But I’d really prefer that project authors are considered adults who can decide that their project root is safe to be added to sys.path. Currently the only alternatives are to use the compat mode that will be removed, or migrate to src layout which can be impractical, or use the strict mode which can be uncomfortable.
As an example, for a project like Odoo, pkgutil.extends_path is important and likely hard to change, and moving to src layout is impractical (if only because of hundreds of open PRs), and the scrict mode is very uncomfortable due to frequent addition of new python files and various resources during development.
Again, this project is only an example, but I suspect a significant number of complex projects will have similar issues.
But I think I’m repeating myself at this point, so these will be my last 2 cents on this topic 
Hi @sbidoul , I understand the frustration.
While personally I would not like to support compat in the long run, I haven’t completely discarded the possibility of reverting the decision.
To push compat beyond the transition period, I suggest to anyone interested engaging with the other setuptools maintainers and get their opinion and explicit support. This can be done I’m the discussions tab in the setuptools repository, for example.
Modern Git is very good at handling diffs through file and directory renames, even including file additions and deletions. This shouldn’t be an issue when choosing project layout.
I think there’s opportunity here to implement something I’ve had on my mind for a while now: on-import module compilation. I could install an import hook which (Cythonises and) compiles the extension modules on import, if it’s been installed as editable (eg using an in-tree build-backend to set a flag).
Hi Laurie, that is a great idea and I would be more than happy to reviewing this as a contribution.
Another thing that I am targeting with the upcoming release is better integration with plugins (that for example could provide this kind of functionality). For that I added some hooks that can be defined in custom build steps: https://setuptools--3488.org.readthedocs.build/en/3488/userguide/extension.html#supporting-sdists-and-editable-installs-in-build-sub-commands.
That seems nice, but the issue is more general than that. In our case it is an associated Typescript library that ships with the package that is compiled.
Hello all, thank you very much for the feedback provided. Given this feature has been a long time in the making, it seems to be the right time to release it. I am planning to do it still this week unless something unforeseen comes up, e.g. a new bug, or another release taking priority (e.g. a distutils update).
I think releasing it would be important to start collecting feedback from the broader community.
Regarding the compat mode, I reiterate that I have no strong feelings against keeping it permanently in the implementation, if we have compelling arguments and the buy in of the other maintainers. We have some time to discuss this in detail after we publish the release (which will also be good, since people that might not follow Python discourse will also have the opportunity to report use cases that might not be supported).
v64.0.0 was just released, including the implementation of PEP 660 hooks.
Thank you all for the discussions, suggestions and help with the tests.