PEP 660 and setuptools

Packaging
1

PEP 660’s current status is that it has been implemented in pip, flit, pdm, hatchling and Poetry.

There is a draft implementation in place within setuptools but it has stalled due to lack of consensus.

Could we use this thread to

  • List and understand all the implications of implementing it in setuptools

  • What requires to be done for a workable solution in setuptools

  • Identify any room for a compromise so that we can build some consensus

I would appreciate feedback from contributors within and outside of setuptools. As this is the only pending issue, moving towards implementation will bring PEP 660 one step closer to completion.

2

This is a pretty good summary about the lack of consensus: Standardising editable mode installs (runtime layout - not hooks!).

Some developers consider the behaviour as a “serious long-standing bug” while others are very happy with the trade-off[1] and would experience frustration with a stricter approach, since it can potentially disturb their workflow.

My opinion is that all depends on the target audience and intended usage:

  1. A user that wants to develop incrementally a package and play/tinker with it in the CLI is probably interested in a “loose/lax” implementation that automatically picks up new files (this is required to support important workflows like splitting existing files in smaller pieces or general refactoring).

    • a) Since the existing setuptools behaviour is to allow new files to be automatically recognized, changing the default implementation to a “strict” approach is very likely to be disruptive and frustrate this audience.
    • b) The existing approach to implement this workflow without editable installs is to perform a full-blown install and then manipulate PYTHONPATH to point to the source tree in development (or relevant subdirectory)
      • When specifically talking about setuptools, this alternative approach might not work because of the subtleties and flexibility of package_dir and the arbitrary mappings it allows between the file system tree and the package tree.

  2. A user that wants to test[2] the package (either the final version or an intermediate version after a big chunk of work) is probably interested in a “strict” implementation that emulates as close as possible the installed behaviour of the final distribution artifact.

    • a) AFAIK a “strict” implementation is not provided by the major build backends or by setuptools’ develop command. Therefore there is no expectation that an editable install would behave this way[3].
    • b) The existing approach to implement this workflow without editable installs is to run tests[2:1] on top of a full-blown installation of the package after it is build. Tools like tox and nox facilitate this, and the advantages/disadvantages of this approach are relatively well known by the community[4].
    • c) Even if a “strict” editable install is performed, chances are that unwary users[5] would end up invertedly “poisoning” PYTHONPATH during the tests and loosing the benefits of the “strict” approach (e.g. by running the tests directly from the top folder in a flat-layout project).

Because of the reasons in (2.c), I would argue that the main benefits of having a “strict” editable install by default are (unfortunately) undermined. To actively take advantage of an “strict” editable install, the users also need to be aware of PYTHONPATH “poisoning” and all the tricks to prevent it.

Therefore, since these users already have to actively take extra measures to perform a “strict” test methodology, I don’t think it would be problematic to require an “opt in” for “strict” editable installs.

On the other hand, as previously mentioned in (1), if we make the “strict” editable install the default, users would be probably be frustrated by setuptools not fulfilling the existing expectations. (In summary it would be a “breaking change” that would likely upset a lot of users).

It is also important to notice that the audience in (1) has pretty good reasons for wanting editable installs to be “lax/loose”, therefore I think that classifying this trade-off behaviour simply as a long standing bug is an oversimplification.

Proposal

In order to address the comments and concerns presented in the preceding discussions about the theme, my proposal is that setuptools should implement both the “strict” and the “lax/loose” approaches for editable installs. However, when not specified otherwise by the users, the installation should be perfomed in the “lax/loose” mode.

Initially, setuptools can expose an environment variable (e.g. SETUPTOOLS_EDITABLE=strict) to allow users to opt-into the “strict” mode, just as a transitional measure. But after some time, it would be nice if the existing installers provide an interface for the users to change the editable mode to “strict” without having to set environment variable[6]. Merely illustrative example of how that could be done:

pip install -e --strict .
# OR
pip install -e --mode=strict .
# OR
pip install --editable=strict .
# etc…

The installer/build-frontend would then call the PEP 660 hooks passing an appropriate/pre-agreed key/value pair in config_settings. Merely illustrative example:

config_settings = {“editable_mode”: “strict”}
# OR
config_settings = {“strict”: True}
# etc …

  1. In order to automatically pick up new files the implementation also looses the ability of recognizing when a file would not be present in the final distribution. ↩︎

  2. tests here mean not only automated tests but also checks performed manually/in a ad-hoc way by the user. ↩︎ ↩︎

  3. The word expectation is used here in the same meaning as in “managing expectations”. Users may wish editable installs would work in a strict way, but currently when they do pip install -e . they don’t expect it to be strict. ↩︎

  4. There are popular blog posts, conference talks, videos, etc that date as far as 2015 (or even further). For example: Tox tricks and patterns | ionel's codelog ↩︎

  5. In this text I use the unwary user expression to refer to users that (a) are not aware that they should be testing the packages as if it was installed to prevent accidental errors, OR that (b) are not actively interested in taking this extra measures. ↩︎

  6. It seems that there is already some positive support in the pip team about improving the UI for this use case: Implement PEP 660 hooks in setuptools/build_meta.py by ichard26 · Pull Request #2872 · pypa/setuptools · GitHub, Implement PEP 660 hooks in setuptools/build_meta.py by ichard26 · Pull Request #2872 · pypa/setuptools · GitHub. ↩︎

2 Likes
3

At the moment, pip’s support for config_settings is weak, but we could easily improve this. The current expectation is that the form would be something like --config-settings KEY=VALUE. I am personally willing to take on the work to implement this.

Some notes:

  1. I’m not particularly comfortable having a dedicated “strict mode” flag for editable installs, as it’s likely to be very specific to setuptools. So I’d stick with the generic --config-settings approach in pip and let backends interpret the settings as they prefer.
  2. It may be necessary for backends to agree on how the config_settings namespace gets managed - if setuptools wants to use a generic key like “strict” for editable mode, will that interfere with other backends which might want to use it for something else? That’s something for backends to sort out, though, and arguably it’s not worth worrying about unless config settings get used enough for it to start to be an issue.
4

Thank you very much @pf_moore. I am perfect fine with the approach you are suggesting, it does not have to be a flag directly under pip’s “option’s space”, something like --config-settings mode=strict could work just fine. Something following build’s fashion (e.g. -C--strict) can also work.

I suppose that something that might be implemented by backends is to raise some kind of exception. For example, Python functions may raise ValueError when they receive an unexpected argument. We could go with that…

5

I agree with your proposal @abravalheri. I was waiting on a decision on how PEP 660 would be implemented before working on the External Data Support feature, since it would affect how those files should be handled. Flit handles them as follows:

“If you install a package with flit install --symlink, a symlink is made for each file in the external data directory. Otherwise (including development installs with pip install -e), these files are copied to their destination, so changes here won’t take effect until you reinstall the package.”

Similarly, I would expect python setup.py develop to create symlinks for each file, where pip install -e . would use the .pth approach.

6

@blink1073 just FYI, I am currently working in this PoC for the proposal above: GitHub - abravalheri/setuptools at editable-link-tree (WIP):

  • For the “lax/loose” approach, either a static .pth file is produced or a MetaPathFinder is added, depending on the project layout.
  • For the “strict” mode, the existing files detected by package_data get linked (symlink if possible, hardlink otherwise).

In both scenarios, the <package>-<version>.data wheel entry is generated normally as it would in a conventional wheel.

I may be wrong here, but after reading the specs for editable installs and the wheel format I was under the impression that there is not much of a choice for the .data directory other than having it baked statically inside the .whl archive.

For Python modules and package_data we can use the trick suggested in PEP 660 of creating a link tree out-off-band and then add adding a static .pth file pointing to it, but I don’t know if adding the *.data directory there would have the intended effects. For example, would sysconfig.get_path work to detect these files?

Currently we don’t have a mechanism to represent links in the wheel, and I have the impression that any of the other well-known approaches, such as a MetaPathFinder or a static .pth file, would be of no use for the *.data directory…

7

Can you explain precisely what the semantics you are after should be? Ideally with an example? During the discussions that led to PEP 660, the focus was on Python code being available (in particular, both PEPs submitted defined an editable install with the description “The editable installation mode implies that the source code of the project being installed is available in a local directory”).

One possible approach for non-Python files (data files, headers, etc) would be to simply install them as normal, along with installing the Python files via a .pth file or hook. Changing non-Python files would require a reinstall, but that’s technically OK, people only expect to be able to change Python code “on the fly” (largely because that’s all that works with the historical .pth based approach). And it’s certainly within the definition of an “editable install” given in the PEP(s).

8

Hi Paul. I am completely fine with the “only Python files limitation”, I was just trying to manage expectations regarding the files inside *.data being editable. For example the following is exactly what I have in the PoC right now:

Maybe @blink1073 can chip in here about which semantics the users would like to have? I don’t think python setup.py develop is going to keep receiving support/improvements after we manage to agree on a PEP 660 implementation for setuptools, so if a specific feature is required, it would be nice if we implement it in a PEP 660 scenario.

1 Like
9

I think it is fine if we ignore external data when using python setup.py develop, since that invocation is deprecated. As it stands today, data_files are not handled at all by python setup.py develop. We had a hack in jupyter_packaging at one point to explicitly handle data_files when running develop, but we found it to be brittle because we were not removing them on uninstall.

10

I opened a PR in Implement PEP 660 allowing both "strict" and "lax/loose" approaches by abravalheri · Pull Request #3265 · pypa/setuptools · GitHub.

1 Like
11

Looking over your PRs, I’d appreciate a clarification—is a MetaPathFinder required for cases that currently only need a .pth file when using setup.py develop? And when, specifically, is it required—for all non-src layouts, or only specific ones?

12

In general, I recommend using a .pth file if at all possible (and setuptools is a clear example of where it’s possible, because that’s what the existing code does). The big problem with a metapath solution is that it doesn’t handle implicit namespace packages. IMO, people wanting to avoid the “extra files may be exposed” limitations of the .pth approach probably want the “strict” mechanism anyway.

Of course, this is a question for the setuptools project to decide, though, so it’s ultimately their call what trade-offs their users want.

13

Based on the previous discussions, it seems that the community aggress that the existing implementation for the cases that currently only need a .pth file when using setup.py develop , has problems (except for src-layout). Specifically:

  • In a flat-layout package, everything under the project root suddenly appears in sys.path.

This means that after develop, people might accidentally be able to do import tests , import setup, from docs import config, etc, and that is an undesirable side-effect.

In the proposed PR, I use a MetaPathFinder to workaround this limitation and improve the editable install for the “lax/loose” mode. I would place this approach under the Identify any room for a compromise so that we can build some consensus umbrella, as mentioned by @smm.

Yes, it would be for all non-“src-layouts”[1]. I cannot think about a situation in which the undesirable side effects discussed above would not happen for a flat-layout package installed via a static .pth file. What do you thing? Am I missing something?

  1. “src-layout” is used here in the general sense… package_dir = {"": "my_weird_folder"} can still be considered a variation of the “src-layout”… ↩︎

14

Do you think we should enforce that even when the package can be safely installed (considering the “lax/loose” interpretation and trade-offs) with a static .pth, for example “src-layout” packages?

The existing PR, uses a static .pth for those scenarios… What I had in mind when implementing this was: (a) we can avoid the complexity of touching the import machinery, (b) we can offer the uttermost editable experience, (c) we sidestep the MetaPathFinder installation cost and the follow up import look up costs (not that this is very relevant…).

For the “strict” interpretation, I went with a link tree because it was looking the most straight forward answer for challenges such as dealing with "package_data" files[1]. (It also seemed easier to implement).

I think I manage to workaround that in setuptools/setuptools/command/editable_wheel.py at 891711f4558d58c1b6ca2ddc291ecdbcebc18a6f · abravalheri/setuptools · GitHub. But I might be missing some important pieces and failing to see the big picture…

  1. I admit that I lack the knowledge here to understand how the MetaPathFinder interplays with importlib.resources, pkgutil, etc… ↩︎

15

Oh cool! Would you be willing to contribute that to the editables project (assuming it does work :wink:) I couldn’t find a way that implemented all of the required semantics.

The key problem is that PEP 420 requires that changes to the filesystem while the Python process is running must be picked up. There’s an example in the PEP here, and I’m pretty sure I wasn’t able to make that work with a meta_path hook.

I agree this is an obscure case. I’ve no idea whether it’s acceptable to break it for editable installs. In editables I chose to document that it didn’t work and let the caller (backend) decide which trade-off they preferred.

16

After the PR review goes ahead and we find all the missing pieces, edge cases, I can definitely give it a go.

There is one fundamental difference between editables and the custom MetaPathFinder I am using: the custom finder will intercept the import of sub-packages/modules, while editables does not seem to be doing that. I don’t know if this difference will impact porting the namespace workaround to editables, but I guess we are just going to know when we start implementing…

I still don’t know exactly how to deal with the __path__ part… PEP 451 seems to indirectly suggest that we can set loader to None for namespace packages… In the existing tests, that seem to work for imports at least.

My guess would be that the existing import machinery identifies the ModuleSpec.loader=None and take it from there, but I haven’t tested how this assumption works with __path__… It also does not help the fact that _NamespacePath, _NamespaceLoader are mostly private[1].

This might be relevant here: Loader for namespace packages · Issue #79854 · python/cpython · GitHub

  1. Even in the latest changes when NamespaceLoader is exposed, the documentation seems to be clear that it should only be used for introspection and comparison… ↩︎

17

I’m currently looking at how to implement this. The implementation shouldn’t be too tricky, but there are a bunch of design-level and policy questions around config_settings that need to be considered. I can simply make expedient choices to make things easier for pip to implement, and that’s likely what I’ll do in the absence of a wider consensus, but at some point we need to reach some decisions.

  1. Is config_settings supposed to be constant across all hook calls? See this thread for more detail.
  2. Are backends expected to ignore settings they don’t recognise? This is important because frontends like pip can be building multiple projects at once (in the case of dependencies, this may even be unavoidable) and those projects may use different backends.
  3. PEP 517 states “it’s up to users to make sure that they pass options which make sense for the particular build backend and package that they are building”. Does this mean front ends must provide fine-grained control over config settings[1] (this contradicts the suggested syntax in the PEP)? If so, is doing so at the cost of user friendliness an acceptable trade-off? If not, again are backends expected to be forgiving of settings “leaking through”?

My personal answers to these questions, from a “what is expedient for pip” perspective, are:

  1. The frontend will only guarantee to pass the specified settings to the relevant build_XXX calls.
  2. The frontend won’t make any particular effort to avoid passing settings to backends (or hooks) that might not expect them.
  3. Pip will allow KEY=VALUE settings that apply to every package in the run, or on a per-package basis, but the latter only if specified in a requirements file.
  4. As a consequence, backends should be prepared to silently ignore anything they don’t expect/want to handle[2].

None of these are set in stone, they are just “what I’ll go with in the absence of anyone saying they need something different”. If any of them would cause problems, please speak up. But I’d rather focus on practical concerns over how to make pip’s support useful (with particular emphasis on setuptools’ editable install support) rather than on broad theoretical debates over what people “might” choose to use config settings for in some as-yet nonexistent backend…

  1. The extreme case here would be settings that explicitly specify package, backend, hook, name and value. ↩︎

  2. It may be that they already do that, in which case simply saying so publicly might be enough to deal with a lot of the risk here. ↩︎

18

Thank you very much Paul for looking into this.

I think this is the most straightforward way of dealing with this parameter. In general terms, I believe that config_settings is needed for both build_XXX and prepare_metadata_for_build_XXX hooks (at least)[1] so it make sense to use the same dictionary across all hook calls.

I am in favour of this approach for the keys in config_settings.
I would say that backends may decide to warn the user about keys it cannot recognise (just for informational purposes), but not fail. This seems to also inline with the existing implementation for both setuptools and flit.

I don’t think so. It should be fair to assume backends will gracefully handle the behaviour in (1) and (2). With that in mind, I would say that if users want fine-grained control over how each package is built, they should perform separated frontend calls (one for each package).

  1. There are a few use cases accross PyPA’s issue trackers implying that users expect to use a similar mechanism to influence the metadata (such as build tags added to the version). Examples: Wheel tags · Issue #202 · pypa/build · GitHub, https://github.com/pypa/setuptools/issues/2491. ↩︎

19

It’s not the most straightforward in implementation terms for pip, but I agree that it’s the most straightforward conceptually. Doing it this way will probably require a change to build, as @FFY00 noted in the linked thread that

Right now, pypa/build will only pass config_settings to the build_wheel , build_sdist and prepare_metadata_for_build_wheel hooks, and not get_requires_for_build_* .

But at the moment, I’m focussing on “what should pip do”, not on “what should we document as required behaviour”, so I’ll leave worrying about other frontends and formalising this to someone else to pick up :wink:

OK, so what I’ll do is:

  1. pip will collect config_settings as with any other option (including the possibility of per-requirement overrides from a requirements file).
  2. pip will supply the config settings to every backend hook call.
  3. I’ll dump the problem of handling this behaviour on backends :slightly_smiling_face: I did a quick check of the flit and poetry source code, and both seem to ignore the config_settings parameter altogether, so I don’t believe they will have a problem here. Hopefully, the same will be true of other backends.
3 Likes
20

@smm I have created an issue (Uncertainty on how to implement namespace packages (as in PEP 420) via import hooks · Issue #588 · pypa/packaging-problems · GitHub) to clarify how to support editable namespaces via import hooks.

I am also waiting for feedback from the other setuptools developers (or other members of the community) that might have different opinions about the implementation.