Help testing PEP 660 support in setuptools

Packaging
29

Looks like the stable version 63.x also has PEP 660 support now, so this still works.
I didn’t see it mentioned in the changelog, but it probably should be?

My experience is mostly like what others have said already.
I converted some simple projects from setup.cfg + setup.py to pyproject.toml only, and it worked perfectly well (the only non-standard thing was setuptools_scm and package_data).
I only tried the “normal” non-strict editable install and it works as expected.

The only thing I don’t like is that it still puts the egg-info folder beside the package folder when doing editable install, but that was a problem in the old way too and probably unrelated to the PEP 660 support.

30

Sorry @saaketp, I think the stable version 63 does not include PEP 660 support. In my calculations there would be no breaking change in setuptools until we landed editable support, but during the weekend there was another change that got in first, requiring a major bump.

Since setuptools uses an automatic version bump system, sometimes it is a but hard to precise when a feature under development will be able to get in. When PEP 660 finally lands, there will be an entry in the changelog.

31

Hmm, okay it does fail now when I try in a new venv, not sure what was wrong in the other venv when I tried it earlier, but that time editable install worked with requires = ["setuptools>=63.0"].

So now we need requires = ["setuptools==63.0.0b1"] in the [build-system] section, >= gets the stable one and fails.

32

Sorry for the disruption

I think that for test purposes, the best would be to stick with "setuptools @ git+https://github.com/pypa/setuptools@feature/pep660" for now, this way we avoid any problems with eventual intermediary releases.

33

The strict mode does work perfectly with regards to pkgutil.extends_path. No worries about that - sorry if I was unclear previously.

Odoo is basically trying to locate. Path(__file__).parent.parent / "addons" as a convenience when running from a git checkout. In strict editable mode it would have to do Path(__file__).parent.parent.parent / "addons", or resolve the symlink.

That’s a subtle side effect of importing symlinked code.

Note that that would not be a problem case with the static .pth.

For the case of Odoo I used for testing, there are workarounds. An in-tree backend is needed anyway and it can enforce the use of the strict mode as well as doing other tweaks.

So, to sum up, there are two separate and independent topics here:

  • pkgutil not compatible with the top level finder mode (as alluded to in the limitations, and forcing to use src layout or strict mode)
  • the fact the code is imported from a symlink farm may leak through one way or another

I’ll leave it up to you, of course, to decide whether these are worth further clarifying in the documentation, or whether the issues would be prevalent enough to warrant exposing the 3 modes via the config setting to provide more control to users.

Thanks again for this effort!

34

Hi @sbidoul, I think I understand a little bit better now, thank you very much for the explanation. Sorry for being so persistent, I really wanted to have a clear idea on what are the shortcomings of the selected approaches :sweat_smile:.

We definitely need to improve the documentation and I will work on that[1].

Would this be a general problem, or would it only happen when the project make assumptions on how the package is installed without considering all possibilities offered by importlib? In the example from last comment, it looks like some parts of the code are relying on circumstances external to the editable installation per se, but I don’t think PEP 660 guarantees anything about the specific installation layout[2].

As long as one of the supported approaches can work for the use cases that don’t make assumptions external to PEP 660, I would prefer keeping only editable_mode=strict (or any bike-shedding equivalent names) as user facing interface[3]. But I don’t feel like I should make this decision alone.

Would you like to create a discussion on setuptools repo and try to involve the other maintainers? I would also welcome any opinions of the other members of the community that have more experience on this topic.

  1. In my mind, something that would really help here is if we could display some warnings to the users after pip install -e . (not all the backend logs, I understand they might be very verbose, but only the warnings).
    In the implementation for all editable strategies, I added a short paragraph trying to express the their main limitations (that can be expanded to cover pkgutil namespaces). But currently these messages are hidden by default… I am not sure what would be the best UI for this and I also understand that this is difficult to implement from pip’s point of view. ↩︎

  2. Could this be considered a backwards incompatible change in setuptools? Under Hyrum’s law, probably. But I don’t think setuptools make many explicit promises about setup.py develop (the existing docs do mention .egg-link and links to your project’s source code but the specifics on how this work are left for the user’s imagination - the only guarantee seems to be that the package will be available on sys.path). Nevertheless the existence of a new PEP should be a good reason to introduce some level of backward incompatibility (which I wish to minimise as much as possible). ↩︎

  3. My interpretation of everything I read when working on this, is that there is some level of consensus in the PEP 660 discussions: in general people seem to agree that simply adding the root project folder to sys.path is not the best idea (understandably, because it leads to effectively installing unintentional packages like docs, tests, examples, or single modules like noxfile, tasks, setup as a side effect), and that a MetaPathFinder or links (as long as they work on Windows) are the way to go. ↩︎

35

I have tried testing this on a project where I use versioningit along with its onbuild feature

The onbuild feature is meant to update a version string in the package when building a wheel/sdist but not when performing an editable install. When performing the editable install I expect the code to remain unchanged and __version__ to be assigned the output of a function that calls git describe under the hood to get the version from git tags and hashes. E.g. my code contains __version__ = _get_version() which should be overwritten with __version__ = 0.0.1 or similar when installing the package (except as an editable install)

The feature is implemented using cmdclass with a setup.py that looks like this:

from setuptools import setup
from versioningit import get_cmdclasses


if __name__ == "__main__":
    setup(
        cmdclass=get_cmdclasses(),
    )

This works fine with the legacy editable install but with this branch I am seeing an error when installing as editable. (Also works as expected using build to produce sdist/bdist and using pip install .

❯ pip install -e .
Obtaining file:///C:/Users/jenielse/source/repos/debugproject
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: versioningit>=1.1.0 in c:\users\jenielse\miniconda3\envs\qcodespip38\lib\site-packages (from debugproject==0.0.1.post3+g1aea130) (2.0.0)
Requirement already satisfied: tomli<3.0,>=1.2 in c:\users\jenielse\miniconda3\envs\qcodespip38\lib\site-packages (from versioningit>=1.1.0->debugproject==0.0.1.post3+g1aea130) (2.0.1)
Requirement already satisfied: packaging>=17.1 in c:\users\jenielse\miniconda3\envs\qcodespip38\lib\site-packages (from versioningit>=1.1.0->debugproject==0.0.1.post3+g1aea130) (21.3)
Requirement already satisfied: importlib-metadata>=3.6 in c:\users\jenielse\miniconda3\envs\qcodespip38\lib\site-packages (from versioningit>=1.1.0->debugproject==0.0.1.post3+g1aea130) (4.12.0)
Requirement already satisfied: zipp>=0.5 in c:\users\jenielse\miniconda3\envs\qcodespip38\lib\site-packages (from importlib-metadata>=3.6->versioningit>=1.1.0->debugproject==0.0.1.post3+g1aea130) (3.8.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\users\jenielse\miniconda3\envs\qcodespip38\lib\site-packages (from packaging>=17.1->versioningit>=1.1.0->debugproject==0.0.1.post3+g1aea130) (3.0.9)
Building wheels for collected packages: debugproject
  Building editable for debugproject (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for debugproject (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      running editable_wheel
      creating C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info
      writing C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info\PKG-INFO
      writing dependency_links to C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info\dependency_links.txt
      writing requirements to C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info\requires.txt
      writing top-level names to C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info\top_level.txt
      writing manifest file 'C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info\SOURCES.txt'
      reading manifest file 'C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info\SOURCES.txt'
      adding license file 'LICENSE'
      writing manifest file 'C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject.egg-info\SOURCES.txt'
      creating 'C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject-0.0.1.post3+g1aea130.dist-info'
      adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
      creating C:\Users\jenielse\AppData\Local\Temp\pip-wheel-g3c7z1_y\tmppzq2ikd2\debugproject-0.0.1.post3+g1aea130.dist-info\WHEEL
      running build
      running build_py
      C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\normal\Lib\site-packages\wheel\bdist_wheel.py:80: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
        if get_flag('Py_DEBUG',
      C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      Traceback (most recent call last):
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 101, in run
          self._create_wheel_file(bdist_wheel)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 247, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 220, in _run_build_commands
          self.run_command("build")
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 317, in run_command
          self.distribution.run_command(command)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\dist.py", line 1217, in run_command
          super().run_command(command)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command
          cmd_obj.run()
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\command\build.py", line 33, in run
          super().run()
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run
          self.run_command(cmd_name)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 317, in run_command
          self.distribution.run_command(command)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\dist.py", line 1217, in run_command
          super().run_command(command)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command
          cmd_obj.run()
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\versioningit\cmdclasses.py", line 69, in run
          run_onbuild(
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\versioningit\core.py", line 593, in run_onbuild
          vgit.do_onbuild(
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\versioningit\core.py", line 442, in do_onbuild
          self.onbuild(
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\versioningit\methods.py", line 160, in __call__
          return self.method(params=self.params, **kwargs)
        File "C:\Users\jenielse\AppData\Local\Temp\pip-build-env-0x_b5npw\overlay\Lib\site-packages\versioningit\onbuild.py", line 64, in replace_version_onbuild
          lines = path.read_text(encoding=encoding).splitlines(keepends=True)
        File "C:\Users\jenielse\Miniconda3\envs\qcodespip38\lib\pathlib.py", line 1236, in read_text
          with self.open(mode='r', encoding=encoding, errors=errors) as f:
        File "C:\Users\jenielse\Miniconda3\envs\qcodespip38\lib\pathlib.py", line 1222, in open
          return io.open(self, mode, buffering, encoding, errors, newline,
        File "C:\Users\jenielse\Miniconda3\envs\qcodespip38\lib\pathlib.py", line 1078, in _opener
          return self._accessor.open(self, flags, mode)
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\jenielse\\AppData\\Local\\Temp\\tmp_asga1s0.build-lib\\debugproject\\_version.py'
      error: Support for editable installs via PEP 660 was recently introduced
      in `setuptools`. If you are seeing this error, please report to:

      https://github.com/pypa/setuptools/issues

      Meanwhile you can try the legacy behavior by setting an
      environment variable and trying to install again:

      SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for debugproject
Failed to build debugproject
ERROR: Could not build wheels for debugproject, which is required to install pyproject.toml-based projects

I am unsure if this is a bug or a feature that is no longer supported (cmdclass).

Ideally an editable install should not try to perform this replace at all and should not care that _version.py cannot be found.

A sample project showing the error can be seen here

36

Thank you very much for the feedback @jenshnielsen, I have contacted the plugin author to discuss what can be done to solve this issue.

The TLDR for why this is happening is the following:

  • Previously the develop command in setuptools would only run build_ext and not build_py. versioningit seems to be taking advantage of this behaviour.
  • In the changes I have implemented in pypa/setuptools@feature/pep660, every build command (this generalisation includes build_py) has the chance to decide to run (or not) on editable installs[1]. This happens inside the run method, which is overwritten by versioningit.

My proposal for versioningit is based on this procedure for implementing custom build steps.

  1. The motivation for this is to improve support for custom build steps. ↩︎

37

Hi @jenshnielsen, it seems that versioningit v2.0.1 is out now and it should support your use case with PEP 660.

38

Thanks a lot @abravalheri for the quick response. I can confirm that this resolves the issue.

39

Hi @sbidoul, sorry for the delay in getting back to this topic.

After some discussion about the topic, I have implemented a compat mode that is meant as a temporary workaround during the transition period, so people have time to adapt to the changes. I am currently considering a period of 6 months, after which I would like to remove this compat mode.

The compat mode will try to emulate the behaviour of the existing develop command (i.e. use a static .pth file).

I have also updated the docs with a more detailed list of limitations and a brief summary of how the editable installation is performed:

https://setuptools--3485.org.readthedocs.build/en/3485/userguide/development_mode.html

I hope this is useful.

40

Mypy does not find py.typed in non-strict mode

While testing the new pep660 feature I have found that mypy does not work as expected. This seems to be a regression compared to the legacy mode.

In short, I have a project in a flat layout containing a py.typed file within the toplevel folder of the source
e.g. debugproject\debugproject\ contains __init__.py foo.py and an empty py.typed file.

The file is included as package data with the following config in setup.cfg

[options]
zip_safe = False
packages = find:

[options.package_data]
* =
    py.typed

I have verified that the py.typed file gets installed as expected using pip install . and is a part of the bdist/sdists produced by python -m build

Now I run mypy a simple script that imports a class from this project.

mypy .\test.py

where test.py contains a single import

from debugproject.foo import Foo

This raises an error when the package is installed in editable mode using the default non strict config.

test.py:1: error: Cannot find implementation or library stub for module named "debugproject.foo"
test.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

With legacy editable install this type checks correctly.
The same is true using the compat and strict mode.

A sample project with this issue can be found here (Updated compared to my last comment)

41

Thank you very much for the help testing Jens, and the well thought reproducer.

I checked your example and I think I have an idea what is going on:

It would seem that mypy is not taking into consideration the existence of finders defined outside of the stdlib.

Maybe I am wrong here, but I reached this conclusion after running importlib_resources.files(debugproject).glob("*") and checking that py.typed is indeed listed as part of debugproject.

I believe that this limitation is closely related to the discussion in a different topic about PEP 660, IDEs and static code analysis. If we look at that thread and replace the occurrences of “IDE” with “static code analysis tool”, everything is pretty much applicable for this use case.

The TL;DR that I got from this other thread is that the packaging community is open to discuss different mechanisms to facilitate interoperating with static analysis tools, but this topic of conversation has not been explored yet.

On the bright side, if you install your project using the strict mode, mypy seems to work well:

pip install -e . --config-settings editable-mode=strict
echo "from debugproject.foo import Foo" > test.py
.venv/bin/mypy test.py
# Success: no issues found in 1 source file

I also believe that projects using the src-layout should work as normally.

42

Thanks @abravalheri I also just discovered Add support for PEP-610 editable packages (#12313) by hashstat · Pull Request #12315 · python/mypy · GitHub / Support finding "editable" type annotated packages made editable through direct_url.json · Issue #12313 · python/mypy · GitHub which looks like it may be trying to solve the issue from the mypy side.

I can also confirm that strict mode does indeed resolve the issue.

1 Like
43

Thanks for working on bringing PEP 660 support to setuptools.

If I understand correctly there’s three editable install modes - strict, lax and compat - but lax, which is the default, does not appear to be documented? Is the “redirecting” hook used for lax installs, or how does lax work? What does compat do?

44

@abravalheri can correct me, but as I understand based on what has been said previously, lax uses a custom import finder hook (at least for implicit layout projects, as implied above, not sure what is used for src ones), compat uses .pth (like the legacy setup.py develop) and strict uses symlinks/hardlinks.

45

Hi @layday thank you very much for the kind words.

The way the installation works it similar to what @CAM-Gerlach have described (thanks Christopher). I will try to summarize it bellow:

  • The default behaviour when the user simply runs pip install -e . will depend on the project structure.
    I tried to implement a trade-off between allowing the users to edit files freely (including renaming, deleting and adding files[1]) and preventing sys.path from being polluted with auxiliary scripts/folders the user might have added to the project repository.
    Currently this means:

    • For a project using a src-like layout that does not mess with the package_dir config: a static .pth file is used (since src layouts are relatively safe in terms of sys.path pollution).
    • Otherwise, a finder is installed taking advantage of import hooks.

  • Users can opt into a “stricter” behaviour with pip install -e . --config-settings editable_mode=strict, which means that setuptools will try to emulate as close as possible a regular wheel installation. Currently this is implemented using a link farm in an auxiliary folder which is added to sys.path.

  • Temporarily users can opt into a behaviour that is compatible with what the develop command would do. When you run pip install -e . --config-settings editable_mode=compat, setuptools will always use a static .pth file (even for flat-layouts, no trade-off is considered).
    However compat is only meant to help during a transition period when will eventually go away (I am choosing the end of the year, happy to revise this date if necessary).

Although I briefly mention all the possibilities of how setuptools achieves an editable installation in this section of the docs, I am deliberately avoiding being categorical in the text, because this is meant to be an internal implementation detail and I don’t want users to rely on the specific method so it can change in the future[2].

  1. Which I am lately calling in the code “lenient” as a direct opposite to “strict”, but the exact name should not matter, since I prefer users to think of it as simply “the default behaviour”. ↩︎

  2. For example, we might decide to revise the implementation once we have more information from Clarification about how to implement namespace packages (as in PEP 420) via import hooks for PEP 660 use case · Issue #92054 · python/cpython · GitHub or when we start getting feedback from real world usage. ↩︎

1 Like
46

The reason why I chose to go with a trade-off instead of simply using static .pth files all the time is to try to achieve some compromise between the two sides involved in the previous discussions about setuptools and PEP 660 (the ones advocating for a strict behaviour by default and the ones advocating for a lenient behaviour by default).

The reason why I did not choose to unify all the different modes into custom finders with complex import hooks, is that (in my opinion) they present several limitations. At least I noticed the following:

  • there is no official answer for implicit namespaces yet (python/cpython#92054)
  • they don’t work with pkgutil/pkg_resources namespaces
  • data/resource files inside package dirs (intentionally excluded from the build via configuration) would probably leak (which is not great for strict installs).

In the end of the day it just seems that a static .pth file or a link farm will match more closely the users’ expectations… So I try to use them whenever possible.

47

Oh, I didn’t realise that this was temporary. Is there any specific reason that we want to go through churn of changing things for users immediately after introducing this functionality to them?

48

Hi Pradyun, thank you very much for the feedback.

I see things from a slightly different perspective:

We are changing the way setuptools work because of a few motivations: to catch up with PEP 660, to avoid making internal scripts/packages/modues available via sys.path, and to address a series of accumulated issues because of the limitations of the static .pth approach (e.g.pypa/setuptools#230, pypa/setuptools#1801, pypa/setuptools#2662, pypa/setuptools#3399, …).

Between the default editable exerience and the strict mode, setuptools should be able to support all kinds of project layouts. The cost of adding these improvements will be (justifiably) some level of churn until the users understand how to deal with this new dynamic[1].

The only use case identified so far that is not solved by either the default behaviour or the strict mode is the one presented by Stéphane: a flat-layout package using pkgutil namespaces that try to reach for files outside of the package directory with Path(__file__).parent.parent.parent.
If I understood correctly, this use case does not really fit the mindset of PEP 660 (I think that no PyPA standard offers any guarantee about accessing files outside the package directory using the value of __file__).

Therefore, the compat mode is not really a functionality that we are introducing or that we want the users to learn about. Instead, it is a temporary escape hatch (I hope that the documentation makes this clear). This way users have some time to implement any required changes or report use cases that are not covered by the other “blessed” editable installation modes.

I am more than happy to reconsider compat if we identify other use cases requiring it.
But since I personally don’t plan to keep providing support for compat[2], I believe the best is to let it go after the end of the year.

  1. Any sufficiently ambitious/complex change or PEP unfortunately will have this side effect in setuptools. ↩︎

  2. Given the limitations of the static .pth approach, fixing issues or adding features would likely require using file links or path hooks, which is already what the default behaviour / “strict” mode do. ↩︎