PEP 660: Editable installs for PEP-517 style build backends

Packaging
1

This is the draft PEP that @dholth and myself have prepared, following the ideas raised in Standardising editable mode installs (runtime layout - not hooks!).

The text became a bit longer as we tightened loose ends and reflected the discussions in the previous thread.

There are proof of concepts in pip, flit, enscons, and possibly hatch (although I don’t find the link right now).

Looking forward to reading your feedback.

PEP: 9999
Version: $Revision$
Last-Modified: $Date$
Title: Editable installs for PEP-517 style build backends
Author: Daniel Holth <dholth@fastmail.fm>, Stéphane Bidoul <stephane.bidoul@gmail.com>
Sponsor: Paul Moore <p.f.moore@gmail.com>
Discussions-To: 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-Mar-2021
Post-History: 


Abstract
========

This document describes a PEP 517 style method for the installation of packages
in editable mode.

Motivation
==========

Python programmers want to be able to develop packages without having to
install (i.e. copy) them into ``site-packages``, for example, by working in a
checkout of the source repository.

While this can be done by adding the relevant source directories to
``PYTHONPATH``, ``setuptools`` provides the ``setup.py develop`` mechanism that
makes the process easier, and also installs dependencies and entry points such
as console scripts. ``pip`` exposes this mechanism via its ``pip install
--editable`` option.

The installation of projects in such a way that the python code being
imported remains in the source directory is known as the *editable*
installation mode.

Now that PEP 517 provides a mechanism to create alternatives to setuptools, and
decouple installation front ends from build backends, we need a new mechanism
to install packages in editable mode.

Rationale
=========

PEP 517 deferred "Editable installs", meaning non-``setup.py``
distributions lacked that feature. The only way to retain ``editable`` installs
for these distributions was to provide a compatible ``setup.py develop``
implementation. By definining an editable hook other build frontends gain
parity with ``setup.py``.

Terminology and goals
=====================

The editable installation mode implies that the source code of the project
being installed is available in a local directory.

Once the project is installed in editable mode, users expect that changes to
the project *python* code in the local source tree become effective without the
need of a new installation step.

Some kind of changes, such as the addition or modification of entry points, or
the addition of new dependencies, require a new installation step to become
effective. These changes are typically made in build backend configuration
files (such as ``pyproject.toml``), so it is consistent with the general user
expectation that *python* source code is imported from the source tree.

The modification of non-python source code such a C extension modules obviously
require a compilation and/or installation step to become effective. The exact
steps to perform will remain specific to the build backend used.

When a project is installed in editable mode, users expect the installation to
behave identically as a regular installation. Depending on the way build
backend implement this specification, some minor differences may be visible
such as the presence of additional files that are in the source tree and would
not be part of a regular install. Build backends are encouraged to document
such potential differences.

The Mechanism
=============

This PEP adds a single optional hook to the PEP-517 backend interface. The hook
is used to build a wheel that, when installed, allows that distribution to be
imported from its source folder. 

build_wheel_for_editable
------------------------

:: 

  def build_wheel_for_editable(
      wheel_directory,
      scheme=scheme, 
      config_settings=None):
      ...

``scheme``: a dictionary of installation categories ``{ 'purelib':
'/home/py/.../site-packages', 'platlib': '...'}``. This makes it possible to
use relative paths to the source code, which might help the interpreter find
the package after the root path changes with ``chroot`` or similar.

Must build a ``.whl`` file, and place it in the specified ``wheel_directory``.
It must return the basename (not the full path) of the .whl file it creates, as
a unicode string.

May do an in-place build of the distribution as a side effect so that any
extension modules or other built artifacts are ready to be used.

The .whl file must comply with the Wheel binary file format specification (PEP
427). In particular it must contain a compliant .dist-info directory.
Metadata must be identical as the one that would have been produced by
``build_wheel`` or ``prepare_metadata_for_build_wheel``, except for ``Version``
and ``Requires-Dist`` which may differ slightly as explained below.

Build-backends are encouraged to append a distinguishable PEP 440 *local version
identifier*, such as ``+editable``, so as to make editable installs easier to identify
by users inspecting installed distributions. Since public index servers reject wheels
containing local version identifiers, this approach has the interesting property of
preventing accidental publishing of editable wheels. The version must otherwise be
identical to what would have been produced by ``prepare_metadata_for_build_wheel``
so as to ensure correct dependency resolution.

Build-backends must produce wheels that have the same dependencies
(``Requires-Dist`` metadata) as wheels produced by the ``build_wheel`` hook,
with the exception that they can add dependencies necessary for their editable
mechanism to function at runtime (such as `editables`_).

The filename for the “editable” wheel need not use the same tags as
``build_wheel`` but must be tagged as compatible with the system.

An “editable” wheel uses the wheel format not for distribution but as ephemeral
communication between the build system and the front end. This avoids having
the build backend install anything directly, but it is probably not useful to
cache or distribute these wheels.

get_requires_for_build_wheel_for_editable
-----------------------------------------

::

  def get_requires_for_build_wheel_for_editable(config_settings=None):
      ...

This hook MUST return an additional list of strings containing PEP 508
dependency specifications, above and beyond those specified in the
``pyproject.toml`` file, to be installed when calling the
``build_wheel_for_editable`` hooks.

If not defined, the default implementation is equivalent to ``return []``.


What to put in the wheel
------------------------

Build backends may use different techniques to achive the goals of an editable
install. This section provides examples and is not normative.

* Build backends may choose to place a ``.pth`` file at the root of the ``.whl`` file,
  containing the root directory of the source tree. This approach is simple but
  not very precise, although it may be considered good enough (especially when
  using the ``src`` layout) and is similar to what ``setup.py develop``
  currently does.
* The `editables`_ library shows how to build proxy modules that
  provide a high quality editable installation. It accepts a list of modules
  to include, and hide. When imported, these proxy modules replace themselves
  with the code from the source tree. Path-based methods make all scripts under
  a path importable, often including the project's own ``setup.py`` and other
  scripts that would not be part of a normal installation. The proxy strategy
  can achieve a higher level of fidelity than path-based methods.

Frontend requirements
---------------------

Frontends must install editable wheels in the same way as regular wheels.
This also means uninstallation of editables does not require any special treatment.

Frontends must create a ``direct_url.json`` file in the ``.dist-info``
directory of the installed distribution, in compliance with PEP 610. The
``url`` value must be a ``file://`` url pointing to the project directory
(i.e. the directory containing ``pyproject.toml``), and the ``dir_info`` value
must be ``{'editable': true}``.

Frontends must execute ``get_requires_for_build_wheel_for_editable`` hooks in
an environment which contains the bootstrap requirements specified in the
``pyproject.toml`` file.

Frontends must execute the ``build_wheel_for_editable`` hook in an environment
which contains the bootstrap requirements from ``pyproject.toml`` and those
specified by the ``get_requires_for_build_wheel_for_editable`` hook.

Frontends must not rely on the ``prepare_metadata_for_build_wheel`` hook when
installing in editable mode. They must use ``build_wheel_for_editable`` and
inspect the resulting wheel.

References
==========

.. _`editables`: https://pypi.org/project/editables/

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
7 Likes
2

Is there a render version available? (e.g. on GitHub) The long code block is pretty difficult to read.

2 Likes
4
2 Likes
5

To be honest, I’d prefer it if anything that isn’t purely typo corrections gets discussed here, as the github tracker is not as easy to follow for discussions (and has a tendency to hide comments a\s “outdated” when the file changes.

Please can we keep anything substantive on here - I will not be reviewing github traffic when it comes to making a decision on the PEP.

3 Likes
6

I am sorry if any of this is repeated from the previous thread, but I did not participate and it was too big for me to read the full history. And also sorry if this isn’t the kind of feedback you wanted on this thread, if that’s the case, let me know and I’ll post in on the original one.

In my opinion, there are 2 issues with the current proposal.

Building a .whl seems like a footgun. These wheels are tied the specific machine and setup, they are essentially internal artifacts of the editable install process.
I do like that we are not asking the backends to do the install themselves, because that would likely result in a huge mess, but using the .whl extension seems wrong.

I would propose to change the extension to something else, we can still ask for the file to implement the format described in PEP 427, but IMO the extension shouldn’t be a .whl.
In doing this, we can also skip all the compatibility stuff. We can simply say that the artifact is meant to only be compatible in the current system, and in the particular current state.

There has been some talk, and I understand it is the intent, of removing the executable functionality of .pth files. How does this proposal account for that? As far as I can tell, it relies on this mechanism to provide update-on-import modules.

.pth files are also not optimal… They only run on the Python module import, which makes it impossible to update package data if the data users don’t import my module.
I can live with this quirk, but I just wanted to point out it is not optimal.

We are designing a new system, so I’d like to not have this kind of issues as a starting point, as we are in a position to fix them. However, this approach is fairly future-proof. It simply builds an install artifact and relies on already existent mechanisms, so we can add a mechanism to fix this later.

With that said, I’d really like to see 1) addressed, and then we could deal with 2) later in a future PEP. Since this has prolonged from a long time, it would make perfect sense to me to split the issue into two PEPs, one introducing the base structure and hooks, and one later introducing a mechanism to replace .pth files, if others agree that we should do that.

7

To me the PEP read that it was only specifying the interface between build front-ends and back-ends when installing packages as editable, like PEP 517. It only suggests 2 example mechanisms for how those packages are editable. .pth files could be phased out, but that’s completely independent of the interface.

Is there a case when a standard user would even see this ephemeral wheel? If the format exactly matches, I would argue that having a different extension would only introduce more confusion for implementers down the line. The compatibility stuff should already be there from the build_wheel part of PEP 517.

8

Ideally they should not, but there’s no way we can’t prevent this.

Wheel installer CLIs should straight up refuse to install such wheels, which is not possible with the current proposal.

Implementers need to look at the spec, this would be a very minimal detail, I really don’t think this is an issue.

9

I don’t think this is correct. There have been discussions to disallow arbitrary code execution from pth files. However, the intent is not to stop having this functionality but create a dedicated interface for it. See PEP 648 – Extensible customizations of the interpreter at startup | peps.python.org for more details. So it’s more of a case of changing the implementation detail then to remove pth.

If your module is not being imported, it’s not being used; so it’s actually good/doesn’t matter that it’s not updated.

IMHO, maybe sometimes. But generally no. These wheels will be mostly ephemeral and we can easily add some metadata that makes them fail on other machines at install time (they would already fail at load time). I don’t like moving away from the wheel because now install frontends need to support a new binary. With the current design the install frontend never needs to change, only the backend that’s generating the blob. This would make it much easier to adopt it.

1 Like
10

I guess this is semantics, but I’d say they are definitely planning on removing the executable functionality of .pth files, it’s just that they are proposing an alternative mechanism to solve the issue (completely outside .pth files).

This is not true, I know several people who are shipping data-only packages. I, myself, am planning to ship data-only packages.

Frontends don’t need to support a new binary, it’s the same binary, the only thing that changes is the name.

Frontends will need to change anyway because nobody is supposed to be pip install(-ing) my-editable-wheel.whl. The way this is intended to be used is by asking frontends to install the project in editable mode, the wheels should be ephemeral. The use case where frontends do not need to change is exactly the use-case that should not be happening, nobody should be manually installing editable wheels – I will not add support for the editable wheel hook in the CLI of pypa/build. The actual use-case that is intended will require frontends to change, as they now will need to use this new mechanism.
I think the argument that frontends do not need to change doesn’t hold almost any value in this situation, so, given that the wheels should not be used directly, but there isn’t really anything enforcing that, IMO we should be preventing issues where try to do just that, it will inevitably happen.

I have a really strong opinion here, for what that is worth.

11

From the POV of this PEP when that PEP is accepted we can just switch from using pth files to the new functionality. The PEP generally are referring to the latest accepted standards, to be independent of each other. For example, PEP 648 might get rejected, in which case pth files remain, and will remain until we create an alternative for them. We generally don’t drop functionality without offering an alternative.

The way CPython 3 is designed (and evolving) those data files should be loaded with importlib.resources and that goes through the import system also. The design here would not need to change to use that. If you want to use resource (data) files without going through importlib.resources then this might not work for you, but in that case, you might first standardize the alternative usage, and then we can come up with a solution for that.

I think you’re conflating a few things, so let’s try to clear this up. Components here are:

  • build backend (setuptools, flit)
  • build frontend (build project),
  • installer (installer ).

pip here is a combination of things, but you can think of as just an orchestartor. When you say pip install -e ., that roughly is translated by pip to:

  • provision a build backend to build an editable wheel (acts as a build frontend),
  • call the backend to build the editable wheel (for this and previous might just delegate the job to the build project),
  • install the editable wheel to the target environment (for this it might have it’s own installation logic, or again just delegate it to the builder - act as an installer).

So in this scope, if we change the extension the installer has the right to reject to install that as a wheel, per PEP-427. It’s no longer a valid wheel (note though this change affects the installer, not the build frontend). The separation of concerns is important because, for example, tox might decide to not use pip, or have its own implementation of the installer to speed up operations (virtualenv already does this when provisioning seed packages).

Maybe not pip install them, but definitely use installer to do so (as said various development workflow must install these generated wheels somehow). I’d argue though there’s no reason why not to allow pip install too.

As said above, build frontends only build the project. Installing wheels is the responsibility of wheel installers. When the wheel gets passed between the build frontend to the wheel installer the wheel needs to be persisted, it can’t be ephemeral.

We don’t need to be that draconian. There’s no real reason not to be backwards compatible here, and allow editable wheels to work when there’s a valid wheel installer available.

Perhaps this discussion and decision should be made together with the other build maintainers, not solely by you.

Kinda redundant, but yeah, me too. Like stronger than titanium.

12

@FFY00 The discussion of “wheel vs not wheel” is literally the bulk of the other thread.

IIUC, that long discussion is where everyone reached concensus on the choice that has been made. Skipping that discussion to bring up something that has been discussed extensively is not helpful IMO, and I’ll request you to read everything before Standardising editable mode installs (runtime layout - not hooks!) - #42 by dholth and the 4-5 posts before Standardising editable mode installs (runtime layout - not hooks!) - #90 by dholth.

If that discussion still doesn’t convince you, please bring up your concerns as a reply on that thread.

13

@ffy00 do I understand correctly that the core of your concern is that you are looking for a way for frontends to detect that a wheel is an “editable” one ?

If there is no such way, I personally don’t think it would be a big problem in practice:

  • these wheels will normally not be exposed to users
  • the PEP recommends adding a local version identifier (such as +editable) so if a user finds one it will be easily recognizable as “editable”
  • even if the user insists on installing that wheel, not much harm will be done, as the only thing an installer frontend should do differently is generate an adapted direct_url.json

That said, if we conclude the risk is significant nevertheless, or that there would be other benefits, then I would prefer another approach than changing the .whl extension. Indeed it is still a valid wheel. As an alternative, we could, for instance, make the local version identifier containing the editable string a MUST for the backend.

2 Likes
14

We could also revive the proposal for a “local” platform tag that means “only valid on the machine it was built on” and mandate that editable wheels use that platform tag. But that can be a follow-up PEP, it doesn’t have to be part of the first iteration of the editable install spec. We could wait until we confirmed there’s a problem in practice before taking that route.

1 Like
15

Installers are used to installing wheels these days, so if you look at the pip source code you’ll see that installing a not-wheel would be a separate and buggy endeavor. Rather than invent an almost-wheel we chose to use the format.

In the example a .pth file for editable would not need to be executable. It would only mention the source folder.

We don’t think it would be useful to copy these wheels to another computer, but if a clever person finds a way to make that useful in their specific workflow that would be fine.

16

My issue is not “whell vs not wheel”, I think it being a wheel is definitely the correct solution in this case, I just think these wheels should be differentiated from normal wheels – they serve a different purpose.

Here’s my issue, there’s nothing enforcing that. There is absolutely nothing preventing user i-like-caos from creating a tool that will build editable wheels and let users re-use them.

The issues we have seen with Python packaging over the years have not generally been due to bad design, but rather due to failure to predict how users will use, and abuse, the mechanisms.

User i-like-caos doesn’t see a problem, for them this is purely informational. They will try to use it as any other wheel.

This can cause some obscure errors. IMO the wheel should not be allowed to be installed in the first place, as it will generally successfully install and likely not work, which is bad UX.

The problem is that any wheel installer that does not know about this will gladly install it. We have issues getting users to move to newer pip versions, a lot of users will have access to such unequipped wheel installers.
Updating the wheel spec to add an editable marker would work.

This is a good idea, and something I have considered, however, I don’t see how that can be properly implemented. We would need to add a machine identifier, actually not only machine (hardware), but system. Most modern Linux distros will have this, systemd will handle it for you, on other distros we can use the DBus machine id, but there are systems that will not have such mechanism, or that it will not be very straight forward. And since we need to describe all this in the PEP, because the wheel builder and installer need to calculate the id the same way, it is not optimal.

An alternative, like I mentioned above, would be updating the wheel spec to include a editable marker. This would be essentially the same as having a different extension, I originally wanted to avoid it, but given the feedback here, it might make more sense.

editable would be different than local because it would only server this purpose. With local I would expect to be able to build a normal wheel in build_wheel and it not work on other systems, because the validation would fail.

17

Other than once installed the import would throw runtime error? Aka there would be no advantage in doing this. That being said you could add a meta-field into the wheel that identifies the machine, and disallow installation on other systems, but that can quickly become an issue with docker images and mounted disks (even with systemic ids and such).

1 Like
18

@FFY00 suggests naming the wheel something like package-1.0-editable-none-any.whl and having the installer accept the editable tag only when it knows it is installing an editable wheel. An interesting idea… although my preference is to enable and not prevent behavior.

4 Likes
19

It’s not as straightforward as that, because editable wheels might system to update the code.

If this is allowed, I can see a build backend trying to be smart when rebuilding native code and guess the files that should be built – let’s say it will try to build all .c files under ~/someproject – which will try to build random files on my machine and will likely error out with obscure errors.

Yeah, as I explained above, getting a system id is tricky.

1 Like
20

Could you describe the situations where you actually want this behavior? I am struggling to come up with anything.
In all situations I can think of, the user would be better off having something like pip generate the editable wheel and install it.
I don’t see much of an use for users to generate and then install editable wheels themselves, like I said in my first reply, this just seems like a footgun.

Anyway, sorry for taking over the thread :sweat_smile:
I will take a step back and come back later.

1 Like
21

I wanted to allow this all the time, and actually thought of this exact problem when proposed the import hook solution to expose projects as editable. So for me, this is a feature, not an issue. It’s up to the build backend to make sure the errors it throws in this instance are not obscure; rather than excluding this use case. Also, note the backend doesn’t have to guess, it can be explicitly configured too. Also can restrict binary builds to be only allowed on in-project sources.

1 Like