PEP 660: Editable installs for PEP-517 style build backends

Packaging
62

I think I need some clarification. Currently the tool “stack” consists of:

  • Build backend - responsible for building the package. Chosen by the package maintainer.
  • Build frontend - responsible for calling the build backend and spitting out a ready to be installed .whl file. Chosen by the package end-user.
  • Installer - responsible for installing a .whl file created by build backend and build frontend working together. Chosen by the package end-user.

We can consider 2 different scenarios regarding the end-user part (build frontend and installer):

  • two separate tools, one being a build frontend, and one working as an installer (e.g. pypa/build and pradyunsg/installer pair).
  • one tool working both as a build fronend and an installer (e.g. pip)

Am I right so far?

Given that, which part of the stack is responsible for creating a direct_url.json? If it’s the installer, how can it know when it’s installing an editable wheel? How can it know what exactly should be inside a direct_url.json file? Of course if the installer is consolidated in one tool with the build fronted, this information can be just passed internally, but it won’t be possible when using 2 separate tools.

If that’s the case, I think we should either:

  • move the responsibility to the build frontend or build backend,
  • provide an interface to pass this information to the installer,
  • restrict the editable builds feature to tools that are working both as a build frontend and an installer (which I think should be avoided).
1 Like
63

Personally, I consider the PEP 517 workflow to be

  1. Build backend - responsible for implementing the PEP 517 hooks and doing the actual build - creating wheels, etc.
  2. Build frontend - any tool that calls the PEP 517 hooks.

Build frontends can be any of a number of types of application: installers like pip, tools like build that manage creation of wheels/sdists, or just adhoc scripts.

Some frontends, notably installers, and in particular pip, want to be able to do an “editable install”, where the installed project references the “live” source code of the project. This PEP adds a new backend hook that says “give me the necessary file structure to make an installation that works as an editable version of this project”. The hook is defined to work by returning a wheel format file, because front ends already know how to process such files (that’s frankly the only reason).

The pradyunsg/installer project doesn’t really fit here at all. It’s not a PEP 517 build frontend at all. In this context it’s simply a library that handles the job of unpacking wheels, which an installer application might use (but doesn’t have to, any more than it has to use any given HTTP library).

The editable install PEP also makes some requirements on tools that install (unpack) wheels returned from build_wheel_for_editable, specifically around writing a direct_url.json file. So as a PEP 517 frontend supporting the various “database of installed packages” specifications, pip would be responsible for writing that file, just like it’s responsible for writing the RECORD and METADATA files.

pradyunsg/installer, as a library for unpacking wheels, might choose to help by writing direct_url.json. But it doesn’t have to - and it doesn’t for any of the other uses of direct_url.json PEP 610 applies to, so there’s no obvious reason why it would in this special case.

Hope that helps - yes, it is complicated :slightly_frowning_face:

1 Like
64

Thanks for the clarification on direct_url.jon @sbidoul, I clearly skimmed over PEP-610 too quickly :confounded:

Your second point on scheme brings an other question. The PEP currently only says:

scheme: a dictionary of installation categories { 'purelib': '/home/py/.../site-packages', 'platlib': '...'}. This makes it possible to
use relative paths to the source code, which might help the interpreter find
the package after the root path changes with chroot or similar.

But it is still unclear for me why scheme is actually needed for building an editable wheel (and/or why it isn’t needed for “classical” wheels and the build_wheel hook).

Finally, I’m uncomfortable with the recommendation to add a +editable local identifier to the version.
It seems like a hacky solution to an unclear problem. What should happen if the version already has a local version identifier ?
I would expect pip install . and pip install -e . to install the exact same version.

The explanation of “so as to make editable installs easier to identify by users inspecting installed distributions” seems weak to me since the direct_url.jon file should already contain this information.

If we want to make editable wheels identifiable, I’d be more comfortable with a new key in {distribution}-{version}.dist-info/WHEEL. And in fact, on a slightly off-topic note, I think that it might make sense to also store there the backend (and its version) used to produce the wheel.

2 Likes
65

Agreed. I’m not clear why this is needed, either. The editables project doesn’t need that information, so I’m not sure what the backend would do with it if it’s using editables.

66

We are calling the wheel editable but it’s the project that is editable.

The wheel is intended to be generated then immediately installed and deleted.

The scheme allows the backend to produce relative paths to the project. Normal wheels do not care about the relationship between the source code and the installed path.

The dist-info/WHEEL file already stores the name and version of the backend.

67

Given that moving virtual environments isn’t supported, is this needed?

68

Is the editable wheel always installed into a virtual environment, and do we want relocating to be even less supported?

69

I don’t think so. E.g. within a docker image you can use the system Python. Relocating is probably out of scope, at least from my POV.

70

No, and yes :slight_smile:

To be more explicit, what is the actual use case that having the schemes (and having relative paths in the wheel) is intended to support? Is it acceptable for backends to ignore the scheme and always use absolute paths? If so, what functionality will the users lose?

Currently setuptools appears to use absolute paths in editable installs, so there’s no precedent for relative editables that I can see. Is there any reason the PEP is trying to add support for something that no existing tools allow?

71

Right, I somehow also missed that (PEP skimming is clearly not my forte :sweat_smile: )
This could then maybe be expanded to also include the used hook and when relevant the config_settings as JSON ?

We currently have:

Wheel-Version: 1.0
Generator: bdist_wheel (0.36.2)
Root-Is-Purelib: true
Tag: py3-none-any

which might become

Wheel-Version: 1.1
Generator: bdist_wheel (0.36.2)
Generator-Hook: build_wheel_for_editable
Generator-Config-Settings: {"CC": "gcc"}
Root-Is-Purelib: true
Tag: py3-none-any
2 Likes
72

Doesn’t the PEP only exist to support things that no existing tools allow? A generic way to link source repositories to Python environments? I thought it might be useful to support relative paths. We could allow scheme to be None, and then both the backend and the frontend could ignore the feature if they choose.

73

Personal opinion:

The PEP should be defining a standard way for tools to implement useful functionality. The existing setuptools setup.py develop mechanism has proven usefulness, and people have explicitly asked for that functionality in a form that isn’t tied to setuptools. So my feeling is that “support the features setuptools offers across all backends” is the baseline here.

I don’t actually think we need anything more than that - the setuptools mechanism appears to have been perfectly successful in spite of its flaws and limitations, so while I personally prefer a more precise and detailed spec, I’m fine with not trying to pin down details that setuptools has survived without.

It’s perfectly OK for the PEP to offer extra features that setuptools doesn’t have, but as a front end maintainer I’d want to limit the amount of complexity I have to support, and I’d be against extra complication just to provide something that “might be useful”. If there’s a good justification, then fine, but otherwise let’s keep things simple. In this case, having to know the install scheme when we request the editable wheel, and not just when we install that wheel, seems like unnecessary extra coupling that might make it more difficult for an installer. At the very least, it adds a risk of bugs if what gets passed to the hook doesn’t match what’s used to install. And currently the PEP doesn’t say what should happen then, so there’s no guidance on how to fix such bugs.

Furthermore, it’s a huge warning sign to me that I’d expect the setuptools implementation of this new hook to completely ignore the scheme argument.

PEP Delegate opinion:

If the PEP contains a feature, I’d expect it to include an explanation of why that feature is needed. “It might be useful” isn’t sufficient, and I’d reject the PEP with a request to provide better use cases if it were submitted for approval in that case.

1 Like
74

That’s no fun at all. Hasn’t setuptools been deprecated since 2013 when wheel came out? :slight_smile:

I have noticed that with Docker & containers I am looking at my code through a different root than the one it will run under more often than not, through a bind mount. That’s the main reason I think relative paths will be increasingly useful.

75

It is true that direct_url.json provides the information that the install is editable, and might suffice for some use cases. Nevertheless I proposed this for the following reasons (which I will clarify in the PEP unless we conclude it’s a bad idea):

  1. a casual look at the list of installed versions, possibly with tools that do not support PEP 610 yet, would reveal the editable-ness by looking at the version
  2. the version found in installed .dist-info metadata may be out-of-date compared to the actual version of the project and the +editable local version identifier can serve as a warning for that situation for users looking at installed distributions
  3. public index servers reject wheels with local version identifiers, so if one escapes and someone tries to upload it, it will be rejected
  4. also in case the wheel escapes, it clearly shows in the file name that it is not a regular wheel

1 and 2 can be covered by tools that enhance their UI using PEP 610, but tools that are not updated for that would readily benefit from the editable local version identifier. 3 and 4 would need alternatives if we remove that from the PEP.

If there is already a local version identifier, I’d recommend appending .editable to it.

IMO, re-reading PEP 440, that usage is not an abuse of local version identifiers.

Your proposal to update the WHEEL metadata sounds interesting to me. Complementary I’d say ?

76

Hmm, this makes me think. How would this work with version specifiers? If foo 1.0+mylocal.editable is installed, as a result of installing 1.0+mylocal in editable mode, that wouldn’t satisfy a requirement foo==1.0+mylocal:

If the specified version identifier is a local version identifier, then the local version labels of candidate versions MUST be considered when matching versions, with the public version identifier being matched as described above, and the local version label being checked for equivalence using a strict string equality comparison.

Yes, it’s an obscure corner case, but PEP 440 goes into a lot of detail about corner cases like this, and I don’t think we want to have to deal with people complaining that installing a local version in editable mode breaks their dependency data…

Because of this, I’m inclined to say that editable installs must have identical metadata to the equivalent non-editable install. That seems far simpler, and ensures that there’s no possibility of differing behaviour (it’s also what setuptools currently does, so we know it doesn’t cause people issues in practice).

The point about public indexes is a valid one, but I don’t think it’s worth the risk of compatibility problems.

So my counter-proposal would be:

  1. Backends MUST ensure that wheels generated by build_wheel_for_editable have exactly the same metadata as if they had been built using build_wheel.
  2. Tools SHOULD properly support PEP 610 and direct_url.json, so that they can correctly report editable installs.
  3. Formally add the local platform tag, noting that it is intended to reflect “the machine the wheel was built on only” (even though in practical terms, tools won’t be expected to try to enforce that) and that installers MUST always include local in their list of supported platforms. Indexes MUST NOT allow uploading of wheels with the local platform tag. Installers should prioritise local wheels over more generic wheels like manylinux or any.
  4. build_wheel_for_editable MUST return a wheel with the local platform tag. As indexes don’t allow uploading local wheels, this protects against editable wheels getting accidentally published.

Yes, it’s more work, and it requires us to change more of the ecosystem to bring in the local tag, but it’s much more in line with existing standards, and (to me, at least) it feels like a much cleaner design with fewer edge cases to worry about.

The local platform tag also has a history of people requesting something like it, so it already has a good justification. The biggest problem might be that I think some people argued that local wheels should have a low priority. If that becomes a sticking point, we could change the tag to editable and say it’s only for editable wheels, if that avoids an extended debate.

3 Likes
77

Requiring the same metadata prevents us from adding extra dependencies, which we will likely need if the package requires any generation/compilation step.

78

This is exactly the architecture which creat it own problems. All of these lengthy discussion cycling back to the same point. Wheel would never cover all the user cases event if one try as hard as they could. It is by design.

79

Well spotted, thanks! Editables adds a dependency so I should have thought of that :slight_smile:

80

@pf_moore pinning on a local version identifier and installing it as editable is indeed an edge of the edge case I had not considered. Well spotted.

I’m going to remove that part of the text and move it to rejected ideas.

To make the wheel generated by build_wheel_for_editable recognizable as such in case it escapes, shall we combine an editable build tag and the local platform tag ?

So: pkg-1.0.0.dev0-editable-py3-none-local.whl ?

I’m not sure which spec needs updating to introduce the local platform tag, nor what the text would look like.

81

The wheel spec says build tag must start with a number. It looks like it’s more intended to be a build number than a textual tag (it sorts by initial numeric part then remaining text), so I fear this may be an abuse of the field, and could cause problems with someone’s weird build pipeline that we don’t know about :slightly_frowning_face:

The platform compatibility tags spec. Which is derived from PEP 425 which is unfortunately pretty vague, so I think writing a bunch of standalone text and bunging it in as a standalone section in the compatibility tags spec would be sufficient.

My suggestion:

Local platform tag

Wheels may use local as a platform tag. This tag should be interpreted as meaning that the wheel is only valid on the exact system it was built on. Using a wheel with this tag on any other system is unsupported.
The local tag should be treated as the highest priority of platform tag by installers, to be chosen in preference to any other platform tag.
The local tag should not be combined with other platform tags in a “compressed tag set” - the interpretation of such a combination is not defined by this spec and installers are allowed to interpret such a combination in an implementation-defined manner (including raising an error and refusing to install such wheels).

Arguably, adding this should be a separate PEP, but I’m personally comfortable with making it part of the editable PEP, for convenience. If anyone wants to insist that it’s a PEP in its own right, please speak up.

One thing I’d note about adding a local platform tag is that it adds a whole bunch of extra options to the list of “Compatible tags” for a system (what packaging.tags.sys_tags() returns). On my Windows PC, it adds 21 new tags to the existing 33 (a local version of every win_amd64 tag), getting the list up to 54 entries. On Ubuntu (wsl) I see 87 at the moment, with 19 more being added. While that’s not a disaster, I do think that the “list every combination you accept” mechanism for tags is starting to show signs that it might not scale indefinitely :slightly_frowning_face:

1 Like