# Discuss PEP 662: Editable installs via virtual wheels

**URL:** <https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071>\
**Category:** Packaging\
**Created:** [June 3, 2021, 12:47pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071 "2021-06-03T12:47:29Z")\
**Posts on this page:** 20\
**Page:** 5

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [June 8, 2021, 3:40pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/83 "2021-06-08T15:40:44Z")

</div>

> [@pganssle](#):
>
> My thinking on the matter is that an editable installation is one where _at least_ the subset of files specified in the mapping are exposed to the interpreter for installation. Front-ends are free to expose more than that, including by simply adding each directory containing a file to expose to the path in the appropriate way (or symlinking in directories, or whatever).

OK, in which case we’re still in the confused situation of not knowing who’s responsible - the user says they expected to be able to create a new file and have it visible, the frontend says the backend didn’t say to expose that, the backend says that it’s the front end’s responsibility to do it, the front end says that it expects the backend to pass the directory back if that’s what’s wanted, etc.

So I guess we’re still no closer to knowing what “editable” is expected to mean. Ah, well. And the “virtual wheel” approach is still no closer to clarifying whose responsibility it is to say what gets exposed.

(A reminder - I’m fine with “we deliberately don’t specify” if that’s the position you want the virtual wheel PEP to take. I’m not trying to tell you how to write your PEP. I’m just explaining what I’ll be thinking about if I have to make a decision between two “competing” PEPs - although I hope it’s not being seen as a competition, we’re all looking for the best solution here 🙂)

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [June 8, 2021, 3:42pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/84 "2021-06-08T15:42:35Z")

</div>

> [@bernatgabor](#):
>
> I think it’s better to propose some way to support it though than not supporting it at all, so I’d still go with that recommendation. If the frontend can’t satisfy the backend’s request it should fail the installation.

OK. Can you explicitly make that point in the PEP please? I don’t want to have to hunt through hundreds of emails to find that recommendation in future…

---

<div class="post-metadata">

**Author:** ![ofek](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ofek/32/1033_2.png) [@ofek](https://discuss.python.org/u/ofek)\
**Post date:** [June 8, 2021, 4:20pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/85 "2021-06-08T16:20:12Z")

</div>

> [@takluyver](#):
>
> > [@bernatgabor](#):
> >
> > You can symlink on Windows. You need a new enough variant of it and symlinks enabled.
> 
> Sorry, I’ve got used to abbreviating this. I know that Windows has symlinks now, but the ‘enabled’ bit essentially kills it, as far as I’m concerned - as a software author, you can’t assume that symlinking works for arbitrary users on Windows, so general purpose, cross-platform tools can’t rely on making symlinks.

As a Windows user, this is correct.

> [@pganssle](#):
>
> My thinking on the matter is that an editable installation is one where _at least_ the subset of files specified in the mapping are exposed to the interpreter for installation. Front-ends are free to expose more than that, including by simply adding each directory containing a file to expose to the path in the appropriate way (or symlinking in directories, or whatever).

For me personally, I will forever continue having a stub `setup.py` if implementations do not automatically detect new or renamed sub-packages or files.

---

<div class="post-metadata">

**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\
**Post date:** [June 8, 2021, 4:23pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/86 "2021-06-08T16:23:58Z")

</div>

> [@ofek](#):
>
> As a Windows user, this is correct.

I guess all we conclude that it’s possible but not in general and is not on by default. As a Windows user I’ve spent 5 minutes post install to enable it and using it ever since 🤔 I’d say from my own POV was well worth the effort considering the benefits. So I think we should give this choice to the developers, to make symlinks a hard requirement to solve their use case. Not by default, but definitely an opt-in.

---

<div class="post-metadata">

**Author:** ![h-vetinari](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/h-vetinari/32/1220_2.png) [@h-vetinari](https://discuss.python.org/u/h-vetinari)\
**Post date:** [June 8, 2021, 4:33pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/87 "2021-06-08T16:33:19Z")

</div>

> [@bernatgabor](#):
>
> escape hatchet

This thread is full of them 🙃 (sorry to pick out an unintentional and understandable typo, it’s just quite apt…)

---

<div class="post-metadata">

**Author:** ![layday](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/layday/32/2771_2.png) [@layday](https://discuss.python.org/u/layday)\
**Post date:** [June 8, 2021, 4:35pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/88 "2021-06-08T16:35:31Z")

</div>

> [@pf\_moore](#):
>
> OK, in which case we’re still in the confused situation of not knowing who’s responsible - the user says they expected to be able to create a new file and have it visible, the frontend says the backend didn’t say to expose that, the backend says that it’s the front end’s responsibility to do it, the front end says that it expects the backend to pass the directory back if that’s what’s wanted, etc.

I don’t think that’s the case. I understood what @pganssle said to mean that the backend does not dictate what’s to be exposed - it still supplies the full set of files it would have included in a built wheel. In this setup, it will be necessary, however, for the backend to categorise the files, so that a frontend which wants to expose packages as a whole does not have to make assumptions about their nature. A rudimentary classification might include: `package`, `module` and `package_resource`.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [June 8, 2021, 4:46pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/89 "2021-06-08T16:46:45Z")

</div>

Sorry, I thought I’d seen a suggestion that the backend might alternatively return a directory if it wanted to expose the whole directory, meaning the backend could take two different approaches and the frontend had to know that. But I can’t find that now, maybe it was someone else.

This would all be a lot easier if this level of detail was included in the PEP 🙁

---

<div class="post-metadata">

**Author:** ![takluyver](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/takluyver/32/513_2.png) [@takluyver](https://discuss.python.org/u/takluyver)\
**Post date:** [June 8, 2021, 4:52pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/90 "2021-06-08T16:52:34Z")

</div>

> [@pf\_moore](#):
>
> Sorry, I thought I’d seen a suggestion that the backend might alternatively return a directory if it wanted to expose the whole directory, meaning the backend could take two different approaches and the frontend had to know that. But I can’t find that now, maybe it was someone else.

That was me, apologies for muddying the waters.

I wish it was possible to do it with _just_ ‘here’s the package directory containing all the code’. But sadly that’s incompatible with namespace packages (if I symlink foo from distribution 1, distribution2 can’t put anything under foo).

---

<div class="post-metadata">

**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\
**Post date:** [June 8, 2021, 4:59pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/91 "2021-06-08T16:59:03Z")

</div>

> [@takluyver](#):
>
> I wish it was possible to do it with _just_ ‘here’s the package directory containing all the code’. But sadly that’s incompatible with namespace packages (if I symlink foo from distribution 1, distribution2 can’t put anything under foo).

Can you clarfiy here a bit?

---

<div class="post-metadata">

**Author:** ![takluyver](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/takluyver/32/513_2.png) [@takluyver](https://discuss.python.org/u/takluyver)\
**Post date:** [June 8, 2021, 5:05pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/92 "2021-06-08T17:05:51Z")

</div>

The vast majority of use cases I’m interested in look something like: 'src/foo represents the entire foo package, so symlink that directory as `.../site-packages/foo`'.

Namespace packages allow different projects to install in the same package, so `foo.bar` could come from one project and `foo.baz` from another. This is OK if `foo` itself is the namespace - you just make `.../site-packages/foo` and create symlinks in there. But if `foo` is a regular package containing your app and `foo.plugins` is a namespace package, then you have to symlink all the other files under `src/foo`, rather than symlinking the whole directory. It’s not impossible, just an extra complication necessitated by a feature I’ve never liked.

---

<div class="post-metadata">

**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\
**Post date:** [June 8, 2021, 5:20pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/93 "2021-06-08T17:20:08Z")

</div>

> [@pf\_moore](#):
>
> OK, in which case we’re still in the confused situation of not knowing who’s responsible - the user says they expected to be able to create a new file and have it visible, the frontend says the backend didn’t say to expose that, the backend says that it’s the front end’s responsibility to do it, the front end says that it expects the backend to pass the directory back if that’s what’s wanted, etc.

No, to be clear it is always the frontend’s responsibility to choose between a wider install and a narrower install. The backend just says, “Here’s where all the files are supposed to be mapped from and to” and my SHOULD language was that they shouldn’t be trying to control how it gets installed. If you want the behavior where all the directories are exposed or some variation on that, that’s up to the front-end (which could be configuration options in a single front-end or many different front-ends specializing in different kind of editable installs).

To be concrete, say we have a backend whose internal rule is "install everything in `/a/b/myproj/src/myproj` as `myproj/`. If `src/myproj` contains ` __init__.py` and `foo.py`, it will pass the following to the front-end (simplified, not in the structure of Bernat’s PEP):

```auto
{
    "myproj/ __init__.py": "/a/b/myproj/src/myproj/ __init__.py",
    "myproj/foo.py": "/a/b/myproj/src/myproj/foo.py"
}

```

The front-end is free to use a mechanism that detects that adding `/a/b/myproj/src/` to `sys.path` is sufficient to expose `myproj` and do that, even though there may be other stuff in `/a/b/myproj/src/`. It’s also free to do something like generate a `.pth` file that creates an import hook that detects any `import myproj` imports and tries to import the relevant symbols from `/a/b/myproj/src/myproj/` (whether or not it’s in the mapping). Or of symlinking the directory `$PYTHON_ROOT/site-packages/myproj` to `/a/b/myproj/src/`. All valid modes of installation that different front-ends (or the same front-end with different configuration options) can choose from, with trade-offs evaluated by the end user.

It also has the option to symlink all the files and require re-installation to add new files, or to do something like generate a `.pth` file to install an import hook that bakes in the mapping, detects if a file _would_ be in the project root and trigger a custom error if someone has added `myproj/src/myproj/bar.py`, like:

```auto
ImportError: The module myproj.bar was not installed, but it is present in
the editable install directory: you may need to re-install the package to
expose new files, and/or update your project manifest.

```

Lots of options here, but at the core the responsibilities are very clear — the front end decides how strictly the installed layout matches the mapping.

And as I mentioned in an earlier reply, if we feel that a mapping of just files is not sufficient for front-ends to use to build robust heuristics that allow the behavior that users want in common use cases, we can take on a bit more complexity and design an optional, limited “here’s where I look for new files to go in the package” mapping. In which case the back-end would expose something like this:

```python
{
    "purelib": {
        "myproj/ __init__.py": "/a/b/myproj/src/myproj/ __init__.py",
        "myproj/foo.py": "/a/b/myproj/src/myproj/foo.py"
    },
    "content_expansion_hint": {
        "myproj": "myproj.*"
    }

```

(or even something slightly more complicated that says, "Everything under this directory except things that start with `tests/*`). In that case it would still always be the front-end’s responsbility to decide what to expose (as long as it always exposes _at least_ what’s specified in the mapping). The back-end isn’t required to provide hints, but it may, and the front-end isn’t required to use the hints, but if it does use them it can assume that they match the mapping.

**Edit** @layday’s suggestion of structuring the mapping may also work as a simpler version of “content expansion hints”. I’m not clear exactly how such a thing would work, but it would basically just be encoding information about the semantics that went into the mapping into the structure of the mapping, which I think is not a bad idea if done right.

---

<div class="post-metadata">

**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\
**Post date:** [June 8, 2021, 5:29pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/94 "2021-06-08T17:29:07Z")

</div>

> [@ofek](#):
>
> For me personally, I will forever continue having a stub `setup.py` if implementations do not automatically detect new or renamed sub-packages or files.

Well, I doubt “setup.py stub” will be a supported mode _forever_, but my whole opposition to PEP 660 was that you don’t need implementation **s** to support automatically detecting new or renamed sub-packages or files or whatever particular edge case you care about if you give this responsibility to the front-end, because you just need to have _at least one_ front-end that supports your use case.

It seems that a lot of people seem to be arguing as if the question is whether editable installs should be able to detect new files or not, whereas I think it’s clear that end users disagree on this, and so the best thing to do is to aim for a situation where each _individual_ (not each project) is deciding what heuristics and trade-offs to make in different situations. In my world, even if `pip` and every other major front-end decided to go with strict installations, you could write your own custom single-purpose tool that _just_ does the looser install you’d like, and it should work for every project out there (probably most of the tools you’d need to build such a thing will be off the shelf, too, like `pypa/build` for creating isolated build environments).

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [June 8, 2021, 6:22pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/95 "2021-06-08T18:22:52Z")

</div>

> [@pganssle](#):
>
> Lots of options here, but at the core the responsibilities are very clear — the front end decides how strictly the installed layout matches the mapping.

OK. Thanks for clarifying. With my pip developer hat on, I await with interest the availability of the various support libraries so pip can decide which one (or more than one) to use to implement this logic. And of course I’m fine if only one reference implementation of the functionality is ever developed, we’ll just use that.

I don’t imagine pip ever implementing the logic itself - it’s clearly (to me) something that needs to be in a library, so that we don’t get trapped in the whole “implementation defined functionality” situation again (if someone _did_ offer a PR to pip to implement this in-place I’d ask that it be split out as a reusable library).

That’s why I developed the editables library, to provide similar “off the shelf” functionality for backends in a PEP 660 world. But I won’t be doing the same for the “virtual wheel” proposal, because I don’t feel confident I know how to implement the logic. I’ll leave that to someone else who _does_ feel comfortable taking on that task.

---

<div class="post-metadata">

**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\
**Post date:** [June 8, 2021, 8:12pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/96 "2021-06-08T20:12:24Z")

</div>

A project will work under the editable strategy its developer happens to use. The other strategy or strategies will be untested by the author and may not work. We don’t know whether a ‘strict’ or a ‘loose’ editable install would wind up being more popular, but we know the strategy `setup.py develop` uses. Under this proposal the more popular editable install strategy would be more likely to work given a randomly chosen project.

Suppose I’m distributing a package on github only and I like to use a ‘loose’ editable install. Other people are doing ‘strict’ editable installs off the main branch of my repository. I work on my package, then make sure it also works under a ‘strict’ editable install every time I commit to the main branch.

If you want a strict install, and you don’t mind typing ‘pip install’ every time you add or remove files, that is an ordinary install. You have to test the real wheel before you can expect your package to work off pypi.

Suppose we figure out a set of globs etc. for a precise ‘loose’ install. What we should then do to make this proposal work is remove the `build_wheel` hook and use the virtual wheel structure only. So pip can guarantee the rules used to make the ‘editable’ install match the distributed wheel.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [June 9, 2021, 2:23pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/97 "2021-06-09T14:23:42Z")

</div>

> [@bernatgabor](#):
>
> A frontend can in this case fail with an error saying editable mode requires working symlinks, check out here on how to enable it for Windows. 🤔 This essentially makes the project require symlink for editable mode if it provide include/data files; which I think is fine, and many project developers would be fine with it.

Haven’t read all the rest of the replies since this one, because I’d like to ask you to stop expecting developers on Windows to be able to enable symlinks. That is simply not a viable assumption to make at the level we’re working at.

Happy to discuss another time all the reasons why, but I’m making this post a simple and direct request to drop this idea completely. Nothing we design can rely on symlinks being usable on Windows.

---

<div class="post-metadata">

**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\
**Post date:** [June 9, 2021, 2:37pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/98 "2021-06-09T14:37:02Z")

</div>

> [@steve.dower](#):
>
> Happy to discuss another time all the reasons why, but I’m making this post a simple and direct request to drop this idea completely. Nothing we design can rely on symlinks being usable on Windows.

If you’re going to make the statement I’d like you to also publish all those reasons when you have time for it. Thanks! PS. I wasn’t saying to make symlinks the only way, but a possible way with some benefits when available.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [June 9, 2021, 3:02pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/99 "2021-06-09T15:02:36Z")

</div>

Immediate answer - corporate PC with admin rights and developer mode locked down so the user can’t make the necessary change.

---

<div class="post-metadata">

**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\
**Post date:** [June 9, 2021, 3:17pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/100 "2021-06-09T15:17:37Z")

</div>

That sounds to me more like shouldn’t be the only way to do it, not that it shouldn’t be allowed at all. So I’m waiting for some other reasons here that I’m sure Steve has.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [June 9, 2021, 3:54pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/101 "2021-06-09T15:54:05Z")

</div>

> [@bernatgabor](#):
>
> That sounds to me more like shouldn’t be the only way to do it, not that it shouldn’t be allowed at all.

This is correct, but in all the other times you’ve referred to it, you’ve never included this nuance.

Feel free to take advantage of symlinks if they’re enabled. Do not require them to be enabled or assume that they will be.

(Another answer is that the vast majority of student laptops these days are as locked down as corporate PCs, so anything that doesn’t work without symlinks - even if it barely works and “we’ll display a message telling them to enable them” - is going to exclude the next generation of devs.)

---

<div class="post-metadata">

**Author:** ![layday](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/layday/32/2771_2.png) [@layday](https://discuss.python.org/u/layday)\
**Post date:** [June 9, 2021, 4:45pm UTC](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071/102 "2021-06-09T16:45:32Z")

</div>

I’ve spent a little time creating a library for the frontend as suggested above, which you can find [here](https://github.com/layday/frontend-editables). Hopefully, this will help inform our choice of editable installation and will form the basis of a more robust implementation should this PEP or an equivalent be adopted.

[Previous page](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071.md?page=4)

[Next page](https://discuss.python.org/t/discuss-pep-662-editable-installs-via-virtual-wheels/9071.md?page=6)