Third try on editable installs

Packaging
41

Thanks, I now understand the argument for listing every single file so that frontends can choose how to implement editable installs. However, I still have a couple of concerns about implementing my preferred way of doing this (a symlink in site-packages pointing to a package directory) on top of this information.

  1. As @dholth asks, how much can the mapping rearrange files? I’d need to find not just a common prefix of a list of paths, but a common prefix from the source files which maps to a common prefix in the ‘virtual wheel’. And check that no other source files are being mapped into my destination prefix. This is already starting to sound messy.
  2. If a backend doesn’t list some files under a package directory, and the frontend does the editable install in a way that exposes the entire directory, including those unlisted files, is that valid? I can imagine backend authors saying that’s a bug in the frontend.

More detail from the backend ostensibly lets frontends choose how to do an editable install. But I’m concerned that it could prove impractical to do anything except what the specification is designed for - symlinking each individual file in newly created directories.

42

Summary: There are a number of things that really need to be clarified, IMO, before we have a workable spec. I’ve noted the key things for me below. Hopefully thrashing those out is the point of doing the POC, and we’ll work on getting answers.

(Edit: Accidentally hit “send” too early. But I was only proof-reading, and I can’t face more editing, so please accept my apologies for any typos or weird formatting).

OK, I’ve re-read it. As you say it’s not that long, but I still find the discussion hard to follow, so my apologies if I’ve misunderstood something. It’s also possible some or all of my issues were discussed at the packaging summit, but in the absence of any details I’ll assume not (and I don’t expect it matters much now anyway, as people will quite probably not remember all of any discussions that were had).

Things that bother me are:

  1. We’re discussing “editable installs” without a proper definition of what an editable install actually is. The obvious definition is “what setup.py develop does”, but we’ve explicitly decided not to copy that behaviour in a number of ways:
    • Adding a new (Python) file needs a reinstall in the proposed approach.
    • The behaviour around excluded directories becomes a front-end problem.
      I don’t believe anyone has actually discussed whether or not these changes are acceptable to people who use editable installs (if that was discussed, I missed it). I’m not going to bring up my own view again, the point here is that we shouldn’t be relying just on our own views if we’re going to change behaviour.
  2. The front end behaviour is badly under-specified at the moment. The proposal suggests that the backend provide a list of files and the front-end “make the files available”. That’s not an implementable spec for front ends. There’s discussions about symlinks or .pth files, but no real meat. That becomes very obvious when talking about the excluded files issue. The proposal doesn’t actually solve that problem, it just makes it the front-end’s issue, and gives front ends no support in how they handle it.

At this point I’m going to specifically put my “pip maintainer” hat on.

I’m not at all happy with how this proposal leaves all of the technical questions on how to implement editables up to the front end1, while passing responsibility for what you yourself describe as a “huge bug” to the front end and yet claiming that “front ends don’t need to fix this bug”. If it’s such a huge bug, then yes, they do, because users will demand it, and appealing to the fact that the PEP says we can ignore it won’t make any difference. And the point of providing full lists of files is to allow front ends to address this issue - so if I advocate for pip to ignore this problem because the PEP lets us, and I don’t view it as particularly critical, then what’s the point in having all that extra mechanism in the proposal?

Similarly, there’s been talk about using symlinks, but not all platforms have reliable symlinks. So any solution either needs to be implementable without symlinks, or front ends need to be willing to offer degraded support on platforms without symlinks. I’m OK with this being a front end choice, but personally, I’d advocate for pip using .pth files. So again, anything that’s in support of front ends using symlinks, who’s it going to be used by?

So what I’d like to see is:

  1. A background section that details what editable mode is, and explains how the standard version differs from the version as implemented currently, so that we’re open about the changes.
  2. A section in the proposal that describes the standard runtime layout for editable installs2, so that front ends have something to work with, and potential consumers don’t have to rely on implementation-defined layouts.
  3. Better clarity over what front ends MUST do, what they SHOULD do, and what they are allowed to ignore3.

Note: I get that we’re only in “write a POC” stage at the moment, and so the standard isn’t written yet. But the POC is triggering these questions, and if we don’t come up with proper answers as part of doing the POC, there’s not much point. And let’s be realistic here - if the POC works, it’s very unlikely we’ll just throw it away and write something different. So the code needs to implement the behaviour we want to have as the reference implementation.

Also, we do have UX specialists currently working on pip’s user interface. We could, I suspect, ask them to collect data on how people use editable mode and what they want from it (@sumanah please correct me if I’m speaking out of turn here and that would be out of scope for the work they are funded to do). But I’d be cautious about doing that unless we have some good, focused questions we want answering. Just asking for ideas will probably get either nothing useful, or too much for us to deal with (we’re already stretched just covering the questions we know about!)

1 We’re supposed to be avoiding implementation-defined behaviour…
2 I’m OK with labelling such a description as “provisional”, or “version 1”, but there will be consumers of this layout, as well as front ends producing it, and we should acknowledge taht and document the format.
3 Given that pip is, in practical terms, the only front end at the moment, I’d argue that any feature in the spec that frontends can ignore and that pip is expected to ignore, should be an obvious candidate for omission on the grounds of YAGNI.

3 Likes
43

I have my own opinions on how an editable install layout should work, but they aren’t really that important here - I structure my code in a way that makes it easy for me to do manually.

One thing we as a Python/packaging community are often accused of is not being opinionated enough. We try very hard to accommodate all scenarios and all preferences, often leading to extended discussions like this and eventual confusion for users.

Perhaps this is a case where we could try being highly opinionated? Simply set down “if you want editable installs, your sources must/must not include/exclude/whatever” and build a tool that works with that.

My general impression of packaging right now is that more than having their cases supported, users just want to know what they should do. Why don’t we just tell them?

5 Likes
44

We could look at getting some of the user experience folks’ time in the second half of the year, in the third phase of the current funded project. Especially if someone can plausibly explain that the intersection of the new dependency resolver with editable installs causes, or will cause, extra confusion.

If there’s a specific deadline someone is trying to meet to ship better editable installs, then it would be good to know that. Otherwise I think it’s ok to start developing some questions for user experience researchers to ask, and pause other discussion and development till we have some answers.

1 Like
45

Thanks @sumanah. My feeling is that we shouldn’t rush into trying to do UX work, we need to be very sure we have specific questions and goals before going down that route. And at the moment the discussion feels too broad to make for a good set of questions. But it’s useful to know the option is there if we do get to the point where we could get value from it.

1 Like
46 
python -c 'import this' | python -c 'import sys; print(sys.stdin.read().splitlines()[-7])'
1 Like
47

in gumby_elf i experimented with a entirely different approach for editable installs

instead of letting the installer handle it, i generated a wheel, which had __init__.py in the packages that would set up __path__ and then execute the real code

that way namespace packages and corret metadata was trivially installable as the installer wuld get a wheel with a +editable local tag and a +editable in the version metadata

1 Like
48

I think this is where we are at the moment under Next steps for editable / develop proof of concept