Okay, but someone will still have to decide whether that mapping can be satisfied and if it’s not the frontend, it’ll be the backend - you’d just be moving that logic from your editable installation library to setuptools, for something that the former might be better equipped to handle, in a manner which is common between all backends. Anecdotally, it was easy to identify when the package names disagreed and error when using a .pth in my own implementation.
What’s the current status on a decision between the 2 PEPs? Are we waiting on wording updates, etc.?
1 Like
Neither of the PEP authors has yet said that their PEP is ready for a decision. I won’t do anything until I hear from the PEP authors saying that their PEP is ready for pronouncement. That should happen via a formal posting to the relevant thread here, mentioning me, with an explicit request for review and pronouncement.
At that stage, I’ll ask the authors of the “other” PEP if they can have it ready within a short period (no more than a week or two) so that I can make a joint decision. If they can’t then I’ll just decide on the one that I do have independently.
2 Likes
This PEP is not ready for pronouncement. The current state is trying to create a POC with pip and setuptools for it. This is expected to raise a few concerns that I’ll address with a subsequent PEP edit. Post that PEP edit I expect a short amount of feedback time before we can finalize it and ask for a decision. Realistically we’re talking here for a month or so timeframe until that happens.
Understood, and I appreciate the indication of the timeframe here. As long as PEP 660 is also not ready there’s no rush.
If the PEP 660 authors (@sbidoul and @dholth) could indicate the sort of timeframe they have in mind for submitting their PEP, that would be useful to me.
1 Like
On a completely different direction compared to the discussion here, can we rephrase this PEP’s abstract to use the phrasing “pyproject.toml based builds”?
This document describes extensions to the pyproject.toml based builds (as introduced by PEP 517) to allow projects to be installed in editable mode by introducing virtual wheels.
This would help establish an nicer name for the interface defined in PEP 517, especially since there’d now be multiple PEPs defining it. This was previously discussed at:
3 Likes
@bernatgabor, can you provide some indication on what you plan to change in the PEP and if the changes address some of my concerns above? I’m broadly supportive of PEP 662 and I’d like to help with steering it in a direction where it is able to rival PEP 660. If @pf_moore could also give a quick rundown on what he thinks about the PEP so far, I think that would be helpful. I don’t want to see the PEP rejected for reasons that could have been addressed but weren’t simply because they were not stated.
My main concerns at the moment with PEP 662 are:
- There seems to be fragmentation between the various people supportive of a “virtual wheel” approach - @pganssle expressed concerns that he didn’t agree with the approach the PEP was taking, and you also have concerns. If the PEP doesn’t end up being a proposal that at least the supporters of the “virtual wheel” concept are comfortable with, then IMO it’s pretty much dead.
- The PEP doesn’t make clear how the interface between the frontend and the backend works and who is responsible for what. There’s currently too much vagueness, meaning that the frontend has to be prepared to accept a wide range of possibilities, and the backend can’t assume that the frontend can handle what it sends.
Both of these are things that have been raised already here, so I hope they are in the process of being addressed.
My other big concern is that PEP 662 feels “rushed”, in the sense that work was only started on it after PEP 660 was published, whereas there’s been a lot of preparatory work done on getting PEP 660 to the state of being ready for publication. I feel that the lack of interest in working on the virtual wheel idea is a reality that needs to be taken into account, and it’s something I will be considering. Unfortunately, that makes the bar for PEP 662 a little higher, but again I’m hoping that @bernatgabor can address this and still come up with a good proposal.
I should also point out that this isn’t just a matter of ticking the right boxes and getting approved. Both PEPs need to make their case that they are the best way forward for the packaging ecosystem - and that means delivering functionality that users need, without placing undue strain on the volunteer support for packaging tools, and setting us up for a sustainable future where people can use the standards to write their own tools without being locked into monolithic “legacy” projects like pip or setuptools. That’s a lot to expect and I’m not looking for perfection here, just the right viewpoint and focus. (But consequently, one thing I’m not interested in, is arguments about why the “other” proposal is flawed - if a proposal is to succeed it should do so on its own merits, not merely because it’s “better than the alternative”).
I won’t say what I think about the two PEPs in relation to each other right now, as it wouldn’t be fair - PEP 660 has been confirmed ready for pronouncement whereas PEP 662 is still a work in progress, so it’s not comparing like with like. All I will say is that I’m perfectly open to being persuaded in favour of either proposal at this point.
1 Like
I’ve updated the PEP with a better defined virtual wheel concept here:
Also finished my rough POC for pip + setuptools, which should give a better idea of where/what happens:
- setuptools - Comparing pypa:main...gaborbernat:main · pypa/setuptools · GitHub
- wheel - Comparing pypa:master...gaborbernat:master · pypa/wheel · GitHub
- pip - Comparing pypa:main...gaborbernat:main · pypa/pip · GitHub
Post addressing feedback I’ll receive on these, going ahead I consider the PEP ready-ish.
PS. I’m not sure what we can do about vagueness on the editable mechanism used by the frontend, it’s purposefully there to allow the frontend to be as flexible or strict as it decides so. And I consider it one of the strengths of the PEP: that’s a bit more flexible than PEP-660 as far as editable installation manifestation goes.
Thanks for the update, and good to see the PoCs are complete. From a quick look:
- Can you explain why you have involved
prepare_metadata_for_build_wheelin this new version? - If the installation will be based around the wheel standard, I feel that the first level of the mapping is redundant; you can simply map in-wheel paths relative to the root of the wheel to paths on disk.
platliborpurelibcan be chosen based on the value ofRoot-Is-Purelib- although, my preference, as stated, is to only mappurelibandplatlib. This should also more closely align the editable installation to that of the wheel. - I’m very happy to see that the paths must now point to files.
- Please clarify whether the target paths should be platform-native or POSIX (i.e. as the path in the wheel would be).
- I suggest to remove the provision that the “backend can take control if it wants to do so”. This is untenable. Consequently the
get_requires_for_build_editablehook would also not be required. This will bring the complexity of the PEP all the way down.
It’s building a wheel so could not come up with a good reason why it should not respect this behavior the same way build_wheel does.
Can you provide a concrete example to visualize why you think so?
I assumed platform-native 
but guess I should state it explicitly 
This is just acknowledging reality, the backend can always just use editables project to inject customer behavior, not?
I can see a case where the backend wants additional dependencies to achieve the editable effect. For example, in files to be compiled (c/c++/rust), it might want to load a tool that knows how to generate a python file that self-replaces and compiles on import. I consider the presence of this a low overhead, high benefit. Otherwise, I’d need to mandate that the frontend needs to reuse get_requires_for_build_wheel, which would break the symmetry and actually increase complexity.
Depending on how a frontend implements the editable mode, some differences may be visible, such as the presence of additional files (compared to a typical installation), either in the source tree or the interpreter’s installation path.
IMO I think both PEPs should outright require that, within packages, new Python files and sub-packages will be detected. Otherwise, I truly think editable mode is mostly useless, especially for newcomers.
I’d go with should rather than must. I think that would be a good reason why popular and general use frontends would not implement certain editable modes . However, some systems might be ok with a less strict approach, not? E.g. in corporate environments you can mandate/educate people that if they add new files they must reinstall making it must feels a bit excesive.
Ultimately it’s just one argument for backends to ignore - no backend will want to bother with ensuring metadata consistency for editable wheels.
If we assume that the paths are relative to the root of the wheel,
as the “virtual wheel” monicker would suggest,
we can model the installation on the wheel format, so that:
foo.py -> {purelib}/foo.py
and
foo-0.0.0.data/data/bar.txt -> {data}/bar.txt
The path mapping can then be expressed as:
{
"paths": {
"foo.py": "/home/layday/foo/foo.py",
"foo-0.0.0.data/data/bar.txt": "/home/layday/foo/baz/bar.txt"
}
}
… and the paths can be expanded as they would have been with a regular wheel.
The benefit of this approach is that (a) the scheme will not have to be encoded
at the level of this PEP and the PEP will not have to be amended to accommodate
changes made to sysconfig, and (b) a “sans I/O” library can perform the path expansion
both for regular wheels and for virtual wheels.
We may then also choose to say that .data paths are unsupported at this time
because they cannot be satisfied in a platform-agnostic manner - the door’ll be left open
for them without actually requiring any changes to be made to the PEP itself
or the editable.json schema.
Well, if it were simply acknowledging reality, the PEP wouldn’t then go on to provide an example on how to accomplish just that 
I think we’ll just have to concede that this isn’t something we can fulfil while keeping a clear delineation between the frontend and the backend.
The virtual wheel name and data structure is fine. The structure can clearly be used to make a normal wheel. It would also of course be fine for the data structure to not say "foo-0.0.0.data/" and only include { category: path } mappings from which a wheel archiving library could construct a concrete rather than an abstract wheel. Would not be very practical for the backend to give the expanded installed paths of each file.
The question for this PEP is to specify exactly how to use the structure to implement pip install -e.
Whether the build must be “in place” in the same shape as an install. How to have predictable editable installs for unpredictable inputs, so that every installer would behave in the same way. …
There was also the idea that “editable” should mean something strict that existing users do not want, that accidentally including setup.py might be a catastrophe rather than a minor inconvenience, and that an editable install might so closely simulate a normal install that you don’t have to test the normal install.
And there’s the idea presumably rejected by PEP 662 that an alternative ‘strict/preflight’ mode possibly used to speed tox testing could be a distinct feature from pip install -e mode used to develop.
If you’re going to make such a claim please specify the reasons for it. Otherwise, I must ask why do you think so?
A backend that would bother ensuring metadata consistency could have faster installs. E.g. pip install -e . does not actually build a wheel and then install it. Because it needs to first use the resolver to determine if the requirement set can be satisfied. So it will always first invoke prepare_metadata_for_build_wheel to get the project metadata to answer whether these project dependencies are satisfiable (as defined in PEP-517).
So we have three options the way I see:
- force build backends to ensure metadata consistency between
prepare_metadata_for_build_wheelandbuild_editable, - define a
prepare_metadata_for_build_editableto be used by the frontend (e.g. pip resolver) - do not provide a fast path, and force build frontends to always fully build virtual wheels to get their metadata.
I went for option 1 because that requires no extra handling on the frontends side. We could also force option 3, but considering normal wheels have a fast path, could not justify why virtual wheels should not. My main driving goal here is to make the difference between a virtual wheel and a normal wheel as little as possible because that’s going to be the simplest to implement and maintain.
Ah, you’re talking about removing the scheme abstraction from within the editable.json. I don’t think that’s a good idea because some categories can be made editable in different ways than others. For example, scripts/data handles differently than purelib/platlib. So I like to keep that abstraction, to give more information for the frontend on what techniques he can use or not. I know this would not be the case if you’re targeting to only support purelib/platlib but this PEP aims to allow more than that.
Changes to the sysconfig paths are very rare, so I don’t see that much benefit. That being said even when that does happen I don’t think we’ll need to amend the PEP significantly because we’re just talking about adding another key into the JSON config, that’s backwards compatible because it does accept additional properties.
I don’t think this is a big benefit and was my experience when implementing this for pip, because making content editable during installation already is custom enough that the expansion is trivial compared to the rest.
Wait, the example just showcases a reality case. So I don’t follow what’s the problem here.
I don’t see how this statement is related to providing a get_requires_for_build_editable or not. From an implementation POV providing this hook is actually helpful, because all existing infrastructure calls some get_requires_for_build_ hook before starting to communicate with the backend. And defining an optional get_requires_for_build_editable is cheap. Again the backend doesn’t have to implement it, but e.g. setuptools must because the wheel building dependencies need to be specified somewhere.
I think this is where Paul and I, and you folks differ in opinion. We do not want to enshrine in PEP the how, we leave that up to the frontend to determine/innovate. We do provide a few non-normative examples how the frontend might achieve this, however, ultimately we allow the frontend to implement multiple ways and then communicate with the user on the benefits/drawbacks of each; and allow the frontend together with the user to select the one best for them. I think this is in line with PEP-660 mostly, the main difference is that PEP-660 puts the unknown how to achieve editable effect into the backends court, while this makes the backends job easy by putting the responsibility into the frontends court.
I think we do say this is not the case because the backend can remap the source tree. We even provide an example, against this.
This would be an ideal place to get to, but we acknowledged that there are cases when this is not possible. So while we strive for this (only during development) I don’t think we can always get there. You’re only allowed to do editable install during development so that’s the place where ideally editable wheels would replace normal wheels. When installing/building from PyPI you must use normal wheels. Also when your use case cannot be mimicked exactly by editable install, you’ll want to do normal installs too. I have seen a lot of tox configs that always enable editable installs, so people only using editable installs to test their projects is much more frequent than you’d think.
I don’t know what preflight mode you’re referring here to so can’t address this point.
Thanks for the update. The two proposals are converging in an interesting way here - if we added a mechanism for having a post-install hook in the wheel, the backend could provide the implementation approach PEP 660 style, by providing both the mapping and the hook. (Post-install hooks in wheels as a general concept have been discussed, but I think the consensus is that we don’t want them in general - for this specific case it’s an interesting possibility though).
I’m inclined to think that the new mechanism shouldn’t even be called a “virtual wheel” as it’s not, it’s just a normal wheel with some extra metadata. If the wheel spec were updated to define that metadata and describe how it should be used on an install, we’d have PEP 660. So this new version of PEP 662 is arguably just PEP 660 where the backend uses a wheel feature that doesn’t yet exist (much like symlink support in PEP 660 would rely on getting symlink support added to the wheel spec).
My concern is support. If a user reports that “I tried to do an editable install of my project and it failed” who should the user report the problem to? The frontend (installer)? The backend? As the PEP stands, we’d be telling the user that the backend is fine because it provided a valid mapping, and the frontend is fine because it chose not to support that mapping. So who breaks the bad news to the user? And if the user feels the failure is unreasonable and makes an issue of it, what happens then? I feel that we’d end up with it looking like the frontend was blaming the backend, and vice versa - which is bad for the reputation of the Python packaging ecosystem, and would ultimately contribute to maintainer burnout and harm the sustainability of the ecosystem. On a personal note, I’m already tensing up at the thought of such issues arising 
Maybe we can’t avoid such tensions, but I think the PEP has to face the fact that it’s a real issue and address it. Because it’s an important distinguishing feature of PEP 662 compared to PEP 660. In PEP 660, the responsibility is solely on the backend, for better or worse. In PEP 662, the responsibility is shared, by design, so this is a cost of that design, which needs to be weighed against the benefits.
This misses one important aspect though of PEP-662. While true implementation might allow PEP-660 to do what PEP-662 wants to provide, the fundamental difference is that PEP-662 thinks it’s the frontend’s responsibility to achieve the editable effect and not the backends. So PEP-662 thinks that the backend is only responsible for communicating what it wants to expose to the frontend. It’s up to the frontend to implement that, and communicate with the user what it can do and what it cannot.
Well, I don’t think there’s a complaining to be made here. The culprit ultimately is the frontend, however, when the frontend detects that it cannot support something it can tell right there that this is unsupported.
The user might open an issue to the frontend, nevertheless. The frontend can say look we don’t support that use case, and you should look at frontend y instead to achieve that or suggest alternatives.
Ultimately I feel this will be less of an issue in practice if frontend developers circle around a frotend_editable library they all just delegate the job.
This is not true in my mind. PEP-662 puts the responsibility solely on the frontend. The backend has a very clear set of information it can provide, and then validating/performing that is up to the frontend. Frontends might center and reuse a frontend_editable library to be as widely compatible as possible.
“preflight” would be an appropriate name for a feature where extra .py files, not included in the wheel/sdist, are hidden from the interpreter. If you are using editable mode to try to determine if your application will work when distributed.
setup.py develop is good enough most of the time. For projects that are never distributed and might always run from their source checkout a “preflight” mode would not be needed.
In PEP 662 my extremely flexible backend would provide imperfect data to the frontend. It would matter how the installer dealt with that imperfect data.
It would be better if PEP 662 says how it works instead of saying “installer must use library X to meet expectations”.
What imperfect data are we talking about? (specific example/use-case that is)