Standardising editable mode installs (runtime layout - not hooks!)

Packaging
#99

I’ve been looking some more at namespace packages in the context of editables. The big issue is that PEP 420 explicitly requires that namespace packages recalculate their path based on sys.path every time they are accessed. This means that you cannot just “add” a development directory ~/work/project/foo as a namespace package - sure, the sys.modules entry is present, but unless the project directory ~/work/project is on sys.path at the time you import from the namespace, it will not be searched.

So the practical implication of this is that namespace packages cannot be supported properly. Note that this is a property of the import machinery, not specific to editable mode (via wheels or otherwise).

Possible workaround include:

  1. Just add ~/work/project via a .pth file. That’s what setuptools does now, and it seems to be acceptable enough. The problem is, it exposes the whole directory, which means it’s not a precise equivalent of a normal install.
  2. Write your own implementation of namespace packages, or steal/monkeypatch the importlib implementation. This looks like it would be tricky to get right, and it would then deliberately not be following PEP 420. I’ve no idea if this is sufficient, as I don’t use implicit namespace packages myself.

The current implementation of the editables package doesn’t support namespace packages at all, and I intend, at least for now, to keep it that way and make it an explicit limitation.

My question for the PEP is, what do we want to say build_wheel_for_editable must do? Is it OK for it to just omit namespace packages when building the wheel? Should it be required to fail, and (in effect) say “project cannot be installed as editable”? Is it allowable to implement some sort of limited-functionality equivalent of a namespace package? I’m reluctant to choose that final option, because we’ll then be sucked into writing our own equivalent of PEP 420 without the “dynamic path” feature.

#100

I don’t see why we wouldn’t allow that, and thereby force the perpetual use of setuptools / setup.py.

I plan to revert to the pth way for hatch (based on an option in pyproject.toml).

#101

OK, so can you define what behaviour you would consider essential in a “limited-functionality equivalent of a namespace package” in sufficient detail that I can write some tests to check that an implementation delivers that behaviour? (Because I’ve tried, and I can’t :slightly_frowning_face:)

The good news is that an editable wheel can, I believe, contain a .pth file, so this is still 100% compatible with the proposed new hook.

#102

A strong motivator for the src layout, it seems to me. An all-around win.

-Fred

3 Likes
#103

Given package a and b under namespace foo, I’d expect foo.a and foo.b to be importable as long as all packages are installed as editable.

#104

The pre-draft PEP now lives at build_wheel_for_editable by sbidoul · Pull Request #1 · sbidoul/peps · GitHub. The phrasing defines the broad expectations of a user for an editable install but otherwise leaves a lot of freedom to backends. .pth and editables are mentioned as possible implementation approaches.

#105

OK. So basically it’s up to the backend, that seems fair. I’m sensing a general reluctance to be too prescriptive here, which is fine.

I’ll probably omit namespace package support from editables in that case, and let backends implement those however they choose. They can still use editables for non-namespace packages. That saves me having to make policy choices on behalf of backends.

PEP 660: Editable installs for PEP-517 style build backends
#106

This makes me feel the root problem is in Python’s import system, and editable namespace package support should and can only be implemented by introducing new mechanisms to the system. A “proper” way to “link” a package into sys.path, instead of adding an entire directory into sys.path, for example. So I’d feel it’s good for the editable PEP to explicitly rule out namespace packages, until a PEP adding that feature is proposed (by someone) and accepted in CPython.

1 Like
#107

Why rule it out explicitly, since the .pth approach works perfectly well with namespace packages? So a solution exist that is on par with what setuptools does, and ruling this out explicitly would create another roadblock for the adoption of PEP 517, IMO.

#108

The pth approach works under certain conditions, but far from perfectly (even in terms of editables where technical perfectness is impossible).

#109

Do we say “it’s an especially good idea to use src/ if you are doing a namespace package”? Does a namespace package or the .pth method fail then? (If you don’t need to exclude anything under src/?)

I’ve never been thrilled with namespace packages. They are fragile because someone can come along and put an __init__.py in their version of the namespace package, and it breaks you. The installer could check that all installed packages in a particular namespace are using the same strategy?

2 Likes
#110

The Python import system is mostly designed statically - by which I mean the design doesn’t really take much account of things like sys.path or the contents of the filesystem changing “on the fly” while programs are running. This is the reason we typically don’t recommend executing pip from a running process, for example. It’s not exactly that the results are undefined, but they are pretty hard to reason about, and documented mostly by implication rather than explicitly.

Namespace packages are an unusual exception here, in that they explicitly do define how they behave if the filesystem content changes at runtime. And the implementation handles this by making the filesystem rescans part of the implementation of the namespace package, rather than something “higher level” that 3rd party code can hook into.

It is possible to implement them, just significantly harder. After all, we could simply take the existing implementation from importlib, modify it to handle the needs of editable installs, and ship that as support code with an editable wheel.

The question isn’t what can be implemented, it’s what trade-offs backends want to make. And more importantly, what trade-offs will be acceptable to end users. To that end, the setuptools .pth solution is simple and proven to be “mostly sufficient”. And it’s easy to implement in a wheel - it’s just a single file to install.

But from a specification point of view, the question is how willing we are to leave the behaviour of an editable wheel unspecified. For example, if a backend returned the exact same wheel from build_wheel_for_editable as it does for build_wheel, is that compliant? The backend could claim, “well, you can edit the files, they are all in site-packages”.

I don’t honestly think we need to protect ourselves against deliberate attempts to subvert the spec. But conversely, if the spec is too loose, we will get people interpreting it differently - we’ve had recent examples with other specs, and it’s a pain.

I think the spec should steer well clear of providing any sort of “advice” to backends, it’s too easy for people to end up claiming that it’s a requirement, not a suggestion.

I don’t have a good answer here, but I feel like it’s important to restrict the PEP to things we can be precise about.

#111

@pf_moore The current release of editables is working well for me! However, I just realized that PyCharm & VSCode are unable to provide auto-completion. I think that may be worth mentioning somewhere.

2 Likes
#112

I have no idea what they might need that isn’t already present to allow them to do so. If you can identify what that is, I can look at adding it.

#113

Sometimes there is an “execute Python code to look at these trickier modules” list that imports the module instead of using static analysis

#114

I was trying to politely say “if you explain what code I have to write, I’ll write it, otherwise feel free to offer a PR” :slightly_smiling_face: I don’t intend to bother researching how to make editables work nicely with IDEs myself right now (I mostly consider it the IDE’s problem to work with properly-written Python code, not the other way around…)

1 Like
#115

I know from experience that PyCharm doesn’t resolve dynamically-loaded modules to get the attributes of the loaded module. There doesn’t seem to be any issues posted to the PyCharm bug-tracker about this.

This seems to me the kind of thing that will likely need a specific solution, like Jinja template file resolving. I imagine they wouldn’t want to release that solution until editable has been standardised.

#116

Ah, if you mean that modules loaded via import hooks aren’t introspected correctly by IDEs, then yes, that’s not a problem in the editables code directly. I’m pretty sure the modules have a source file loader, so the source code is accessible - if that’s not the case then that would be something that can be improved, but I’d need some means to demonstrate the issue (ideally in a way that can be added as a test) before that would be possible.

By the way, these sorts of questions would be better raised on the editables issue tracker…

#117

Correct, and none of the auto-completion providers support this sort of thing, so no one is going to be able to make it work without some configuration of your paths in your preferred editor for it to know where to look. This is the one thing that .pth files have going for them since they edit sys.path and that can be programmatically picked up with python -c "import sys; print(sys.path)".

1 Like
#118

Cool. So simply having a loader that gives access to the source code isn’t enough? One thought I’d had was to check that I was returning a SourceFileLoader where possible, in case that helped…