Ah, I see. Yeah, this isn’t really a concern worth worrying about - there are plenty of way to achieve this during package installation. It’s probably easier to detect uploaded packages that write outside of their own space.
The best option is to always run pip from a zipapp, and to clean up sys.path so that the stdlib can’t be shadowed.
On the main topic of should we have -X lazy_imports=none - I no longer think we should.
I think it’s better if - like the initial PR - we can move all imports out of functions and get away from some of the horrible patterns we have to use at the moment. Some of these are currently avoiding the circular imports that I brought up on that issue. This is only an issue if we have the none mode.
For tools that do have concerns like pip, moving away from hiding imports inside functions will actually make it easier to resolve imports before performing some action if necessary as the lazy_import objects will be discoverable at runtime. Tools could call an equivalent of resolve_all_lazy_imports() in order to then know that calling subsequent functions shouldn’t trigger any imports[1].
Edit: I’ll note that the original PR moving all imports to lazy does have some other issues, it moved some try/except or conditional imports to being lazy imports that would be unresolvable depending on platform.
Providing we have actually moved away from hiding imports in functions and other evil tricks. ↩︎
Unfortunately until PEP 794 the definition “own space” is completely undefined, and PEP only provides optional metadata.
It’s even standard practice amongst certain packages to overwrite other name spaces, uv has going been going through several rounds of just trying to warn users when packages do this and it’s been tricky for them to even turn that on.
This is fine, but the PEP’s security implications needs to be updated to reflect this and give advise to Python package installers that install wheels into the same environment they are executing in.
Currently the only solution provided in this thread that I think works in all situations is to ban importing right before or after the installation step. It could mean that pip will break with forced lazy imports, but that’s on users who use that option.
Not entirely - we can define it as “not pip unless you are pip”[1] and the only people who will complain are obviously malicious. It doesn’t require a PEP for us to say that a particular package is obviously trying to overwrite someone else’s files.
There are other, equally problematic ways to end up in the same situation, though. Or are you implying that all ways to disable lazy imports should be actively discouraged?[2]
“Horrible patterns” a.k.a. magic inside a library is much better than magic in user’s code. We the library/language developers can handle magic, and it can easily be commented when it’s in a single place like this. Scattering bits of magic throughout user’s code is worse.[3]
Those are literal "pip"s, not variables. ↩︎
I considered writing “removed” but felt that was putting words in your mouth that you didn’t say. Substitute “removed” if you like - my emphasis is on the “all ways” part. ↩︎
This is the primary reason I didn’t like this lazy imports idea in the first place, FTR. ↩︎
I think the behavior of -X lazy_imports=none can be modified such that everything (or at least a lot of stuff) just works. 
What breaks when lazy imports are disabled are circular from-imports - everything else stays functional, except with a potential performance penalty.
E.g.
# a.py
lazy from b import foo
def bar():
print("bar!")
foo()
# b.py
lazy from a import bar
def foo():
print("foo!")
bar()
This code breaks because the symbols foo and bar are looked up at the point where the input happens, at which point the other module is only partially initalized.
However, all we need to have happened at that point is that the import was started - we don’t yet need to resolve foo and bar.
So instead we again use proxies similar to what the proposal uses for normal lazy imports: These proxies just have a strong reference to the already imported module and just look up the attribute on demand. Effectively the code is turned into this:
# a.py
import b
def bar():
print("bar!")
b.foo()
# b.py
import a
def foo():
print("foo!")
a.bar()
Which is a perfectly valid way of avoiding circular import errors that is already used sometimes and has no significant security concerns.
This is a big assumption, I have worked with packages that intentionally overwrite pip to add functionality. They probably should have gone about it in a different way, but it’s not correct to say they were obviously malicious.
My point is that this problem can (and should) be flagged, detected, and resolved at a completely different level. We don’t need to pull it into deciding whether lazy imports are okay or not.
The problem is that the “horrible patterns” are unique, different modules use different patterns and these are generally not discoverable at runtime[1]. The lazy_import objects that would replace it are.
If a user cares about not triggering imports it’s easier to resolve them all if the lazy_import objects are all available at module level where they can be found and resolved.
I think this is fine, even with forced lazy imports. You use something like resolve_all_lazy_imports to resolve any existing lazy imports before blocking import entirely before the installation step.
You could scan the source code for them but then you need to individually handle each case. ↩︎
Yeah, this is my point: -X lazy_imports=none breaks the ability to use lazy import to avoid circular dependencies, which is a super common use of lazy imports.
Sure, but again, this is less than ideal. The whole point of PEP 810 is to avoid the need for these mechanical solutions.
Yes. I know, my point is that we can implement this so that the exact same code works.
We already have magic proxy objects. We can just make these magic proxy objects do something slightly different if lazy imports are disabled.
To be clear: I am proposing a modification of PEP 810 and the currently implemented runtime behavior.
Sorry, I misunderstood your initial post.
It sounds to me like you’re proposing that lazy from imports continue to be lazy under -X lazy_imports=none. Two questions/concerns with that approach:
lazy import statements? Won’t that still experience the circular dependency problem?-X lazy_imports=none? Based on the feedback from pip maintainers, we don’t need it at all.No. They eagerly import the other module. But this is not actually an issue as long as no symbols from that module are used.
Those also eagerly import the module. As long as no attribute is accessed, this is not an issue. And the only way to access an attribute is to use the module, at which point it would have been imported either way.
pip is not the only package that exists and might want to disable lazy imports. If we can provide perfectly predictable semantics while keeping the security benefits the option could provide I think that’s a net benefit.
This is important. Apart from pip (and other package managers, since they can use the same solution as pip), what projects want to disable lazy imports, and why?
IIRC there were discussions about disabling lazy imports for low-latency applications where a high startup cost is acceptable, but response times ones the service is running should be reliably low. pip wasn’t the only potential user mentioned in the original threads.
In general, disabling lazy imports (even with my suggestion) makes the behavior far more predictable, which would also help with debugging.
Do you have a link? The PEP 810 thread is pretty large, and the only user I saw mentioned was pip when I looked at it earlier.
Either way, those low-latency applications should simply not use lazy imports. If there’s somehow a bottleneck in their dependencies, they can use the resolve_all_lazy_imports function described earlier.
Sorry, I disagree. I think the behavior is unintuitive (because a partially lazy import isn’t a concept that currently exists) and will likely cause subtle issues that only occur when -X lazy_imports=none is enabled.
I agree with this. I think -X lazy_imports=none is a possible way for pip to deal with part of the issues around installing wheels that overwrite code that pip might later import, but it’s not the only way, and it may not even be the best way. I do think that lazy imports have the potential to make it harder for pip to address this issue, but that’s not really relevant to the discussion here.
I don’t have a view on whether -X lazy_imports=none is problematic or not, but I don’t think pip should be used as a reason to claim that it needs to stay.
My suggestion does not introduce any new concepts, it only uses “concepts” that work even before the lazy imports. It appears you don’t understand what I am suggesting. You might want to review what circular imports are and when and why they are a problem. (hint: Not all import circles are problematic circular imports, even without lazy imports)
Here one of the authors explicitly mentions that both variants (fully enable, fully disable) are needed for some usescase, although he doesn’t elaborate: PEP 810: Explicit lazy imports - #92 by pablogsal
My understanding is that you’re proposing the following:
# Under -X lazy_imports=none, foo is imported, but bar is not accessed
lazy from foo import bar
# bar is reified and accessed from foo
xyz = bar
Is that wrong?
Please try to be civil. I’m not sure where I implied that all “import circles” are problematic, or what point you’re trying to make here.
Thanks, this is helpful. From what I see in that post, Pablo is basically saying that libraries are free to break -X lazy_imports=none to avoid circular import issues and will, in turn, annoy people who try to use it.
Given that pip has alternative solutions from this thread, this is exactly what I expect from the ecosystem: -X lazy_imports=none will essentially be a broken option that nobody uses, because everyone wants to use PEP 810 to resolve circular import problems. Unfortunately, CPython will be the only one that can’t do that, and speaking with my CPython maintainer hat on, that’s quite frustrating.
Obligatory ping @pablogsal – would you mind expanding on the uses here?