This is correct. But if you understand this, then I am confused what you are referring to here:
What is a “partially lazy import”? The import is eager. The exact same as if you had written import foo. What changes is the behavior of the proxies, which is fully local to the module.
What kind of “subtle issues” are you imagining? The semantic meaning of every piece of code stays unchanged: With this proposal all currently eager imports in all files could be replaced with lazy import statements and not changes in behavior would be observed when -X lazy_imports=none. That’s far more predictable than lazy imports could ever hope to be. We are already accepting that -X lazy_imports=all is an option that exists.
I’m concerned about the magic in lazy from. I say “partially lazy” because the module is imported eagerly, but the attribute is looked up lazily. That kind of separation leads to edge cases.
For example, a module might set a global variable depending on the value of an environment variable. Let’s pretend we’re designing something similar to _colorize.
# my_colors.py
import os
COLORS_ENABLED = 'NO_COLOR' not in os.environ
If we were to lazily import this and then set the environment variable, we’ll see a difference depending on whether -X lazy_imports=none is set:
lazy from my_colors import COLORS_ENABLED
import os
os.environ["NO_COLOR"] = "1"
print(COLORS_ENABLED) # False with lazy imports on, True with lazy imports off
Easily fixable? Yes! Just move the os.environ modification above the import. But, that’s beside the point – there are always going to be subtle differences if lazy imports can be modified by a flag, and I’m arguing that there aren’t any uses for -X lazy_imports=none now that we have package managers out of the way, so why support the flag at all, especially if it makes CPython maintenance more difficult?
You are describing a weirdness of lazy_imports=normal and lazy_imports=all, not of lazy_imports=none.
There are a billion ways to make code like this give weird behavior if lazy imports are on. The behavior of this code if lazy imports are off is perfectly predicablty.
But also, we are going to need to support something equivalent to lazy_imports=none anyway:
sys.set_lazy_imports_filter(lambda *args: False)
Unless you are also of the opinion that the above usage is not supported by the stdlib. In which case we should provide a well defined set of imports that need to be allowed to be lazy for the stdlib to function correctly.
Hm? Are you referring to the current behavior or the behavior in your proposal?
This seems subjective. If lazy imports are off, I would expect it to behave like an eager import rather than mixing the two behaviors. The behavior is only predictable if you know the (subjectively, unintuitive) semantics. Are we still talking about your proposal?
I don’t think the stdlib needs to support this because, at the moment, it seems like nobody will do it. I don’t see much point in discussing this until we’ve clearly determined whether there’s a legitimate reason to arbitrarily disallow all lazy imports.
To be clear, my view is that lazy imports will, even with -X lazy_imports=all, make this easier for something like pip. Keeping -X lazy_imports=noneas it is currently implemented hampers this effort, by forcing lazy imports which involve import cycles to be kept hidden inside other objects.
With current Python, any function anywhere can have an inline import, or you can do some of the evil things typing.py tries to do for performance with its lazy imports. The important thing is that without special casing, it’s not obvious what attribute access or function call might trigger a lazy import.
With PEP-810, the easiest way to do lazy imports is now with the lazy import statement. We benefit more if we can do this without consideration for every lazy import, including circular imports. PEP-810 lazy imports can only be used at module level which means that if you look at sys.modules and look at every module’s globals, you will find all of these lazy imports.
This means you can trigger every ‘pending’ lazy import. This is what the resolve_all_lazy_imports function sketch I posted earlier is designed to demonstrate[1]. The idea is that if it’s more important that no further imports occur, either for latency or for security, you call this and then block off imports. Providing everyone has switched to PEP-810 imports and you yourself don’t import anything, no further imports should occur.
There is one issue I know of in that hastily sketched function, you actually need to make a copy of the module dict instead of using it directly. Resolving one lazy import may trigger a __getattr__ in the original module which can mutate the module dict. ↩︎
You have not shown an example where the behavior differs from an eager import. You are correct that those examples do exists, but I strongly doubt that anything close to that pattern exists anywhere in the stdlib.
So you are saying that the official policy should be that lazy imports can always assumed to be active. Since the stdlib is doing this, many third party libraries will also start doing this with zero concern and reject bug reports in this regard.
This is a significant change from the conditions under which the PEP was accepted and needs a formal approval by the Steering Council. Multiple people in the original thread have expressed concerns about wide spread usage of lazy imports - making it a fundamentally required part of the stdlib is a non-trivial change.
I think lazy imports should be more widespread, but then I also probably look at the output of -X importtime more than is healthy. I also think it will need SC approval, but I also think that was the point of the discussion in the first place.
I’ve essentially come to the conclusion that it’s more useful to be able to force all imports to resolve at any point in an application than it is to force every import to do so from the start.
-X lazy_imports=noneas it is currently implemented discourages some use cases for lazy imports that would otherwise be beneficial, both in maintenance of the code and in making the lazy imports discoverable for tools that care about disabling lazy imports.
I think libraries will ignore it regardless of what the standard library does. -X lazy_imports=none or arbitrarily disabling all lazy imports via a filter doesn’t seem useful, so there’s no use in supporting it.
The standard library already has tons of lazy imports, and all we want to do is make the syntax cleaner. But we can’t, because PEP 810 comes with a setting that breaks it. I’m arguing that we don’t need that setting and can therefore widely use PEP 810 in the standard library. That’s what this thread is about.
Or we can make the setting more useful and not have to worry about it in the stdlib.
PEP 810 also comes with a setting to in general break most code (lazy_imports=all) and very easy ways to disable all lazy imports without lazy_imports=none. I don’t see how removing the setting has any benefit over saying “stdlib might not work with this setting” in the docs.
There are tons of ways to break imports, some of which were even suggested in this thread as solutions to the security problem. Edge cases having weird behavior clearly can’t be a major concern if PEP 810 got accepted in the first place.
Ultimately, I think that any change that majorly affects lazy_imports=none requires SC approval. If no Core Dev supports my idea, then so be it, but I think just removing the setting completely is going to be worse long term (e.g. the PEP explicitly spells out using lazy_imports=none for debugging).
I think removing lazy_imports=none makes sense here.
I was one of the people with a concern about latency, but that can be solved by people that have specific needs of imports having been eager by reifying imports as part of application startup, my use case doesn’t need an interpreter flag.
I think the security side is actually a red herring and the actual root issue here is that pip and other apps like it make assumptions that aren’t safe with or without lazy imports. pip shouldn’t run from the environment it is modifying.
Changing that is a large change to push onto pip, and there are other options that preserve the current security boundary without costing the main gain (such as pip reifying imports prior to processing remote files during install-like commands, but not prior to then, allowing response time to not incur this for things like --help)
Your idea doesn’t necessarily make it more useful; it just reduces the risk of causing problems, at the cost of making PEP 810 even more complicated, for a feature with a very ambiguous use case.
You’re right, it has no effect on users, but it does affect core developers who have to maintain the code.
We can document that the standard library does not work with -X lazy_imports=none, but then we have a broken flag just sitting around. I’m requesting that we remove it during the alphas so we don’t have to go through the trouble of deprecating it.
Yeah, that’s the plan. I’m not going to change anything without the blessing of PEP 810’s authors and the SC.
Well, just about anything can be used for debugging. This was brought up on Discord as well, but I don’t think -X lazy_imports=none will really help debugging in practice. It can confirm that your lazy imports broke something (though it won’t even be able to do this if most libraries ignore support for the flag), but it can’t tell you what, why, how, or where things went wrong.
My guess is that the PEP cites debugging because one of the authors found it useful while implementing lazy imports, as they could quickly compare output between eager imports and lazy imports. But, since we shouldn’t have to do that again, it shouldn’t need to stick around. (And if we do need to reimplement lazy imports for some reason, we can add the flag back on debug builds.)
sorry, that was too harshly worded and unfair to pip and it’s maintainers.
It’s a fragile assumption, and one that has been load-bearing on multiple recent attempts at improving other parts of python (including with me making the argument that you’re making).
The concern isn’t a red-herring, but the conclusion that pip needs this flag seems to be wrong here because pip can reify imports from runtime as needed, keeping the current security boundary.
This is in contrast to the issues other proposals like plugins during install have where the proposal has relied on pip and other installers being able to install then run something that was installed, where that doesn’t seem possible to maintain that boundary.
I agree, I think it’s important that the security concern is highlighted, this flag seemed like the obvious solution but it’s more important just state the security boundary and ask Python package installers to enforce that boundary.
You could have a dedicated CLI option (e.g. pip install --allow-overwriting-pip) for those special use cases and still forbid it by default, just like you already have --break-system-packages for another case where the user may legitimately want to do dangerous things.
But you still need to document what imports can and can’t be made eager using the filter functionality. My point is that you don’t actually gain anything meaningful in terms of avoiding bug reports.
Would you not also have to remove sys.set_lazy_imports("none") for the same reason?
I’m OK with leaving the rest of the filter system in place, as it’s reasonable to argue that anyone using sys.set_lazy_imports_filter is deliberately choosing to rely on implementation details of the modules being forced to load eagerly, and therefore there’s no need to try to document for every single module whether it’s “safe to force eager loading”. The key difference with the “none” forms is that they don’t have an “escape hatch” - it’s all or nothing.
Anyone who really wants to force all imports to be eager can still do sys.set_lazy_imports_filter(lambda *args: False) - and that form makes it very clear what you’d change if you needed to add an exclusion list of modules that could be imported lazily.
Also, isn’t -X lazy_imports=all just as problematic? Maybe the stdlib doesn’t break in that case[1], but after all, isn’t it doing precisely the same as PEP 690 proposed, making all imports lazy, which was rejected because it would cause a split in the community over whether import-time effects are allowable? If the argument that “projects will need to support -X lazy_imports=none, otherwise we’ll just have a broken option” just as applicable to the “all” case?
To put this another way, I’m hesitant to single out the stdlib here. It’s an important special case, but it’s far from the only case of widely used library code.
Yup, that too. I’m also referring to that when I say -X lazy_imports=none, because it’s just more tedious to type out every time.
Yeah, I totally agree. The filter makes it much clearer that you’re basically hacking Python’s import system.
I’m less concerned about that option because it doesn’t actively prohibit using PEP 810 in place of old-style lazy imports. It might also break someday, but it seems useful enough in the short term that it’s worth keeping around.
The ‘all’ method works with the lazy imports filter, I actually think that’s the filter’s original purpose. There’s not an equivalent when using the ‘none’ mode - the equivalent is just using the filter again in normal mode.
A user who wants to force lazy imports to be eager everywhere except in modules it breaks can use a filter like this with lazy_imports=normal (the default)
def eager_filter(importer, name, fromlist):
if name in modules_that_break_under_eager:
return True
return False
While a user who wants lazy imports everywhere would essentially do the inverse, with lazy_imports=all:
def lazy_filter(importer, name, fromlist):
if name in modules_that_break_under_lazy:
return False
return True
I’m having a hard time coming up with a coherent threat model where lazy imports meaningfully affects things. I understand the idea that lazy imports means that someone could swap out a piece of pip at runtime due to a delayed import— but without a delayed import they’re just swapping out a piece of pip the next time pip is invoked. So you’re only potentially affecting situations where pip has lazy imports, and is never invoked again within that environment.
Even then, they could just drop a sitecustomize or a fake python binary or something, that means that you’re only safe if you never invoke Python itself again in that environment, at which point why did you even install something?
Historically Python packaging has not attempted to try and treat a malicious package that a user has chosen to install as being part of the things that are in scope for security considerations (different than just downloading them, etc), because at the end of the day if you’re installing something then there’s not much we can do to meaningfully secure it if the thing you’ve chosen to install is malicious.
I fail to see how lazy imports with pip meaningfully changes anything in that regards. Yes technically it shifts the point of when an attacker gets their code executed forwards a tiny bit, so you could come up with some scenario where it’s technically making the situation less secure— but I’m struggling to think of a situation where that’s actually a realistic case.