Normal global variable and module-attribute access still work. It’s just that accessing module.__dict__["lazy_imported_name"] doesn’t give you the real object. The cases where this is an issue should be few. It’s IMO similar to obj.__dict__["a"] and obj.a not necessarily giving you the same thing because of the descriptor protocol. (e.g. see class methods)
1 Like
I think I have a way to make lazy import work in a try statement.
How about we allow and compile a lazy import statement in a try statement into IMPORT_NAME with a new flag of oparg & 0x02 so that when the interpreter executes it, it will call importlib.util.find_spec() to find its spec, and stash the spec into a slot of the lazy module object, so that when the lazy module is reified it will not need to find the spec again?
In other words, make lazy import less lazy when the compiler finds it in a try statement.
I think the major problem with this idea isn’t that it is hard to implement (though that is also true), the semantics and effect on lifetimes and various other things are, in my opinion, enough to doom the idea of try/except “just working”. For one thing, like Brett, I would find it very surprising to have this “action at a distance” where you have a disembodied try block that follows your lazy import around. Imagine:
try:
lazy import foo
foo_imported = True
except ImportError:
foo_imported = False
some_global = 3
else:
print("Is this allowed?")
Does else ever get executed? If so, when? Does it get executed when foo gets reified? What does foo_importedget set to? When does some_globalget set? Does it start as Truebut change to False if the reification fails? If my code later looks like this:
if foo_imported:
def _foo_bar():
# implementation
else:
_foo_compat = foo.bar
Now what happens? Obviously you can decide on how all of this stuff is going to work, but as you can see it gets complicated very fast, and the key to designing something like this is to make it simple and do what people expect. Any “try/exceptworks magically” solution is so far outside of normal Python semantics that I think there’s no way to make it have semantics that we can expect people to reason about effectively.
6 Likes
There is no “action at a distance” with what I’m proposing.
To be clear, what I’m proposing is for:
try:
lazy import foo
bar = 1
except ImportError:
bar = 0
to be roughly a shorthand for:
lazy import foo
try:
globals()['foo']._spec = importlib.util.find_spec('foo')
bar = 1
except ImportError:
bar = 0
except that when foo is reified it will skip finding its spec and reuse the _spec attribute.
So with the exact semantics documented there should be no ambiguity in how your questions above should be answered.
3 Likes
I’ve been waiting for the thread to settle, and many points have already been raised. One topic that hasn’t been discussed yet is how to handle exceptions that occur during module import.
In the case of a normal import, exceptions are raised while the program is being loaded. In contrast, inline imports are usually wrapped in a try/except block, since they occur at runtime rather than at startup. In other words, no one would be there to notice the exception when it happens, unlike with regular imports. That’s another reason I personally avoid them, along with the reasons mentioned in the PEP.
Lazy imports share the same drawbacks as inline imports, since they are essentially syntactic sugar for inline imports. Although they may appear like normal imports at the top of a file, under the hood they function just like inline imports.
There’s a workaround to make inline imports more foolproof: avoid executing code in the imported module and focus only on definitions, which aligns perfectly with the rejected PEP 690.
How should exceptions be handled during reification?
1 Like
Don’t forget the possibility of overwriting a __getattr__ to intercept incorrect names for mod.<name>.
There are a few things to consider here, and different ways to access attributes are definitely on there.
Ah, sorry, I hadn’t understood what you were proposing. Thank you for taking the time to explain it further. I agree that what you are proposing is a lot more feasible, but I’m still -1 on it because while it’s easier to reason about, it now gives people two completely different regimes and import mechanisms to think about.
-
Within
try/exceptblocks, you now have eager binding to module lookup but late importing, so you can still have uncaughtImportErrors, as long as they are raised at module import time rather than spec finding time. -
Within
try/exceptblocks you also lose a lot of the laziness in situations where “find the module” is the slow part (or one of the slower parts), and for the most part this will happen competely silently and people will not notice that their modules are only half-lazy. -
Would we need to special-case
except ImportErrorand have everything else be aSyntaxError, or wouldexcept WhateverErrorstill be essentially a no-op? People would again be confused whyexcept ImportErrorappears to be caught correctly but everything else is not.
I do hope than an ergonomic pattern for what you are trying to do here can be found, but I don’t think this is the right one.
3 Likes
Ah, okay, that’s fine.
Is there a reason not to do this in any case, regardless of whether it’s
in a try block or not?
It seems a bit magical to have lazy import behave differently depending
on its context.
Maybe a special construct try lazy import (and a corresponding try import) could be worth it:
bar = 1
try lazy import foo
else:
bar = 0
Would be the equivalent to @blhsing example.
Of note:
- No body for the
tryclause: If you need more complexity, be specific. - No “if-succesful” clause either. This is the one I am least sure about. There should almost always be an equivalent design as I showed above. But maybe this limits the usefulness too much
- No handling of different errors. The goal is to handle the ImportError if that module is not installed, i.e. if find spec fails.
- Yes, this would change the behavior of a lazy import - IMO this is enough surrounding extra syntax that it wouldn’t be too surprising.
Anyway, this would probably be a separate PEP.
1 Like
If the intention is to provide a way of ensuring that a module exists before trying to lazily import it, seems to me it would be better to provide something you can explicitly call to achieve this, e.g.
if importlib.module_exists("mymodule"):
enable_mymodule_features = True
else:
print("Warning: Some features not available")
It could still cache the spec somewhere for when the module is later imported, either eagerly or lazily.
3 Likes
Well the idea is to make imports as lazy as possible, hence the “magical” context-aware behavior, which I personally think matches the user’s likely intention intuitively though I can understand why it may be deemed too magical for some.
I’m not sure what your snippet is supposed to show, but the lazy import lel is reified there when getattr(lel, "foo") is called.
Why would it raise the import exception in the try block? On the other hand, another exception might be raised eagerly, especially if the try block spans multiple lines. Why disallow that?
(I’m not Pablo but I think that) It shows that it’s not getattr() on a module object that reifies the lazy import within that module (lel lazy imports blech). It clarifies a previous example, and shows that calling getattr is not like accessing the module’s dict.
I’m personally on edge regarding the thing of denying lazy imports within a try block. I understand that it prevents an obvious footgun, but it also feels like a bit of an arbitrary restriction on what the user might be trying to express. If they understand how lazy imports work, they will know that they don’t raise ImportErrors on the statement itself. This doesn’t seem harder to teach than anything about deferred side effects of imports. Someone already suggested that it might be the linters’ responsibility to warn when try/lazy import are combined in an ineffective way.
On the other hand, restricting it now means it can be extended later. Either by just lifting the restriction if experience shows it’s too arbitrary; or possibly with a more thoughtful design regarding the possibility to capture ImportErrors, although I can’t think any right now.
2 Likes
This has been discussed afaiu (although the thread is long) and one suggestion was that if you want to import one package in case the import of the first fails, you write another module containing only
try:
import a
except ImportError:
import a_fallback
And the you import that module lazily.
Also, in my experience inline imports are just raw import statements causing import error at some point during runtime, so nothing changes there with this proposal.
4 Likes
I am normally a bit worried of new language changes but I have to say that this PEP is everything I’ve been waiting for! I maintain three physics simulation packages and a bunch of other scientific libraries and the problems described in the PEP have been affecting us for years. It’s embarrassing, honestly. And Jupyter notebooks? Forget about it. The first cell in every notebook is just import statements, and users sit there waiting while the kernel spins up and loads everything. I’ve had colleagues joke that they time their coffee runs by how long it takes their notebook to finish starting.
Here’s where I want to push back a bit though: the restriction on with blocks needs reconsideration as someone was pointing out before. We’ve already got custom lazy import machinery scattered across our codebases (most of it fragile and hacky, if I’m being honest). Having a clean migration path it’s fundamental. Scientific packages move at a glacial pace because we support Python versions going back years. If I can’t write code that works cleanly on both 3.14 and 3.15+ without maintaining two completely different implementations, we won’t be able to adopt this until maybe 2028 or 2029. The context manager pattern actually makes sense here. Yes, contextlib.suppress exists, but I’ve literally never seen anyone use it with imports in real production code. It’s theoretically possible, sure, but it’s such an edge case. Meanwhile, most with usage is about managing resources or scoping behavior, which feels totally different from exception handling semantics. Linters can catch the weird stuff.
What really matters is this: if lazy imports work in with blocks from day one, I can write a simple compatibility shim and start using this feature immediately. Something like a context manager that’s just nullcontext on 3.15+ but falls back to our current hacky loader on older versions. Without that, we’re stuck waiting years to drop 3.14 support before we can touch this.
I haven’t been this excited about a Python language feature since f-strings! Thanks to all the authors for pushing for this.
3 Likes
Some quick questions I have after reading the document:
-
What’s the technical reason lazy imports aren’t allowed inside functions? Is it a scoping issue with the proxy objects, or more about implementation complexity with local namespaces?
-
Are lazy imports allowed inside
ifstatements ormatchblocks? I saw the restrictions on try/except and with blocks, but what about:
if sys.platform == "win32":
lazy import windows_module
else:
lazy import posix_module
Does that work, or do you need to put both at module level and just conditionally use them?
- If I have a module that does
lazy import numpyat the top level, then somewhere deep in the code there’s a function that accessesnumpy.array, does the reification happen synchronously on that thread?
Perhaps, the ImportError is raised at (first) attribute access, because the module we want to access might be modified at runtime (although it is not recommended to open() files you’ll import).
So that’s the reason no ImportError (LazyImportError?) is raised there.
1 Like