I meant, an importer may have made an assumption, or made promises to users, about getting triggered at the import statement. This is essentially another case of import side effects, so I suppose the PEP covers the topic. It might make sense to note that it affects importers in the same way, since the impact to such importers would be potentially trickier.
2 Likes
This is definitely the right approach. (/me, speaking from experience)
3 Likes
This is exactly the same problem as having an import not at the top of the file:
lazy import my_module
def foo():
my_module.b()
is effectively the same as:
def foo():
import my_module
my_module.b()
and the impact of side effects on import is the same as well. So you don’t have to know more or less about a module that you do now.
3 Likes
Thank you for the great proposal. I’m thinking about these questions:
It’s always easier to forbid something first, and then allow them later. Context managers can be used to suppress or otherwise redirect exceptions (such as contextlib.suppress ), so if we don’t want to allow lazy imports in try/except,we shouldn’t allow them in with either, unless we can think of a way to do it nicely.
- Should there be a way to access module
__dict__without reification from outside the module* Something like__raw_dict__has been suggested for introspection use cases. Any other options
This will be required on day one, mainly for the benefit of people profiling their code for performance changes with the use of lazy import.
- Should we include something like a free function to force reification on objects (or the entire module) instead?
Such a function should ideally not be required for correctness in any code, but I can think of it being useful in cases akin to those of gc.collect(), i.e.
- introspection, profiling
- to take @sirosen 's server example, ensure the expensive import operations happen when you can afford to, not when you’re actually serving a request in a function(though in this case, just globally disable lazy import)
On performance, one thing that happens to my mind is that we should probably advertise less about performance benefit of lazy import because in many cases it’s not really a benefit: when you actually use a module, sooner or later the module itself needs to be imported and its code executed anyway; lazy import just shifts the import burden from the import statement to wherever the module is first mentioned. Lazy import also means that the first invocation of a function may have very different performance characteristics than subsequent invocations (much like an import statement within the function), and this might have implications on future optimizations and JIT efforts.
4 Likes
A reify function would be sufficient. It could be used to reify an object or module when the target module is not under one’s control. This way, you wouldn’t have to clutter otherwise clean code with verbose try/except blocks that hurt readability.
This isn’t a blocking issue, just a convenient feature to have.
By the way, this is a very well-crafted PEP that tackles a long-standing issue. I guess it’s time to retire my own dynamic module loading hacks.
2 Likes
This is actually a good use of LLMs; summarize the topic as well as asking it if your questions have already been brought up.
7 Likes
Quick update: We’re preparing a PEP revision that addresses some of the clarifications and concerns raised in this thread, including valuable feedback from an out off band long conversation with @eric.snow (thanks a lot for your time 
). Thank you all for the thoughtful engagement!
To help us converge towards a decision, we kindly ask everyone to read PEP 810: Explicit lazy imports - #242 by pablogsal and focus the discussion on the genuinely open questions outlined there.
Please help us by respectfully not reopening discussions on topics we’ve indicated are closed (syntax, new semantics, try/except blocks, etc.). We understand some of you feel strongly about these points and I promise you all that we have spent a lot of time re-evaluating that and I have read and consider every single comment, but revisiting them after we have made a decision on what we believe is the best option makes it much harder to reach consensus and move forward.
Thanks again for your patience and understanding!
— Pablo (on behalf of the PEP 810 team)
12 Likes
Since it may have gotten lost, I have a post here outlining why I think that conceptually with blocks are different than try/except even though you can use them in place of try/except, and why in this case it’s actually important to get it right from the beginning: namely that context managers are probably our best shot at an ergonomic functional backport, and if PEP 810 launches without support for them it makes it that much harder to write code that works in all versions.
5 Likes
I’ve found a workaround for it, and I think it’s good enough for me to not need __raw_dict__ as long as the workaround can be guaranteed to work:
# m.py
lazy import math
# t.py
import m
def raw_dict(m):
return {name: object.__getattribute__(m, name) for name in dir(m)}
print(raw_dict(m)['math'])
print(m.__dict__['math'])
Running python t.py outputs:
<lazy_import 'math'>
<module 'math' from '/home/blhsing/src/cpython-lazy/build/lib.linux-x86_64-3.15/math.cpython-315-x86_64-linux-gnu.so'>
Although the workaround performs much slower than an exposed __raw_dict__ if it were made available, I can’t think of a use case where one would need this introspection ability in a high-performing production code.
Reification on a specific object is just a matter of evaluating the name isn’t it?
lazy import math
math # reifies math
It would be nice, however, to have a simple function that reifies all lazy objects in the entire module, even though it can be done in a less readable way by accessing the module’s __dict__:
lazy import math
import sys
sys.modules[__name__].__dict__ # reifies all lazy objects
print(globals()['math']) # <module 'math' from ...>
1 Like
IMO, debuggers that permently display the state of some context would fall into this usecase: They want to be as fast as reasonably possible while impacting the semantics of the program as little as possible. If it’s easy to expose something like __raw_dict__, than it should be done.
Pablo has been very unclear on this. AFAICT, what is meant is this:
globals()['foo'] returns a lazy imported object. We then have a method on these objects, reify, that can be called like this: globals()['foo'].reify(), that does the lookup/import if still needed. Since it can’t know where it was referenced from it wouldn’t replace the original variable reference.
Two notes:
- if there is still a significant amount of people who don’t like the current solution then it’s completely correct that “it’s hard to reach consensus”. Just saying “please don’t complain about this anymore” doesn’t mean those people suddenly agree with you (as would be needed to claim to have consensus).
- However, I accept that if the PEP authors don’t want to change their mind, continued discussions aren’t helpful. Instead people who have strongly feeling about a specific topic should state “I am -1 on this PEP because (the syntax/the semantics/…)” and leave it at that. This is just so that these voices aren’t drowned out, responding to them here is not going to be helpful. This is just so the SC can later on decide whether or not to take these voices into consideration.
4 Likes
Yes, fair enough.
But there is already a .get() method for this exact purpose.
EDIT: Well I just tested it but found that reification though the get method doesn’t trigger a name rebind:
lazy import math
print(globals()['math'].get()) # <module 'math' from ...>
print(globals()['math']) # <lazy_import 'math'>
Should it? You’re right that it can’t be done robustly though since the lazy object doesn’t know the name that references it.
Thanks for raising this. I apologise if my message was
unclear in this regard. Allow me to try to clarify it:
We’re not claiming consensus or trying to silence dissent. We recognize that many people support the PEP overall but feel strongly about one or two specific aspects, and that’s often what drives the back-and-forth. If people have serious concerns about particular elements, those voices need to be heard by the Steering Council, and we’re not trying to drown them out.
What we’re asking is to avoid extended back-and-forth debates on topics where we’ve indicated our position. State your concerns clearly for the SC’s consideration, but repetitive rehashing makes it hard for anyone (including the SC) to identify the actual points of contention.
In any case I think you articulated this very clearly in your message so thank you for that and for raising it!
8 Likes
For the open questions:
withstatements, yay or nay?
Nay, as the backport syntax will necessarily use regular import statements, not lazy ones. One potential implementation strategy would be to override the regular __import__ inside the context with one that maps to __lazy_import__. That would require some careful design to avoid having things blow up if reification is attempted while the context is still open, though. Maybe __lazy_modules__ = ["*"] could be allowed, so the context manager could just temporarily set that instead of messing with __import__?
__raw_dict__
Seems worthwhile to me, and parallels nicely with globals().
- Standalone way to force reification of a namespace
Also seems worthwhile to me. We may even want to update gc.freeze to reify everything in sys.modules by default (with a new flag option to opt out of the global reification).
3 Likes
So I was less than clear in my reasoning here, but my understanding is that in the current model, when __lazy_modules__ is set, it will not lazily import in a context, but it also won’t be a syntax error, so:
__lazy_modules__ = ["foo", "bar"]
import foo # lazy
with blah():
import bar # eager
If we change it so that both of these are lazy, then I think that lazy import … should also be allowed in the with block as well, because otherwise I think people will think “well lazy importing isn’t allowed in context managers so obviously this is eager”.
If however we allow lazy imports in a context manager, you don’t need any hacks in Python 3.15 and no careful thought needs to be done about replacing __import__ with __lazy_import__, because it will just work out of the box once __lazy_import__ exists, and the hacks can all go into the backport.
3 Likes
Yes, and it’s worth noting that from @cekopic’s post it is clear that the expected use case is for blah to be a context manager that injects a lazy loader hack into the meta path in <=3.14 versions, and nullcontext in 3.15+.
3 Likes
I don’t follow that reasoning. The cross-version compatible syntax can’t use the new keyword at all, as that would cause a syntax error on older versions.
Since the lazy import machinery won’t exist on those older versions (other than LazyLoader and friends), the context manager will either be a no-op, or override __import__ to force partially lazy loading.
On 3.15+, allowing lazy imports inside arbitrary context managers creates the same problems as allowing them within try/except statements.
Your point about needing to be consistent in the way nominally eager imports are executed is valid, though.
Based on that, my suggestion would be that any context manager should be based on overriding __import__, with the lazy import protocol adjusted to make it possible to do that without triggering infinite recursion. For example, during reification, we could check for __reify_import__ before falling back to checking for __import__. Then the context manager would be able to copy __import__ to __reify_import__, __lazy_import__ to __import__, then revert those changes when done. Trying to enter the CM when __reify_import__ was already set would raise an exception.
After the implementation of PEP 810, this:
__lazy_modules__ = ["foo"]
import foo
Is equivalent ot:
lazy import foo
So my proposal is that I or someone else will implement a backport that in Python 3.14 replaces __import__ with some function that more or less looks like this one in Python 3.14 but in Python 3.15 is equivalent to contextlib.nullcontext, so that you can do this:
__lazy_modules__ = ["foo"]
with backports.lazy_imports(__lazy_modules__):
import foo
Maybe the context manager can also try and find out what __lazy_modules__ is by handwave handwave magical introspection or whatever¹, but the idea is that whatever hack you need to do to make it work in Python 3.14, you don’t need to do any tomfoolery in Python 3.15+, because the native lazy loading mechanisms would be in place, so backports.lazy_imports doesn’t actually have to do anything, it just needs to not disable lazy importing.
If we maintain things as they are, backports.lazy_imports needs to do the hacky stuff in Python 3.14 and it needs to do the hack you are describing in Python 3.15, because lazy imports are disabled in with blocks, which is the main driving reason I have for allowing these things in with blocks (it’s a relatively minor one in that you could work around it, but I also think the benefits of disabling lazy imports in with blocks are pretty modest as well).
¹I realize that for these purposes we don’t actually have to look at __lazy_modules__ at all in Python <3.14 because we can be assured that anything imported in that context would be something someone intended to import lazily, but doing so reinforces that this is a backport of the functionality, so I think it’s helpful to include the part where you need __lazy_modules__, but the details of this kind of thing don’t matter much since there will probably be multiple versions of this.
4 Likes
People will need to interact with the proxy objects at some point (for a large quantity of obscure reasons; inspection, testing, optimization, fallback tricks, path checks, compatibility tests…).
Probably a lazy imported object can have a __lazy__ attribute, explicitly centralizing an interface with the proxy. I can foresee a bunch of uses now :
lazy import mymodule
mymodule.__lazy__.check() # perform any kind of quick tests to check whether the import system will succeed to import (module init errors apart) and return boolean if yes
mymodule.__lazy__.get_path() # seek and return path to module root file or folder, like importlib.util.find_spec
mymodule.__lazy__.raw_dict() # seek the __raw_dict__ by parsing
mymodule.__lazy__.reify()
hasattr(mymodule, '__lazy__') # check if mymodule was reified
and this interface might be a place of first choice for future improvements.
Firstly, hats off to everybody involved 
! An excellent PEP as well as the iterations of it over the last few days - phenomenal handling of the (extensive) feedback 
.
I’m very much +1 on the PEP, and think it is quite a major step forwards for the language, especially for typing related imports. The big use case that I think this can resolve in my org is with JPype, where imports need a JVM to actually work. lazy from jpype.java.utils import Properties can now have non-trivial requirements (like needing to have a running JVM via the JNI interface) that need to be met for import to actually work, without forcing the user to have those requirements setup at (lazy) import time.
Given the motivation of the PEP to mitigate an immediate cascade of imports, one (possibly very controversial) thing that should probably be covered in the PEP is the import of parent packages when importing subpackages. The import mechanism documented in 5. The import system — Python 3.14.0 documentation pre-dates lazy imports, and it isn’t obvious that a lazy from a.b import Thing; Thing() should need to have imported a/__init__.py in a lazy world. Making this change after the PEP is implemented would be a breaking change, and so I think it is worth considering now (and strictly only for the lazy import case). This is a big existing import side-effect behaviour baked into the language which could be removed, but I acknowledge that a huge drawback of what I’m proposing is that it would result in different import behaviour for lazy vs non-lazy imports, which would require those depending on the side effect behaviour to become more explicit if they wish to support lazy importing. I think this aligns with other behaviour shift documented in the PEP under Observable behavioral shifts (opt-in only).
For completeness, I was originally in the camp that basically all imports are going to become lazy, and we will inevitably setup linters to always replace normal imports with lazy ones, so why not just transition to lazy imports by default? Having actually played with the implementation a bit, I’m happy that this has been ruled out in the recent iterations of the PEP, can see how it can become a future PEP, and think this is a good call for this one[1].
The experience of lazy imports in the REPL is not so good if your imports have side effects (like writing to stdout), so I can totally imagine wanting to not have lazy imports by default in the REPL, but still have lazy imports work within the modules that you import such that optional dependencies work out nicelyFootnotes ↩︎
4 Likes
For now? Yes. This can be easily added in the future.
I appreciate that you’ve discussed this a lot, and studied the effects on large codebases, but… could __dict__ itself stay raw?
Yes, __dict__ “crosses module boundaries”. But so does moving imports into functions, or other kinds of “local” code reorganization. Introspection needs to be able to handle changes like that already.
I imagine research in private monorepos focused a lot on avoiding forking of introspection-heavy libraries, which is less of a concern now. For a PEP, we primarily want a clean data model; the libraries are likely to accept PRs to adapt to a core feature.
Entire module? Yes! Don’t miss an opportunity for raising ExceptionGroup ;)
(If __dict__ stays raw, with this we don’t need an extra __reified_dict__: you can call the reification function and then access __dict__.)
Imdividual objects? Also yes. If there’s no free function for get(), IMO the method should be made into a dunder that’s a no-op on real module objects (__reified__?). Otherwise you always need an isinstance check before calling it:
# action at a distance -- perhaps in another module, maybe in usercustomize for REPLs
import json
... # ... some time later ...
lazy import json
globals()["json"].get() # AttributeError! #[line edited to add globals()]
No. AFAIU, that would mean lazy module objects would need to be tightly coupled with “their” module. Also, consider this:
lazy import foo
lazy_foo = globals()["foo"]
foo = "something unrelated"
lazy_foo.get() # should not rebind `foo`; cannot rebind `lazy_foo`
3 Likes