We are very exited to share with you PEP 810: Explicit lazy imports: We’re proposing an opt-in lazy import syntax that defers module loading until first use, aiming for faster startup, lower memory, and clear semantics with zero overhead when not used. We worked hard to balance user value, an elegant implementation, and predictable behavior, and we’ve included alternative implementation paths where trade-offs are worth discussing. Not only do we believe this is important for Python, but the fact that multiple companies (Meta, Google, HRT …) and communities (e.g., Scientific Python) are rolling their own imperfect solutions or even forking CPython, highlights that this is a critical feature that the language needs. We’ve explored the design space extensively and we think this is the best chance to land explicit lazy imports in the Python language; help us make it as good as possible. There’s a substantial FAQ: please read that first.
Thanks to Paul Ganssle, Yury Selivanov, Łukasz Langa, Lysandros Nikolaou, Pradyun Gedam, Mark Shannon, Hana Joo, teams at Google/Meta/HRT/Bloomberg, the Scientific Python community, and everyone who contributed feedback
While we found from foo lazy import bar to be a really intuitive placement for the new explicit syntax, we quickly learned that placing the lazy keyword here is already syntactically allowed in Python. This is because from . lazy import bar is legal syntax (because whitespace does not matter.)
Not without violating PEP 387 or adding a __future__ import, which seems very heavy-handed for this. It’s an option to consider if the majority opinion is overwhelmingly that the breakage is worth the slightly different placement of the keyword. (I prefer having lazy or whatever the keyword is at the front, to make it immediately obvious what the line is doing.)
As someone who‘s in the business of codegen, I’m very excited by this. I’ve spent already a lot of time thinking about making attrs lazy for better startup performance but this would essentially solve it for all practical purposes. Dataclasses et al would suddenly lose their only downside
I also like (for DX) to allow access to all features of my packages to reach them from the root package. This would practically improve import times of all of them (I’ve even implemented lazy imports in stamina due to a request) so I can’t wait to deploy __lazy_modules__ in all of them.
Maybe this is just “new feature looks weird” goggles, but the existence of the space makes this seem like a “keyphrase" made up of two keywords. Are there other examples of this already? (None are coming to mind outside of not).
My ugly suggestion, but not any uglier than many other names in the language/stdlib, I think considering (and/or rejecting) a new keyword taking the place of import could work: i.e. lazyimport.
There’s also the possibility of a different verb, like (well apparently I can’t be creative this morning, so please accept this placeholder) from x gimmie y.
With regards to the technical details, it’s not worth bike shedding over too much, but this is the very first thing people will see of the change (the syntax). So it does deserve some care (in this case I think just a stronger argument, in terms of rejected alternatives)
This looks super exciting. I can see it helping CLIs, TUIs and things like running web applications locally with code reloading on changes a lot.
One question about the global lazy import control: is this expected to be a provisional or a permanent thing? I would prefer it to be permanent since I would like to run network services in production with lazy imports forcefully disabled. Simply because in that particular context I prefer potential errors to bubble up right away, at startup.
I do agree that the syntax looks a little foreign at first glance, but I suspect that might just be because the PEPs website isn’t highlighting lazy as a keyword, which is making my brain confused. I think it will look a lot more natural once it becomes highlighted, the same way async def and friends do.
I’d support this from a teachability perspective, it is easier to explain that prepending import with lazy is always the right thing to do to create a lazy import. It also creates an easier pattern to look for, rather than having to do something like lazy (import|from).
I particularly like that this populates sys.modules only after the proxy is reified.
With existing lazy import tools, I already have some applications which test that modules don’t accidentally break performance guarantees by importing things too early. This is important when A is slow to import, B imports it lazily, but B also imports C, D, E. Testing that C, D, E don’t import A eagerly maintains correctness of B. sys.modules inspection has proven to be the most reliable test.
I think explicit would be a better keyword than default, as then I don’t need to look up what default means for my version of Python. If the default changes in the future, I expect to get that with no -X flag, but if i set it to -X lazy_imports=explicit, then I want to keep that behavior even if a future version becomes implicit. default could be an alias for whatever mode is default, for simplicity of shell scripts passing -X lazy_imports=$LAZY.
Can sys.set_lazy_imports() override -X lazy_imports=...?
Hi! Love the PEP. A few questions about the spec that are not clear to me on first read:
The “Syntax restrictions” specify “only allowed at the global (module) level, not inside functions, class bodies, with try/with blocks, or import *". It’s not clear to me if this is a list of examples or a specification list. For example, if a module contains if foo: lazy import mymod, would that import be “at the module level” and thus valid, or not?
The tryrestriction seems to prevent using this feature in the very common pattern where an “optional” module is attempted to be loaded, and a fallback mechanism (either providing an alternative source or dummy replacement) is provided in the except clause. I’m talking about:
try:
from typing import LiteralString
except ImportError:
LiteralString = str
I understand why this is hard to do lazily, but are there any suggestions on how to approach this?
When a module has a lazy import foo, and ”foo” is already on sys.modules because some other code already needed it, does the spec say if my current module will get a real module or a proxy in its globals?
The behaviour of argumentless dir()is not clear. If my module contains lazy import foo; dir() , does the call force the import of foo? (The document only clarifies what happens with dir(foo)which, quite reasonably, forces the import.)
Finally, given that there’s likely some existing code that would benefit from using lazy imports, but they may also be using globals() (for example, in an eval()call). The PEP says “if you add lazy imports to your module and call globals(), you’re responsible for handling the lazy objects”. What would be the recommended code changes in that scenario for the owners of the module if they don’t want to have lazy objects (which I imagine is a common scenario)?
I think the PEP looks great. Like this post above mentioned, I think a common use case for deferred imports is for optional dependencies, so I wondered about the best pattern for testing for import errors. Before one might have:
def some_func():
try:
import numpy
except ImportError as exc:
raise RuntimeError("numpy is required for using some_func!") from exc
It seems like the new pattern would be:
lazy import numpy
def some_func():
try:
globals()["numpy"].get()
except ImportError as exc:
raise RuntimeError("numpy is required for using some_func!") from exc
Perhaps for optional imports where you want to check for import errors there is not much benefit to the new syntax over the old pattern of putting an import into each function that needed the optional import? There is still a modest benefit to grouping all imports at the top of the file instead of hiding them in functions.
I love this. It is badly needed. Many style guides tell people to always import at the top level, but when writing a command line application that makes using –help take multiple seconds if you have heavy imports. It also makes it really tricky when you want to use dynamic analysis of the code to help populate your argparse.ArgumentParser (i.e. discover other modules that could add subparsers for modal CLIs).
This will go such a long way to alleviate some of these issues. However, it might not solve everything. For instance, if I want my CLI to depend on arguments introspected from the signature of a class’s __init__ function, but that class inherits from something heavyweight (e.g. torch.Module) then my CLI will still have a slow startup time… although I suppose torch itself could make use of the lazy keyword to make importing Module much quicker.
In any case, this is the best Python news I’ve heard since ordered dictionaries in 3.7. I really hope this lands.
I love that:
you can disable lazy imports and make them all eager
lazy import * is not allowed.
the word lazy was chosen instead of defer, although I wouldn’t complain either way.
I’m not sure if:
disallowing lazy statements inside classes or functions is necessary. In fact lazy imports inside a function could be very useful if there is a control path that avoids needing the import. It lets me put all local imports at the top of the function. I suppose I could put it on the top of the module and it would be effectively the same, but I don’t see the downside. Does it make the implementation harder?
dir should always reify the variables. Maybe even .__dict__.keys() should try to avoid reifying as we are not actually accessing the variables, we are just asking what we could access. (especially because globals does not reify). At least for dir, wondering about more details on what the rational is here.
lazy import numpy
def some_func():
try:
numpy
except ImportError as exc:
raise RuntimeError("numpy is required for using some_func!") from exc
Edit: I’d also like to make it clear that it would make just as much sense to replace the code within the try block with whatever business logic you were intending to execute using numpy from within some_func. In addition, if you’re okay with the user seeing the ImportError in lieu of the RuntimeError, you could remove the try/except entirely.
My only reservation about this is lazy is a pretty generic word to add as a keyword that will be restricted to being used in just one context space. If a feature come down the line where it would make sense to use the keyword lazy taking such a generic keyword might make implementing said feature more difficult in the future. I know its a soft keyword so the context space it can be used in could always expand in the future but its just something that I think should be considered. At the end of the day this reservation is fairly tiny but just something I think should be at least discussed.
Hi everyone,
