Or can it?
from importlib import lazy_import
np = lazy_import('numpy')
Static analysis tools know that lazy_import does importing - first string argument being the package name.
I like being able to lazily import by just adding lazy in front of my import statements.
Although, my first instinct is to want to have this modifier work in other cases, like when calling functions:
result = lazy expensive_function()
if conditionA:
print("A", result)
elif conditionB:
print("B", result)
else:
print("No expensive function execution")
I wonder if Python is gonna end up implementing this in a few years because users start to expect they can lazily do other things too.
I assume the migration path depends on the mechanism and it would be a bit difficult to give general advice but my view is that:
If you’re using lazy_import() from an existing library like lazy_loader, the library itself just needs to update to detect Python 3.15+ and use native lazy imports internally. Your code doesn’t change at all: you keep calling np = lazy_import('numpy') and it automatically uses the native implementation when available. You get the benefit for free when you upgrade Python.
If you wrote your own custom lazy_import() function, same principle applies. Update that function to check sys.version_info >= (3, 15) and use native lazy imports when possible. All your existing call sites continue to work unchanged.
If you’re doing function-level imports (importing inside functions to defer loading), you can gradually migrate those to top-level imports with __lazy_modules__ on newer Python versions while keeping the function-level imports as a fallback.
I assume most of the conditional import/lazy import can be done via __import__ and __lazy_import__ checkinf if the later is available.
This is my main point. There is a certain need for a more general concept.
If it so happens that more general “deferred evaluation” is deemed to come into fruition there is a very high risk that decisions being made for this, in contrast limited feature, will be deemed suboptimal in that new light.
I would be pro some lazy import idea that seemingly has no overlap (e.g. LazyLoader path based), but for this specific proposal there is non-negligible conceptual overlap.
Have you written or used these solutions? Because I have, and I’m enthusiastically in favor of this PEP.
I contest the view that this has only small benefits.
I work for a small team. We can’t afford the large corporations’ “fork CPython” approach.
And I’ve been writing lazy modules since 3.7 added __getattr__ because import time is significant to several applications.
Here’s a classic use case: a web app with an admin CLI which uses the same application code. The service context doesn’t care about startup time, but the CLI context does tremendously.
Existing lazy loading is sufficiently verbose and cognitively expensive that we can only apply it at well defined module boundaries and only where it is absolutely necessary. And it can’t be used with from ... import ... style at all, since that forces eager loading.
It may seem small as a benefit based on your needs, but surely the large amount of community support for the idea of lazy imports – notwithstanding disagreements about how exactly it should be done – speaks to the contrary?
This is extreme. When the thread started, clearly weeks to months of work by multiple people had already gone into a solution. Not to mention years of past proposals in this space. Expecting them to wait years for people to bat around alternatives is simply asking them to let the idea die slowly.
I agree that this has seemed quite fast from the outside perspective. But I’m not surprised, given that this is following a failed PEP and is being driven by people highly familiar with the process, that it’s a fast one with a relatively short comment period.
If people have specific notes about what’s wrong with this change which have not already been noted, or want to voice support for ideas that the PEP authors have rejected, now is an appropriate time. But if you say you’re against it without any more information, that feedback is of limited value not only to the community but also to the SC when they consider accepting, rejecting, or deferring this proposal.
20 Likes
There has been 100 new messages since that post, and you are definitely not the first to say that this PEP feels rushed, but I think that that is indeed not the point.
There have been multiple posts about this PEP not being rushed. There have also been a few posts that t tried to get this back to topic, being PEP 810, not the discussion about its timing.
IMHO, this is not as rushed as people try to say it is, if it even is at all. This topic has been discussed multiple time before, but just discussing how wired the timing is is not really on topic.
And about giving it a few more years, that will only make it harder to make a PEP that makes everyone happy (or at least feel heard). Imo my opinion, it would be smarter to make this now, with the community being interested. I think 400+ messages is more that on a lot of other PEP threads, which usually have ~150 messages (mostly those PEPs I’ve looked at, which have been accepted).
3 Likes
To add to this: the authors actually requested a review period extension almost immediately after posting the PEP (I cannot find the post because all these comments but is there) and the irony is that for the next several days, the thread was dominated by meta-discussion about timing rather than technical feedback on the proposal itself.
5 Likes
Yeah, that’s what I was talking about.
1 Like
Yeah the notion to wait years is just plain ridiculous. Even if you want to consider that, this idea has already been given years. LazyLoader was added in Python 3.5 (2015), and PEP 690 attempted to address this in 2022 and was rejected. The design space has been explored for years and we count with several forks around implementing the same thing (and seems that they all agree on this proposal based on the messages in this thread). Making everyone happy is not the goal and never will be: the goal is finding the best technical solution, and community interest is at its peak right now with 400+ comments, far more engagement than typical PEPs get.
5 Likes
I actually don’t think it’s rushed as a concept since it really builds on PEP 690, addressing what people generally didn’t like about that proposal, keeping much of what they did, and coming up with a something that’s potentially more appealing. PEP 690 was proposed over 3 years ago. So the idea and basic semantics have been around – and discussed – for quite a while now.
15 Likes
That’s always going to be the case. Nobody has a crystal ball, and I think history has shown that until a feature – designed with the best intention, considered with all due diligence, and with the best vision that we have at the time – is in the hands of actual users, we just can’t know if a) we got it exactly right; b) we haven’t painted ourselves in a corner; c) there’s room for building even new wild features on top of it. Think about the evolution of metaclasses, async, decorators, typing, and on and on. We do the best we can with what is in front of us, and we see where it goes. Kind of like life I guess 
15 Likes
Yes many.
From user’s perspective, I am as enthusiastic about this as the next guy. But from POV of design, simply too risky.
What is wrong with up LazyModule?
It is based on mutation, not implicit global variable change and doesn’t suffer from large long term risks.
All it needs is a bit of work and a nice wrapper importlib.lazy_import?
It is to me. I didn’t have time to properly evaluate, discuss it and the fact that it moved so quickly made me change context rather abruptly to make sure I have a say in time. I wonder how many people didn’t have time to zoom out from their own rabbit holes? Yes, the topic is long going, but this is regarding a dump of this specific implementation.
The reason for number of threads is arguable. It might just as well be:
- Very good advertisement and presentation
- Many people of influence involved
- Community is simply excited about lazy imports with the hope for “final end of pain” for this.
And the number of posts has no relevance in evaluation on whether this specific proposal is optimal from long term design perspective.
I know.
Regardless of the history, simply reflecting on the current state of things -
all I am saying is that it needs couple more years, that is all.
I am just warning about much higher than average risk of this being the case for this specific instance. I might be wrong, ofc, that is why I suggest to slow down and not claiming that this is “definitely bad idea”.
I appreciate that, but from my experience I can always set “the level of perfection” that I am aiming for. And the “time to completion” is a function of it. Some things are quick to nail down. And for others, to achieve desirable level of perfection, one needs to be ultimately patient. All I am saying is that this specific case is likely the latter.
I have spent a lot of time thinking about deferred evaluation in general, researched, experimented, implemented quite a lot over past 2-3 years. I would say 15-20% of that time was dedicated to this topic. And my best bet is to hold the horses for a bit longer to minimise the risk.
I appreciate the number of posts, the excitement and convictions.
If everyone is happy then I am in the minority.
I completely appreciate that community consensus is what it is.
But I need to say what I need to say:
“In contrast to the general beauty of Python that I see, this hasn’t yet reached the bar in terms of conclusiveness. Not from POV of high level design.”
4 Likes
Did you have time to talk in PEP 8? No, probably not, but I suppose you still use the principles defined within it. Some people will always say “I didn’t have time…”, but that’s always the case. Someone who joins just after a PEP, he would care about, is accepted would also be angry about it, but again, not everyone can (and perhaps should) be made happy.
Well, if a thread is 2 years old, with 10 replies, does that make it a better PEP discussion than one 2 weeks old with 100s of replies?
No, I think both parts are important. But the time since the first proposal of this has indeed been years too, and this thread builds on top of some rejected PEPs / ideas. If this were one continuous thread, there would likely be 1000s of replies, and years of time put into it.
Imo, this topic has been going on for long enough now, and it is time to actually focus on the PEP proposal again, not it’s timing.
You said your piece, you disagree, and we’ve listened. but your taking a lot of space and making this so much harder to read by jumping between different unrelated concerns (asyncio, typing, timing, deferred evaluation) and even responding to your own posts, which tbh shows (in my opinion) that you havent even understood most of the PEP and the discussion. it makes it really difficult for everyone to follow the actual technical discussion when the same vague concerns keep getting repeated.
Several people have told you your concerns have been adressed and explained why they disagree. Please be respectful to everyone and stop cluttering the discussion with the same broad “this needs more time” and “design risks” complaints that have already been responded to multiple times?
8 Likes
How exactly would the lazy_import function do this if it receives a string as the module name and does not use global variables?
I assume it would need to use the __lazy_import__ function somehow but this is what I get with the reference implementation:
>>> sys = __lazy_import__('sys')
>>> sys
Traceback (most recent call last):
File "<python-input-0>", line 1, in <module>
sys = __lazy_import__('sys')
ImportError: deferred import of 'sys' raised an exception during resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<python-input-1>", line 1, in <module>
sys
TypeError: 'module' object is not subscriptable
I accept things are done and concentrate on things that are pending. And this is pending in both - acceptance and consideration of whether to give it more time.
How many people do you represent?
Understandable.
Everyone is involved in technical details while I did a bit of push for a step back.
Please, don’t get too personal - although you might not like what I say and how I do it, but you need to agree that I talk ideas and decisions, not people.
I think I am taking my fair share of this discussion.
You can count number of my posts in total, compare to posts of others and report results.
In no way I have shown disrespect to anybody (except you now).
Using false accusations to put someone down in matters of design and technical decisions is what is disrespectful.
To put it as simply as possible one last time:
- There is no burning issue
- The matters can be improved in different ways for the time being
- It has a high risk of overlap with long term related endeavours in the same space, while in contrast is very small in scope and is not fit to be a vanguard in this area.
- Why not give it more time?
All the reasoning for above statements are in my previous posts.
1 Like
Set the topic to slow mode per PEP authors’ request.
12 Likes
You may have missed it in this monster of a thread[1], but this was one of my motivations for suggesting that lazy imports be allowed in context managers, see this post. My plan is to write a context manager that works as similar as possible to PEP 810-style lazy imports in Python <3.15 that is essentially nullcontext in Python >3.15, so you’ll be able to do something like:
__lazy_modules__ = ["foo", "bar"]
from backports.lazy_imports import lazy_importer
with lazy_importer(__lazy_modules__):
import foo
import bar
That was the most ergonomic pattern I could find.
One reason I am in favor of getting this in front of the SC sooner rather than later — the design space here has actually been pretty well explored in this thread, it’s just going in circles now because no one has time to read a novel before replying. ↩︎
11 Likes
Thanks for pointing this out! This is just a bug, we’re grabbing the builtins module instead of it’s dictionary and so we’re looking in the wrong spot for the __import__ function to call. It should be fixed with Grab builtins dict from module by DinoV · Pull Request #17 · LazyImportsCabal/cpython · GitHub and I’ll add a test on Monday.
5 Likes
Just for reference a minimal version of this idea (which only works for the restricted cases where LazyLoader works) looks like this:
import importlib.util
import sys
import threading
from contextlib import contextmanager
_thread_local = threading.local()
_original_import = __import__
def _lazy_import_wrapper(name, globals=None, locals=None, fromlist=(), level=0):
lazy_modules = getattr(_thread_local, 'lazy_modules', None)
if lazy_modules is None or level != 0 or fromlist or name not in lazy_modules or name in sys.modules:
return _original_import(name, globals, locals, fromlist, level)
try:
spec = importlib.util.find_spec(name)
if spec is None or spec.loader is None:
return _original_import(name, globals, locals, fromlist, level)
spec.loader = importlib.util.LazyLoader(spec.loader)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)
return module
except Exception:
return _original_import(name, globals, locals, fromlist, level)
if isinstance(__builtins__, dict):
__builtins__['__import__'] = _lazy_import_wrapper
else:
__builtins__.__import__ = _lazy_import_wrapper
@contextmanager
def lazy_importer(lazy_modules):
if sys.version_info >= (3, 15):
yield
return
prev = getattr(_thread_local, 'lazy_modules', None)
_thread_local.lazy_modules = lazy_modules
try:
yield
finally:
_thread_local.lazy_modules = prev
Then you can use it like this:
from lazy_import_backport import lazy_importer
__lazy_modules__ = {"this", "collections"}
with lazy_importer(__lazy_modules__):
import json # This is eager because is not in __lazy_modules__
import this # This is lazy (prints nothing even if normally would)
from collections import namedtuple # This is eager because the from ... import ...
In both <3.15 and under this PEP it will make this lazy (via different mechanisms).
For this idea you can use something like:
import sys
import importlib.util
def lazy_import(name):
if sys.version_info >= (3, 15):
return __lazy_import__(name)
if name in sys.modules:
return sys.modules[name]
spec = importlib.util.find_spec(name)
spec.loader = importlib.util.LazyLoader(spec.loader)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)
return module
Then in both this PEP and <3.15:
>>> import lazy_import
>>> this = lazy_import.lazy_import("this")
>>> this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
<module 'this' from '/home/pablogsal/github/lazy/Lib/this.py'>
10 Likes