I want to express my view that it would be best if lazy imports was not to use implicit global variable setting and it should be the last resort.
So yes, either that or delay until everyone got on board to sufficient degree and had a say.
Funny. And false. This person needs to check his facts.
I can understand that.
I am not.
It can be easily made to work either way.
It is. A critique of certain aspects of it.
Yes, this is a possibility, but it doesnât need to be. Easy to have either / both ways.
This is from design perspective. Keyword is modularity.
Agreed. With the general âsomething complex may be better than a more complicated way of doing thingsâ. The rest has no supporting reasoning or evidence.
I have no issues with either: dedicated syntax or the keyword lazy.
And this is what the PEP authors have done, they delayed the PEP submission by a few weeks so thereâs more time for feedback. But more than that is not really required IMO, and looking at the other thread it seems most either agree that itâs sufficient for the community to get a say. But, the community must not agree for a PEP to be submitted and accepted. The authors should try to build consensus, but plenty of PEPs disliked by a large portion of the community have been accepted before and those who disagree just gotta accept that. Same goes for you if PEP 810 gets accepted.
Asking for what is an indefinite pause based on vague ideas (maybe not vague to you but to me) is to me absurd. You really need to sit down and make that proposal for deferred evaluation happen before PEP 810 gets accepted if you want a chance for people to understand the ideas and see if that proposal is worth delaying a severly wanted feature for. If you donât, you gotta accept that the early bird gets the worm, even if it sucks.
You are right, to me this is BAU high level consideration / decision making, taking all perspectives into account.
I donât see it as a âwormâ, it is simply âworkâ.
This is simply a consideration / contribution for the sake of the best possible outcome for Python (whatever it is).
This thread is meant to be a challenge to certain aspects of the proposed PEP. And is mostly targeted at those who know how import machinery works and are able to incorporate implications into their high level perspective.
From users POV, there is only one drawback: âyou might not get from module import attribute pattern. At least for the time beingâ. The rest are benefits.
If from ... import attr is of such high importance that it negates all raised concerns and opportunity costs, then I can respect that.
As long as we are clear on what and why things are happening the way they are and that everyone is aware of it to sufficient degree.
By the way, for the sake of clarity, I can recall things from back then.
Most likely it was someone else who was looking at âdeferred evaluation at the timeâ. I can understand how it could have been a concern at the time given it takes time to digest.
But I wasnât part of those discussions, only read them after.
I got involved quite a bit later.
And after short while it was clear to me that PEP671 and âdeferred evaluationâ are orthogonal. And I have expressed this at least 2 times, you being present in discussions.
I donât think support for from ... import attr negates âall raised concernsâ. I just think that a proposal that doesnât attempt to address this isnât an alternative.
Iâve actually found thereâs a significant benefit performance-wise from lazily importing as much as possible. The cumulative import time of a large number of small modules does add up. The only places where I know I wouldnât use lazy imports would be for things CPython always imports at startup and things Iâm going to use at module level.
Strictly speaking âit is an alternative that omits this part of functionalityâ, but I respect that it is not an alternative to you personally.
In this case, shouldnât community aim for PEP690-like âeverything is lazily imported as much as possibleâ?
If you want as much lazy as possible, what I have presented here has an option to make imports lazy, even if dependencies do them eagerly (both ways can be made available). If you are ok with not having from module import attribute, then this path is able to make bigger portion of imports lazy.
It would need to be optimised though, how many libraries do you think lazy imports should be able to handle smoothly?
If we were starting from scratch, yes. The performance benefits from doing so are significant. However weâre not starting from scratch and there is established code that relies on import side-effects so thereâs going to be compromise somewhere.
Even if I would like to import something lazily, I donât think that should change how a different module imports something unless thatâs explicit.
Also Iâm still not OK with not having from module import attribute 
. I consider (eventually) replacing things like lazy_loader (or ducktools-lazyimporter for me) to be a key purpose of changing anything with regards to lazy imports. A proposal which doesnât cover those use cases wouldnât be a significant improvement over the status quo (it might actually make things worse as people feel there isnât more to be done as their use case is covered).
Thanks for this!
Yes, there are a fair amount of possible variations with this proposal that can be exposed via different functions / arguments:
And there is generally scope to explore possibilities.
Added one more drawback to this. It might not be possible to optimize the work that is done at import definition to the degree that one can just make all imports lazy without consideration.
As I said, that is honest preference and I can not argue with that.
For those that donât know, Iâm the creator of importlib, one of its primary maintainers, and I am the author of importlib.util.LazyLoader.
I disagree with this; I donât see how functions are more explicit than syntax. For instance, linters will have a much harder time working with these functions if they were to adopt some lazy, import-specific rule compared to syntax.
How so?
Thatâs only partially true. While youâre more likely to get a ModuleNotFoundError at the import call in the default case (i.e. finding the module), that doesnât cover anything that occurs when the module is loaded (e.g. syntax errors or runtime errors).
As the author of importlib.util.LazyLoader, I would never claim that it isnât magical; it does some tricky stuff that isnât ânormalâ.
This isnât important to me. Plus PEP 810 still uses importlib for importing, it just delays calling it.
⊠which doesnât exist and has no proposal to consider. That isnât an argument that has ever held since I got involved in Pythonâs development over 23 years ago. And considering how long people have bandied about the idea of deferred evaluation, itâs not something to hold up any part of Python for.
As the person who has merged PRs to fix things with importlib.util.LazyLoader, I can tell you there have definitely been issues.
Once again, I disagree here. The import system isnât magical, itâs complex. What importlib.util.LazyLoader does is way more magical.
If thatâs true then youâre writing a lot more code than I am as a huge amount of Python is just syntactic sugar. Personally, I like the syntax we have for all the benefits of increased productivity.
As you have pointed out, itâs not exactly equivalent to what is being proposed in PEP 810, such as fully delayed imports, working on objects being imported, etc. Plus the tooling benefits of it being syntax is not the same as if it were all just functions.
As the person âwho know[s] how [the] import machinery worksâ better than almost anyone and has thought about lazy importing from a âhigh level perspectiveâ and its âimplicationsâ for years, I am emphatically stating that I prefer PEP 810 over this solution.
This is regarding underlying mechanism, not âsyntax vs functionsâ.
One can do lazy import, some object is added to sys.modules. Then there is possibility to check it and decide what to do with it depending on what was found. Simply less things hidden and more things inspectable, providing space to build upon, new ideas emerge, etc.
Indeed.
Agreed. It is magic, but more like grey - much less is hidden. In comparison to everything. Where nothing is inspectable, everything is in limbo state without any error checks and it is imposed on one of the most basic operations - to say that it is âunusualâ is a slight understatement.
Eventually. But it is not making use of it for lazy features. It is just wrapping its call as it could any other operation. As I said, if this is the road, then why limit this to imports? Letâs just have full blown âdeferred evaluationâ based on this, given we allow this methodology to creep in and be part of Python.
And one of the main reasons why it didnât go through is because there is reluctance to introduce mechanisms as PEP810 is relying on and strife to find better ones.
Have they been fixed by now?
That is fair, LazyLoader adds magic, but I was referring to the fact that it simply gets added to sys.modules same way as standard import, but delays certain operations. From certain perspective, this as straightforward and least magical way that this can be approached. But I can appreciate another one as well, where it can be said that simply wrapping operation in independent delayed mechanism is simpler. But if we take this one, then my issue is that given this is independent mechanism of such power, then why not introduce lazy everything? It will be harder to build upon partial feature than to premeditate the whole thing. Few year later its going to be âif we were to do this from scratch, then yeah, but nowâŠâ
I am just ok with functions - making use of LazyLoader opens possibilities to different variations of it and I might slightly prefer freedom in customization to dedicated syntax. But I am not against syntax, if it is one best way that covers all that is needed, then why not. I am just unsure whether it will for me. After looking into possibilities of LazyLoader I can see myself building on top of it, extracting more and more convenience and value over time.
Canât argue with the first part.
Regarding the second part, the way proposed PEP works, feels more along the lines of generic âdeferred evaluationâ than âlazy importsâ.
But maybe you have thought about it from this POV as well, I simply can see how starting from the point of âlazy importsâ, this perspective can be overlooked, so just making sure.
This is not a âsolutionâ, just something to show potential in this direction to make an argument.
Who knows what the final version could look like.
Whilst I see where youâre coming from regarding PEP 810âs use of magical lazy evaluation while generic lazy evaluation is repeatedly rejected for similar magic, I donât see that the concerns made about the latter really apply to the former.
With generic laziness, every function already in existence could potentially start receiving lazy objects, introducing the potential for inconsistency if the object mutates between being passed to the function and being de-lazified by the function. This also introduced the need to special case lazy objects in isinstance(arg, sometype) and arg is someobject so that arg is evaluated and the expressions donât incorrectly return false. With PEP 810, in the unlikely event of someone running lazy import numpy and then passing numpy to a function, it just reifies numpy before the function call and weâre back onto normal footing â the function doesnât know nor need to know that one of its arguments used to be a voodoo lazy object.
Generic laziness also wanted all kinds of ways to (sometimes conditionally) handle a lazy object without evaluating it so that it could cover passing expensive-to-calculate arguments to functions that may not use them. This led to rules that most of us couldnât follow about when a lazy object really gets evaluated. PEP 810 on the other hand is content with a comparitively simple if you touch it, itâs reified â even at the expense of otherwise possibly being able to support something like def foo(default=lazymodule.something): or class Foo(lazymodule.Base): without upfront reification.
To the best of my knowledge, yes, but so would bugs related to PEP 810 so I donât see why this is even an important point. If youâre trying to say importlib.util.LazyLoader is somehow better or more stable because itâs been used more, I can guarantee that wonât hold for long if PEP 810 lands.
I disagree, to the extent that I believe arguing from the perspective of some mythical deferred evaluation future is weakening your position of arguing against PEP 810.
If this isnât a proposed solution then I personally donât consider it as a legitimate counter-proposal to PEP 810 which is a solution and is ready to go. Python development has always preferred to be pragmatic and prefer concrete solutions to hypothetical ideas when thatâs the two options.
As such, Iâm choosing to view what youâre proposing as âfunctions over syntaxâ, and I choose syntax. Framing this in any other way makes me consider it too hypothetical to consider viable.
Yes, this is the main obstacle.
Yes, and given above can not be solved, this might be the only path for more general âdeferred evaluationâ. Which is known, I just decided to explore all possible alternatives and giving it as much time as possible before going this direction.
If this is so, then it might be worth leaving space for it, by making things more generic in PEP810.
No, this was was my response to criticism, not original argument against PEP810.
I need more time to digest it and connect the dots. Yes, given the momentum of it, there simply isnât enough time to do it properly. Maybe this is not the issue for others, but given I am slowly working on this, this is the situation I am in.
This is a request to hold the horses for a bit. Give it couple more months for those who are keen on doing a bit of work, there are 2 directions things to digest:
From strictly procedural POV, you are completely correct. This is the procedure, PEP810 has nailed down all the necessary stuff and what I did here in a day doesnât measure up to extensive and profound work of several expert people. Obviously.
But I am coming from a bit different POV: âwe all want what is best for Python, could you please slow down a bit, because I need a bit more time to double check things from my POV? Intuitively something doesnât sit well, I can not yet pinpoint what exactly it is and the time of exposure that you allowed is not sufficient to do it.â
So I just did this to have some observable and concrete content to back up my claim, that there might be something. This is regarding (1) above, regarding (2), I just need to dig into it.
Because, is there any chance that you would take simple âcould you please wait for couple more months?â without any supplementary material / evidence? Given that I am an odd one out without any support here.
This was false. I take it back.
FYI, and Iâm not sure if you already include this, any lazy import proposal must have some option to force all imports as eager, so Python packaging installers can install wheels securely.
I explain the scenario for pip in detail here: PEP 810: Clarify security implications by notatallshaw · Pull Request #4660 · python/peps · GitHub
To be clear, it isnât me thatâs controlling the timeframe.
If I was still on the SC my answer would be ânoâ. The only way I would see the SC even considering doing that is if someone who has time and again shown their gut feeling was accurate over the years along with proven follow-through was making that sort of request because youâre asking the entire community to trust you based on vague feelings and no plans on how to come to a resolution. And to be frank, Iâm not sure I could even get the SC to hold off more than a month if I made a similar request.
Supporting @brettcannon here, one of the things to consider is that we (in the narrow âimport system maintainers and contributorsâ sense) have had LazyLoader in the stdlib for quite some time, along with 3rd party function based lazy loaders before that, and never promoted any of them to syntactic sugar because they failed the âtoo much pain for not enough expressive gainâ test. Lazy loading is inherently confusing (that was a big part of the proposal to make imports lazy by default getting rejected), and existing lazy import semantics can be expressed as functions, so thereâs no huge benefit to giving them dedicated syntax (aside from giving static code analysers a hint as to what is going on).
What makes PEP 810 so interesting is the fact that it not only proposes dedicated syntax, but also takes advantage of the compiler being aware of what is going on in order to offer semantics that mitigate some of the annoyances associated with using lazy imports in a way that function based proposals canât reasonably provide.
Importantly, existing function based lazy imports continue to work as they do today, so weâre not losing any capabilities, and weâd potentially be reducing the barriers keeping folks from making effective use of lazy imports to reduce the runtime overhead of initialising irrelevant modules.
Note that PEP 810 has been accepted: PEP 810: Explicit lazy imports - #466 by barry
Thank you to everyone who participated here.
@brettcannon, could you please close this?
This serves no purpose anymore.