How about checking sys.modules? You could do that in a subprocess as part of your unit tests.
>>> lazy import email
>>> import sys
>>> sys.modules['email']
Traceback (most recent call last):
File "<console>", line 1, in <module>
KeyError: 'email'
>>> email
<module 'email' from '/lib/python315.zip/email/__init__.pyc'>
>>> sys.modules['email']
<module 'email' from '/lib/python315.zip/email/__init__.pyc'>
I don’t think there’s a good way to do this automatically on behalf of users, because it requires knowing whether or not their use intended to import.
I maintain different internal tools at work, some where we eagerly import things, including intentionally triggering lazy loaders in scipy, because it’s part of a long lived internal api, so we’d rather pay a predictable extra cost on startup to not have the first api request that hits a lazy code path be slower, but others where we want everything not used to not be imported (internal, frequently used cli tools. We don’t want our nondev users of these tools feeling like these are sluggish to respond and have to wait for someone else to fix it for them)
In terms of ensuring certain things are lazy or eager, once you define the behavior that’s desirable for your use case, you can write tests for it so long as it’s possible to introspect the lazy objects. The pep authors have made changes that address it being possible to check if you have the actual object or the deferal of the object, so as long as that remains the case, after whatever changes are required during implementation and acceptance, my answer to this would be to write a test for this if you have to care.
I’d be mildly in favor of a function in importlib that the only purpose of was to give a yes or no on if a module has been eagerly imported or not yet if it’s undesirable to have users poking at this themselves (it might be prudent to allow the implementation details to not be guaranteed on this in case issues arise later)
If the lazy module type gets exposed in types then I would expect an isinstance() check to be enough to determine if a module has been reified or not.
Thank you for your patience and civility.
I could have done a bit better on my end.
Don’t think I am stopping this thread.
In my opinion, what I said legitimately belongs here.
In any case, I have opened Alternative path for explicit lazy imports with complete lay out.
If anyone wants to discuss anything that I have said in this thread, please reply there, so not to take any more space here.
Perhaps the slow-mode (8h per post) can be removed now that there is a different threat, as to allow the natural discussion of PEP 810 (not other ideas) to continue?
Done, let’s see how it goes.
Freudian slip?
I have a couple questions about what should be suggested best practice for library authors. First, if I want to add lazy import functionality to my library but still support old versions of Python, how could I do it? There is no __future__ import and even if I put the lazy import inside a if sys.version > (X, Y): block, I would guess you still get a syntax error.
It would be nice if we could have a recommended way to do this and it’s not too clunky. Or, do we think using global lazy import control is sufficient? To me, it seems better that libraries are able to opt-in to the feature yet still be usable by old Python versions. If we had importlib.lazy_import_module() then we could use that.
A related question would be what is our advice about making a library work with -X lazy_imports=<mode> set to either all or none. My gut feeling is it would be best practice if libraries work with either mode. Ideally you would have CI tests that run tests in both modes. However, one thing that seems likely to trip that up is circular imports. In my experience, a lazy import is a convenient way to break the circularity. However, it would break if you set the global mode to none.
This might be solvable with another function, e.g. have both importlib.lazy_import_module() and importlib.maybe_lazy_import_module(). If I’m breaking circular imports, I can use the function where laziness doesn’t get disabled by the global mode.
That’s the idea behind the __lazy_modules__ dunder. If you want to have lazy imports in 3.15 and eager in older versions, you can just use the syntax of regular imports and add the modules you want to be lazy to the list. Older versions will just ignore the variable while 3.15 recognises it as enabling lazy imports without any new syntax.
Regarding working with the global flags: I don’t think the intent is that libraries work with the flags set arbitrarily. The idea behind the flags is that if you have some tightly controlled setup where you can somehow guarantee that everything works lazily/eagerly then you can use them, but in general you don’t touch them. There are plenty of perfectly legitimate coding practices that break if you force the import to be lazy or eager.
I don’t think we need functions that differentiate between “lazy but only if the global flag isn’t set” abd “actually always lazy”. If you’re using the flag you’re intentionally changing the semantics of what your libraries want. It might end up working fine in your use case, but it shouldn’t be encouraged to defensively code around that.
Ah, I see. It’s a long PEP, to be fair. 
That seems like a good solution.
The __lazy__ attribute could also fit this, assuming paths are resolved at lazy initialization eagerly, a module could have a __lazy__.py file that is eagerly executed, thus providing mitigability of eager checks and lazy heavy ops.
Also this
could be
assert hasattr(mod, '__lazy__')
They are not. See “Making lazy imports find the module without loading it” under the Rejected Ideas section for the detailed reasoning.
Just a short idea concerning library support of lazy imports.
If a library maintainer is aware of lazy imports and knows that he does not support them, is there an API planned to detect that the import was done lazily, so that he can issue a warning or rise an exception?
Something like
if is_imported_lazily():
raise RuntimeError("Lazy import not supported")
I’ll quote my earlier post on this topic (which, to be fair, was almost 300 posts ago!)
And if you’re worried about users turning on the “global lazy imports” option, the pep makes clear this is a “you broke it, you fix it” problem and not something library authors need to concern themselves with supporting.
(And the fact that you can check the type of imported modules without reifying gives you ways to do runtime checking of import types if you’re really paranoid.)
And the chance of causing unexpected problems when attempting to enforce those requirements programmatically is high enough to make such enforcement a bad idea, IMHO. Like how calling sys.exit(1) when user code imports an “internal-use only module” that’s not meant to be imported directly may seem like a good idea, but it crashes pydoc -k scans.
Dear PEP 810 authors. The Steering Council is happy to unanimously[1] accept “PEP 810, Explicit lazy imports”. Congratulations! We appreciate the way you were able to build on and improve the previously discussed (and rejected) attempt at lazy imports as proposed in PEP 690.
We have recommendations about some of the PEP’s details, a few suggestions for filling a couple of small gaps, and we have made decisions on the alternatives that you’ve left to the SC, all of which I’ll outline below. If you have any questions, please do reach out to the SC for clarification, either here, on the SC tracker, or in office hours.
Use lazy as the keyword. We debated many of the given alternatives (and some we came up with ourselves), and ultimately agreed with the PEP’s choice of the lazy keyword. The closest challenger was defer, but once we tried to use that in all the places where the term is visible, we ultimately didn’t think it was as good an overall fit. The same was true with all the other alternative keywords we could come up with, so… lazy it is!
What about from foo lazy import bar? Nope! We like that in both module imports and from-imports that the lazy keyword is the first thing on the line. It helps to visually recognize lazy imports of both varieties.
Leveraging a subclass of dict. We don’t see a need for this complicated alternative; please add this to the rejected ideas.
Allowing ’*’ in __lazy_modules__. We agree with the rationale for rejecting this idea; it can always be added later if needed.
One thing that the PEP does not mention is .pth files, which the site.py module processes, and which has some special handling for lines that begin with the string 'import' followed by a space or tab. It doesn’t make much sense for .pth files to support lazy imports, so we suggest that the PEP explicitly says that this special handling in .pth files will not be adapted to handle lazy imports.
There currently is no way to get the active filter mode, so please add a sys.get_lazy_imports() function. Also, do you think appending _mode to their names makes the purpose of these functions clearer? We leave that up to the PEP authors.
The PEP should be explicit about the precedence order between the different ways to set the mode, i.e. $PYTHON_LAZY_IMPORTS=<mode>, -X lazy_imports=<mode>, and sys.set_lazy_imports(). In all expectation, it will follow the same precedence order as other similar settings, but the PEP should be explicit.
We agree that the PEP should take no position on any style recommendations for sorting lazy imports. While we generally like the idea of grouping lazy imports together, let’s leave that up to the linters and auto-formatters to decide the details.
That should just about cover it. Again, thank you for your work on this, as it’s been a feature so many in the Python community have wanted for so long. Given the earlier attempts and existing workarounds, we think this strikes exactly the right balance.
-Barry, on behalf of the Python Steering Council
4 votes, as Pablo cannot vote ↩︎
Congrats! Looking forward to using this feature!
After switching most of my attention to this several days ago, I have covered the essentials in time.
And I agree, this will inevitably bring substantial benefit to most users.
While strongly supportive of lazy imports, I recommend two key changes to ensure smooth adoption: (1) add eager import syntax, and (2) enable minimal error checking by default.
I am still uncomfortable with the pace of this.
It has been said in relation to this: “The biggest change to Python in a decade is coming.”
I would add “with the least proportional community exposure time”.
E.g. PEP572 seems to have had ~4-5 months of solid exposure.
Concerns regarding overlap with more general “deferred eval” eliminated.
In line with Alternative path for explicit lazy imports - #32 by bwoodsend.
In the remote case (remote, because inability to pass deferrals into functions would unlikely cut it) that this type of deferred eval is wanted in the future, this doesn’t obstruct it.
Not storing placeholders / proxies in sys.modules makes this orthogonal with LazyLoader.
As accurately pointed out by Alternative path for explicit lazy imports - #38 by ncoghlan.
I think this could use some polish to be better suited for adoption by average use case:
eager import“we don’t want to encourage use of the global flags”
I think this is the biggest value it brings - making large portion (if not all) lazy.
If only few sparse lazy imports are needed, then LazyLoader (or one of other existent solutions) is sufficient.
The very reason why this is worthwhile is the ability to make most of imports lazy.
And the concept itself seems to be taken from places where this was the primary concern.
Thus, scenario:
10 independent modules that all import “single side-effect module”.
(Largely in line with flat structure of Python’s standard library.)
No natural place to add filters.
importlib.import_module does the trick, but then it needs to be imported in all 10 modules.
And given that this had a big weight of it being syntax-based, eager import does seem like a very convenient option. I think this would cover common cases for most of users without needing to add any filters and allow minimum effort switch to global lazy imports, which delivers maximum benefits.
This was the first thing that I wanted to do when trying to adapt to this.
And the fact that I needed to either use filters (think about where to put them (if there is a good place at all) and how to use them) or import importlib everywhere and resort to functional imports alongside the syntactic ones did not feel smooth.
ModuleNotFound errorThere were 3 reasons indicated for not doing minimal error checks:
While I agree that “no error checks at all” is a good option to have, I don’t think it should be default.
This does not represent average use case.
I propose to have a flag to “opt out from minimal checks”.
It is not about error checks, but about keeping the default as much in-line with standard imports as possible for as smooth switch as possible for average use case.
I tried adapting it to my libraries and the biggest change that is needed to adapt is to replace all top level module logic on detecting what is installed and what is not.
So, given that:
a) I know that I have not made any syntactic mistakes
b) I know that all is configured properly
The pain point of adoption is:
try:
import compression.gzip as gzip
except ModuleNotFoundError:
import gzip
Alternative would be to provide a builtin or utility placed conveniently somewhere else:
if module_locatable('compression.gzip'):
...
However, I don’t think such sacrifice is very optimal given that the most common case is locally stored imports.
That is true, but that does not mean that there shouldn’t be an attempt to align the behaviour to the maximum possible degree for the sake of familiar transition.
It is an import statement, if no checks are done, then it is an import wrapped in independent delay mechanism and its functioning has nothing to do with import, but is only a generic wrapper for any operation, while __import__ just happens to be the one at hand.
I don’t think “majority opt-in” for this case represents optimal benefits for the community.
If this is tailored for the use case of “remotely stored imports”, then there needs to be evidence of the frequency and significance of this. Otherwise, I think it is safe to assume that the average use case is “local imports”.
Apart from the above, this addition, IMO, is close to the best that can be done (which is as it should be, given that it has already been tested in other places and the number of experts behind this):
from ... import ... is useful. Although not implementing this would reduce major portion of complexity, but I think the complexity is more than justified given manifestos of its desirability and the fact that complexity is not that big in relation to complexity that already exists in places of implementation.ns pass +core.py
------------------------
3.13 : 190 +150
3.14 : 190 +130
lazy : 20 +120
lazy=on : 20 + 70
where core.py is common dependencies of 90% of my stuff.
Thanks for your engagement with this feature @dg-pb. Speaking for myself, both as a member of the SC and as the sponsor for the previous PEP 690 (which was rejected while I was not on the SC), I personally felt like PEP 810 took the best ideas of 690 and addressed the biggest negative sentiment leading to 690’s rejection. As I’ve mentioned before, I don’t feel like accepting 810 is in any way hasty, if you take the totality of “the lazy import” discussions over the years.
I voted to accept this PEP now because I think the proposal was very thorough, with a large benefit[1].
We’re also early enough in the 3.15 development cycle to be able to get this in the hands of testers and developers. While the PEP authors and other very early adopters have already been running this feature on their own code bases, we’ll soon get the chance to bring it to a wider audience, once the implementation lands in the main repo. I encourage people to put it through its paces, discover the rough edges, and provide feedback to make the implementation rock solid when 3.15 comes out late next year. Nothing beats real-world experience.
I don’t personally think this is necessary, and don’t really want to see lots of eager sprinkled throughout my code, but if said real-world experience – hopefully before 3.15 beta 1 feature freeze – shows that this is really useful, there’s still time to add it, but now with much more concrete justification.
There may indeed be some utility in eagerly performing the finding of modules separate from the loading of them. You mention an important data point with your experience in adapting your libraries to lazy loading. It could be that eagerly-found-lazily-loaded is a useful mode for globally testing code, and I think if that’s the case, it too could be added, and again, there’s time to figure this out while 3.15 is in early alpha development.
perhaps obvious, since I’ve been a fan of the concept of lazy loading for many years, and with my experience as a co-maintainer at times of importlib ↩︎
Thank you for the reply, I hope my raised points will be considered more closely in stages to come.
Maybe, then, for the next time, it would be good to consider putting some earlier “alpha” version for community to see a bit earlier.
Which would not impact the decision timelines.