I don’t, it’s hand-waving It matches my experience with many Python projects, but YMMV, of course. I would note that importing modules is also usually (again, in my experience) without side effects, you just use a type/function from it later, so I don’t see that being different from importing a function directly.
I think that’s fine on its own. I just wondered if the potential community aspect of people spamming PRs to random projects to “optimize” them, or Python developers proactively using lazy import everywhere, was considered. It’s something that we have been thinking about a lot for new opt-in perf-oriented features in Rust, so I thought I’d mention it here, as I didn’t see it being discussed.
But ignoring all of that, making eager the opt-in is a bit hostile towards beginners where reasoning about lazy imports is harder than lazy imports. In that spirit, I don’t think asking us to type 5 more characters when the perf matters is a big deal.
I think that the reasoning is complicated in both situations. It’s not trivial to benchmark the effect of making a single import in a Python program lazy, it might be difficult to understand the performance characteristics, which could lead to the “premature lazy optimization” behavior I described above.
Features normally have a set of trade-offs. If developers knew that this feature can make their code 5x slower or 5x faster, it’s clear that they would have to think hard about its usage, and not use it willy-nilly everywhere. But if the story is more like “this either has essentially no performance downside or it can make your code faster and take less memory” (which, as far as I understand, is the case), then that might lead to a world where the optimal behavior is to simply use a lazy import everywhere.
Maybe @thomas and I will still need to duke it out, but here’s my suggestion for addition to the filter section:
The filter function is called at the point of execution of the lazy import or lazy from import statement, not at the point of reification. The filter function may be called concurrently.
There’s still the performance impact of the filter function to address. I’m assuming that if there is no filter function (likely the common case), there’s a fast short-circuit. If so, it begs the question of how to unset the filter function, and I assume that calling set_lazy_imports_filter(None) will do that. It might be worth mentioning in the PEP, but it’s definitely worth mentioning in any future documentation.
It’s a “secret” type instantiated internally by the interpreter/runtime - to me, that says types. Save importlib for things that are part of an API that users may need to use/implement directly.
Oh, and if you’re going to call it a “lazy object type” and not a “lazily imported module type”, then you’d better think about how it will one day be extended to apply to any object at any time (or just name it something specific to imports)
Once concern I have, andpip doesn’t have this problem, since it’s usually not used as a library, is library authors using sys.set_lazy_imports(“disabled”) inside their module and then not re-enabling it. It’s also unclear to me if they can actually re-enable it in all use cases, since there’s not really anywhere to put it, if they want it disabled for all calls to their functions.
This will diminish the lazy-import performance gains for the rest of the application (even more so if the user set it to enabled).
It seems destructive to me to keep this in it’s current state - I’m worried some library down the chain will do this and it’ll make the feature much less useful to a lot of people, without them even knowing. I tried to come up with a solution, but I couldn’t think of one that works well.
I read the entire thread and hasn’t seen anyone mention this, so please let me know if I’m missing something here and it’s not actually an issue
Library authors should not disable lazy imports inside their libraries. Please see the previous discussion regarding the global flag to disable and the PEP updates but short story: it’s an advanced feature for end users (here pip is an end user because its exposed as a program on its own) and you need to be aware of the consequences. See also this section for some related extra context
types has the advantage of having the “lazily imported module type” living together with types.ModuleType.
I like the general idea of lazy import, and as many posters noted the purported benefits of lazy import is quite large and people including me will try to use it everywhere. I get the impression thata most modules, except in a few cases, would support being lazily imported out-of-the-box.
Still, users will want find out if a module can be lazily imported. All is good if the library author explicitly states that a module can be lazily imported (or not). If that’s not the case, as with legacy modules, the PEP states
Test that side-effect timing changes don’t break functionality.
and
When in doubt, test lazy imports with your specific use cases.
Testing is obviously important but nonetheless certain forms of breakages are things getting subtly wrong timings and thus hard to test, for example reading a config file at a wrong time, initializing an event loop or database connection from a wrong thread. These are all classic heavyweight operations that will certainly benefit from lazy import when someone just wants --help from the command line. But what should users and authors do about them? How can library authors make their code compatible with lazy import? Will there be a HOWTO documentation on lazy import patterns?
Yeah but importlib is where the rest of the features are, so I think that’s not an direct winning argument.
As I understand the PEP, library authors don’t need to do anything to “support” lazy imports in the default mode. When someone writes lazy import your_library, that’s the importing code’s decision and responsibility not yours as the library author. The important part is that your library code doesn’t change at all internally by someone lazy importing it. Whether someone imports your library with import your_library or lazy import your_library, your module executes exactly the same way and the only difference is when it executes (at import time vs. first use). This timing shift is the importer’s concern to manage, not yours.
The guidance in “How to Teach This” (avoiding import-time side effects, using explicit initialization functions, etc.) is primarily for those who want to use the advanced global mode out of the box. Even then, as someone noted before library maintainers can reasonably decide not to support global mode.
The PEP’s “How to Teach This” section already provides concrete points for supporting the global mode. But I assume this will translate into a better “How to” document about it. Indeed: if we sneakily spy on this in progress PR by the authors you can see that they indeed want to do exactly that.
Since top-level module code is run when a module is imported, the “import convention” should be considered an entry point and thus part of a library’s public API. It is as much a responsibility of the library author to document the sensitivity of their module-level code (which gets run on import) to external state and thus requires additional care on import.
This thread contains direct feedback from maintainers of several projects such as attrs, FastAPI, Scientific Python, stamina, PySide, and other major libraries stating they would immediately benefit from this feature. The PEP documents that several major companies Meta, Google, HRT, and Bloomberg already use similar implementations in production at scale with measured performance gains. Some users have shared resources in articles, bug tracker items and other links.
Python doesn’t require real-world adoption before accepting syntax changes. For example pattern matching was accepted without requiring projects to prove adoption. The evidence here (production use + direct enthusiasm from library maintainers in this discussion) exceeds typical standards.
The “burden” you mention isn’t being shifted. I think your requests here are unfair
I think you’re misunderstanding how this works. As a library author, you don’t write your code any differently. Your module executes identically whether someone uses import your_library or lazy import your_library.
The lazy keyword affects the importer, not the library being imported. When someone chooses to lazy import your library, they’re accepting the timing change. That’s their responsibility, not yours.
Unless I am missing something you don’t need to document anything special or make your code “compatible with lazy imports.” (unless you want it to work under the global flag). With the normal mode your library’s behavior is unchanged.
Thus: if your module has special requirements about when it can be imported, it’s already incumbent on you to clearly document these to consuming code. The only thing that would change is to include consideration of lazy imports into that documentation.
Switching to a fallback/alternative upon a failed import is a rather common pattern and it’d be somewhat of a bummer if one has to revert to using eager import for handling ImportError.
Maybe there can be an extended syntax for registering a block of exception handler that’s executed in the same frame upon ImportError when the name is reified:
try lazy from fastfoo import bar except: # implicitly handles ImportError only
from oldfoo import bar
# or: bar = more_compatible_bar
# or: raise RuntimeError("Friendlier error message.")
if action == "print":
print(bar())
elif action == "errorcode":
sys.exit(bar())
else:
print("Usage: foo <print | errorcode>")
so that there’ll be no need to handle ImportError in multiple places downstream where bar may be first used.
Is the following a correct rephrasing ?
“”"
Reification imports the module in the same way as it would be done eagerly at reification time, employing the current state of the import system (i.e. of sys.path, sys.meta_path, sys.path_hooks, __import__, importlib, sys.modules and PYTHONPATH).
“”"
It matches my experience with many Python projects, but YMMV, of course. I would note that importing modules is also usually (again, in my experience) without side effects, you just use a type/function from it later, so I don’t see that being different from importing a function directly.
