This is inspired by @encukou post here: Sys.lazy_modules clarification PEP 810 says sys.laxy_modules is there but it doesn’t really explain why. I’m wondering if we just want to remove it because it is a little wonky as pointed out.
I think the original reason for it to exist was for tooling support. It would allow you to introspect the process and see what modules have been lazily imported but not yet reified. Technically though this can almost completely be reverse engineered - you can look at what has been loaded and look at any imports those modules contain to determine what they would have imported if they used them. There’s also tooling which is being devloped (e.g. GitHub - facebook/Lifeguard: Lifeguard is a static analyzer to detect Lazy Imports incompatibilities and ease the adoption overhead for Lazy Imports in Python · GitHub ) which is showing that static analysis alone can drive tooling for lazy imports.
So at this point it seems like sys.lazy_modules isn’t providing anything useful. If at some point there’s an identified use for it we could always add it back. But taking it away in the future would be much more difficult. So, should we just remove it entirely?
It’s obviously a significant last minute change so @hugovk would need to be on board and we’d need to talk to the steering council for the change to the PEP.
Without this variable, how would you identify currently lazy loaded modules to e.g. trigger their actual initialization or to determine whether your code is still up for surprises entering an important section which could use such modules ?
We already found very late in the adoption process that disabling lazy imports no longer is possible, since not even the stdlib survives such a setting anymore.
If you now remove that last safety net against non-deterministic runtimes, I’m afraid that Python will no longer be the stable platform you’d want to build large applications on.
There has to be a way to say: I want all lazily import modules loaded now and sys.lazy_modules is the only way to make that happen, AFAICT.
It is also needed for debugging applications, since side effects of importing modules may well affect the runtime behavior of other aspects of you application.
I agree there should be a way to resolve all lazy imports but I think that sys.lazy_modules is to some extent unhelpful. The version in RC1 is better than some of the earlier ones but doesn’t necessarily distinguish modules from other names[1] and can contain names that have already been resolved.
The solution I’ve found from the Python side, without needing sys.lazy_modules is to walk the __dict__ of every module in sys.modules, calling getattr on every types.LazyImportType instance found in the dict. Looping this until you don’t find any or all those that you find are known to fail.
As the lazy import... syntax only works at module level you shouldn’t need to look any deeper.
If there’s a better way this could be implemented using internal state details this would be nice, but we don’t have no way of resolving this (although a helper like this would probably be a good thing to have in importlib somewhere).
lazy from X import Y puts X.Y in sys.lazy_modules. ↩︎
Without this variable, how would you identify currently lazy loaded modules to e.g. trigger their actual initialization or to determine whether your code is still up for surprises entering an important section which could use such modules ?
That’s definitely a use-case I hadn’t considered so definitely could be a good reason to keep it around even w/ it’s warts, But does @DavidCEllis solution seem like a reasonable way to trigger that instead?
I could also see other ways this could be provided as well - e.g. we could provide some runtime support for doing explicitly this (as the runtime could track the lazy import objects and then force them all to be reified) - but it’s too late to do that in 3.15.
@DavidCEllis 's approach would probably work, but it’s really a hack and nothing guarantees that it won’t change in a few releases down the road, while sys.lazy_modules is a documented API.
We have to provide people with supported, reliable and readily usable tooling.
I’d prefer to focus on clarifying what sys.lazy_modules really means.
I do think we could do with a helper function provided by the stdlib somewhere like importlib.util to resolve lazy imports. I know pip will probably need something like this as it currently has a plan to block all imports when installing for security reasons. As lazy import proliferate it’s probably going to need to resolve those first.
Sketch implementation
import sys
import types
def get_all_lazy_imports(
*,
exclude_imports: set[tuple[types.ModuleType, str]] | None = None
) -> list[tuple[types.ModuleType, str]]:
if exclude_imports is None:
exclude_imports = set()
return [
(module, name)
for module in sys.modules.values()
for name, obj in module.__dict__.items()
if isinstance(obj, types.LazyImportType)
and (module, name) not in exclude_imports
]
def resolve_all_lazy_imports() -> list[tuple[types.ModuleType, str, Exception]]:
"""
Attempt to resolve all pending lazy imports found in all loaded modules
including those loaded during resolution.
Returns a list of (module, name, exception) tuples for failed imports.
"""
failed_imports = set()
failed_imports_reasons = []
while pending := get_all_lazy_imports(exclude_imports=failed_imports):
for module, name in pending:
try:
getattr(module, name)
except Exception as e:
failed_imports.add((module, name))
failed_imports_reasons.append((module, name, e))
return failed_imports_reasons
My view on this is that it’s always been a red herring for the problem of actually resolving pending imports. It may have had some internal meaning at one point, but the versions of it I’ve seen in the betas don’t seem to be of much practical use.
This is a significant security/performance/UX issue for pip, we must be able to reify all lazy modules before we install a wheel. Walking the __dict__ of every module in sys.modules sounds very expensive.
In the next version of pip we expect pip to crash if an import is triggered once wheel installation starts, this seems likely in a world of lazy modules being reified
I tested it and a significant chunk of the time of resolve_all_lazy_imports on pip (placed after _eagerly_import_modules) appears to be that it finds and triggers the import of _colorize (and to a lesser extent tomllib._re).
There’s about a 1% difference by calling the function if those two modules are already imported. It’s about 5.5% if you include the two imports that are currently lazy that aren’t being triggered. So my view is that the actual imports will probably outweigh the walking.
Edit: Forgot to mention this was doing pip install attrs with attrs already downloaded.
On my machine the reify pass with sys.lazy_modules is about ~0.01ms, whereas walking sys.modules is ~10ms, a ~1000x increase in cost.
10ms cost on every pip install (presumably billions per month) is not insignificant, and given we’re slowly trying to catch up to uv which is often finished installing in about 10ms, this is a pretty dramatic performance penalty.
That’s the optimization with sys.lazy_modules working, the point I am making here is you can leverage it to do less work.
I had another look at walking sys.modules and found an optimization for it as well: Skip stdlib-owned namespaces in the walk · notatallshaw/pip@aaf1c16 · GitHub. Which speeds it up to about 0.8ms on my computer, which is a lot better, but still ~80x slower than with sys.lazy_modules, and an unnecessary nearly 1ms on every pip install is still pretty undesirable, if not quite as dramatic of a performance penalty as my first implementation of this.
I’m not going to argue that it’s ideal. Needing to reify at all isn’t ideal (I am aware of why pip needs to do this currently). I’m just not sure it’s worth committing to what is a messy API over.
The current sys.lazy_modules is a set of names that may or may not actually be module names and may or may not have already been imported. Sometimes things will get removed from it, but not always.
That said, if it’s going to be removed I’d like to know that it’s going to be replaced with something more useful (and more consistent) and not just removed with no plans for replacement.
Yeah, cost is relative to who is paying it. The lack of a good API here is going to be outsourced to all pip users.
For comparison, I would hazard a guess that if a new version of rust came out, and it cost 1ms on every single install command for uv, they would simply not upgrade and would compile only against older versions of rust until it was fixed, pip does not have such a luxury.
I’d also like to add the timeline from my perspective:
And now sys.lazy_modules is apparently being removed, long after feature freeze, and even after the beta period, with the suggestion being pip walk the entire Python object tree (and I guess hope nothing has been injected into the Python environment that has nasty side effects when it is touched)
Personally, but driven by my obligations as a pip maintainer and PSRT member, I would prefer to see lazy modules delayed to Python 3.16 than providing no clean way for pip to secure itself. But I’m very aware that’s unrealistic.
And now sys.lazy_modules is apparently being removed, long after feature freeze, and even after the beta period, with the suggestion being pip walk the entire Python object tree (and I guess hope nothing has been injected into the Python environment that has nasty side effects when it is touched)
This was really just a discussion to see if we had over-designed things with no useful use-case in mind for sys.lazy_modules. It seems like there is definitely a use-case for it even if it’s not the one we had in mind when we designed the PEP (and that’s not surprising as that’s how these things frequently go).
I’m happy to table this discussion and say let’s keep it. It’s always going to be a little bit wonky because in from foo import bar we can’t disambiguate if bar is a module or just a random attribute. But that seems like a tolerable limitation and/or something we can see if anyone comes up with clever ways to address in the future.
Is it too late to consider renaming the object, given that it the contents can’t be known to be modules?
sys.lazily_imported_names seems like a perfectly adequate name for this structure. I think that may also help to clarify whether or not certain things qualify as bugs.
For example:
>>> lazy import itertools.cycle
>>> "itertools.cycle" in sys.lazily_imported_names
True
would be more obviously not-a-bug. It is a lazily imported name.
I’m also okay with it being explicitly marked as “experimental” / “here be dragons” / “behavior may be changed or outright removed and/or replaced in a future Python version”.
I’ll note that I expressed my concerns in that thread that sys.lazy_modules wasn’t fit for this purpose. Which at that point it definitely wasn’t as it missed the lazy typing import entirely.
It’s more that everyone else gets much faster but due to the fact that pip installs into its own environment, it has to disable the feature while installing.
For perspective, against your reify-sys-modules-walk branch:
$ hyperfine -w3 -r50 -L python .venv_312/bin/python,.venv_313/bin/python,.venv_314/bin/python,.venv_315/bin/python --prepare "{python} -m pip uninstall attrs -y" "{python} -m pip install attrs"
Benchmark 1: .venv_312/bin/python -m pip install attrs
Time (mean ± σ): 137.4 ms ± 0.9 ms [User: 109.0 ms, System: 24.8 ms]
Range (min … max): 135.5 ms … 141.3 ms 50 runs
Benchmark 2: .venv_313/bin/python -m pip install attrs
Time (mean ± σ): 139.6 ms ± 0.6 ms [User: 112.7 ms, System: 23.3 ms]
Range (min … max): 137.8 ms … 140.7 ms 50 runs
Benchmark 3: .venv_314/bin/python -m pip install attrs
Time (mean ± σ): 147.4 ms ± 0.5 ms [User: 119.9 ms, System: 24.0 ms]
Range (min … max): 146.3 ms … 148.4 ms 50 runs
Benchmark 4: .venv_315/bin/python -m pip install attrs
Time (mean ± σ): 147.5 ms ± 1.0 ms [User: 119.8 ms, System: 24.3 ms]
Range (min … max): 145.9 ms … 151.8 ms 50 runs
Summary
.venv_312/bin/python -m pip install attrs ran
1.02 ± 0.01 times faster than .venv_313/bin/python -m pip install attrs
1.07 ± 0.01 times faster than .venv_314/bin/python -m pip install attrs
1.07 ± 0.01 times faster than .venv_315/bin/python -m pip install attrs
I’m actually now more curious what the change was in Python 3.14 that made it so much slower.
Some sense of what you might get if you could take advantage of lazy imports.
$ hyperfine -w3 -r50 ".venv_315/bin/python -X lazy_imports=all -m pip list" ".venv_315/bin/python -m pip list"
Benchmark 1: .venv_315/bin/python -X lazy_imports=all -m pip list
Time (mean ± σ): 46.3 ms ± 0.4 ms [User: 38.7 ms, System: 6.7 ms]
Range (min … max): 45.2 ms … 47.1 ms 50 runs
Benchmark 2: .venv_315/bin/python -m pip list
Time (mean ± σ): 73.6 ms ± 0.7 ms [User: 62.6 ms, System: 9.8 ms]
Range (min … max): 72.5 ms … 77.2 ms 50 runs
Summary
.venv_315/bin/python -X lazy_imports=all -m pip list ran
1.59 ± 0.02 times faster than .venv_315/bin/python -m pip list
Others already pointed out tooling that reifies all lazy imports at runtime; I’ll add that this was what inspired me to investigate and write that post :)
It’s useful, even with the extra entries. My docs PR probably reads a bit snarky, sadly; I did not mean it that way.