PEP 810: Explicit lazy imports

PEPs
268

Notice the hypothetical __raw_dict__ only makes sense across module boundaries when inspecting module objects:

# x.py

lazy import a
lazy import b

def foo(): ...

# y.py

import x

# If we do here x.__dict__ then both x.a and x.b will be reified so we
# need some way to know that they are there but without triggering the
# reification

print(f"names in x are: {x.__raw_dict__.keys()}")

We are not closed to this if people agree that this is the best option. It means we are technically exposing lazy objects a bit more, also outside the module. Apart from the options highlighted in the PEP we didn’t went with this option initially so is also a bit easier to activate global mode, but I am happy to reevaluate it as its another option to the __raw_dict__ conundrum. I think @brittanyrey may have more info on how much this matters in practice in the wild.

Notice the get() method only happens if you somehow stumped into a proxy object via globals() the new hypothetical __raw_dict__ or __dict__ if we use your proposal.

>>> lazy import exampl
>>> print(globals()['example'])
<lazy_import 'example'>
>>> print(globals()['example'].get())
<module 'example' from '/Users/pgalindo3/github/python/main/example.py'>

Also note that any attempt to use will cause it to reify so its really hard to ever see a lazy object:

# lel.py
print("FORTRAN 77 ROCKS")

def foo(): ...

then:

>>> lazy import lel
>>> x = globals()['lel']
>>> print(x)
FORTRAN 77 ROCKS
<module 'lel' from '/Users/pgalindo3/github/python/main/lel.py'>

or

>>> lazy from lel import foo
>>>
>>> x = globals()['foo']
>>> def print_type(x):
...     print(type(x))
...
>>> print_type(x)
FORTRAN 77 ROCKS
<class 'function'>

Perhaps this can motivate why trying to reify on module.__dict__ is worth doing and exposing __raw_dict__ or some free function that does that: it “hides” the lazy objects more. Not that they are a problem if they appear (that’s fine!) but is nice that is rare for them to show up.

2 Likes
269

The idea I have under the attribute __lazy__ is that it explicitly gathers everything that can be called on the proxy without reification. It feels really visual in the code.
Thus maybe a natural way for the raw dict will become mymodule.__lazy__.__dict__.

1 Like
271

I think I get what you say but just to be absolutely clear: the __lazy__ attribute must then be on the module object not on the proxy. Note that this is because __raw_dict__ would be an attribute on the module object not on any proxy.

1 Like
272

Thanks for the reply!
I hope my thoughts are useful; I won’t mind if you let practicality beat purity.

I now see more clearly where this clashes with my mental model: when I type __dict__, I’m asking for the internals.
Currently, __dict__ is the way to get an object’s raw namespace – uncooked by descriptors or inheritance. Adding __raw_dict__ layer below that[1] means I’ll need to remember what’s cooked at which layer.

  1. regardless of whether it’s exposed as __raw_dict__ or globals() ↩︎

14 Likes
273

I agree that module.__dict__ is supposed to give us the module’s raw namespace to begin with, but we should also recommend a canonical way of obtaining the “cooked” namespace of a module going forward. Should vars(module) be the function to return a cooked namespace then? Or should there be a new helper function?

2 Likes
274

Yeah, it seems vars() would fit the bill; decoupling vars() from __dict__ could be an explicit change in the PEP. Free builtins generally do cook the things they access.

That’s another option: after reifying the whole module you can use __dict__ (or __raw_dict__, or globals() in the current PEP), and getting that would stay O(1). [edit: It would also stay practically infallible].

2 Likes
275

Great PEP.

About the from X lazy import Y syntax alternative: If I see

from A.B.C.D lazy import E

then I expect the resolution of A.B.C.D to be eager. After all, Python is generally left-to-right, so I don’t expect anything to the left of the lazy soft keyword to be lazy.

IMHO the PEP gets it right, and even if there weren’t a syntactical ambiguity, I would still prefer lazy to go first.

10 Likes
276

Since the main reason to allow them is a backwards compatibility shim (see this post and this one), it actually is important that this be allowed from day one if it’s going to be allowed.

7 Likes
277

Is a backport actually going to have the same behavior as a PEP 810 lazy import? I thought part of the motivation for this PEP is that the behavior can’t be implemented the same way by a package that modifies the import machinery[1].

A backport that has subtly different behavior sounds like a trap.

  1. if it can, why would we need new syntax? ↩︎

2 Likes
278

They syntax has no relation to why this needs to be build into the language.

The true “innovation” in this PEP isn’t the lazy import system itself, but the proxy object that has unique interactions upon being looked at. [1]

That part can’t be back-ported. But other things can. Specifically you can make a pure-python proxy object that just forwards attribute accesses and only proxy modules. This is a change in behavior, and most notably, from imports aren’t going to be lazy.

With the idea that lazy imports are only for performance improvements, this shouldn’t be an issue. The performance behavior/import order differs between python 3.13 and 3.14, but that can be fine

  1. I am honestly surprised there haven’t been complains about this. Previous discussions for proxy objects always had ideological opponents. ↩︎

2 Likes
279

I was trying to understand what a “backport” can accomplish that the other implementations can’t do. It’s not that lazy imports are currently impossible–there are multiple ways to do it.

This was in the context of the “should lazy imports be allowed in with statements” discussion. One argument for allowing it was that it enables a particular style of backport. It doesn’t seem to me that such a backport would be more useful than using any other lazy import system. Either way, you’re using it until PEP 810 is available for all the versions you support, and then I imagine you’d refactor to use the new syntax and drop a dependency.

2 Likes
280

For many backports and backwards-compatibility shims, the goal is for the semantics to work as closely as possible as you need, and you just keep adding epicycles to your shims until you get it working, and then test as well as you can on the entire supported matrix.

You don’t always need to have identical semantics before and after (though it is nice if you can achieve that). In this case, one concrete example I can see where semantic differences are fine is the global flag. That flag doesn’t exist in Python 3.14, so you know no one running your library will be using it and you don’t have to try to implement it in earlier versions. If you try to keep consistent semantics and just use whatever lazy imports you are using already, you will actually be thwarting the global flag in Python 3.15, so you actually would prefer to switch to the standard lazy import system in 3.15 even if the semantics are slightly different from the one you were able to implement in 3.14.

281

I’d prefer the pep remain simpler and focused on consistent behavior for the future, and not concern itself with backportability. There are plenty of clever tools available that forbidding the lazy import within a contextmanager shouldn’t block anyone doing “reasonable effort” backporting. (it can’t be backported in whole). Additionally, Many libraries who have a need for lazy imports already have something in use, with no clear benefit to switch to a backport.

At the very worst, exec(_invalid_in_3_15_import_version_as_str, ...) is trivial.

More elegant solutions like generating a second version of the file and picking which one to use when a library is installed (once) are also possible.

3 Likes
282

Something occurred to me earlier that isn’t specifically related to runtime behavior but will nonetheless affect everyone. I think the PEP should offer guidance or a rejection of guidance on formatting import statements.

I would be extremely perturbed if formatters that support import groups (e.g. stdlib, 3p, 1p, relative, etc.) didn’t put lazy imports at the beginning or end of each group.

8 Likes
283

It actually does require a be imported. When you import a submodule or subpackage, two key things happen:

  1. The module is added to sys.modules, e.g. "a.b" in sys.modules
  2. The module is added as an attribute on the containing pacakge, e.g. hasattr(a, "b")

That step 2 would be confusing in a lazy world (e.g. what if there’s name conflict between something in the __init__.py and the submodule?).

But regardless, the fact that you always import the package(s) above you is so old that not doing it would break a ton of code. And if your code wouldn’t break then your __init__.py is probably empty and thus you don’t need it to be lazy for performance reasons. And if it does have stuff then there’s a good chance it has side-effects and thus you need it to be imported before the submodule is.

Personally, I would find the semantics harder to understand. As they stand now, I know that if I import something that everything above it is also imported. But making everything lazy means never knowing what is or is not lazy and then having to try and reason about the effects of that.

4 Likes
284

I would argue that allowing lazy imports is the simpler option, since forbidding them requires a more complicated implementation and requires specifying things like “when a lazy import appears in a context manager using the __lazy_modules__ mechanism, it is not an error but rather imported eagerly”.

This complexity is worth paying if there is a good reason, but the primary reason is that you can use context managers to catch exceptions and in the absence of a good reason to import things in a context manager it’s enough to say, “This will probably be misused if it’s used at all”. This analysis holds up well for try/except because there’s no good reason to put a deferred operation like that into a try/except block and it’s a footgun, but with blocks are more general than that, and we’ve already identified at least one good reason to use it.

I note that on Github I see about 10k uses of suppress(ImportError) and 3.5M uses of except ImportError. It is also not necessarily a situation where it makes no sense to put a deferred operation in a context manager, since there actually is stuff that is triggered. One could imagine a context manager that replaces __lazy_import__ to do some sort of weird and horrible magic for some reason.

At the end of the day, this is an empirical question, in my opinion. If there exists a better way to implement the backport than a context manager and that is the only use or context managers that people can see and there is real danger of people accidentally thinking they are doing one thing with context managers when they are actualy doing another, then I will gladly change my mind.

But at the moment, I don’t think we can vaguely gesture at a clever future that will find a perfectly ergonomic pattern for this when we have a very clean and ergonomic pattern already. This:

__lazy_modules__ = ["foo", "bar"]

with backports.lazy_imports(__lazy_modules__):
    import foo
    import bar

has several advantages over anything you have mentioned and anything else I can think of:

  1. It syntactically looks like imports, which means by default linters, highlighters, dependency checkers, etc, all get “for free” information about what is going on there.
  2. All code is centralized in one place and not generated, so readers of the code can look at the file itself and see exactly what is going to be executed.
  3. The code is the same in all versions
  4. It gives the backport implementer the freedom to choose what semantics apply pre-Python 3.15
  5. It requires no hacks to seamlessly update you to the new mechanism in 3.15.

The PEP itself doesn’t really need to be concerned with backporting per se, but by restricting the use of with statements it is cutting off a pretty fundamental mechanism in Python for encapsulating complexity. The backporting case is an example of that and perhaps it is the only example of that, but also the only example of with statements causing problems that people have come up with is contextlib.suppress, and I think it is just as fair to say, “This is not an especially common pattern and it is easy to encapsulate in a lint rule and the PEP doesn’t need to concern itself with elevating lint errors into syntax errors”

9 Likes
285

That will make lazy importing much less useful than it could be, though. Take this example:

lazy import numpy.random

If that imports numpy eagerly then the benefit becomes very slim, since:

>>> %time import numpy
CPU times: user 1.7 s, sys: 11.9 ms, total: 1.71 s
Wall time: 79.5 ms
>>> %time import numpy.random
CPU times: user 2.45 ms, sys: 1.98 ms, total: 4.43 ms
Wall time: 4.23 ms

(yes, importing NumPy can consume more than one second of precious CPU time due to bloody OpenBLAS).

Even with OMP_NUM_THREADS=1 there is still a 20x difference between the parent import and the submodule import:

>>> %time import numpy
CPU times: user 63.1 ms, sys: 5 ms, total: 68.1 ms
Wall time: 67.5 ms
>>> %time import numpy.random
CPU times: user 2.98 ms, sys: 995 μs, total: 3.98 ms
Wall time: 3.79 ms

So this current iteration of lazy importing would work poorly with NumPy, one of the most widely used Python packages in the world.

Edit: I had misunderstood @brettcannon 's comment, see correction by @DavidCEllis below.

2 Likes
286

The arguments for allowing with are indeed convincing, thanks!

Meanwhile, I found another issue with the reifying __dict__: reification can fail. Consider a foo.py like this:

lazy import fast_foo
lazy import slow_foo

def do_work():
    try:
        # maybe we can use the fast implementation
        fast_foo
    except ImportError:
        # no? no worries
        return slow_foo.do_work()
    fast_foo.do_work()

(Not realistic as-is, but you can extend the idea.)
With the current PEP, you can’t get this module’s __dict__. The failure cascades down to all the places where hiding lazy objects is meant to help introspection:

>>> import foo
>>> help(foo)
No module named 'fast_foo'

This suggests that reifying the whole namespace should either:

  • be an explicit operation – something you can skip doing or put a try around, or
  • swallow issues, and leave lazy objects around on failure. But that’s icky (ignoring the swallowing exceptions part: it’d be good if lazy objects show up rarely, but IMO it’s more important for them to show up consistently).

For me, that means vars() should not reify. But a “reify whole module” function that raises ExceptionGroup would be nice.

6 Likes
288

I think maybe you missed the Thing() at the end of the line in the example which would trigger the import of a.b which requires the import of a.

lazy import numpy.random won’t import numpy or numpy.random but using numpy.random.default_rng() will import numpy.random and numpy.

To be clear, using importlib.util as an example on the current build:

import sys
lazy import importlib.util

print("importlib" in sys.modules)  # False

importlib.util

print("importlib" in sys.modules)  # True
6 Likes
289

Ok, thanks for the correction! It appears this is doing what I hoped it would do, then.

2 Likes