PEP 810: Explicit lazy imports

PEPs
164

Not looking up the import at all leaves errors (imports not found) that get raised conditionally, so they might not be noticed right away.

Yes, most people that worry about that are probably using type checkers, which already check imports, but some might not be using them. Delaying import lookup is a possible way to not directly show bugs that exist. It might however be something “good” too, as a running python script might modify the contents of the other module at runtime.

Generally I’m not too sure which side to be on here, but it is something worth discussing again, at least in my opinion.

165

My understanding is that reification imports the module in the same way as if the import statement immediately preceded the line that is triggering the reification. Just without the pain of both writing the import statement before every line where the import is used and without the performance impact of triggering the import machinery every time.

For the case where try/except is being used to check if a module is installed/exists I think you can replace:

try:
    import new_module
except ImportError:
    import old_module as new_module

With:

if importlib.util.find_spec("new_module") is not None:
    lazy import new_module
else:
    lazy import old_module as new_module

If this is correct, perhaps this should be documented?

There’s some text on PEP-649/749 annotations that demonstrates that reification wouldn’t occur until annotations are accessed. Is it worth noting this is also true for aliases created by the type statement until something accesses __value__?

4 Likes
166

Thank you for the PEP! I love how it combines recent-ish Python features; it’s like we were building up towards this for some time :‍)

One thing that wasn’t clear is what happens when the “real” import fails. The reification section only talks about how the exception will look.
I expect the name stays bound to the lazy object, so the next use will attempt reification again?
(The PEP says “first use” a lot, and I agree it’s a good simple term, but I’m missing a clarification that it’s technically “all uses before reification”, or “failures don’t count as use”.)

It may be good to clarify “use” – the real-world examples are great but they kinda imply you need attribute access or function calls. Would it work to e.g. extend the dumsp example with:

# The function call is not necessary; any use of the name will trigger reification:
dumsp  # raises ImportError
serializer_function = dumsp  # raises ImportError
9 Likes
167

The PEP says the name to a lazy object is rebound to the real object once reified:

The proxy approach is simpler: it behaves like a placeholder until first use, at which point it resolves the import and rebinds the name. From then on, the binding is indistinguishable from a normal import.

However, when I tried the following code with the reference implementation, the name remains bound to the original lazy module object:

lazy import m
print(globals()['m'])
print(m)
print(globals()['m'])

which outputs:

<lazy_import 'm'>
<module 'm' from '/home/blhsing/src/cpython-lazy/m.py'>
<lazy_import 'm'>

I expected the last line to output <module 'm' from '/home/blhsing/src/cpython-lazy/m.py'> because the name m is supposed to have been rebound to the reified object in the global namespace.

Am I interpreting the PEP incorrectly? If so, is globals()['m'] going to stay as a lazy object no matter how many times the global name m is loaded?

I haven’t spent time debugging the code yet but I think this line is supposed to be what rebinds the global name indeed:

2 Likes
169

This is covered in the document:

However, calling globals() does not trigger reification – it returns the module’s dictionary, and accessing lazy objects through that dictionary still returns lazy proxy objects that need to be manually reified upon use. A lazy object can be resolved explicitly by calling the get method. Other, more indirect ways of accessing arbitrary globals (e.g. inspecting frame.f_globals ) also do not reify all the objects.

Note also that the implementation may be out of date as it main purpose it’s to evaluate the idea but the canonical document and behaviour is what the PEP says. We try to keep both more or less in sync but it’s quite challenging so please understand :slight_smile:

1 Like
170

Yes, but I thought I’ve triggered reification by explicitly referencing the name:

print(globals()['m'])
print(m) # explict reference supposedly reifies m
print(globals()['m']) # m should bind to the real object at this point

Yes, but I thought this line is supposed to be what implements the rebinding:

But yeah it may just be a bug in the reference implementation that doesn’t represent the intention of the PEP. Thanks for the response!

2 Likes
171

Yeah indeed is a bug. Notice the code you copy is for LOAD_GLOBAL that happens in functions:

lazy import lel

def foo():
    print(globals()['lel'])
    print(lel) # explict reference supposedly reifies m
    print(globals()['lel']) # m should bind to the real object at this point

foo()

this prints:

<lazy_import 'lel'>
<module 'lel' from '/Users/pgalindo3/github/python/main/lel.py'>
<module 'lel' from '/Users/pgalindo3/github/python/main/lel.py'>

We will fix the bug don’t worry :slight_smile:

Edit: Fixed

9 Likes
172

If the module foo both defines constants and also loads some heavy machinery (usually by importing other submodules), maybe it could use lazy imports itself, like replacing

# module foo
DEFAULT_PORT = 8080
from . import heavy_machinery

with

# module foo
DEFAULT_PORT = 8080
lazy from . import heavy_machinery

and the constant can be used without incurring the whole import. It extends the optimization step to the library, but if (when?) this feature gets widely used, this will be common practice.

173

You can find workarounds for this such as:

if some condition:
    lazy from my_module import impl1 as impl
else:
    lazy from my_module import impl2 as impl
1 Like
174

Yeah, that’s basically it. We updated the PEP to make that section clearer. Hope it helps!

Separately, we wanted to bring a topic for discussion here: Should lazy import be allowed in with blocks. Currently, the PEP does not permit it. The primary rationale is that with can also be used to perform exception handling, so it’s “more like” try/except. Consider the use of contextlib.suppress

MY_MODULE_AVAILABLE = False
with contextlib.suppress(ImportError):
    import my_module
    MY_MODULE_AVAILABLE = True

This example is obviously somewhat contrived but represents the point. Replacing import with lazy import there seems to be an obvious foot-gun: MY_MODULE_AVAILABLE would always be True, and depending on the rest of the code in the module, you’ll either get ImportError in a place that might not expect it or other secondary crashes (e.g. NameError). However, while the mechanics in this example are equivalent to try/except, most other uses of with in the language are semantically different. Python allows other deferred operations to happen in with blocks — you can close over variables, call asynchronous functions, etc. And we think there might be at least one concrete use case for allowing this, to enable a functionally-compatible backport to earlier Python versions.

So we wanted to survey the larger community on this point. Should we relax the restriction of lazy import in with blocks?

1 Like
175

I understand that. The point of the example was more that top-level simple assignment does not have to reify (exactly because a [lazy] import can be seen as a sort of an assignment). But the PEP is setting the semantics for this now, and it might be harder to change later. Should it be guaranteed that any expression that evaluates to a lazy import/module is instantly reified, or should it be an implementation detail?

176

TL;DR (Edited): I think that lazy imports should probably be allowed in with blocks*, and it’s important that that be in this PEP*, because the main use case is ease of backporting, which is defeated by forbidding it in any release.

First off, this is a great PEP. It’s clearly super well thought-out and it’s going to solve a lot of real problems that people are currently working around. dateutil not only has this thing (which solved the perennial problem of “how come import dateutil; dateutil.tz.tzfile doesn’t work?” without making all the modules eager), but also has had stuff like this (lazy imports between modules) for over 20 years now. It will be very nice to update that stuff to use a native mechanism.

That said, the biggest stumbling block I’m hitting with the specifics of the PEP here is that, as a library maintainer, I don’t see a great way to have a well-scoped backwards-compatible implementation that is lazy both pre-Python 3.15 and post-Python 3.15. The __lazy_modules__ mechanism allows you to opt in to lazy semantics for 3.15, but I already have lazy semantics, and switching to __lazy_modules__ for Python <3.15 would be a regression for my users. I believe @stefanv expressed something similar here.

I can think of two ways to ergonomically accomplish this, but in both cases I would want to use a context manager like this one, so that you can do something like this:

__lazy_modules__ = ["foo", "bar"]

from ._compat import lazy_importer

with lazy_importer(__lazy_modules__):
    import foo
    from bar import blah

I’m guessing that in Python 3.14 you would implement this with some kind of dirty hack or custom loader (epyutils does it like this, which is less dirty than I expected but also doesn’t look especially thread-safe). With the PEP as it is now, because lazy imports are forbidden in with blocks, you would also have to use a hack in Python 3.15, but I think you could at least base that one on __lazy_import__ instead of __import__, so you would get some gain, but you won’t be able to avoid hacks until you fully drop anything lower than 3.14.

If lazy imports were available in with blocks, however, in Python 3.15+, lazy_importer could just fall back to being contextlib.nullcontext. So my question is this: do we actually need to forbid lazy imports in with blocks?

I think that while it is possible for context managers to do exception handling, the kind of exception handling done there is pretty different from the kind done in try/except blocks: usually you don’t use them to try to handle exceptions so much as to prevent exceptions from interfering with your management of the lifetime of some resource (though contextlib.suppress and presumably warning-suppression context managers are significant counter-examples here).

The most salient points to me:

  • The benefits of forbidding lazy imports in with statements are modest but real, and much lower than the benefits of forbidding them in try/except blocks.

  • I can think of at least one potential advantage to allowing this pattern — backwards compatibility context — but I can’t think of any others.

  • The backwards compatibility use case only works if with statements are allowed from the beginning — if they are forbidden in 3.15 and then allowed in 3.16+, we are still stuck with a hack in 3.15.

  • One advantage of forbidding lazy imports in with blocks is that for people who want to run with -X lazy_imports=enable, you can do something like this to force eager imports:

    with eager():  # alias for contextlib.nullcontext
    import whatever

    Whereas if they are allowed, I think the best you could do is:

    try:
    import whatever
except Exception:
    raise

    Though, again, perhapse there is a better way to do this, and also maybe global mode is not something that needs to be driving the semantics here.

At the moment I’m leaning towards recommending that lazy imports be allowed in with blocks, but it’s very much an empirical question, so I’d ask the PEP authors, people who have rolled their own lazy import mechanisms and library maintainers to weigh in on these questions:

  1. Is this context manager idea even the best way to do a backwards-compatible polyglot lazy import?
  2. Are there other circumstances you’ve seen where on balance people are going to be bitten by doing lazy imports in with blocks? (Bonus points if these are real scenarios and not things that just could go wrong)
  3. Are there other uses for with blocks that I’m missing that would weigh in favor of allowing lazy imports to occur within them.
12 Likes
178

I would appreciate two clarifications about this PEP.

Clarification 1

Would the use of lazy be allowed inside an if statement? This could be used to conditionally lazy-load a fallback implementation, for example. (Note that I am not asking about using lazy inside try-except!)

try:
    # This line is intentionally not lazy, because
    # we need to know if foomod is actually installed
    from foomod import do_stuff
    foomod_supported = False
except ImportError:
    foomod_supported = True

if not foomod_supported:
    # If foomod is not installed, try barmod instead
    lazy from barmod import do_stuff

# do_stuff is either non-lazy reference to foomod or lazy reference to barmod

It seems like this could be supported without triggering the issues caused by allowing try-except.

Clarification 2

Unless lazy imports are disabled or suppressed (see below), the module is not loaded immediately at the import statement; instead, a lazy proxy object is created and bound to the name. The actual module is loaded on first use of that name.

What does ‘suppressed’ mean in this context? What might suppress a lazy load?

From the replies to this thread, I infer that this is talking about the Lazy imports filter section. Is that correct? If so, I think that could be made more clear by either changing “suppressed” to “filtered,” or by adding a link to the relevant section.

1 Like
179

Yes, if statements are not included in PEP 810 – Explicit lazy imports | peps.python.org. Unless you feel very strongly I don’t think we will explicitly state this since I don’t want to have to list all possible places where is allowed but just the ones where is not.

Yeah its talking about the filter. We will clarify it. Do you want to make a PR? :slight_smile:

4 Likes
180

I think it’s not clear. When I read, “The soft keyword is only allowed at the global (module) level, not inside functions, class bodies, with try/with blocks, or import *,” I interpreted that in my head as, “You can’t use this in any situation that would require indenting.” I don’t think there’s any example in PEP that uses lazy within an indented block, except in the Rejected Ideas section.

I’d like to - I will warn you that I have not made a contribution to the peps repo before, so I’ll need to figure that out before I send a PR.

2 Likes
181

Don’t worry then. We will consider adding this clarification in the next batch of updates.

2 Likes
182

As someone who’s working on Python language tooling I’m really excited to see this proposal, I think it’s a great addition to the language and hope it gets accepted.

I’m working for Google maintaining the infrastructure and tooling around Python language and we’ve also had issues with Python applications startup time caused by imports that are very slow: we found that imports alone cost us hundreds of hours of compute per week, and latency costs multiple software engineer years, not to mention the unquantifiable benefits you get from lower latency operations. This was a big enough deal that directives to fix it came from high in our leadership.

Thus we now have built and started deploying our own implementation of lazy imports, originally based on the rejected PEP 690 to turn lazy imports globally.

Our experience was that the PEP 690 approach of turning on imports globally would require an enormous effort to implement, since most existing code was unsuited for lazy imports, so our implementation turns on lazy imports at the module level, which is simlar to — and compatible with — the approach taken in PEP 810 (with one of the major differences being driven by the level of abstraction we were working at: we wanted to minimize changes to the language itself).

Having read through all the discussions about PEP 690, we end to agree that the outcome in PEP 690 was correct; enabling lazy imports globally was an attractive idea, but even in a monorepo like ours with a high degree of control over the full codebase, it was too much work to adapt to an opt-out approach. Our experience with a targeted, opt-in approach similar to the one taken here is that it is dramatically more tractable, and brings huge benefits.

Given that this problem is pervasive enough that so many companies and ecosystems have created their own version of lazy importing, it seems like this really should be solved at the language level, and PEP 810 seems like the right way to do that.

12 Likes
183

(I apologize in advance if this question has already been answered, I don’t see that in the PEP text but might have missed a comment here)

Let’s start with a simple directory layout:

.
└── foo
    ├── bar.py
    └── __init__.py

And then

echo 'bar = 1' >| foo/__init__.py

This is a common pitfail with current module system: depending on whether the codebase has import foo.bar somewhere, the outcome of import foo; print(foo.bar) is different:

import foo
print(foo.bar)  # 1
import foo.bar
print(foo.bar)  # <module 'foo.bar' from '...'>

Okay, that’s what we all already know. What happens in the following case (the grammar specifies dotted_as_names there, so it is allowed)?

import foo
lazy import foo.bar
print(foo.bar)

From my reading of the PEP, it will print <module ...>. Is this correct? Or will lazy import make that snippet print 1, making lazy import foo.bar essentially dead code forever, never triggered?

And in the opposite case, will lazy be essentially a no-op below?

lazy import foo
import foo.bar
print(foo.bar)
1 Like
184

I think this case should generally be rework in a way where no matter what happens,the last import statement is used.

This should then obviously apply to any kind of lazy import as well.

185 
>>> import foo
foo.__init__ imported
>>> lazy import foo.bar
>>> foo.bar
bar imported!
<module 'foo.bar' from '/home/pablogsal/github/lazy/foo/bar.py'>

The lazy import foo.bar is not dead code. Even though foo is already eagerly loaded, the lazy import creates a valid lazy reference to the submodule. When you access foo.bar, it triggers reification and loads the bar submodule.

>>> lazy import foo
>>> import foo.bar
foo.__init__ imported
bar imported!
>>> print(foo.bar)
<module 'foo.bar' from '/home/pablogsal/github/lazy/foo/bar.py'>

Here the eager import foo.bar reifies the lazy foo import and loads both the package and submodule.

The PEP’s reification section explains that:

… when a package is reified and submodules in the package were also previously lazily imported, those submodules are not automatically reified but they are added to the reified package’s globals (unless the package already assigned something else to the name of the submodule)…

This means lazy import foo.bar creates an independent lazy binding for the submodule that works regardless of the parent package’s load state (check the PEP for details).

1 Like