Alternative path for explicit lazy imports

[Jost (Jost Migenda)]

Thanks for sharing a concrete proposal and suggested implementation! This has been really interesting to read through and you may have nerdsniped me a bit with the draft PR …

While this is intended as an alternative to PEP 810, I would rather avoid splitting that discussion up into multiple threads (the existing thread is already long enough to make the SC’s job unenviable); so I’ll try to avoid comparisons and focus on your idea here on its own merits.

@dg-pb

from importlib.util import *
turtle = lazy_import('turtle')
print(type(turtle))     # lazymodule

# Any further non-eager imports simply pick up `lazyimport`
# from `sys.modules`

import turtle as turtle2
print(type(turtle2))     # lazymodule

This provides a mechanism to set lazy imports at the main application,
and all subsequent dependencies will simply pick them up from sys.modules:

# main.py
from importlib.util import *
tkinter = lazy_import('tkinter')
import submodule        # @ submodule: lazymodule
submodule.foo()         # I am not using tkinter

# tkinter is still lazy
print(type(tkinter))    # lazymodule


# submodule.py
import tkinter

The potential for a single lazy_import() call to change the meaning of later import statements elsewhere in the code (including in other modules and third-party dependencies) makes it extremely hard to reason about what code does.

Imagine I write a library and use import statements to import my dependencies. Now any user of that library could, in their code, lazy_import() one of those same dependencies before importing my library. Suddenly the execution order of code in my library would change in ways that are basically impossible for me to predict and test for.

As the library developer, how would I fix this? If I have any non-trivial dependencies, it is not realistic for me to understand the implementation details of each of them well enough to know which (if any) import has any side effects that may cause issues if lazily imported^[1]. So … would I need to replace all my import statements with calls to eager_import(), just to be safe?^[2]

(Or the other way around: Let’s say my code imports two third-party libraries, a and b, in that order; and a also imports b internally. Then, whether b is lazily or eagerly imported in my code depends on an implementation detail of a.)

While more selective than PEP 690’s global approach to lazy imports, this shares some of the effects and issues cited in that discussion. The reasons cited in the rejection message for that PEP apply here as well.

@dg-pb

Error handling:

In sync with non-lazy imports:

import sys
from importlib.util import *
try:
    lazy_import('no_turtle')
except ModuleNotFoundError:
    print("'no_turtle' package not found, install by $ pip install no_turtle")
    sys.exit(1)

That only applies to ModuleNotFoundError, though—right? Something like

try:
    lazy_import('my_module')
except ImportError:
    # handle as appropriate

might not work as in the eager case, because ImportError can be raised during execution of the imported module. (I.e., long after the lazy import.)

That may be an acceptable trade-off, if sufficiently documented, but it takes away some of the simplicity and makes it harder to teach and reason about. Suddenly, learners have to understand internal steps of the import mechanism to understand which types of errors may be raised at what stage of the import process.

@dg-pb

Cons:

1. Does not support from module import attribute. However, PEP810 does not support from module import *, so this is just a slight functionality downgrade

This is not quite a fair comparison to me. from x import * has a few legitimate use cases, but I would consider it bad style in most situations and try to avoid it. (I can think of perhaps a handful of cases where I actually use it in my projects^[3].) In contrast, I use from x import y very commonly; often multiple times per module.

@dg-pb

Pros:

1. Simpler
2. More explicit
3. More interactive
4. Provides more control

In some ways, yes; but as noted above, in other ways things have the potential to become quite messy and have far-reaching side effects.

@dg-pb

5. Manages errors at definition

As above—only ModuleNotFoundError, not necessarily other ImportErrors.

@dg-pb

9. Does not require dedicated syntax for functionality to work

In fact, to some degree it breaks the existing dedicated syntax (because users can no longer be certain whether an import statement is eager or not). Introducing a new function^[4] and recommending it to replace an existing statement seems pretty unprecedented to me.

@dg-pb

10. Is orthogonal to any future endeavours in “deferred evaluation”

As @ajoino said above (and, I believe, a few people said in the PEP 810 thread), if there are no specific ideas currently being worked on, this is much too handwave-y to be of much use.^[5]

Also, there are already some kinds of deferred evaluation in Python (e.g., with lambdas); as far as I’m aware those don’t share a fundamental mechanism with any lazy import implementation (existing or proposed), so it’s not a priori clear to me that any future deferred evaluation proposals are especially likely to do so.

---

1. in more complex cases, I’d even have to consider the relative import order of multiple dependencies, which could lead to a combinatorial explosion ↩︎

2. Or call exclude_lazy() on everything I import, which I think is equivalent? ↩︎

3. including a few legacy instances that I’d like to clean up at some point ↩︎

4. in importlib.util no less, not even a builtin function ↩︎

5. It’s a fully generic counterargument that works against basically any change to the core language. “A built-in syntax for f-strings? Nah, let’s stick with str.format() in case there are any future endeavours in string interpolation.” ↩︎