PEP 690: Lazy Imports Again

[Kronuz (Germán Méndez Bravo)]

@encukou, thank you for your comment, it’s indeed a valuable thing to add to PEP.

@encukou

Lazy imports aren’t fully transparent, as they have “necessarily observable effects”. With that in mind, is preventing any other observable changes worth the implementation complexity?

The complexity worries me. I understand that it can be easily added to current dicts, but it’ll be a burden for any future optimizations and implementations.

I agree there is some amount of complexity added to dictionaries. However, I’ve been through two implementations of it (on top of 3.8 and on 3.12, and between the two I saw changes the way dictionaries worked, a lot of them!), and my experience was it gets easier every time. We only need to control how lazy objects can enter and leave a dictionary, and it’s done, more or so, in a simple way that doesn’t add too much burden, in my opinion. Only input and output of key functions need to be controlled.

@encukou

To make things clearer, consider semantics like the following. (I don’t see anything similar in Rejected ideas, hopefully it wasn’t floated earlier):

• import foo creates a global variable __lazy:foo (specially named, but otherwise normal), and sets it to a lazy object. (Another possibility is using a global dict: __lazy__['foo'].)
• LOAD_GLOBAL for potentially lazy objects (which are known at compile time) becomes LOAD_LAZY_GLOBAL, which:
  • tries loading foo, and if it doesn’t succeed:
    • loads __lazy:foo from globals (not builtins), resolves it and stores the result as foo
    • deletes lazy:foo
  • replaces itself with LOAD_GLOBAL, if the specializing machinery allows that
• module __getattr__ tries resolving lazy objects the same way
• is_lazy_import, eager_imports, set_lazy_imports would work as in the PEP
• importlib.resolve_lazy_imports(mod_or_dict) or a globals(resolve_lazy_imports=True)

Yes, we did consider, and even tried implementing the type of approach you are suggesting. We found that it makes the incompatibility barrier too high and too many lazy objects escape where they aren’t supposed to (especially due to the presence of C extensions).

@encukou

So, my concrete question is: how important is the “transparency”?

The “transparency” in the implementation is crucial. Even the slightest changes in behaviour can cause bigger than expected incompatibilities, after a lot of iterations this is what we believe to be the right amount of transparency to allow things to work and to be easy to reason about.