PEP 690: Lazy Imports

__future__ flags are for introducing features that will become normal always-on features in the future (hence the name), not for always optional features.

We don’t have a __future__ flag for (e.g.) the -E and -O command line switches.

3 Likes

According this this, subclasses() is an optimization, and in the context of another optimization (import time), it may make sense to trade one for another. You’re right that decorators and meta classes may lead to side effects, but I believe my point still stands - it is possible to determine those at the AST level.

In C++11 constexpr was introduced as ways to do some computation at compile time, and initially its scope was extremely small. Over time, the set of things that were provably correct at compile time grew, so the set of features supported by constexpr also grew. I’m imagining the same sort of story for python lazy module imports.

Yes, this is a very good point. In fact we’ve already had to add an “is lazy imports enabled?” API to the implementation for testing reasons, we need to add this to the PEP also.

4 Likes

That’s the main problem others are suggesting: users asking library authors to support lazy imports may significantly increase the support burden on many Python developers, arguably reducing the quality of the ecosystem.

I think we should heavily discourage such requests, and find a way to make the community think of lazy imports as a potentially-available feature that only application developers (not end-users nor library authors) can enable. Maybe they can only enable it if their CI test-suite has 100% path coverage :smiling_imp:.

In my proposal, both end-users and library developers ideally have no control over lazy-importing. Raising an exception after discovering that lazy importing is in-effect is a form of control (although I don’t see a way to prevent this if sys.lazy exists).

Would it make sense to enable lazy-imports in the entry-point hook, without adding a command-line flag, environment-variable or sys function? For example, a lazy marker in the console_script entry-point definition, which then tells Setuptools to add a #py-pragma: lazy line early in the hook, which then tells the parser to enable lazy-imports.


The main one to look out for is __init_subclass__, which I suspect most users expect to be run on import of the subclass’s module.


Another idea: rename Lazy Imports to something more clandestine: eg “Dangerous Dragons mode”, “Unsupported Deferring of Imports Mode (UDIM)”, “Potentially Problematic Postponement of Imports (P3I)”

2 Likes

The idea of adding something to the console_script entry point specification has merit. Even if we keep the -L flag, you still need a way to tell the backend to create the script using that option (but also other options might be useful and I’m not aware of a standard for specifying that for entry points, in either PEP 621 or the packaging guide.

However, even if the thing that triggers lazy imports is a pragma comment, then it’s always going to be possible for programmer misuse. So I don’t see that as an argument for removing the -L option, which will still be useful when the script is not created by an entry point.

3 Likes

Beyond Mercurial, I know of at least one very large Python code base that (at least at one point) used LazyLoader for their entire test suite.

Note you will need to update Declaring project metadata — Python Packaging User Guide as that’s the packaging standard for declaring entry points.

Yep, the shebang will need the flag on Unix. Not sure what would need updating for Windows.

And I’m not a fan of the pragma idea as it now makes this somewhat syntactic and it would be the first comment that changes runtime semantics (I don’t count the encoding cookie). The AST doesn’t even keep comments, so that would make this even more of a change.

3 Likes

Presumably, the actual Entry Point Specification would also have to be modified, no?

2 Likes

Probably both. The example in that spec, for instance, is setuptools-specific, so it’s already somewhat outdated.

1 Like

You may be aware the Scientific Python community is using lazy loading (via LazyLoader in some cases) e.g. in skimage, tensorflow, SciPy, napari, cuda, … libraries:

Maybe they should be asked about their experience and impact on them if this is deprecated?

4 Likes

Thanks @petersuter. I posted at Lazy loading has landed! - #6 by carljm - scikit-image - Scientific Python to invite attention to PEP 690, and included some thoughts there about how PEP 690 can cover the SPEC 1 use case. I think in general it would be an improvement, but to fully cover it might require that we add the capability for a library module (typically an __init__.py in the SPEC 1 use case) to locally declare all its own imports lazy by default.

First of all, I think there is real value in lazy imports, and in adding them to the language rather relying on various third-party implementations.

However the proposed PEP just feels too magical. Controlling import by configuration, rather than being explicit,
makes it really hard to reason about and test code.

I don’t think that having things that do or don’t exist, depending on how you look at them, is a good idea.
(I’m referring to the hidden thunks that a lazy import introduces).

I feel that explicit approaches have been rejected with insufficient justification.
Using an explicit lazy as a keyword prefix to import makes it very clear what is going on, and allows packages precise control over what they import lazily and what they import eagerly, and the ability to test their packages properly.

lazy import mod

could simply create a lazy module object that would convert itself into a real module (performing the import) when it was used.

I understand the desire to force packages to be loaded lazily in a large application, rather than waiting for package maintainers to implement it themselves, which might take years.
However, if the feature is as valuable as claimed, package maintainers will be eager to adopt it.

The PEP as proposed has too many rules as to when a module is imported lazily or not. It is impossible to know at a glance whether a module will be loaded eagerly or lazily. A keyword makes it very clear.

One thing that this PEP does that an explicit lazy import could not is defer the loading of a module when an attribute is accessed from it without changing the class of that attribute, so this

from mod lazy import foo

can only work if foo is a module.

However, if lazy imports are wanted, then changing to code to

lazy import mod

and changes uses of foo to mod.foo doesn’t seem a big deal.

4 Likes

Really? I think all these issues have been discussed quite extensively, both here and in the PEP. I acknowledge that there’s different opinions about it, but I don’t think the PEP itself is lacking in justification. That said, I think the PEP is on target for what it proposes. If there are advocates for the explicit keyword based approach, then let’s see a competing PEP with all the necessary details, and we can debate the alternatives!

2 Likes

I don’t know if they have been discussed here.
The PEP should stand alone, IMO there isn’t sufficient justification in PEP.
But, as you say, it is a subjective judgement.

With regards to testing, I can’t see how it is much more than adding a lazy-imports toggle to the CI matrix. Perhaps Pytest can learn a command-line flag to run the test suite twice, once with lazy imports.

The point of this PEP is that library authors shouldn’t care about whether lazy imports are enabled, just like right now where libraries don’t care they’re running in a separate thread. The application developer has the authority to enable lazy imports, but also has the responsibility to test the application thoroughly once enabled.

I think preventing application developers from demanding library authors support lazy imports is the social issue that should be addressed.

3 Likes