PEP 810: Explicit lazy imports

[pganssle (Paul Ganssle)]

@mikeshardmind

I’d prefer the pep remain simpler and focused on consistent behavior for the future, and not concern itself with backportability.

I would argue that allowing lazy imports is the simpler option, since forbidding them requires a more complicated implementation and requires specifying things like “when a lazy import appears in a context manager using the __lazy_modules__ mechanism, it is not an error but rather imported eagerly”.

This complexity is worth paying if there is a good reason, but the primary reason is that you can use context managers to catch exceptions and in the absence of a good reason to import things in a context manager it’s enough to say, “This will probably be misused if it’s used at all”. This analysis holds up well for try/except because there’s no good reason to put a deferred operation like that into a try/except block and it’s a footgun, but with blocks are more general than that, and we’ve already identified at least one good reason to use it.

I note that on Github I see about 10k uses of suppress(ImportError) and 3.5M uses of except ImportError. It is also not necessarily a situation where it makes no sense to put a deferred operation in a context manager, since there actually is stuff that is triggered. One could imagine a context manager that replaces __lazy_import__ to do some sort of weird and horrible magic for some reason.

@mikeshardmind

There are plenty of clever tools available that forbidding the lazy import within a contextmanager shouldn’t block anyone doing “reasonable effort” backporting. (it can’t be backported in whole). Additionally, Many libraries who have a need for lazy imports already have something in use, with no clear benefit to switch to a backport.

At the very worst, exec(_invalid_in_3_15_import_version_as_str, ...) is trivial.

More elegant solutions like generating a second version of the file and picking which one to use when a library is installed (once) are also possible.

At the end of the day, this is an empirical question, in my opinion. If there exists a better way to implement the backport than a context manager and that is the only use or context managers that people can see and there is real danger of people accidentally thinking they are doing one thing with context managers when they are actualy doing another, then I will gladly change my mind.

But at the moment, I don’t think we can vaguely gesture at a clever future that will find a perfectly ergonomic pattern for this when we have a very clean and ergonomic pattern already. This:

__lazy_modules__ = ["foo", "bar"]

with backports.lazy_imports(__lazy_modules__):
    import foo
    import bar

has several advantages over anything you have mentioned and anything else I can think of:

1. It syntactically looks like imports, which means by default linters, highlighters, dependency checkers, etc, all get “for free” information about what is going on there.
2. All code is centralized in one place and not generated, so readers of the code can look at the file itself and see exactly what is going to be executed.
3. The code is the same in all versions
4. It gives the backport implementer the freedom to choose what semantics apply pre-Python 3.15
5. It requires no hacks to seamlessly update you to the new mechanism in 3.15.

The PEP itself doesn’t really need to be concerned with backporting per se, but by restricting the use of with statements it is cutting off a pretty fundamental mechanism in Python for encapsulating complexity. The backporting case is an example of that and perhaps it is the only example of that, but also the only example of with statements causing problems that people have come up with is contextlib.suppress, and I think it is just as fair to say, “This is not an especially common pattern and it is easy to encapsulate in a lint rule and the PEP doesn’t need to concern itself with elevating lint errors into syntax errors”