PEP 690: Lazy Imports

[steve.dower]

@carljm

I think the question of what this would mean for libraries has been a bit vague in this thread, and would benefit from more concrete examples. Specifically, I don’t think the idea of libraries “opting themselves in” to lazy imports, which has been mentioned a few times, is actually relevant or helpful to the real use cases I’m aware of.

The example that comes to mind for opting-in is this one (dataclasses → re → enum), but I’m inclined to agree that library opt-in isn’t an important piece here. If it matters enough, it’s easy for the library to import explicitly in a function (and re should be good enough at caching that pre-compiling expressions is rarely a huge benefit, so it can just be imported when needed).

Opting-out is my bigger concern. Who is supposed to do the opt-out that triggers mimetypes.init() or protects against sys.path modifications when those are buried 2-3 levels deep in someone else’s library? The application developer obviously owns the final responsibility, but realistically their only available mechanism isn’t going to be very fine-grained (other than filing bugs against library devs, which is what we want to minimise).

@carljm Do you have a sense of how deep the lazy imports need to be to get the benefits you’ve seen? Or at least most of them? My own experience optimising CLIs has suggested that it’s quite diminishing returns after your “top level” imports - the ones that are used directly by command implementations, which seems to be what is implied by some references to __main__ above.

Would requiring CLI developers to use an equivalent of with import.shallow_lazy around just the imports they control offer all/most/some/none of the benefits? ^[1]

I would have guessed “most”, and that the rest could be helped with a few updates to libraries, rather than implicitly switching all imports to this behaviour, but would buy a few counterexamples of libraries with many transitive imports but non-overlapping sets of functionality (such that you can use the library for something useful while only needing some of those imports).^[2]

---

1. And could it be spelled with importlib.util.lazy(): and be implemented today…? ↩︎

2. One such example is the inspect module, which doesn’t seem to need both enum and importlib.machinery for the same scenarios. Is this common? Should it be encouraged at the cost of less predictable side-effects, or should we instead be encouraging separation of distinct functionality as a performance (and arguably, readability and maintainability) improvement? ↩︎