My other observations…
– Opting In –
From my point of view, the recommended way for an application to opt in should be with sys.enable_lazy_imports()
at the top of the script or main.py.
-L
would be useful to try it out for an app, but we should actively discourage it as the primary way for an application to opt in.
Also, count me as part of the group that would rather not support an env var.
– Focus on Opting Out? –
It may make sense to put the focus on opting out, instead of opting in, especially if we eventually want to make lazy imports the default behavior. I noticed that a number of comments in this thread make a bit more sense in terms of opting out.
For example, instead of asking libraries to opt in, whether globally or per-module, why not recommend that they explicitly disallow use with lazy imports:
# mylib/__init__.py
import sys
if sys.is_lazy_imports_enabled():
raise ImportError('mylib does not support lazy imports (yet)')
...
We could even provide something like importlib.disallow_lazy_imports()
to do that as a helper. It could also be something like importlib.lazy_import_unsafe()
to make it sound more ominous.
This would give library authors a simple solution that they can use temporarily, which avoids any hard-to-diagnose failures for their users. We would make this the prominent recommendation for now.
It may also make sense to focus the various proposed API on the operation we expect to go away. So:
-
sys.is_lazy_imports_enable()
→ sys.is_eager_imports_enabled()
-
importlib.disallow_lazy_imports()
→ importlib.require_eager_imports()
-
sys.enable_lazy_imports()
→ sys.disable_eager_imports()
- (keep
importlib.eager_imports()
, etc.)
– Changing the Default Behavior –
Perhaps I missed it, but there didn’t seem to be any discussion about if/when lazy imports would become the default. It makes sense to plan for that. At the least the PEP should say “This proposal does not include any considerations for making lazy imports the default behavior.”, though I’d expect it to at least define a basic plan.
– Backward Incompatibilities –
Clearly these are the sticky points to this proposal.
Import Side Effects
IMHO, things like registration-as-a-side-effect are a code smell. I’m definitely a big fan of being explicit about composing things together and activating functionality. However, it may not be so clear to everyone, nor clear what to do about it.
One thing that the PEP could do to help library authors (and probably application/script devs) is to give clear guidance (best effort) on how to replace import side-effect patterns with a corresponding side-effect-free alternative. (It wouldn’t hurt to identify the pros and cons of the side-effect patterns.) I can imagine an enumeration of each pattern, with use cases and examples of the before & after. The docs would benefit from a similar treatment.
Dynamic Paths
This is definitely a tricky one. It could be really hard to diagnose resulting failures (spooky action at a distance, etc.).
The only solution that comes to mind is to cache (on the module) the full import state relative to each import statement and use that later for the corresponding lazy imports. However, that seems pretty expensive, especially for something that won’t be a problem often.
Perhaps a less expensive solution would be to remember the dict/list version of each part of the import state where the import statement is. Then when the lazy import happens, emit a warning if those versions changed. (A dict watcher would probably reduce the cost, but wouldn’t help with the lists, e.g. sys.path
.)
Half-lazy imports would mostly resolve the problem, but the perf penalty is too high.
Deferred Exceptions
One thing came to mind that could help with this: preserve a traceback (or similar) relative to the import statement. Then, if an ImportError gets raised for the lazy import, set __cause__
(or would it be __context__
?) on that error to a copy with the preserved traceback.
– Rejected Ideas –
Per-module opt-in
I agree that this isn’t worth the trouble. It would be nothing more than busy work in most cases.
Also, I’d favor opt-out (e.g. my importlib.disallow_lazy_imports()
example above).
Explicit syntax for lazy imports
I agree that this wouldn’t pay for itself. The context manager would be more than enough and doesn’t affect readability all that much. IMHO, the only advantage of the syntax would be the little bit of help the compiler could provide to make the operation more efficient.
If we were to have explicit syntax, I’d argue it would make more sense to (again) focus on opt-out, e.g. eager import
.
Half-lazy imports
agreed on the perf penalty
I was going to suggest caching the module spec, but just can’t get past the performance hit.
Lazy dynamic imports
agreed
Deep eager-imports override
agreed (plus it’s easier to add later if needed than to remove it if not)
Other Thoughts
- consider making builtin/frozen modules always eager (probably too messy)
- …