Let me tell you the story of Lazy Imports regarding that global flag, so we can have some context about why we are proposing it that way.
When I started working in lazy imports at Instagram, my first intention was to be able to flag each import that I wanted to be deferred as lazy, with a comment or with a new keyword as the lazy keyword you’re suggesting.
The ideal to me was, obviously, that eventually everything was supposed to be lazy to get the full benefits of laziness, but I doubted that was going to be possible because of the large number of third party packages and huge amount of modules we use (it was too big of a change in the semantics of imports, I thought). I felt it was going be prohibitively difficult to make that ideal happen. I initially settled for adding a way to toggle lazy imports on and off in a per module basis and see if that was enough, so I implemented that in Cinder as from __future__ import lazy_imports.
I started enabling the flag in a few modules of our own codebase after analyzing them and figuring out if they were going to work or not. I kept doing this for more and more modules and started adding the future import in whole packages, then on whole directory trees!
When I realized it was working for almost all our internal modules I was trying it on (most of the time without any changes). I added an option to enable it everywhere, and I also added from __future__ import eager_imports, to disable lazy imports in a per module basis when needed. When I saw things were broken I started digging into seeing what was missing.
I fixed some bugs in Lazy Imports, also fixed some unneeded incompatibilities in the implementation and I progressively saw less and less issues. Then I started seeing common things that weren’t working. It mostly became a matter of fixing few patterns: import side effects related to the registry pattern; bad imports such as importing a module and not its submodules and then expecting the submodule attribute would be there (e.g. import foo; foo.bar.Bar → import foo.bar; foo.bar.Bar); cycles in custom import loaders (when lazy imports were triggered midway another import); and modules that mangle with sys.path and/or sys.modules in certain ways (such as adding a path to sys.path doing a lazy import and removing the path before resolving the lazy import).
Finally we ended up not using any explicit way of enabling lazy imports for our codebases and in the PEP we removed the from __future__ import lazy_import because it’s not clear whether lazy imports will ever be the default or not for Python.
There are many different types of uses of Python and some communities have different patterns, but all the evidence we do have is that the percentage of modules we tried and worked without any issues out of the box with lazy imports enabled was high, and that just enabling lazy imports in a few modules doesn’t yield many benefits at all. The true power comes when you enable laziness in whole systems. We’ve saved terabytes of memory in some systems and reduced start times from minutes to just a few seconds, just by making things lazy. When we saw this is when we thought of the insanely great amount of memory, time and other resources this could save us all, if we make this available for everyone!
Having said that, I’m not totally against having an lazy import mod or from mod lazy import foo. However, there are a few considerations::
- Perhaps most obvious, is that a new syntax for imports would require changes in parsers, linters, debuggers, etc.
- One bigger concern I have has to do with implementation details and what the new syntax could require: what happens with
lazy importinside a function (inner lazy imports)?. They run with optimized fast locals and, in the current implementation of lazy imports, that would mean we’d lose that for lazy imported names, or introduce performance penalties on fast locals. It’s also unlikely that having inner lazy imports supported would bring additional or significant wins, because, many times, inner imported names are immediately used, defeating lazy imports purpose there. - Most importantly, needing to explicitly call for a lazy import would mean we won’t see any of the benefits of the feature until the new syntax is generally adopted, and that could take years! Not to mention that very good (but unmaintained packages) could never get the (some times) free wins.
Edit: I’m also sharing a link to the blog post I wrote back in June, I don’t think I had shared it here before, and it could give some additional context: Python Lazy Imports With Cinder