Alternative path for lazy imports v2 and PEP810

Ideas
21

Thanks for pointing out. Got focused on name in sys.modules and missed the fact that the common way is sys.modules.get.

Could make __bool__ return False for lazymodule to keep this behaviour.

But special utility is also an option - a bit more forward looking and opens space for evolution.

There are more hits for this:

g 'sys\.modules\.get' | grep -v import | grep -v test | grep -v rst | wc -l

26

And yeah, most of these would need special handling. Simply implementing __bool__ would not be enough. And even incorrect as some checks do explicit is None:

globals = getattr(
                sys.modules.get(self.__forward_module__, None), "__dict__", None)

Utility module_is_loaded would most likely be needed.
But it is fairly simple change - most of hits are concentrated - 8 modules in total.

22

So ok, did a circle (went through use cases) and back to the same place.

No code is broken by lazymodule introduction.
It only does not take advantage of it.

  1. Making __bool__ return False would cover some of the cases as they are.
  2. Other cases would need to opt into benefits of lazymodule by more explicit check.

The simplest solution would be to make __bool__ return false for entries in sys.modules that do not represent loaded module and make this a one correct way to check if a module is loaded:

module_exists = bool(sys.modules.get('numpy', None))

The total number of cases is small: 7 + 26 = 33.
It is neither rooted badly, nor it is a difficult adjustment.

23

__bool__ is not a good idea, but isloaded utility works well.

24

This is pretty cool.

Folks who have to support a wide range of Pythons (e.g. library authors and maintainers) may very much like your library… provided that it upgrades nicely to lazy import if the runtime supports it.

Aside, and totally minor, what does mby= kwrag mean, “maybe”?

1 Like
25

Yeah, this is optional thing. If package can not be located the return value is None, for both lazy and eager imports. Just a tiny convenience.
if mby == 1, then it doesn’t raise for ModuleNotFoundError, if mby == 2 then it doesn’t raise for Exception.

Currently both modes are functional:

  1. LazyLoader-like
  2. PEP810-like, where no checks at import are being done.

However, PEP810-like doesn’t raise any errors on call, thus mby has no impact.

26

PEP 690 rejected global lazy imports.
PEP 810 repackaged the same concept, adding explicit imports to the forefront.
But the real value - and the original driver - is global mode (even if officially discouraged).

I raised concerns during the PEP process because I had strong intuition there was a better path for global imports. And I still think there is.

I had some doubts along the way and stressed out a bit thinking maybe I have overextended with my criticism (especially in the beginning), but after following through, I think my reaction, given my POV, was closely proportionate.

Any lazy import implementation inevitably causes a fracture, adoption period and various nuances that will be felt throughout ecosystem.

If I am right and implementation of PEP810 did not take the optimal path, it will be messier than it needs to be and will deliver less value.

The LazyLoader approach has proven reliable in practice - all fell nicely into places and is in line with how import machinery works.
And for local imports new benchmarks show it is faster.

I’ve enabled global lazy imports in my shell RC. Everything is lazy from now on.

For those hoping for a PyPI package: there won’t be one - this wasn’t my intent and I would only make effort if it had impact on CPython.

I engaged with the PEP process in good faith, provided evidence and working alternatives, and was told my concerns were “noted” but would not change the outcome.

I’ve chosen to continue this work privately.
If you have serious interest contact me.

@moderators, could you please close this thread?