# Deferred computation/evalution for toplevels, imports and dataclasses **URL:** \ **Category:** Ideas\ **Created:** [September 20, 2023, 7:10am UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173 "2023-09-20T07:10:25Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![RonnyPfannschmidt](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ronnypfannschmidt/32/1714_2.png) [@RonnyPfannschmidt](https://discuss.python.org/u/RonnyPfannschmidt)\ **Post date:** [September 20, 2023, 7:10am UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/1 "2023-09-20T07:10:25Z")
when investigating import times for pluggy and their effect on hatch, we took note that importing importlib.metadata, typing and inspect in a pretty way, it makes a import time difference that’s very noticeable - 0.003 s !! when deffered - 0.046 s when eager (rounding errors apply to the components) - 0.024 s from importlib-metadata - 0.012 s from typing - 0.004 s from inspect typing was only used for declaring type aliases and type annotations importlib.metadata was only used very late for entry-points currently there are quite some pain points for deferring those imports sensibly while importing - loss of sane usage of typeddicts (those need eager import of typing or utterly reprehensible hacks) - painful deferring/local importing of importlib\_metadata/inspect i would like to propose expanding the deferred calculation of just type annotations to also be applied for imported names and globals like classes with dependencies decorated functions ```python # excessively simplified example import typing import dataclasses @typing.final @dataclasses.dataclass class Test(): foo: int def make_test() -> Test: return Test() # only by the time this is called the imports should trigger and the class should be made make_test() ``` ---
**Author:** ![BrenBarn](https://avatars.discourse-cdn.com/v4/letter/b/74df32/32.png) [@BrenBarn](https://discuss.python.org/u/BrenBarn)\ **Post date:** [September 20, 2023, 7:17pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/2 "2023-09-20T19:17:41Z")
This would be a large change to Python’s runtime semantics. Currently, libraries can do anything they want on import (i.e., imports can have side effects); this change would mean anything that happens on import now happens at some future time which may be very non-obvious. This is even more true of class definitions with decorators, where for instance a decorator may add a class to some kind of global registry. To me the proposed change smacks of spooky action at a distance, which would be worse than Python’s current semantics, which are relatively straightforward (viz., the entire file is run top to bottom). ---
**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\ **Post date:** [September 20, 2023, 9:02pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/3 "2023-09-20T21:02:36Z")
Is this [_PEP 690_](https://peps.python.org/pep-0690/)? ---
**Author:** ![sirosen](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sirosen/32/8478_2.png) [@sirosen](https://discuss.python.org/u/sirosen)\ **Post date:** [September 21, 2023, 4:41am UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/4 "2023-09-21T04:41:22Z")
Would putting most of these modules’ attributes behind ` __getattr__ ` be worthwhile? With some basic profiling, the slowest parts of these modules could be isolated from the rest. I think there are some very minor overheads related to adding internal modules (more modules need to be loaded), but if the result is that `import typing` is nearly instantaneous, that could be a pretty big win. With deferred annotations as a related topic, the case seems strongest for doing this with typing. ---
**Author:** ![effigies](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/effigies/32/35835_2.png) [@effigies](https://discuss.python.org/u/effigies)\ **Post date:** [September 21, 2023, 1:05pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/5 "2023-09-21T13:05:45Z")
It does seem that, even if we’re not prepared to adopt PEP 690 because of the changes in semantics, stdlib modules that do not have side-effects could be lazily loaded, either through ` __getattr__ ` or something like [GitHub - scientific-python/lazy\_loader: Populate library namespace without incurring immediate import costs](https://github.com/scientific-python/lazy_loader/). Much of the code there handles cases where imports fail, so it could be stripped down for use in stdlib to something quite compact. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [September 21, 2023, 1:28pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/6 "2023-09-21T13:28:21Z")
Honestly, I’d just report these as performance bugs. That’s likely to be much easier to address than any sort of deferred loading proposal - as has been noted, that’s basically PEP 690, and it was rejected because the semantic implications were too great. ---
**Author:** ![DavidCEllis](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/davidcellis/32/9789_2.png) [@DavidCEllis](https://discuss.python.org/u/DavidCEllis)\ **Post date:** [September 21, 2023, 5:09pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/7 "2023-09-21T17:09:01Z")
I’m not sure you can always avoid the performance issue without some form of lazy loading inside the libraries or a backwards compatibility break. I’m not so familiar with `importlib.metadata` but I know `dataclasses` for instance pulls in `inspect` which is fairly significant\[1\]. However you kind of end up following a chain downwards. Is the issue that `inspect` is a heavy import and so `dataclasses` should avoid it or is it that `inspect` should be made faster, possibly by avoiding or delaying heavier imports itself (can the `ast` import be delayed for instance)… I think that both this suggestion and the rejected PEP690 proposal both have issues with changing existing behaviour in ways that can cause unexpected side effects. However, the `lazy_loader` module linked earlier and other methods I’ve seen seem like a lot of work to get around the lack of direct language support (and can be a pain when static analysis tools get involved). * * * 1. It takes about half the import time if you look at `-Ximporttime`, but if some of the other direct dependencies were removed from `dataclasses` they end up being imported by `inspect` anyway. ---
**Author:** ![AlexWaygood](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/alexwaygood/32/4722_2.png) [@AlexWaygood](https://discuss.python.org/u/AlexWaygood)\ **Post date:** [September 22, 2023, 9:50am UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/8 "2023-09-22T09:50:50Z")
FWIW, I’ve filed [gh-109653: `typing.py`: improve import time by creating soft-deprecated members on demand by AlexWaygood · Pull Request #109651 · python/cpython · GitHub](https://github.com/python/cpython/pull/109651), which cuts down the import time of `typing` by around a third according to my measurements. ---
**Author:** ![RonnyPfannschmidt](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ronnypfannschmidt/32/1714_2.png) [@RonnyPfannschmidt](https://discuss.python.org/u/RonnyPfannschmidt)\ **Post date:** [September 22, 2023, 5:28pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/9 "2023-09-22T17:28:03Z")
i wonder if something like ```py lazy import inspect @somedecorator lazy def foo(...): ... ``` would make it more allowable it would be nice if there was a namespace/dict variant that could contain lazy computed members it would help for type annotations, heavy imports one needs later (code surrounding that is horrendous) it would also help for delayed imports the alternative would be to set up something like apipkg to generate certain objects in that case there would be `.pyi` files with import specs and a code-generator would translate them to `.py` files with lazy imports (with the caveat, that class/function definitions would be disallowed, only imports and very basic constants ---
**Author:** ![DavidCEllis](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/davidcellis/32/9789_2.png) [@DavidCEllis](https://discuss.python.org/u/DavidCEllis)\ **Post date:** [September 23, 2023, 4:58pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/10 "2023-09-23T16:58:08Z")
> [@RonnyPfannschmidt](#): > > i wonder if something like > > ```auto > lazy import inspect > > @somedecorator > lazy def foo(...): > ... > > ``` > > would make it more allowable I think in this case I would still expect the decorator to be evaluated, and the wrapped function (or class) could be depending on what the decorator does? You could make a decorator wrapper that only does the import and work on ` __call__ ` or ` __getattr__ ` and replaces itself at that point. It wouldn’t work for something like `@typing.final` that needs to be there for static analysis but you could make a `@deferred_dataclass` decorator that works at runtime for instance. ---
**Author:** ![RonnyPfannschmidt](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ronnypfannschmidt/32/1714_2.png) [@RonnyPfannschmidt](https://discuss.python.org/u/RonnyPfannschmidt)\ **Post date:** [September 23, 2023, 5:12pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/11 "2023-09-23T17:12:44Z")
Basically only the expression that’s creating the decorator can be executed Else one would have to execute the just lazy defined Thing to apply the decorator ---
**Author:** ![sirosen](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sirosen/32/8478_2.png) [@sirosen](https://discuss.python.org/u/sirosen)\ **Post date:** [September 24, 2023, 3:18pm UTC](https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173/12 "2023-09-24T15:18:30Z")
Looking at this thread again, I think getattr laziness is quite powerful for these cases as a client as well as for potential use in the stdlib. I’ve only used the following trick at the top level of a package, but it can be used anywhere: ```auto import typing if typing.TYPE_CHECKING: from ._foo import Foo else: def __getattr__ (name): ... ``` For this to work, the import speed of typing is the most important thing. There’s a funny thing going on here if you wanted to defer the import of TypedDict because typing imports are slow. If `Foo` above is a TypedDict, then it would work just fine as an optimization if typing itself deferred everything except for the TYPE\_CHECKING variable. But presently, of course it doesn’t work well for that case. This use case may impact how maintainers think about the typing module. If this usage pattern is one we want to encourage, then typing could begin chunking slow imports into internal modules, to make TYPE\_CHECKING very fast to access. (There’s also the `MYPY = False` variable, which needs no import, if that’s still supported. But I’m not sure I like relying on that anymore, even if it does work.)