That’s pretty clever although I worry that this might break anyone who was relying on the real paths for things in typing. To be clear, I don’t know how that would actually work, it just feels like the kind of thing that someone is relying on somehow.
As a sidenote, it would be nice if the PR was set up such that typing.py -> _typing.py was a git mv rather than a massive diff.
That breaks (ironically) type checking, as now type checkers cannot statically catch typos like from typing import Anyy. It also breaks IDE completion. It might be solvable by adding __all__, not sure.
It’s one of the most important stubs in a way, because typing.py (ironically) is very hard to understand for type checkers—there’s a lot o runtime magic going on.
I do like the idea of lazy loading, but there is also an overall problem where, as soon as one of your import-time imports does an unguarded from typing import Any, you’re back to square one.
I wonder if typing could be split into some lighter, mostly orthogonal submodules, e.g.:
typing/
__init__.py <- Mostly __getattr__, maybe Any and a couple other ubiquitous imports
_base.py <- Any and NoReturn and other non-generics, commonly used
_generic.py <- TypeVarLikes and generics, somewhat less common
_runtime.py <- Forward reference evaluation and similar, rarely needed
(This organization is probably not a good one. I just had a quick glance through the typing.py module and it seemed like a plausible carving.)
Lazy loading via __getattr__ is a clever trick, but makes all other imports slower, and without careful management can break star-imports, dir(), the __module__ attribute on objects, etc. These can all be fixed, but it makes the ‘simple’ code currently more complex and harder to maintain.
Importantly, there’s nothing preventing a future contributor to typing.py reverting all of the lazy loading for ease-of-maintenence, so I’d not be confident relying on it as a downstream library/application author who needs to support multiple releases of Python.
Whilst we agree it has drawbacks, I continue to support the simple ‘add a constant to builtins’ approach, as the simplest and most straightforward.
Considering the actual change (if merged) would contain a comment explaining why the code exists as-such, wouldn’t that just be a borderline malicious/inappropriate change? I think you could say that about any singular change that doesn’t fail the test suite if reverted. (Which, I’d argue this discussion and prior ones are proof enough a test SHOULD be added to the “performance” part of the test suite that this wasn’t accidentally broken going forward)
Improving the import time of typing and other stdlib modules is generally a good thing that can be done any time without a PEP.
Improving import time is not exclusive with adding e.g. __typechecking__ as a magic variable that does not need to be imported though. Generally I would like “zero cost static typing” and I think that the cost of importing the typing module will never be zero.
There are also reasons for using TYPE_CHECKING that are not about import time so it is not going to go away and if it stays then a __typechecking__ builtin is a better spelling for it. Even better is if it is impossible to set it to true at runtime: that should never have been allowed with TYPE_CHECKING.
Agreed that the alternatives that have been shown so far aren’t really suitable for a “Rejected Alternatives” section. (And we should definitely make it faster - a quick look shows that annotationslib is 3-4x more expensive than typing itself, and is now imported by typing, so if we can remove that then there’s an easy win.)
However, I do think the PEP needs to make a stronger case that moving TYPE_CHECKING out of the typing module will actually (eventually) cause all code to never import typing.
The primary benefit is cited as being “won’t have to import typing anymore”, but most code I see using typing is using it unconditionally in annotations. As long as those have to evaluate successfully, importing typing won’t really be optional, so this PEP provides no benefit over status quo. And of course, everyone needs to change their code to get the benefit - as soon as one module hasn’t converted, there is no benefit.[1]
There’s a hypothetical future suggested where the compiler can entirely exclude code under an if TYPE_CHECKING block, but it’s still going to have to be parsed, so I doubt there’s a huge benefit.[2] But it also seems like something that can be tried before making a permanent change to how people write their code. Also worth checking with people who know the optimiser well whether they could detect the variable from the module at compile time (which I’d hope isn’t much more complex than detecting whether a builtin has been overridden).
So I don’t think the rejected alternatives are as important as the rejected status quo. And I’m not convinced we need to reject the status quo.
I’m still waiting to see if deferred evaluation of annotations takes longer than normal evaluation. But of course, when I care about perf and don’t have a choice but to add annotations, I just use string literals. ↩︎
Improving marshal performance is another thing we should “just do”, and would have much wider benefits than we’d get from excluding if TYPE_CHECKING blocks from .pyc files - the only case I can think of where we save time from not producing bytecode. ↩︎
It won’t cause all code to never import typing but if you write a simple Python script you can not import typing. If you want to not import typing in a library to bring down the import time of the library then you can etc.
There are already lint rules for this e.g. the flake8-typechecking ruff rules. This moves all of your typing-only imports into if TYPE_CHECKING blocks:
ruff check --select=TC --fix --unsafe-fixes
It does not currently move the imports from typing itself (Callable, Any etc) presumably because there is no point doing that if you first need to from typing import TYPE_CHECKING.
This both won’t and can’t happen currently, even with moving the symbol out. currently, the typing module contains more than just symbols that are useful only to static analyzers and introspection based tools (such as ORMs using annotations), but also a named tuple constructor (that has to be used to have typing support for named tuples) as well as decorators that have runtime effects (overload, final).
This can at least start the process for more libraries to reasonably not import typing, but we would need other larger changes that get more things split out of typing that aren’t just static info to be used in annotations, or for the ecosystem to stop using any of the things that are more than that in typing [1]
As to your footnote about string annotations for performance… I, unfortunately, have to agree with you there, but that ship seems to have sailed long ago. That would have put the cost on only those introspecting by forcing them to resolve the strings only if and when introspecting, and had (the future import of annotations was effectively this) a simpler implementation than that of 649/749
I for one don’t use namedtuples or dataclasses, but handwritten classes specifically to avoid these problems and to allow others to use my code without having this decision already effectively made for them. ↩︎
You can configure it to move typing imports as well, it’s just that typing is by default part of exempt_modules, mostly because the fix for TC001-003 will currently import TYPE_CHECKING from typing if it needs to add a new if TYPE_CHECKING block in order to move the import.
Ruff did however recently add support for:
TYPE_CHECKING = False
if TYPE_CHECKING:
...
And one of my goals is to make the fix for TC001-003 smart enough to use that idiom instead of importing TYPE_CHECKING from typing, if typing isn’t part of your configured exempt_modules[1].
it might also be possible for the fix to import TYPE_CHECKING from a first party module that is part of both exempt_modules and typing_modules↩︎
the repr of the type alias type is misleading here.
it would not be a stretch to have type import for typing only imports, this works exactly as one would expect in wrapping this so that you have to introspect to cause the import. it would also be interesting if type checkers understood this as-is.
(thanks to @Liz for some prior idea brainstorming and @Sachaa-Thanasius for bringing it up again in discord)
but I acknowledge that Josh can’t do anything about it.