I don’t see any suggestion in PEP 747 that an arbitrary variable typed as TypeForm can then be used as a static type annotation. PEP 747 addresses, in a sense, the “opposite” problem: recognizing the runtime values resulting from runtime evaluation of a valid static type expression, and allowing runtime code to accept and operate on those values.
2 Likes
One possibility that would still permit many type-mapping style use cases, but without requiring type expression semantics to depend on full type evaluation of arbitrary value expressions, would be to allow custom __class_getitem__ but require the slice to be a valid type expression. That is, both the argument and the return type of the __class_getitem__ must be a TypeForm.
This would not quite allow the AstroPy use case as it currently exists, because it allows Quantity["length"] and "length" is not a type expression. I think it could allow Quantity[u.km].
(FWIW the AstroPy use case is also happy with the much weaker pyright version of this, where the type checker can ignore the slice and the __class_getitem__ return type entirely, and just always infer the receiver class object as the type. But I don’t think this is a good version of the feature – it’s confusing/unclear why some parts of the annotation are ignored.)
What about comprehensions? Subscripts of arbitrary objects? Bare strings? Any other construct that is syntactically permitted in annotations but is not valid in type expressions? If those should all be disallowed as well, then I think this ends up with my suggestion above, that the slice must be a type expression. If you would just disallow function calls, why are they special?
PEP 827 extends the language of type expressions substantially, but it specifically and intentionally does not open the door to arbitrary value expression inference being a requirement to understand the static type meaning of a type expression. It actually goes to great lengths to avoid this.
Rather than restrict the slice to being a typeform, can we go slightly more relaxed to “type form or literal value”? (literal value here meaning how python the language uses that, not typing.Literal)
That still restricts it to a set of things that should be reasonable for typecheckers to handle, but does enable the various ergonomic improvements other libraries are creating using this already.
1 Like
Hmm, I’m not sure what exactly you mean with “slice” here. Are you talking about something shaped like
def __class_getitem__(cls, tp: TypeForm[_] | tuple[TypeForm[_], ...], /) -> TypeForm[_]: ...
or would tp also accept slice[TypeForm[_] | None, TypeForm[_] | None, TypeForm[_] | None]?
By “slice” I just meant “whatever is inside the subscript brackets”. Sorry, terminology borrowed from the Python AST, maybe not the clearest. So yes, that would imply that the __class_getitem__ could effectively only accept typeforms and tuples of typeforms.
That version would not gain the same concision benefits as your hypothetical numpy example above. @mikeshardmind’s extension would help, since a slice literal (here I mean the syntactic construct with a colon that constructs a runtime slice object) is a literal and would therefore be allowed.
2 Likes
Phrasing it that way, I don’t think function calls are any more special, and I would be fine allowing them. The only issue with any of these is that the new annotations aren’t yet on every supported python version. Once those are, all of these are resolvable without executing the expressions, even by runtime typecheckers. Each of these are also possible to statically analyze.
The type manipulation pep wants to be able to do all of these things as well, so if you think it should be ruled out, I’m curious if you think it needs to be ruled out forever, or just is something that you don’t want to open up right now.
I like the middle ground of TypeForm as input to __class_getitem__ with a TypeForm output. I don’t need the literal concession, and I’m not sure if that makes it harder in some unacceptable way.
I think to support string values, it requires the __class_getitem__ to be annotated as accepting it and not accepting an arbitrary TypeForm or else it’s not possible to statically determine if a string is a forward reference or a value. Other literals don’t have this issue as far as I can tell.
Your example does in fact work in pyrefly (the link should open a slightly modified version of your snippet):
Though it violates the “typeforms only” rule that seems to be emerging as consensus here.
That’s pretty cool! But unless all type-checkers support this, I’m afraid we won’t be able to use this in NumPy.
I’m not sure what consensus you’re talking about, because so far I haven’t seen any objections against @mikeshardmind 's literal value extension.
To be clear, I’m not concerned about difficulty of implementing this in ty; I think it would actually be pretty simple. The restricted “typeforms only” or “typeforms only plus literal types” versions would be a bit more work, not less (though also not bad.)
I think the full version is a significant increase in the complexity of what can be represented in type annotations, and brings a lot of power to customize the syntax and expressiveness of type annotations. Combined with PEP 747, it effectively gives the full expressive power of Python itself to writing type expressions (as long as you wrap it in a subscript). Used well, this can potentially make some annotations less verbose and nicer to read.
The downside is that it can also make type annotations a lot harder to read. Today, if Quantity is a generic class, then its subscript is always a generic specialization, which is always a type expression, meaning "length" in Quantity["length"] must be a stringified forward reference. With this proposal, you really have no idea what a subscript in a type annotation means, without inspecting the definition of the receiver class to see whether it is generic or has a custom __class_getitem__, and if the latter, what it does.
There is also an inherent ambiguity with stringified forward references to allowing bare strings in this position. Type checkers would have to use contextual inference based on the annotated parameter types of __class_getitem__ and potentially try both options to see whether a bare string is supposed to be a stringified type or simply a string.
This addition would be awesome, especially with the overloads!
It would be a perfect addition for my proposal to add specialized versions of Templates and would probably simplify it a lot too!
This is the main part that worries me, even for the more restricted version that only accepts literal values in addition to TypeForm. Type checkers are not the only static analysis tools that need to know whether a string is a forward reference or just a string. Some linters like ruff, pyflakes and flake8-type-checking need to know it as well, so they don’t emit too many false negatives/positives for unused imports, references to undefined symbols etc.
If we make this ambiguous we make it impossible to detect some of these issues without complex multi-file-analysis or configuration. It’s not really a clear improvement in ergonomics if you end up needing to litter your code with pragma comments to ignore false positives or need to know enough details about configuring your linter to get rid of the false positives reliably.
While it seems somewhat reasonable to expect type checkers to have to go the extra mile to support this, since they already need to be able to support it anyways in other places, it doesn’t seem as reasonable to inflict that burden on more simple static analysis tools, since they’re not type checkers.
1 Like
I had Claude Opus 5 investigate exactly how much it could simplify in NumPy’s bundled stubs and in scipy-stubs. After grinding at it for a while, it concluded:
numpy ~52-59%, scipy-stubs ~38-43% LOC reduction. Both lose ~2/3 of their
@overloads.
This seems very plausible to me. These are not small gains either, because e.g. scipy-stubs currently counts 97,945 LOC (according to scc, excluding comments and whitespace, and excluding type-tests).
I’d argue that these overload-heavy stubs are significantly harder to read than what they’d look like if they could be simplified using these type constructors.
Also note these are simplications that could be done without the literals extension. Those I’m mostly interested in to make shape-typing more ergonomic for downstream users. So it’s something that would certainly be very nice to have, but without it I’d already be very happy to say the least.
3 Likes
I wonder if as an additional restriction, one could say __class_getitem__ should only return subtypes of (or rather, types that are compatible with) the class that __class_getitem__ is on. Your original reported case would then still be allowed because Annotated[Quantity, u.km] is compatible with Quantity.
(This restriction seems orthogonal to the input-restrictions of “only typeforms”, “typeforms + literals”, or “typeforms + literals but no string literals”.)
I’ve also just remembered this old thread on typing-sig where someone from numpy wanted to return np.types.Float64DType from the __class_getitem__ in np.dtype[np.float64] (where Float64DType is a subclass of dtype). This would then also be allowed.
We could solve this and other issues of forward reference ambiguity by deprecating the use of strings for forward references. They shouldn’t be needed anymore with the various improvements to how annotations are stored and resolved, but we wouldn’t be able to immediately use this without the ambiguity unless we also gave a way to indicate a project wasn’t using bare strings as forward references.
Sort of re-raises some prior concerns about how to effectively handle phasing out things we know cause problems of some sort now that we have multiple typecheckers in the ecosystem, and these things may be in public API (so no, a typechecker setting isn’t enough)
I’d love to deprecate forward references, and recent improvements in Python (native type aliases and generics in 3.12; deferred annotations in 3.14; and lazy imports in 3.15) mean there are a lot fewer places where they’re needed. Still, we’re probably not ready for it. For one, Python versions without these features are still supported for a few years.
There are still some places where types are evaluated at runtime and you might need forward references in case of circular references: NewType definitions, cast() calls, functional TypedDict definitions (which are needed with certain keys), and base classes (for defining something like class str(Sequence["str"]): ..., if we were to define that in Python).
A useful intermediate step could be to deprecate nested forward references, so allow x: "list[str]" but not x: list["str"]. Nested forward refs are where much of the complexity comes from, because they mean that anywhere in a type expression a string may suddenly be a type. We do need them for the base class case, but perhaps that is rare enough in practice that it’s not worth the trouble.
1 Like
The NumPy and scipy-stubs use-cases (Should non-generic classes with `__class_getitem__` be subscriptable in a type expression? - #32 by jorenham) would not be possible with this restriction, so I’d rather not do this.
Also, I personally think that it’s the responsibility of the library authors to avoid their typing utilities from becoming difficult to read. And if done correctly, I’m sure it can significantly improve readability, rather than hurt it.
Even if this would be possible, I don’t think I would do this, because I expect this to be more confusing than helpful in practice.
We might be able to avoid much, if not most, of the confusing ambiguity if we disallow custom __class_getitem__ for generic classes (i.e. treat Generic.__class_getitem__ as @final), while still allowing custom __class_igetitem__ for non-generic classes. That way generic classes will remain predictable when subscripted, while opening the door to user-definable polymorphic type(-form) constructors. And since these type constructors are a new concept, we can simply disallow string-based forward references in them, so that e.g. ToQuantity["length"] will be unambiguous (where ToQuantity is a type constructor, not a generic type).
1 Like
That restriction unfortunately only really helps tools that already have to perform deep multi-file static analysis, like type checkers, since you need to know what’s being subscripted, everyone else will only be able to determine whether ToQuantity is a Generic in simple cases.
We already suffer from this problem for subscripts occurring in value expressions, so we already fail to detect some forward references. But since forward references are very rare in value expressions, this is less of an issue, than this would be.
@Jelle’s suggestion of deprecating nested forward references would help a lot more and even reduces implementation complexity in the long term, compared to what we have to do today, once we can stop supporting nested forward references.
But I’m not fully convinced we can actually stop supporting them, unless we provide an alternative way to spell nested forward references via explicit use of ForwardRef, so something like Foo[ForwardRef("Bar")] can be used, when there’s no other way to spell things correctly.
Why would "Foo[Bar]" not be a correct spelling?
Sorry if I was being unclear, that is also a correct spelling, but depending on what you pass it into, it might not be supported. E.g. a function that expects a type[T], so it needs an actual runtime type, but Bar is not (yet) resolvable, so it needs a forward reference. And type checkers need the subscripted type in order to infer the correct return type.
To be clear this is an uncommon edge-case, but just because it’s uncommon, doesn’t mean we can just stop supporting it altogether.
1 Like