Should non-generic classes with `__class_getitem__` be subscriptable in a type expression?

Typing
1

The AstroPy library documents a suggested pattern to type-annotate using an expression like Quantity[u.km]. Quantity is not a generic class, but it implements __class_getitem__ to return Annotated[Quantity, u.km] – effectively a shorthand for an Annotated annotation.

AstroPy itself does not type-annotate the return value of Quantity.__class_getitem__ and doesn’t seem to publish type stubs, but recently a someone inquired about how to write type stubs for this pattern that will work in ty.

ty requires a class subscripted in a type expression to be defined as generic (that is, have PEP 695 type variables, or inherit typing.Generic, or typing.Protocol with type variables), otherwise it emits invalid-type-form. Mypy and zuban similarly error on this example:

from __future__ import annotations

class U:
    def __class_getitem__(self, value: int) -> type[U]:
        return U

def _(x: U[0]):  # error, can't subscript a non-generic class in a type expression
    reveal_type(x)

Pyright and pyrefly allow this without error and reveal U on the last line, though their support differs in scope. Pyright always reveals U on the last line, even if the __class_getitem__ is modified to return type[int] – it doesn’t actually respect the return type. Pyrefly reveals int instead with that return type change.

My reading of the current spec language is that it strongly suggests that this pattern should not be allowed. The type expression grammar is explicitly restrictive, and makes no allowance for subscripts in type expressions to be something that is not itself a type expression. 0 in the above example is clearly not a type expression.

I think the main risk in explicitly allowing this is that it opens the door for confusing/problematic mismatches between runtime behavior and what the type checker can understand. For example, consider this code:

class Factory:
    def __class_getitem__(cls, key: int) -> type[object]:
        return int

x: Factory[0] = "hello"

At runtime, Factory.__class_getitem__ returns int, but since type is covariant, it can be annotated as returning type[object]. So the annotation actually evaluates to int (and runtime annotation introspection will see int), but the type checker can only see object, and will allow the assignment of "hello".

I would be interested in hearing especially from pyrefly and pyright maintainers about your reasons for permitting this, and from anyone with opinions about whether this should be allowed or not. I think it would be useful to clarify and specify this one way or the other. It would not be hard for ty to allow this, if we decide that’s the right path. I assume the __class_getitem__ would be required to return a valid TypeForm (which type[...] is), or else it should be an error? (Though currently pyright and pyrefly are also fine with no return annotation at all, in which case they both infer the subscripted class type.)

5 Likes
2

I would say it should be allowed, and that the return type annotated should be used, no guessing (that means using Any in the absence of an annotated return type, not assuming generic behavior)

I’m not concerned about the potentially confusing outcomes related to covariance of type, people manually implementing __class_getitem__ rather than relying on the default generic semantics are already in advanced usage territory.

3 Likes
3

I took a quick look at the history of __class_getitem__ support in Pyrefly, and it doesn’t look like we had much discussion on the topic. The relevant issues and PR are:

Basically, we started from the assumption that we ought to support __class_getitem__ because it works at runtime and proceeded accordingly.

IMO using __class_getitem__ as basically a way to get static typing to compose with other use cases for annotations, as AstroPy seems to be doing, is reasonable, and I’d be in favor of allowing it.

2 Likes
4

I also think it makes sense to allow this, as long as the __class_getitem__ returns some kind of TypeForm. For example, in this variant:

from __future__ import annotations
from typing_extensions import TypeForm

class U:
    def __class_getitem__(self, value: int) -> TypeForm[int]:
        return int

def _(x: U[0]):
    reveal_type(x)

I’d expect to reveal int (as pycroscope does, though I can’t say I consciously designed for it).

As @mikeshardmind suggested we can use Any if the __class_getitem__ is unannotated. Using one that returns a non-TypeForm should be an error (e.g., if it is annotated as -> int).

1 Like
5

Thanks all! Sounds like the balance of opinion is to allow this.

In considering it further, my hesitation is that this weakens the distinction between type and value expressions, and allows arbitrary dynamic code to determine the type spelled by a type expression.

Should there be any limits on what code can appear inside the subscript in this case? Are arbitrary Python value expressions valid there? Including e.g. calls (normally banned in type expressions), other subscripts (of arbitrary objects, not necessarily classes with __class_getitem__), etc? Or should there be some limits? If so, what should the limits be? (As far as I can tell, pyright and pyrefly do not apply any limits.)

We already support arbitrary dynamic Python expressions in value expressions, via Annotated – but that feels a bit different, since a type checker does not even need to evaluate those expressions, it can ignore them. This would be the first time (AFAICT) that we allow arbitrary dynamic code in a Python type expression to determine the type spelled by that type expression. (Pyright’s version of “supporting” this does not actually do that, but pyrefly’s does, and that’s the one it sounds like people would prefer.)

This makes me a bit uncomfortable. I’d have to say that my preference would be not to open this door.

6

My understanding was that whilst this might allow a dynamic object to be used as a type, since the return type of __class_getitem__ has to be declared statically, it’s not really making it more dynamic?

Though I guess that might cause different behaviour between static type checkers which don’t narrow the __class_getitem__ return type based on the code inside, and dynamic ones which do.

1 Like
7

Quantity is a subclass of numpy.ndarray which is a subclass of Generic in the stubs, but not at runtime:

>>> import numpy as np
>>> np.ndarray.mro()
[<class 'numpy.ndarray'>, <class 'object'>]

IIUC, a type checker would consider Quantity as Generic and have inherited 2 type parameters with defaults.

Question
If u.km is not assignable to tuple[int, ...] [1], is a type checker expected to ignore the issue because of a __class_getitem__?

The runtime implementation makes me think they’d be better served by (PEP 835: Shorthand syntax for Annotated type metadata).

Using preserving-units as an example, something like this possible today and not much of a mouthful [2] if you alias some symbols:

Show a few generics 
from typing import Annotated as A, Literal as L

import numpy as np


class Quantity(np.ndarray): ...


class Unit[T]:
    def __init__(self, arg: T) -> None:
        self.arg: T = arg


class PhysicalType[T]:
    def __init__(self, arg: T) -> None:
        self.arg: T = arg


type m = Unit[L["m"]]
type length = PhysicalType[L["length"]]

>>> A[Quantity, m]
typing.Annotated[__main__.Quantity, m]


>>> A[Quantity, length]
typing.Annotated[__main__.Quantity, length]

  1. _ShapeT_co’s bound ↩︎

  2. which seems like the goal for defining __class_getitem__ in the first place ↩︎

8

That’s because numpy.ndarray is defined in C. It does, however, implement __class_getitem__, which returns a types.GenericAlias:

>>> import numpy as np
>>> np.ndarray[tuple[int], np.dtype[np.float64]]
numpy.ndarray[tuple[int], numpy.dtype[numpy.float64]]

So for all intents and purposes it’s a full-fledged generic type, both at runtime and in the stubs.

But that doesn’t mean that this should also be the case for all of its subtypes. For example, class MyVec(np.ndarray[tuple[int], np.dtype[np.float64]]): ... is not a generic type.

1 Like
9

Ah I just came back to make an edit that I’d confused myself with this part

But you’ve beaten me to it :wink:

Perhaps the ndarray part isn’t relevant because of the “inherited” defaults

10

I wouldn’t support it if it required this, but as @dr_carlos said, using the declared return type of __class_getitem__ prevents it from being arbitrary dynamic code that typecheckers are forced to evaluate, it still has to follow the normal semantics of function return type annotations.

specifying/requiring inference behavior is something I ruled out, stating that it should use Any in the absence of an annotation. A specific type checker applying inference behavior here is still fine if it only does it in local inference, not for library code, but really not ideal IMO

2 Likes
11

@dr_carlos @mikeshardmind This proposal does allow arbitrary dynamic value expression evaluation to determine the meaning of the type expression, unless we also add an explicit rule forbidding __class_getitem__ from being generic or overloaded.

Pyrefly accepts this code, and conditionally interprets the type of x as int or str depending which overload is selected:

from typing import overload

class C:
    @overload
    def __class_getitem__(cls, o: int) -> type[int]: ...
    @overload
    def __class_getitem__(cls, o: str) -> type[str]: ...
    
    def __class_getitem__(cls, o: object) -> type[object]:
        return type(o)
    
def returns_int() -> int:
    return 1
    
def _(x: C[returns_int()]):
    reveal_type(x)
1 Like
12

I don’t see an issue with this example, the intended type can be evaluated statically using only the annotations. returns_int doesn’t need to be executed here; this isn’t arbitrary execution.

1 Like
13

I’m not claiming that actual Python runtime execution would be required; obviously not. But this requires type evaluation of arbitrarily complex Python value expressions in order to understand the meaning of a type annotation. This is something the Python type system has, until now, explicitly avoided. It means that the type expression grammar is no longer sufficient to understand the meaning type expressions.

It’s certainly possible for type checkers to do this, but it has consequences both for type checker architecture and for the human comprehensibility of type annotations as a static type system. I don’t think it’s a bridge we should cross without careful consideration.

1 Like
14

I think the use cases for this are pretty clear, and it’s an obvious extension of what type checkers already understand.

I think any “this is hard for users to understand” argument comes down to “library authors should be responsible with this to cases that are naturally understood”. The case prompting the discussion is naturally understood.

When it comes to tooling complexity/architecture, tooling is for the benefit of users, the use here is clearly a more ergonomic composition for users, and if we determine that because some tools might not want the complexity, users should have a worse experience, I think we’ve lost sight of the fact that the complexity for tools is precisely to create a better experience for users.

1 Like
15

I’m pretty strongly against mixing inference and static type definitions and I don’t like the fact that this works in Pyrefly. I used to allow more dynamic type definitions in Zuban with type inference and it was a really bad idea, because there can be very weird cycles that are hard to understand for both users and type checkers (lots of bugs).

I can kind of see us allowing non-generic TypeForms as a return of __class_getitem__, where we can ignore the expression like in Annotated, but I’m also not a fan of that and think that it’s a bad idea. I also think that x: C[returns_int()] looks really bad.

1 Like
16

I’d just keep the restriction on not having function calls in type expressions, while allowing the annotation on __class_getitem__ to declare the actual resulting TypeForm. I agree with @mikeshardmind about this being something that library authors should be responsible about, and also with what @Jelle wrote about still restricting the result to type forms.

I have no issue with AstroPy’s use, the part that goes beyond what I think should be allowed in your latest example is allowing the function call.

Even with function calls and only using the annotation in the type system, this would be tamer than other open proposals about type transformations on the effect of:

though the grammar was also never sufficient alone to understand the meaning of a type expression. Some symbols have their own rules, and they aren’t keywords encoded into the grammar.

17

Numba uses a pretty neat syntax for declaring N-dimensional arrays of certain dtype when specifying the signature of a @jit function, where you write e.g. float32[:] for a one-dimensional single-precision array, or int8[:, :] for a two-dimensional array of 8-bit integers (docs). I first thought that this might be a good use-case for allowing non-generic __class_getitem__, but I then realized that these numba types like numba.float32 are instances rather than types, so they can just use __getitem__.

However, for a long time I have wondered whether it would be somehow possible to also use a similar terse syntax in in NumPy, i.e. being able to write np.int8[:, :] instead of the painfully verbose np.ndarray[tuple[int, int], np.dtype[np.float32]] [1]. But this always seemed unrealistic to me, and thought that it would require numpy-specific special-casing to be accepted as part of the typing spec.
This thread made me realize that this is actually not as far-fetched as I initially thought. Because by allowing __class_getitem__ to also be used to return TypeForms that are allowed to be used in annotations, then we’d effectively be able to define custom dependent type constructors. For the shaped dtype constructors in NumPy, that could look something like this:

type Dim = slice[None, None, None]
type Float32ND[ShapeT: tuple[int, ...]] = ndarray[ShapeT, dtype[float32]]

class float32:
    # --snip--

    @overload
    def __class_getitem__(cls, key: tuple[()], /) -> TypeForm[Float32ND[tuple[()]]: ...
    @overload
    def __class_getitem__(cls, key: Dim, /) -> TypeForm[Float32ND[tuple[int]]: ...
    @overload
    def __class_getitem__(cls, key: tuple[Dim, Dim], /) -> TypeForm[Float32ND[tuple[int, int]]: ...
    @overload
    def __class_getitem__(cls, key: tuple[Dim, Dim, Dim], /) -> TypeForm[Float32ND[tuple[int, int, int]]: ...

    # --snip--

    @overload
    def __class_getitem__(cls, key: tuple[Dim, ...], /) -> TypeForm[Float32ND[tuple[Any, ...]]: ...

I realize that this wouldn’t be easy to implement in type-checkers, to say the least. But it would make for an immensely powerful addition to Python’s type system, as it will allow us to overload types themselves. It would also be direct solution to What would it take to implement type mapping for generics? without requiring any language changes whatsoever.

Clearly, this __class_getitem__ + TypeForm stuff will require more research and discussion before going forward with it. So for now let’s just consider it a “potential use-case” that allowing non-generic __class_getitem__ definitions will enable. Then after that, we could discuss this TypeForm-driven dependent type constructor stuff separately.

And to be clear: it’s not my intention for this to be a serious proposal, just a potential one. I’m only bringing it up here because it requires us to allow __class_getitem__ in non-generic-type contexts. Then, if we decide on this, I’ll open a thread where we can discuss this in more detail [2].

  1. This could be made less painful using type aliases, but then you’d end up with a huge number of type aliases (one for each combination of dtype and rank of up to 3 or 4 or something), which clearly also isn’t ideal. ↩︎

  2. That is, unless it turns out not to be feasible, after all. ↩︎

3 Likes
18

I started poking around at what can be done with TypeForm today, and I realized that mypy, pyright, ty, and zuban all error on this:

from typing_extensions import TypeForm
X: TypeForm[int] = int
x: X  # Error: variable not allowed in type expression
x = 0

(Pyrefly also errors but on the next line, which seems like a gap in our implementation of TypeForm rather than any deliberate decision.)

So now I’m curious: is this restriction on the use of TypeForm intentional? For me, I think that changes whether this use of __class_getitem__ (with some limitations on what’s accepted and returned) feels like a natural extension of existing capabilities or opening the door to something totally new.

1 Like
19

pyright presents the same error on the following, and the error predates TypeForm being accepted. The error message also explicitly states it is about variable use in a type expression, which the propsed relaxation on __class_getitem__ doesn’t require relaxing.

X: type[int] = int
x: X
x = 0
20

Yes, the fact that the error predates TypeForm is why I’m wondering whether it was extended to TypeForm intentionally or essentially by historic accident.

I was trying to figure out how novel it would be to say that a value annotated as TypeForm can be used as a type by checking if TypeForm already works this way anywhere else.