Note that Annotated and Annotated[int, ""] are considered incompatible with type[T] by both mypy and the latest version of pyright.
However, when used in an traditional type alias, mypy does not always flag this as an error.
AnnotatedOptInt = Annotated[int | None, ""]
takes_type(AnnotatedOptInt) # Error
AnnotatedInt = Annotated[int, ""]
takes_type(AnnotatedInt) # Mypy generates no error here
I’ve been working to tighten up pyright’s error detection for special forms, and the latest version now generates errors for the last case above. Should it?
I think this should be an error. An object that is compatible with type[T] should be an instance of type, and Annotated[int, ""] is not an instance of type.
I think underlying issue is lack of TypeForm[T]. I’d be happy to use TypeForm[T] for functions that do runtime type introspection and can handle special forms, but at moment type[T] is best thing I have. There’s no good way to write code like this today,
where from_config can handle special forms correctly. Ideally it’d be TypeForm[T] as input type. So tightening this does fix some false negatives, but there is code that liked laxer pyright handling and did handle special forms.
Interestingly, Annotated appears to call through to the constructor of its first type argument, so instantiation via a constructor call actually works.
MyList = Annotated[list, ""]
print(MyList) # Prints typing.Annotated[list, '']
print(MyList([1, 2, 3])) # Prints [1, 2, 3]
It apparently doesn’t call through to other magic methods though. For example, it doesn’t work for instance checks.
On closely related topic which special forms are passed to type[T] for my codebase,
Union (and Optional) is most common. Code like from_config(Union[dict[str, int], list[int]], obj).
Literal is pretty rare but occurs.
Annotated/Final/ClassVar/Required/NotRequired I pretty much never pass to type. Final/Required/NotRequired/ClasVar all share property they don’t normally make sense except for attribute declaration. Annotated I can see making sense, but rarely has come up in this kind of context for my codebase.
So only special form I’d hope we stayed lax til TypeForm exists is Union.
Edit: One notable function in the standard library that’d make sense to use TypeForm/lax interpretation is cast.
Similarly, I have a code base for Pydantic which heavily uses this pattern:
from typing import TypeAlias
SomeType: TypeAlias = Annotated[<some simple type>, <some pydantic class representing a restriction>]
There are at least around ~100 such cases and I also don’t think I pass any of them to type.
Just some data to help you make a good decision.
This made me realize I don’t understand at all how metaclasses belong/behave in the type system and neither the typing spec nor PEP 484 explain that well (at least what I could find). Before this I would have assumed that all types were of type type transitively but I take it that even if a metaclass is type type that doesn’t mean it’s subclasses of type type? Were can I read more about this and how it relates to the Python type system?
Metaclasses are subclasses of type, so instances of metaclasses are still instances of type.
Not sure if this is what’s causing your confusion, but worth noting that the meaning of the word “type” is overloaded here. In Python generally, it refers to the type() of a variable, which is always an instance of the class type. But the word can also be used to refer to members of the “type system”, like unions. int | str (the union of int and str) is a type in the second sense, but not in the first sense.
If you wanted something compatible, it would need to be something that was assignable to Annotated[type[int], ""] by my understanding of Annotated. The first element being the type system’s annotation use.
I probably should have provided some examples. I expect all of the things you said fail to fail.
Consider this:
def takes_type[T](t: type[T]) -> T:
return t()
takes_type(1) # should error
takes_type(int) # should work
x: Annotated[int, ""] = 1 # works
y: Annotated[type[int], ""] = int # works
z: Annotated[int, ""] = int # errors
final_x: Final[int] = 1 # works
final_y: Final[type] = int # works
final_z: Final[int] = int # errors
The special forms there don’t expect a type in the way type is used in annotations. They produce something that expects a value of that type.
From my perspective here, this is all working as intended and pydantic appears to want to use annotated in a way that isn’t compatible with its definition. (pydantic appears to be trying to use annotated to attach an annotation to a type rather than to attach extra non-type information to an annotation, this isn’t inline with my understanding of annotation or these special forms)
These special forms should not behave as if they are type even if they produce something that is a subtype of type at runtime, they exist specifically to express concepts about typing.
PEP 593 introduced Annotated as a way to attach runtime metadata to types without changing how type checkers interpret them. Pydantic takes advantage of this to allow you to create types that are identical to the original type as far as type checkers are concerned, but add validation, serialize differently, etc.
From the README of the annotated-types library (which uses Annotated the same way Pydantic does):
PEP-593 added typing.Annotated as a way of
adding context-specific metadata to existing types, and specifies that Annotated[T, x]should be treated as T by any tool or library without special
logic for x.
Pydantic’s docs and annotated-types both appear to be incorrect here about what PEP 593 has actually specified.
Annotated[T, x]should be treated as T by any tool or library without special
logic for x .
This doesn’t follow from the PEP or its examples, but I can see how that could happen. There have been many instances where people are mixing up that it should be the same as T in an annotation with it should be the same as T everywhere. int has specific meaning in an annotation which is different to using it at runtime. This difference is in fact the motivation for the form type[T], as we need a way to spell when we actually want the type, not a value of that type
From PEP 593:
Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the Annotated type. A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance() ).
This is quite clear to me that the PEP is only considering encountering Annotated in an annotation, this also tracks with the fact that all of the examples of assigning Annotated are being used as type aliases that are later in an annotation.
I think the best possible way forward here would involve allowing functions that operate on more complex info to specify this rather than making it the job of the type checker here to reconcile special forms in a way that may not compose well long term.
To be a valid way to pass inspected Annotations or otherwise runtime accessed instances of Annotated around. (Edit: TypeForm seems to be potentially more appropriate here)
This is explicit in taking a runtime use of Annotated (rather than Annotated in an annotation that’s inspected) and using the inner type. Doing it in implicit ways will have negative consequences when looking at how this would compose with other features.
For instance, Intersection.
Acccording to PEP 593, things which don’t understand the purpose of things in Annotated should just ignore them. If we go with allowing this implicitly as a runtime type rather than only allowing it when it can be done unambiguously, we run into situations where being consistent with definitions has negative outcomes:
The type system doesn’t know the purpose of “…” or how to combine them or not, by specification, it would just be Intersection[T, V]. Clearly, the correct use from a user would require something like Annotated[Intersection[T, V], "... and ..."] It’s quite reasonable to require more explicit handling by advanced libraries doing runtime use. (note the more explicit form suggested above here only changes something for Pydantic’s types, not for users of Pydantic)
I don’t think the second usage is incorrect as such; it’s a reasonable use of Annotated in a runtime context. What’s problematic though is that pydantic expects Annotated[int, "something"] to be compatible with type in an annotation. They need the long-awaited TypeForm special form instead.
I started this discussion by asking “Is Annotated compatible with type[T]?”. Here’s a related question: “Is Annotated callable?”
Nothing that I can find in any documentation (either in PEP 593 or in the official Python documentation) indicates that Annotated should be callable. In fact, Annotated by itself is not callable (Annotated() raises an exception), but when it is specialized (e.g. Annotated[int, ""]) an instance of the _AnnotatedAlias class is created at runtime. This object happens to be callable by virtue of subclassing from _GenericAlias. I suspect this was an unintended behavior — effectively a bug in the implementation.
When _AnnotatedAlias is called, the behavior depends on the first type argument of Annotated. If the first type argument happens to be an instantiable type form, the constructor is called, and an instance of that type is returned. For example, Annotated[tuple, ""]([1, 2, 3]) returns a tuple object.
However, many type forms are not instantiable. For example, Annotated[int | str, ""]() raises an exception.
If this is an unintended and undocumented behavior, my inclination is to file a cpython bug and suggest that this unintentional behavior be eliminated — perhaps by overriding the _AnnotatedAlias.__call__ method with a method that always raises an exception.
I don’t think Annotated should be callable. Allowing this just because people relied on unintended and undocumented behavior in a typing special form will drastically increase complexity with currently in-progress and awaited type features, and the people relying on it can access the inner type
For instance, with Intersection, what’s the behavior Intersecting a Callable with Annotated if Annotated is Callable? This may have many unintended effects, not all of which may be obvious now and does not appear to have been considered.
Comments and suggestions are welcome. For typos or wording changes, you can comment directly in the PR. For more substantive feedback, please post to this thread for more visibility.
Would it be possible to add TypeHint[T] in the same move
That’s a much bigger topic that would require a full design proposal — and potentially a full PEP. It’s definitely out of scope for this minor clarification to the behavior of Annotated.
There has already been some discussion about a TypeForm annotation (which I think is a better name than TypeHint). If you would like to champion such a proposal, you could build on those earlier discussions.