PEP 835: Shorthand syntax for Annotated type metadata

PEPs
22

Doesn’t it? What if I do

MY_SENTINEL @ np.array([1, 2])

?

By the way, at the moment, @ usage on numpy arrays doesn’t seem to be checked in any way by type checkers. Probably because of these overloads:

2 Likes
23

My previous suggestion with the same idea of using @ for annotation moved the responsibility of handling matmul into the annotation object instead of attaching it to the type.

This would resolve these runtime concerns and improves backwards compatibility, at the cost of requiring libraries that use Annotated already to modify their types.

There would be a baseclass somewhere in the stdlib/typing_extension that implements __rmatmul__ (and __(r)or__)

2 Likes
24

It’s a good suggestion. It is a choice between two different sets of trade-offs.

Moving the logic to __rmatmul__ on the annotation object neatly sidesteps the NoneType issue. It also avoids touching core CPython types.

The main downside is the transition path. If we modify type, the new syntax works for all existing libraries immediately in Python 3.16. If we rely on __rmatmul__, we face a fragmented rollout. The syntax would only work as individual maintainers update their libraries.

It also breaks support for primitive types (str, ctypes…) as metadata.

If modifying NoneType proves unpalatable, this is pobably our best alternative.

1 Like
25

Side Note: I don’t think this is a good idea anyway because using something that looks like a type as an annotation is probably going to lead to confusion.

It’s not really a downside, it’s more of a sidegrade. Currently you need to wait for a python version to start using the syntax, but if it’s the annotation object you just need to wait for a library upgrade. Especially relevant if you want to support all active python versions. This variant makes that easier.

1 Like
26

+1

To me this is a step in the right direction, but specially because of what libraries will be able do with such an easy syntax.

1 Like
27

I think it (i.e. relying on __rmatmul__) breaks support for instances of primitive types (incl. type). And while int @ str definitely would be confusing, int @ “count” could be considered less so.
We would then need to use int @ Doc(“count”), which is clearer, but also slightly more verbose (although much less than Annotated[…]).
That limitation is a small price for having the @ shorthand and compatibility for None and others.

3 Likes
28

Is this a common pattern? I’ve never actually seen primitives used in Annotated.

What if we added typing.Metadata, which would be a base class for Annotated metadata. It can be special-cased such that typ @ Metadata(obj) (i.e. not a subclass) returns Annotated[typ, obj].

If we make it generic, it can also be an alternative to PEP 746:

class Metadata[T = object]:
    def __rmatmul__(self, typ: TypeForm[T], /) -> TypeForm[T]:
        return Annotated[typ, self]


class Le(typing.Metadata[int]):
    ...


x: int @ Le(10)  # no issues
y: str @ Le(10)  # typing error: `str` is not assignable to `TypeForm[int]`
3 Likes
29

I’m uncomfortable with using matmul in this way. At least with |, there was a strong correlation with set-theoretic types.

This also clobbers a valid interpretation of matrix multiplication involving types used for dimensional analysis (and while the code involved is private, I am aware of code that actually uses this)

1 Like
30

Heyy,
Coming from a machine learning background, I use the @ operator heavily for matrix multiplication in my daily work.

I completely understand that there is zero contextual overlap between matrix math and type metadata, but from an aesthetic and readability standpoint, I don’t think overloading @ just for a shorthand is the right move.

I’m not sure what the official policy is for expanding operators into typing, but personally, I feel this makes the code harder to reason about.
It’s just not intuitive at first glance; if I stumbled across this in a codebase next month without having read this PEP, I would have absolutely no idea what was going on. :sweat_smile:

On the other hand I totally understand that some new syntaxes might be hard to introduce but while they have been introduced people loved it!!!

It’s hard for me to imagine that I use thi syntax. :sweat_smile:

3 Likes
31

My most common use case of annotated is with strings for documentation. So x @ “definition of x” is a pattern I have hundreds of times at my work codebase.

This specific pattern is mentioned here PEP 727 – Documentation in Annotated Metadata | peps.python.org although that pep chooses not to use string directly due to backwards compatibility concerns. But the idea is still natural one. Annotated is for metadata about a parameter that’s easy to read at runtime. Documentation is pretty common metadata and that can just be represented with str.

32

Thanks for the feedback. You make a strong case for the library-driven transition path. Here is how I plan to adjust the PEP to address these points.

1. Dropping None modifications

I am dropping the proposal to add __matmul__ to NoneType. The runtime risks are too high. We also have precedent for None lacking native operator support (e.g., None | None fails at runtime).

2. Syntax and Formatting

I agree that int @ annotation can look visually jarring at first. However, type @ annotation consistently surfaces as the leading candidate syntax in typing discussions.

To improve readability, we can encourage writing it without a space: int @Field(...). I can work with tools like Ruff and Black to format it this way automatically when the @ operator is in a typing position.

3. Metadata Base Classes

Shifting to __rmatmul__ opens a much better door for static analysis.

In a recent private conversation, Eric Traut highlighted a core problem with Annotated: type checkers cannot tell what the metadata actually targets. They cannot distinguish if an annotation applies to the value itself (like bounds or shapes) or to the symbol binding (like ReadOnly). He shared an older document where he proposed an abstract base class (TypeMetadata) to give checkers a clean boundary.

We can adapt Eric’s idea to handle the __rmatmul__ requirement. I propose a phased approach:

  • In PEP 835: We introduce a base typing.Annotation class (and add it to typing_extensions immediately). Frameworks can inherit from this to support the @ syntax out of the box.

  • In a future PEP: We expand this architecture with typing.ValueAnnotation and typing.SymbolAnnotation.

This lays the groundwork for static type checkers to safely evaluate metadata. To be clear, this does not suggest or standardize any specific annotations today. It simply provides the structural foundation that tools need.

3 Likes
33

I’m not sure this is the best name. Currently, “type annotation” is used interchangeably with “type hint” (PEP 484 – Type Hints uses “annotation” more times than “hint”!). I understand why typing.Annotated was named that way, but I think we should try to move away from that ambiguity instead of embracing it.

7 Likes
34

I understand the (need to) compromise.

Not to object that, but just for clarity:

This approach (ad 3) would make @ ... an alternative syntax for a (growing and ultimately hopefully a very large) subset of Annotated[...] uses, i.e.:

  • in type1 @ ann1 (and type1 @ ann1 @ ann2, …), ann1 (and ann2, …) must be instances of subclasses of Annotation (however it will be named, as per @BenjyWiener)
    e.g. int @ Doc('number of items') @ Ge(0)

  • whilst in Annotated[type1, ann1] (and Annotated[type1, ann1, ann2], ann1 (and ann2, …) can be any objects (incl. strings, as per @mdrissi)
    e.g. Annotated[int, 'number of items', Ge(0)]

Is my understanding correct?

35

Thank you for reasons against my ideas!
I hope they make it to the Rejected Ideas section, to make the PEP more accessible to people like me, who aren’t that involved in typing :‍)

It would still conflict with runtime uses of __matmul__ on types, though.
For example, making a matrix library where Matrix_3x4 @ Matrix_4x5 gives Matrix_3x5 is probably not the best idea, but it is a place where I’m worried about typing further limiting the design space…

At that point, does it need to be @? Could you subclass Annotated instead?

username: Field[str, "User's login name", MaxSize(32), Unique]

and/or perhaps allow call syntax?

username: Field(str, "User's login name", max_size=32, unique=True)
36

How would it conflict? The example you would still work under this suggestion (assuming appropriate metaclasses). (Note that __rmatmul__ would be implemented on the metadata object, not typing.Annotated.)

If you mean that it just “consumes” a design pattern that could be used for other uses, I still don’t think that’s actually a problem. No type checkers currently allow arbitrary computation in type hints, so something like Matrix_3x4 @ Matrix_4x5 probably wouldn’t appear in the same codebase as typ @ metdata anyway.

That defeats the point of the proposal. It’s not much easier to read or write, and worse, it restricts you to one item of metadata.

1 Like
37

I think if Python ever gets “shape typing” it’s more likely to look like this:

def matmul[N: int, M: int, K: int](
    a: Matrix[N, M], b: Matrix[M, K]) -> Matrix[N, K]: ...
def concat[N: int, M: int, K: int](
    a: Matrix[N, K], b: Matrix[M, K]) -> Matrix[N + M, K]: ...
def tile[N: int, M: int, K: int](
    a: Matrix[N, M], reps: K) -> Matrix[N * K, M]: ...

a: Matrix[Literal[3], Literal[4]] = ...
b: Matrix[Literal[4], Literal[5]] = ...
c = matmul(a, b)  # type: Matrix[Literal[3], Literal[5]]

and I think addition and multiplication are enough for these purposes. At least I can’t think of a shape-level operation that’d need matrix multiplication.

2 Likes
38

I am heavy -1 on this. In my opinion, it does not solve any existing real problem. Typing Annotated[] on a keyboard is not that hard and letters are free (especially nowadays).

While it creates several real problems:

  1. It requires to add __matmul__ magic method to most builtin types. For a very niche usecase. It is not something fundamental as | does.
  2. Some existing custom types might already have @ defined to do something different. In this case using them with @ would be impossible and would still require using Annotated directly
  3. It creates complex operator precedence rules int | str @ Metadata and (int | str) @ Metadata are two different things
  4. It would also require us to either port Annotated to C code to use something similar to Py_GenericAlias or to just always call Python code from C code, which is really slow and error-prone

Let’s please do not do this :slight_smile:

However, your PEP work is really impressive and the idea is worth discussing! Thanks a lot for raising this topic.

25 Likes
39

From a style point of view, I like the @ operator better than any proposed alternatives so far. But the objections a couple people have raised so far about incompatibility with types that already implement matmul in their metaclass are pretty concerning, particularly @mikeshardmind mentioning that he’s aware of actual production code that relies on the pattern.

Annotated is a tool I reach for quite rarely (since my efforts in typing tend to focus on typing, not other metadata) so I’m not entirely convinced the shorthand is worth the drawbacks. I primarily only use it when working with Pydantic or FastAPI, and I’m comfortable with the verbosity essentially just being part of their APIs.

2 Likes
40

1., 2., 4. do not apply if we instead use rmatmul on the annotation/metadata object, as is (AFAICT) the latest suggestion by the author, the PEP just hasn’t been updated.

4 Likes
41

This is pretty compelling. I should catch up with the authors of that PEP to see how they feel about this.

The current plan is still to update type, sentinel… to add support for __matmul__ (but not None). If this proves too unpopular, I will pivot to the __rmatmul__ approach. I’m sorry about the confusion.

In a follow up PEP, we would introduce TypeMedata/SymbolMedata with __rmatmul__ (see below).

It’s a chicken and egg problem. Annotated is underused because it has bad ergonomics and is a bit obscure.

If we adopt the TypeMetadata/SymbolMetdata proposal, features like ReadOnly/Final/ClassVar could be prototyped in a way that degrades gracefully

For annotations on the symbol:

    title: str @ReadOnly @Required

And for annotations on the value:

list[str @Literal["yes", "No", "Maybe"]]

To be clear: I’m not suggesting we change the existing standards, I am just showing that Annotated has compelling use cases if we polish its ergonomics.

That is exactly what our cpython prototype does :slight_smile:

This is problematic. Under current semantic for generic type aliases, your example evaluates roughly to:

N = TypeVar("N", bound=int)
M = TypeVar("M", bound=int)
K = TypeVar("K", bound=int)
def matmul( a: Matrix[N, M], b: Matrix[M, K]) -> Matrix[N, K]:
   ...

It means that they are type variables bound to int, not that M, N and K are integer literals.

If you wanted to build a prototype to experiment with shapes checking, you could imagine something like:

def matmul(a: np.array @Shape["N", "M"], b: np.array @Shape["M", "K"]) -> np.array @Shape["N", "K"]: ...
1 Like