PEP 835: Shorthand syntax for Annotated type metadata

[JoBe]

I think out of those, T[#...] can be discarded immediately, as the # (comment) would not match with the language as a whole. Not only would it be hard to decide whetherT[#...] means invalid syntax and a comment following, or means the new syntax, but anything with multiple lines would be worse yet, e.g.

T[#...]
...]

or

T[
    #...
    <Perhaps actual data>
]

The @ syntax feels more natural imo, as @ident is used to wrap function/classes, often appending metadata but returning the same object. T @ Meta would work similarly, attaching metadata to the type while still returning the same type (mostly, just wrapped in Annotated).

I do however get the idea of using //, which relates to the comment syntax in c-like languages, commenting on a type, and therefore adding metadata to it.

[abebus (Albert Namaev)]

I’m personally -1 on this proposal.

Python already exposes a mechanism for attaching metadata to types through Annotated. If a library wants a more ergonomic syntax, it can provide its own abstraction using @ operator overloading and ship type-checker plugins that understand the resulting types.

This seems like a very niche use case that can be solved at the library level today. In contrast, adding new language syntax introduces ecosystem-wide costs.

To me, the burden of proof for changing the language should be much higher than “some libraries would become slightly nicer to write”. If the pattern is genuinely valuable, libraries can already experiment with it and demonstrate widespread adoption before we consider baking it into Python itself

[duncathan (Dunc Graham)]

@abebus

ship type-checker plugins that understand the resulting types

This is not as viable an option as a lot of people tend to think. Of the major type checkers, I believe only mypy supports general third party plugins. Pyright, zuban, and Ty all indicate that they have no intention of supporting general plugins. I’m not sure about Pyrefly’s plans for the future but I don’t believe it supports them at present?

The correct approach for typing is usually in fact to pursue standardization.

[tjreedy (Terry Jan Reedy)]

If the problem is writing out the word ‘Annotated’, one can abbreviate it locally on import or thereafter.

from typing import Annotated as A
age: A[int, Field(ge=0, le=150)]  = 10  # Works since variable annotations were added.
age: int @ Field(ge=0, le=150) = 10  # Only 2 of 37 chars('[]') shorter.

I don’t see the need to abbreviate further, especially given the new issues introduced. The effort saved might take years, if ever, to match the effort needed to make the change and to deal with the confusion introduced, including teaching and learning time.

[MegaIng (Cornelius Krupp)]

Please actually read the thread instead of just repeating already made points. Yes, this thread is already somewhat long, but this issue is not fixed by random people commenting without reading it.

[pawamoy (Timothée Mazzucotelli)]

@xmw

Since it’s designed for arbitrary expressions, and it’s majorly used with runtime introspection, and a big use of it is to attach conditions to allowed values such as the examples in the PEP, I think it’s reasonable to consider readability of infixed @ with arbitrary operators in this PEP.

Sure. Maybe it’s majorly used for runtime introspection, for example in Pydantic models. But people do expose these models in their public API, and do want to auto-document them as such, ideally with tools that understand Annotated, Field, etc., such as Griffe + the griffe-pydantic extension. That’s a reason why I think it’s generally not recommended to use expressions that must be evaluated in annotation metadata. Hence why I’m asking if people do use arithmetic operations in annotation metadata in practice. I’ll stop there as I feel I’m off-topic

[provinzkraut (Janek Nouvertné)]

Repeating some points I already made in the other topic:

Ergonomics

Seeing as the motivator behind this is ergonomics and readability, I’m not convinced this proposal meaningfully improves things.

The PEP cites, as it’s primary real-world example of negative impacts of Annotated:

This ergonomic barrier was notably evident in the withdrawal of PEP 727 (Documentation Metadata). The extreme verbosity of the syntax in function signatures was a primary factor in its community pushback. A native shorthand makes such metadata-heavy standards significantly more viable.

This is, in my opinion, simply not correct. Looking at the proposed PEP 727, we can compare actual FastAPI source code with, and without PEP 835:

    def __init__(
        self: AppType,
        *,
        debug: Annotated[
            bool,
            Doc(
                """
                Boolean indicating if debug tracebacks should be returned on server
                errors.

                Read more in the
                [Starlette docs for Applications](https://www.starlette.dev/applications/#instantiating-the-application).
                """
            ),
        ] = False,

vs. how it would look with a shorthand syntax:

    def __init__(
        self: AppType,
        *,
        debug: bool @ (
            Doc(
                """
                Boolean indicating if debug tracebacks should be returned on server
                errors.

                Read more in the
                [Starlette docs for Applications](https://www.starlette.dev/applications/#instantiating-the-application).
                """
            ),
        ) = False,

The verbosity is identical, and does not come from Annotated, but from the proposed Doc type, putting documentation inside the function signature.

Other examples included are also neither more terse nor readable, and some, seem even worse:

 async def read_items(q: (str | None) @ Query(max_length=50) = None):
     ...

is more involved than

 async def read_items(q: Annotated[str | None, Query(max_length=50)] = None):
     ...

This particular instance could also be solved in a different way, without requiring new syntax, if it were allowed to dynamically produce TypeForms to be used in annotations, subclass Annotated, or any other method of marking a type “transparent” to the type checker. Then we could write:

 async def read_items(q: Query[str | None](max_length=50) = None):
     ...

Usefulness to the broader ecosystem

The PEP mentions only the Pydantic + FastAPI ecosystem as benefiting from this, and while libraries making use of type annotations are become more and more common, support for this seems to be focused on a particular niche (although one has to consider the massive popularity of those libraries mentioned in the PEP).

Many potential downsides and pitfalls have been brought up here already, so the usefulness of this should probably outweigh those. Hearing from users / maintainers of other libraries making extensive use of Annotated would be great for this.

[provinzkraut (Janek Nouvertné)]

My key point is that we shouldn’t be attaching non-type information to types, not that there’s a runtime/static difference.

It’s not type information and is, mostly, also not consumable by type checkers, but instead just a way to attach arbitrary metadata to a parameter / attribute.

The reason why people like this is locality. Traditionally, information regarding e.g. field validation was stored independently from the field. For classes, this continues to work well via dataclass semantics:

class MyThing(BaseModel):
    field: int = Field(gt=1)

This is still local and easy to read.

For functions, it becomes more indirect:

@validate(Field(name="field", gt=1))
def some_func(field: int) -> None:
    ...

This duplicates information, makes it less portable, and you now have to look in 2 places to have all the relevant information about how field gets processed.

[mikeshardmind (Michael H)]

Many of the use cases of annotated types are technically type information, just type information that isn’t standardized within the type system.

Sized ints, for example, are refinement types.

I don’t see refinement types as likely to reach standardization soon, but I do wonder if there aren’t better approaches to supporting use cases like this

For example, we could change the execution semantics of type alias statements so that the below would work as expected.

type SizedInt[Low, High] = Annotated[int, Meta(ge=Low, le=High)]
u16 = SizedInt[0, 65_535]

Currently, at runtime, this results in u16 having a value of Annotated[int, Meta(ge=Low, le=High), With Low and High being TypeVars, not the actual values parameterized with, even though as far as the static type system is concerned, u16 isn’t generic, and __parameters__ is an empty tuple reflecting such.

The values SizedInt are parameterized with aren’t stored at all on the second line, even though both statically and the runtime implementation can see that the values passed in must be values. Simply storing them for runtime tools to retrieve, even without changing the existing semantics for execution of the alias would also allow this use.

[mikeshardmind (Michael H)]

Prior post had a mistake, they are stored at runtime. Which means simply standardizing a meaning for the above would work with zero language changes required, and support already working for a minimum of python 3.12.

This won’t really help the people pushing to shove documentation into annotations (though see 3 posts above for how @ for annotated won’t either), but the results of the attempt to propose typing.Doc seem to indicate that shouldn’t be pursued for standardization given the divisiveness of it, and if this not significantly helping that specific use case is a deal breaker in some way, then this proposal starts to feel like an attempt to do a run around the community view on the prior propsal.

[BenjyWiener]

@provinzkraut

vs. how it would look with a shorthand syntax:

    def __init__(
        self: AppType,
        *,
        debug: bool @ (
            Doc(
                """
                Boolean indicating if debug tracebacks should be returned on server
                errors.

                Read more in the
                [Starlette docs for Applications](https://www.starlette.dev/applications/#instantiating-the-application).
                """
            ),
        ) = False,

This feels slightly disingenuous due to the extra parentheses. It should be:

    def __init__(
        self: AppType,
        *,
        debug: bool @ Doc(
            """
            Boolean indicating if debug tracebacks should be returned on server
            errors.

            Read more in the
            [Starlette docs for Applications](https://www.starlette.dev/applications/#instantiating-the-application).
            """
        ) = False,

(Although I’m personally -1 on Doc annotations for other reasons.)

@provinzkraut

This particular instance could also be solved in a different way, without requiring new syntax, if it were allowed to dynamically produce TypeForms to be used in annotations, subclass Annotated, or any other method of marking a type “transparent” to the type checker. Then we could write:

 async def read_items(q: Query[str | None](max_length=50) = None):
     ...

This doesn’t quite solve the type-locality issue: to someone scanning this code, it’s not immediately clear what the runtime type of q is.

@provinzkraut

The PEP mentions only the Pydantic + FastAPI ecosystem as benefiting from this, and while libraries making use of type annotations are become more and more common, support for this seems to be focused on a particular niche (although one has to consider the massive popularity of those libraries mentioned in the PEP).

Many potential downsides and pitfalls have been brought up here already, so the usefulness of this should probably outweigh those.

Please correct me if you disagree, but I think almost all of the concerns raised here are solved and or sidestepped by limiting the scope of this PEP to an opt-in Metadata base class.

@provinzkraut

Hearing from users / maintainers of other libraries making extensive use of Annotated would be great for this.

Anecdotally, my experience has been that Annotated is considered “highly-advanced Python,” and is quite confusing for more casual Python users (even those highly experienced in other programming languages). Suddenly mostly self-explanatory type hints become buried in a complex construct, with new terms that feel like more like leaked implementation details.

Using @property doesn’t require fully understanding descriptors, and using type hints doesn’t require understanding __annotate__/__annotations__. If attaching metadata didn’t require fully understanding the mechanics of Annotated and runtime annotations, it would be much more approachable.

x: int @ Interval(1, 10) may be magic, but it’s magic that doesn’t require much background to “get,” or at the very least, to ignore. The same cannot be said of x: Annotated[int, Interval(1, 10)].

[ZeroIntensity (Peter Bierma)]

I will say, I was really excited when I first saw this PEP, and it disappoints me to see how many people dislike it. I share the opinion that a dedicated syntax encourages the use of Annotated, which I have also avoided due to verbosity. I think the @ syntax is intuitive and uninvasive.

Many people seem to agree with Nikita’s post, so I’ll simply put my counterarguments here. (I’ve only skimmed through this thread, so I apologize in advance if I’m repeating what others have already said.)

@sobolevn

1. It requires to add __matmul__ magic method to most builtin types. For a very niche usecase. It is not something fundamental as | does.

I’m having a hard time understanding why this is a problem. | is surely more common, but there’s not any cost of adding __matmul__ to built-in types (other than reserving it for something else, but I can’t imagine what else we’d want to use it for). If you’re concerned about the maintenance overhead, see my remarks at the bottom of this post.

Furthermore, | is arguably already a niche use-case. There are millions of Python users who have not and will not touch type hints, and they haven’t seemed to notice nor care about the existence of | on some types. This applies to __class_getitem__ too. (Out of curiosity, I looked this up online – I could not find a single person asking why these existed!)

@sobolevn

2. Some existing custom types might already have @ defined to do something different. In this case using them with @ would be impossible and would still require using Annotated directly

Do you have any examples of this? This sounds a lot like a theoretical issue. It is very uncommon to see the @ operator outside of math libraries in the first place, so it’s extremely uncommon to see it implemented on a metaclass where this would apply.

If there was a major library that had already defined type @ whatever as something else, then I’d be receptive to this argument, but otherwise, it seems like a non-issue.

@sobolevn

3. It creates complex operator precedence rules int | str @ Metadata and (int | str) @ Metadata are two different things

Is this not a problem for any operator? Would you consider 1 + 2 * 3 being different than (1 + 2) * 3 an issue as well?

The operator precedence here is consistent with all other operators in Python. I do agree that it can be annoying at times, but static typing in particular usually means instant feedback for the developer (because it will typically show up in their editor). If you get the operator precedence wrong, you’re one hover away from realizing that.

@sobolevn

4. It would also require us to either port Annotated to C code to use something similar to Py_GenericAlias or to just always call Python code from C code, which is really slow and error-prone

This argument can be made for just about anything we add to CPython, but in this case I don’t think it’s a real issue.

The best reference here is Py_GenericAlias and the C implementation of GenericAlias. Looking at the commit history of genericaliasobject.c, it has had 63 changes in the six years that it has existed. Of those, only about ~40 of them are actual fixes or new features (the rest being general housekeeping changes across all of CPython). In the ~2200 days since PEP 585 was implemented, that’s about one real change every two months or so. For something with that few changes, I absolutely think the benefits of this proposal outweigh the maintenance cost.

[BenjyWiener]

@ZeroIntensity

I’m having a hard time understanding why this is a problem. | is surely more common, but there’s not any cost of adding __matmul__ to built-in types

I’m strongly +1 on this PEP in general, but I’m -1 on type.__matmul__ because of the None @ x issue. While uncommon, I’ve personally used Annotated[None, …]^[1]. While I’m pretty sure that’s the only exception currently, I don’t like that @ would differ from | in such a seemingly arbitrary way.

On the other hand, I see multiple advantages to Metadata.__rmatmul__:

1. It elegantly solves PEP 746.
2. Subclasses can override __rmatmul__ to add multiple metadata objects (useful for, say, a utility wrapper that generates both Pydantic’s Field and WithJsonSchema metadata).
3. It makes explicit what objects are meant to be used as metadata.
4. It allows metadata to perform some validation (duplication, invalid type, invalid combinations) at the point of definition, instead of only at the point of consumption, potentially making debugging easier.

To allow using arbitrary objects with the new syntax, Metadata can be implemented such that T @ Metadata(1, 'a') is equivalent to Annotated[T, 1, 'a'].

---

1. for dependency injection with only side effects ↩︎

[DavidCEllis (David Ellis)]

@ZeroIntensity

I will say, I was really excited when I first saw this PEP, and it disappoints me to see how many people dislike it. I share the opinion that a dedicated syntax encourages the use of Annotated, which I have also avoided due to verbosity. I think the @ syntax is intuitive and uninvasive.

As a counter-point, I was unenthused when I saw this PEP and was happy to see how many other people disliked it to know I wasn’t alone in my apprehension .

Mostly this is due to this having some of the same runtime issues that | has in the presence of forward references, but with the potential to lead to additional issues with runtime relevant information being lost. But I also just don’t like putting more information in annotations as I find it makes it more difficult to connect names with the values they’re being assigned.

---

The runtime issue is that annotations may contain objects that are not yet defined and as such this is ambiguous:

unknown_1 @ annotated_thing

While this is not:

from typing import Annotated

Annotated[unknown_1, annotated_thing]

To avoid suddenly breaking things when switching to the proposed syntax if the first object is unknown, it is necessary to make the assumption that either the unknown object implements __matmul__ to create Annotated or that it doesn’t implement __matmul__ and so we can rely on the __rmatmul__ of annotated_thing (or in the case that both are unknown, that this is still fine somehow).

Supporting this requires a new annotation format (FORWARDREF_STRUCTURAL in the PEP) and I’m not convinced the convenience is worth adding this complexity (or this guesswork).

If we allow a new syntax like this but exclude the additional annotation format we’ll need to make it clear that it will break in the presence of forward references and that users will either need to avoid them or wrap them in type aliases. Tooling will likely need to be aware of this either way.

[till-varoquaux]

@BenjyWiener

I’m strongly +1 on this PEP in general, but I’m -1 on type.__matmul__ because of the None @ x issue.

You make a strong case for __rmatmul__. It is definitely worth considering. I am waiting for the dust to settle before updating the PEP, but this will be included.

Both approaches have tradeoffs.

• __matmul__ is more consistent. It arguably provides a better path for static tracking.
• __rmatmul__ handles None cleanly. It also enables some validation ^[1].

They are not mutually exclusive. We could ensure type checkers support:

def __rmatmul__[T](self, typ: TypeForm[T], /) -> TypeForm[T]:
    ...

My current inclination would be to add __matmul__ and make sure type-checkers support __rmatmul__.

Down the line, if we add base classes for annotations (to perform stricter validations and/or enable static tracking), these base classes should implement __rmatmul__.

---

1. Validating things like Java’s @Target(ElementType.METHOD) would still require extending the type system ↩︎

[till-varoquaux]

@DavidCEllis

Supporting this requires a new annotation format (FORWARDREF_STRUCTURAL in the PEP) and I’m not convinced the convenience is worth adding this complexity (or this guesswork).

Production libraries already handle forward references and infix operators like | during type resolution.

All the examples in the PEP were tested against Pydantic and FastAPI. They worked perfectly without modifying those libraries at all. Crucially, they worked entirely without FORWARDREF_STRUCTURAL.

FORWARDREF_STRUCTURAL is not a requirement for the @ syntax to function. It is simply a proposed convenience to help authors write new libraries. Tools will handle @ using the same resolution mechanisms they currently use for | and __getitem__.

[DavidCEllis (David Ellis)]

@till-varoquaux

All the examples in the PEP were tested against Pydantic and FastAPI. They worked perfectly without modifying those libraries at all. Crucially, they worked entirely without FORWARDREF_STRUCTURAL.

What examples that use forward references? The format is only relevant in the presence of forward references.

You give one example here which for some reason uses a string in place of an actual forward reference. Run as is, attempting to resolve the annotation will fail with a TypeError even if you have defined NotYetDefined as @ between a literal string and a field doesn’t work.

Pydantic requires the annotation to be resolvable by the time you create an instance, so it essentially needs VALUE annotations to work by the time the class is used even if not at definition. The issue is for cases where partial information may be enough for a tool to function and would be lost.

Take cattrs’ override option which works with Annotated.

from typing import Annotated
from attrs import define
import cattrs

@define
class Example:
    original: Annotated[Unknown, cattrs.override(rename="new")]

ex = Example(42)

print(cattrs.unstructure(ex))

{'new': 42}

This requires the partial information from Annotated that would be lost if the entire object was still an opaque forward reference. I’d also note that it doesn’t seem to work correctly on your fork even with the use of Annotated directly but I’m not sure why?

[till-varoquaux]

Thank you for the clarification, I misunderstood your original point. I assume these issues also currently affect | or, in the cattr case, is this not a problem because the Annotated is always at the top level?

@DavidCEllis

I’d also note that it doesn’t seem to work correctly on your fork even with the use of Annotated directly but I’m not sure why?

I’ll look into it.

[DavidCEllis (David Ellis)]

I’m not sure if there’s anything that makes use of partial Union information - the existence of | for Union predates 3.14 annotations or this may have been more of a concern. Technically a runtime checker could know that a str would be valid for Union[str, Unknown] but wouldn’t know for str | Unknown because it doesn’t partially evaluate.

My concern is in not adding something else which has the same issue where syntax that is supposed to be ‘equivalent’ actually has corner cases where it isn’t. Especially if well-meaning ‘upgrade’ tools will make this change for you.

The current Annotated[...] form shouldn’t have issues with forward references unless the annotation is actually invalid in a way other than being an undefined name (say an AttributeError for a deprecated alias that has been removed - this would turn the entire annotation into an opaque ForwardRef).

[ZeroIntensity (Peter Bierma)]

@DavidCEllis

But I also just don’t like putting more information in annotations as I find it makes it more difficult to connect names with the values they’re being assigned.

I think you have a skewed idea of where this will be used. It sounds like you’re envisioning a case like this:

def whatever():
    a: int @ ValueRange(1, 10) = ...

In which case, I agree. I think a should be its own special object in the above example, rather than storing the information in the annotation. But that’s not what this proposal is for, nor do I think it will encourage it – Annotated becomes useful in cases where the type acts as metadata for something else that can’t be described as a value, such as in a Pydantic class. Something like this doesn’t work or make sense:

class Foo(BaseModel):
    a: int = ValueRange(1, 10)

And this is exactly why Annotated was added in the first place. This proposal simply makes that more accessible, which encourages people to actually use Annotated instead of inventing their own special types with special meanings to whatever is parsing them. I think philosophical arguments about the efficacy of Annotated should have been made when PEP 593 was in discussion, but we’re long past that.

@DavidCEllis

Supporting this requires a new annotation format (FORWARDREF_STRUCTURAL in the PEP) and I’m not convinced the convenience is worth adding this complexity (or this guesswork).

Hm, okay, but what would convince you? I agree that this PEP should come with a new annotation format for all the reasons you described, but I don’t see why adding it would be too complex. More generally, the “this is complex” argument against this proposal feels like a cop-out. I’d love to hear Jelle’s opinion on this as the annotationlib maintainer, but, among other PEPs, this one doesn’t seem very complicated.

What are some concrete problems that this proposal would introduce? Looking through this thread, I can’t find anything beyond “Annotated isn’t used that much”, which is exactly the problem this proposal is intended to solve, isn’t it? The FastAPI use case alone is enough to convince me that this is necessary.