PEP 835: Shorthand syntax for Annotated type metadata

[tmk (Thomas Kehrenberg)]

@sirosen

If, for example, “annotate” is a new diglyph, <@,

Another natural choice for a digraph is @@. There’s already precedence for doubled symbols as different operators: / vs //, * vs **.

As for single letters, I think only $ and ? are currently unassigned, from the ASCII range.

[jorenham (Joren Hammudoglu)]

@tmk

There’s already precedence for doubled symbols as different operators: / vs //, * vs **.

Following that logic, @@ should then be used for the matrix power.

[jorenham (Joren Hammudoglu)]

I don’t think that inventing a cryptic symbol or combination of symbols is the way to go here. It’ll look more like Haskell than Python that way.
A dedicated keyword like int annotate Interval(1, 10) or something would have my preference.

[sirosen (Stephen Rosen)]

Thank you for using “digraph”! I didn’t notice that I wrote “diglyph”. Oops.

I don’t think it’s very important to debate the use of symbol if the PEP authors don’t agree that a syntax change is the right choice here.

@jorenham

A dedicated keyword like int annotate Interval(1, 10) or something would have my preference.

This is a bit more different than an operator.

There are interesting cases to consider for a new soft-keyword, where readability may be worse:

annotate = False
choice = (Foo | Bar) annotate Tags["type_field"]

The idea has merit, IMO, even though it’s longer than a single character. The biggest readability issues with Annotated are that it’s prefix rather than infix and adds more nesting – wrapping due to long lines is a secondary problem.

As with operator spelling, I’d hold off on going deep on this idea in this thread unless the PEP authors are convinced that new syntax is the best solution.

[till-varoquaux]

@pf_moore

Isn’t this very much related to Guido’s talk at the typing summit - specifically the question of whether we’re still bound by the “no new syntax” rules? A new operator is the least intrusive form of new syntax, in many ways.

I’m open to alternatives. The reason I went with @ was that it seems to be the strong favorite every time this topic arises.

If we decide to add a new infix operator or keyword, I’m happy to do the integration work in cpython, ruff…

[decorator-factory]

I like MegaIng’s idea of offloading the resposibility for supporting __matmul__ to libraries that want to attach metadata. Maybe it could be done without any additions to the standard library? For example, what if type checkers understood this form:

class Field:
    def __rmatmul__[T](self, typehint: TypeForm[T], /) -> TypeForm[T]:
        return Annotated[typehint, self]

type PositiveInt = int @ Field(gt=0)
# `PositiveInt considered to be `int` by type checkers

If the library author doesn’t like the @ operator, they can use something else, as long as it has the interface of [T](TypeForm[T]) -> TypeForm[T]:

Rename = sentinel("Rename")

def rename[T](typehint: TypeForm[T], name: str) -> TypeForm[T]:
    return Annotated[typehint, (Rename, name)]
# <...>
class Entity(otherlibrary.BaseModel):
    uid: rename(str | int, "id")  # same as `uid: str | int` to type checkers

[jason-s (Jason Sachs)]

@kkirsche

@sobolevn

Typing Annotated[] on a keyboard is not that hard and letters are free (especially nowadays)

True, but I’m not sure the writing argument is truly the value this is adding, more the reading side. In libraries that lean heavily on annotations, Annotated[] starts adding visual noise that makes the code harder to parse at a glance. Since most of us spend way more time reading code than writing it, legibility is where I’d weight the tradeoff.

It’s the reading that’s the problem — both in the reading, and in writing code, since we have to read our own work to be satisfied with it.

[jason-s (Jason Sachs)]

@pf_moore

But it’s still not type information, is it? Unless you’re suggesting that Annotated[int, ValueRange(-10, 5)] would be a different type than int? And if so, does that mean that Annotated[int, Doc("some random integer")] is also a different type than int?

My key point is that we shouldn’t be attaching non-type information to types, not that there’s a runtime/static difference.

It’s annotated type information — information which is important to attach to a type, for introspection or analysis purposes, without changing the type itself, as seen by the language.

As for the syntax: I don’t really like the @ syntax. The existing Annotated syntax would be fine if it were possible to create “type macros” at static-analysis time, for our own shorthand: (pardon my syntactic sloppiness)

def doc(T: type, text: str, *args): T
    return Annotated[T, Doc(text), *args]

def q12_int(text: str): int
    return Annotated[int, Doc(text), ValueRange(-2048, 2047)]

and then

class Foo:
    zipcode: doc(int, "ZIP code", ValueRange(0, 99999))
    x:       q12_int("x-coordinate")
    y:       q12_int("y-coordinate")
    voltage: q12_int("battery voltage")

It’s possible today to use the type statement for constant annotated types that are used in several places, but not ones that take a parameter.

type q12_int = Annotated[int, ValueRange(-2048, 2047)]

Or

>>> from typing import Annotated, get_type_hints
>>> type q = Annotated[int, "Hey there"]
>>> def f(x:q) -> q:
...     return x+1
...
>>> get_type_hints(f)['x'].__value__.__metadata__
('Hey there',)

[till-varoquaux]

@decorator-factory

I like MegaIng’s idea of offloading the resposibility for supporting __matmul__ to libraries that want to attach metadata. Maybe it could be done without any additions to the standard library?

This is an appealing way to test-drive the syntax without a PEP, but it comes with three major drawbacks:

1. Ecosystem fracture: Some libraries will implement it; others won’t. Annotated should be standardized.

2. Bypassing consensus: People have expressed valid concerns about @. Relying on a decentralized, library-level workaround feels like going behind the community’s back rather than reaching a consensus.

3. Static analysis friction: It complicates static tracking. We would need complex workarounds (although we could add support in type checkers for something like Annotated[TypeForm[T], Self]) just to verify the metadata is applied correctly.

@jason-s

class Foo:
    zipcode: doc(int, "ZIP code", ValueRange(0, 99999))

The primary advantage of an infix operator (whether @ or something new) is that it instantly visually isolates the base type.

When I read v: doc(int, ...) or v: FooBar[int], I have to mentally parse the function signature to figure out if v is ultimately an int, a doc, or a FooBar.

With v: int @ FooBar, the underlying type (int) is immediately obvious at a glance.

[sobolevn (Nikita Sobolev)]

It’s the reading that’s the problem — both in the reading, and in writing code

>>> from typing import Annotated
>>> A = Annotated
>>> A[int, 1, 2]
typing.Annotated[int, 1, 2]

There’s nothing different from the proposed @, except the scope and precedence are clearly defined.

[MegaIng (Cornelius Krupp)]

@sobolevn

There’s nothing different from the proposed @, except the scope and precedence are clearly defined.

There is a difference; It’s significantly less readable to me. It requires non linear parsing of the expression.

Fundamentally, an infix operator is going to be more readable than the function-call syntax. You might think that the costs of the infix operator are too high, but please stop trying to say it has no benefits.

[duncathan (Dunc Graham)]

I’m interested in the idea of a whole new infix operator. It has all the benefits of using an infix operator, without the drawback of potentially breaking existing valid code. There are some outstanding questions about it that would probably need answering, though:

• How to spell the operator? @>, @@, etc?
• What would the name of the operator be? __annotate__, perhaps?
• Would we be able to override its behavior in our classes?
• Is overriding it something we could expect to be meaningful, either at runtime or statically?

Recognizing that there’s no need to bikeshed on these questions if the PR authors are against the idea. Just figured they might be worth discussing if the authors are interested.

[till-varoquaux]

@duncathan

Just figured they might be worth discussing if the authors are interested.

I am completely open to a new operator. While I have a slight preference for @, getting a readable syntax for Annotated is the main goal. I will gladly align with the community.

Regarding the risk of __matmul__ breaking existing code: a collision only happens if a type uses a custom metaclass that explicitly implements __matmul__ , or if an existing object already implements __rmatmul__ to accept type forms. I would be happy to run some analysis on the ecosystem to see how often that actually occurs in the wild.

That said, I suspect some of the technical anxiety is amplified by how jarring the syntax feels. Seeing an operator repurposed is inherently uncomfortable..

(Though it is worth noting the irony here: PEP 465 faced the exact same pushback from developers who felt that using @ for math was jarring because it repurposed Python’s decorator symbol!)

Syntax tastes are deeply personal, which is exactly why I am perfectly happy to pivot to a new operator if that is where the consensus lands.

[xmw (xw)]

I’m +1 for use of the @ symbol, but -1 for it being used as an infix operator, because it’ll greatly decrease readability for attaching metadata that uses any variety of arithmetic operators.

If we’re open to grammar changes, I wouldn’t mind just seeing @ in place of Annotated, e.g. var: @[T, meta + data, some * expression], rather than T @ (meta + data) @ (some * expression). Commas are still great for delimiting, because they’re not an operator in Python.

[till-varoquaux]

Brackets for type metadata are fairly common. C++, OCaml, Rust, and PHP all use them, and all the languages I am aware of place the base type explicitly outside the metadata brackets.

For example, C++ uses T[[meta]], OCaml uses T[@meta], and Rust uses T #[meta].

If we adapted that pattern for Python, it could support both grouping and chaining: var: int[@Gt(0), Lt(10)] or var: int[@Gt(0)][@Lt(10)].

Python

class MyClass:
    age: int[@Gt(18)]
    factors: list[int[@Predicate(is_prime)]]
    my_list: list[int][@Len(0, 10)]

We probably don’t want to put the metadata before the type (like [@required] list[int]), it makes it hard to read when combined with generics.

Under the hood, this syntax could be powered by a new __annotate__ dunder method, operating similarly to __getitem__.

[DavidCEllis (David Ellis)]

@till-varoquaux

Under the hood, this syntax could be powered by a new __annotate__ dunder method, operating similarly to __getitem__.

__annotate__ already exists and does something quite relevant to the PEP.

In its current form I don’t find the shorthand a significant improvement in readability to justify the additional complexity required to support it and I think the assumptions required to handle forward references in order to make this work are undesirable.

For anything else I’ll wait until there’s an updated PEP.

[BenjyWiener]

@till-varoquaux

Under the hood, this syntax could be powered by a new __annotate__ dunder method

Well __annotate__ was already taken by PEP 649.

@till-varoquaux

If we adapted that pattern for Python, it could support both grouping and chaining: var: int[@Gt(0), Lt(10)] or var: int[@Gt(0)][@Lt(10)].

I think using square brackets would be too similar to generic syntax (especially when combined with generics).

Just some more bike shedding, but what about curly-brace subscripting?

x: int {Gt(10), Lt(20)}
y: int {Gt(10)} {Lt(20)}
z: str {
  Field(
    max_len=10,
  ),
}

(I still think shifting responsibility to the metadata is the best option, whether via __rmatmul__ or some new infix operator.)

[pawamoy (Timothée Mazzucotelli)]

@xmw

because it’ll greatly decrease readability for attaching metadata that uses any variety of arithmetic operators

Does it exist in practice? Metadata making/recommending use of arithmetic operators? Could you give real-world examples? I suppose the use-case is runtime introspection, because static analysis tools wouldn’t be happy with such metadata.

[xmw (xw)]

The metadata in Annotated was not designed to be (just) recognised by static analysis tools, it was designed to accept arbitrary value expressions (see the original text of PEP 593). AFAIK current popular frameworks making use of the metadata do so by runtime introspection only; this applies to the examples in the PEP (e.g. pydantic.Field(gt=0, le=1000)).

Unless there are recent features in static analysis tools, AFAIK only mypy has the capability of statically analysing the metadata in Annotated, and that would be through the use of mypy plugins.

Since it’s designed for arbitrary expressions, and it’s majorly used with runtime introspection, and a big use of it is to attach conditions to allowed values such as the examples in the PEP, I think it’s reasonable to consider readability of infixed @ with arbitrary operators in this PEP.

[sirosen (Stephen Rosen)]

@till-varoquaux

We probably don’t want to put the metadata before the type (like [@required] list[int]), it makes it hard to read when combined with generics.

I agree; I’d say that one of the stronger arguments in favor of a dedicated syntax for Annotated is that we want to give the type, not its modifiers, the primary position.

x: T ...

allows the reader to focus on T first.

I think a syntax which starts with [ will be confusing to read, given that we already have generics using brackets.

The possible options appear to be

• a repurposed operator like @ or //
• a new operator like <@ or ::
• a new soft keyword
• a new bracketing syntax like T{...} or T[#...]

Of these, I can’t strictly disqualify any, but a repurposed operator is my least favorite choice.