PEP 835: Shorthand syntax for Annotated type metadata

Following up on our initial feedback thread, PEP 835 is now officially live. I am proposing T @ Metadata as a built-in shorthand for typing.Annotated[T, Metadata].

# Before
class Person:
    age: Annotated[int, Field(ge=0, le=150)]
    ...

# After
class Person:
    age: int @ Field(ge=0, le=150)
    ...

Working Prototypes

I have built reference implementations across the core toolchain so you can test the syntax today:

  • CPython (Includes a new annotationlb.Format: FORWARDREF_STRUCTURAL to resolve parsing issues with stringified compound types).
  • Mypy
  • Pyright
10 Likes

Thanks for the PEP! I have some thoughts on it:

The @ operator is implemented by adding nb_matrix_multiply to the metatype (type) and to several typing-related types. The operator is supported for any left-hand operand that currently supports the | operator for making a union.

Does this include type(None)? The reference implementation adds nb_matrix_multiply to it, but calling NoneType a “typing-related type” is a bit of a stretch.

This proposal introduces a new format, Format.FORWARDREF_STRUCTURAL.

Could you put that in the Specification section?
To me it looks that this proposal could be a PEP by itself, as a solution to the increasing friction between regular syntax & typing annotations. It’s quite surprising to find it as a Backwards Compatibility note.

Notably, if FORWARDREF_STRUCTURAL becomes the state of the art, we … don’t really need __matmul__ on the runtime types anymore.

The shim is scheduled for removal in Python 3.18.

The standard deprecation period is 5 years; is there a reason to rush this?


Were chained colons considered?

class Project(BaseModel):
    name: str: Field(title="Project Name"): Len(1)
    url: HttpUrl: Field(description="The project homepage")
    stars: int: Field(ge=0) = 0

Would it make sense to standardize naked strings as docstrings/descriptions? Frameworks could define how exactly they treat these, but assigning a default meaning might help across projects.

class Project(BaseModel):
    name: str @ "Project Name" @ Len(1)
    url: HttpUrl @ Field(title="Project Homepage") @ "The project homepage, including the scheme (like http://)"
    stars: int @ Field(ge=0) = 0

Just read the PEP, nice work!

I’d love to see examples of usage that span multiple lines, if you believe that’s worth showing. Typically, would we wrap metadata in parentheses?

int @ (
    Hello() 
    @ World(
        """
        very long string
        """
    )
    @ Bye()
)

Looks like the Python syntax highlighter here doesn’t like it: it sees World and Bye as decorators. This isn’t something new (operator already exists) so maybe the PEP shouldn’t concern itself about it.


I suppose the rejected idea in PEP 727 applies here too:

this would create a predefined meaning for any plain string inside of Annotated, and any tool that was using plain strings in them for any other purpose, which is currently allowed, would now be invalid

This doesn’t apply to new syntax though. There will be no ambiguity if we defined a meaning for bare strings for a new Annotated syntax.

As a code reviewer, the chained colons look like a typo or syntax error to mii. I definitely prefer the proposed @ operator haha. I’m sure eventually I could get used to the colons but it seems like it would be a lot more friction

1 Like

Not all types are inside annotations, and that will remain the case for the foreseeable future. (Places where you might need a type outside an annotation include in NewType definitions; in generic parameters to base classes; in calls to cast() and more generally in places where TypeForm is used at runtime; in the argument to NewType(); in type definitions for the functional form of TypedDict and NamedTuple; and in old-style type aliases.)

3 Likes

Both old and new syntaxes (Annotated[int, ...] and int @ ...) evaluate to the same object at runtime: types.AnnotatedType. Unless you recommend having different behavior for strings in the new syntax versus the old one (for example wrapping them in Doc vs leaving them untouched), giving a special meaning to strings in the new syntax is giving the same meaning to strings in the old syntax too.

1 Like

Apologies but the link to PEP 835 is broken, it simply points to a Google search of the “link” word

EDIT: also the link pointing to the previous discourse thread is broken

2 Likes

Thank you! Fixed

Binary operator @ is documented here.

The @ (at) operator is intended to be used for matrix multiplication. No builtin Python types implement this operator.
This operation can be customized using the special __matmul__() and __rmatmul__() methods.

The first 2 sentences can be revised in various ways as needed. In regard to the 3rd, which should best remain as is: in an annotated assignment statement, the annotation is an expression. So I would expect the expression T @ Metadata to resolve to either T._matmul__(Metadata) or Metadata.__rmatmul__(T). Since there is no base class for types T, is Metaclass an actual base class for classes like Field? Or is there a finite set of classes which could each gain a new rmatmul method?

I have no idea what stringified compound types are, but parsing issues might suggest that you are trying to apply @ too broadly.

Good point. Both None @ some_annotation and types.NoneType @ some_annotation are explicitly supported. None is an oddity since it’s not strictly a type. But since it acts as one in type hints, we support it for consistency with |. I’ve updated the PEP to clarify.

Will do. I’ve moved it up into the main Specification section.

Good catch. Bumping the removal to 3.21 to align with PEP 387.

I wanted to stick to existing precedent and avoid CPython grammar changes.

Chained colons also make runtime type manipulation awkward. For example:

def get_machine_int():
    max_val = ...
    return int @ Le(max_val)

This is a niche use case. But it’s cleanly supported by other typing operators (__getitem__ for generics, __or__ for unions). With chained colons, doing this outside a variable annotation would be a syntax error.

1 Like

The PEP implements this by adding __matmul__ directly to the type metaclass (specifically PyType_Type in C).

This automatically supports T @ Metadata for any runtime instance of type (like int, str, object, or custom classes). We don’t need to modify object or the right-hand metadata classes. No __rmatmul__ is required.

This just refers to PEP 563 (from __future__ import annotations). Under PEP 563, annotations are stored as strings.

The “parsing issues” mentioned are only about how tools evaluate those strings at runtime. They are not about Python’s core grammar parser. I’ve updated the wording in the PEP to make this clearer.

And what to do with the return value for functions?

def f() -> int: ...
def f() -> int: Ge(0): ...

Example:

x = Ge(0)
y = int
def f() -> int : x : y = 3 

It

x = Ge(0)
y = int
def f() -> int:
    x: y = 3 

or

x = Ge(0)
y = int
def f() -> int : x:
    y = 3 

I think a shorter deprecation cycle can make sense here because the thing we’re deprecating is private already. I’ve previously used 2-year deprecations for other typing constructs that are private but had some usage in the wild.

None is actually a bit more subtle. It does not support |; constructs like None | int work only by way of __ror__. But for Annotated, you can’t really do that, since the right-hand side of the matmul could be anything, so you’d have to implement __matmul__ on None itself. And that could hide some bugs:

def matmul_arrays(a: np.array | None, b: np.array):
    return a @ b # oops, forgot to check for None

Currently, this would blow up at runtime if a happens to be None. With your proposal, it would instead return an Annotated object.

10 Likes

I see we need to add special handling for None objects because they can show up directly in an annotation.

Does that mean we need to do the same for PEP 661 sentinels?

3 Likes

Shouldn’t the parentheses be like this instead?

class A:
    x: (
        int
        @ Hello()
        @ World(
            """
            very long string
            """
        )
        @ Bye()
    )

I wonder actually whether one should embrace the parallel with decorators and skip the space after the @:

class Person:
    age: int @Field(ge=0, le=150) = 25
3 Likes

Good point. I suppose they are both equivalent in the end (since metadata gets flattened), and that’s up to formatters to make it pretty :slight_smile:

I just read PEP 835 and came to show my very strong and excited support. :star_struck:

This would greatly simplify the syntax needed in libraries that use Annotated, and the effort required from developers to learn how to use them.

If this is accepted, as soon as there’s a supporting Python version available (3.16), I would use it as the main recommended syntax in the docs for my libraries: FastAPI, Typer, SQLModel. :tada:

Small link note:

The PEP at PEP 835 – Shorthand syntax for Annotated type metadata | peps.python.org has a link:

Discussions-To: Discourse thread

That points to:

https://discuss.python.org/t/pep-835-shorthand-syntax-for-annotated-type-metadata/107795

…which is a 404 instead of pointing to this same thread:

https://discuss.python.org/t/pep-835-shorthand-syntax-for-annotated-type-metadata/107814

2 Likes

Thanks, fixed!

2 Likes

PEP 661 sentinels natively implement __or__ (to support unions like Sentinel | int), so we just add __matmul__ to them directly. This doesn’t have the same runtime risks as adding __matmul__ to None.