I disagree. Annotated is not used often because it’s not type information[1], and so putting it with types feels wrong. Making it easier to do simply encourages non-type usage of annotations.
I thought we had this debate years ago. Annotations were originally for arbitrary data, but were later dedicated to be specifically for type information. It feels like we’re trying to backtrack on that decision here (or at least, work around the consequences of it).
When we introduced Annotated we left this intentionally vague. It was mostly designed to be an escape valve and provide a good mechanism for degradation. If you look at the examples (e.g.: Annotated[int, ValueRange(-10, 5)]) some of those annotations could be consumed by a static analysis tool (e.g.: using z3 to check bounds).
Right now, python doesn’t support any good extension point for type checkers. Annotated could be a good place to do that. This is outside of the scope of the current PEP but I wanted to push back against the notion that Annotated should be cordoned off to runtime uses only. It could very well do both. And, in some cases, we could use the same annotations for static analysis and runtime checks.
But it’s still not type information, is it? Unless you’re suggesting that Annotated[int, ValueRange(-10, 5)] would be a different type than int? And if so, does that mean that Annotated[int, Doc("some random integer")] is also a different type than int?
My key point is that we shouldn’t be attaching non-type information to types, not that there’s a runtime/static difference.
True, but I’m not sure the writing argument is truly the value this is adding, more the reading side. In libraries that lean heavily on annotations, Annotated[] starts adding visual noise that makes the code harder to parse at a glance. Since most of us spend way more time reading code than writing it, legibility is where I’d weight the tradeoff.
Well, it’s type information IFF you are using a type checker that knows how to consume that specific kind of metadata.
IMHO type checkers should ignore annotations they do not know how to handle. In a type checker that doesn’t do bounds checking int @Interval(ge=-10, le=5) (switching to annotated-types constraints) would be the same as int.
The main point of introducing SymbolMedata/TypeMetadata would be to add very generic sanity checks (e.g.: disallow list[int @ ReadOnly] because ReadOnly appears nested in a type and is a SymbolMetadata) and agree on a framework to track some annotations statically.
I think projects like Pydantic and FastAPI make a pretty compelling case for the usefulness of attaching metadata to symbols. Instead of framing this as “shoehorning metadata into types,” I’d argue that this PEP is an acknowledgment of the fact that types are simply one form of annotation.
Unless we want to try to devise some alternate syntax for attaching metadata directly to symbols[1], types will need to learn to share the annotation space.
I will note that annotationlib may be a more appropriate home for a potential Metadata class than typing.
Not really? I don’t use it often because I don’t often need to attach non-typing metadata to types. In cases where I do, such as FastAPI or Pydantic, I have no problem reaching for it.
I don’t really see how this proposal is supposed to make Annotated more frequently used. Certainly it might make it more ergonomic when it is used, but this syntax doesn’t create any new reasons to want to use the feature. It doesn’t make it any easier for consumers of these annotations (static or dynamic!) to consume them.
You provided these examples, but I don’t see how this syntax has anything to do with them? Why can’t these be implemented without the @ operator?
In previous (admittedly pet-) projects I explicitly decided against using Annotated because of it’s verbosity and unwieldiness, and if this proposal had been in the language, I would have used it. I don’t think I am alone with that, otherwise this wouldn’t be a topic that comes up again and again.
Not quite. NewType and Annotated serve fundamentally different purposes.
NewType creates a strictly distinct type. As you noted, you must explicitly cast to it (e.g., User(cast(UserId, 4))).
Annotated attaches metadata to an existing type. It does not create a new type. To a type checker that ignores Interval, int @ Interval(...) evaluates simply as int. You can pass a literal 4 to it directly without casting. They are not competing features.
They absolutely can be implemented without the @ operator. You could write Annotated[str, ReadOnly, Required].
The @ operator does not change the underlying semantics; it solves a readability problem.
The proposed @ syntax and the future metadata base classes (SymbolMetadata / ValueMetadata) are technically independent, but they gain strength from each other:
Annotated has room to grow: It has potential as a standardized extension point for static type checkers.
Ergonomics drive adoption: Without the @ syntax, Annotated remains verbose and kludgy. If the ergonomics remain poor, it makes less sense to invest heavily in standardizing static metadata support.
The syntax makes the feature readable and usable at scale. The base classes will make it suitable for static analysis (and could be used to implement PEP 746).
I think that if a new syntax for Annotated is added, it should be a new operator. It solves the matmul compatibility problems, and I argue that it reads more easily.
If, for example, “annotate” is a new diglyph, <@, then the following expression tells us more than it would with @: alpha <@ beta.
Because there’s no conflict with matmul, it’s clear that alpha is a type and beta could be anything valid for use in Annotated.
I understand that there are downsides to adding a new operator, and I’m sure the PEP authors have weighed this… But at least right now, I can’t convince myself that @ is better than a new operator.
I’d like to hear, was this idea, which I’m sure was considered at last briefly, rejected because the authors consider the end result worse or because they think a new operator is harder to justify?
__matmul__ takes priority over __rmatmul__, so at runtime, if __matmul__ is defined on the type, it would win. Unless it returns NotImplemented, in which case @ falls back to __rmatmul__.
Will type checkers be expected implement this logic (which is rather close to arbitrary compuation)? If they don’t implement all of it, at what point will the discrepancy between runtime and type-checking behaviour be too much?
It is almost entirely about ecosystem cost. Adding a brand new operator (like <@) requires modifying Python’s tokenizer, parser, and AST. That instantly breaks every syntax highlighter, linter, formatter, and static analysis tool in the ecosystem. Repurposing an existing operator avoids this massive churn.
There are precedents in the standard library for repurposing operators. For example, pathlib uses division (/) for paths. The context in which an operator is used clears up ambiguities.
Python’s use of @ for math is quite unique. If you look at other major numerical computing environments, they typically use different syntax:
MATLAB and Julia: Use * or .*
R: Uses %*%
Fortran: Uses matmul()
Mathematica: Uses .
In the broader programming ecosystem (Java, TypeScript, C#, Ruby), the @ symbol is often associated with metadata, decorators, and annotations. Repurposing it for Annotated puts us in line with other programming languages and Python’s use of @ for decorators.
/ for paths doesn’t involve putting a definition for __truediv__ on every type; it’s just implemented on instances of a type that would otherwise not participate in division at all.
As for context clearing up ambiguity, that would be true if and only if there were either no valid preexisting interpretation of matrix multiplication involving types (there is) or that we had entirely different behavior in annotations from runtime (we don’t).
In fact, the library authors who seem most active in pushing for this have expressed wanting it specifically for runtime annotation use.
Isn’t this very much related to Guido’s talk at the typing summit - specifically the question of whether we’re still bound by the “no new syntax” rules? A new operator is the least intrusive form of new syntax, in many ways.
There are precedents, but very few of them. And in other languages, repurposing operators is generally not well regarded (the C++ I/O operators being a good example). It’s also important to note that the @ operator was added explicitly to be reserved for matrix multiplication, for 3rd party library use. We can’t know what types 3rd party libraries might have implemented the @ operator on, but we can be sure that it’s matrix multiplication (assuming the code respects PEP 465).
Sure, but that choice was made in PEP 465, and I assume those other numerical environments existed at the time, and their syntax was considered. I’d expect the reasons for the choice to be in the PEP. Also, decorators already existed, and the use of @ in other languages for annotations isn’t new, so again the PEP will have taken that into account.
But PEP 465 explicitly noted that @ for decorators and the operator @ were distinct use cases, which was an important reason that the @ symbol was acceptable. Using operator @ for annotations weakens that argument, and will increase user confusion (assuming the argument in the PEP is correct).
And @ for annotations is far from universal. Rust uses #[...]. C# uses [...]. Where @ is used for annotations, they are typically very similar to Python’s decorators, which already do use @.
I think returning ‘NotImplemented` in this case is the correct thing to do anyway.
Okay, I think I understand your point now.
(Note that I’m -1 on adding __matmul__ to type, +1 on TypeMetadata.) I think it would be reasonable for type checkers to reject A @ x when type(A) implements __matmul__ itself. In general, I don’t really see a world where A @ B (where A and B are types) is accepted as a type hint by type checkers, so I don’t think this would be much of a problem.
I’m currently -1 on the idea of a new operator (mainly because it would take much longer to become widely available). But if we’re exploring that route, I’d like to propose infix ~:
class A:
x: int ~ Interval(1, 10)
y: ~Field(…) # Equivalent to `Any ~ Field(…)
One advantage of a new operator is the opportunity to set a precedence that’s lower than | (and &).
~ already exists for the __invert__ method when used on a single object, like ~obj (it’s implement for Ints only iirc, books had them but it got deprecated). Using the same operator for something not related that’s used in a binary context seems confusing.
~ is not established in any way already. For inversion, it is. Similarly, the two other unary- operators, + and - (__pos__ and __neg__) are established mathematically both for making an object positive/negative and adding/subtracting two objects. ~ is not used in any sich way. Additionally, as @till-varoquaux already mentioned, changing tooling, and the AST /parser for such a minor change that already has a proposed operator is a weak reason. Here the lexer/tokenizer wouldn’t need to change, but that’s the only difference between this idea and the one of using <@.
Therefore I’d be a strong -1 for adding new operators for this. Reusing ‘old’ ones seems fine though.
To me this is another proof of the necessity for a separate AST and a type-hint-specific mini-language suggested in Idea: Simpler and More Expressive Type Annotations. There is no scenario where @ needs to work as a math operator[1] in a type hint, and there’s equally no scenario where @ needs to work as an Annotated operator in a non-typing context. The fact that the two have to share the same AST is causing all these problems that are being discussed here, and IMO this fact alone is massively hindering typing in Python.
the same issues would arise with any other operator too ↩︎
I don’t agree that this is not worth the churn, although it is a real downside. We pay these costs for new syntax fairly regularly, recently with match-case, the type keyword, generic syntax, and lazy imports. It’s not that these changes to the language don’t take effort to support, but they’re considered worth it because the result is a better language.
I give proportionately much more weight to the long term state of the language. Tooling will heal with time, but if we make an inferior choice in design, we’re stuck with it.
(Also, I’m not sure how much it’s relevant, but most parsers and tools will find it pretty easy to add support for a new infix binary operator.)
