If type implements __matmul__, then __rmatmul__ is pretty much irrelevant. (I guess type.__matmul__ could first check if the RHS is an instance of Metadata and if so defer to it, but that feels pretty backwards, and can easily see such a follow up proposal getting pushback for that weirdness.)
With __rmatmul__, the issue only appears if type(T) in T @ x implements __matmul__.
It’s somewhat unfortunate, but I think it’s okay to add a disclaimer to Format.FORWARDREF that non-standard operator implementations on meta types can lead to unspecified behavior.
Deferred evaluation has plenty of footguns by itself, and partial-evaluation via forward refs has even more. Making some explicit assumptions about the behavior of types used in such annotations seems like the responsible thing to do anyway.
Compared to some of the other issues, this one is probably the easiest for type checkers to warn about.
I’m in full agreement that the current ergonomics of Annotated aren’t good, but that doesn’t mean this needs to be the solution to it.
I’d like a real answer to what I pointed out above with just adding a meaning to something that already exists and works, and improves this use case, but that static typecheckers reject and runtime tools currently dont handle.
I’d think that going from
from typing import Annotated
from some_lib import BaseModel
class Foo(BaseModel):
a: Annotated[int, ValueRange(1, 10)]
to
from some_lib import BaseModel, RangedInt
class Foo(BaseModel):
a: RangedInt[1, 10]
is a bigger ergonomic improvement offerable by libraries to their users than offered by this proposal, and the libraries in question that would benefit the most from it just need to provide a few type aliases + agree that “when given a type alias type, runtime tools may interpret the synthesized type variables as value placeholders when dealing with replacement within annotated.” and tell static type checkers “Annotated is the escape hatch for things you aren’t standardizing, if a typevariable is only used as a value in annotated metadata within a type alias, it’s not for you, dont error on it”
How would this work for something like pydantic.Field or fastapi.Query? And what about multiple metadata items? This would only solve a small part of the problem, and you haven’t yet specified what the actual semantics of these value-TypeVars should be (I don’t think it’s as simple as you make it seem).
And, again, it makes it more difficult to see what the runtime type is, not less.
I’m envisioning a case like the FastAPI Doc case as the worst example. Some type hints can already get rather verbose.
A choice of syntax that doesn’t have an ambiguous meaning.
The concrete problem is that not breaking in the case of the cattrs example I gave when switching to the new syntax requires making additional assumptions about the undefined object (in that you have to assume __matmul__ isn’t defined).
I’m not saying I’d be pro the new syntax, but I wouldn’t be concerned about its addition.
This isn’t the case because __matmul__ takes precedence over __rmatmul__ if it exists and you can’t know it doesn’t exist if you don’t know what T is (you have to make the assumption that it doesn’t define __matmul__).
Edit: I guess you did say that and I kind of misread this. But yes - this is still a required assumption.
I’m going to be blunt, I avoid both of these libraries. I don’t like them, and think there are better options in the same space that do structuring and destructing less intrusively, with better control knobs. I’m not well-positioned to give you a full example that is a 1:1 for these.
I’m not a believer in shoving as much data as possible in annotations, and it was somewhat intentional that this best maps to things that are still types in theory, just not in the static type system.
The above also could support things like
PatternValidatedStr[r"some|regex"]
The entire point of what I was saying was “let the libraries provide the better ergonomics by wrapping Annotated for the user, and provide those aliases.” The semantics should intentionally be as loose as possible, because any runtime tool should be able to define the semantics for their own metadata types. Annotated is a standardization escape hatch, we shouldnt be attaching meaning where we dont have to.
See the above linked post. It’s just an alias to Annotated and saying that static type checkers shouldn’t be blocking what can already be done.
That’s subjective, and it’s also subjective whether seeing the runtime type is even a useful goal for the given use case of dataclass-like libraries. This still presents the type first, it just gives it a useful name (via alias) that encapsulates both type and constraint on valid values of the type (which is a form of type refinement)
This is something I’ve found myself wishing for in the times when I do want to use Annotated. It would make a bigger impact on the ergonomics of Annotated for mii than the change in the PEP, tbh.
Agreed, but how would you envision this being implemented? I can’t imagine a good way to describe all non-standard Annotated types that would still be usable to type checkers. The closest I can think of is PEP 827, but even that might not be enough for something like RangedInt.
As you said, the whole point of Annotated is to denote something non-standard. There are definitely things used in Annotated that could be standardized (such as ValueRange), but that doesn’t mean we should avoid or remove Annotated entirely. In fact, this proposal might help improve Python’s type system over time; if we notice a very common case for Annotated, we can consider standardizing it. But to take that first step, we have to make Annotated easy to use.
Could you elaborate on that example? I’m still having a hard time understanding how it applies practically. I might be missing some context, sorry!
Is the issue that you think annotationlib should be aware of types that implement __matmul__, so if type does its own weird thing rather than the default Annotated behavior, then it retains that? If so, I’ll reiterate my point from my initial post: this is a hypothetical problem. I would be very, very surprised if you could find a library that implements __matmul__ on a metaclass, uses it in an annotation where this would break, and also uses annotationlib to inspect those annotations.
But even if that’s not the issue you were describing, I don’t understand how any potential problems that apply to @ wouldn’t also apply to |, and | has seemed to work out pretty well. I’m inclined to quote “practicality beats purity” here.
To also be blunt: I think that means you’re missing a lot of the context behind this proposal. We want third-party libraries to be able to use type annotations to ease developer experience, and users of FastAPI and Pydantic are two of the major beneficiaries here. Let’s not allow personal opinions about these libraries to leak into the perspective on this proposal; many Python users who use both FastAPI and Pydantic would largely benefit from improved Annotated syntax, regardless of whether anyone considers them good or bad libraries. Even if FastAPI and Pydantic were objectively terrible, I would still be +1 on this proposal, on the basis that we would be improving the experience of millions of Python users.
The whole point of annotated is for metadata not intended for static type system use.
If we just say that type aliases are allowed to parameterize annotated values, and that when it happens, the values parameterized are explicitly for the use of the library whose metadata object is being parameterized, then type checkers don’t have a reason to error for the example I gave.
ie. I’m proposing to allow type alias statements to be shipped by libraries like pydantic that wrap annotated, and that typecheckers should allow this, and continue only using the part of the annotation that’s for them, even when aliases are involved.
That also means pydantic could have a public module containing simplified aliases for “common” uses of Annotated that would work the moment pydantic was updated, rather than waiting for python 3.16+, and we can focus on looking for what that can’t do before looking for new syntax that might be a better fit with the existing options more explored.
I don’t think I have here. I’m 100% for improving on use of annotations, Annotated or otherwise, to make them more useful, but I’m against this proposal. I’m not against the proposal because of pydantic, I like a handful of libraries that would benefit from this as well, but when someone asks me how it would look for users of a specific library, as I’m not a user of that library, I’m not the best person to try and adapt that example.
I mentioned above that I’m aware of existing code that uses matmul on types for matrix multiplication. It’s unfortunate that the code isn’t public to point to and that I can’t share it, but this proposal clobbers a valid use of the operator for what it was originally designed for, and is effectively saying that specific existing use isn’t valid, despite it actually using the operator for what it was introduced for because the authors of a handful of big libraries want to use it for something else too.
Maybe. I’m not really one to dive into whether Annotated is the best solution, and again, I think this sort of discussion should have taken place when PEP 593 was being proposed. My stance is simply that Annotated is being used and is cumbersome, so we should improve it. If you want to create an alternative proposal, you’re already on the right forum .
I missed that post; thanks for the pointer. That does sway my opinion to some extent, but it is really quite unfortunate that the code isn’t publicly available, because I’m essentially changing my opinion on a feature I’d really like based on no concrete evidence. I still have two questions, though:
Why was __matmul__ implemented on the type? Matrix multiplication is something that happens on a value, not on a type. (I’m just trying to understand the context in more depth – it seems quite odd that someone would actually be using @ on a type in practice.)
Would the __rmatmul__ switch make this easier to swallow? If we do that, it sounds like the code in question would continue to work, just not when inspected by annotationlib.
It’s perfectly valid on types. The simplest example I can give is M[1, 2] @ M[2, 3] resulting in M[1, 3]
It’s proven useful in various python notebooks to have various assumptions checked and written out in this way (with some level of testing that outputs match expectations during development), and having the types actually match the way the values will be interacted with helps with ensuring nothing is lost in translation. There’s a mix of people who I would say are more focused on software development, and those more focused on physics collaborating on the project involved.
Maybe, it depends on if that means typecheckers are now angry that there’s use of @ somewhere that isn’t for annotated involving types. It’s also undecidable in an open type system with subtyping whether __rmatmul__ and/or __matmul__ are not defined, so relying on __rmatmul__ seems like it poses thorns for the static type checker portion.
If static typecheckers are willing to in the absence of a visible __rmatmul__ or in the presence of a visible __matmul__, not treat this as if it’s a shortcut to annotated, it goes from “I’m opposed” to “I won’t use this, and will continue to use Annotated”
PEP 593 explicitly includes an example of using Annotatedto implement new typing constructs. The goal was for these constructs to degrade gracefully in type checkers. I stepped away from working in Python shortly after releasing PEP 593. However, my hope was always that Annotated would be used for static analysis.
If we secure a cleaner syntax, I will explore adding formal static support.
Thank you for pointing this out, We should carefully document the edge cases in FORWARDREF_STRUCTURAL. I think this behavior is consistent with how type checkers handle infix operations in type hints.
AFAIK static type checkers do not currently track dunder methods in a typing context. They operate purely on syntax. For example, mypy, pyright, pyrefly, ty and zuban all accept this:
def f() -> None | None:
return
They accept this even though None lacks both __or__ and __ror__. Type checkers would handle @ by recognizing the AST node, not by looking for a dunder method.
The current __matmul__ proposal is trivial for type checkers to adopt. Support for __rmatmul__ might be significantly harder. It would require a typing spec change and force every checker to track more information. We should carefully consider this implementation burden.
This is an interesting use case. Currently, standard type checkers seem to reject M[1, 2] @ M[2, 3] as an invalid type expression regardless of how M is defined. Do you mind sharing more details about your code? Maybe you have a custom type checker plugin?
Assuming this is primarily for runtime evaluation, this PEP should not break your workflow. You can still use a custom metaclass on M to overload __matmul__. That will preserve your custom matrix logic. This PEP only defines the default behavior for standard types.
I can’t share as much as would be necessary to fully demonstrate. It’s an uncomfortable position to be in, needing to argue that closed-source code shouldn’t end up broken by this.
The internal type checking tool we use is standards compliant in what it must accept, but doesn’t reject some things that people are used to having rejected by the “standard driving” static typecheckers, and it understands slightly more expressions on types that result in types.
Without a working implementation and typecheckers agreeing on how they will handle this (or mandating how they do in the proposal), it’s not possible for me to say if this will or won’t cause problems given the intersecting issues here, and the amount to which typing has been pushed out of the language and onto tools. I’m not optimistic that this will be handled well, and if it isn’t and interacts poorly with other tools as a result, I’m going to still be in a position where I can’t even fully share context.
It might be possible to paper over any issues further with stubs for other teams consuming the library, but without more specified, it’s hard to be sure.
Given existing issues with typecheckers and dunders and existing issues where behavior diverges in an annotation context from a runtime context, I’d much rather see a new operator or new keyword, rather than reusing matmul in any capacity for this if it’s already possible to determine that expanding what is allowed in type aliases would be insufficient.
Sorry, that was poorly worded on my part. I am in full agreement on that. My own uses for annotated are mostly refinement types. by “Static type system” there, I do mean the standard, not all possible tools, and I view the existing errors for certain uses of annotated in type aliases as a failure to degrade gracefully.
I completely understand wanting to protect a workflow that works well for your team. It is stressful when standards discussions seem to threaten internal tools.
However, looking at the Expression Grammar section of the Python Typing Specification, your example already falls outside the official standard in two ways:
The @ operator is not in the allowlist of valid type expressions.
M[1, 2] is invalid because literals (like 1 and 2) can currently only appear as arguments to Literal or Annotated.
Since your parser already implements its own custom grammar rules, this PEP will not force you to change them. PEPs define the baseline for public standard tools, not the ceiling for private ones.
I understand that, I’m not concerned about the use in an annotation context because that’s effectively constrained to our custom tooling already. I’m more concerned with how this interacts with runtime use and various tools that build on typing based standards and already prefer the typeshed or various typing standards over accurately modeling runtime (Such as language servers). I’m concerned about the ambiguity that exists in a value context.
Python’s type system is open (type checker only ever has partial knowledge, things are assumed to be swappable with any compatible type), so dunder priority between LHS and RHS and use of reverse dunders is technically undecidable without the LHS and RHS’s dunder implementation being from provably disjoint types.
What I’m worried about here is that in a runtime context, not an annotation one, users suddenly go from tooling seeing a matmul definition and understanding M[...] @ M[...] = M[...] to seeing a standard for matmul on types, and ending up with tooling using that standard rather than the definition of the type, and getting M[...] @ M[...] = Annotated[M[...], M[...]] for inference behavior, or complaints about incompatible definitions.
|does have the same problems, but in general Union not being partially resolvable doesn’t have the same issues as Annotated might.
from annotationlib import call_evaluate_function, Format
from typing import Union
type a = Union[unknown, str]
type b = unknown | str
print(call_evaluate_function(a.evaluate_value, format=Format.FORWARDREF))
print(call_evaluate_function(b.evaluate_value, format=Format.FORWARDREF))
Let’s make a new example, based on something people have expressed a desire for in attrssee this issue[1].
from annotationlib import get_annotations, Format
from types import SimpleNamespace
from typing import Annotated, get_origin, get_args
def validate_notnone(instance, attribute, value):
if value is None:
raise ValueError(f"{value!r} must not be None")
class FieldLike(SimpleNamespace):
def __rmatmul__(self, other):
return Annotated[other, self]
class Example:
a: Annotated[Unknown, FieldLike(validator=validate_notnone)]
a_anno = get_annotations(Example, format=Format.FORWARDREF)['a']
print(get_origin(a_anno))
print(get_args(a_anno))
So far, so ‘good’. We can extract FieldLike from Annotated because this is partially resolvable - unlike Pydantic our theoretical library is more like dataclasses and will just treat the forward reference for the actual type as something we don’t care about rather than as an error.
Now we drop support for those old versions of Python that didn’t support @ and ‘upgrade’ to the new syntax.
The partial information is gone - the outcome of which is that our class is no longer validating the input. The whole annotation is treated as one opaque ForwardRef which is essentially ignored.
In order for this to work, you would have to make the assumption that Unknown doesn’t define __matmul__ which may not be correct and isn’t made for __or__.
Again, I don’t like this for runtime reasons (it’s slower to need to extract and re-combine the information from annotations and doing so brings in the complexities of the PEP-649 annotations space). ↩︎
No. If you make the two objects type aliases you are wrapping them in TypeAliasType objects. TypeAliasType.__or__ creates a union at runtime. You, the user who put the objects inside the type aliases, made that choice[1].
The unknown objects in an annotation are not required to be types. Type checkers will complain, but just because it’s an invalid type hint doesn’t mean it’s an invalid annotation. The annotations Format a library uses may not be in control of the person who wrote the annotation.
In the current situation if some annotation uses M1 @ M2 where they are undefined, this either fails (under VALUE) or becomes ForwardRef("M1 @ M2", ...) under FORWARDREF. Under FORWARDREF_STRUCTURAL this would be typing.Annotated[ForwardRef("M1", ...), ForwardRef("M2", ...)] - if it turns out that these have their own definition of __matmul__ then you’ve ended up with the incorrect result. This is worse if you subsequently pass this result onto something that relied on the original behaviour.
Format.FORWARDREF_STRUCTURAL feels more like something that should be a typing feature to use on a ForwardRef because the underlying assumption is that the annotation is a type.
This way you retrieve the annotations as Format.FORWARDREF and get ForwardRef("M1 @ M2"), try typing.evaluate_forward_ref(ref, assume_forwardrefs_are_types=True) and see Annotated[ForwardRef("M1"), ForwardRef("M2")], and see there’s no metadata you need but keep and pass on the original ForwardRef("M1 @ M2").
Of course my preference if we have to go down this road would be to do this on something like Format.DEFERRED (essentially what is now Reannotate) rather than FORWARDREF as it means you could get the annotations in this format, call a function to evaluate in this manner without needing to check if it’s a ForwardRefand as a bonus you’d be able to create a correct __annotate__ function with the annotations you have[2].
My overall preference is still to either come up with something that doesn’t suffer from this ambiguity or to keep the status quo.
You could also argue that within a type statement, the evaluation should also assume the contents are types and convert | to Union but it currently operates on the same basis as for annotations. ↩︎
Edit: Yes maybe I’m a bit grumpy that this isn’t possible with existing annotation formats or with FORWARDREF_STRUCTURAL. ↩︎
Why? If it works in your project, I don’t see why you would want to use the old way, other than out of spite.
I feel like I’m repeating myself on this point. There are very few people who implement __matmul__ on types; the only example we have is in closed-source Jupyter notebooks, and it doesn’t sound like they were using them in annotations.
Fundamentally, I think it’s a bad idea for annotationlib to maintain support for forward references with weird, non-standard operators that people made up, because that will arbitrarily constrain Python’s type system in the future (like it potentially is here!). When we inevitably want to add intersection types to Python, will you be against using & because it would break people who implement __and__ on a metaclass?
Consider the very limited number of cases where this actually applies. This only breaks with forward references, which are a weird form of black magic specific to static typing. If I were using annotations for something completely unrelated to static typing, I wouldn’t expect forward references to work with it, because that’s a typing-specific construct.
It just absolutely seems like a hypothetical problem to me. I apologize if I’m sounding terse, but I really don’t want to continue entertaining this argument unless I see a concrete, practical example of where this becomes a real problem, rather than an artificial one.
Sure, but they can certainly choose not to care. If I were a maintainer of a dataclasses-like library and got a complaint from a user that foo: ImplementsMatmulNotYetDefined @ anything doesn’t work correctly with my library, I would just tell them, “garbage in, garbage out”.
Think about this from a user perspective. If you know that @ is used for Annotated, the library you’re using takes advantage of Annotated for metadata, and your editor is already yelling at you for doing M[1, 2] @ M[2, 3] in an annotation, do you really think it’s Python’s fault that your code doesn’t work and not your own?
.