My ‘dislike’ of the PEP largely stems from the fact that it’s chosen to reuse @, but isn’t about how the symbol looks. @ in a normal expression has a different, defined, meaning and this reuse causes problems that wouldn’t exist if a new unused symbol or symbol pair were used.
In order to try and work around the problem this creates, the PEP proposes a new annotation format that doesn’t solve existing problems with PEP-649 annotations but instead creates new problems and quirks along with the churn of yet another new way of obtaining annotations.
As the PEP seems determined to go ahead with this operator reuse I’m going to try to be constructive and illustrate the issues this new annotation format creates and propose a smaller, internal change to Format.FORWARDREF that could still support it (without breaking the ‘shaped’ types at runtime - or at least providing them with an escape hatch).
Problems with the current Format.TYPE proposal
My biggest objection to the current PEP is to making @ ‘work’ in the case where objects are known and the operation should fail. This may seem nice, but as a consequence this will essentially break anything that currently handles type annotations using VALUE and FORWARDREF.
- Supporting
None @ <object> creates new ‘valid’ annotations without undefined names that fail under VALUE.
>>> from annotationlib import get_annotations, Format
>>>
>>> class Example:
... a: None @ "Arbitrary String"
...
>>> get_annotations(Example, format=Format.VALUE)
Traceback (most recent call last):
File "<python-input-15>", line 1, in <module>
get_annotations(Example, format=Format.VALUE)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/ducksual/src/temp/cpython/Lib/annotationlib.py", line 1189, in get_annotations
ann = _get_dunder_annotations(obj)
File "/var/home/ducksual/src/temp/cpython/Lib/annotationlib.py", line 1381, in _get_dunder_annotations
ann = _BASE_GET_ANNOTATIONS(obj)
File "<python-input-14>", line 2, in __annotate__
a: None @ "Arbitrary String"
~~~~~^~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for @: 'NoneType' and 'str'
>>> get_annotations(Example, format=Format.TYPE)
{'a': None @'Arbitrary String'}
VALUE fails even if there are no forward references due to the annotation using the new syntax.
As Format.TYPE also allows for forward references this means VALUE can no longer be used as a way of checking for forward references if you also potentially use the new syntax. There is currently a pattern in some places where annotations will be gathered as Format.FORWARDREF and then later when it’s necessary to know all of the information, they are re-evaluated as Format.VALUE and its success or failure indicates if the reference is resolvable - as Format.TYPE supports forward references you can’t make this distinction if you’re relying on the new syntax.
- The current implementation also supports
arbitrary_object @ arbitrary_object which leads to some other strange behaviour.
>>> s1 = "StringRef"
>>> s2 = "Other String"
>>>
>>> class ExampleVariable:
... a: s1 @ s2
...
>>> get_annotations(ExampleVariable, format=Format.TYPE)
{'a': ForwardRef('StringRef', is_class=True) @'Other String'}
Note that the ForwardRef created in this case is unresolvable but you could consider that an implementation bug. Using the literal strings just fails as it does under annotations now (as it should). For some reason if the LHS is a string, literal or otherwise, it always gets converted into a broken ForwardRef.
- The format currently assumes any kind of subscripting that fails makes a GenericAlias for some reason?
>>> class Example:
... a: str[int, float]
...
>>> get_annotations(Example, format=Format.TYPE)
{'a': str[int, float]}
>>>
>>> # Weird dict example
>>> d = {'a': "String a"}
>>>
>>> class Example:
... a: d['a']
... b: d['b']
...
>>> get_annotations(Example, format=Format.FORWARDREF)
{'a': 'String a', 'b': ForwardRef("d['b']", is_class=True, owner=<class '__main__.Example'>)}
>>> get_annotations(Example, format=Format.TYPE)
{'a': 'String a', 'b': {'a': 'String a'}['b']}
Note that that’s not a literal dict for ‘b’ under Format.TYPE, that’s a generic alias of a dictionary. I don’t know what purpose this functionality is supposed to serve? If you’re evaluating an annotation, what use is it to make a generic alias of something if you don’t know what the origin is? This just makes it more complicated to evaluate the forward reference later.
I think this behaviour of making things work between objects that are known and where the failure is known at runtime should be dropped.
As far as I can tell this behaviour primarily exists to support the rare case of annotating something which can only have the value None but None is more likely to be part of a union at which point the union will support @. I wouldn’t be surprised if the most common place where None @ <annotated detail> would occur would be in error due to writing T | None @ <annotated detail> and getting bitten by operator precedence attaching it to None rather than the union.
If it is necessary to annotate None on its own it’s possible to use a type alias, the actual NoneType or the existing Annotated[...] syntax but I don’t see this as a common or serious enough problem to make it worth breaking __annotations__ and the VALUE format over.
Linters and/or type checkers should indicate that None @ <thing> doesn’t work if someone does try to use it. Upgrade tools should not ‘upgrade’ Annotated[None, ...] to the new syntax.
Problems with PEP-649 annotations that are made more difficult to fix with Format.TYPE
Adding a new format that’s subtly different makes the ‘annotation intermediary’ problem that already exists under PEP-649 annotations worse.
This is where something from library A has to get the annotations but they end up used by library B (like the earlier dataclasses and cattrs example).
There are essentially currently 4 states an annotation under PEP-649 exists in:
- Fully resolved, as retrieved from
__annotations__ or Format.VALUE
- Completely unresolved, as they exist inside the
__annotate__ function but inaccessible separately.
- Partially resolved, as retrieved from
Format.FORWARDREF
- Resolved into strings, as retrieved from
Format.STRING
The current proposal adds another state which is:
- Partially resolved with additional “structural” assumptions which are not guaranteed to hold when names can be resolved with existing annotations.
An existing problem libraries have is that there’s no (stdlib) way to get the completely unresolved annotations that would allow them to be evaluated later. This generally means libraries either ‘leak’ ForwardRef(...) into VALUE and STRING annotations, lose annotations where they’ve modified __annotations__ after a tool has processed a class, or fail to evaluate __annotations__ in cases where it should be possible to succeed. (Note: these are specific examples I’m aware of but I don’t want to call out libraries here.)
The new TYPE format and @ syntax as they currently are adds a new dilemma where Annotated may wrap something the tool does need to check so it would be forced to use the new format which then bakes in the potentially incorrect assumptions made to anything downstream. So this essentially just forces everything to switch to Format.TYPE without adding access to the unresolved annotations that they actually need.
You will end up with some tools supporting it and some tools semi-supporting it when there are no forward references and more churn and compatibility hacks as tools now need to support old annotations, __future__ annotations, Format.FORWARDREF annotations and now Format.TYPE annotations under 3.16.
Proposed change
My suggestion here is then to drop Format.TYPE entirely and allow Format.FORWARDREF to evaluate @ in a more restricted way than the current proposal that will still cover the majority of actual use cases without breaking in the rare cases where @ has some other meaning.
Resolving A @ B in Format.FORWARDREF:
- If both
A and B are unknown → leave this as ForwardRef('A @ B')
- Both objects are opaque, there’s nothing to be gained by resolving this to
Annotated[ForwardRef('A'), ForwardRef('B')]
- Resolving this to
Annotated can only break cases where this isn’t what it will eventually resolve to.
- It’s also easier to resolve later as one
ForwardRef than as separate objects inside Annotated
- If
A is known and B is known or unknown → Resolve A @ B normally
- If
A is a type this becomes Annotated[A, ...] as expected at runtime, there is no need to special case
- In the same way that
Format.FORWARDREF fails now, if A @ B fails it gets wrapped in a ForwardRef
- If
A is unknown and B is known → check B.__rmatmul__
- If there is no
B.__rmatmul__ then resolve to Annotated[ForwardRef('A'), B]
- If it exists, resolve to
ForwardRef('A @ B')
- If
type actually defines __rmatmul__ in the final version of this it will need to be special cased here
- This is the only actual change to how
Format.FORWARDREF resolves annotations
This would allow things like unknown @ Field(...) to work while preventing M[1, 2] @ M[2, 3] from breaking even if M is not yet defined when the annotations are retrieved. I think this would be much less disruptive than adding a new format that requires a greater number of additional assumptions about what’s in an annotation.
If necessary we could also add a decorator along the lines of @no_annotated_syntax in either typing or annotationlib that wraps the __annotate__ function and prevents this resolution under Format.FORWARDREF.
Something equivalent could be done for | if it’s deemed necessary but I’m not sure it is. Sure you can get some runtime information out about the type but are there actually runtime tools that can do anything with partial information about a type? I would expect most need the entire type to be resolvable when they operate, the same for if type intersections were ever added.
As a side note on the current implementation I really don’t like the change from T @ extra to T @extra in the representation. The original PEP had this as T @ extra. The operator that’s used nearly always belongs to T, it seems wrong to attach the symbol to the object that doesn’t define it.
The PEP claims this is similar to decorators which are associated with matadata but I’ve never associated decorators with “metadata”. I usually expect them to perform some kind of runtime transformation on whatever they’re decorating (for example: @property, @staticmethod, @dataclass, @cache, …) I would suggest changing this back.