My view on further thought is that actually requiring users to do AST transformations at runtime is just going to make performance worse and make handling annotations more difficult.
Any tool which works with annotations at runtime will essentially be forced to perform the transformations in order to work, at which point I think it’s better to do the transformations before they are baked into the __annotate__ functions. @MegaIng brought up a similar thought in that thread.
As such, adding additional ways to represent types is not a goal of this proposal. The goal is to add a format which can both be easily evaluated in all of the supported annotation formats and can be used to create new __annotate__ functions that work correctly and completely.
I disagree with the characterisation that this is adding “this much complexity”. Both the evaluation logic and the construction logic underlying DeferredAnnotation are already part of annotationlib.
Format.DEFERRED is built from the implementation of Format.STRING, just without the final step of string conversion. The evaluation logic is mostly extracted from ForwardRef and moved to EvaluationContext in order to be shared.
The change DEFERRED brings is that it makes it possible use to these internals to make creating new __annotate__ functions much easier. This moves the complexity out of dataclasses and into annotationlib and makes it reusable for anything else that needs to construct an __annotate__ function.
Other examples of issues
You characterise these as “a few dataclass gotchas” but note that the third dataclasses example is what SQLAlchemy was doing that caused its use of dataclasses to break in 3.14.1 when the logic I’d written for the __annotate__ function mistakenly assumed the annotations that were there when dataclass(...) was called would still be there. The patch in 3.14.2 just prevents this from breaking if the annotation isn’t there, it doesn’t make the annotation work correctly as it does with DEFERRED.
Some other examples I’m aware of:
`attrs` leaks forward references in VALUE and STRING annotations
Currently attrs is getting PEP-649 annotations ‘wrong’ and it’s not possible to easily fix it for the same reason dataclasses doesn’t have a proper fix.
from attrs import define
from annotationlib import get_annotations, Format
@define
class Example:
examples: list[Example]
print(get_annotations(Example.__init__, format=Format.VALUE))
print(get_annotations(Example.__init__, format=Format.STRING))
Output
{'return': None, 'examples': list[ForwardRef('Example', is_class=True, owner=<class '__main__.Example'>)]}
{'return': 'None', 'examples': "list[ForwardRef('Example', is_class=True, owner=<class '__main__.Example'>)]"}
This is because attrs doesn’t write an __annotate__ function, it’s just putting the Format.FORWARDREF annotations into __annotations__. As such it leaks ForwardRef objects into VALUE and STRING annotations.
Handling this with DEFERRED annotations is a much simpler implementation change than rewriting all of the attribute gathering logic purely to attach to __init__. The logic would be more complicated for attrs than for dataclasses as it also has to care about the annotations in any converter functions and it would be just as broken as dataclasses implementation.
This could broadly use the same fix as my dataclasses changes.
My own class building library uses STRING if VALUE fails to avoid leaking ForwardRef
My own class builder currently ‘cheats’ and uses VALUE annotations where possible, when those fail it uses STRING annotations because FORWARDREF annotations lead to the issues attrs has.
from ducktools.classbuilder.prefab import Prefab
from annotationlib import get_annotations, Format
class Example(Prefab):
examples: list[Example]
print(get_annotations(Example.__init__, format=Format.VALUE))
print(get_annotations(Example.__init__, format=Format.STRING))
Output
{'examples': 'list[Example]', 'return': None}
{'examples': 'list[Example]', 'return': 'None'}
Constructing a ‘correct’ __annotate__ function for this is about as complicated as it is for attrs, but slightly different - __prefab_post_init__ works as a partial __init__ function and changes its annotations.[1]
I’d probably replace the use of STRING annotations with DEFERRED annotations. VALUE annotations would likely still be used where they succeed for performance reasons.
Bonus digression on current annotations implementation details
In PEP-749 as part of the implementation of deferred annotations, the way of recognising if an annotate function can be called in the ‘fake globals’ context is to call it with VALUE_WITH_FAKE_GLOBALS. If it it supported, this outputs the same annotations as VALUE.
Unfortunately, you can’t check this within a fake globals context as names like NotImplementedError and annotationlib.Format don’t exist in that context. Thus you have to call the annotate function in the standard globals space first to see if it raises NotImplementedError.
As such, the other formats are by their nature always going to be slower than Format.VALUE in the cases where VALUE would succeed.
DEFERRED actually exploits this required call and uses successful results to pre-fill it’s cache for VALUE evaluations.
A Pydantic developer wants to be able to distinguish *which* annotations fail to evaluate
Another example is this helps to handle functions such this one, based on an example given by @Viicos in this comment on an annotations issue
This is given as roughly the logic that Pydantic has to use:
def get_model_type_hints(cls):
hints: dict[str, tuple[Any, bool]] = {}
for base in reversed(cls.__mro__):
anns = get_annotations(base, format=Format.FORWARDREF)
for name, type_ann in anns.items():
try:
evaluated = typing._eval_type(type_ann, globalns=..., localns=..., type_params=...)
hints[name] = (evaluated, True)
except Exception:
hints[name] = (type_ann, False)
return hints
First, this requires the library handle the evaluation contexts itself from somewhere and second it requires using an internal typing._eval_type which special cases traversal of some typing objects.
The current[2] DEFERRED equivalent of this would be:
def get_model_type_hints(cls):
hints: dict[str, tuple[Any, bool]] = {}
for base in reversed(cls.__mro__):
anns = get_annotations(base, format=Format.DEFERRED)
for name, type_ann in anns.items():
value = type_ann.evaluate(format=Format.FORWARDREF)
hints[name] = value, type_ann.is_evaluated
return hints
This no longer requires handling of evaluation contexts directly and doesn’t do any special case traversal of annotations to try to uncover nested ForwardRef objects like _eval_type.
I’d be interested if there are other examples, but my primary interaction with runtime annotations is through dataclasses, attrs and ducktools-classbuilder. All of which essentially have some variation on the same ‘constructing __annotate__ for a generated __init__’ problem and all need their own bespoke solutions without something like DEFERRED.
Yes, static tools don’t understand this correctly. The annotation for
__init__should be that of the argument to__prefab_post_init__, not the class annotation. ↩︎I only just added the public
.is_evaluated- previously you would have had to check internals. This indicates if aVALUEannotation has been cached. ↩︎