Idea: Simpler and More Expressive Type Annotations

Typing
81

Yeah, you’re right, we do need a namespace object to properly resolve variables.

I’ve been testing out a bunch of variations of string or AST based annotations. I’ve been meaning to make a more comprehensive post, but keep getting sidetracked with more alternatives (that mostly don’t really lead to meaningful benefits unfortunately). But my observations mirror yours, string based annotations are pretty slow, both compared to the current approach and AST objects. They do have the advantage that they’re slightly more memory efficient (about 1% smaller modules), with the AST approach using about as much ram as the current approach.

I’ll try to wrap up my testing and unnecessary optimising and make a new post with more explanations of them and test results. I’m feeling confident that if we compare our results we’ll be able to get an implementation together that’ll work out nicely!

2 Likes
82

You can also get this one to fail on current Python if you also put in an annotation that will fail with a non-name error and force get_annotations into a fallback mode. (This also revealed a bug with conditional annotations and Format.STRING).

from annotationlib import get_annotations, Format

x = "global"
def outer(broken=False):
    x = "nonlocal"
    class Cls:
        global x
        ann: x
        ref: undefined
        if broken:
            break_annotation: object.does_not_exist  # force an AttributeError
    Cls.x = "class"
    return Cls

print(get_annotations(outer(False), format=Format.FORWARDREF)['ann'])  # global
print(get_annotations(outer(True), format=Format.FORWARDREF)['ann'])  # class

The ‘almost but not quite enough’ context is what’s being used here for the case where the annotations are broken by an attribute error. It’s also what I currently use for deferred annotations in reannotate, which is my current implementation of the deferred annotations I’ve mentioned before.

You need the namespace per annotation. Take the example of dataclasses __init__ that can contain annotations that come from different modules. If its __annotate__ function is to support the new format in a useful manner it needs to be able to provide the context for each annotation separately.

The DEFERRED format I’ve been speaking of is essentially designed for this. It’s a pairing of some arbitrary object - currently most commonly either an AST expression or a string - with an ‘EvaluationContext’ object required to evaluate it.

>>> from annotationlib import call_annotate_function, Format
>>> from reannotate import get_deferred_annotations, ReAnnotate
>>> class Example:
...     a: int
...     b: list[str]
...     c: unknown
... 
>>> annos = get_deferred_annotations(Example)
>>> print(annos)
{'a': DeferredAnnotation('int'), 'b': DeferredAnnotation('list[str]'), 'c': DeferredAnnotation('unknown')}
>>> annos['a']._obj  # String as it's just a name
'int'
>>> annos['b']._obj  # AST expression
Subscript(value=Name(id='list', ctx=Load()), slice=Name(id='str', ctx=Load()), ctx=Load())

Now, in its current form it’s not enough for expressions not supported by Format.STRING but that could be an internal change rather than a public format.

Draft PEP: More Expressive Type Expressions
83

Why can’t we just use nested namespaces? Like if we have {"a": MyClass} in namespace first_namespace and {"b": MyClass} in second_namespace we can just merge them to effectively be {"a": first.MyClass, "b": second.MyClass} in a namespace {"first": first_namespace, "second": second_namespace}. I don’t think we really care about preserving the exact shape of the AST, just that they evaluate correctly.

But regardless, I think that’s an implementation detail that can be considered later and seperately. No matter how we implement the new annotation mechanism, we’ll need some kind of namespace lookup (or some pretty complex duplication of symtable rules). And we can implement that lookup however we want, independently of how we implement the annotations.

I’d have to look into how annotations are most often consumed and annotate functions generated to make a judgment on what variant is the nicest to use. Having the namespaces folded into the annotation objects also has the advantage that all annotation formats just return a dict mapping variable/attribute names to stuff, rather than the new format being special in that it returns such a map and a namespace. But it also duplicates a bunch of information in the vast majority of cases.

84

Regarding the dataclasses example, as I recall that currently works by having a custom __annotate__ function that calls the various original __annotate__ functions. Why is that not possible under Imogen’s idea?

85

I don’t consider the dataclasses __annotate__ function something to aspire to. It fails in some cases where it could succeed and isn’t guaranteed to give the same annotation information that was there when the class was constructed. It’s also more complicated than it could be.

  1. The values it gives reflect the annotations that are there when someone calls get_annotations, which isn’t necessarily the same as the annotations that were there when dataclasses itself called get_annotations to construct the dataclass (as we discovered when the initial implementation broke something SQLAlchemy was doing with temporarily replacing the annotations).

  2. It can’t handle Format.VALUE if there’s an annotation on a class in the MRO that isn’t part of __init__ that has a forward reference even though the resulting annotations under Format.FORWARDREF won’t contain any forward references.

  3. Field.type should also be backed by a deferred annotation so it can correctly resolve self-referential annotations.

For the AST format to be usable it needs to be possible to at some point evaluate the actual objects. You could return the AST in the same way as STRING annotations are returned in the dataclass __annotate__, but then you’ve lost this context so you can’t correctly evaluate them any more than you could under __future__ annotations. This is why I suggested the annotations should be paired with their context.

Also in general I don’t want creating __annotate__ functions to be as messy as it is in dataclasses. I want it to be as simple as it used to be to manipulate __annotations__. With a format that pairs the unevaluated object with the context necessary to evaluate it you can do this.

This is how it works with reannotate:

from annotationlib import get_annotations, Format
from reannotate import get_deferred_annotations, ReAnnotate

Vector = list[float]

class A:
    a: Vector

class B(A):
    b: undefined

def init_func(a, b): ...

# To generate a new __annotate__, simply merge the relevant annotations
# and create a ReAnnotate instance.
a_annos = get_deferred_annotations(A)
b_annos = get_deferred_annotations(B)
annos = a_annos | b_annos | {'return': None}
init_func.__annotate__ = ReAnnotate(annos)

undefined = int

print(get_annotations(init_func))
print(get_annotations(init_func, format=Format.STRING))

{'a': list[float], 'b': <class 'int'>, 'return': None}
{'a': 'Vector', 'b': 'undefined', 'return': 'None'}

For dataclasses this would mean creating the annotate function would be something like this:

def make_annotate_function(cls):
    annos = {f.name: f._raw_type for f in fields(cls) if f.init}
    annos['return'] = None
    return ReAnnotate(annos)
1 Like