PEP 849: More Expressive Type Expressions

PEPs
1

I’m excited to present PEP 849, which adds some new background capabilities to the way type annotations are processed in order to let us use arbitrary expressions while maintaining runtime introspection.

Currently, you technically can use arbitrary Python expressions in an annotation, so e.g. even var: f(x) if Something else 1 + 17 wouldn’t raise a compiler error. But if we inspect the annotations dict at runtime, we would only ever see the result of calling f or 18, depending on what Something evaluates to. This means that the typing spec can’t really assign any meaning to an expression like that because it wants to make sure that runtime tools will always see some representation of the full annotation.

This PEP introduces a new format for annotations, Format.AST. When an annotate function is called with it, it will not return the result of evaluating the annotation in some way, but the annotation’s AST and namespace. E.g. for the above example, we’d get something like this:

AnnotationAST(
    ast=ast.IfExp(
        test=ast.Name("Something"),
        body=ast.Call(ast.Name("f"), [ast.Name("x")]),
        orelse=ast.BinOp(ast.Constant(1), ast.Add(), ast.Constant(17)),
    ),
    namespace={"f": ..., "Something": ..., "x", ...},
)

In the most common use case, runtime introspection tools will not work with this AST directly, but use it to construct the correct typing objects. To make that easier, we introduce helper functions similar to annotationlib.get_annotations, which perform those computations under the hood.

There are still some implementation details to iron out, most notably how exactly the annotate functions are stored and how they construct the AST objects. But even in the current implementation, import times and memory usage are reduced compared to the current behavior, but executing the annotate functions takes longer in some common cases. Note that this slow down is comparing just the time it takes to construct the relevant typing objects. In most use cases that is just a small step of whatever introspection machinery is using annotation data. So the actual performance impact will be positive for many users and only a marginal decrease in some situations.

The full PEP is published here, the reference implementation is being worked on in my CPython fork and previous discussion around this idea took place in this thread and this thread.

20 Likes
Draft PEP: More Expressive Type Expressions
Should non-generic classes with `__class_getitem__` be subscriptable in a type expression?
Idea: Simpler and More Expressive Type Annotations
2

I rarely speak up on typing proposals, but I do really like the premise of this.

I’ve said before that I wished we’d made annotations stored as ASTs rather than values at any of the times where we redefined how they work (a.k.a. broke compatibility assumptions), as I think that leads to far more interesting uses of them than import-time evaluation.

All the best with this PEP!

6 Likes
3

I’ve been cautiously following the discourse around this PEP. I know I’m way out of my depth here but one of the comments from the previous thread picked my interest. It was something about the possibility of implementing an extension module capable of interacting with annotations for faster analysis (I think).

I’ve also seen the changes on the CPython side that touch a bit of the C-code. Would it be possible to also add a C-API that would provide introspecting annotations from an extension? Unless this is already possible today of course.

EDIT: for the record, any kind of work towards more expressive typing is always welcome on my end, so thank you for your work!

4

Thanks, this is an interesting PEP.

IMO, it needs to focus more on clarifying (and probably strenghtening) ast’s stability guarantees, both across CPython versions and across Python implementations.

Currently, the AST is not very stable – it might change with each Python release; Python 3.8, for example, unified the ast.Num,Str,Bytes etc. nodes into ast.Constant. CPython compiler maintainers might want to do something similar again.
Other implementations of Python might have slightly different ASTs – for example, they could do constant folding early, combining 1 | 2 into 3. MicroPython doesn’t have an ast module at all, which currently is not much of an inconvenience.

If the AST does stabilize and become an integral part of the language, it would be useful for much more than type annotations.
Take the numexpr package, which takes an expression string and evaluates it (often more efficiently than Python can):

>>> import numpy as np
>>> import numexpr as ne
>>> a = np.arange(10)
>>> b = np.arange(0, 20, 2)
>>> c = ne.evaluate('2*a + 3*b')    # (*)
>>> c
array([ 0,  8, 16, 24, 32, 40, 48, 56, 64, 72])

With this PEP, and some straighforward (AFAICS) glue code, the (*) line can turn into a proper expression, with syntax highlighting & IDE support:

>>> c: 2*a + 3*b
>>> c = evaluate_annotation('c', __annotate__)  # (more magic is possible here)

Or alternative compilers (like Numba) might want a decorator-like syntax that makes a function’s AST available.

The reason we don’t have such things now is that AST is currently not an integral part of Python-the-language. It’s more of a window into the internal workings of the compiler, meant for debuggers, code generators, and the like.

The main change the PEP asks for isn’t a Typing change; it’s closer to PEP 638 than to PEP 649. That doesn’t make it a bad change, but it’s certainly ambitious.
Best of luck in the PEP process!

5 Likes
5

How is Format.AST supposed to handle forward references?

In the example in the implementation you give a concrete flat namespace with resolved names. The PEP talks about adding annotationlib.create_annotate_function but if the AST format annotations aren’t actually capturing the relevant scopes I don’t see how this is going to work? The way dataclasses creates its __annotate__ function doesn’t generalise (and also fails in cases where it could succeed).

I am also concerned about the performance impact on constructing dataclasses, but would need to see actual numbers.

6

Big +1 from me! This will give much more flexibility for type annotation syntax in the future.

PEP 827 – Type Manipulation | peps.python.org more or less needs this; (the current draft tries to manage just based on carefully calling __annotate__, but that has severe drawbacks; I will update it now to depend on PEP 849).

PEP 835 – Shorthand syntax for Annotated type metadata | peps.python.org had also wanted to add a Format.TYPE operator, though there was some opposition to it and they eventually decided they could live without it. The ASTs from this should also be able to give them what they wanted from Format.TYPE. (Though maybe that’s a double-edged sword and some of the complaints about that would apply to this? I’m not sure.)

I might have some nitpicky technical followups, but I’m very happy to see this PEP!

3 Likes
7

Thanks for the detailed feedback!

Conceptually, the AST format is largely internal. Most users will call get_type_annotations and never see the ASTs themselves. It still is useful for the annotate functions to not construct the typing objects themselves because that allows us to still backport typing features through typing_extensions and it also lets us combine typing semantics with the other annotation formats easily, i.e. users can call get_type_annotations(something, format=Format.FORWARDREF) and get forwardrefs that still maintain the typing semantics. Libraries might also want to implement the AST evaluation logic via e.g. a C extension to save time. Though I don’t think that will be super common since that part of the evaluation isn’t super time consuming.

I’m honestly not super familiar with the exact policies around stability guarantees of the AST and related functionality. Maybe it would be better to explicitly document the AST format as being unstable and to recommend to only use the helper functions? Or if that is too weak of a warning to outside users, even define the returned objects as just being some opaque data that get_type_annotations uses to create the proper typing objects? I think it’s a good idea to keep the mechanism open for library optimization in principle, but if that collides with stability guarantees then that would have to go.

Regarding this PEP being closer to the introduction of macros rather than a typing change: While the underlying mechanics can be used to implement macro-like features, that’s very much not what this is trying to support. I would see it as similar to it being possible to make a function like in your example via disassembling the bytecode of a function.

1 Like
8

It’s already documented as such.

9

Right, I worded my question badly. What I meant was whether the AST itself being documented as unstable is enough to make it acceptable that the protocol that annotate functions follow would then also be similarly unstable? Or should the PEP take further steps like making the data in the returned objects be completely opaque to the outside (i.e. stored in private variables, etc) to make sure that the AST’s instability doesn’t leak out into the annotation handling code.

1 Like
10

I don’t think that making the returned objects completely opaque is necessary; I think documentation about AST stability is probably enough.

3 Likes
11

I have updated the PEP to reflect the new reference implementation. In summary, it now doesn’t generate the usual function objects for each __annotate__ but rather instances of a new AnnotateObejct type. These are very simple structs that store the AST data and some other namespacing info.

This approach gives us pretty excellent performance numbers. Import times are the same as they are now (though the noise on my machine is larger than I would have expected, maybe there’s a better way of measuring this that reveals something else). RAM usage is slightly reduced and .pyc sizes are down about 6%. The only downside is that annotation evaluation times are increased significantly, about 10x slower than baseline. But those function calls are a pretty small portion of real-world use cases, a more realistic number is the time it takes dataclasses to do its magic. This is up by between 10% and 45%, depending on the annotations that are used. I suspect that most real world annotations are closer to the lower end of that spectrum. There also still is a lot of room for improvement, e.g. I was able to get this down to between 5% and 25% with some code that was too ugly and fragile.

I can also adress some of the points that have been raised here:

The annotate objects use that standard __call__ interface and can thus be called from C code using the usual functions. I don’t think there’s a real use case in implementing something more specialized because the external API are the Python objects that the call returns. Compiler generated functions could have some capabilities added to return C level structs, but any code using that then wouldn’t work with synthesized or user generated annotate functions.

The returned AnnotationAST contains the AST and an AnnotationNamespace which references all scopes relevant to the annotation and correctly resolves names in them. Names are looked up only when the ASTs are evaluated and will then use whatever definitions exist at that time. For example, this works:

def f(a: undefined): ...
annos = f.__annotate__(Format.AST)
undefined = int
assert annos["a"].evaluate(Format.VALUE) is int

Finally, there’s a small question that I’ve come across: what exactly should the Format.STRING contain? The current docs describe it like this:

Values are the text string of the annotation as it appears in the source code, up to modifications including, but not restricted to, whitespace normalizations and constant values optimizations.

The exact values of these strings may change in future versions of Python.

But for a function like def f(a: int, b: "str"): ... the returned dict is {"a": "int", "b": "str"}, which doesn’t include the enclosing quotes around str even though they appear in the source code. Small differences like this also happen in other situations, e.g. when using mangled names. As far as I know, this wasn’t exactly an intentional design decision and is just a consequence of how the format is currently implemented.

Since we would store the AST directly, we could change that behaviour so that the strings match the actual source code more often. But that would also mean that sometimes it gets harder to eval those strings and get the correct result. So now the question is wether the strings are supposed to be the characters that appear in the source code or strings that can easily be used as forward references.

2 Likes
12

Given the existance of value and ForwardRefs, I feel that the main use-case for getting a string annotation would be to print it out to the console, log or UI somewhere.

For debugging, it’s most helpful in my mind to have a string which is grep-able, and one which is easy to read - both of which are more likely to be the case for the exact AST reconstruction.

In the case of string annotations themselves (a: "str") actually meaning the value within the quotes (str) that feels like a legacy feature we need to continue supporting for now to cause a bunch of backwards compatibility issues upfront.

1 Like
13

While lower memory use is nice, up to 45% slower dataclass construction on based on their current construction time really doesn’t sound good. dataclasses are already slow to construct and libraries that use them heavily are already noticeably slower to import.

2 Likes
14

The dataclass decorator only needs to check for ClassVar, InitVar, and KW_ONLY, right? Maybe it could extract that information directly from the AST?

15

Yes but it needs to check the value for ClassVar, InitVar and KW_ONLY not the name, so it still needs to evaluate them to some extent. I’m not sure traversing the AST for these to avoid evaluating the entire expression will be an improvement over evaluating the whole thing.