Type annotations, PEP 649 and PEP 563

Core Development
1

(For visibility, posted both to python-dev and Discourse.)

Over the last couple of months, ever since delaying PEP 563’s default change in 3.10, the Steering Council has been discussing and deliberating over PEP 563 (Postponed Evaluation of Annotations), PEP 649 (Deferred Evaluation Of Annotations Using Descriptors), and type annotations in general. We haven’t made a decision yet, but we want to give everyone an update on where we’re at, what we’re thinking and why, see if there’s consensus on any of this, and perhaps restart the discussion around the options.

First off, as Barry already mentioned in a different thread, the SC does not want to see type annotations as separate from the Python language. We don’t think it would be good to have the syntax or the semantics diverge, primarily because we don’t think users would see them as separate. Any divergence would be hard to explain in documentation, hard to reason about when reading code, and hard to delineate, to describe what is allowed where. There’s a lot of nuance in this position (it doesn’t necessarily mean that all valid syntax for typing uses has to have sensible semantics for non-typing uses), but we’ll be working on something to clarify that more, later.

We also believe that the runtime uses of type annotations, which PEP 563 didn’t really anticipate, are valid uses that Python should support. If function annotations and type annotations had evolved differently, such as being strings from the start, PEP 563 might have been sufficient. It’s clear runtime uses of type annotations serve a real, sensible purpose, and Python benefits from supporting them.

By and large, the SC views PEP 649 as a better way forward. If PEP 563 had never come along, it would be a fairly easy decision to accept PEP 649. We are still inclined to accept PEP 649. That would leave the consideration about what to do with PEP 563 and existing from __future__ import annotations directives. As far as we can tell, there are two main reasons for code to want to use PEP 563: being able to conveniently refer to names that aren’t available until later in the code (i.e. forward references), and reducing the overhead of type annotations. If PEP 649 satisfies all of the objectives of PEP 563, is there a reason to keep supporting PEP 563’s stringified annotations? Are there any practical, real uses of stringified annotations that would not be served by PEP 649’s deferred annotations?

If we no longer need to support PEP 563, can we simply make from __future__ import annotations enable PEP 649? We still may want a new future import for PEP 649 as a transitory measure, but we could make PEP 563’s future import mean the same thing, without actually stringifying annotations. (The reason to do this would be to reduce the combinatorial growth of the support matrix, and to simplify the implementation of the parser.) This would affect code that expects annotations to always be strings, but such code would have to be poking directly at function objects (the __annotations__ attribute), instead of using the advertised ways of getting at annotations (like typing.get_type_hints()). This question in particular is one in which the SC isn’t yet of one mind.

Keeping the future import and stringified annotations around is certainly an option, but we’re worried about the cost of the implementation, the support cost, and the confusion for users (specifically, it is a future import that will never become the future). If we do keep them, how long would we keep them around? Should we warn about their use? If we warn about the future import, is the noise and confusion this generates going to be worth it? If we don’t warn about them, how will we ever be able to turn them off?

One thing we’re thinking of specifically for the future import, and for other deprecations in Python, is to revisit the deprecation and warning policy. We think it’s pretty clear that the policy we have right now doesn’t exactly work. We used to have noisy DeprecationWarnings, which were confusing to end users when they were not in direct control of the code. We now have silent-by-default DeprecationWarnings, where the expectation is that test frameworks surface these warnings. This avoids the problem of end users being confused, but leaves the problem of the code’s dependencies triggering the warning, and thus still warns users (developers) not necessarily in a position to fix the problem, which in turn leads to them silencing the warning and moving on. We need a better way to reach the users in a position to update the code.

One idea is to rely on linters and IDEs to provide this signal, possibly with a clear upgrade path for the code (e.g. a 2to3-like fixer for a specific deprecation). Support for deprecations happened to be brought up on the typing-sig mailing list not too long ago, as an addition to the pytype type checker and hopefully others (full disclosure, Yilei is a team-mate of Thomas’s at Google).

This sounds like a reasonably user-friendly approach, but it would require buy-in from linter/IDE developers, or an officially supported “Python linter” project that we control. There’s also the question of support timelines: most tooling supports a wider range of Python versions than just the two years that we use in our deprecation policy. Perhaps we need to revisit the policy, and consider deprecation timelines based on how many Python versions library developers usually want to support.

The SC continues to discuss the following open questions, and we welcome your input on them:

  1. Is it indeed safe to assume PEP 649 satisfies all reasonable uses of PEP 563? Are there cases of type annotations for static checking or runtime use that PEP 563 enables, which would break with PEP 649?
  2. Is it safe to assume very little code would be poking directly at __annotations__ attributes of function objects; effectively, to declare them implementation details and let them not be strings even in code that currently has the annotations future import?
  3. Is the performance of PEP 649 and PEP 563 similar enough that we can outright discount it as a concern? Does anyone actually care about the overhead of type annotations anymore? Are there other options to alleviate this potential issue (like a process-wide switch to turn off annotations)?
  4. If we do not need to keep PEP 563 support, which would be a lot easier on code maintenance and our support matrix, do we need to warn about the semantics change? Can we silently accept (and ignore) the future import once PEP 649 is in effect?
  5. If we do need a warning, how loud, and how long should it be around? At the end of the deprecation period, should the future import be an error, or simply be ignored?
  6. Are there other options we haven’t thought of for dealing with deprecations like this one?

Like I said, the SC isn’t done deliberating on any of this. The only decisions we’ve made so far is that we don’t see the typing language as separate from Python (and thus won’t be blanket delegating typing PEPs to a separate authority), and we don’t see type annotations as purely for static analysis use.

For the whole SC,

Thomas.

6 Likes
2

I won’t help much on the typing/annotations side, but I’d like to make one thing explicit:

Features gated by a from __future__ import are stable. They’re covered by the backwards compatibility policy: and their behavior isn’t expected to change before it becomes Python’s default behavior.
Here, the SC is deliberating a one-time exception to the general policy, just for from __future__ import annotations.

Is that right?

Oh boy, this should probably be in another thread…

IMO, if an old API isn’t very dangerous nor hard to maintain, it would be great to only add PendingDeprecationWarning (and document its deprecation), and wait until the preferred replacement is available in all currently supported versions of Python. That way library maintainers can switch to it rather than go through try/except, sys.version, or some other shim. (That’s assuming library maintainers follow Python-dev’s EOL dates, but if they don’t, they can still use a wordier workaround.)
Only then add a DeprecationWarning, and remove the API after 2 more years.

IDEs, linters and test runners could then, hopefully, somehow, signal PendingDeprecationWarning in the currently developed project to the developer.

3

I think it would be more accurate to say that we probably have a need to change a stable API, and the question is how to best go about it. It may be a one-time exception for this specific case, but maybe the end result of the discussion is that we have to re-think how we handle these kinds of changes – or all potentially backward-incompatible changes.

(The other question is how much of PEP 563 is the API. There are lots of things that are clearly implementation details, and lots of things that clearly aren’t implementation details. The question here is about the grey area in-between.)

4

But the thing is, if PEP 649 becomes the future, PEP 563 never will.

5

I see it as any other feature. We decide that from __future__ import annotations is a feature to be deprecated, we can do that in Python 3.11, then we can remove it in 3.13. I’m sure we can make it report a DeprecationError (or a SilentDeprecationError if need be). If we feel that it’s used by many people we can extend the deprecation period. But just because it’s spelled using __future__ I don’t think doesn’t mean we need to treat it differently. After all if we were to introduce a new builtin function or new syntax, and then later we decided to get rid of it (as has happened on numerous occasions, though perhaps not so soon after the introduction) we could do that too.

1 Like
6

I just wanted to say thank you! :bowing_man:

On behalf of those of us doing these things with type annotations at runtime (FastAPI, Pydantic, Typer, SQLModel, etc), thank you to the SC and everyone involved for giving this so much importance, and for paying attention to these (quite new) Python sub-communities and use cases. :tada:

6 Likes
7

If I’m not being too tunnel-visioned, I think changing the behavior of from __future__ import annotations in this case is harmless. Accessing the annotations from the __annotations__ attribute is discouraged anyways. Turning on PEP 649 instead of 563 shouldn’t break too many codebases.

In my case, I only add the __future__ import to use Python 3.10 type annotation features in the earlier Python versions like 3.8 and 3.9. For example, currently, I can easily use this on Python 3.8 although it’s a Python 3.10 feature:

from __furure__ import annotations
from collections import Sequence # Py3.10: Collections mixins are generics.

a : Sequence[str] = ['a', 'b', 'c']
b : dict[str, int] = {'a': 1, 'b': 2}  
c : tuple[str, ...] | None = None

As far as I understand, this would still keep working if the __future__ import turns on PEP 649 instead of PEP 563. Right?

8

Oh, not, it isn’t harmless. Despite the recommendations there are people who depend on this behavior. We can deprecate it and eventually get rid of the feature altogether, but we should definitely not change its behavior at this stage. It needs to keep generating strings as long as it is supported.

6 Likes
9

Hi Thomas and SC,

Thanks for this clarity on where your thinking stands and the remaining open questions.

I think so too.

Other replies have mentioned the desire to use annotation syntax that relies on newer runtime features (e.g. __or__ on types to support X | Y unions) under older Pythons that don’t yet have those runtime features. Since this means using syntax in annotations that isn’t part of the language version you are using, this could be construed as running afoul of the SC’s stated intention to “keep typing syntax a part of the language,” which I agree with.

One use case I haven’t seen mentioned is self-documentation tooling that wants to access and display annotations as strings. With PEP 563 it is very easy to access the annotation as a string; with PEP 649 this requires re-parsing source code, or else reifying the annotation as a Python object and then hoping its repr does something reasonable. I’m not sure Python needs to cater to this use case; its a bit niche and probably OK to require extra work for it. In general its reasonable for introspection of the results of Python expressions at runtime to be Python objects, not stringified source. You can’t easily get stringified source of the code of functions at runtime, either, even though that might also be something you’d want for a self-documentation tool; you have to use the inspect module to go re-parse the source.

Regarding forward references, a few points:

  1. Both PEP 563 and PEP 649 support “forward references” just fine, in the sense of names in the module that aren’t yet defined when the annotation is encountered by the compiler but are defined later in the module, as long as the annotation isn’t accessed & reified at runtime until after the module namespace is fully populated.
  2. For names that will never be present in the module namespace at runtime (e.g. imports guarded by if TYPE_CHECKING to avoid import cycles or just avoid unnecessary additional runtime dependency fanout), the PEPs are also fairly similar I think (though I only realized this recently.) With both PEPs, such annotations are fine if you never reify them at runtime. With both PEPs, reifying such annotations will result in NameError by default. With both PEPs, it should be possible to augment the relevant namespaces with the needed names at runtime before reifying the annotation if you put in some extra work.

__annotations__ is also documented and not particularly discouraged in the documentation; I don’t think it can really be considered private. And as others have said, __future__ imports are just as subject to backwards-compatibility commitments as anything else; not that they can’t be removed, but they need deprecation like any other changed or removed feature. We certainly have code that will break if from __future__ import annotations silently became PEP 649 instead of PEP 563. It seems wiser to provide a new __future__ import for the new behavior and a deprecation path for the “old new” ( :wink: ) behavior.

For those not using PEP 563, there will be no noise or confusion. For those who are using it in ways that PEP 649 supports equivalently, the deprecation warning will provide a very easy quick fix (switch to the new __future__ import) that will Just Work. For those already relying on PEP 563 in ways that PEP 649 doesn’t transparently support, “noise and confusion” from this change is already inevitable, it’s just a question of whether it takes the form of immediate breakage or a deprecation warning and some time to adapt (and the ability to adapt modules incrementally rather than a flag-day change.)

I don’t think at this point that I have any such use cases to propose.

No, unfortunately I don’t think that this would be safe.

We definitely care about the overhead of type annotations. At one point pre-PEP-563 I think our codebase spent 1% of total CPU on executing annotations. That’s a lot, although I believe most of it was due to the old GenericMeta implementation that is also gone in Python 3.7 thanks to __class_getitem__.

I don’t think we know yet how the performance compares, or whether the difference is significant. I tried to do some performance comparisons a few months ago on a large well-annotated codebase, but I wasn’t able to get the reference implementation of PEP 649 to work at all so didn’t make those comparisons. I think that it would be valuable to have a working reference implementation and some performance data (and then perhaps some work on perf tuning: possibilities have been suggested in earlier threads about PEP 649) before making a final decision on PEP 649.

I think Inada Naoki’s comparison of annotations with docstrings elsewhere in this thread is apt, and an optimization flag or level that removes annotations is worth considering if the overhead is significant.

I don’t think we need to or should keep PEP 563 support around indefinitely, but I do think this is a significant change to semantics of documented language features that will break code in the wild, and the existing PEP 563 __future__ import needs a deprecation path.

I don’t have strong feelings about this. I think the deprecation could probably go faster than some, given the cost of keeping the feature, and the fact that it was a __future__ import that never became default. One or two releases would be fine for us. After the deprecation I think it would be better if the __future__ import became an error rather than silently doing nothing; that seems like a pretty small and low-maintenance stub to keep around for a while to avoid confusion.

Carl

10

I think this here needs special handling to support libraries. For normal features deprecate and then remove is fine as library authors can do if/else per python version. Future imports though cannot be version guarded, but instead must be at the top of the file. So the earliest we can remove it is when 3.10 is EOL (so a library author can do the transition by removing the future import safely altogether).

7 Likes
11

As a developer who doesn’t rely on runtime annotations, I do appreciate the opportunities that PEP 563 provides. E.g. it would otherwise be impossible for me to start using X | Y now while still supporting older Python versions. Same with builtin generic types added in Python 3.9

Most typing features are meant to help developers with type checking their code, so I do think it would be a missed opportunity to limit the use of new syntax to new version alone, even if it’s only in type annotations.

It can be complicated at times but in my experience, both pyright and mypy, I’m sure others too, do a good job detecting invalid uses. Once someone has understood the limitations, it’s fairly strait forward. There are also linters that can help with that, pylint with the typing extension just to name one.

1 Like
12

I agree with this as well. Types shouldn’t depend on the runtime implementation. I should be able to use features like X | Y and collections.Sequence[str] in the earlier Python versions. Initially, I think that was the goal of PEP 484—separating out type-hints from the runtime.

13

Since this got split into Discourse and python-dev, I wrote my take on the issue on the blog so that I could post it in both spaces with proper formatting and everything:

6 Likes
14

However, in the current situation, it’s not that clear because I find it very important to allow writing a single codebase that works on Python 3.7 - 3.11. If we can secure this behavior, I’m +1 to accepting PEP 649.

This perfectly summarizes my point as well. Thanks!

15

One issue PEP 649 doesn’t solve is with code that only wants to partially evaluate annotations - for instance dataclasses needs to know if members are annotated with ClassVar so it can ignore them, but doesn’t care exactly what type it is (other than preserving the value to copy to the field object). On the other hand PEP 563 causes issues locating the correct scope for evaluating local objects, if it’s possible at all.

Perhaps Łukasz’s suggested third scheme of string annotations + ensuring the required closure values are stashed would be the best approach. Doing something similar to PEP 649, storing that in a keys tuple and cell values tuple pair and lazily constructing a dict might help with memory concerns.

2 Likes
16

The idea of having a tighter tie-in to the parser to help resolve non-local names is interesting. How much of that would solve problems for the Pydantic-and-friends group? @tiangolo, any thoughts ?

2 Likes
17
  1. If we do need a warning, how loud, and how long should it be around? At the end of the deprecation period, should the future import be an error, or simply be ignored?

I want to reiterate the point I wrote on python-dev, because I believe it’s an important point that the discussion so far seems to skip over: from __future__ import annotations should not show any warnings or be removed before Python 3.10 reaches end of life (expected to be October 2026). As future imports can’t be version gated or shim’ed, removing it before that point would force code bases that want to support both older and newer Python versions to temporarily regress on their type annotations.

For example, def foo(x: int | str): ... requires either Python 3.10+ or the future import. If, for example, Python 3.13 removes the future import, this code can’t be supported on both Python 3.9 and 3.13, whose lifespans are expected to overlap by a year. Same problem, to a lesser degree, with visible warnings. Most libraries only remove supports for a Python version after its end of life.

3 Likes
18 
from __future__ import annotations

is not like any other import. It changes the language, and is supposed to be at the top of the file; if it is removed in Python 3.13 or even just starts spewing a warning, codebases will have to either do a conditional import after importing sys with sys.version_info, a try/except import, or remove the feature and transition back to true Python 3.7 compatible type hinting, which is dramatically worse. NumPy and pretty much the whole data science community and 130K instances on GitHub are using future annotations to make annotations much more usable and readable on Python 3.7. PyUpgrade automatically converts all annotations to mypy compatible 3.10 style if from __future__ import annotations is used, doing things like Optional[Dict[str, str]] → dict[str, str] | None.

Last I checked, this issue and PEP 649 was still because Pydantic users sometimes make locally defined types that are then hard to resolve as strings. While other libraries keep getting bundled in, are they actually affected? Typer, for example, doesn’t seem to encourage this model; is there a single issue about future animations in Typer that is still broken? I found exactly one issue about the future import, and it was immediately fixed and released in version 0.2.1 over a year ago. This will affect corner cases in Pydantic, but “all libraries that use runtime types” are not affected. Placing a type inside a local definition then using it inside an annotation is not going to affect all libraries using runtime types. PEP 649 seems like a very complex solution to a very specific problem. It’s much easier to reason about type hints as strings than it is about lazy objects.

The idea of having a tighter tie-in to the parser to help resolve non-local names is interesting.

This sounds really interesting, IMO. :slight_smile:

3 Likes
19

First, thanks @ambv for the great write-up, I learned things!

Thanks @brettcannon for the ping. :bowing_man:

The idea of having a tighter tie-in to the parser to help resolve non-local names is interesting. How much of that would solve problems for the Pydantic-and-friends group?

I agree this sounds very interesting. I don’t understand the internals enough to know if the proposed alternative would solve it as it’s not obvious to me (e.g. it’s probably implicit that it would be solved with the last example, but I wouldn’t know).

If the proposed alternative would support this:

from __future__ import annotations

def main():
    from pydantic import BaseModel, PositiveInt

    class TestModel(BaseModel):
        foo: PositiveInt

    # crashes with current PEP 563
    typing.get_type_hints(TestModel)

main()

…I think that’s the main use case I would care about (and extensions of it), so that would be perfect.

I would still like to see Samuel Colvin’s point of view, as he’s the one that has actually done all the heavy work with type annotations at runtime in Pydantic. I mainly inherited or copied that work in other libraries. He’ll hopefully get the chance to reply in the next few days.

Additional, less important comments below :point_down:

I also see that if using types imported inside functions was supported (e.g. as in the example above), the cases that depend on if TYPE_CHECKING: to avoid cyclic imports would now be somewhat usable even at runtime too. Those possibly cyclic things could now be imported inside of functions (instead of in if TYPE_CHECKING: guards). Which is one of those compound use cases that would not be supported by the current PEP 563, but would be solved by the alternative (I think).

For example:

  • module_foo.py
# module_foo.py
from __future__ import annotations

from pydantic import BaseModel
from module_bar import helper_func

# Importing and using this here creates the cyclic import
helper_func()


class Foo(BaseModel):
    answer_of_life: int
  • module_bar.py
# module_bar.py
from __future__ import annotations
from pydantic import BaseModel


def helper_func():
    print("Not helpful and cyclic but helps with the example")


def get_bar_model():
    # Can't import this at the top level, as helper_func is imported
    # in module_bar
    from module_foo import Foo

    class Bar(BaseModel):
        foo: Foo

    return Bar(foo={"answer_of_life": 42})
  • main.py
# main.py
from module_bar import get_bar_model

m = get_bar_model()
print(m)

…running python main.py will crash, with the current state of from __future__ import annotations.

Of course, this example is terrible code with bad practices all over the place, but it’s the shortest way I found to make something simple that shows the problem.

About using future features of Python (e.g. 3.10) in previous versions (e.g. 3.7), I would love to be able to do that, also at runtime. But I don’t see a way to support it (at runtime) without making get_type_hints() part of an external library that can be updated independently and used in previous versions of Python (e.g. like mypy). Something like that would probably be a much more controversial idea with probably a lot of drawbacks and opinions, etc. Probably not even worth mentioning or discussing it, but :person_shrugging:.

1 Like
20

I’m very much a bystander in this discussion, so please don’t take this as anything more than idle curiosity, but why is it important that the from pydantic line is inside the main() function, rather than at the top of the file, which is (in my experience) far more common and idiomatic? Is it a particular pattern that pydantic users are encouraged to follow? The code works fine if the import is at the top level.

(To be clear, I do find it annoying that the example as quoted doesn’t work, but to be honest, I’d come to the conclusion a long time ago that what I thought was reasonable didn’t match what people who were enthusiastic about typing were comfortable with. So I assume my instincts on what’s OK to accept aren’t very good here).