(For visibility, posted both to python-dev and Discourse.)
Over the last couple of months, ever since delaying PEP 563’s default change in 3.10, the Steering Council has been discussing and deliberating over PEP 563 (Postponed Evaluation of Annotations), PEP 649 (Deferred Evaluation Of Annotations Using Descriptors), and type annotations in general. We haven’t made a decision yet, but we want to give everyone an update on where we’re at, what we’re thinking and why, see if there’s consensus on any of this, and perhaps restart the discussion around the options.
First off, as Barry already mentioned in a different thread, the SC does not want to see type annotations as separate from the Python language. We don’t think it would be good to have the syntax or the semantics diverge, primarily because we don’t think users would see them as separate. Any divergence would be hard to explain in documentation, hard to reason about when reading code, and hard to delineate, to describe what is allowed where. There’s a lot of nuance in this position (it doesn’t necessarily mean that all valid syntax for typing uses has to have sensible semantics for non-typing uses), but we’ll be working on something to clarify that more, later.
We also believe that the runtime uses of type annotations, which PEP 563 didn’t really anticipate, are valid uses that Python should support. If function annotations and type annotations had evolved differently, such as being strings from the start, PEP 563 might have been sufficient. It’s clear runtime uses of type annotations serve a real, sensible purpose, and Python benefits from supporting them.
By and large, the SC views PEP 649 as a better way forward. If PEP 563 had never come along, it would be a fairly easy decision to accept PEP 649. We are still inclined to accept PEP 649. That would leave the consideration about what to do with PEP 563 and existing
from __future__ import annotations directives. As far as we can tell, there are two main reasons for code to want to use PEP 563: being able to conveniently refer to names that aren’t available until later in the code (i.e. forward references), and reducing the overhead of type annotations. If PEP 649 satisfies all of the objectives of PEP 563, is there a reason to keep supporting PEP 563’s stringified annotations? Are there any practical, real uses of stringified annotations that would not be served by PEP 649’s deferred annotations?
If we no longer need to support PEP 563, can we simply make
from __future__ import annotations enable PEP 649? We still may want a new future import for PEP 649 as a transitory measure, but we could make PEP 563’s future import mean the same thing, without actually stringifying annotations. (The reason to do this would be to reduce the combinatorial growth of the support matrix, and to simplify the implementation of the parser.) This would affect code that expects annotations to always be strings, but such code would have to be poking directly at function objects (the
__annotations__ attribute), instead of using the advertised ways of getting at annotations (like
typing.get_type_hints()). This question in particular is one in which the SC isn’t yet of one mind.
Keeping the future import and stringified annotations around is certainly an option, but we’re worried about the cost of the implementation, the support cost, and the confusion for users (specifically, it is a future import that will never become the future). If we do keep them, how long would we keep them around? Should we warn about their use? If we warn about the future import, is the noise and confusion this generates going to be worth it? If we don’t warn about them, how will we ever be able to turn them off?
One thing we’re thinking of specifically for the future import, and for other deprecations in Python, is to revisit the deprecation and warning policy. We think it’s pretty clear that the policy we have right now doesn’t exactly work. We used to have noisy DeprecationWarnings, which were confusing to end users when they were not in direct control of the code. We now have silent-by-default DeprecationWarnings, where the expectation is that test frameworks surface these warnings. This avoids the problem of end users being confused, but leaves the problem of the code’s dependencies triggering the warning, and thus still warns users (developers) not necessarily in a position to fix the problem, which in turn leads to them silencing the warning and moving on. We need a better way to reach the users in a position to update the code.
One idea is to rely on linters and IDEs to provide this signal, possibly with a clear upgrade path for the code (e.g. a 2to3-like fixer for a specific deprecation). Support for deprecations happened to be brought up on the typing-sig mailing list not too long ago, as an addition to the pytype type checker and hopefully others (full disclosure, Yilei is a team-mate of Thomas’s at Google).
This sounds like a reasonably user-friendly approach, but it would require buy-in from linter/IDE developers, or an officially supported “Python linter” project that we control. There’s also the question of support timelines: most tooling supports a wider range of Python versions than just the two years that we use in our deprecation policy. Perhaps we need to revisit the policy, and consider deprecation timelines based on how many Python versions library developers usually want to support.
The SC continues to discuss the following open questions, and we welcome your input on them:
- Is it indeed safe to assume PEP 649 satisfies all reasonable uses of PEP 563? Are there cases of type annotations for static checking or runtime use that PEP 563 enables, which would break with PEP 649?
- Is it safe to assume very little code would be poking directly at
__annotations__attributes of function objects; effectively, to declare them implementation details and let them not be strings even in code that currently has the annotations future import?
- Is the performance of PEP 649 and PEP 563 similar enough that we can outright discount it as a concern? Does anyone actually care about the overhead of type annotations anymore? Are there other options to alleviate this potential issue (like a process-wide switch to turn off annotations)?
- If we do not need to keep PEP 563 support, which would be a lot easier on code maintenance and our support matrix, do we need to warn about the semantics change? Can we silently accept (and ignore) the future import once PEP 649 is in effect?
- If we do need a warning, how loud, and how long should it be around? At the end of the deprecation period, should the future import be an error, or simply be ignored?
- Are there other options we haven’t thought of for dealing with deprecations like this one?
Like I said, the SC isn’t done deliberating on any of this. The only decisions we’ve made so far is that we don’t see the typing language as separate from Python (and thus won’t be blanket delegating typing PEPs to a separate authority), and we don’t see type annotations as purely for static analysis use.
For the whole SC,