Should we decide about deprecation now?
How about providing an option to enforce PEP 649 for all modules (even if from __future__ import annotations) and wait 2 or 3 years for feedbacks?
3 Likes
The cases folks are concerned about are the ones where PEP 649 doesn’t quite give the same behaviour as from __future__ import annotations does. My own preference would be:
from __future__ import annotationsstarts emitting PendingDeprecationWarning as soon as PEP 649 is implemented (i.e. 3.14). The mandatory release field in__future__.annotationsis also set toNoneat this point (see below).from __future__ import annotationsstarts emitting DeprecationWarning as soon as it stops having any effect and you get PEP 649 semantics even with this option specified
We would then never reach a point where from __future__ import annotations outright breaks - it would just emit DeprecationWarning indefinitely.
For the handling of __future__.annotations.getMandatoryRelease(), setting that to None for features that end up being dropped instead of becoming the default behaviour is explicitly documented: __future__ — Future statement definitions — Python 3.12.3 documentation
1 Like
That means, for example, that pytest displays PendingDeprecationWarning for hundreds of modules. The application programmer cannot easily resolve that warning until all dependent libraries stop from __future__ import annotations.
I don’t know how difficult to stop using from __future__ import annotations. That is why I want to wait for two or three years.
If there is no hard problem for transition, we can start emitting DeprecationWarning after Python 3.13 become EOL. I prefer it over displaying noisy PendingDeprecationWarning for all developers as soon as Python 3.14 is released.
Additionally, an option to enforce PEP 649 for all modules helps many developers a lot. They can easily run test or application with the option and see what is broken. Easy to try is important to collect information about how backward compatibility breackage affects to real world.
This is why I have added PEP 597 optional EncodingWarning. I enabled it at my .bashrc and see how many warning is shown. That option helped me to write PEP 686 – Make UTF-8 mode default a lot.
I think it’s important that we have a plan, so the future import doesn’t stay in limbo. However, PEP 749’s current plan leaves at least five years until we actually start emitting DeprecationWarning, so there is plenty of time to propose an alternative.
I like this idea. I think it would be useful for __future__ imports generally to be able to forcibly turn them on or off globally. For example, for from __future__ import generator_stop (the most recent future import), an option to turn the future import on globally would have made it easier to test for issues caused by the change.
Maybe we could do:
python -X future=foo # turn on future 'foo' everywhere
python -X nofuture=foo # turn off future 'foo' everywhere
The documentation for this feature should make it clear that it’s primarily for testing, since libraries you import may rely on the future statement.
3 Likes
I guess the unique nature of __future__ statements does make it genuinely hard to provide any advance programmatic notice (as PEP 749 argues), so consider the PendingDeprecationWarning suggestion withdrawn 
1 Like
Makes me wish (again) that PEP 690 had been accepted 
In any case annotationlib seems like a reasonable approach, given that I agree annotations would be too confusing.
8 Likes
Why member of Format enum is named FORWARDREF and not FORWARD_REF? It is not compatible with class name (ForwardRef) which distincts Forward and Ref as 2 separate words
1 Like
I ran into a few interrelated bugs with the current implementation:
>>> class X(type):
... a: int
...
>>> class Y(metaclass=X):
... b: str
...
>>> X.__annotations__
{'a': <class 'int'>}
>>> Y.__annotations__
{'a': <class 'int'>}
Accessing the annotations on the metaclass caches them in the metaclass’s __dict__, and subsequently, lookups on classes that are instances of the metaclass return the metaclass’s annotations.
And if the class itself doesn’t have any annotations:
>>> class X(type):
... a: int
...
>>> class Y(metaclass=X): pass
...
>>> Y.__annotate__
<bound method X.__annotate__ of <class '__main__.Y'>>
Instead, Y.__annotate__ should be None.
I’ve stared at this problem for a while now and I’m not sure what to do about it yet; I’d appreciate any opinions.
I’ll note that some related behavior is also buggy on existing versions (this is 3.12.2):
>>> class X(type):
... a: int
...
>>> class Y(metaclass=X): pass
...
>>> Y.__annotations__
{'a': <class 'int'>}
Metaclasses having annotations do seem to lead to surprising results (3.12.3):
class Meta(type):
a: int
class X(metaclass=Meta):
b: float
class Y(X):
pass
print(Y.__annotations__) # {'b': <class 'float'>}
It appears that defining __annotations__ on a metaclass puts you back in something like Python 3.9 annotations behaviour? I would have expected an empty dict in both cases for 3.10 or later.
I don’t have a build of the current __annotate__ implementation. Is it set to None in the case where there’s no metaclass and no annotations?
Looking at the current __get__ implementations for the __annotate__ and __annotations__ descriptors, they’re not paying attention to their final type/cls argument (which is how a descriptor can tell the difference between “accessed on an instance” and “accessed on the class”).
The descriptor methods in typeobject.c are a bit weird in general, due to the whole “type is its own metaclass” situation, but I’m pretty sure paying attention to that final argument will provide enough info to be able to get Y.__annotations__ to report Y’s annotations properly.
The implementation of classmethod.__get__ may provide some ideas for restructuring the logic to detect and handle cases like this one.
1 Like
I don’t think anything we do in the descriptor alone can fix this, because the problem is that __annotations__ is stored in the metaclass’s __dict__, and as a result we never even get to type.__annotations__.
That makes me think that the way to solve these bugs is to never store anything in the __annotate__ and __annotations__ keys of a type’s __dict__. We’ll have to store the annotate function and the cached annotations dictionary somewhere else, and rely exclusively on the descriptors. A metaclass that intentionally overrides the descriptors could still create a different behavior if it wants.
Huh, I segfaulted Python 3.12 trying to demonstrate that this should be possible by showing that type.__base__ would ignore an entry injected into the type object’s instance dictionary:
>>> class BypassProxy():
... def __ror__(self, other):
... other["__base__"] = 42
... return other
...
>>> type.__dict__ | BypassProxy()
... [snip output] ...
>>> type.__dict__["__base__"]
42
>>> type.__base__
Segmentation fault (core dumped)
It genuinely should be possible though, due to that whole “type is its own metaclass” aspect, which means data descriptors on type and other type instances should still be invoked even if there’s an entry in the instance dictionary under the same name (if it wasn’t possible, my attempted demonstration with __base__ would have just returned 42 instead of segfaulting).
(While its been years since I really had to work on it, typeobject.c has always made my brain hurt, since the assumptions that apply when working on any other class, even ones with a custom metaclass, often aren’t quite true due to the special role that type plays in the runtime type system)
Ah, wait, I just realised I had misunderstood the problem. It isn’t the lookup on X that is getting confused, it’s the lookup on the next step in the metaclass chain, Y. Ugh.
1 Like
To be clear, I am talking about this example:
>>> class X(type):
... a: int
...
>>> class Y(metaclass=X):
... b: str
...
>>> X.__annotations__ # as a side effect, sets X.__dict__["__annotations__"]
{'a': <class 'int'>}
>>> Y.__annotations__
{'a': <class 'int'>}
When Y.__annotations__ is executed, the name is looked up in the dict of type(Y), but type(Y) is X, not type.
Perhaps there’s a possible resolution in using a different name like __type_annotations__ as the key for caching the annotations in the instance dictionary, and let the data descriptor for __annotations__ handle looking that up?
That way _PyObject_LookupSpecial won’t get confused the way it is now, since cls.__dict__["__annotations__"] won’t get in the way of locating the descriptor inherited from type.
Yes, that’s a solution I was thinking of. We’ll have to come up with a name for __annotate__ too, and I’m not sure adding “type” is appropriate, since annotations at this level are not reserved only for types.
I’ll have to add this to PEP 749; I’ll probably work on it tomorrow.
1 Like
Making sure I understand the __annotate__ variant of the problem correctly: since the class compilation adds the compiled annotation building descriptor to the class dictionary under __annotate__, accessing Y.__annotate__ sees that entry in type(Y).__dict__ rather than the type.__annotate__ descriptor, and hence the X.__annotate__ descriptor ends up being invoked directly rather than Y.__dict__["__annotate__"] being processed by type.__annotate__.
If I’ve understood correctly, then a name like __exec_annotations__ could make sense as a way of keeping the compile-time generated code in metaclasses from shadowing the type.__annotate__ descriptor definition.
Similarly, __annotations_cache__ would be a more semantically neutral name for the dict entry that avoids shadowing type.__annotations__.
When writing the related test cases, it’s also worth considering that metaclass chains can technically be arbitrarily deep (Python 3.12 interactive session):
>>> class X(type):
... a: int
...
>>> class Y(type, metaclass=X):
... b: int
...
>>> class Z(metaclass=Y):
... c: int
...
>>> X.__annotations__
{'a': <class 'int'>}
>>> Y.__annotations__
{'b': <class 'int'>}
>>> Z.__annotations__
{'c': <class 'int'>}
Since the existing direct storage works without issue, I think simple separation of the dict entry names from the descriptor names will suffice to deal even with deeper chains, but it would still be good to have the test suite ensure that is the case.
1 Like
When exploring the impact of order on a build from Main made this morning I found that the example @ncoghlan gave actually ends up revealing the underlying descriptor in some evaluation orders.
Script:
import sys
from itertools import permutations
print(sys.version)
def make_classes():
class Meta(type):
a: int
class X(type, metaclass=Meta):
b: float
class Y(metaclass=X):
c: str
class Z(Y):
pass
return Meta, X, Y, Z
def demo_annotations():
classes = make_classes()
class_count = len(classes)
for order in permutations(range(class_count), class_count):
print()
classes = make_classes() # Regenerate classes
for i in order:
classes[i].__annotations__
names = ", ".join(classes[i].__name__ for i in order)
print(f"__annotations__ access order: {names}")
for c in classes:
print(f"{c.__name__}.__annotations__ = {c.__annotations__}")
demo_annotations()
From a build of 3.14a0 from Main this morning:
3.14.0a0 (heads/main:73dc1c678e, Jun 18 2024, 11:57:21) [GCC 11.4.0]
__annotations__ access order: Meta, X, Y, Z
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Meta, X, Z, Y
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Meta, Y, X, Z
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Meta, Y, Z, X
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Meta, Z, X, Y
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Meta, Z, Y, X
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: X, Meta, Y, Z
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {'b': <class 'float'>}
__annotations__ access order: X, Meta, Z, Y
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {'b': <class 'float'>}
__annotations__ access order: X, Y, Meta, Z
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {'b': <class 'float'>}
__annotations__ access order: X, Y, Z, Meta
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {'b': <class 'float'>}
__annotations__ access order: X, Z, Meta, Y
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {'b': <class 'float'>}
__annotations__ access order: X, Z, Y, Meta
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {'b': <class 'float'>}
__annotations__ access order: Y, Meta, X, Z
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Y, Meta, Z, X
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Y, X, Meta, Z
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {'c': <class 'str'>}
__annotations__ access order: Y, X, Z, Meta
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {'c': <class 'str'>}
__annotations__ access order: Y, Z, Meta, X
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Y, Z, X, Meta
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Z, Meta, X, Y
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Z, Meta, Y, X
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Z, X, Meta, Y
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {}
__annotations__ access order: Z, X, Y, Meta
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'b': <class 'float'>}
Z.__annotations__ = {}
__annotations__ access order: Z, Y, Meta, X
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = <attribute '__annotations__' of 'type' objects>
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
__annotations__ access order: Z, Y, X, Meta
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
In Python 3.13b1 or earlier for every order:
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {'c': <class 'str'>}
The output I would have expected:
Meta.__annotations__ = {'a': <class 'int'>}
X.__annotations__ = {'b': <class 'float'>}
Y.__annotations__ = {'c': <class 'str'>}
Z.__annotations__ = {}
Is this correct? If so should this also be raised as a bug against Python 3.13?
1 Like
I spent some more time thinking about this and concluded that we should go with a solution that @AlexWaygood suggested to me. This solution sets __dict__["__annotations__"] to a special descriptor that behaves like a lazily evaluated mapping.
I wrote up the motivation and solution in a PR to PEP 749:
I’ll work next on implementing this behavior in CPython.
Thanks, that code sample is very helpful. I will put a variation of it in the test suite.
I agree this is a bug in Python 3.13 and earlier. I expanded on this and another buggy behavior in the proposed change to the PEP.
5 Likes
For a pure-Python proof of concept of this idea, I’ve posted a gist at Demo for an `__annotations__` solution · GitHub.
3 Likes