Hello! This is my first time on these forums, though I’ve lurked for a long time.
I understand that PEP 585 lets us type hint the standard collection objects, like list[str], starting in 3.9 by default and optionally in 3.7 and 3.8 with __future__. This is great! I’d been looking forward to this feature for a while.
I’m kind of confused about how it works in practice, though. This code:
from __future__ import annotations
x: list[str] = ["a", "b"]
works today, as expected, in my various Python envs (one in 3.7.4 and one in some version of 3.8). But it didn’t prior to October 5th, when 3.9 was released.
How did this happen? How did the interpreter know that a future version of Python had been released, or that this code should somehow start working? What is it within __future__ that made this start working on October 5th?
I looked at the following places online:
The CPython source code: https://github.com/python/cpython/blob/3.7/Lib/__future__.py (sorry, Discourse only lets me put two links in a post?)
but wasn’t able to find a satisfactory answer. I apologize if this is a stupid question, and hope that I’ve expressed myself clearly, but please let me know if I haven’t.
__future__ imports are special and must appear early because they modify the behavior of the Python bytecode compiler for the current module scope. The magic happens in Python/future.c and Python/compile.c. The Python module __future__.py is more or less for show. The future flag FUTURE_ANNOTATIONS sets the compiler flag CO_FUTURE_ANNOTATIONS. This instructs the compiler to call compiler_visit_annexpr instead of compiler_visit_expr for annotations. compiler_visit_annexpr turns the annotation expression into a string. The flag turns list[str] into the string "list[str]".
Thank you very much for the response. That clarifies things a lot!
I believe that’s the same future flag that allowed us to type hint class definitions with their own name (PEP 563), right, by turning the annotations into strings as you describe and evaluating them later?
So I understand that __future__ import had been changing how evaluations were processed, but it seems like Python 3.9’s release caused it to do even more. What was it that changed when 3.9 was released, then? How did list[str] go from being a TypeError to working as you helpfully described, seemingly overnight?
If I have understood you, you are suggesting that installing Python 3.9
has somehow backported a change in behaviour to your existing 3.7
installation, without you changing or updating the 3.7 installation in
any way. And you want to know how this is possible.
Is that what you are asking?
If it is, then I would say, no, it is not possible. In principle if
parts of the standard library were over-written, then the libraries
could change behaviour. But not a change in interpreter behaviour.
Are you sure you are calling the correct version? If you are using a
file system link, alias or shortcut, perhaps the link changed and is now
pointing to 3.9 instead of 3.7.
In 3.7, the behaviour I expect is:
list[int] will be a runtime TypeError outside of annotations.
With a call to from __future__ import annotations, it will be
accepted inside annotations, but remain a TypeError as an exception
outside of annotations.
Thank you for the response Steven. That cleared up my confusion—implicitly in what I was asking was a conflation of annotations vs. non-annotations.
What makes x: list[str] work (not raise a TypeError) in Python 3.7 with the __future__ import isn’t related to PEP 585, or Python 3.9. It’s because (as Christian described) the __future__ flag, which has been available since PEP 563 was implemented in 3.7, stringifies the annotation so that it does not raise a TypeError.
That makes sense; it just never occurred to me that x: list[str] would have been accepted by the Python interpreter with the __future__ import, because type checkers didn’t accept it and we were told to use typing.List. I had never thought that it would work before now, but it makes perfect sense that it does.
Today I observed that saying x = list[str] (a type alias) doesn’t work on 3.7, even with the __future__ import, but works on 3.9. After reading your response and the implementation section of PEP 585, it also makes sense that these are different; x: list[str] is an annotation, but x = list[str] is not.
Thank you both for your help and your work on Python. I appreciate it.