Please consider delaying, or even rejecting, PEP 695

PEPs
1

PEP 695 proposes a new syntax for type parameters.
The motivation is that the scoping rules for type parameters are hard to understand,
are prone to misuse because of that, and that it forces developers to understand variance.

These are reasonable points. Type parameters can be hard to use.
However, the cure is worse than the disease.

The new scoping rules in PEP 695 are unclear and far too complicated

PEP 695 doesn’t solve the core issue that scoping rules for type parameters are hard to understand.
In fact, it makes the situation worse.

Currently, scoping is easy to understand: use the LEGB rule Python Scope & the LEGB Rule: Resolving Names in Your Code – Real Python.

The new scoping rules as described in the PEP are very hard to understand.
I would not be unable to implement PEP 695 from the PEP alone.

So we need to look at the implementation.

The implementation is large, but looking at the symbol table it appears that four new scopes are introduced, “annotation”, “typevar bound”, “type alias” and “type parameter”.
Previously there were only three: “function”, “class” and “module”.
How does more than doubling the number of scopes make anything easier to understand?

The Real Python article on scopes can be understood by any Python user: engineer, scientist, architect or student. I don’t think it would be possible to write such an accessible article covering the new scope rules.

Additional complexity in the VM.

The interpreter

The implementation also adds 12 instructions to the VM. Currently there are 119 instructions (not including instructions for specialization and instrumentation). That means that PEP 695 requires increasing the base instruction set by 10%. That is a lot, and totally out of proportion for any benefit it brings.

For comparison, here’s a list of some other features in terms of instructions used:

Imports: 4
With statement: 2
Async and await: 4
Async with: 1
Async for: 1
Pattern matching: 4
Except star: 2
Walrus operator: 0

Either the PEP needs to be simplified w.r.t to scopes a lot, or the implementation is taking the wrong approach.
Whichever it is, this shouldn’t be added in 3.12.

The function object

Any use of inner function and list comprehensions (until PEP 709) creates a new function object.

The PEP 695 implementation also adds four fields to the function object struct.
The function object is important for performance and streamlining these objects is important for performance. We have put a lot of work into reducing the size of core objects and streamlining their construction and destruction. It is vexing to have this undone for a feature of such limited benefit.

Conclusion

Understanding scopes is key to understanding Python.
Making the scoping rules more complex limits who can understand, and therefore use, Python.

I believe that the simplicity of Python is fundamental to its popularity and will continue to be so.
Adding features that make it less accessible and harder to learn is going to hurt all of us, even the professional developers who might benefit from more powerful features like PEP 695.

Any change to the language should be in proportion to the benefit it brings.
Type variables aren’t that commonly used. If we really feel that changes are needed, then we need to work hard to make sure that those changes are in proportion to the benefit they provide.

25 Likes
2

I like the syntax proposed by PEP 695, and I hope we end up with something like it eventually. But, I also support delaying it until at least 3.13.

I know there’s been a lot of incredible work done recently to try to get the implementation ready in time for 3.12, and I really appreciate that work. However, the PEP is a huge change that deserves to be well tested. Testing the PEP with real code – rather than in isolated unit tests – could possibly reveal flaws in the design that would require rewrites of the initial implementation. I don’t think we should go through that process during the 3.12 beta phase.

Moreover, while I support us reaching this kind of syntax eventually, I also care a lot about the stability of Python’s syntax and Python’s typing system. It was recently indicated to some core developers that the Steering Council was planning not to make a decision on PEP 696 for now, because they preferred to see how PEP 695 played out first. I strongly believe that this is the wrong order in which to do things. PEP 696 does not require any syntax changes to the language if it is implemented before PEP 695. But if PEP 695 is implemented in Python 3.12, and PEP 696 is implemented in Python 3.13, we will need TypeVar-related syntax changes in two consecutive versions of Python. I think this will be really harmful to the stability of the language’s syntax and Python’s typing system.

I have previously expressed concerns that Python’s typing system was not stable enough to consider “freezing” the status quo in new syntax; that implementing PEP 695 now would make it harder to propose new typing features relating to TypeVars; and that the syntax changes proposed by PEP 695 would not be extensible without introducing further syntax changes:

I feel like these concerns have been brushed aside a little bit, and that the current situation (where PEP 696 is being deferred at least in part because of PEP 695) is exactly the situation I was worried about. There has also been a recent proposal for a TypeVarDict: it is unclear how that would interact with PEP 695, and whether accepting that idea would require yet more syntax changes if PEP 695 is implemented.

There are really three distinct ideas in PEP 695, and I wish it had been split into three PEPs:

  1. Allowing type checkers to infer the variance of TypeVars rather than forcing users to manually specify it when defining a TypeVar
  2. New syntax for declaring type aliases
  3. New syntax for declaring generic classes and functions

Idea (1) in the PEP would be a huge usability improvement to the type system and, if implemented on its own, would not require any changes to Python’s syntax. The infer_variance keyword argument (which is all this idea needs to be implemented, if it is isolated from the other two ideas in the PEP) has already been backported to typing_extensions.TypeVar. I would feel much happier if we implemented only idea (1) for Python 3.12, and took some more time to further consider and work on implementing the two other parts of the PEP.

21 Likes
3

In fact it adds three, not four. And the only reason there are three is for error messages: the three new scopes have identical semantics. These three scopes behave very similarly to function scopes, except that they can see names in the immediately enclosing class scopes and that they cannot contain constructs such as the walrus operator.

The implementation adds only three new top-level instructions, not 12. In addition it adds 9 intrinsics, mostly because the PEP introduces a large number of new concepts. The new instructions are needed to implement the correct behavior around class scopes.

Two fields, not four. I agree that it’s unfortunate that these are new fields on the function struct for relatively rare use cases. Would it be better to put them in the function’s __dict__ instead?

1 Like
4

Intrinsics are still instructions. There are distinct only to give us more opcode space, and to keep down the size of PyEval_EvalDefault() for performance reasons.

I stand by my 10% figure, as an increase in the instruction set size.

In fact it adds three, not four.

That’s still two or three too many.

Ideally, we would add no new scoping rules at all. Adding more than one seems like a design failure.

Currently annotations and defaults are a bit odd, in that they evaluate in the enclosing scope, but are semantically strongly tied to the function. Adding an extra scope tightly for these values might make sense. But just one scope, and defined in a way that can be easily understood.

Two fields, not four.

My mistake, sorry.

1 Like
5

One other thing.
PEP 649 plays around with the temporal aspects of scopes, deferring the evaluation of annotations.
It too wants to add fields to the function object, and add some new instructions. Some similar (or the same?) as PEP 695.

The interplay between PEP 695 and PEP 649 doesn’t seem to have discussed much. If there are problems, it would be a lot easier to sort out in 3.13 alpha than in 3.12 beta/rc.

6

PEP 649 should reuse most of what PEP 695 provides, including the three new top-level instructions and one of the two new funcobject fields. PEP 649’s annotation scopes should behave identically to the new scope types added by PEP 695 (except for slight variations in error messages as appropriate).

1 Like
7

I’m not sure how things are normally done, but would it make sense to express some of these concerns in the pull request to see if they can be addressed. Or do you think they would be impossible to address?

In particular you mention the addition of too many intrinsics, opcodes, scopes, and function object fields. Perhaps these can be reduced or justified?

Type variables don’t currently follow LEGB. Most type variables are defined in the global scope, but are meaningless in that scope. It’s very hard to figure out in which scope a type variable is meaningful. So I’m not sure that it’s fair to say that “scoping is easy to understand: use the LEGB rule”.

I’m also not sure why looking at the implementation is the best way to understand the new scoping rules. Perhaps, some of the confusion about any new scoping rules can be illustrated with Python code examples? Do you find the new scoping rules in the PEP problematic, or the implementation problematic?

1 Like
8

I think that having this discussion on the PR risks excluding those of us who don’t follow typing developments but are concerned about the amount of complexity being added in the name of typing. I think it’s absolutely the right move to raise these issues here, where the wider community has a chance to comment.

For example, the additional scope(s)[1] are a concern. It’s very easy to claim that “you don’t need to care about them unless you’re writing type declarations” (I’ve no idea if that’s even true, but I assume so) but the reality is that a lot of library developers are being asked to either add type annotations, or accept PRs that do so. And having to research a whole bunch of complexity that you’ve so far been ignoring, just to deal with such requests, isn’t good - particularly as you typically end up having to maintain the annotations that get added, so you get exposed to bug reports on the edge cases that the original proposer missed.

I commented previously on PEP 695, to the effect that “this seems very complicated” and the response was mostly the standard “if you don’t use it you don’t need to worry”. I wasn’t entirely convinced by that even then, and now it appears from what @markshannon is saying that there are further complexities in the implementation, ones that may block optimisation opportunities that I would personally find much more beneficial than the new typing features.

Anyway, I run the risk of sounding like a broken record if I say too much. But I’m +1 on this being discussed here rather than on the tracker, and I’m also +1 on delaying PEP 695 until Python 3.13, so that we have time to properly understand all of the implications. It’s only 12 months - what’s so urgent that it can’t wait that long?

  1. Whether it’s four, three, or “three but they are all just variations on one concept” matters little to me. ↩︎

30 Likes
9

Reiterating my similar concerns as a maintainer who would need to learn, add, and support this to a lot of libraries. Seeing how it interacts with the lazy evaluation PEP, trying out applying it to popular libraries, reducing the implementation complexity, etc. all seem like good reasons to delay this.

7 Likes
10

Basically a +1 to Alex’s comment. I’m supportive of the idea of merging infer_variance=True in 3.12 and delaying syntax changes till 3.13. This what I had anticipated in my long post:

Defer syntax changes until the dust has settled on autovariance and PEP 696 (defaults for type variables), or reject the syntax changes outright

I’m impressed by and grateful for the work it took to make PEP 695 viable in 3.12. But I was surprised the implementation of scopes didn’t get more discussion when the PEP was being discussed, and it’s important that we feel comfortable with the implementation. I agree with others that 3.13 is a good release to nail down the interactions with PEP 696 and 649 (and e.g. to avoid slightly different syntax in the case of 696). We’ll get there when we get there.

6 Likes
11

I’ll share a bit about my mental model for how scoping works under PEP 695, which might allay some concerns. This will be based on the proposed changes in PEP 695: Lazy evaluation and other small changes by JelleZijlstra · Pull Request #3122 · python/peps · GitHub (which have already been submitted to the SC), so this is slightly different from the current version of the PEP.

There are four cases:

  1. Non-generic type aliases
  2. Generic type aliases
  3. Generic functions
  4. Generic classes

For each, I’ll provide an example, along with a rough translation into equivalent current Python code. These translations don’t exactly run as written, since the syntax enables creating a few objects in ways that are not accessible from normal Python code.

All of these translations are subtly incorrect in a few ways, which I’ll discuss more at the end: expressions like yield are disallowed; qualnames are different; and hidden functions have access to enclosing class scopes.

1. Non-generic type aliases

type Alias = int

Equivalent to:

def __evaluate_Alias():
    return int
Alias = typing.TypeAliasType(name="Alias", evaluate_value=__evaluate_Alias)

typing.TypeAliasType is new. It does not actually have a public constructor in the current implementation; it can only be created through the type ... = ... syntax.

The value is lazily evaluated and can be accessed through the Alias.__value__ attribute. Lazy evaluation allows for an intuitive definition of aliases that contain forward references, such as mutually recursive aliases:

type Expr = SimpleExpr | tuple[Expr, BinOp, SimpleExpr]  # e | e + e
type SimpleExpr = str | int | ParenthesizedExpr  # "x" | 1 | (e)
type ParenthesizedExpr = tuple[LeftParen, Expr, RightParen]  # ( e )

2. Generic type aliases

type Alias[T] = list[T]

def __generic_parameters_of_Alias():
    T = typing.TypeVar("T")
    def __evaluate_Alias():
        return list[T]
    return typing.TypeAliasType(name="Alias", evaluate_value=__evaluate_Alias, type_params=(T,))
Alias = __generic_parameters_of_Alias()

For generics, we wrap the evaluation of an alias in a hidden, immediately evaluated function. The type parameters of the alias are set as locals in this hidden function.

Things are a little more complicated if the TypeVar has a bound:

type Alias[T: int] = list[T]

def __generic_parameters_of_Alias():
    def __bound_of_T():
        return int
    T = typing.TypeVar("T", evaluate_bound=__bound_of_T)
    def __value_of_Alias():
        return list[T]
    return typing.TypeAliasType(name="Alias", evaluate_value=__value_of_Alias, type_params=(T,))
Alias = __generic_parameters_of_Alias()

The bound of TypeVars created through this syntax is lazily evaluated, unlike the bound of current typing.TypeVar. It can be evaluated by accessing T.__bound__. (The Python constructor of TypeVar does not have an evaluate_bound parameter.)

Bounds are lazily evaluated because they may often contain forward references.

TypeVars can also have constraints; those work similarly at runtime to bounds.

I’ll omit TypeVar bounds in the next two examples, but they work in the same way as for generic aliases.

3. Generic functions

def func[T](arg: T) -> T: ...

def __generic_parameters_of_func():
    T = typing.TypeVar("T")
    def func(arg: T) -> T: ...
    func.__type_params__ = (T,)
    return func
func = __generic_parameters_of_func()

Like generic aliases, generic functions are defined in a hidden immediately evaluated function.

Defaults and decorators are evaluated outside this function:

@decorator
def func[T](arg: T, arg2=SOME_DEFAULT) -> T: ...

__arg2_default = SOME_DEFAULT
def __generic_parameters_of_func():
    T = typing.TypeVar("T")
    def func(arg: T, arg2=__arg2_default) -> T: ...
    func.__type_params__ = (T,)
    return func
func = decorator(__generic_parameters_of_func())

4. Generic classes

class Cls[T](Base):
    def method(self) -> T: ...

def __generic_parameters_of_Cls():
    T = typing.TypeVar("T")
    class Cls(Base, typing.Generic[T]):
        __type_params__ = (T,)
        def meth(self) -> T: ...
    return Cls
Cls = __generic_parameters_of_Cls()

The base classes are defined within the hidden function. As with functions, decorators are evaluated outside the hidden function (not shown here).

5. Minor differences

All of the translations above involve hidden functions. They differ from normal functions in one major way that I’ll discuss in the next section, and a few minor ways:

  • The names of these functions don’t show up in the __qualname__ of objects defined within them, though you may see it in stack traces.
  • Expressions within these functions cannot contain:
    • yield expressions
    • yield from expressions
    • await expressions
    • := (walrus) operators

The motivation is that these expressions would have unexpected effects and there is no use case for them in typing generics.

6. Class scopes

The more important way in which PEP 695 hidden functions differ from normal functions has to do with class scopes: all hidden functions can see names defined in the immediately enclosing class scope.

Consider this example:

class Cls:
    class Nested: ...
    type Alias = Nested

If we translate that using the example from above, we get:

class Cls:
    class Nested: ...
    def __evaluate_Alias():
        return Nested
    Alias = typing.TypeAliasType(name="Alias", evaluate_value=__evaluate_Alias)

However, if you access Alias.__value__ with this code, you’ll get a NameError: Nested is not defined, because normally, functions nested within class scopes do not have access to their immediately enclosing scopes.

You can construct similar problematic examples with generic methods:

class Cls:
    class Nested: ...
    def meth[T](self, arg: Nested) -> T: ...

Here the annotations for meth are evaluated within a hidden function, so Nested would normally not be visible there.

To fix this problem, we change scoping rules so that all of these hidden functions have access to their immediately enclosing class scope. This allows names to be resolved in a more intuitive way.

The runtime implementation for this scoping change accounts for a lot of the complexity of the implementation: the function objects for the hidden functions have to carry around a reference to the class scope that defined them (the new func_class_dict attribute), and there are new opcodes that load names while checking the func_class_dict field first.

2 Likes
12

I’m afraid I could barely follow any of that. I lost most of my sense of what was going on around the end of item (2).

One thing that’s not at all clear to me - as a (pretty experienced, but with only a superficial use of type annotations) Python programmer, how do I determine whether I need to understand this stuff? Specifically, what must I do to ensure that my lack of understanding of essentially everything you posted in your message, doesn’t hamper my ability to maintain and support code that I write? Must I avoid annotations altogether? Must I avoid generics? Even something relatively basic like def f[T](arg: list[T]) -> T? What about the current syntax for the same thing:

T = TypeVar("T")
def f(arg: list[T]) -> T:
    ...

The more this discussion goes on, the less I feel that I should be using types at all, as I apparently know far less than I thought I did (which wasn’t much in the first place :slightly_frowning_face:).

13 Likes
13

Does your code have a TypeVar in it? If so, then use the new syntax, else don’t worry about it.

I think a key point to make is Mark’s points, while valid, are very technical around low-level details the vast majority people I would argue will never need to care about (descriptors, anyone?). Remember, we are discussing the specification, not the HOWTO, on how to use the thing. Most of us have not read the entire Python language specification and yet we are all comfortable using Python, so I don’t see why typing is any different. Do you need to understand how type checkers will resolve this down to the smallest detail, or just how to use the feature day-to-day?

2 Likes
14

A lot of these weird edge case behaviors sometimes pop up for runtime annotation usage. In practice unless you are writing a library like pydantic or cattrs, this is mostly implementation detail. I do write cattrs like library and sometimes have to look into some of these details but many of them I don’t handle/give up as complex. PEP 649/563 saga similarly involves a lot of complexity and handling questions of closures buried inside your annotation usage can matter for standard library implementation, but a user should not review unless you want to be maintainer of typing.py.

In practice even for runtime typing most of it can get by using get_type_hints/get_annotations and treating how it extracts them out as black box. I hope as a user that as time goes on, I can call get_type_hints and otherwise trust standard library to have done hard messy details of getting annotations to behave as documentation/tutorials explain. Less user manual tracking with scopes/namespaces the better.

edit: As a static typing only user at end how def foo[T] gets interpreted by runtime doesn’t impact much as long as it’s defined, available for you to use as a hint inside, and code runs same way as if you didn’t use type hint. For code that uses type hints and only runs mypy/other static type checkers like before it remains gradual typing without impact on your runtime usage.

15

OK, so my takeaway from this is that if I’m not using annotations at runtime, I don’t really need to care. And the user-facing documentation (i.e., the stuff that isn’t the language reference!) will be readable without going into these sorts of edge cases. (Let’s skip over for now the fact that the typing docs are somewhat difficult to find if you don’t realise that the mypy docs are the de facto user documentation for typing in general, not just for mypy…)

Thanks for helping me understand the context a bit better here. I still think there’s been enough reservations expressed here to suggest that deferring to 3.13 is a reasonable idea, but my reservations have been (mostly) addressed[1], and the question of deferring isn’t my call to make, so I’ll leave it at that.

  1. The general point about how much complexity this adds remains, but it’s a matter of principle rather than a practical issue for me at this point. ↩︎

6 Likes
16

I’m not very proficient in the typing internals, and the PEP does not discuss these scoping changes afaict, so I wonder: If

class ClassA[T: str]:
    def method1(self) -> T:
        ...

is supposed to be equivalent to


_T_co = TypeVar("_T_co", covariant=True, bound=str)

class ClassA(Generic[_T_co]):
    def method1(self) -> _T_co:
        ...

why are you not changing the compiler to give the same bytecode? I assume there is a good reason but I don’t understand the internals enough to see it.

17

We want the scoping rules to be a little different. A problem with the current syntax is that TypeVars are too dissociated from the class definition: you define them in the global scope, you can use the same TypeVar for multiple classes, etc. The new syntax means that a TypeVar is defined only in association with one specific class, which makes it much clearer what the intended scope for the TypeVar is.

The runtime scoping rules are discussed in the section PEP 695 – Type Parameter Syntax | peps.python.org.

2 Likes
18

Thanks for the pointer, I looked at the runtime implementation section in the PEP but turns out it was described elsewhere.

I reread the post andlooked at the PR you mentioned in that same post because I was curious if this

was described in further detail, but I couldn’t find any description of these error messages. Would you mind elaborate on that (if it is not OT). Also, afaict, the PR still says that only one lexical scope is introduced. If it is the case that three scopes are added, the PEP should be updated to reflect that three scopes are added IMO.

19

Then generate them with a special name?

_T_co-for-ClassA = TypeVar("T", covariant=True, bound=str)

class ClassA(Generic[_T_co-for-ClassA]):
    def method1(self) -> _T_co-for-ClassA:
        ...

Yes, I put hyphens in the variable name, because it’s fine to do that as long as you do it after the tokenizer has run (and after any additional validation, but I’m pretty sure we trust the tokenizer to split up tokens). It means normal code can’t accidentally modify the variable.

You could even name it something like $001 and just increment a counter as each one is encountered. Type checkers won’t see this, because they run out of band, so they only see the source code, and anyone inspecting at runtime is only going to see the name attribute which is passed into the instance anyway.

There shouldn’t be any need to introduce actual scoping rules for code generated by the compiler.

2 Likes
20

The error messages in question are these:

>>> type T = (x := 3)
  File "<stdin>", line 1
SyntaxError: 'named expression' can not be used within a type alias
>>> def f[T: (x := 3)](): pass
... 
  File "<stdin>", line 1
SyntaxError: 'named expression' can not be used within a TypeVar bound
>>> class X[T](a := 3): pass
... 
  File "<stdin>", line 1
SyntaxError: 'named expression' can not be used within the definition of a generic

The PR description at gh-103763: Implement PEP 695 by JelleZijlstra · Pull Request #103764 · python/cpython · GitHub doesn’t say anything about the number of scopes, so not sure what you’re referring to there.