Thoughts on PEP 695, rejection of PEP 677 and the future of typing syntax
Since @thomas / the Steering Council asked the community for more thoughts on PEP 695, I thought I’d write something up.
First and foremost, I think PEP 695 is really well thought through, and I admire all the effort and cleverness that went into it. I think the scoping challenges were formidable and the proposed solution addresses them well. The new syntax for generic classes and functions overall feels natural to me as a user of typed Python.
Summary of PEP 695
I saw a couple messages on Discord and here about the PEP being hard to follow, so here’s a quick summary of the PEP (still contains jargon, but is at least shorter). PEP 695 comprises of three things:
- A new syntax for generic classes and functions
- A new syntax for type aliases
- A way for type checkers to automatically infer variance of type variables
How do these things relate / why is this one PEP instead of two or three?
The new syntax isn’t just syntactic sugar, but an effort to clarify the scoping of type variables.
Type variables are conceptually meaningless outside of a scope that binds them. However, today’s syntax for type variables confuses that fact. This is particularly confusing when the type variable has properties attached (like variance or bounding), because those aren’t properties of the type variable itself, as much as properties of the class, function or alias that binds them.
The syntax changes thus target the places where you’d bind a type variable, mirroring precedent in other languages. The automatic inference of variance is primarily a usability improvement, but is part of this PEP because including it means we don’t need new syntax to explicitly express variance.
Rejection of PEP 677
Okay, I’ve sufficiently buried the lede, so here it is: I would be very surprised if the Steering Council accepted PEP 695. The Steering Council is largely the same as when it discussed PEP 677 (syntax for callable types). Every reason for rejecting PEP 677 applies to this PEP as well, except often stronger.
PEP 677 rejection notice. I excerpt the points below, as per my interpretation, but click through to the notice if unfamiliar.
Let’s go through the reasons the SC provided for rejecting PEP 677:
- We feel we need to be cautious when introducing new syntax.
[…] A feature for use only in a fraction of type annotations, not something every Python user uses
This still clearly applies.
With PEP 677, you could have argued that it’s intuitive and mirrors existing Python syntax, that it’s a smaller change, that it’s easier for users to ignore. But PEP 695 changing how functions and classes can be declared is a big deal… It’s amongst the more personal changes you can make to a language. I expect Python users to have feelings about PEP 695 syntax.
And while I think the SC was wrong on PEP 677, they are absolutely right to be wary of syntax changes.
For example, the small syntax change in PEP 646 maybe flew under the radar, but now it’s causing a little bit of trouble for PEP 649.
PEP 695’s estimate is use in 14% of files with typing. I measured prevalence of TypeVar on the corpus of GitHub - hauntsaninja/mypy_primer: Run mypy and pyright over millions of lines of code . This is 127 projects (often name brand) that use typing in CI and close to ten million lines of code. It depends on how you count, but Callable was used in about 22% of files with typing and TypeVar was used in about 9%. PEP 696 (defaults for type variables) could change how much generics are used, but that remains to be seen.
- While the current Callable[x, y] syntax is not loved, it does work.
This PEP isn’t enabling authors to express anything they cannot already.
[…] We can imagine a future where the syntax would desire to be expanded upon.
Unlike PEP 677, which was just sugar, PEP 695 does make generics conceptually clearer. But it’s still true that the syntax changes in PEP 695 do not enable things that aren’t already expressible. (For what it’s worth, this should be a point of pride. It’s great that users on all Python versions can typically benefit immediately from new typing features)
While type variables are relatively mature, I think they are still more susceptible to future changes than PEP 677 (there was basically only one direction to take PEP 677, which is the extended syntax that PEP 677 discussed). For example, PEP 696 would involve a (straightforward) syntax addition to PEP 695. PEP 695 itself is adding automatic variance. One could imagine future kinds of TypeVarLike’s (things like ParamSpec). Or maaaybe even a future where we model mutability explicitly, which would have implications for variance.
- In line with past SC guidance, we acknowledge challenges when syntax
desires do not align between typing and Python itself.
[…] shifts us further in the direction of typing being its own mini-language
Changing function and class declarations for a typing specific feature doesn’t assuage this fear.
- We did not like the visual and cognitive consequence of multiple
-> tokens in a def
The odds of someone proposing a syntax change that people aren’t concerned about the visual and cognitive consequences of is lower than the odds of Python no longer being dynamically typed
While I think the syntax is better than the status quo and blends quite nicely into existing typed Python, I empathise with worries about a relatively implicit way of defining symbols or more soft keywords and the overloading of
And of course, the Steering Council may have additional concerns that are more specific to PEP 695 (I can think of a few that might come up).
Future of typing syntax
Given the above, my best guess is that the SC will:
- Ask for automatic variance inference to be its own PEP and then accept that PEP (variance is confusing, this is a thing that helps, it’s low burden for type checkers to implement since we already have it as part of PEP 544)
- Defer syntax changes until the dust has settled on autovariance and PEP 696 (defaults for type variables), or reject the syntax changes outright
The rest of this section is written assuming this outcome.
I think if PEP 695 is rejected for effectively a superset of the reasons PEP 677 was rejected, this would be somewhat frustrating, on both “sides”. On the typing “side”, because a lot of effort and ingenuity is spent on PEPs like this and because these changes have the potential to help users (of typing). On the Steering Council “side” because it sucks to say no to a lot of well thought through work that benefits some users but may have global costs — especially if saying no for similar reasons to previous proposals.
I’d love more guidance from the Steering Council on syntax changes, particularly syntax changes that are aimed at ergonomic benefit. The rejection reasons for PEP 677 are quite broad:
- re point 1, every syntax change will be a syntax change
- re point 2, as mentioned, it should be a point of pride and strength that we can typically find kludgy ways to express things without new syntax. I found this point confusing at the time of PEP 677 too, especially since IMO PEP 677 did a good job anticipating future extensions: see Mailman 3 [Python-Dev] Re: PEP 677 (Callable Type Syntax): Rejection notice. - Python-Dev - python.org and the reply
- re point 3, seems to rule out most syntax changes that are about typing ergonomics.
- re point 4, every syntax change will have visual and cognitive consequences. This is subjective and it’s unclear a priori what the Steering Council’s bar here is.
I understand that it’s hard to give guidance here and it’s important to preserve optionality in both ways (to reject things for subjective reasons or accept things that in some ways contradict previous rejections).
To make things more concrete, here are a few random ideas that could be in the guidance action space:
- Syntax changes targeting typing ergonomics should only touch parts of the type system that have not been changed in X years (addresses point 2 of PEP 677 rejection)
- Syntax changes should look more like PEP 637 rather than being ergonomic focussed (PEP 637 was also rejected, but the SC did say the typing argument was the strongest argument for that change) (addresses point 2 and point 3 of PEP 677 rejection)
- Syntax changes should not make use of PEG features (mentioned in PEP 677 rejection) (addresses point 4 of PEP 677 rejection)
- Syntax changes that likely affect
<X% of Python files are unlikely to be considered (addresses point 3 of PEP 677 rejection)
Finally, and this is getting off topic, I think there’s often a nebulous desire expressed for typed Python to feel more cohesive with untyped Python (I think SC might even have said something on these lines, but I can’t find the source). I’d love opinions from everyone on what that means and recommendations on how to go about it (maybe in another thread), for instance:
- On the typing side, this often results in a desire for ergonomic syntax, because ergonomic syntax is a way for something to feel native and cohesive.
- For some users, cohesion means the ability to blur lines between runtime and static type checking. There are limits to what is even theoretically possible here, but for what it’s worth I think we’ve made good strides in recent years to make things more introspectable and future proof the runtime aspects of typing.
- For some users, cohesion means powerful static type checking primitives that look more like writing Python than writing a declarative DSL.
- For some users, cohesion could just mean better resources and documentation. Most non-typing features of Python have a decade (or two) headstart on typing features when it comes to building these resources.
- For some developers, cohesion could mean building static analysis libraries that are easy to build tooling or custom static analysis on top of.
- Finally, for some people, maybe this is just a polite way to say “this stuff looks different, get off my lawn” But don’t worry, we’ll win you over