PEP 695: Type Parameter Syntax

PEPs
87

Is it possible to use type comments (# type: ...) for PEP 695 instead? These have the advantage of being both non-intrusive (minimal or no effect on either runtime code and readability, no changes to standard Python syntax) and recognised by the compiler already.

The following is a syntax error when parsed with ast.parse(<source code>, type_comments=True), because type: ... is not allowed to be on a single line on its own:

# type: int

This leaves an opportunity to use the line above a function or class declaration (or either on top or below @decorators) as a place to bind type variables to the definition underneath.

Proposed syntax (taken from various examples in PEP 695):

# Upper bound
# type: [T: str]
class ClassA:
    def method1(self) -> T: ...

# Type variable in function signature
# type: [T]
def func(a: T, b: T) -> T: ...

# Constrained type specification (string quotes omitted for forward reference)
# type: [T: (ForwardReference, bytes)]
class ClassB: ...

# Parameterisation
# type: [T]
class ClassA(BaseClass[T], param=Foo[T]): ...

This has some of the same drawbacks as the rejected idea PEP 695: Prefix Clause, especially regarding scope clarity. However, some of the feedback on this thread about the information-denseness of the current proposal suggests that there is a fine balance between clarity of scope and a flood of square brackets appearing in a class or function declaration, and there’s not much consensus on where to draw the line.

1 Like
88

The comment would not bind the name T, so this would not work without changes to the compiler. And if we’re changing the compiler, we shouldn’t do it to introduce syntax that looks like a comment.

6 Likes
89

Hi, I am a long-time lurker and created an account to express my support for this proposal. I know that one user’s perspective is unlikely to make a difference in such a long and complex discussion, but I do believe that this PEP makes generics and type variables much easier to work with than they are today.

Most important to me is that, under PEP 695, parameters are declared exactly when they are needed and their usage is localized to there and there alone. This is addressed in the PEP as the second paragraph of Points of Confusion:

The scoping rules for type variables are difficult to understand. Type variables are typically allocated within the global scope, but their semantic meaning is valid only when used within the context of a generic class, function, or type alias. A single runtime instance of a type variable may be reused in multiple generic contexts, and it has a different semantic meaning in each of these contexts. This PEP proposes to eliminate this source of confusion by declaring type parameters at a natural place within a class, function, or type alias declaration statement.

I find this to be most compelling.

  • It’s confusing to declare a type variable in the global scope and have its meaning potentially change wherever it’s used.
  • It’s confusing that, despite this, the “bounds” of a type variable sometimes have to be defined ahead of time, but then also only narrowed by the type checker when the type variable is used.
  • It’s confusing that T = TypeVar("T") can refer to a class-scoped type parameter in one place and a function-scoped one in another, or that it can be imported into another module and take on semantics that are further detached from its declaration.

PEP 695 makes this all much clearer. As a user of typed Python, this example from the PEP is anecdotally very common:

# Here is an example of a generic function today.
from typing import TypeVar

_T = TypeVar("_T")

def func(a: _T, b: _T) -> _T:
    ...

# And the new syntax.
def func[T](a: T, b: T) -> T:
    ...

The new syntax more closely mirrors that found in other languages (described in the Appendix). It is more obvious with the new syntax that T is meaningful in the function, but not outside.

The post that I’m replying to says: “The new syntax for generic classes and functions overall feels natural to me as a user of typed Python.” I wholeheartedly agree. Having talked to developers IRL who are more familiar with other languages, this new syntax makes more intuitive sense in that way, too.

Finally, I believe the case for new syntax is stronger here than it was in PEP 677; there, the proposed syntax was conceptually equivalent to the existing syntax. PEP 695, by contrast, clarifies longstanding conceptual confusion about type variables and generics. It’s not “just” syntax—this is a meaningful improvement for users of typed Python.

I hope the SC accepts it.

8 Likes
90

(Apologies for the double post—my previous one was already long, and this is a reply to a post much further up.)

This was my experience (and that of coworkers who were unfamiliar with this) as well. Having to inherit from Generic and declare TypeVar is surprising and kludgey. This, in contrast, feels like the “one—and preferably only one” way to do this, or the way that things “should” work. This is especially true when considering the survey of other languages in the Appendix.

1 Like
91

If the main objection is that the proposed syntax is too “magical” then I think this might be a good compromise.

I do think it’s probably okay to hide the instantiation of the TypeVars behind some magic, though.

A proposal in a similar vein:

class A(Generic[T]) with (T: Any):
    def __init__(self): ...

def reduce(
    function: Callable[[T, S], T], sequence: Iterable[S], initial: T
) -> T with (T: Any, S: Any): ...

I think it might be good to make specifying the bound mandatory, and if there is no bound then you can use Any or object (using object actually seems a bit more correct to me).

Also, instead of re-purposing “with”, it could be done with “where” as a soft keyword:

class ClassA(Generic[T]) where (T: str):
    def method1(self) -> T:
        ...
2 Likes
92

On behalf of the Steering Council, we’d like to report that we are happy to accept PEP 695. Thanks to everyone for the reinvigorated discussion in the last few weeks – I look forward to this step forward for typing in Python.

21 Likes
93

:exploding_head::grin:

Yes and no. As you pointed out, “With PEP 677, you could have argued that it’s intuitive and mirrors existing Python syntax, that it’s a smaller change” while “PEP 695 changing how functions and classes can be declared is a big deal”. But the key detail there is while PEP 677 proposed something that I would argue was a bit of syntactic nicety for something that functioned fine w/o the PEP, PEP 695 has a much broader impact in terms of improving your typed code. So I think a better concept to compare this to in terms of impact compared to cost is decorators; not necessary but there’s a more demonstrable improvement to how your code reads, the general semantics, and minimizing errors than compared to simplifying how you write Callable.

Or at least that’s how I approached the two PEPs when thinking about whether to accept/reject them.

8 Likes
94

This never got addressed.

95

I expect this is something that the implementation work will reveal.

Slices are not expected to ever be used in function or class definitions so it seems like there is sufficient information for to reason about what is what.

1 Like
96

Doesn’t this [T: str] syntax conflict with how slices are parsed.

The grammar changes introduced in this PEP are unambiguous. The : token is never interpreted as a slice when used in the context of a type parameter definition, just as : is never interpreted as a slice when it’s used today for parameter type annotations.

1 Like
97

But it would break my super important and not at all contrived class, as far as I understand.

class BoundedMeta(type):
    def __instancecheck__(self, obj):
        return isinstance(obj, float) and self.LO <= obj <= self.UP

class Bounded(metaclass=BoundedMeta):
    def __class_getitem__(cls, rng):
        lo, up = rng.start, rng.stop
        return type(f'Bounded[{lo}:{up}]', (cls,), {'LO': lo, 'UP': up})

print(isinstance(3.1, Bounded[2.3:5.5]))  # True
98

But it would break my super important and not at all contrived class, as far as I understand.

This PEP has no impact on how : is interpreted in the grammar today. That’s true for your code sample above or any other code that works today. So no fear, your super-important not-at-all-contrived class would continue to work just fine. :slight_smile: .

The grammar changes this PEP introduces are for type parameter definitions only. For example, if you were to change your class to be generic with a type parameter named T, you would use the following syntax.

class Bounded[T: str](metaclass=BoundedMeta): ...
3 Likes
99

I might have missed it but I believe this wasn’t addressed as well. It’s likely just an implementation detail. However the AST change would make it easier for linters (written in Python) to highlight the name specificly without needing to parse the expression themselves (using regex).

100

Congrats to the authors! One more thing to look forward to in 3.12.

101

Congrats! :partying_face: I’m so excited to see it was accepted and can’t wait to see it implemented! :smiley:
Couple of questions though:

  1. When will the PEP’s status be officially updated?
  2. Does it mean we will be able to see it in Python 3.12? After all, we’re about to enter the beta phase meanning that new features aren’t supposed to be added and this feature will need some time to be implemented.
  3. Is there any way for outside users like myself to be updated with the schedules of when such decisions are made? From my perspective, my only option is to enter this (and other pep threads im interested in) every day and check if there is an update, but is there some public schedule which we can look at to know when such decision should be made?
2 Likes
102

You could subscribe to the official issue for the SC request.

103
  1. When somebody makes a PR to the PEPs repo to update it.
  2. Yes, it should go into 3.12, and yes, that means we should start implementing it soon. Hopefully @erictraut can submit his draft implementation and we can iterate on that.
  3. I think subscribing to the GitHub - python/steering-council: Communications from the Steering Council repo is the best way to stay up to date with SC decisions. There is no set schedule; the SC is a small group of volunteers who do things when they have the time to do so.
104

@erictraut Congrats with the SC acceptance! We have only about 3 weeks to get this into 3.12 – the beta 1 feature freeze is May 8 (PEP 693 – Python 3.12 Release Schedule | peps.python.org). And during that time falls PyCon, which means I have less time to focus on this. Do you want to start by merging the current main branch into your prototype, or would you prefer to start over? (The merge alone could be a big job.)

I hope there are volunteers to help with this process. (IIRC Eric’s prototype is linked from the PEP.)

3 Likes
105

My earlier prototype is in this branch, but it implements an older version of the spec (prior to incorporating input about scoping rules). The cpython compiler code has also changed significantly in the past year, so I don’t think it’s worth trying to carry the prototype forward in its entirety.

There are parts of the prototype that can be used with little or no modification. This includes:

  1. Grammar changes
  2. AST changes (Note: these changes don’t yet include the suggestions that @cdce8p made.)
  3. Unit tests
  4. Changes to typing.py classes

I’ve manually integrated these changes into a new branch that is synced with the latest cpython main branch.

This leaves the following work:
5. Symtable changes
6. Compiler changes
7. Moving TypeVar, TypeVarTuple, ParamSpec and Generic implementations from Python to C

I’m hoping that we can get a core Python developer or someone who has expertise with the symtable and compiler components to volunteer to implement the necessary changes. I’m happy to collaborate and share learnings from my proof of concept, but I’d prefer that someone else take the lead here. Any takers?

One consideration is whether PEP 696 (Type defaults for TypeVarLIkes) is going to be pursued for Python 3.12. If so, it might make sense to combine the implementation effort for 695 and 696. Given the short time window available, PEP 696 may need to wait until Python 3.13. @Gobot1234, what is the status of 696? Have you submitted it to the SC for review?

3 Likes
106

It has been submitted and is at the top of our list for discussion next week.

6 Likes