PEP 695: Type Parameter Syntax

Can you elaborate on what you mean by “unprecedented density of information”?

For reference, here’s an example of the existing syntax and proposed new syntax for a typical generic class.

# Existing syntax
class Foo(Generic[T]): ...

# New syntax (proposed)
class Foo[T]: ...
4 Likes

As Jelle mentioned, the syntax form type X = int | str reads naturally and consistent with type alias declaration forms in other popular languages. It’s also consistent with how type aliases are declared historically in Python (e.g. X = int | str and X: TypeAlias = int | str), so it should feel natural to current Python users.

Using as is an idea that didn’t come up in any of the lengthy syntax discussions we had in the typing forums when discussing this PEP. I think it’s an interesting idea, but I see it as problematic because it moves the type variables to the end of the statement. This is inconsistent with the proposed class and def variants. I find it difficult to read because I need to scan to the end of the statement to first determine which symbols are type variables. With the proposed syntax, I can simply scan from left to right, and the information needed to understand the statement is presented in that order.

3 Likes

Sure! There’s a new use of square brackets in the proposal, particularly in the case of callables that can lead to very cryptic function signatures. One example would be the decorator from PEP 612:

def with_request[R, **P](f: Callable[Concatenate[Request, P], R]) -> Callable[P, R]:
    ...

In this example I feel that:

  • the first square brackets throw off intuition on how to read a signature, it’s surprising to see the square brackets;
  • it’s not immediately clear what **P is;
  • this signature is already 84 characters long even though it’s using single-character typevarlikes and just one function argument.

The end result is that it’s harder to read this sort of function signature than it was before. It’s additional information inserted into the mix. In contrast, the following existing “bolted on” syntax solves all three problems:

P = ParamSpec("P")
R = TypeVar("R")

def with_request(f: Callable[Concatenate[Request, P], R]) -> Callable[P, R]:
    ...
3 Likes

For what it’s worth, I agree with @ambv here - the version with explicit named type variables is much easier to read, even though it basically only removes [R, **P] in favour of a couple of explicit assignments. And I can look up ParamSpec to find out the semantics of P, whereas in [R, **P] I don’t have the first clue what to research to understand the syntax.

3 Likes

I think primary issue with current syntax is type variables/paramspecs often are many lines away from definition and reused across several functions/classes that it becomes easy to get lost where they come from. If you have your paramspec 3 lines away from your function then I agree change looks smaller benefit. When you have 100 lines away or in a different file then defining it separately becomes more confusing. Especially for any type variables that use other arguments (bound/variance). Today type variable may have arguments important for its meaning but not even be on same screen as function/class where you use it.

One option is always define it next to function/class in which case you’d got a lot of duplicate definitions. While generic types may be less common across typing files, for files that do use them they tend to have many of them near each other.

1 Like

How is this different from using a class or function from a different module or further up a file?

(To be clear, this is a legitimate question, not a snarky retort or anything like that. Remember that I know nothing about typing :slight_smile: )

2 Likes

I think this is mainly due to variance. Whether type variable is covariant/contravariant matters sometimes but is irrelevant in other usages that it’s fairly easy to get them mixed up. This pep does offer separate solution to that of automatically infer variance that would be helpful and is independent of syntax change.

T = TypeVar("T", covariant=True)

class Foo(Generic[T]): # Here covariance is important
  ...

def my_sum(x: list[T]) -> str: # Here covariance has no meaning at all.
  ...

Other ways type variable can have meaning is bounds/constraints. I do hit this issue less with them as usually you can name variable more meaningful then T. Like there I would instead have,

MessageT = TypeVar("MessageT", bound=Message)

I think the one other place where having definition tied to function/class matters is making scoping clearer.

T = TypeVar("T")

class Foo(Generic[T]):
  ... # 100 lines later
  def combine(self, x: T, y: T) -> T:
    ...

vs

T = TypeVar("T")

class Foo:
  ... # 100 lines later
  def combine(self, x: T, y: T) -> T:
    ...

have two very different meanings. In first the type variable is scoped to class but in second it is independent of Foo and combine function can always be used generically. If type variable where defined like,

class Foo:
  def combine<T>(self, x: T, y: T) -> T:
    ...

then it’s explicit that combine’s T is generic on it’s own.

1 Like

OK, that makes some sense to me. As P is simply a placeholder, there’s little information in P = ParamSpec("P") beyond "this is a ParamSpec, and that’s not something you’d reuse in the same sense that you reuse a global function or class. But conversely, as it’s a throwaway name, what’s wrong with duplicating it before each function/class declaration? It’s a temporary name, just like a temporary local variable.

So I accept that explicit assignment is a bit more verbose, as well as that it’s not typical in strongly-typed languages. But Python has optional typing, so there are going to be compromises. And you haven’t addressed my other point about discoverability - with types being optional in Python, and advanced type constructs having no runtime effect, the typical user is much less likely to be familiar with type syntax, so being able to easily look up the meaning of a type annotation is much more important (IMO).

And to address the inevitable response that if you don’t use types (and hence don’t know about them) you won’t need to understand them - in my experience a pretty common scenario is a project mandating type annotations (because of the maintenance benefits) but typical contributors having little familiarity with typing. So far from being a case of “if you use them, people will know how they work”, complex, difficult to understand types can risk being a deterrent for community contributions to a project.

2 Likes

For discoverability my work pattern is mainly in an IDE where I rely on hovering over variables/objects to see more about them. For vscode I know if hover over P/T it’ll show a hint it’s type variable/paramspec and let me click to see where it’s defined. I’d guess other IDEs like pycharm have similar support while some editors may lack it.

Otherwise I’m unsure of discoverability for syntax. How do people discover meaning := expression? That feels comparable to me with introducing syntax. I find it from reading python release notes + teammates teaching me, but I’m aware that both are things a beginning may lack.

1 Like

I think both of these forms are horrible compared to this:

def with_request(f):
    ...

So what is the best way back to that while retaining the type information? One thing that I might have missed in the discussion would be a C++ inspired decorator (exact syntax debatable):

@generic[R, **P]
def with_request(f: ...) -> ...:
    ...

I think that takes some of the information density out of the signature, while not separating the declaration of R and P potentially thousands of lines from their use.

The real ugliness in this specific example comes from the callable syntax. Now this strays far off topic, but if you also imagine some Callable shorthand (exact syntax debatable), you get something almost readable for such a complex annotation:

@generic[R, **P]
def with_request(f: (Request, **P) -> R) -> (P -> R):
    ...

The difference here is that you need to understand := to accurately read Python code.[1] Because types have no runtime behaviour, you can ignore them when reading code (at least when you can assume the code works). They’re there so that the machine can validate your changes to the code, and also to help reduce the amount of reading you need to do in order to make a correct change.


  1. Though for the record, I opposed := at the time on this same basis - that it makes it harder to read Python code. ↩︎

2 Likes

This is typically where I lose interest in typing discussions. I accept that technically co/contra-variance may be important, but in my experience, no-one except “typing experts” use or understand those terms. Rust has a strong type system, with extensive use of generics, and I don’t recall ever seeing covariance mentioned in the documentation of how Rust’s types work. If Python generic types require you to be familiar with this terminology, then to be blunt, you’ve lost a big chunk of your audience right there. I basically can’t understand any of the rest of your post, although I can (of course) go and look up the docs on TypeVar and covariant. But if all I have is [R, **P] style syntax, I have nothing to look up even.

So all I’m getting in any practical sense at this point is “this is all too complicated, don’t use it”. And that’s coming from a fan of Rust’s typing system, and someone who read Haskell’s typing system and thought “that’s quite cool”. So anything but a “typing skeptic” :wink:

A lot of my work involves doing PR reviews in github’s interface - where there’s no IDE support. And in any case I strongly object to the idea that Python needs IDE support to be usable (I had enough of that with commercial Java environments).

They are familiar with it, typically. That’s my point about typing being optional. If it’s embedded in syntax which isn’t intuitive or easily discoverable, it’s no longer optional - you have to be familiar with it or you’re stuck.

6 Likes

On 3/28/2023 2:05 PM, Eric Traut :

[erictraut] Eric Traut https://discuss.python.org/u/erictraut
March 28

Jacob Nilsson:

Do you have any thoughts on about using |as| instead of |=|?

As Jelle mentioned, the syntax form |type X = int | str| reads naturally
and consistent with type alias declaration forms in other popular
languages. It’s also consistent with how type aliases are declared
historically in Python (e.g. |X = int | str| and |X: TypeAlias = int |
str>), so it should feel natural to current Python users.

This doesn’t look at all natural to this current Python user.

(OK, I don’t use typing, I’m relying on the promise that it is optional)

I think main disagreement is variance is reasonable concept for typing intermediate to learn and is documented well in mypy’s usage. Understanding why this code,

def foo(x: list[float]) -> None:
  ...

y = [1]
foo(x)

is a type error, but this code is safe

def foo(x: Iterable[float]) -> None:
  ...

y = [1]
foo(x)

is safe I think is something python typing user will hit early. It is common place for confusion, but I do view it as fundamental and place where new prs often will need some help to learn when to prefer list vs Sequence or dict vs Mapping. I do think for newer type features that rely heavily on generics it is necessary to learn prior typing concepts first. The rest of this comments a bit more on theory/rust/haskell side and less relevant to python variance question.

Rust has variance and common types like Cell vs Vec behave differently because of it. Haskell has far more complex type system features related to generics that I am curious about, but are reasonable to call esoteric. On Haskell side current generic types in python can roughly be called non-higher order, non-higher kind, and non-dependent types. Haskell has support for first two (third in progress) with higher kinded types probably being one of the longest typing discussion threads.

Functors, Applicatives, Monads, and many other type classes in Haskell strongly rely on higher kinded type support to work. Why are functors/monads (many confusing tutorials) a major thing in haskell but not in other languages? One big reason is that very few languages have complex enough generics to support monad like usage. There is a python library that tries to partially emulate them. I looked for a few links to haskell pages explaining HKT/higher order types and sorta came empty for a good explanation. The python typing discussion on it is probably better explanation for higher kinds then more theory pages. My guess is one day python may consider higher kinds, but is pretty unlikely to do dependent types/higher rank types.

OK. I don’t know why one is safe and the other isn’t (BTW, I assume the call site has a typo and should be passing y). I can do some research and find out (I consider explaining it to me to be off-topic for this thread, so I’m happy to go and find out for myself). But right now, it’s not a distinction that I can immediately grasp.

Where does that leave me? Am I unqualified to have a view on this PEP? And that’s not sarcasm - I genuinely don’t know. I came here because @thomas called for more input, but I know I have limited understanding of Python’s typing, which is why I’d not provided feedback before now. I assumed that the call for feedback implied that the SC wanted the discussion to not be just between typing experts.

Long story short the distinction is in mutability. If a function accepts an immutable iterable of floats, you can pass a list of ints to it just fine because there is no risk anything will break. If that same function would accept a mutable collection of floats, it would potentially be mutating that collection. So if it put a float at the end of your list of ints, it would no longer be a list of ints, right?

I don’t think people should be discouraged from opining on matters that will affect them, and the look & feel of function signatures affect everybody who uses Python.

That being said, it’s for sure helpful to understand context on why extra features are needed in typing and what they allow to express that was impossible or hard to express before.

In the case of PEP 695 I don’t think any entirely new form of expression is enabled by the proposed syntax. The innovation seems to be bringing the definition of typevarlikes closer to where they are used, and making variance implicit (which IIUC is already possible without changing syntax).

I think you are fully welcome to give feedback as one of many that would encounter the syntax in future if accepted. The example is not meant to be a criticism of anyone’s choice of usage of typing. I do think value of new generic syntax will come more to users who do follow that example and similar examples in mypy page. This is mainly for generic specific syntax (<T>, [T]). So I guess one question is does new syntax feel comparable enough to old that if it mainly benefits subset of typing users it would be good? I also don’t know what percent of typing users that is.

Similar recent typing pep is variadic generics pep. That pep also adds an advanced generics feature I expect fewer to use. It comes with a syntax change but it is smaller change and one unlikely most would hit. Pep behind paramspecs is also similar in I expect most code uses them relatively rarely but it is necessary for some functions to have good type inference.

I think other changes in pep (type alias + auto variance) are orthogonal here and may have value to broader user base.

That seems like a naming problem, though; why can’t you use a more descriptive name in that case instead of T?

That won’t work (literally) because decorators don’t introduce names, so referencing those type hints later would lead to a NameError. Basically you would have to update the parser to special-case your @generic and that isn’t acceptable in my opinion.

2 Likes

You can. In more full examples the type variable may already have meaningful name for another reason though. I often works with type variables representing callable like things with names like

InputT = TypeVar('InputT')
OutputT = TypeVar('OutputT')

Usually InputT is covariant, sometimes it can’t be. You could then,

InputT # Or maybe InputT_inv although never seen invariant suffix.
InputT_co

is one way to resolve it. It’s sorta like having all your variable names have type as suffix like Hungrarian style. I do see people write _co/_contra suffix so it is common way to handle with it.

The languages with type declarations that I have experience with have separate namespaces for values and types.

OCaml:

# type ty = int;;
type ty = int
# let ty : ty = 5;;
val ty : ty = 5

Rust:

fn main() {
    type ty = u64;
    let ty : ty = 5;
    dbg!(ty);
}

These both work.

So does the C code

typedef int ty;
ty main() {
  ty ty = 5;
  return ty;
}

and the C++ code

using ty = int;
ty main() {
  ty ty = 5;
  return ty;
}

The criticism I could formulate on this PEP is that it gives a feeling of similarity with other languages with the type syntax, but at the same time, other languages use that syntax to add definitions to a separate namespace, that of types, while the PEP’s doesn’t.

1 Like