Please consider delaying, or even rejecting, PEP 695

PEPs
41

Wouldn’t T be captured in the closure in this case? So deleting it from the global namespace is fine.

42

Globals aren’t captured in closures.

>>> T = "global"
>>> def f():
...     return T
... 
>>> print(f())
global
>>> del T
>>> print(f())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
NameError: name 'T' is not defined

We need a new scoping mechanism to make scoping work as expected for PEP 695. The mechanism I implemented works for all cases that I am aware of, and nobody so far has proposed anything else that works as well.

2 Likes
43

Ignoring the issues of containing scope and lazy-loading, and trying to solve only the more immediate problem of the scopes: Why not work the type parameters into the signature of the PEP 649 __annotate__ function (ignoring format here for simplicity) and the equivalent __value__ function for aliases?

Non-generic type aliases:

type Alias = int
# =>
def __value__():
    return int

Generic type aliases (with bound):

type Alias[T: int] = list[T]
# =>
def __value__(T=TypeVar('T', bound=int))  # could be bound=(lambda: int)
    return list[T]

Generic functions:

def func[T](arg: T) -> T: ...
# =>
def __annotate__(T=TypeVar('T')):
    return {'arg': T, 'return': T}

Generic classes:

class Cls[T](Base):
    def method(self) -> T: ...
# =>
T = TypeVar('T')  # only visible to interpreter while building class
class Cls(Base, Generic[T]):
    def __method_annotate__(T=T):
        return {'return': T}

This seems to me as if it has three benefits:

  1. No scoping, type parameters are stored as default values of the annotate functions and available in the __type_params__ of the objects.
  2. It’s clear how the type specification can be used by the third-party annotate functions mentioned in PEP 649. [1]
  3. Since these are real variables, one could in principle use __annotate__(format, int, (float,), [str]) to inform about the annotations of myfunc[int, float, [str]] at runtime.

  1. How does that work with the new scopes, anyway? ↩︎

1 Like
44

That’s not enough; the type parameters are not only used in annotations. For example, they may be used in base classes or cast() invocations inside generics.

In your generic alias example, you still need the new PEP 695-style scope so that the __value__ function has access to an enclosing class scope.

I don’t know what the __method_annotate__ method on the class is supposed to do. If somehow the annotations on the class’s methods need to be aware that they are using the T from the class’s type parameters and generate an additional parameter to __annotate__, that sounds an awful lot like you still need scoping semantics like the ones I already implemented.

1 Like
45

My point wasn’t actually so much about the defaults, but about making everything explicitly locally scoped on the Python side. The interpreter still needs to keep track of things – of course – but I would not call that “scoping semantics”. From the user perspective, everything is a local variable everywhere.

If you also want to use the symbols in functions without going through __type_params__, introduce the name explicitly in both class and function scopes (the existing ones).

class Cls[T](Base):
    clslist: list[T] = []
    def method[R](self, x: T, y: R) -> tuple[T, R]:
        return cast(tuple[T, R], ...)
# =>
_T = TypeVar('T')  # only known to interpreter
class Cls(Base, Generic[_T]):
    T = _T  # class scope

    clslist: list[T] = []

    _R = TypeVar('R')  # only known to interpreter
    def method(self, arg):
        T, R = _T, _R  # function scope
        return cast(tuple[T, R], ...)
    def __annotate__():
        T, R = _T, _R  # function scope
        return {'arg': T, 'return': tuple[T, R]}
    method.__annotate__ = __annotate__
    del _R
del _T

The point here (and with the defaults) is that the compiler would actually bake T into the class and both T and R into the method, e.g. via a LOAD_CONST (I guess there’s probably a better way to prepopulate the local variables).

46

In your example, when the __annotate__ function is run (under PEP 649, that would be if someone accesses __annotations__), _R is already gone, because you del it right after. So this won’t work.

47

No, because this is just an illustration. The compiler would physically store _R and _T in the function it compiles. Defaults is one way, co_consts is another way. The actual symbols _R and _T are not meant to make it into the Python bytecode in the first place.

You can ignore the del statements altogether. They were just meant to say “these symbols don’t make it out into the wild”. They would be Decrefs on the interpreter side.

48

Nicolas, it looks like you’re just making up rules as you go. That’s not helping.

8 Likes
49

In the end, there do only exist integer types and float types (both available in various widths in the CPU).

Anything beyond that is an abstraction. I’d argue that thinks like str, bytes, int etc are helpful abstractions and annotations to use, but that things like TypeVar[T] are fundamentally unhelpful as they do not inform me about what the program is actually doing with actual real-world data.

I therefore think having to type:

from typing import Generic, TypeVar

_T_co = TypeVar("_T_co", covariant=True, bound=str)

class ClassA(Generic[_T_co]):
    def method1(self) -> _T_co:
        ...

is a feature, not a bug. It is good that the wrong thing to do is hard. Organise your code around your data, not around your other code. Please show me your tables, and it will be obvious.

50

@larry drew my attention to another edge case. He brought it up in the context of annotations for PEP 649, but the same idea applies to PEP 695’s lazy evaluation.

Consider this code:

class X:
    A = "A before"
    B = "B before"
    type T = (A, B)
    A = "A after"

X.B = "B after"

What should X.T.__value__ be? Larry argues, and he is probably right, that both of the reassignments should take effect and the value should be ("A after", "B after"). However, in my current implementation, only the assignment that happens within the class scope takes effect, and the output is ('A after', 'B before').

Larry has also suggested a way to fix this behavior, which I will try to implement.

2 Likes
51

Implemented this in Use a __classdict__ cell to store the class namespace by JelleZijlstra · Pull Request #3 · JelleZijlstra/cpython · GitHub.

52

For the record, the SC has been following this discussion and understands the concerns (old and new). We’re still comfortable accepting PEP 695 in 3.12 (and Jelle’s latest changes to the PEP), as long as the implementation can be finished on time (preferably by next week, but the hard deadline is now May 22).

5 Likes
53

Thanks Thomas! The implementation (in https://github.com/python/cpython/pull/103764) is ready from my perspective and has already received review from several people, but I’d be glad to get more feedback from anyone interested. I’m also happy to write up more explanation of the changes in the PR, if that would be helpful for anyone.

2 Likes
54

Sure, but Paul has historically asked about writing typed code due to his maintenance of pip and other projects, so I answered from that perspective.

55

My intention was to ask about maintaining code. A lot of which is reading other people’s code (co-maintainers or contributors submitting PRs) or writing code that follows standards (such as “use types”) that I wouldn’t necessarily use in my own code.

And hence, when I say “I feel like I shouldn’t be using typing” I also mean that I should argue more forcefully against complex type usage in shared projects.

2 Likes
56

I’ll chime in a bit from the perspective of Pydantic.

I’ve reviewed the PEP and believe that not only will it not hurt runtime type checking but it seems like in general it would be of great benefit to runtime type checking, especially in combination with PEP 649. In brief:

  • TypeVar defaults help us because we need to do stuff at runtime in situations where they can’t be parametrized directly (def foo(x: MyModel[T]) for example). We currently use the TypeVar bound or Any, but a default would be more explicit.
  • The new generics syntax and associated scoping rules should make it easier to parametrize models. When someone does GenericModel[T1, T2][T2, int][str] and whatnot we have to trace that through that model, all sub models, etc. I think we mostly handle this now but there may be situations where re-using a TypeVar in multiple places makes it hard to keep track of things. Having more “unique” TypeVars would help.
  • We use to PEP 593 heavily to add runtime typing information without messing with static type checking, but you can’t name type aliases created this way. If you have Users = Annotated[List[User], MaxLen(10)] and use users: MyAlias in a FastAPI function you probably want the parameter to be called Users in your OpenAPI schema, but that name is lost at runtime.

While I personally like the new syntax a lot and would love to be able to upgrade non-library code to use it in a couple of months, I have to recognize that all of the points but the new generics syntax could be enabled without the syntax changes, and maybe even backported through typing-extensions. That’s a lot more interesting for Pydantic and it’s users because otherwise it wouldn’t benefit our users for years (if we have to wait for all Python versions to support these features) or at least require a lot of effort from us to support things on only some versions. TypeVar defaults are already available in typing-extensions. I think it should be possible to introduce the new TypeAlias class so that Foo = TypeAlias(Annotated[int, …], name=“MyInt”) can be understood by type checkers. And if Pydantic (and similar projects) can’t test out these new features or has little incentive to do so we loose valuable feedback (like I said above I read the PEP but it is indeed complex and I can’t promise we won’t encounter issues until we support and use it).

I won’t opine on delaying the PEP or not, I leave that to those more knowledgeable, but I can say that in general we are both supportive of the changes but are also in favor of exploring ways to get them without syntax changes to get users more immediate benefit and make implementation easier for us. I hope this can be explored even if the features ship in 3.12.

3 Likes
57

That’s an interesting idea. My current PEP 695 implementation doesn’t allow you to instantiate typing.TypeAliasType (other than through the type statement). If others would also feel this is useful though, I can add a public constructor for Python 3.12, and we can provide a backport in typing-extensions.

3 Likes
58

I wrote a detailed account of my implementation of this PEP in Implementing PEP 695 | JelleZijlstra.github.io. I hope this can help answer some of the questions in this thread.

5 Likes
59

I would be very supportive of a way to backport new TypeAlias. Having some way to preserve TypeAlias at runtime would help here as a lot of type aliases I work with are Unions. It’d be nice for runtime documentation tooling and makes some inspection issues with type alias simpler especially if get_type_hints could return new alias object instead of value of it.

1 Like
60

Small semi-related question: if there is a new type keyword, what happens to the current builtin function of the same name? (couldn’t see any mention of it above or in the PEP)