Please consider delaying, or even rejecting, PEP 695

PEPs
21

I considered a name mangling approach; it’s also discussed in the PEP at PEP 695 – Type Parameter Syntax | peps.python.org. As the PEP calls out, that technique is difficult to reconcile with string forward references. It also causes problems with class scopes, because we can’t determine at compile time what names are defined in class scopes. When I tried it, it also turned out to require rather invasive changes to the symtable.c code. The current approach requires relatively more modest symtable changes.

Even if we used name mangling for type parameter names, we’d still need the new scoping rules related to class scopes, because we want lazy evaluation for type alias values and TypeVar bounds (and in the future, with PEP 649, for all annotations).

22

Thanks for the elaboration.

Lazy evaluation

This first thing I note is the inclusion of lazy evaluation.

The original PEP makes no mention of lazy evaluation. This is a very large addition.
Adding lazy evaluation changes the PEP enough that it should be fully reconsidered.

I note that your PR is titled “Lazy evaluation and other small changes”
Adding lazy evaluation is no small change.
Lazy evaluation is complex and subtle; just see the many and long discussions around PEP 649.

New scope(s)

It isn’t clear from your hidden functions what the scope rules are, and how many new scopes are being added.
You state that only one new scope is being added, so assuming that’s true, let’s suppose a let syntax.

let var = value:

Where var would shadow vars in enclosing scopes, but other variables have their normal scope, such that:

x = "outer"
let x = "inner":
    y = "hi from inner"
    print(x)
print(x)
print(y)

would produce

"inner"
"outer"
"hi from inner"

Just to be clear, I am not proposing adding a let keyword, I’m just using it to express the semantics.

So, using the imaginary “let”, and without lazy evaluation:

class Cls[T](Base):
    def method(self) -> T: ...

becomes

let T = TypeVar("T"):
    class Cls(Base, typing.Generic[T]):
        __type_params__ = (T,)
        def method(self) -> T: ...

and

def func[T](arg: T) -> T: ...

becomes

let T = TypeVar("T"):
    def func(arg: T) -> T: ...
    func.__type_params__ = (T,)

It should be possible to define all the new syntax in terms of a single, simple, new scope (which doesn’t have to be what I describe above).
Otherwise, it is just too complicated.

Conclusion

The inclusion of lazy evaluation definitely means that the PEP should be deferred to 3.13.
We need to consider how to approach lazy evaluation, so that it works for all use cases:
PEP 649, PEP 695, possible lazy imports, and other uses.

If lazy evaluation were dropped from the PEP, then I still think it should be deferred.

If it absolutely must go for in 3.12, for whatever reason, then the following needs to happen:

  • Drop lazy evaluation
  • Specify the behaviour using explcit scope notation (as described above)

But, why the mad rush?
Regardless of what happens for 3.12, later versions will be better for a less rushed design and implementation.

6 Likes
23

To be fair here, I don’t necessarily think that the design and implementation have necessarily been rushed. The problem is that the work (because it’s mostly happened on typing-specific lists) hasn’t been particularly visible to people who are working on the interpreter in general, but who have limited interest in the esoterica of typing itself. So we now have a lot of last-minute questions, and there’s a sense of “we haven’t got time to properly consider this feedback” - which feels like rushed design, but is in fact artificial pressure caused by the 3.12 deadline. Hence my support for deferring to a later release.

But having said that, I do think that typing proposals would benefit a lot from wider discussion, earlier in the process. And if that means simplifying the proposal overviews, framing them for non-expert audiences, or putting up with dumb questions from people like me who aren’t familiar with type theory, then so be it - 99% of Python end users will be in that category when the feature lands, after all.

4 Likes
24

Oh I didn’t mean to suggest that it shouldn’t be posted here. I just thought it might be nice to give Jelle an opportunity to respond to and think about the criticisms of his implementation before having the whole world code review a work-in-progress.

I don’t want to step on any toes. I’m just very interested in the evolution of this feature :smile:

The PEP doesn’t use the term “lazy”, but it does mention it:

Type aliases can refer to themselves without the use of quotes.

# A type alias that includes a forward reference
type AnimalOrVegetable = Animal | "Vegetable"

# A generic self-referential type alias
type RecursiveList[T] = T | list[RecursiveList[T]]

Is it not true that forward-references and self-references require lazy evaluation? Or is there a better way to implement them?

The scoping rules are given in this giant section in the PEP. I think Jelle’s reply was more about the implementation details. I’m still not sure if the criticism in this thread is about the PEP or the implementation, although it seems to be more about the implementation? So it makes sense to me that he would explain the details of his implementation.

That seems fair to me. I’m just trying to understand the criticism.

25

I’m mostly just watching from the sidelines, so forgive if this is a naive suggestion, but wouldn’t the deferred annotations from PEP 649 mean that no additional scope was necessary if these “locals” were passed to __annotate__? With __annotate__ effectively being the additional scope?

1 Like
26

The PEP doesn’t use the term “lazy”, but it does mention it

I did a text search of the PEP, and “lazy” doesn’t show up.

Is it not true that forward-references and self-references require lazy evaluation?

Lazy evaluation or something similar, yes. But the mechanism needs to be spelled out, not just glossed over.

The scoping rules are given in this giant section in the PEP.

Giant and incomprehensible.
Jelle’s explanation is a considerable improvment.

27

This might be a naive question, but would the introduction of a new intermediate scope have performance implications for global variables referenced within methods?

I was reading through the PEP’s scoping rules and found it surprising that there was an example where the TypeVar was accessible from within a method’s body.

The section in question, particularly the line with the comment # Prints 'T'.

T = 0

# T refers to the global variable
print(T)  # Prints 0

class Outer[T]:
    T = 1

    # T refers to the local variable scoped to class 'Outer'
    print(T)  # Prints 1

    class Inner1:
        T = 2

        # T refers to the local type variable within 'Inner1'
        print(T)  # Prints 2

        def inner_method(self):
            # T refers to the type parameter scoped to class 'Outer';
            # If 'Outer' did not use the new type parameter syntax,
            # this would instead refer to the global variable 'T'
            print(T)  # Prints 'T'

    def outer_method(self):
        T = 3

        # T refers to the local variable within 'outer_method'
        print(T)  # Prints 3

        def inner_func():
            # T refers to the variable captured from 'outer_method'
            print(T)  # Prints 3

The example is from a nested class, but I couldn’t find an example of what would happen in the case of an ordinary, top level class.

28

That’s what my previous post does, where I provide translations into “hidden functions”: the syntax is mostly equivalent to these hidden functions, except for the change in access to enclosing class scopes (as well as disallowing yield/walrus/etc.).

Your imaginary let syntax muddies the waters further; it’s not needed to explain the semantics.

29

Names are resolved at compile time within methods, so there should be no performance implications for existing code. In your example T would be resolved like any existing non-local variable, and globals would still go through LOAD_NAME.

1 Like
30

Why “mostly equivalent”? Why not be exact?
You are adding a new scope. Adding a new scope seems the obvious way to explain the semantics.

31

I hope my long post with translations was exact. The new scope behaves the same way as function scopes, except that:

  • Names defined in immediately enclosing class scopes are visible within the new scope.
  • yield/yield from/await/walrus are disallowed in the new scopes.
  • The __qualname__ for objects defined within the new scope is as if the objects were defined in the immediately enclosing scope.
1 Like
32

I just now noticed that the example I quoted does in fact include an example of a top-level method.

It does make sense after a bit more thought, it just took some time to wrap my head around it since I don’t personally use nested classes to any extent.

I apologize for jumping to conclusions.

33

What you say doesn’t square up with the implementation.
If the translations in the long post are exact and are in terms of existing syntax, why does the implementation require all the new instructions for name lookup?

The PEP introduces at least one new kind of scope. The PEP needs to very clear about how it (they?) works.
Given that needs to be done anyway, why not use it to explain how the new syntax works?

34

This might be a naive question, but would the introduction of a new intermediate scope have performance implications for global variables referenced within methods?

Always worth checking.

I benchmarked the PR and there is no performance impact.

4 Likes
35

You are right that the PEP could use a more concrete specification of scoping. I pushed such a concrete specification in this commit: PEP 695: Lazy evaluation, concrete scoping semantics, other changes by JelleZijlstra · Pull Request #3122 · python/peps · GitHub. The main difference from the long post above is that the “hidden functions” are introduced with a “def695” pseudo-keyword, indicating that they are almost, but not quite, like regular functions. The most important difference is the behavior around class scopes. This behavior is also the motivation for the new (non-intrinsic) opcodes in my PR.

1 Like
36

Do you never have to understand other people’s code?

5 Likes
37

so, assuming the PEP is (at least) postponed and we get a couple months to discuss it -

Talking about the new scopes - was “variable deletion at block end” even considered instead of all those new scopes?

except blocks have been using those for decades, and very few people get hurt by it.

38

That wouldn’t work because the new variables need to remain accessible in nested scopes after the definition has finished executing. For example:

def make_converter_function[T](typ: type[T]) -> Callable[[object], T]:
    def converter(obj: object) -> T:
         return typ(obj)
    return converter

make_int = make_converter_function(int)

If T was deleted after make_converter_function is created, this would throw a NameError when the inner function is created.

1 Like
39

And if T is deleted after the body of make_converter_function?

40

I’m not sure I understand exactly. I think the suggestion was to do essentially this:

T = TypeVar("T")

def make_converter_function(typ: type[T]) -> Callable[[object], T]:
    def converter(obj: object) -> T:
         return typ(obj)
    return converter
del T

make_int = make_converter_function(int)

In that case, when the body of make_converter_function executes, T is no longer defined.

1 Like