PEP 718: subscriptable functions

PEPs
,
102

Thanks for your work on this PEP. The idea certainly has a compelling consistency, and I do think there are use cases for it.

That said, I don’t feel that this PEP is ready. I did a review just now, including of the discussion thread, and I think there are critical questions (including some that were already discussed extensively in the thread) that are still not addressed at all, or are not addressed with adequate precision, in the PEP text. Any significant topic that has been raised in the discussion should appear in the PEP.

The most important thing that I think this PEP needs before it can be considered is a reference implementation in at least one type checker. This is both the hardest and the most valuable part of any typing PEP. It invariably uncovers (and helps to answer) tricky edge cases and design questions, like some of the ones I mention below. In recent years a reference implementation has been a baseline requirement for all typing PEPs; some have had multiple reference implementations!

(The reference implementation also serves another useful role, which is that finding someone sufficiently motivated to put in the time to do it helps indicate that the use cases in the PEP really are motivating.)

Since I did my review, I’ll go ahead and post my notes below, in case they are useful (sorry in advance for the length!). But I would recommend prioritizing a reference implementation above all else.

  • The PEP should clarify that the runtime __getitem__ does not validate whether the function is generic, let alone how many type parameters it has; this is left purely for static verification by type checkers. (This can be discovered by looking at the runtime implementation, but the PEP should be clear about it.)
  • Should type checkers allow a symbol typed with a Callable type to be subscripted, or only objects the type checker knows to be instances of one of the five mentioned “function types”?
    • If so, that for sure will lead to runtime errors not caught by the type checker. Many objects are callable (and thus can inhabit Callable types) and will not be subscriptable at runtime, including many common standard library types: functools.partial, functools.lru_cache, functools.cache… If the proposal is that this risk is acceptable, the PEP needs to make a good case for why.
    • If not, that will dramatically limit applicability. Most decorators can’t return anything more precise than a Callable type, which would effectively mean that most decorated functions can never be subscripted, which will feel to users like a strange and arbitrary limitation.
    • This was previously asked in the discussion thread and there was a fair bit of discussion around it, but the PEP remains entirely silent on it.
    • In general the PEP relies quite a bit on an assumed “we already have GenericAlias for classes, so it’ll work just as well here”, but this issue doesn’t exist in the same way for classes, so it’s important for the PEP to address it in detail.
  • Another runtime issue: functools.wraps crashes if used on instances of types.GenericAlias (their attribute proxy is read-only), so if a generic alias is ever passed to a function accepting a callable and using functools.wraps, that will cause a crash.
  • I personally find the section on “monomorphization and reification” very unconvincing. It’s fine for something to be left as separate future work, but there’s so little concept of how this could even possibly work presented in the PEP, that I don’t even find it justified to claim that this PEP meaningfully “opens the door” for it. In a certain sense, that door has already been opened and we already walked through it with PEP 747. But suggesting that PEP 718 can improve on PEP 747 in some way requires at least suggesting an answer to the question of how a function that expects reified generics would receive them if it is called without being explicitly specialized via subscript. Or would such functions only work if called via a GenericAlias?
  • For introspection purposes, there should be a publicly supported way to tell whether a GenericAlias is for a function or class, without having to isinstance check its __origin__ against five different types.
  • The overload section needs a clearer specification of the modification to the overload algorithm, including precision about what is meant by “subscription may succeed.”
  • TypeVarTuple section should include an example of how to successfully subscript a function generic over a TypeVarTuple.
  • Can you subscript a function generic over a ParamSpec? Example? What about a function generic over both a TypeVarTuple and a ParamSpec? What about a TypeVarTuple followed by a regular TypeVar? Maybe there are existing rules for some of this in the typing spec for classes that we can follow; that should be said explicitly.
  • Is it ever allowed to subscript like foo[*tuple[int, str]] in order to disambiguate when subscripting a function parameterized by a TypeVarTuple? PEP 646 allows this in some contexts.
  • The example C[int].method[str](0, "") is marked # OK but is wrong, method should be called on an instance of C or an explicit self must be passed.
  • There was a fair amount of discussion in the thread about performance, some measurements were done, and there was a lot of talk about why it does or doesn’t matter. The topic isn’t mentioned in the PEP.
15 Likes
103

Another useful addition to the PEP would be a list of links to reported GitHub issues that this PEP would address. For comparison, PEP 747 has a section on use cases at PEP 747 – Annotating Type Forms | peps.python.org. The existing examples in the PEP are fairly abstract, but strong motivating examples are especially important for this PEP because it touches the language core, not just typing.py.

5 Likes
104

The first workaround mentioned in the “Motivation” section seems wrong (all type checkers reject it). You would first need to create an instance of my_list before being able to call it. That is, you would need to write my_string_list = my_list[str]()([]) with additional parens, which makes it a lot less appealing. I think you should consider using the __new__-based workaround instead (mentioned higher up in this thread, works with pyright/ty/pyrefly, not with mypy):

class my_list[T]:
     def __new__(cls, arr: Iterable[T]) -> list[T]:
         # do something...
         return list(arr)

my_string_list = my_list[str]([])
3 Likes
105

Strong +1 for this. Rust does this very well and I often wish to be able to do the same in Python

1 Like
106

I still want this to happen. It’s on pause until I’m recovered from my vacation. Timeframe August-September.

3 Likes
107

I should have more time to finish it soon. Just been reading Carl’s feedback.

108

One of the main selling points of this PEP is to make generic functions more like generic classes. But one thing that’s missing from this PEP and discussion are type parameter defaults (spec). For example:

def constrained_addition[T = int](a: T, b: T) -> T: ...

Without type parameter defaults this PEP could be summarized as “Generic functions are now subscriptable in the same way as generic classes are, except in case of type parameter defaults.”, and it’d require yet another follow-up PEP to add this later on.

With type parameter defaults this PEP would be more well-rounded, and it would be way more convincing to be able to state that “Generic functions are now subscriptable in the same way as generic classes are <period>.”

6 Likes
109

In retrospect it looks like a glaring omission, but we can’t blame anyone, because this discussion started eight months before the acceptance of PEP 696. I’m sure the original authors would agree to add this.

110

I’m sure they (we) would; but can you explain more to this Bear of Very Little Brain what the issue is here? What do we have to add or change?

111

The specification does not mention type parameter defaults, which one can do with classes:

class identity[T = int]:
    def __call__(self, x: T) -> T:
        return x
112

Since Python 3.13, anyway.

113

Why would PEP 718 need to change? It is about calls, not definitions. What am I missing?

114

In case of

def f[T]() -> T: ...

x = f()

type-checkers will complain that f is missing a type argument. But in case of

def f[T=int]() -> T: ...

x = f()

You’d expect that type-checkers won’t complain.

A more involved example:

def g[S, T=S]() -> tuple[S, T]: ...

Also, the PEP mentions that this opens the doors to reification, for which well-defined runtime mechanics are essential. So that should then also be well-defined in case of type parameter defaults, i.e. in a way that will at least make it possible for users to be able to tell from within f that T = int if the default is used, and T = <user-provided> otherwise.

115

Sorry for the oversight. I really should get more sleep…

116

Okay, so let’s fix some semantics. Both at runtime and in type checkers:

def f[T](x: T): ...

f  # generic function
f("")  # infers T as str
f[int]  # concrete function, T=int
# The following don't use inference
f[int](0)  # correct
f[int]("")  # type error

Attributes are probably as follows (this is a runtime feature, not a type-checker thing):

f.__type_params__ == (T,)  # unchanged
f[int].__args__ == (int,)  # analogous to classes

It gets more interesting when defaults are present:

def g[T, U=int](x: T, y: U): ...

g  # generic function, as before
g("", "")  # infers T=str, U=str
g[str]  # binds T=str, U=int
g[str, bytes]  # binds T=str, U=bytes
# The following don't use inference
g[str]("", b"")  # correct
g[str]("", "")  # type error: U is bytes, y is str
g[str, int]("", 0)  # correct

Attributes are similar as before:

g.__type_params__ == (T, U)  # unchanged
g[str].__args__ == (str, bytes)  # analogous to classes
g[str].__parameters__ == ()  # ditto
g[int, int].__args__ == (int, int)  # ditto
g[int, int].__parameters__ == ()  # ditto

I still believe that the original PEP 718 covers most of this, with the possible exception of disambiguating that g[str] in the last example binds both type vars (though again, this is the same as for classes, isn’t it?).

4 Likes
117

Did you mean g[str].__args__ == (str, int)?

1 Like
118

That clears things up; perhaps I was more worried than necessary, so that’s a win :slight_smile:

If the desired semantics just roll out naturally, as they appear to, then that’s a good sign overall that this PEP fits in nicely with the rest of the Python semantics.

Anyway, I still think that defaults should at the very least be mentioned in the PEP, so that the reader knows that this is intended to be compatible with PEP 696. That should also help others like me, that also don’t directly see how these desired semantics can be naturally derived from definitions in the PEP, from wondering whether there are might issues in that regard.

1 Like
119

The spec leaves the semantics of type parameter defaults on functions unspecified (Generics — typing documentation; this comes directly from PEP 696 as I recall). If there is some natural semantics for type parameter defaults on functions then we should specify those semantics, either in this PEP or in an independent change to the spec.

120

Sorry, I went back and forth when editing this. . I think the fix is ‘def g[T, U=bytes]…’

121

I think what I wrote clarifies parameter defaults. Would one sentence in the PEP suffice?

“This PEP specifies the semantics of function type parameter defaults precisely and thus requires a change to the typing spec (link), which currently leaves the semantics up to the individual type checker. Defaults on an unsubscripted function call should be used when inference provides no binding. On subscripted function calls all type parameters are bound by the subscription and the call does no inferencing.”