PEP 718: subscriptable functions

It is perfectly possible in maybe 20-30 lines of code to write a decorator that adds runtime generics in a really easy to use way. [1] PEP 695 makes this quite convenient. Currently the biggest blocker for its usage is that type checkers don’t like it. Implementing the same thing for classes is a bit harder, but very much possible.

IMO runtime access not being possible shouldn’t be considered a problem if this feature, this feature not existing is a blocker for runtime access.


  1. I can’t provide the code right now, I am on my phone ↩︎

1 Like

Thanks for the comments and sorry I took so long to get to this.

Yeah I don’t entirely disagree with this sentiment though I do think it does have merit to stand on its own feet as outlined at the end.

Some of the examples from the PEP were taken from real world code this wasn’t one of these pieces of code though I don’t think it’s actually that unreasonable to see a case where this could come up

Yes this came up from a library online that a friend is writing GitHub - Zomatree/Kine: React like agnostic GUI framework for Python

I’m curious if you’d be able to dig these up because they’d be useful for adding strength to the argument.

My main reasons for writing the PEP

  • Completely agree with Gudio’s point about the complexity of the types, I think this feeds into the next point nicely.
  • Symmetry with classes. I’ve had multiple people be surprised that this wasn’t already a feature. I think it is a logical change post PEP 585.
  • Prospect of runtime access. I know that this isn’t really brought up in the PEP though it is something I am working on, I did see this however as a large enough change that is should be separated from this PEP.
5 Likes

My two three cents:

  • static typing asymmetry between functions and classes is a big pain point in general for our team (mostly data scientists). I would say, generally speaking: people love the benefits of having types, but they don’t like writing them because it’s “hard.” And a big reason it’s hard is that there are a lot of minor inconsistencies, where I have to tell people the equivalent of “oh, yeah, you can’t do it that way because of X - you need to do it like this” (which is always followed by writing 5 extra lines of something they’ve never seen before). Note: I’m not trying to convince any readers of this - if you’ve experienced it, I’m sure you’ll know what I’m talking about, but it’s beyond the scope of this discussion to walk through a bunch of concrete examples. Therefore: any concessions toward consistency (why can I do this with classes but not with functions - I thought Python had first class functions?), while they may seem minor in “functionality” benefit, are cumulatively major in making Python pleasant to use with static types.

  • giving access to this parameter at runtime seems like it would be a nice-to-have, and because I am in favor of this precisely because of the advantages of consistency, I would lean toward throwing it in. The thing that would be unpleasant is if we ended up with a situation where tools ended up requiring users to do both: e.g. my_foo = deserialize_obj[Foo](some_bytes, Foo) - i.e. passing the type as a function argument in addition to using the ‘standard’ annotation. That said, I am in favor of this also primarily as a concession toward consistency; I’ve not personally gotten to use cls.__orig_bases__ yet, so I can’t speak to the usability of the class side of things directly.

  • performance concerns of the added __getitem__ call, etc, really should not be relevant to this topic. The vast majority of real world use cases for this, I predict, will be in cases where very dynamic (and slow) code is running under the hood. The slowness from this form will be dwarfed by the slowness of whatever is actually being run. Furthermore, as has been pointed out, if there’s any need to run this inside a loop, it will be possible to construct the specialized version outside the loop and use it inside the loop.

This PEP doesn’t seem like a huge step forward, but the top two things Python static typing could use more of are:

  • consistency
  • expressivity
    and this helps with both to a small degree.
9 Likes

@Gobot1234 Are you still interested in pursuing this PEP? Maybe you could do a round of revisions based on the latest feedback and link to the current preview? (The preview link from your OP is dead.)

(This PEP was just flagged as one of several typing PEPs whose target revision has been moved to 3.15 since 3.14 is in feature freeze as of today, the (planned) release date of 3.14 beta 1.)

I’d like to follow up on this part, because I don’t think the text in the current draft addresses the question sufficiently.

Consider this example from the typeshed:

class str:
    @staticmethod
    @overload
    def maketrans(x: dict[int, _T] | dict[str, _T] | dict[str | int, _T], /) -> dict[int, _T]: ...

    @staticmethod
    @overload
    def maketrans(x: str, y: str, /) -> dict[int, int]: ...

    @staticmethod
    @overload
    def maketrans(x: str, y: str, z: str, /) -> dict[int, int | None]: ...

The first overload is generic with a single typevar _T . The other two overloads are not generic. What would str.maketrans[int] mean, according to this PEP?

There’s a clear runtime behavior, since the overloads don’t actually “show up” at runtime. The definitions shadow each other, and so str.maketrans only actually refers to the last definition. (Which would presumably be the implementation in a non-stub file.)

But to a type-checker, str.maketrans actually refers to all of the overload definitions (plus any implementation) collectively. And the explicit specialization is only valid for the first overload.

So I think there’s only two possible interpretations:

  • Disallow explicit specialization for all overloaded functions, full stop.
  • Only allow the specialization if every overload is generic, and the specialization is consistent with each of them. (Arity matches, taking into account any typevar defaults; each specialized type satisfies any bounds/constraint of the respective typevars)
1 Like

Your example is from a .pyi stub, so there is no implementation, and no runtime.

But in case of an inline overloaded function definition in a .py, it is not difficult to obtain the type parameters of the individual overload signatures:

>>> from typing import Any, get_overloads, overload
>>> @overload
... def f[T](x: T, /) -> T: ...
... @overload
... def f() -> None: ...
... def f(x: object = None, /) -> Any:
...     return x
...     
>>> get_overloads(f)[0].__type_params__
(T,)

I think there is a third: If you provide an explicit specialization, only overloads that have a compatible parameter list can be matched, so it restricts the list of available overloads to whichever could be valid. Type checkers should then ofcourse provide good error messages. (i.e. in the maketrans example you quoted, maketrans[int]({'a':0}) would be valid, but maketrans[int]('a', 'b') wouldn’t be).

But I am not completely sure this is something that needs to be solved in the first iteration of this concept: Make it a static-type-checker error for now to use it on any overloaded function, and ofcourse allow it at runtime. Then changing the behavior to whatever turns out to be most useful is a type-spec-only change that can be done with just a little coordination between the type checker authors (as is provided by the typing council).

1 Like

Will this work with context managers too? It’s not mentioned in the PEP.

There is no simple way to annotate the a in with asd() as a: right now.

Does this not work under the PEP?

@contextmanager
def asd[T]() -> Generator[T]:
  ...
  yield some_t
  ...

@samwgoldman brought up a potentially tricky interaction between this PEP and the typing rules for TypeVarTuples

The typing spec has a restriction that “only a single type var tuple may appear in a type parameter list” to avoid confusion about which TypeVarTuple a particular type argument should match with.

This appears to only apply for subscriptable generic types like classes or type aliases. Functions aren’t currently subscriptable, so using multiple type var tuples works in both Pyright and Mypy.

If we made functions subscriptable, existing functions that are parameterized by multiple TypeVarTuples can no longer typecheck under the current rules.

1 Like

If we made functions subscriptable, existing functions that are parameterized by multiple TypeVarTuples can no longer typecheck under the current rules.

A possible solution is to disallow subscripting such functions. That is, a function can only be explicitly parametrized if it has at most one variadic type parameter.

3 Likes

The question is how do you call it? (Specify T)

While working on Pyrefly, I noticed another case that might be tricky with this PEP. Mypy will synthesize a generic function when accessing a class method from a generic class.

class C[T]:
    @classmethod
    def classm[U](cls, x: T, y: U) -> None:
        pass

    @staticmethod
    def staticm[U](x: T, y: U) -> None:
        pass

C[int].classm[str](0, "")   # OK
C.classm[int, str](0, "")   # Error: too many targs
C.classm[int](0, "")       # OK? Require targs to C, instantiate T=Any? Is the function still generic?

C[int].staticm[str](0, "")  # OK
C.staticm[int, str](0, "")  # Error: too many targs
C.staticm[int](0, "")       # OK? Require targs to C, instantiate T=Any? Is the function still generic?

I suggest that the type argument provided by subscripting functions must match the number of type parameters appearing on that function. In other words, the syntactical introduction and elimination forms should line up.

There’s a question about what to do when the class does not have explicit type arguments, but the method does. Currently, only mypy and pyrefly implement the implicit generic class method logic, as far as I can tell, so maybe we can just find some agreement between us?

3 Likes

This PEP would be great if accepted, it is a natural extension for the generic typing in classes:

my_empty_list = list[int]()

def empty_container[T]() -> list[T]:  
   return list()

my_empty_container = empty_container[int]() # feels natural but doesn't work ...

But where it excel the most it is in generators, more specifically in consumer generators with send, as they are imposible to type properly.

The most naive example is example is generator equivalent to reduce(operator.add, iter) meaning a generator that sums if int / float, broadcast if np.array and concatenates if str or list:

from typing import Protocol

class SupportsAdd[T](Protocol):
    def __add__(self, other: T) -> T: ...

def gen_sum[T: SupportsAdd]() -> Generator[T, T]:
    # yield None raises issue in mypy but is never used, so don't do T | None
    s = yield None # type: ignore 
    while True:
        s += yield s

# Ideally with this pep we could do:
gen1 = gen_opadd[float]()
next(gen1)
# reveal_type(gen1) = Generator[float, float] # but doesn't work
# reveal_type(gen1.send(1.2)) = float # but doesn't work

gen2 = gen_opadd[str]()
next(gen2)
# reveal_type(gen2.send("2")) = str # but doesn't work

Other example that is not working and maybe doesn’t even need the pep:

def raw[T]() -> Generator[T, T]:
    last = yield None # type: ignore (Annoying but neccesary till now)
    while True:
        # do smthg like print or whatever
        last = yield last

raw_procces = raw()
next(raw_procces) # prime generator
b = raw_procces.send(1) # reveal_type -> Unknown , :( when is clearly int 

# but at least with the pep we could save the day  partially for cases where send has always the same type.
raw_procces = raw[int]()
next(raw_procces) # prime generator
b = raw_procces.send(1) # not working currently
# and maybe even at each call (though i don't expect this to be efficient)
b = raw_procces.send[str]("t") # ?? maybe

I hope this illustrate how useful this syntax would be in the generator world, of course as you can expect everything applies in the same way for AsyncGenerators.

I only wrote naive examples that are not that useful and where the type of the send and yield did not change, but i think the idea is clear, these generators have been a bit overlooked, but would benefit massively from this pep.

Also I’m looking for ideas for how to solve in a elegant manner the # type: ignore in the yield None, so if anyone knows, pls tell me.

1 Like

I’m not sure why you would want to deal with the yield None and next(gen1) hack.
We don’t need anything special or complex to implement this:

gen1 = GenSum(float)
reveal_type(gen1)  # GenSum[float]
reveal_type(gen1.send(1.2))  # float

gen2 = GenSum(str)
reveal_type(gen2)  # GenSum[str]
reveal_type(gen2.send("t"))  # str

basedpyright.com link

Thanks for the example! of course there is no question that it can be implemented using a class, the whole point of this pep is avoiding it :sweat_smile: we could implement all functions as classes with callable.

The yield None, is kind of standard in this type of generators with send, is just the typehinting what would be nice to handle.

It should be like any function subscription, there’s no reason it should be any different?

with asd[int]() as a:

Am I missing something?

1 Like

Thanks everyone for this round of feedback. I’ll update the PEP with a couple more of these examples. Hoping to get this all through before the 3.15 feature freeze.

3 Likes

These are Great news! count with me if you need anything in this regard

+1, this addresses a real inconsistency that’s been bothering me.

The asymmetry between generic classes and generic functions has always felt arbitrary. I can write list[int]() but not make_list[int]() for a generic function? Having to use intermediate variables or type annotations for every generic function call gets verbose fast, especially when the return type is complex.

The symmetry argument alone is compelling. People expect functions to work like classes when it comes to generics, and explaining why they don’t always feels like defending a historical accident rather than a design choice.

The factory callable example resonates particularly well. There are legitimate cases where type checkers genuinely can’t infer the right type, and providing explicit type arguments is cleaner than workarounds with cast() or extra variables.

Looking forward to seeing this land.

4 Likes