PEP 718: subscriptable functions

PEPs
,
82

This may be out of scope for this PEP, but has the use case of specializing generic functions been discussed? I have a generic implementation that takes various type variables, but parts of the application only use a fixed value for that type variable and I want to save users from having to know the underlying implementation is generic. I essentially want an alias for the function but with some type vars clamped.
Example

def mygenericfun[A, B](a: A, b:B) -> A: ...

## Specialization (desired)
myfun[B] = mygenericfun[str, B]

# Users import `myfun` not `mygenericfun`
### LSPs/type checkers/linters now can show a signature for myfun as myfun[B](a: str, B) -> str ```

Currently this possible with possible with classes through inheritance and to some degree with type aliases. AFAIK for functions the only solution is to write a full wrapper.

EDIT: Added an example here

2 Likes
83

Great observation, I didn’t realize that this was a thing. But that function[TypeVar] returns a GenericAlias I think it is already part of this PEP because it is how it works in existing constructors:

my_list = list[int]
a = my_list([]) #  a: list[int]
type(my_list) # types.GenericAlias

We can also create a partial generic of dict:

my_dict = dict[str] # works but some lints complain, dict[str,Any] better
a = my_dict()  #  a: dict[str, Unknown]. or  dict[str, Any]
type(my_dict) # types.GenericAlias

Your “generic specialization” is also possible and already allowed but the TypeVar B as of now needs to be defined beforehand:

from typing import TypeVar

B = TypeVar("B")
my_specialized_dict = dict[str, B]
my_very_specialized_dict = my_specialized_dict[int]

a = my_very_specialized_dict() # a: dict[str, int]

Are you proposing to allow defining TypeVars in assignations?

my_specialized_dict[B] = dict[str, B]

For me this looks slightly odd.

84

It works to create an alias, but ideally there is a way to create something callable too

I don’t know what the syntax should be. But I think specializing a function would be nice. The workaround is to create classes and abuse __new__, and use pyright since mypy is very opinionated on what __new__ is allowed to do. This works today

class my_generic_function[A, B]:
    def __new__(cls, a: A, b: B) -> tuple[A, B]:
        return a,b

class my_specialized_function[B](my_generic_function[str, B]): ...

my_specialized_function('hello', 1) # OK
my_specialized_function(1, 1)  # Pyright correctly complains

but maybe there are more idiomatic solutions.

Is it ok to use async __new__?
85

what is wrong with the example i gave you? (the one of my_specialized_dict) of course it will only work when PEP718 is approved or using any of the workarounds you can read in this thread: callable class or typeform.

86

what is wrong with the example i gave you?

Nothing. You are right. If after this PEP regular functions behave like list/dict then we can specialize generic functions to domains.

87

I had not considered using a traditional TypeVar. Given that this PEP would enable support for specializing functions, I think this use case should be added explicitly to the PEP. This is not something “smarter linters” could provide.

88

I also think it is interesting to document this already @Gobot1234, plus possibly the usage in generators. and maybe a discussion on the alternatives that were found.

I think for this usage to works we only depends on the linter knowing subscription on functions will return a GenericAlias so I think it would probably be straight forward.

1 Like
89

Hi! all, I will be helping coauthoring the last changes on this PEP as @Gobot1234 currently has little time to finish it.

I have already some examples to motivate this PEP (the ones affect my use-cases), but just for completeness, if anyone has more examples, both easy and advanced, or examples that affect important libraries please let me know.

Special mention to @rsdenijs, @jorenham, @mikeshardmind as you looked somewhat interested on this.

Besides that any help is welcomed as I didn’t participate before in writing a PEP.

2 Likes
90

@PabloRuizCuevas I want to thank you for the help here :slight_smile:

I think one of the most important ones that I have missed is the ability to skip a cast where you know an ABC ahead of time (this example is from discord.py and was something I found myself trying quite naturally

    channel = guild.get_channel[discord.TextChannel](1132731894954475561)
    message = await channel.fetch_message(1430262684795928798)
3 Likes
91

Sorry to drag out the discussion more here, but I’ve been thinking about this PEP in connection with this recent idea (basically, a DSL for typing): Idea: Simpler and More Expressive Type Annotations and the newly accepted PEP on lazy imports. For both it would be great if the type parameter to a function call could be treated as special by the compiler. This would require specialized syntax and couldn’t just rely on __getitem__.

For the sake of this discussion, let’s assume this syntax (inspired by Rust):

def make_list[T](*args: T) -> list[T]: ...

lst = make_list::[int]()

Relevance to lazy imports

We could then write something like this

lazy from collections.abc import Sequence

list_of_sequences = make_list::[Sequence[int]]()

and the import can be skipped because we would have deferred evaluation for the type parameter.

Relevance to Imogen’s proposal

In Imogen’s proposal, tuple[int, str] can be written as (int, str) as long as it’s in a place where the compiler knows that only type annotations may appear. This would then allow

list_of_tuples = make_list::[(int, str)]()

which couldn’t be supported with the __getitem__ syntax because it can’t be treated differently by the compiler.

As a competitor to PEP 747

PEP 747 enables functions that take complex type expressions as runtime parameters. This PEP could fill the same role if the value of the type parameter could be made available in the function body at runtime. This would then allow writing function like this:

def checkcast[T](value: object) -> T:
    typ = T.__value__  # (or something like that)
    assert isinstance(value, typ)
    return value

which can be called like this

an_int = checkcast::[int](maybe_an_int)

I think this wouldn’t cover all the use cases of PEP 747, but a large majority of them.

1 Like
92

Maybe a simple example, let me know if this is useful.

# Generic function for creating email messages. Signatures with type variables and bounds can be intimidating to non-expert users. 
def create_email[T: Hashable, S: DestinationType](content: T,  destination: S) -> Message[T, S]: ...

# In practice, in one part of the code where we only create simple email messages. Thus we setup a GenericAlias.
create_regular_email =  create_email[str, str]

# In a different part of the code we always send rich emails to multiple senders with some custom types
create_rich_bulk_email =  create_email[RichString, EmailList]

# When operating in these domains, users see restricted signatures in the IDE and when validating the calls through the static type checker

create_email(content: str, destination: str) -> Message[str, str]
create_rich_bulk_email(content: RichString, destination: EmailList) -> Message[RichString, EmailList]

This use case can’t be solved with smarter linters, and already works today with classes. IMO functions deserve the same level of support.

1 Like
93

I really hope this makes it in. It solves a real pain point for some scenarios, and adds helpful consistency between class calls and function calls (both of which may invoke generics, but only class currently can specify the type variables).

18 Likes
94

@samwgoldman did you find the agreement, I’m now working on this part, and prefer to have your informed opinion.

For now I think it makes sense that is possible to define the type of the method (cls,static) even without defining the one of the class (letting it generic)

95

Type-annotations are already lazily evaluated in 3.14+, so some x: Sequence[...] works, even if I don’t (yet) have Sequence imported.

This might also be transferred to subscription of function, where the __getitem__ function of some function object (if that would exist at runtime) just returns the function itself. That would mean, that some x[...] = x (or a copy of x, with __getitem__ set to none now, so xcan't be subscripted twice). So ifx[…] == x, why not make x[…]not even callgetitem`. Because if we do that, any expression (with valid syntax) can be passed, and your concerns (and new syntax?!) are addressed.

Also, this would probably be worth a new topic, if you want to go after this idea further. It combines multiple ideas unrelated to this topic itself. Generally though, I find it unlikely that a new syntax is added for lazy evaluation of type hints (which can also be done with ForwardRefs). And the idea of evaluating the expression within [...] lazily, can work, if we make functions special in that case, which would mean that neither new syntax, nor new attributes or similar would be required, and existing code would still work.

96

I somehow completely missed this PEP until now. Just wanted to add that I am a strong supporter of this idea. There are many cases where type inference is ambiguous and type checkers are forced to rely on various heuristics (often inconsistent between type checkers).

An additional benefit is that this gives type-checkers more flexibility w.r.t. how to handle variables with and without annotations. For example, use traditional/rigid “variable“ semantics for variables with annotations, and use more pythonic/lenient “name“ semantics for variables without annotations. Currently this option is limited, because users are forced to add annotations in situations with ambiguous inference.

5 Likes
97

PEP 718 has been updated. The PEP now includes additional examples, justifications and binding rules.

We believe this PEP is complementary to the recently accepted TypeForm (PEP 747). TypeForm addresses cases where types need to be inspected or passed as values at runtime, while PEP 718 addresses a different problem: giving a consistent and ergonomic way to guide the type checker where inference is ambiguous or unsolvable:

# Already works for classes
list[int]()  # list[int]

# PEP 718: the same consistency for functions
make_list[int]()  # list[int]

Unlike TypeForm or callable class workarounds, PEP 718 works universally at runtime for all existing functions, including stdlib and third-party libraries — with no changes to their source code. Enabling full type inference for specific functions may still require typeshed stub updates, but that is significantly lighter work than rewriting functions as callable classes or extra arguments.

The proposal has received strong support from Guido van Rossum (sponsor) and Ivan Levkivskyi, and has been discussed extensively with no fundamental technical objections to the runtime mechanism itself.

We would appreciate the Typing Council’s recommendation on PEP 718.

9 Likes
98

Please open an issue on Issues · python/typing-council · GitHub to get the Typing Council’s recommendation.

99

I’m definitely for accepting this PEP. I feel like it makes Python more consistent and therefore better.

I feel like there is a small oversight in overloads that might need to be clarified. There is the example:

@overload
def serializer_for[T: str]() -> StringSerializer: ...
@overload
def serializer_for[T: list]() -> ListSerializer: ...

def serializer_for(): ...

At runtime serializer_for does now not have any type parameters. This is a problem, because the implementation is not actually subscript-able (unless you plan to make non-generic functions subscript-able, which I would oppose). I’m not sure there is a great solution for this. Type checkers can error here, but this is not specified in the spec. Should this get a bit more of an explanation in the PEP?

Is there a plan how the transition of builtin functions with function/method generics (e.g. def aiter(async_iterable: SupportsAiter[_SupportsAnextT_co], /) -> _SupportsAnextT_co: ...) works? I’m generally a bit worried that if typeshed (partially) migrates to this new syntax, which then seems usable from a type checker perspective, but is not actually, because CPython does not support subscription for that specific function yet. I think the hard part here is probably not the type checkers, but keeping typeshed and Python in sync.

I’m glad functions with multiple TypeVarTuples was mentioned, because that was a potential problem. I think now the only weird thing is that classes cannot have multiple TypeVarTuples and functions can. That is unfortunate, but I’m not sure we can easily get out of that.

3 Likes
100

I think we can address this by using (...) instead of () in the overloads and in the implementation, for both examples (serializer_for and make).

(I don’t think we need to alter the PEP text regarding multiple TypeVarTuples, and that’s fine with me.)

101

The PEP as written (along with the runtime reference implementation) makes all functions and builtin functions subscriptable.

Making only generic functions subscriptable is an interesting way to narrow the impact of the PEP, but it would require some mechanism to deal with functions in untyped code that are treated as generic for type checkers.