Here’s another underspecified area that the PEP should clarify (credit to @decorator-factory who brought up some related cases on Discord).
The PEP says
Type checkers should support subscripting functions and understand that the parameters passed to the function subscription should follow the same rules as a generic callable class.
But what if you have a Callable? Should type checkers accept:
T = TypeVar("T")
def f(x: Callable[[T], T]):
x[int](1)
This would work at runtime if the x is actually a function object at runtime, but not necessarily if it’s some other kind of callable.
This can also come up easily with decorators:
def deco[**P, T](func: Callable[P, T]) -> Callable[P, T]:
def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
return func(*args, **kwargs)
return wrapper
@deco
def f[T](x: T) -> T: return x
f[int](1) # allowed?
There’s a few possible answers:
- Subscription only works on function objects (types.FunctionType instances), and not necessarily on other callables. So type checkers should reject subscription if they can’t prove that the callable is actually a function. But this would break reasonable use cases like the above decorator.
- Type checkers allow subscription on any callable. But arbitrary callables don’t actually necessarily support subscription at runtime, so this is unsafe.
The PEP doesn’t specify any changes to pre-existing Callable instances; The rationale section specifies that “Function objects in this PEP is used to refer to FunctionType, MethodType, BuiltinFunctionType, BuiltinMethodType and `MethodWrapperType
This is an interesting point, but looks more of a problem of Callables than this pep as typing using subscription (of callables) is already working for constructors such as “list” “set” “tuple”. So your issue can be rise in current 3.14 python for deco(list)[int], deco(tuple)[int] etc.
For me that just means that we should type your decorator as accepting callableAndSubscriptable instead of Callable, but i think that is something we can consider independently of this pep.
I think the problems you illustrate are a result of 2 things. One, the subscript types of a callable can’t be statically specified outside the callable’s definition. Two, that subscription for functions with multiple subscript types haven’t been specified by the PEP.Consider the following code snippet.
def f[T](x: Callable[[T], T]) -> T:
x[int](1)
def simple_example[M, N](x: tuple[M, N]) → tuple[M, N]:
return x
f(simple_example) # What should happen here?
In terms of types, f here can only specify that its x argument has the type of a unary callable, but it cannot specify that x is a unary callable with only 1 subscript type.Since simple_example is a unary callable (with 2 subscript types), f(simple_example) would be a false negative if the current PEP were implemented.
This could be solved with intersection types, right?
from types import FunctionType
def deco[**P, T](func: Callable[P, T]) -> Callable[P, T] & FunctionType:
Though it’d be unfortunate if every decorator had to be written like that. Maybe a type alias like
type Function [**P, T] = Callable[P, T] & FunctionType
could help.
No, because the subscript types for the func argument aren’t statically accessible. To be more specific, there is no way to specify (using types only) that the deco function takes a callable func and returns a callable with the same list of subscript types as func. Intersection types alone wouldn’t be able to help specify that.
Hmm, but pyright seems to know here that f is generic:
from collections.abc import Callable
from typing import reveal_type
def deco[**P, T](func: Callable[P, T]) -> Callable[P, T]:
def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
return func(*args, **kwargs)
return wrapper
@deco
def f[T](x: T) -> T: return x
x = f(1)
reveal_type(x) # `int`
y = f("foo")
reveal_type(y) # `str`
So pyright should also know that it’s subscriptable, no?
Though I’m unsure; maybe this is just special treatment of decorators by pyright. I cannot really see how pyright knows that f is generic.
EDIT: just confirmed that mypy has the same behavior on the above code
I don’t want to make it doable on all arbitrary Callables because it’s unsound and there’s a reasonable way to solve this problem, I think the ideal solution is to make FunctionType generic Making functions/methods subscriptable · Issue #1410 · python/typing and then let developers chose
One advantage of Callable[[int, bool], str] & FunctionType over FunctionType[[int, bool], str] is that if Callable ever gets a new syntax, as was proposed in PEP 677 – Callable Type Syntax | peps.python.org , then the former option might become more expressive (with named arguments, etc.)
I somewhat agree but there’s nothing stopping adding a def keyword to PEP 677 to do FunctionType as well.
Intersections still have a lot of work that needs doing before I’d feel like I was proposing them responsibly. I believe it’s better to make the language more consistent and allow subscripting of any callable at runtime, than to assume the remaining work on intersections will reach a point of agreement, acceptance, and implementation by typecheckers in a timeframe that does not block progress on subscriptable functions.
For this to happen, it just requires that both methods and functions are always subscriptable. We have benefits to this for consistency in user expression whether or not other kinds of expressibility can be improved on with intersections later.
I think that PEP 718 may be a solution in search of a problem. The use cases it addresses such as empty containers, factory callables, and ambiguous type inference and more can already be handled with existing mechanisms like type annotations (x: list[int] = make_list()) or typing.cast, which are more explicit about their intent. The proposal adds significant runtime overhead (benchmarks show 2x slowness compared to cast) for marginal ergonomic benefit, while several motivating examples demonstrate functions that cannot even be implemented in a type-safe way (like foo[T](x: Sequence[T] | T) where the runtime can’t actually distinguish which branch to take).
More fundamentally, I think this treats symptoms rather than causes (if type inference is failing, the answer should be improving type checkers, not adding new syntax that makes Python more complex).
The thing is that this syntax already exist for constructors, x = list[int]() so if we are concern about slowness we can just avoid using it, but when the concern is not the marginal performance, but safety, we are forced to do a class with callable instead of just a function, which is even slower.
class make_list[T]:
def __call__(self, it) -> list[T]:
return list(it)
m = make_list[int]()
PEP 718 is giving consistency to the language in the way one expects, is later up to anyone to decide wether to use it or not.
I was initially generally positive on this proposal, but after learning that Swift also doesn’t have the ability to explicitly specialize generic functions (there have been several proposals in the past but they quickly petered out), I’m thinking that it’s maybe not needed after all. And in my experience, a type annotation on the return value is usually enough to solve the problem in Python (more rarely you need to add a type annotation on an input parameter).
Rust does have a syntax for this, the turbofish, but in my experience it’s usually only needed in cases where the type is actually used inside the function (i.e., what the PEP calls reification, IIUC):
assert_eq!(4, size_of::<i32>());
But the PEP is explicitly not proposing reification, so that use case is out.
Currently, in Python you would just pass the type as a normal function argument instead:
def size_of(typ: type) -> int: ...
This pattern also works for the make_list function mentioned in the PEP:
from typing_extensions import TypeForm
def make_list[T](*args: T, typ: TypeForm[T] | None = None) -> list[T]: ...
reveal_type(make_list(typ=int)) # list[int]
reveal_type(make_list(1, 2, 3)) # list[int]
EDIT: pyright can even handle this:
make_list_int = partial(make_list, typ=int)
reveal_type(make_list_int())
Very interesting, it seems that PEP747 and PEP718 compete in functionality, but for me there is a clear winner.
Let’s check a example that is just a bit more complex with a nested list.
# Constructor that already works and widely used
a = list[list[int]]() # list[list[int]]
# Making your own constructor
class my_list0[T]:
def __call__(self, it) -> list[T]: return list(it)
b = my_list0[my_list0[int]]() # list[list[int]]
# PEP 747 TypeForm, already working with typing_extensions
from typing_extensions import TypeForm
def my_list1[T](it, typ: TypeForm[T]) -> list[T]: return list(it)
c = my_list1(None, typ=type(my_list1(None, typ=int))) # list[list[int]]
# PEP 718, if approved
def my_list2[T](it) -> list[T]: return list(it)
d = my_list2[my_list2[int]]() # list[list[int]] if PEP 718 approved
Also the TypeForm trick is not a good solution for generic returns.
def test() -> mylist(None, type=int): ... # won't work
def test() -> mylist[int]: ... # no problem, though probably rarely used
To be clear, TypeForm may still has its use case beyond this one, but I think that for this specific use case TypeForm is not great:
- Mixes kwargs with typeargs, which can only cause problems
- Introduces a new semi standard argument name “typ” that is subject to be named inconsistently across different code bases
- Makes function signatures large and hard to read.
- Requires knowing and understanding
TypeForm[T]and the difference withType[T], and is not inmediately obvious - Doesn’t work in return types cause requires calling the function
- Doesn’t compose nicely as it it very verbose
my_list1(None, typ=type(my_list1(None, typ=int)))
The typeform trick also doesn’t work well for existing codebases, especially in the case of composition.
Subscriptable functions are a significant improvement for generic functions that return generic functions. While some type checkers apply some special casing for decorator factories, they don’t handle bounds on potential inference properly in many other contexts, and do require more information.
There’s an argument that typecheckers should just be smarter and track appropriate lower and upper type bounds, rather than synthesizing a solution and binding to it eagerly, and I would personally strongly agree with that, but no typechecker currently behaves that way.
Even though I believe typecheckers should be smarter in cases like this, there are also some issues with the current type system only checking certain things at function boundaries that hampers potential inference of generics by destroying relevent context.
You can achieve the same syntax if PEP 747’s TypeForm is accepted (pyright demo):
from typing_extensions import TypeForm, Any
class ListMaker[T = Any]:
def __call__(self, /, *args: T) -> list[T]: ...
def __getitem__[U](self, /, t: TypeForm[U]) -> ListMaker[U]:
return self # type: ignore
make_list = ListMaker()
reveal_type(make_list[int]()) # list[int]
The only difference is some extra ceremony in defining a function definition as a __call__ instance method and making an instance of the callable.
IMO, using TypeForm to do this is more transparent, more immediately flexible (you can define callback protocols which have __getitem__-support, bypassing the issue of arbitrary Callables not being subscriptable) and less intrusive (PEP 718 seems to imply that all function definitions, including dunder methods, will become runtime-subcriptable regardless of static typing application my_function.__getitem__[“!@#$”](), and eventually some library will rely on such behaviour).
I also think having this PEP accepted would generally lead to harder-to-read code. Subscriptable callables currently are restricted to class constructor expressions, and you know that they go through __class_getitem__ at runtime. Having types being routed through __getitem__ without TypeForm makes subscripted functions look the same as mappings which delegate callables.
That’s a really interesting point and it might be worth digging deeper into why this happens in the first place. You may be right that the language itself should provide more help here, but it would be useful to explore that more carefully and understand when this kind of feature is genuinely needed versus when smarter type checkers or better patterns could solve the same problem.
I’ve found it to be mostly useful for passing generic const type arguments. For example in the xsf crate (github, docs.rs) the Bernoulli numbers can be calculated as follows:
use xsf::bernoulli;
assert_eq!(bernoulli::<0>(), []);
assert_eq!(bernoulli::<1>(), [1.0]);
assert_eq!(bernoulli::<2>(), [1.0, -0.5]);
assert_eq!(bernoulli::<4>(), [1.0, -0.5, 1.0 / 6.0, 0.0]);
The signature is pub fn bernoulli<const N: usize>() -> [f64; N], and the const generic type parameter makes it possible to know the size of the returned array at compile-time.
But of course, in Python there are no const generics. However, if we ever were to add them, then we wouldn’t be able to use them for functions like this, because that would require subscript-able functions. An important use-case would be shape-typing, i.e. statically annotating the rank of multidimensional arrays and tensors (and perhaps even the lengths of individual axes).
So even though the number use-cases might be limited at the moment, having subscriptable generic functions will open up the doors to other, arguably more interesting, new features like const generics.