So, I think what you’re claiming is that non-function callables are sufficiently rare that we can just treat all generic callables as functions and allow subscription? That doesn’t really match my experience. I constantly work with callables where I’m not sure whether it’s a function or not. Just as a random example with jax:
>>> import jax
>>> @jax.jit
... def f(x):
... return x * 2
...
>>> from types import FunctionType
>>> isinstance(f, FunctionType)
False
I’m sure I could fine more.
I believe you’re misunderstanding Michael here. The idea is to let these sorts of wrappers do manual forwarding as needing. functools.wraps() should be updated to forward __getitem__(), and library code that implements something like it should be updated to match. In your jax example, that would mean jax updating jax._src.util.wraps() to forward __getitem__(). The argument (as I understand it) is that this sort of wrapping is already something that takes significant consideration, and it’s not unreasonable to ask library authors to take a bit more consideration to handle this forwarding.
Just want to remark that there are legitimate use-cases for that, and the type-variable can be resolved if outer context is given:
class nn_Module: ...
def instantiate_from_json[T](arg: dict) -> T:
raise NotImplementedError
m: nn_Module = instantiate_from_json({"_target_" : "module.klass"})
But that just underlines the point that these solvability[1] issues should get raised at the call-site, not at the defintion.
And yes that totally is a word, mathematicians use it all the time 
↩︎
I’d argue that the correct type here would be Any, if this result is what you want.
One concern that I have, and did not see it addressed in neither the PEP or the thread, is the ambiguity created with:
class C:
def __call__[T](t: T) -> T:
return t
def __getitem__(i):
return i
c = C()
c[int](3)
I would assume that this can be expanded to:
c.__getitem__(int)(3)
and not:
c.__call__[int](3)
as that would be a breaking change, but I think this needs to be specified.
Additionally, it does mean that if this PEP is accepted, adding __getitem__ to a class that already implements a generic __call__ would be a breaking change, right?
You’re talking about arbitrary user-defined callable objects, but this spec is explicitly about callable functions, so it seems out of scope to me.
C is a user-defined callable object, but C.__call__ is a function. So I’m wondering about the interaction between the two if this is accepted.
No, because (as has been discussed extensively though you are excused for not having read 145 posts) the [int] syntax will only work on functions and not on callable objects. So, if a class with a generic __call__ does not implement __getitem__ then c[int]() will fail with a runtime error. And if the class does implement __getitem__, then c[int]() means you call whatever __getitem__ returns.
There’s a PR almost ready[1] that updates the PEP to take almost all of the feedback into account. You can already preview the rendered output.
Waiting for a CLA signing ↩︎
The new version of this PEP leaves some important pushback from @Jelle and @carljm incompletely answered.
The problem is that only actual function objects support subscription, and a decorator whose signature is
def deco(func: Callable[[int], int]) -> Callable[[int], int]: ...
does not say whether it returns a function object – only that it returns a callable object. And not all callables are subscriptable – in particular, objects with a __call__ method aren’t, unless their author also added a __getitem__ method (and if they did so before this PEP was accepted, it would surely do something unrelated).
The new version answers this by saying “yeah, we know, it’ll just be a runtime error, no big deal”. But Jelle and Carl don’t like this answer much.
A partial solution would be to make your decorator generic, like this:
def deco[F: Callable](func: F) -> F: ...
This allows the type checker to realize that if the argument is a types.FunctionType instance (which all objects created by def and lambda are), the returned object is also such an instance, and peace returns to the kingdom.
But if the return type preserves the “function-ness” but not the exact signature, or if it takes any callable and wraps it with an actual function (def), this hack won’t work.
Jelle’s proposal is to make a change to typeshed, where the actual specification for FunctionType lives, to make it generic, so you can write things like this:
def deco[**P, R](func: Callable[P, R]) -> FunctionType[P, R]:
def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
print("Calling", func.__name__)
return func(*args, **kwargs)
return wrapper
(A small change to type checkers will be needed for this to work.)
We can then see FunctionType as a subtype of Callable that adds a bunch of additional dunder attributes, rather than a stand-alone object definition that happens to define __call__ (and adding __getitem__ per the new PEP version).
Obviously the same treatment should be applied to all function types supported by the PEP (FunctionType, MethodType, BuiltinFunctionType, BuiltinMethodType, and MethodWrapperType, all in types).
Thoughts? @Jelle, did I capture your idea correctly?
That’s one solution to the problem, but wouldn’t it be nicer if we had a minimal version of intersection types (one which is forward-compatible with a full version) such that we could write
def deco[**P, R](func: Callable[P, R]) -> Callable[P, R] & types.FunctionType:
def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
print("Calling", func.__name__)
return func(*args, **kwargs)
return wrapper
?
The only thing that a minimal version of intersection types would need in order to support this, is the intersection of a concrete type (FunctionType) and a protocol (Callable), which is, I believe, basically the least problematic part of intersection types.
The above already works with ty: Playground | ty
Such a type would not have the same semantics, because FunctionType currently has an Any-typed __call__. Thus this creates a gradual intersection, which would allow more calls.
Generally I don’t think limited intersections are worth it. We need to think through the whole behavior, and limiting to smaller areas doesn’t make those cases go away.
The new version of the PEP is merged. You can now read PEP 718 on the official peps site.