Proposal: more kinds of Callable types

Typing
1

Summary

I propose adding different variants of Callable to typing or other relevant parts of the stdlib, which use the same syntax as Callable but allow differentiating between different kinds of callables which may have different semantics.

Motivation

Callable is relatively broad, which means type checkers and authors have to make assumptions which makes it harder to write correct types for many cases where Callable is currently used. The exact semantics of Callable have also grown over time, sometimes in different ways across type checkers.

There have been a relatively large number of discussions where people generally agree that types which differentiate between the different kinds/variants of callable would solve the respective issue and otherwise be desirable. My hope is that this proposal can serve as a place to collect various peoples opinions and concerns.

In no particular order:

This list is far from exhaustive, since there are many issues raised against the various type checkers where the broadness of Callable has caused issues/confusion/type-unsafety. The last link, the issue by Glyph, iteslf contains a list of issues that are related to this.

Proposal

The core proposal is to introduce multiple new types which, for the most part, work like Callable. They differ in some specifics, which allows type checkers to be more strict while users can be more specific, and design harder to abuse APIs. Nothing here should change how Callable is treated, to maintain compatibility.

Part of the goal of this proposal is to collect ideas and concerns, instead of asking for a strict list of new types. However, i’ve come up with what i think would be a good starting point:

  • Function: a defined, named, function. Function and Lambda should never apply to the same type.
    • __name__: str (except <lambda>,)
    • __qualname__: str (except <lambda>)
    • __doc__: None | str
    • __annotations__: dict[str, type|types.GenericAlias]
  • Lambda: a anonymous function defined using lambda syntax.
    • __name__: Literal["<lambda>"]
    • /__qualname__: Literal["<lambda>"]
    • __doc__: Literal[None]
    • __annotations__: dict[typing.Never, typing.Never] (i’m unsure if Never is a good way to show “this dict cannot contain values”, since Literal[{}] is not an option)
  • UnboundMethod: a callable that takes self as the first parameter, but is not bound to an instance yet. Calling it requires passing self explicitly or turning it into a BoundMethod.
    • __get__: Callable[[SelfParam, type[SelfParam]], BoundMethod[OtherParams, Return]] (or a protocol/type corresponding to method-wrapper)
  • BoundMethod: a callable that is already bound to an instance, so calling it does not require passing the self parameter.
    • __func__: UnboundMethod[Params, Return]
    • __self__: typing.Self
  • ClassMethod: a callable that takes cls as the first parameter, and was defined on a class.
    • __self__: type[typing.Self] (unlike BoundMethod)

Potential Alternatives

Some of these can be approximated in the form of Protocols. This has been a relatively common solution across the various issues around Callable, but there are some things a Protocol can’t currently be used for. Even if a Protocol is sufficient, writing it requires in-depth knowledge about the way python deals with functions/methods.

Many of the Callable variants i proposed have equivalents in types (MethodType, FunctionType, LambdaType). Making these usable with type params like Callable has been suggested before but since there were apparently some issues implementing that, i chose to focus on new typing types instead.

2 Likes
PEP 718: subscriptable functions
2

Thank you for posting this! Great job on the thorough research for relevant comments and proposals!!

This recently came up in a PR I just did where we had to loosen some of the type annotations precisely because we don’t have a FunctionType. See here for the symptom and the unfortunate solution whereby the parameters P must be discarded.

I think your breakdown into Function, Lambda, UnboundMethod, and BoundMethod is a good start. However, the lack of Callable subtypes is most painful because there is no good descriptor binding (__get__). (See the above problem and solution.) Therefore, if we go this route, can we combine it with my breakdown: OrdinaryCallable, StaticMethod, ClassMethod?

Alternatively, all of this complexity is a special case of the more general tool PartialApplication with which we can build proper annotations for __get__ as well as annotations for functools.partial and its various specializations like jax.tree_util.Partial.

Of all the directions typing can go, for me this seems like one of the biggest holes with the biggest payoff for the things I’m looking at. :grinning_face:

Edit: oh wow, and this is your first post!! Welcome to the community :tada: Phenomenal first post!

1 Like
3

Thank you! Your mention here that PartialApplication can build proper annotations for __get__ finally helped it click for me. The __get__ of functions/methods means that just accessing the callable does a partial application over it, which is how methods get bound! wow! I even mentioned __get__ in UnboundMethod with that annoyingly complex type, but somehow it didn’t click for me that the reason this works is that python always goes through __get__, but the __get__ implementation only does the partial application when its actually needed (method/classmethod).

Tho i’m not quite sure what differentiates a StaticMethod from a OrdinaryCallable in your breakdown. It seems like StaticMethod never does a partial application, while OrdinaryCallable can. But i can’t think of a case in which a OrdinaryCallable (which i’m interpreting as module-level function or lambda) would become whats essentially a bound method, or why, if it can, a StaticMethod can’t.

4

This also came up in the PEP 718 discussion, since the proposed syntax would only work for actual function objects.

1 Like
5

The title confused my a bit at first, because In type theory, a “variant type” actually means something very specific (roughly speaking it’s the same as a sum type, disjoint union in set theory, or a coproduct in category theory). This proposal is about something else though, so maybe it’s worth updating the title to prevent others from also having a “huh?” moment :slight_smile: .

6

Exactly: the distinction becomes visible when the callable is stored in a class namespace.

A module-level function normally isn’t accessed through the descriptor protocol, so its binding behavior is invisible. But that same function (or lambda) will bind if assigned directly to a class attribute:

def f(first, value):
    return first, value

class C:
    ordinary = f
    static = staticmethod(f)

c = C()

c.ordinary(1)  # Equivalent to f(c, 1)
c.static(1)    # Equivalent to f(1): nothing was bound

Python effectively performs:

C.__dict__["ordinary"].__get__(c, C)
C.__dict__["static"].__get__(c, C)

A function’s __get__ conditionally binds the instance. The staticmethod wrapper’s __get__ deliberately returns the wrapped callable unchanged.

So the three descriptor policies are:

  • OrdinaryCallable: bind the instance on instance access, but bind nothing on class access.
  • StaticMethod: never bind anything.
  • ClassMethod: bind the owner class on both class and instance access.

This is why PartialApplication fits so naturally: it provides the missing type-level operation needed to express the removal of the bound leading parameter.

1 Like
7

Done, thanks!

Thank you, i hadn’t considered this case.

Hmm, this is basically the same as the concept i called UnboundMethod earlier, so the only distinction would be where it was originally defined. I think theres a bit of value in being able to differentiate “ordinary function that was assigned to a class attribute” but really the only usecase i can think of is to allow library authors to nudge users in the right direction.

I can see why, yea. Looking at that issue, it seems functools.partial already being handled was the only real argument against it. Maybe this usecase can provide some more motivation. Tho since its a few years old at this point, i think discussing alternatives still makes sense.

1 Like
8

100% agree. Putting yours and my ideas together, I think this is the right topology (push the expand table button to hopefully see it in one screen):

Object type Direct call __get__(None, C) __get__(c, C) Distinguishing members
FunctionType[P, R] Callable[P, R] Function[P, R] BoundMethod[type(c), P, R] See the type-shed.
StaticMethodType[P, R] Callable[P, R] Function[P, R] Function[P, R] __func__: Function[P, R], __wrapped__,__isabstractmethod__
ClassMethodType[P, R] No __call__ BoundMethod[type[C], P, R] BoundMethod[type[C], P, R] __func__: Function[P, R], __wrapped__, __isabstractmethod__
BoundMethodType[T, P, R] Callable[PartialApplication[P, tuple[T]], R] Self Self __func__: Function[P, R], __self__: T

Note that def and lambda both produce the same Function type, and binding a Function or ClassMethod both produce the same BoundMethod type. I believe this is consistent with Python’s semantics.

I used PartialApplication for clarity, but this topology doesn’t depend on it. If BoundMethod exists as a type in the typing library, then type checkers could implement the correct behavior without having to expose PartialApplication.

1 Like
9

Yeah, i think i broadly agree.

I do think theres quite a bit more value in being able to distinguish functions created by def and lambda, since the latter have various limitations that can make them undesireable. For example, a tool which uses the __annotations__ of a callback at runtime might want to only accept def defined functions, since lambda inherently can’t have annotations.

But this is a special enough case i think it’d be enough if type checkers allowed writing a custom protocol that doesn’t accept lambda (someting pyright in strict mode, at least, currently can’t do. It seems to allow lambdas in a protocol with __doc__: str even tho they can only ever have None there).

Though it’d be even better if it was possible to type that __annotations__ needs to be non-empty… thats getting into dependent types, which is quite offtopic here. It’d be nicer if __annotations__ was None when empty, like __doc__, instead of an empty dict, but alas.

10

As far as typing is concerned, there is almost no difference between functions created by def and by lambda:

x = lambda: None
def y(): pass
# x.__annotation__ = y.__annotations__ = {}
# x.__name__: Literal['<lambda>']
# y.__name__: str
# x.__doc__ = y.__doc__ = None

I’ve never need to distinguish these.

11

Overall, the things here would be welcome.

I’d suggest against this particular bit, just leave the type of __annotations__ as the type the data model prescribes here, not more restrictive in an attempt to model the empty version, there are no guaranteed keys in any case, and there are ways to augment a lambda that gets passed into a function that transforms functions.

12

One example i’m thinking of is a framework which makes heavy use of dependency injection by reading the annotations at runtime and passing the values to a function which the function “asked for” in its annotations. Lambda there is mostly just one case of “doesnt have __annotations__”,

Yeah, i think ultimately i’ve scope crept a bit with that. It’d be more like a theoretical HasAnnotations (or HasDocs) protocol a library might want to write, which is quite dissimilar from the distinct types of callable that have more clear boundaries.

13

Note that under 3.14+ you also have __annotate__ is None for both lambdas and unannotated functions. But this doesn’t mean there are no annotations as setting __annotations__ directly has the same effect.

>>> f = lambda x: x
>>> def g(): return x
...
>>> def h(x: int) -> int: return x
...
>>> f.__annotate__ is None
True
>>> g.__annotate__ is None
True
>>> h.__annotate__ is None
False
>>> h.__annotations__ = h.__annotations__
>>> h.__annotate__ is None
True

On the whole this seems like a good idea though, I definitely have some places where this would improve existing attempts at annotating functions.

14

I’m wary of adding more types that are basically protocol versions of specific standard library types. I think we should ensure the actual concrete types (types.FunctionType etc.) are well supported, and rely on general features like protocols and (in the future) intersections for the rest.

1 Like
15

How would you solve the problem I linked with protocols?

How do protocols help us annotate the output of staticmethod, classmethod, descriptor usage, or function decorators?

16

Sorry for speaking out of turn, but I’d go with something like this:

from typing import Callable, Concatenate, Protocol, Self, overload


class JitDeco[**P, R](Protocol):
    def __call__(self, /, *args: P.args, **kwargs: P.kwargs) -> R: ...

    @overload
    def __get__(self, obj: None, ower: type, /) -> Self: ...
    @overload
    def __get__[ObjT, **P1, R1](
        self: JitDeco[Concatenate[ObjT, P1], R1],
        obj: ObjT,
        owner: type | None = None,
        /,
    ) -> Callable[P1, R1]: ...


def jit[**P, R](fn: Callable[P, R], /) -> JitDeco[P, R]: ...

# ... elsewhere

class Foo:
    @jit
    def bar(self, x: int) -> int:
        return x * 2
    
reveal_type(Foo().bar(2))  # int

basedpyright playground

2 Likes
17

Very cool!!

Unfortunately, this doesn’t work with Pyrefly, which Jax uses, but I filed a bug.

18

I’ve had similar thoughts in the thread PEP 677 with an easier-to-parse and more expressive syntax but I’ve ultimately concluded that these problems are better solved with intersection types. The following already works today in ty:

from types import FunctionType
from collections.abc import Callable

from ty_extensions import Intersection

def f(func: Intersection[Callable[[int], str], FunctionType]) -> None:
    print(func.__name__)

class C:
    def __call__(self, x: int, /) -> str:
        return f"{x}"

f(C())  # err: type `C` is not assignable to element `FunctionType` of intersection `((int, /) -> str) & FunctionType`
f(lambda x: f"{x}")  # OK

Playground link

The advantage of this approach is that this can benefit from new syntax for Callables.

The downside of this approach is that intersection types aren’t a thing yet. That’s why I’ve been pushing for at least a simplified version of them: Would a reduced form of intersection types get into the spec faster? (this simplified form would be sufficient for the above example).

And as Jelle says, it’s not really sustainable to keep adding new special callable types. Callable types aren’t rare in Python. For example, I regularly work with Modules in PyTorch which are callable, and which can be annotated with the intersection type approach:

from types import FunctionType
from collections.abc import Callable
from typing import Self, Protocol

from ty_extensions import Intersection


class HasForward[**P, R](Protocol):
    def forward(self, *args: P.args, **kwargs: P.kwargs) -> R:
        pass

# You are expected to subclass `Module` and to then implement the `forward()` method
class Module:
    def __call__[**P, R](self: Intersection[Self, HasForward[P, R]], *args: P.args, **kwargs: P.kwargs) -> R:
        # (do some stuff here like keeping track of gradients)
        return self.forward(*args, **kwargs)

class MyModule(Module):
    def forward(self, x: int, /) -> float:
        return 4.0

type IntModule = Intersection[Module, Callable[[int], float]]

def f(m: IntModule) -> float:
    return m(3)

f(MyModule())

Playground link

This looks very complicated but that’s only because we need to forward the signature of forward() to __call__ (because PyTorch does it like that). Without that, it would be quite simple: an intersection of a nominal type and Callable.

19

I’m not sure I follow.

The four types I proposed are fundamental Python types. They likely already exist in every type checker within their implementations. I don’t think we’re “regularly” adding any new callable subtypes.

Even if you have intersections, you cannot annotate more complex decoration patterns easily (e.g., suppose you want to wrap classmethod with more functionality, then you ideally want f: Callable[[FunctionType[P, R]], ClassMethodType[P, R]]).

And while @jorenham’s code above is very cool, I feel like we might have had a simpler solution if we had access to the basic callable types. The type annotations are already hard enough to reason about without forcing people to annotate __get__ IMO.

20

I understand the concern, however, one of the main reasons i wrote this proposal to assume they’d be protocols or at least act like protocols (since protocols alone are not enough right now) versions is the following quote by you:

If types.FunctionType can be made to effectively work as a resolution to this issue, i’d be perfectly happy with that.

But if your answer is that every typing user who wants to properly type a method decorator should re-create some of the rather complex protocols i’ve seen be shown as partial solutions to this, i’ll have to respectfully disagree. That level of complexity means that most users just wont ever do it. It means that for any smaller script, i might end up with more typing code than executable code. That doesnt seem like a good solution to me.

2 Likes