The structural type of typing.Callable

Typing
1

I originally asked this question in python/typing, but I thought it would be useful to revisit it here.

The structure of typing.Callable seems a bit of a gray area at the moment. It is defined as a special-form in typeshed, so I assume that all type checkers have their own internal definition somewhere.

In particular, I would like to understand if a value of type typing.Callable

  1. has a __call__ method,
  2. has any attributes beyond the ones available on object.

It looks like pyright and mypy currently do not allow __call__ lookups on a typing.Callable, whereas pytype does:

def f(fn: Callable[[], int]):
    reveal_type(fn.__call__)

All three type checkers pass the following snippet, even though it’s a runtime type error:

def f(g: Callable[..., None]) -> str:
  return g.__qualname__

class A:
  def __call__(self) -> None:
    ...

f(A())  # AttributeError: 'A' object has no attribute '__qualname__'
4 Likes
2

A callable should probably be something that can be called and nothing else should be assumed.

Separately, I think it makes sense to have a “subtype” (in the structural sense) of Callable for functions, which then has all the function attributes:

  • __name__
  • __qualname__
  • (__annotations__?)

This would additionally help with determining when a callable is bound as a method. (Functions defined within a class are bound as methods, but other callables are not bound as methods.)

from typing import Function

def f(func: Function[[int, str], bool]) -> None:
  print(func.__qualname__)  # OK
1 Like
3

Yes, I think it’s good to get more clarity here.

One approach could be to specify that Callable is exactly equivalent to a Protocol with only a __call__ method; i.e., the following two are the same type:

X1 = Callable[[int, str], bool]

class X2(Protocol):
    def __call__(self, x: int, y: str, /) -> bool: ...

As @tmk said, it could then be useful to have some way to express a function type, which has a few more attributes.

4 Likes
4

I think it’s always safe to access any object attribute/method on a Callable, e.g. __module__.

5

For context, pyright currently uses the “fake” (type_check_only) function class defined in builtins.pyi for all callable types. I copied that behavior from mypy, which explains why they both type checkers allow access to attributes __qualname__, etc.

I welcome additional clarity in the spec about the behavior of typing.Callable and callables in general.

6

One approach could be to specify that Callable is exactly equivalent to a Protocol with only a __call__ method

I like this. The protocol has a __call__ method and inherits all of object’s attributes, which matches my intuition for what a callable thing should be. I guess there’s a question of how many users are currently depending on Callable having attributes like __qualname__.

it could then be useful to have some way to express a function type, which has a few more attributes.

Maybe types.FunctionType could be used for this?

7

I occasionally access fields like qualname. There’s a problem with current definition of FunctionType. It’s not generic. I can’t do FunctionType[P, R] for paramspec P and typevar R (or concrete types).

If functiontype was generic then I’d be fine adjusting some callable annotations to it.

8

Making FunctionType generic as you suggest makes sense. I think we may have tried it in typeshed earlier and run into some practical issues.

types.FunctionType is the type of functions created through def at runtime, so under the principle that typing should reflect the runtime behavior, ideally type checkers should infer this type for def functions.

A complication, though, is that builtin functions (including ones created in extensions) are not instances of FunctionType, but of types.BuiltinFunctionType. However, stubs just use def for those, so type checkers can’t tell that these functions are instances of BuiltinFunctionType, not FunctionType.

Does that difference matter? Usually not, but it may matter if users are accessing attributes that exist only on one of the two.

9

Some previous discussion: Why is Callable assumed to be a user defined function? · Issue #14392 · python/mypy (github.com)

One major issue with stripping attributes from Callable and using types.FunctionType (or some other nominal type) is how type-checkers currently infer plain functions (def) - which are just Callable. It seems like type checkers would need to rework what is inferred so that the following doesn’t issue false positives:

import types

def my_func() -> None: ...

def accepts_callback[**P, R](f: types.FunctionType[P, R]) -> types.FunctionType[P, R]:
    return f

accepts_callback(my_func)  # currently a typing error

But, if types.FunctionType is inferred, this doesn’t work in cases where Python source code is used to build extension modules. For example,

  • def functions in mypy or other modules compiled by mypyc are <built-in function>s
  • def functions compiled by Cython in pure python mode are <cyfunction>s

There might be some good heuristic, like avoiding inferring types.FunctionType if (1) a compiled binary with the same name exists as a Python source file or (2) if a Python source file can’t be found at all (and the type definition is taken purely from a .pyi); this could also solve issues with functions originating from Python built-in modules.

Otherwise, I agree with this,

a structural “function” type stripped down to a few commonly-accessed attributes (I frequently use __qualname__ and __module__ for introspection) seems like a simpler first step while people are changing their annotations to work with functions.

10

Is this indeed true? Callable only supports positional-only parameters, so I would assume any type checker will need a distinct type to model def-defined functions and lambdas.

11

Ah - I was only thinking of mypy, which has non-public API in mypy_extensions that you use to build a Callable type with parameters other than positional-only. Maybe other type-checkers don’t do this, but I would expect them to also use a structural type, not a nominal type.

A quick experiment indicates that pyre does something similar to mypy, though (pyre playground).

12

Another question with Callable is how does it behave as a descriptor. Can ClassVar[Callable]] become a bound method? If so should it be an error to assign an already bound method?

13

I think that’s a totally separate idea. I proposed a PartialApplication type form to solve many problems including accommodating the descriptor protocol, described here.

14

Speaking of bound methods and equivalence of Callable and callback protocols, that’s another thing we should iron out in the spec. See also this mypy draft PR

15

Couldn’t we consider something like SupportsCall for _typesheed?

from collections.abc import Callable
from typing import Protocol


class SupportsCall[**P, R](Protocol):
    __call__: Callable[P, R]