(When) should we assume callable types are method descriptors?

Some common callable objects (functions, lambdas) are also bound-method descriptors. That is, they have a __get__ method which returns a bound-method object that binds the receiver instance to the first argument (and thus the bound-method object has a different signature, lacking the first argument).

Other callable objects (staticmethod objects, instances of classes with a __call__ method but no __get__ method) are not bound-method descriptors. If accessed as class attributes via an instance, they are simply themselves.

Both kinds of objects can inhabit the same Callable type. (For example, def f(x: int) -> int and staticmethod(f) both inhabit the type Callable[[int], int].) Normally, this would suggest that we should not assume that a Callable type is a bound-method descriptor, because not all of its inhabitants are.

But the bound-method descriptor behavior violates Liskov substitutability: it is not safe to substitute an object which is a bound-method descriptor for one that is not, or vice versa, even if they are both callable with exactly the same signature. When accessed as attributes, they will behave in incompatible ways.

So this is a soundness hole in the type system, both for subclassing in general (implementing __get__ on a subclass to return something that isn’t a subtype of Self makes it an unsound subclass, but no type checker errors on this) and specifically for Callable (a given Callable type includes both bound-method descriptor subtypes and non-bound-method-descriptor subtypes, and no matter which behavior we choose for Callable, some of those will become unsound subtypes.)

I suspect this underlying unsoundness is not worth fixing at this point; the disruption would outweigh the gain. But type checkers still need some answer to the question “should we consider a Callable type to be a bound-method descriptor?”

The simplest answer would be “Callable implies only __call__, not __get__ – assume Callable is not a bound-method descriptor”. Both ty and pyrefly (new in-development type checkers) currently implement this. But neither pyright nor mypy do. Instead, both implement a hybrid heuristic, where a Callable type as a class attribute is considered to be a bound-method descriptor if it is explicitly wrapped in ClassVar, and not otherwise.

This special treatment of ClassVar seems to be specific to Callable. Both mypy and pyright will happily model the descriptor protocol on other types assigned as class attributes but not explicitly marked with ClassVar.

This ClassVar heuristic doesn’t appear to have any particular correctness advantage: it is just as unsound (although in different scenarios) as “assume Callable is always a bound method descriptor” or “assume Callable is never a bound method descriptor”. One possible advantage is that it at least allows the annotation author to spell either possibility (albeit not in a particularly obvious way.)

(Mypy seems to also implement an additional special case, where a __call__ attribute annotated as Callable is always assumed to be a bound method descriptor, unlike other attributes. Pyright does not do this.)

As far as I can tell, the spec and conformance suite are currently silent on this question.

I would be interested to hear perspectives from typing users and developers of type checkers regarding which behavior is desirable here, and why. Is this something we should standardize in the typing spec?

In case the above isn’t clear, I’ve put together some sample code which illustrates many of the relevant scenarios (ty, pyrefly, mypy, pyright). The sample code is written to ensure that everything it does is correct at runtime, but due to the underlying unsoundness described above, every type-checker mis-understands the runtime behavior in some cases.

from typing import Callable, ClassVar, assert_type

class Descriptor:
    def __get__(self, instance: object, owner: type) -> int:
        return 1

def impl(self: "C", x: int) -> int:
    return x

class C:
    descriptor: Descriptor = Descriptor()
    classvar_descriptor: ClassVar[Descriptor] = Descriptor()

    callable: Callable[["C", int], int] = impl
    classvar_callable: ClassVar[Callable[["C", int], int]] = impl
    static_callable: Callable[["C", int], int] = staticmethod(impl)
    static_classvar_callable: ClassVar[Callable[["C", int], int]] = staticmethod(impl)

c = C()

# Establish a baseline that type checkers generally respect the descriptor
# protocol for values assigned in the class body, whether annotated with
# ClassVar or no:
assert_type(c.descriptor, int)
assert_type(c.classvar_descriptor, int)

# The calls and assignments below are all correct per runtime behavior;
# if a type checker errors on any of them and expects a different
# signature, that indicates unsound behavior. Note that the static_*
# variants are annotated exactly the same as the non-static variants,
# but have different runtime behavior, because Callable does not
# distinguish descriptor vs non-descriptor. Thus, it's unlikely that any
# type checker can get all of these correct.

# If a type-checker assumes that callable types are not descriptors,
# it will (wrongly) error on these calls and assignments:

c.callable(1)
c.classvar_callable(1)

x1: Callable[[int], int] = c.callable
x1(1)
x2: Callable[[int], int] = c.classvar_callable
x2(1)

# If a type-checker assumes that callable types are descriptors,
# it will (wrongly) error on these calls and assignments:

c.static_callable(C(), 1)
c.static_classvar_callable(C(), 1)

y1: Callable[["C", int], int] = c.static_callable
y1(C(), 1)
y2: Callable[["C", int], int] = c.static_classvar_callable
y2(C(), 1)

# Now let's look specifically at annotated `__call__` attributes:

def cm_impl(self: "CallMethod", x: int) -> int:
    return x

class CallMethod:
    __call__: Callable[["CallMethod", int], int] = cm_impl

def cmc_impl(self: "CallMethodClassVar", x: int) -> int:
    return x

class CallMethodClassVar:
    __call__: ClassVar[Callable[["CallMethodClassVar", int], int]] = cmc_impl

def cms_impl(self: "CallMethodStatic", x: int) -> int:
    return x

class CallMethodStatic:
    __call__: Callable[["CallMethodStatic", int], int] = staticmethod(cms_impl)

def cmcs_impl(self: "CallMethodClassVarStatic", x: int) -> int:
    return x

class CallMethodClassVarStatic:
    __call__: ClassVar[Callable[["CallMethodClassVarStatic", int], int]] = staticmethod(cmcs_impl)

# Again, all of these are correct per runtime behavior; type checker
# errors indicate an unsound interpretation:

# Type checkers which assume callables are not descriptors will (wrongly)
# error on these:

CallMethod()(1)
cm: Callable[[int], int] = CallMethod()
cm(1)

CallMethodClassVar()(1)
cmc: Callable[[int], int] = CallMethodClassVar()
cmc(1)

# Type checkers which assume callables are descriptors will (wrongly)
# error on these:

CallMethodStatic()(CallMethodStatic(), 1)
cms: Callable[["CallMethodStatic", int], int] = CallMethodStatic()
cms(CallMethodStatic(), 1)


CallMethodClassVarStatic()(CallMethodClassVarStatic(), 1)
cmcs: Callable[["CallMethodClassVarStatic", int], int] = CallMethodClassVarStatic()
cmcs(CallMethodClassVarStatic(), 1)

3 Likes

It doesn’t sound like you are proposing to fix this but the situation where I frequently run into the descriptor vs callable inconsistency is with the following that all type checkers incorrectly reject:

class A:
    abs = abs

a = A()

print(a.abs(-10))
1 Like

Thanks for starting this discussion. This is a complex topic, and I welcome clarity and standardization in this area.

The typing spec is pretty clear that the Callable special form describes a structural type that has a __call__ method. In other words, it’s comparable to a protocol class with a __call__ method. No other methods (such as __get__) are implied.

Internally, pyright models function types with a set of flags that differentiate between normal functions, static methods, class methods, and instance methods. It uses these flags to determine the behavior when binding an object (either a class object or instance) to an attribute or method.

The challenge with attributes that contain callable objects (or more generally, descriptor objects) is that their behavior can change depending on whether the value has been stored on the class object or an instance.

You mentioned that pyright and mypy use a heuristic based on ClassVar, and this approach has no correctness benefit. I can’t speak for mypy, but in the case of pyright, the behavior you see is rooted in how it treats class and instance variables. If an attribute is annotated as a ClassVar, pyright considers it a “pure class variable” and enforces that it cannot be set on an instance. The behavior of a descriptor or a callable can be statically determined in this case. When an attribute is not annotated as a ClassVar, pyright assumes that it’s OK for someone to set the value on an instance. The runtime behavior is then unknowable because it depends on state that cannot generally be known statically.

This can be demonstrated by extending the sample code that you posted above.

c.classvar_callable(1) # OK (no possibility of false positive or false negative)
c.classvar_callable = impl # Assignment not allowed because classvar_callable is a ClassVar

c.callable(1) # Type error but no crash (false positive)
c.callable(C(), 1) # No type error but crashes (false negative)
c.callable = impl # Assignment allowed because callable is not a ClassVar
c.callable(1) # Type error and crash
c.callable(C(), 1) # No type error, no crash

This issue is related to the recent conversation we had about differentiating between class and instance variables. The challenge stems from the fact that Python allows an instance variable to shadow a class variable of the same name. Most languages require that a class-scoped symbol is either a class variable or an instance variable, not both.

One way to guarantee soundness is to completely disallow shadowing in typed Python, but I think there would be a lot of pushback since this practice is very common. Perhaps we could devise a less draconian restriction that preserves soundness in most (all?) cases. The type system already imposes the limitation that when an instance variable shadows a class variable, the type must be the same for both. For example, you can’t declare the type of a class variable to be a str but then indicate that it should be shadowed by a instance variable of type int. Developers generally find that restriction to be reasonable. Since some types (notably, some descriptors and callables) behave differently when stored on the class object and the instance, maybe we should disallow shadowing specifically for these categories of types?

In any case, progress in this area probably requires us to first formalize the concepts of “pure class variables”, “pure instance variables” and “class variables that can be shadowed by instance variables” in the typing spec.

6 Likes

Based on what Eric points out, if it’s annotated without ClassVar, that means it needs to be able to be treated as an instance variable.

So this assignment should be rejected, because you can’t call it with that signature from an instance.

class C:
    callable: Callable[["C", int], int] = impl

This assignment should be accepted, because it can be called with this signature from an instance.

class C:
    callable: Callable[["C", int], int]

    def __init__(self) -> None:
        self.callable = impl

I agree with @erictraut, solving the issue of correctly formalising and inferring pure class variables vs. pure instance variables vs. mixed state will go a long way, doing this will cause most other issues with descriptor types (not just Callables) to be naturally solved without having to add more and more heuristics.


The runtime behaviour of descriptors (including function definitions) are very clear:

  • __get__ only activates if the item that implements it is a class variable, and has no effect if it’s an instance variable;
    class D:
        def __get__(self, instance, owner):
            raise RuntimeError
    
    class A:
        cd = D()
        def __init__(self):
            self.id = D()
    
    a = A()
    a.id  # OK
    a.cd  # RuntimeError
    
  • Data descriptors (items which implement __set__ and/or __delete__) will always activate (if the item is a class variable) upon assignment in an instance:
    class D:
        def __delete__(self, obj): ...
    
    class A:
        d = D()
    
    a = A()
    a.d = 1  # AttributeError: __set__
    
  • Non-data descriptors can be shadowed in an instance:
    class A:
        def f(self): ...
        def __init__(self):
            self.f = 1
    
    a = A()
    print(a.f)  # 1
    

It follows that all function definitions in a class are implicitly ClassVar, because they activate __get__ to return a bound method when accessed through an instance. Shadowing a function definition via assignment in an instance (self.f = <r>) also has a well-defined behaviour at runtime (because functions are not data descriptors), and is actually type-safe if <r> is a Callable which matches the signature of the bound method, but type-checkers prevent the shadowing due to the same reasoning as f being a pure ClassVar - you shouldn’t assign a class variable via an instance, even if the types match.

Callables are then method descriptors if they are intended to behave the same as a function definition in the class body. This means, for an assignment-like statement a in a class body:

  • a: ClassVar[Callable] is currently unambiguously a method descriptor, because Callable is currently the canonical replacement for the type of a function definition, and as established above, function definitions are implicitly ClassVar;
  • a: Callable where a is in __slots__ = ("a",) is definitely not a method descriptor;
  • self.a: Callable is also definitely not a method descriptor;
  • a = lambda *args, **kwargs: None is a method descriptor at runtime, but whether a static type checker should treat it as such is dependent on whether they think unannotated assignments are ClassVar.

The remaining points of contention is then the interpretation of the following (the second example is chosen to deliberately crash when called via an instance):

  • a: Callable[[], None]
  • a: Callable = lambda: None

These points of contention circles back to the ambiguity of class-vs-instance variables for annotations on a class body.


I think the issue can be considered solved when classmethod, staticmethod, etc. aren’t special-cased, but expressed purely through type annotations. IMO there is no reason for this code to fail type-checking, but only pyright seems to pass at the moment (and its passing is dependent on the presence of __slots__):

class A:
    __slots__ = ("m",)
    m: staticmethod[..., int]

a = A()
m: staticmethod[..., int] = a.m  # Most type-checkers error here

The problem here is that abs is a builtin function (implemented in C), and this is another kind of callable object that (unlike Python-defined functions) doesn’t implement __get__ to make itself a bound-method descriptor. But type checkers have to rely on type stubs, and in typeshed abs is simply defined as a Python function, so type checkers assume it is a bound-method descriptor.

I think it might be possible to reflect this in the stubs in some way; a hacky way that would likely work would be to simply mark builtin functions with the @staticmethod decorator. But I think it’s a separate issue from the topic of this thread.

3 Likes

Thanks for the replies!

I agree. I believe that this is the interpretation that ty and pyrefly already implement. If we were to agree on this interpretation, I think it would indicate a change in the current behavior of pyright and mypy: they should not interpret Callable-annotated attributes as bound-method descriptors, regardless of ClassVar. The question of “pure class variables” vs “class-and-instance variables which might be shadowed on the instance” (while certainly important in general) is irrelevant to the topic of this thread, if we don’t consider a Callable type to implement the descriptor protocol in the first place, because a non-descriptor doesn’t behave any differently when set on the class vs on an instance. (Note, though, that agreeing on this doesn’t fix the unsoundness, it just means we agree on one particular version of the unsoundness. I don’t see any way to fix the unsoundness without splitting Callable into two separate incompatible types, which I don’t think is very palatable.)

I don’t think the unsoundness here depends on the static unknowability of instance-level shadowing, and I don’t agree that forbidding instance-level shadowing would provide soundness. The code example in my post demonstrates this. If we forbid instance-level shadowing, it remains true that both impl and staticmethod(impl) are assignable to Callable[[int], int], but at runtime they behave in incompatible ways. Given these two attributes:

def impl(x: int) -> int:
    return x

class C:
    is_a_descriptor_at_runtime: ClassVar[Callable[[int], int]] = impl
    is_not_descriptor_at_runtime: ClassVar[Callable[[int], int]] = staticmethod(impl)

Both of those assignments are valid. There is no possibility of instance-level shadowing here, because we are using ClassVar. (We could give the same example without ClassVar in a hypothetical world where we’ve outlawed instance-level shadowing of class attributes.)

Is a type checker supposed to treat those two attributes differently, despite them being annotated identically? If it does not, then it will definitely be unsound on one of them. (And we can follow up with valid assignments C.is_a_descriptor_at_runtime = staticmethod(impl) or C.is_not_descriptor_at_runtime = impl to easily invalidate whatever inferred distinctions the type checker might try to make in order to be sound.)

Our uncertainty (and unsoundness) about whether it is correct to treat a Callable type as a descriptor, or not, is inherent in the fact that we allow both runtime descriptors and runtime non-descriptors to be assigned to the same Callable type. It doesn’t require the possibility of instance-level shadowing.

1 Like

Another point of divergence is attributes - __doc__, __code__ etc. Usage of those are rare enough that it’s probably fine to ignore, but it does show another issue. Callable feels like it’s a bunch of types in a trenchcoat, but it seems difficult to break up now.

If I understand you correctly, you’re saying that because the Callable structural type doesn’t include a __get__ method in its definition, we should assume that no __get__ method is present on an object that conforms to this structural type. Is my understanding correct? If so, that doesn’t sound right to me. If we treat it like other structural types, then Callable says nothing about either the presence or absence of a __get__ method. That means we need to use other information and rules to determine the binding behaviors for an attribute that has a Callable type annotation, which is what mypy and pyright are doing.

I’m open to changing pyright’s behavior if it more accurately models the runtime behavior and we can get consensus from the community (including the mypy maintainers). However, we’d need to take into account the impact of such a change. I suspect there are many stubs, libraries, and source bases that rely on mypy’s and pyright’s current interpretation. Have you done any experiments to see what the fallout would be if we were to make a change here? My educated guess is that it would be a really painful change for users to swallow.

Here’s another idea. What if we added a few new special forms like Method, ClassMethod and StaticMethod? These would be subtypes of Callable but they would also include a __get__ method as appropriate. Like Callable, these would be structural types, as opposed to the the nominal types defined in types.pyi such as MethodDescriptorType and ClassMethodDescriptorType. They’d support the same generic parameterization as Callable. Perhaps that’s roughly what you had in mind when you said “splitting Callable into separate incompatible types”? If we were to do this, we could keep the current (mypy and pyright) behavior for Callable but implement the improved behavior for these new subtypes.

Another thing to consider is how any such change would affect typed decorators — those that use either TypeVars with a Callable upper bound or Callable with a ParamSpec. I mentioned above that pyright uses flags internally to track whether a function type is an regular function, an instance method, a class method, or a static method, and it uses these flags to inform the binding behavior. These flags are also captured and propagated by the constraint solver, so if a callable object is passed as an argument to a generic decorator function, the return result of the decorator retains these flags. I think mypy does something similar.

Code sample in pyright playground

In addition to these flags, pyright also captures the docstring of the pre-decorated function so when you hover over references to the post-decorated function, you’ll still see the docstring in your editor. Are you doing something similar in ty and pyrefly? If not, you may find that existing typed decorators don’t work in a compatible way.

4 Likes

This looks less to do with Callables and more to do with descriptors in general. Here’s a non-Callable example:

from typing import Any, ClassVar

class IntDescriptor(int):
    def __get__(self, instance: Any, owner: type[Any], /) -> str:
        return str(self)

class A:
    val1: ClassVar[int] = 0
    val2: ClassVar[int] = IntDescriptor()  # Type-checkers currently don't flag this
    val3: ClassVar[IntDescriptor] = IntDescriptor()

a: A = A()
reveal_type(a.val1)  # int
reveal_type(a.val2)  # int
reveal_type(a.val3)  # str

How about making type-checkers flag such an assignment if either the instance or class-accessed type isn’t assignable to the given annotation? This is assuming that we’re still talking about the possibility of Callable of being method descriptors, but it should still apply to other descriptor cases regardless.

def impl(x: int) -> int:
    return x

class C:
    is_a_descriptor_at_runtime: ClassVar[Callable[[int], int]] = impl
    # Error: Class variable expression of type `staticmethod[[int], int]` is incompatible with instance attribute of type `Callable[[], int]`
    # Hint: Use `ClassVar[staticmethod[[int], int]]`
    is_not_descriptor_at_runtime: ClassVar[Callable[[int], int]] = staticmethod(impl)
class A:
    val1: ClassVar[int] = 0
    # Error: Class variable expression of type `IntDescriptor` is incompatible with class attribute of type `str`
    # Hint: Use `ClassVar[IntDescriptor]`
    # Error: Class variable expression of type `IntDescriptor` is incompatible with instance attribute of type `str`
    # Hint: Use `ClassVar[IntDescriptor]`
    val2: ClassVar[int] = IntDescriptor()
    val3: ClassVar[IntDescriptor] = IntDescriptor()

I’ve found myself wishing for precisely this before, in instances where my code handles bound methods in a different way than it handles other sorts of callable. I’ve had success rolling my own Protocol to describe a bound method more completely, but it’s such a fundamental element of the language that a dedicated special form would be preferable, IMO.

2 Likes

Adding new flavors of Callable as suggested by @erictraut sounds more promising than tweaking than the semantics of Callable. We’ve tried to improve the behavior of Callable in mypy in the past, but it’s very easy to break some important existing use cases. I feel like we’ve reached this local maximum where it works for most (but not all) users, but it’s really to hard to make significant further progress while retaining sufficient backward compatibility.

I think that estimating the impact of proposed changes empirically would be good thing, however – maybe backward compatibility is worth breaking if the benefits are big enough.

1 Like

Having a MethodLike protocol that includes both __call__ and the habitual __get__ semantics would be great for creating precise types for things like cache decorators (which often slap a couple extra attributes onto a function but are still returning a function with the binding behavior):

class MethodLike[**P, RT](Protocol):
    def __call__(*args: P.args, **kwargs: P.kwargs) -> RT: ...

    def __get__[SelfType, **RestArgs](
        self: "MethodLike[Concatenate[SelfType, RestArgs], RT]",
        instance: SelfType,
        owner: object | None = None,
    ) -> "Callable[RestArgs, RT]": ...

^ I ended up doing something like this in one case earlier this week, and it’s been working well enough.

In the general case it’s not clear to me what the “proper” generic return type for __get__ would be here though, since the protocol shouldn’t prevent the MethodLike from implementing a custom __get__ that returns another MethodLike of a specific shape.

For cases where you really are relying on function’s binding behavior, maybe even a protocol that is purely focused on the binding behavior?

class FunctionTypeInstanceBinding(Protocol):
    def __get__[SelfType, **RestArgs](
        self: "Callable[Concatenate[SelfType, RestArgs], RT]",
        instance: SelfType,
        owner: object | None = None,
    ) -> "Callable[RestArgs, RT]": ...

Something that might be even more precise is being able to have FunctionType in typing with generics support so one could build their interfaces/protocols off of FunctionType[P, RT]. That feels precise and matches what people are doing in the “create a function and then slap some attributes onto it”.

I personally would love to have a lot more publicly accessible type sigs for these cases. I would love a type specificly for bound methods, for FunctionTypes, etc so when writing library code I can be very pedantic.

(EDIT: after typing this up I found a whole slew of nits and things wrong with all of my code examples… someone with a better understanding than I of all of this should be in charge of determining the actual cutpoint. I do still think that if possible having a name for these protocols would be very good and I believe can be sound?)

1 Like

One possible resolution to this issue might be to support annotations of partial applications in general. This could then be used to generally create all of the flavors of callables that Eric was talking about, which I illustrated here.

I understand that Pyright currently handles this using flags—likely for performance reasons—which makes sense. That said, using flags can sometimes lead to edge cases where the behavior is hard to get exactly right. For example, the propagation of flags through decorators is not explicitly defined, invisible to the user, and can lead to type errors.

As type checkers continue to improve in speed, it could be worth revisiting solutions that handle these edge cases more robustly.

1 Like