Allow self binding for generic ParamSpec

Typing
1

We ran into some issues while trying to use ParamSpec to type lru_cache in typeshed. See

Basically, while self / cls is bound in case the ParamSpec is directly passed through as Callable, that isn’t the case for the generic version.

from typing import Callable, Generic, ParamSpec

P = ParamSpec("P")

class Wrapper(Generic[P]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> None: ...

def decorator(f: Callable[P, None]) -> Callable[P, None]: ...
def lru_cache(f: Callable[P, None]) -> Wrapper[P]: ...

class A:
    @decorator
    def method1(self, val: int) -> None: ...
    
    @lru_cache
    def method2(self, val: int) -> None: ...

    def test(self) -> None:
        reveal_type(self.method1)  # def (val: int)
        reveal_type(self.method2)  # Wrapper[[self: A, val: int]]

        self.method1(2)
        self.method2(2)  # Error: Missing positional argument "val"

Although all type checkers seem to agree, this makes using generic ParamSpec in these cases difficult (if not impossible). One option would be to remove the first argument with Concatenate and using overloads but that creates additional issues with @staticmethod and @classmethod.

The solution IMO would be to allow self binding even for generic ParamSpec, so that it doesn’t matter if the return type is a Callable or a custom generic. I.e.

    def test(self) -> None:
        reveal_type(self.method1)  # def (val: int)
        reveal_type(self.method2)  # Wrapper[[val: int]]

        self.method1(2)
        self.method2(2)

A reference implementation for mypy can be found here:

3 Likes
2

I believe this is one part the typing spec currently doesn’t cover almost at all. So it may be more helpful if we first add specification for how self-binding in conjunction with ParamSpec is supposed to work and then can extend that from just an explicit Callable type or function declaration to arbitrary types satisfying the Callable protocol, including descriptors that return such types from __get__, where the headaches usually really start to happen.

Part of the problem certainly is ambiguity, since the type system is trying to abstract away Python’s object model[1] through special-casing callables. So it’s more of a question of finding out where the abstraction currently holds up and where it doesn’t and whether or not it can be fixed to cover more or all the possible cases accurately, without creating other problems.

  1. which creates MethodType and other wrapper types ↩︎

1 Like
3

This is a long-standing problem, so thanks for starting the conversation. However, I think the problem is more complicated than you’ve indicated. Here’s some additional context.

In your code example, the signature captured by P includes the self parameter of method1 and method2. In both cases, P captures the signature (self: Self@A, val: int).

When this captured signature is applied to a Callable (as with decorator in your example), the specialized callable type is:

def (self: Self@A, val: int) -> None: ...

When you bind this to an instance of A, it works as you would expect for an instance method.

When the same captured signature is applied to the Wrapper class (as it is with lru_cache in your example), the specialized type of the Wrapper.__call__ method becomes:

def __call__(__self: Self@Wrapper, self: Self@A, val: int) -> None: ...

When you call self.method2(2), the first parameter (which I renamed __self) is bound to an instance of Wrapper. There is no opportunity to bind the second parameter (self) to an instance of A.

I’m not sure what you’re proposing when you say “allow self binding for generic ParamSpec”. Are you suggesting that the self parameter of an instance method should never be captured by a ParamSpec? If so, that would break the Callable case. Are you suggesting that the extra self parameter should be dropped under certain circumstances when applying a captured signature during specialization? If so, under what circumstances?

I’ll note that this gets even more complicated when a return type is captured along with a signature and that return type depends on the type of the self parameter. For example:

class Wrapper[**P, R]:
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R: ...

def decorator1[**P, R](f: Callable[P, R]) -> Callable[P, R]: ...
def decorator2[**P, R](f: Callable[P, R]) -> Wrapper[P, R]: ...

class A:
    @decorator1
    def method1(self) -> Self:
        return self

    @decorator2
    def method2(self) -> Self:
        return self

In this case, the type of self (which is implicitly Self) must be preserved in the transformed method type to allow the return type (Self) to be solved at the time the method is called. It’s not clear how this would work in the case of decorator2.

4

I think of it as implicitly passing Self@A as argument if method2 is called with self, type[Self]@A in case the method is a @classmethod and not passing it for @staticmethod (same as for A.method2).

Return Self is indeed an interesting case. Notably, it doesn’t work currently with neither pyright nor mypy. For reveal_type(self.method2() pyright returns Unknown, whereas mypy even complains about a missing positional argument “self” and infers it as Never. This At least for mypy this suggests to me that the Self@A instance isn’t passed along which this proposal aims to resolve.

With the mypy change above, it would be inferred as R'5 i.e. Any.