Typing generic callback protocol with *args and **kwargs

Python Help
1

I’m trying to do some possibly weird stuff with decorators and I’m struggling to get the typing right.

I have created a decorator to which you provide some metadata, and when it’s called on a function it will put that function in a container with the associated metadata and then store that container with the same name as the original function.

The functions that I store in the container are generic in their arguments and return types, and I cannot seem to get everything working together.

Here is a simple example of what I’m trying to do

from typing import Protocol, TypeVar, Callable, Generic
from dataclasses import dataclass

T = TypeVar("T", contravariant=True)
S = TypeVar("S", covariant=True)

class TestProtocol(Protocol[T, S]):
    def __call__(self, test: T, *args, **kwargs) -> S:
        ...

@dataclass(frozen=True, slots=True)
class TestContainer(Generic[T, S]):
    a: int
    b: str
    func: TestProtocol[T, S]


def test_decorator(a: int, b: str) -> Callable[[TestProtocol[T, S]], TestContainer[T, S]]:

    def inner_decorator(func: TestProtocol[T, S]) -> TestContainer[T, S]:
        return TestContainer(a=a, b=b, func=func)

    return inner_decorator


@test_decorator(1, "b")
def test_func(test: int) -> str:
    return str(test)

Mypy gives the following error:
mypy_test.py:26: error: Argument 1 has incompatible type "Callable[[int], str]"; expected "TestProtocol[<nothing>, <nothing>]" [arg-type]
However if I remove the *args, **kwargs in TestProtocol.__call__ Mypy and Pyright are both happy.

But, my code requires these functions to take arbritrary *args and **kwargs, and I’m not sure how to make the type checkers happy in this case. This seems like a place to use ParamSpec but I’m not sure how to use ParamSpec together with Protocol.

Anybody knows what I should do to make the checkers happy?

2

Hi,

are you adding the ‘’ or ‘**’ along with the arguments when calling the methods? For example, if I have list and a function that takes the square of every item in the list, when I call the function, I have to include the asterisk '’ along with the argument otherwise an error will be called.

oneList = [1,2,3,4,5,6]

def someFunc(*args):

      aList = []

      for i in args:
            singleResult = i**2
            aList.append(singleResult)

      return aList

# Call the function to print function values
print(someFunc(*oneList))  # Include the '*' along with the argument that is being passed in

print(someFunc(oneList))   # Without the asterisk * will get error

Give it a try and see if this is what is causing the error.

3

I’m not sure I understand what you are saying. The problem is not at the call site of the decorated function (`TestContainer.func’), which is what I think you’re showing in your example. The decorated function is called at some other place in another file, but my error shows on the line

@test_decorator(1, "b")

and appears because, afaict, Mypy cannot figure out the type of TestProtocol in the decorator based on the decorated function.

4

Right, wherever outside in the program the class method is being called, you have to include the asterisks along with the arguments that you are passing in. Take a look at my previous response. I have added an extra line in the test code.

I don’t know your code. But I do have experience running into a similar type of error when I didn’t include the asterisks as part of arguments when calling a function using the *args / **kwargs type of arguments in my function definition. If you don’t include asterisks along with the arguments, when calling the functions, you will get an error.

How are you instantiating the ‘TestProtocol’ class? Can you show the code?

Also make you have proper indentation. I am assuming you copied your code into the panel. When I copied and pasted the code into my test file, there were a lot of indentation errors.

5

I still don’t understand, so could you modify my example to make the type checkers happy, while keeping the ability of TestProtocol.__call__ to take a generic first argument and an arbritrary amount of other arguments? Not asking you solve my homework (though that would be nice) but rather as an explanation of what you mean.

6

Your protocol requires that the callable accept an arbitrary number of both positional and keyword parameters, and test_func does not meet this requirement. That’s why you’re seeing the error.

You can use a ParamSpec to capture the specific parameters that the decorated function accepts.

Code sample in pyright playground

from typing import ParamSpec, Protocol, TypeVar, Callable, Generic
from dataclasses import dataclass

T = TypeVar("T", contravariant=True)
P = ParamSpec("P")
R = TypeVar("R", covariant=True)

class TestProtocol(Protocol[T, P, R]):
    def __call__(self, test: T, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

@dataclass(frozen=True, slots=True)
class TestContainer(Generic[T, P, R]):
    a: int
    b: str
    func: TestProtocol[T, P, R]

def test_decorator(
    a: int, b: str
) -> Callable[[TestProtocol[T, P, R]], TestContainer[T, P, R]]:
    def inner_decorator(func: TestProtocol[T, P, R]) -> TestContainer[T, P, R]:
        return TestContainer(a=a, b=b, func=func)

    return inner_decorator

@test_decorator(1, "b")
def test_func(test: int) -> str:
    return str(test)
2 Likes
7

Disclaimer: I’m very much not a typing expert and this is a bit above my pay grade, but with regards to Mypy, while I can reproduce this type check failure in Mypy 1.6.1 and below, running Mypy 1.7.0 and up on the above code results in no type errors. I’m not totally certain which of the scores of changes in 1.7.0 is the direct cause of this (it’s not the new type inference, as --old-type-inference has no effect).

However, it seems it may actually be a regression in Mypy to not error on this, as python/mypy#16568 and python/mypy#16569 seem related here, and Pyright and past versions of Mypy are actually behaving correctly in warning this, if I understand correctly, since test_func is actually incompatible with the protocol as it does not take *args and `**kwargs``

I defer to the relevant experts on the best way to handle this as its out of my league, but I suspect ParamSpec is the way to go here instead of, e.g., (*args: Any, **kwargs: Any) if python/mypy#5876 is any guide. EDIT: See Eric Traut’s answer, posted since I started writing mine, for a far more authoritative explanation and solution here.

1 Like
8

Thank you Eric and @CAM-Gerlach, ParamSpec did the trick in Pyright. I did try ParamSpec before but must have mucked it up in my frustration (tried using ParamSpec, Concacenate, and Callable and got super weird inconsistent results).

I can also confirm the mypy regression in 1.7.0. In 1.6.0 (last one i tried but I assume it’s the same in 1.6.1) both my and Eric’s examples fail type checking in mypy, but in 1.7.0, both pass. So older version give false negatives and the newer version give false positives (I think that’s the right way around but I’m not entirely sure about the terminology). Should I add this example to one of the issues on the tracker or do you think they have enough information to go on?