Generic class depending on a generic callable

I’m maintaining a python library that has recently introduced typing in its code base. It used to work well until a regression was introduced in mypy 1.7. It took me a while to investigate this issue, mostly because the library uses some meta-programming patterns (e.g. turning function into fully fledged classes). However, I think I reached the bottom of the issue and I can illustrate it with a rather simple example:

Consider the following Convertor class:

@dataclass
class Convertor(Generic[I, P, O]):
    convert: Callable[Concatenate[I, P], O]

    def prepare(self, *args: P.args, **kwargs: P.kwargs) -> Callable[[I], O]:
        return lambda source: self.convert(source, *args, **kwargs)

This achieves something similar to functools.partial, in the sense that parameters are applied in a two-step process. For instance, it can be used to transform a division function into a divide_by_two function:

def division(arg: int, divisor: int) -> float:
    return arg / divisor

division_convertor = Convertor(convert=division)
divide_by_2 = divide_by_convertor.prepare(division=2)
reveal_type(divide_by_2)  # def (builtins.int) -> builtins.float

This works fine with both mypy and pyright, nothing weird so far.

It gets more subtle when the provided function is generic. For instance, consider a function turning an element into a list of said element:

def as_list(source: X, repeat: int = 1) -> list[X]:
    return [source] * repeat

as_list_convertor = Convertor(convert=as_list)
repeat_twice = as_list_convertor.prepare(repeat=2)
reveal_type(repeat_twice)

Now that’s when things start to break:

  • mypy 1.6 produces requires a type annotation for repeat_twice and reveal repeat_twice as a def (Any) -> builtins.list[Any]
  • mypy 1.7 and later requires a type annotation for as_list_convertor and also reveal repeat_twice as def (Any) -> builtins.list[Any]

However, pyright does reveal repeat_twice as (X@as_list) -> list[X@as_list] and even correctly infer the result of repeat_twice, by using the type of the provided argument:

result = repeat_twice(1)
reveal_type(result)
assert result == [1, 1]

Here pyright reveals result as list[int]. Here’s two playground links including the full example:

As a side note, a regression has been introduced in mypy 1.7 that causes even Convertor.convert to break (no need for the prepare method to be defined). You can see that in this playground and the corresponding issue I opened on the mypy repo.

Now, it’s clear that the example with as_list_convertor is a bit tricky: it is a Convertor[I, P, O] parametrized with:

  • I = X@as_list
  • P = (repeat: int = 1)
  • O = list[X@as_list]

This means that I and O cannot be fully resolved. They’re merely passed to the callable produced by as_list_convertor.prepare and only when this callable is called, X can be resolved.

So my questions are:

  • is this a valid use case?
  • is it too dynamic, or is the plan to fully support it in the future?
  • am I doing this wrong?

My suspicion is that we’re in unspecified territory, as @erictraut mentioned in a loosely related issue I opened on the pyright repo.

Any answer or help would be much appreciated since I have yet to find a way/workaround to fix the typing in the python library I maintain.