Subtyping rules for `...` in callable types

I’m planning to write a new chapter for the typing spec that focuses on subtyping rules for callable types. This is an area that has always been underspecified in the spec and the typing PEPs.

One topic that I’d like to get feedback on before writing the chapter has to do with the subtyping rules for .... The ellipsis token can be used in the Callable special form. It can also be used to specialize a generic class or type alias parameterized with a ParamSpec. And it can be used with or without a Concatenate. This is all well documented and specified.

The meaning of ... is not precisely specified in the typing spec, but all type checkers appear to treat it as a gradual type form. That is, it’s analogous to the Any type, but it applies to callable signatures (or partial signatures, if used with concatenation). As a gradual type, it implies bidirectional type consistency with any callable signature (or partial signature).

For example, the types of all of the following functions are bidirectionally compatible with the type Callable[Concatenate[int, ...], None].

def func1(x: int, /) -> None: ...
def func2(x: int, /, y: int) -> None: ...
def func3(x: int, /, *args: int) -> None: ...
def func4[**P](x: int, /, *args: P.args, **kwargs: P.kwargs) -> None: ...

f: Callable[Concatenate[int, ...], None]
f = func1  # OK
f = func2  # OK
f = func3  # OK
f = func4  # OK

So far, so good. All of the major type checkers agree on the above.

The question I have is whether ... can be specified in any other manner. If you’re defining a callback protocol, is there a way to define the __call__ method in a def statement such that the semantics are the same as if you had used ... in a Callable annotation?

Mypy appears to apply an undocumented rule (heuristic?) here. If the __call__ signature includes both a *args: Any and a **kwargs: Any parameter with no intervening parameters, then mypy treats the signature (or partial signature) as though it’s a .... If the *args or **kwargs parameters have any other type annotation (such as object) or have any intervening parameters, then mypy doesn’t apply this rule. It also applies this rule in cases where the *args and **kwargs parameters are generic but are specialized to Any.

T = TypeVar("T", contravariant=True)

class Proto1(Protocol):
    # Mypy evaluates (x: int, ...)
    # Pyright evaluates (x: int, *args: Any, **kwargs: Any)
    def __call__(self, x: int, *args: Any, **kwargs: Any) -> None: ...

class Proto2(Protocol[T]):
    def __call__(self, x: int, *args: T, **kwargs: T) -> None: ...

class Proto3(Protocol):
    def __call__(self, x: int, *args: object, **kwargs: object) -> None: ...

class Proto4(Protocol):
    def __call__(self, x: int, *args: Any, y: int, **kwargs: Any) -> None: ...

class Concrete:
    def __call__(self, x: int, *, y: int) -> None:

f1: Proto1 = Concrete()  # OK (mypy), Error (pyright)
f2: Proto2[Any] = Concrete()  # OK (mypy), Error (pyright)
f3: Proto3 = Concrete()  # Error (mypy and pyright)
f4: Proto4 = Concrete()  # Error (mypy and pyright)

Currently, pyright does not apply any special-case rules because there is no such provision anywhere in the typing spec or in any PEPs, at least to my knowledge. If you want to use ... in a protocol in pyright, you’d need to use a ParamSpec and explicitly specialize it.

class Proto5[**P](Protocol):
    def __call__(self, x: int, *args: P.args, **kwargs: P.kwargs) -> None: ...

f: Proto5[...] = Concrete()  # OK (mypy and pyright)

Is mypy’s special-cased behavior here defensible and desirable? What about pyright’s behavior? I can make an argument for either, but I think it’s important that we choose one and document it. I find mypy’s special-casing here to be surprising and unintuitive, but I can understand why it might have been added at some point.

If we adopt some variant of mypy’s behavior, then I question whether TypeVars specialized with Any should be in-bounds for the rule. I also question whether intervening keyword-only parameters should affect the rule. I’m guessing these were unintended behaviors in mypy’s implementation.

1 Like

From your examples, regarding where mypy and pyright differ:

With a structural type as an annotation, the contract for assignment is “the object assigned must be usable as the protocol defines”, and the contract for later use is “you only know it is safe to use it as the protocol defines”

f1 and f2 both violate this

Proto1 says the following is valid: (edited in other reasons this doesn’t work)

p1: Proto1
p1(1, 2, 3, 4)
p1(1, "this", that="this")
p1(1, y="Not an int")

Instances of Concrete cannot be called like this, so they are invalid values for variables with Proto1 as an annotation

similarly so for p2

1 Like

f5 (parameterized with ...) should also fail here, same reasoning as above

Yeah, mypy made this change a few years ago, in response to Make any callable compatible with (*args: Any, **kwargs: Any)? · Issue #5876 · python/mypy · GitHub. That issue remains one of the most popular mypy issues of all time. The original implementation is here.

In particular, this came up a lot when subclassing. Users appear to often want a gradual type in a parent class method; without the non-standard behaviour, child classes would have to handle all possible args and kwargs to avoid the LSP error.

When I implemented that PR (mypy 0.981), mypy’s special casing only applied to exactly (*args: Any, **kwargs: Any); additional parameters removed the special casing. You’re right that TypeVars specialised with Any would be affected, I didn’t intentionally choose that behaviour.

This was intentionally changed later (mypy 1.7) in Lenient handling of trivial Callable suffixes by ilevkivskyi · Pull Request #15913 · python/mypy · GitHub to allow “suffixes”. Reading that PR back, it’s unclear how strongly people felt about it. I’m not sure I recall too much demand for more things like Concatenate[int, ...] (relatedly, note intervening keyword args is a construct no longer equivalent to Concatenate[int, ...]). So if folks dislike this, you probably won’t get much pushback.

I’d be in favour of incorporating some version of this special case into the typing spec. Users who want non-gradual behaviour are served well by (*args: object, **kwargs: object).


Answering more than what makes sense from just theory, I don’t think this is pragmatic for the examples given above. I have significant concerns about this special casing as it would apply to the work in progress for intersections, specifically for decorators that wrap callables while augmenting them if the special casing is applied broadly

I don’t actually think this particular case is a special case, and it’s a level of equivalence that makes sense for these as structural types.

class A(Protocol):
    def __call__(self, *args: Any, **kwargs: Any) -> Any:  ...

Callable[..., Any]

are identical in what’s allowed, the other things allowed from further changes seems to have caused soundness issues, adding other parameters here causes no longer matching signatures.

This line from your example explains why none of these cases should work.

I’m not seeing an issue for intersections, it would just propagate in this case, no?

@hauntsaninja, thanks for the historical perspective. Given that context, it makes sense to me that it’s desirable to interpret (*args: Any, **kwargs: Any) as equivalent to .... I’m fine with that interpretation, but I’d like to tighten up the specification a bit. In particular:

  1. I don’t think this interpretation should apply in cases where the annotation is anything other than Any. That means TypeVars or unions with Any would not fall under this special case.
  2. I don’t think that preceding or intervening parameters should affect the interpretation. In other words, I think (a: int, /, x: int, *args: Any, z: int, **kwargs: Any) should be interpreted as (a: int, /, x: int, *, z: int, ...).
  3. If *args and **kwargs are unannotated, they should be treated the same as if they are annotated with Any, which means (x, *args, **kwargs) should be interpreted as (x: Any, ...).

@mikeshardmind and @Liz, I’m struggling to understand your responses. I think you may be confused about my question. I’m not asking how to interpret ... in a callable signature. I think that’s already well established, and I’m not proposing to make a change here. Perhaps the title I used for this thread is confusing. If so, I apologize.

My intent was to simply ask whether a def statement that includes parameters *args: Any and **kwargs: Any should be interpreted as .... Based on the context provided by @hauntsaninja, I think there’s a good argument for interpreting it as such.

The ... concept is already well established for callables, so this should have no impact on intersections or any other typing features.

I was following the examples you gave, and these examples came up out of band in another discussion that was being had. The potential confusion around this is likely more my fault.

The cases mypy allows that pyright doesn’t currently are incorrect for the reasons above, so it would appear that at least some portion of the special casing mypy has is undesirable.

There also seems to be a shared issue between mypy and pyright in this case

If you think that this case should be fine, I think we need to pump the breaks a moment and discuss the behavior. This appears wrong. This is saying that f can only be assigned callables that accept an int as their first parameter, and must accept arbitrary other args and kwargs. Concrete does not fit this shape.

1 Like

I understood ... to be equivalent to (*args: Any, **kwargs: Any) I brought up the contravariance your own typevar had because it’s important in why these examples are broken.

While you can Call a Callable that accepts anything with anything, you can’t assign a callable that doesn’t accept just anything to something expecting a callable that does.

This is unsafe. But it is consistent with Callable[…, object] behaves.

def foo():

x: Callable[..., object] = foo # Allowed
x(1) # Crashes but allowed.

def prepend_int(f: Callable[P, R]) -> Callable[Concatenate[int, P], R]]:
  def f_(x: int, /, *args: P.args, **kwargs: P.kwargs):
  return f_

foo2: Callable[..., object]
prepend_int(foo2) # This has type Callable[[int, ...], object]
prepend_int(foo2)(1, "foo") # May crash on second argument and allowed.

The unsafety in Proto5 case is same as unsafety for ... in callable in general and is even explicit with specialization. ... is explicitly allowed to mean any argument signature and allow for holes similar to Any.

1 Like

if ... isn’t just shorthand for (*args: Any, **kwargs: Any) but can fill any hole, then I think now would be a very good time to reconsider that. This makes the more ergonomic option the less safe option.

I don’t think ... is well established enough here. Even if it’s allowed to fill a hole, there’s at minimum underspecification similar to what I brought up with Any.

When it’s no longer a purely gradual type, but a partially known type, the partially known part should definitely be required to remain. The problem with this is that Concrete has a specific keyword argument. There’s no way to express to users “any kwarg except y”, that requires type negation, so what’s left is that this should be incompatible.

Considering the capabilities we have with ParamSpec, concatenate, and typevartuples now, I think this may actually be something that should be revisited.

Do I understand the following correctly?

  • Mike is onboard with a potential equivalence between Callable[..., X] and (*args: Any, **kwargs: Any) -> X as gradual types.
  • It’s unclear whether Mike is onboard with a potential equivalence between the gradual type in Callable[Concatenate[int, ...], X] and (x: int, /, *args: Any, **kwargs: Any) -> X.
  • Mike is not onboard with there being a gradual type in (x: int, *args: Any, **kwargs: Any) -> X, or if there was an intervening keyword argument
  • Mike also wants to think more about cases where ParamSpec substituted by ....
  • Liz does not think ... should be a gradual type at all and so the equivalence should be between Callable[..., X] and (*args: object, **kwargs: object) -> X

(sorry for deleting and undeleting, I posted this right as others posted. Since this post is now quoted I’ll leave it up)

Here’s what the specification currently says. It makes no note of Any-like compatibility behavior:

It is possible to declare the return type of a callable without specifying the call signature by substituting a literal ellipsis (three dots) for the list of arguments:

def partial(func: Callable[..., str], *args) -> Callable[..., str]:
   # Body

Note that there are no square brackets around the ellipsis. The arguments of the callback are completely unconstrained in this case (and keyword arguments are acceptable).

A plain reading of this does not allow functions that have constraints for their parameters.

I think ... should be a gradual type, but that it is inherently problematic to assign something accepting more limited type to a contravariant structural gradual type. This is taking something known to have a specific interface and lying to say it can be used more broadly, this should require a cast or just use of Any

This seems fine to me for equivalence.

with it being positional only, this is also fine for equivalence.

There’s definitely a problem of stating an expectation that the type system knows isn’t true, but I feel slightly less strongly about it than @Liz expressed. I think that because it’s a structural type and does not need to 1:1 match any actual runtime class when used gradually, this should require instead being spelled out with a protocol rather than ... to avoid ambiguity. This wouldn’t be unique, and is already covered by the specification, which says:

Since using callbacks with keyword arguments is not perceived as a common use case, there is currently no support for specifying keyword arguments with Callable . Similarly, Callable does not support specifying callback signatures with a variable number of arguments of a specific type. For these use cases, see the section on Callback protocols.

This is where I forsee potential problems with intersections specifically. I’d appreciate being able to get back to you tomorrow about this in particular after I’ve had time to explore it in more depth.

1 Like

I know I said tomorrow, but this was one of those things that nagged at me and I ended up just finding 2 examples that I thought it could negatively affect, evaluated them, and it did not. it shouldn’t be a problem for intersections. The remaining concern from me is on ... allowing incompatible kwarg use. I don’t think it’s something that is clearly specified right now. While I’d prefer if we don’t allow it, and I think there’s a rationale that works for this, I’m fine with the outcome being allowing it as long as it’s clearly specified.

Edit: see comment here for the full related context, the most relevant part is below. It took me more to piece together what wasn’t sitting right with me…

The safe subtyping of gradual types in general is more complicated than we have language for.