Expanded typing spec chapter for callables

Typing
1

I’ve written a draft version of an expanded “Callables” chapter in the typing spec. It attempts to define detailed subtyping rules for callables. It also address several other previously-unspecified rules for callable types.

Most of the new content should be uncontroversial, as it’s simply codifying existing rules that all major type checkers already follow. However, there is one area that might be contentious: Should a function with the signature (*args: Any, **kwargs: Any) be considered equivalent to ... in a Callable? This was discussed in some detail in this thread.

In the draft chapter, I’ve taken the stance that (*args: Any, **kwargs: Any) should not be equivalent to .... This interpretation is consistent with pyright’s current handling but inconsistent with mypy’s. The special-case behavior in mypy was added in response to feedback from mypy users to address a real issue they were facing, but after thinking about this more, I don’t think the solution mypy adopted is the right solution. If this is a real problem for users, I’d prefer to devise a better solution — one that doesn’t involve a hard-to-explain special case. Special cases like this tend to beget yet more special cases and create problems for composability of typing features. One potential solution is to support (*args: ..., **kwargs: ...) as a way to indicate ... in a def statement.

If you’d like to report typos, minor clarifications, bugs in code samples, or suggest minor wording changes, please add comments directly to the PR.

For more substantive feedback, please post to this forum.

2

I agree with your interpretation. I think mypy’s behavior is a crutch for the insufficiency of callback protocols. It’s also worth noting that mypy’s rules for when this is treated as a gradual extension are very odd and inconsistent. You can e.g. do def __call__(self, foo: int, *args: Any, **kwargs: Any) -> str: ... for something similar to Callable[Concatenate[int, ...], str], except the concatenated parameter is named. But as soon as you use a / to make foo postional-only it is no longer treated as a gradual extension.

I think having a spelling for a gradual extension of a signature inside protocols is valuable, since Concatenate only allows a gradual extension of positional-only arguments, but it needs to be unambiguous. I like your idea of reusing ... for that purpose.

3

mypy’s current behaviour is a special case, but it’s in response to one of the most popular mypy issues of all time. I don’t know that I’d be willing to remove it entirely. I don’t know if this feels any better to you, but for what it’s worth I would be willing to e.g. remove the (x: int, *args: Any, **kwargs: Any) → Concatenate[int, ...] bit, and only keep the core (*args: Any, **kwargs: Any) → ... bit. Like Guido said on that mypy issue, this might be a case where practicality beats purity.

(*args: ..., **kwargs: ...) is an interesting suggestion. It would still be a breaking change for mypy users and it suggests a different semantics to me (maybe something like this thread), but I’m curious to hear what other folks think.

4

One thing that jumps out to me from reading the mypy issue is that for a while there was no way at all to express a Callable ... in a def statement, first because it was inexpressible, then because mypy’s support for ParamSpec was incomplete.

Now it sounds like the following works? (Copying from Subtyping rules for `...` in callable types)

class Concrete:
    def __call__(self, x: int, *, y: int) -> None:
        pass

class Proto5[**P](Protocol):
    def __call__(self, x: int, *args: P.args, **kwargs: P.kwargs) -> None: ...

f: Proto5[...] = Concrete()  # OK (mypy and pyright)

I wonder if we could get rid of the special case and suggest using ParamSpec instead.

I don’t love the (*args: ..., **kwargs: ...) idea because I think it’s rather non-obvious what ... means in this context. In this issue, there’s some support for making a ... parameter annotation mean “inferred type.” I’m not advocating for this, just using it as an example of an also-plausible and entirely different interpretation.

2 Likes
5

As a thought experiment, let’s assume for the moment that the backward compatibility concern for mypy makes it impractical to eliminate this special case.

In some ways, special-casing Any for (*args: Any, **kwargs: Any) is similar to special-casing tuple[Any, ...]. You may recall that we updated the typing spec several months ago to special case tuple[Any, ...] and treat it as a gradual type. The spec now indicates that this form is “bidirectionally compatible with all tuples of any length”. We opted for this interpretation to preserve backward compatibility with mypy even though some of us were uneasy about the inconsistency it introduced. That case feels similar to the problem at hand where we again find ourselves asking whether we should codify a special case because of a choice made in the past by mypy developers.

If we were to adopt this special case, how would we word it such that the specification is clear? Here’s an attempt…

If the input signature in a def statement includes both a *args and **kwargs parameter and both are typed as Any (explicitly or implicitly by virtue of having no annotation), a type checker should treat this as the equivalent of .... Any other parameters in the signature are unaffected and are retained as part of the signature.

def func1(*args: Any, **kwargs: Any) -> None:
    pass

def func2(a: int, /, *args, **kwargs) -> None:
    pass

def func3(a: int, *args: Any, **kwargs: Any) -> None:
    pass

class Proto1[**P](Protocol):
    def __call__(self, a: int, *args: P.args, **kwargs: P.kwargs) -> None: ...

assert_type(func1, Callable[..., None])  # OK
assert_type(func2, Callable[Concatenate[int, ...], None])  # OK
assert_type(func2, Callable[..., None])  # Error
assert_type(func3, Proto1[...])  # OK

class A:
    def method(self, a: int, /, *args: Any, k: str, **kwargs: Any) -> None:
        pass

class B(A):
    # This override is OK because it is consistent with the parent's method.
    def method(self, a: float, /, b: int, *, k: str, m: str) -> None:
        pass

Thoughts?

My preference is to avoid creating a special case here, but if there is consensus that such a special case is needed for mypy backward compatibility, I could live with this compromise. As I said above, it feels similar to the compromise we made for tuple[Any, ...].

6

Carving out a special case could be valuable if it is made to be as powerful as Eric’s proposal above, since we cannot achieve the same kinds of gradual signatures with ParamSpec + Concatenate currently, in which case you can consider me a +1. [1]

The other way to make the special case more consistent is to only allow it for exactly (*args: Any, **kwargs: Any) like @hauntsaninja proposed, although that would introduce some new corner cases. E.g. methods when accessed from an instance vs. when accessed from a class. So you would probably end up with more special cases to make it less surprising, so I’m -0 on this version. It feels slightly better than mypy’s current behavior, but still pretty bad.

  1. Although my hope is that we can express constraints on signatures more elegantly in the future, so it feels like putting the cart before the horse a little bit. But I can also recognize that in all those years we haven’t come up with a better solution, so having something in the meantime, is better than waiting for something that may never arrive ↩︎

7

I’ve incorporated the proposed change into the draft language.

@hauntsaninja, I think that addresses your concerns. Let me know if you have any other thoughts or suggestions.

1 Like
8

The TC has signed off on this change, and it has been incorporated into the typing spec. Thanks to everyone who contributed to the discussion and suggested improvements.