Questions related to `typing.overload` style

Python Help
1

In our project we have a decorator that can be used both as a marker without parameters like @keyword and with parameters like @keyword(name='Example'). You can see a slightly simplified implementation below and the real code on GitHub.

def keyword(name=None, tags=()):
    if isroutine(name):
        return keyword()(name)

    def decorator(func):
        func.robot_name = name
        func.robot_tags = tags
        return func

    return decorator

We’d like to add type hints to the decorator and obviously need to use typing.overload because the return value depends on the arguments. I have some questions related to its usage:

  1. Must argument names match in all implementations? In our case the first argument can either be a name or a function, and using names like name and func would be better than using name everywhere.
  2. Must all arguments be listed with all implementations? In our case no other argument is accepted when passing a function as the first argument.
  3. Is it ok to omit typing from the actual implementation altogether? With complex signatures proper typing will get rather complicated.
  4. Is it ok to use just Callable instead of ugly Callable[[...], Any] when we don’t know anything about the signature of the callable? This isn’t really related to overload, but is relevant in this particular case.

I hope answers to the above would be “no”, “no”, “yes” and “yes”, because it seems to result with pretty clean and easy to understand code (which isn’t always the case with complex type hints):

from inspect import isroutine
from typing import Callable, overload, Sequence, TypeVar


F = TypeVar('F', bound=Callable)


@overload
def keyword(func: F) -> F:
    ...


@overload
def keyword(name: str | None = None,
            tags: Sequence[str] = ()) -> Callable[[F], F]:
    ...


def keyword(name=None, tags=()):
    if isroutine(name):
        return keyword()(name)

    def decorator(func):
        func.robot_name = name
        func.robot_tags = tags
        return func

    return decorator



@keyword
def example1():
    pass


@keyword(name='Example')
def example2():
    pass


print(example1.robot_name, example2.robot_name)

The only problem MyPy reports with the above code is accessing the robot.name attribute at the end and that’s something I’m fine with. Pyright (VS Code) doesn’t seem to like argument names being different, though. That would be trivial to fix, but using name: F when the argument is a function feels wrong. Is Pyright right that argument names must match or is it overly strict here?

2
  1. The argument names are significant, because someone could do @keyword(func=45). What you should do is use / to mark that parameter as positional-only, then type checkers will be perfectly happy with the name changing.
  2. Overloads do not have to have matching signatures to each other, it’s very handy to be able to omit parameters which aren’t valid. Each overload needs to be a valid call for the implementation, of course.
  3. You can omit typing for the implementation, but that’ll mean the types inside the implementation function are all going to be Any. I’d recommend adding types, but it’s just for the internals, it doesn’t affect API. Also, on stricter settings type checkers are going to warn about the missing annotations.
  4. A bare Callable is equivalent to that yes, but similarly strict settings are going to warn about that. You could do AnyCallable = Callable[..., Any], to make it a bit cleaner.
1 Like
3

Here’s a solution that passes type checking (both pyright and mypy) and correctly models the fact that the code is modifying the decorated callable and adding new attributes.

Note that I’ve switched from inspect.isroutine to builtins.callable because the latter is supported by type checkers for type narrowing, whereas the former is not.

from typing import Callable, ParamSpec, Protocol, cast, overload, Sequence, TypeVar

P = ParamSpec("P")
R = TypeVar("R", covariant=True)

class RobotProto(Protocol[P, R]):
    robot_name: str | None
    robot_tags: Sequence[str]

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

@overload
def keyword(func: Callable[P, R], /) -> RobotProto[P, R]:
    ...

@overload
def keyword(
    name: str | None = None, tags: Sequence[str] = ...
) -> Callable[[Callable[P, R]], RobotProto[P, R]]:
    ...

def keyword(
    name: Callable[P, R] | str | None = None, tags: Sequence[str] = ()
) -> Callable[[Callable[P, R]], RobotProto[P, R]] | RobotProto[P, R]:
    if callable(name):
        return keyword()(name)

    def decorator(func: Callable[P, R]) -> RobotProto[P, R]:
        robot_func = cast(RobotProto[P, R], func)
        robot_func.robot_name = name
        robot_func.robot_tags = tags
        return robot_func

    return decorator

@keyword
def example1():
    pass

@keyword(name="Example")
def example2():
    pass

print(example1.robot_name, example2.robot_name)
1 Like
4

Thanks fro replies! It’s a good point that we don’t want @keyword(func='xxx') to be valid. Thanks also for the tip to use positional-only parameters to avoid that while being able to use a meaningful argument name.

I look at the solution by Eric with both awe and horror. It’s awesome that this kind of somewhat complex typing issue can be solved, including attributes added to the function, but all this extra code is far from trivial. We have an open source project and code like this may make it harder for less experienced programmers to contribute, so I wouldn’t like to use it in internal code. This is, however, a public API where typing is important and worth the added complexity.

One reason I’d like to use Eric’s solution is that I’ve read ParamSpec documentation few times, but only with this example I think I got it. Having an example of its usage in our code base would ease using it in other places in the future. Unfortunately ParamSpec requires Python 3.10 and we still support 3.8. We could use typing_extensions, but to keep installation without Internet connection easy, we don’t want to have mandatory external dependencies. I think we can live without the special attributes being recognized for now.