Trying to type-annotate decorator that adds default value

Typing
1

Coming from languages with static types (C, C++, Java) I really welcome the addition of type checking to Python. So far, though, I often fail to type annotate things that are easy to the Python core language but can’t be expressed to MyPy (or I am missing something).

For example, in our code base, we use decorators to inject parameters from context when not passed explicitly (mostly thread-local variables). This is used for progress reporting, which is too complicated for an example here.

Consider this simple Python decorator to reduce this to basics:

def inject_chunk_size(body):
    def wrapper(*args, chunk_size=None, **kwargs):
        return body(*args, chunk_size=chunk_size or 4, **kwargs)
    return wrapper

@inject_chunk_size
def copy_file(source, target, chunk_size):
    print(f"Copying {source} to {target}, chunk size {chunk_size}.")


copy_file("src.py", "tgt.py")
copy_file("backup.zip", "/nas/backup.zip", chunk_size=128)

Output:

Copying src.py to tgt.py, chunk size 4.
Copying backup.zip to /nas/backup.zip, chunk size 128.

I fail to add type annotations to this. I tried to simplify the task, changing it by

  • make chunk_size a position argument
  • make it the first argument
  • drop the ability for override it
  • add type annotations

ending up with this code:

from typing import Callable, Concatenate, ParamSpec, TypeVar

_P = ParamSpec("_P")
_T = TypeVar("_T")

def inject_chunk_size(body: Callable[Concatenate[int, _P], _T]) -> Callable[_P, _T]:
    def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _T:
        chunk_size = 4
        return body(chunk_size, *args, **kwargs)
    return wrapper

@inject_chunk_size
def copy_file(chunk_size: int, source: str, target: str) -> None:
    print(f"Copying {source} to {target}, chunk size {chunk_size}.")


copy_file("src.py", "tgt.py")
copy_file("backup.zip", "/nas/backup.zip")

Obviously, that’s not what I want but given that I can type-annotate it when using the first positional argument for the chunk size, I thought it must be possible to do it when using a named argument?!

A similar discussion was opened here: Type hints for kwarg overrides.

The difference is that I try to motivate that it is possible with positional arguments and argue that it should work therefore with keyword arguments as well.

2

Forgot to add links:

3

Changing the signature of a decorated function, such that a parameter gains a default value, requires the use of a callable protocol, but even that seems to have limitations:

from collections import abc
from typing import Protocol, Concatenate


class ChunkSizeDefault[**P, R](Protocol):
    def __call__(
        self, chunk_size: int = ..., *args: P.args, **kwargs: P.kwargs
    ) -> R: ...


def inject_chunk_size[**P, R](
    body: abc.Callable[Concatenate[int, P], R], /
) -> ChunkSizeDefault[P, R]:
    def wrapper(chunk_size: int = 4, *args: P.args, **kwargs: P.kwargs) -> R:
        return body(chunk_size, *args, **kwargs)

    return wrapper


@inject_chunk_size
def copy_file(chunk_size: int, source: str, target: str) -> None:
    print(f"Copying {source} to {target}, chunk size {chunk_size}.")

image

Does not seem possible to place the parameter anywhere but the beginning of the parameter list.
(And as the signature in the image suggests, to make use of the default you’re forced to pass the remaining arguments as keywords)

4

There unfortunately isn’t a way to do what you want to. A function annotated with an unpacked ParamSpec can only accept additional arguments if they are positional only. That is, even when using protocols you can only write this:

class Something[**P, R](Protocol):
    def __call__(self, extra_arg: int, *args: P.args, **kwargs: P.kwargs) -> R: ...

and not

class Something[**P, R](Protocol):
    def __call__(self, *args: P.args, extra_arg: int, **kwargs: P.kwargs) -> R: ...

and the extra_arg will be treated as though you actually wrote

class Something[**P, R](Protocol):
    def __call__(self, extra_arg: int, /, *args: P.args, **kwargs: P.kwargs) -> R: ...

It works this way because we could otherwise end up declaring invalid functions because the added name already ocurrs within the ParamSpec. E.g. let’s say you have this:

class WithCount[**P, R](Protocol):
    def __call__(self, *args: P.args, count: int, **kwargs: P.kwargs) -> list[R]: ...

def call_often[**P, R](func: Callable[P, R]) -> WithCount[P, R]:
    def inner(*args: P.args, count: int, **kwargs: P.kwargs) -> R:
        return [func(*args, **kwargs) for _ in range(count)]
    return inner

@call_often
def do_thing(a: int, b: int) -> int:
    return a + b

@call_often
def do_thing_with_count(a: int, b: int, count: float) -> int:
    return count * (a + b)

The definition of do_thing works exactly as you’d expect, P captures (a: int, b: int) and the new signature becomes (a: int, b: int, *, count: int) -> list[int]. For do_thing_with_count we have call_often say that P is bound to (a: int, b: int, count: float). But then how do we add a new parameter also called count of a different type to that signature?

Even if we did make some rule for this, consider how the arguments will be passed when the function is invoked. We call the decorated function with something like do_thing_with_count(1, 2, count=3.5). This effectively becomes a call of inner(1, 2, count=3.5), i.e. its args are (1, 2) and kwargs is an empty dict. It then calls the original do_thing_with_count like this do_thing_with_count(1, 2). This will be a type error since we’re only passing two arguments to a function expecting three.

To make this work you’d need the signature of call_often to say that its argument is a callable whose keyword arguments do not already include one called count. Then you can safely add that parameter to it without causing any typing or runtime problems. AFAIK the reason this hasn’t been implemented is that it’s somewhat niche and wasn’t a core part of the original ParamSpec PEP. It also comes with a bit of bagagge since you’d want some kind of nice syntax for it, but we’re already looking for good syntax for function types and signatures in general. So it’s kinda stuck behind first getting that right and then being able to work something out that works with it.

5

Ok, but that could be a type error at the wrapping site. We’re throwing the baby out with the bathwater if we just forbid this completely.

1 Like
6

How does the type checker see this? We have a value of type (a: int, b: int, count: float) -> int and a function of type (Callable[P, R]) -> ... that is being applied to it. Unification says that we just have P capture (a: int, b: int, count: float) and R gets int. The type checker can’t see that we actually don’t want P to capture signatures that contain a parameter count.

Of course, it is theoretically possible to implement this. But we then run into the issues I mentioned in the last paragraph, namely that someone would have to actually do the work, and that we’d need syntax for it (and probably also actual syntax for signatures/callables in general). I agree with you that it’d be great if this were possible to do this in Python right now, I’m just informing Torsten that it currently isn’t.

7

I aplogize if my comment came across dismissively; it wasn’t meant in that tone. I’m just communicating from a place of frustration; more than one person has asked me in the last year about decorators that add keyword args to functions and it’s very unfortunate typed Python has no answer for this currently. To me it sounds like there’s no theoretical barrier against this, just practical ones.

8

No worries. And I totally share your annoyance with this. AFAIK pretty much everyone agrees that this only has some implementation details that would need to be ironed out, but the problem is that no one is able to come up with a good solution to them. As annoying as it is for syntax to be responsible for blocking a feature, we also don’t want to end up with a horribly unergonomic way to write this and be stuck with it. The steering council has also held the stance that type annotations should stay regular Python expressions since they are a completely optional add on to a fundamentally untyped language. So we can’t just declare whatever syntax we’d want for signatures/callables and ParamSpec specifiers. It’s a pretty unfortunate combination of these restrictions that there isn’t a great way to do this, or at least no one has proposed one so far.

1 Like
9

The ParamSpec PEP mentions this and that it was decided to defer in order to produce a simpler PEP.

I suspect this design space is hairy to get right.