Extract kwargs types from a function signature

Let’s say I have some function

def do_some_stuff(
    *,
    foo: int,
    bar: float | None = None,
    baz: BazEnum = BazEnum.SPAM,
) -> int:
    ...

And some wrapper around it, for example to add default/error handling:

def do_some_stuff_safe(
    *,
    default: int = 3,
    **kwargs: Any,
) -> int:
    try:
        return do_some_stuff(**kwargs)
    except ValueError:
        return default

I’d like to express that **kwargs must match do_some_stuff signature using the static type system.

Current solutions

In Python 3.12, I see two ways to achieve this:

  • Explicitly re-expose do_some_stuff kwargs in do_some_stuff_safe:
    def do_some_stuff_safe(
        *,
        foo: int,
        bar: float | None = None,
        baz: BazEnum = BazEnum.SPAM,
        default: int = 3,
    ) -> int:
      ...
    
  • Use Unpack + TypedDict combinaison introduced by PEP 692:
    class DoSomeStuffKwargs(TypedDict):
        foo: int
        bar: NotRequierd[float | None]
        baz: NotRequierd[BazEnum]
    
    def do_some_stuff_safe(
        *,
        default: int = 3,
        **kwargs: Unpack[DoSomeStuffKwargs],
    ) -> int:
        ...
    

These solutions have the same issue: we have to duplicate do_some_stuff’s signature information (argument names + types + default value in solution 1), with no mechanism that ensures they don’t go out of sync if we add, rename, delete… an argument in do_some_stuff.

(some possible mechanism)

One could write something in the lines of

if TYPE_CHECKING:
    kwargs = cast(DoSomeStuffKwargs,{})
    do_some_stuff(**kwargs)  # type error if DoSomeStuffKwargs is incompatible with do_some_stuff signature

but that feels pretty hacky, and it does not detect addition of non-required argument.

Idea

I’d like to have a way to state that those kwargs are the ones of do_some_stuff, something like

def do_some_stuff_safe(
    *,
    default: int = 3,
    **kwargs: Unpack[Kwargs[do_some_stuff]],
) -> int:
    ...

This reminds somewhat of ParamSpec in it’s usage, so we could maybe write it like ParmSpec(do_some_stuff).kwargs— allowing to use it with *args too!

Problem / question

My main issue with this idea is that having a function object inside a type annotation feels wrong, at it is not a type nor a special typing object— I think that’s not something done anywhere?

However, the function’s signature is a static available info, so it looks legitimate to rely on it… but I see no way to denote it other than through the function object.

In that extend, I would feel more comfortable with something ParamSpec()-ish (with parentheses) rather than Kwargs[]-ish (whith square brackets).

9 Likes

Maybe just Parameters[do_some_stuff].kwargs (so, similar to Parameters<> in typescript):

def f(x: int, /, y: str, *, z: bool) -> None: ...

FParams = Parameters[f]
def g(*args: FParams.args, **kwargs: FParams.kwargs) -> None:
    f(*args, **kwargs)
4 Likes

I would like to see this solved, exact naming and syntax TBD.
The notable extension to your use case is to consider what happens if I’m defining a wrapper over some 3rd party method.

import foo

def snork(
    *args: Parameters[foo.snork].args,
    *kwargs: Parameters[foo.snork].kwargs,
    encoding: str = "utf-8",
) -> foo.Snorker:
    x = foo.snork(*args, **kwargs)
    x.set_encoding(encoding)
    return x

Questions I have by raising this:

  • can it be extended to cover the return type too? (Too much scope creep?)
  • what happens if foo adds encoding to the method kwargs? I assume that’s a type error, but want to confirm
1 Like

I thought a bit more about that, and it’s likely a bad idea: ParamSpec represent the parameters of a function call, which is quite different of the arguments of a function signature: notably, what “keyword argument” mean is not obvious for a signature:

Since both f(3, 4, z=5) and f(3, y=4, z=5) are valid function calls, what to do with y? (can it be both in FParams.args and FParams.kwargs?)

Yeah, I suppose it’d make sense!

Yes, I think it’s one cool effect if this proposal: if there is a new kwargs “collision” between the function and the wrapper, type checkers would detect it!

3 Likes

I don’t think you’re allowed to use .args without .kwargs. The whole parameter-spec has to be dumped in.

Anyway, I like your idea. It’s related to my super-args idea, which is a common kind of parameter forwarding. If someone wants to write a PEP, it would be great to address both simultaneously. So, for the super case, we could have:

class C:
    def __init__(self, *args: Parameters[super].args, **kwargs: Parameters[super].kwargs):
        ...

Still seems too wordy. Maybe someone can come up with something more compact? Maybe don’t annotate *args, or annotate it with ...?

One extension that might be worth addressing is the case where the caller wants to fill in some of the parameters. E.g.,

def f(*, x: int, y: int) -> None: ...

def g(*args: ..., **kwargs: Parameters[f, ('y')]) -> None:
    return f(*args, y=2, **kwargs)
2 Likes

That’s true for ParamSpec, but for this idea, would that mean that we would have to specify .args even if the function has only kewford-only parameters?

To come back to my original example,

Params = Parameters[do_some_stuff]

def do_some_stuff_safe(
    *,
    default: int = 3,
    **kwargs: Unpack[Params.kwargs],
) -> int:
    ...

would it be mandatory to add *args: *Params.args ? Doesn’t looks very neat :confused:

Yeah, why not! Since super() returns the parent class, not function, I’d imagine an eventual Super type would refer to the parent type (in the style of Self), and so Parameters[Super] would be the parameters of super().__init__.

If I understood you well, it’s like omiting a key in a mapping (like TypeScript’s Omit)? If so, I guess that would be considered as a feature on it’s own and a bit out of scope for a first version of this!

1 Like

One relevant use case is annotating pandas functions like apply, which accept an arbitrary function func and separate arguments for said function’s args and kwargs: pandas.DataFrame.apply — pandas 2.1.1 documentation

At present, I am not aware of a way to properly annotate this function. It should be possible to statically verify that you do not pass in args or kwargs that are incompatible with the func you passed in.

1 Like

I feel like this is a good case for ParamSpec’s behavior to be expanded on, but last I recall, attempts to do so were rejected. (I think last time it was “what if we don’t want to use both kwargs and args”)

Paramspec feels super limiting to only the case where you are transparently passing both *args and **kwargs through and that your function which passes them through accepts them as *args and **kwargs, with only minimal modifications allowed to *args via concatenate and just doesn’t mesh well with a lot of real-world code.

2 Likes

Yes, I think that’s the current state of affairs. As you and the last two comments have noted, that’s an unfortunate limitation.

Would lifting the args/kwargs limitation belong in this PEP or a different one?

I think we would want forwarding for any method. For example:

class Base:
  def f(self, x: int) -> None: pass

class Derived(Base):
  def f(self, *args: Parameters[Super].args, **kwargs: Parameters[Super].kwargs) -> None:
    super().f(*args, **kwargs)  # Type checks okay by definition.

Or perhaps Parameters[Super.f]? Or SuperParameters?

Yes, exactly like omit! Great comparison and great point. Definitely does not need to be in a PEP about parameters.

Also, we may want to make the parameter forwarding more succinct. Instead of:

def f(*args: Parameters[f].args, **kwargs: Parameters[f].kwargs)

maybe we should allow:

def f(*args: Parameters[f], **kwargs: ...)