Type hints for kwarg overrides

Typing
1

Opening this discussion to talk about whether is it reasonable to issue a PEP for type hints on kwarg mutations/overrides.

Right now, Concatenate explicitly prohibits overriding existing kwargs. This combined with the fact that the typing spec doesn’t allow implicit override such as…

def component(function: Callable[P, T]):
    def constructor(
        *args: P.args, key: str | None = None, **kwargs: P.kwargs
    ) -> Component:
        return Component( ... )

    return constructor

is a heavy limitation for library maintainers. For example, over at ReactPy, we have several APIs that wrap/mutate the a user’s provided kwargs in a fashion similar to the example above.

2 Likes
Trying to type-annotate decorator that adds default value
2

It seems sensible to have something that performs the same purpose as functools.wraps, but for typing information, including kwargs. I’ve no idea how it should, or would be implemented though, or what the ramifications would be.

3

IIRC the ParamSpec PEP handled this at some point but ultimately cut it out to limit scope. The Rejected ideas section of the PEP is informative: PEP 612 – Parameter Specification Variables | peps.python.org.

4

Perhaps it’s worth exploring whether the described heuristics for banned_names is something that a type checker could automatically infer by inspecting how the ParamSpec is used.

5

I think this is worth exploring, but the reason it isn’t already allowed seems to be that this has the potential to be very complicated. That doesn’t mean we can’t do this, but it does mean that we probably need a PEP that provides a precise specification. Two questions that would need to be answered (there may well be other problems):

  • What is the return type of component, and how would users write it? Conceptually, it’s Callable[new-P, T], where new-P is like P but with a new kwonly arg key: str = None, but you need to find syntax for that.
  • What happens if I pass a function to component that already takes a key kwarg? Should type checkers reject that? If so, how do we encode that restriction in component’s input type annotation?
6

Throwing shots in the dark for potential APIs here…

  • What is the return type of component, and how would users write it? Conceptually, it’s Callable[new-P, T], where new-P is like P but with a new kwonly arg key: str = None, but you need to find syntax for that.

Maybe this is leaning us in the direction of a dedicated KwargSpec that can mutate existing ParamSpec.kwargs values. Since that interface might get overloaded with a lot of functionality in the future, it might be forced to use parenthesis rather than square brackets.

Here’s a quick draft of what that could look like: KwargSpec(P.kwargs, overrides=( ("key", str | None, None) , ( ... ) , ( ... ) )

  • What happens if I pass a function to component that already takes a key kwarg? Should type checkers reject that? If so, how do we encode that restriction in component’s input type annotation?

I could argue that this could be left up to the library maintainers to manually validate. After all, in a technical sense **kwargs could have accepted any value.

If the PEP really had to push for an implementation for this, maybe something like KwargSpec(P.kwargs, excludes=("key", "example_2")) could be a solution.

7

If we ignore hybrid (kw-or-positional) parameters¹, a decorator/higher-kinded-function can do one of the following:

  1. prepend/remove-from-front positional arg(s) (already supported)
  2. append/remove-from-end positional arg(s) when there’s no varargs
  3. add/delete keyword-only arg(s)

The supported use supports excluding functions from being used, e.g. def decor(f: Callable[Concatenate[int, P], R]) -> ... expresses “f needs to have at least one positional argument, and its first positional argument needs to be of type int”.

For keyword-only parameters, we just use name instead of position. So I don’t see how removing positionals from the end or, adding/removing keyword-only args are in any way more complex.

# pop first pos argument of type int.
# Needs≥1 arg, first arg to be int
def pop0_pos(f: Callable[Concatenate[int, P], R]) -> Callable[P, R]: ...

# pop last pos argument of type int.
# Needs≥1 arg, no varargs, last arg to be int
def pop_pos(f: Callable[Concatenate[P, int], R]) -> Callable[P, R]: ...

# pop keyword arg “size” of type int. (placeholder syntax, don’t worry about it)
# Needs a kwarg named “size” of type int
def pop_kw(f: Callable[Concatenate[P, "size" : int], R]) -> Callable[P, R]: ...

The only case where prepending is simpler than the rest is when Concatenate appears in a return-type-only position, but again, we already see it having requirements for the function’s signature in all other cases, so I don’t see why that would be special:

# prepend pos argument of type int.
# No requirements
def prepend_pos(f: Callable[P, R]) -> Callable[Concatenate[int, P], R]: ...

# append pos argument of type int.
# Needs no varargs
def append_pos(f: Callable[P, R]) -> Callable[Concatenate[P, int], R]: ...

# add kwarg “size” of type int.
# Needs `f` to not have an argument named “size”.
def add_kw(f: Callable[P, R]) -> Callable[Concatenate[P, "size" : int], R]: ...

Things like replacing the type of a keyword argument works the same way as replacing the type of the first positional argument, e.g.:

def replace_arg0(f: Callable[Concatenate[int, P], R])
                 -> Callable[Concatenate[str, P], R]:

def replace_kwarg(f: Callable[Concatenate[P, "size" : int], R])
                  -> Callable[Concatenate[P, "size" : str], R]:

¹I don’t think it’s valuable to expand the use of keyword-or-positional arguments, so I think it’s fine to reduce complexity by only having syntax for adding/removing positional-only and keyword-only arguments

1 Like
8

Is “add” available for free if ordered intersection gets added to the language?

9

I don’t know what you’re referring to.

10

See here for example An argument against intersections. Or, having our cake, and eating it too. · Issue #38 · CarliJoy/intersection_examples · GitHub

11

Thanks, that explains the intersection part. But I’m still unsure what you mean with “for free”.

12

As in, could you use an intersection to add keys to the kwargs typeddict you get from a ParamSpec, instead of adding a dedicated new construct? It feels like you should be able to since it can add keys to a typeddict. But looking again at your example, it probably won’t end up with another ParamSpec so maybe not :frowning:

13

As far as I can tell intersections will not help with function signatures. Signatures in general are something we’re just woefully under equipped to express in annotations. The callback protocol route is pretty much a cop out and it does not always work as well as Callable, even when you express the exact same signature.

But Python function signatures are unfortunately very complex, so defining simple, consistent operations on them that can mutate them in some way is very challenging as there are many corner cases to consider, especially once you’re talking about a ParamSpec and not a specific signature. Concatenate just happened to be the easiest case to reason about, since you’re just prepending a fixed number of positional-only arguments to the signature, which always works. Everything else comes with caveats. Even the next most simple thing, using Concatenate for keyword only arguments is amongst the rejected alternatives for PEP 612 for that very reason and you could imagine much more complex cases, such as arbitrarily inserting a new parameter at position N with name X or extending the ParamSpec for VarArgs or KwArgs.

Ultimately we’re already stuck at just being able to come up with a syntax to express a complex signature on its own, without having to use a callback protocol, so we can directly parametrize any generics that use a ParamSpec, maybe once we can do that, more complex mutations on those signatures will be easier to spell out and reason about as well.

14

I still think decorator kwarg spec mutations should be possible with a ParamSpec-like interface.

Here’s my latest API suggestion that I believe is both robust and easy to use.

from typing import ParamSpec, Mutate, Callable, TypeVar

P = ParamSpec("P")
T = TypeVar("T")

# `remove` argument is used to define keys that no longer allowed to exist.
# The value of this argument is a list of keys that is no longer allowed to exist 

# `add` argument is used to define keys that can now exist
# The value of this argument is a list of tuples. Within the tuple, the first value is the kwarg name, second is the kwarg's type, third value (if provided) indicates that the key is optional since it has a default value.
# If defining a key that already exists within ParamSpec.kwargs, then it is assumed to be overridden with the new type hint.

P_Mutated = Mutate(P, remove=["example_key_1"], add=[("example_key_2", str, "default_value")] )

# Here's what I would expect that to look like in practical use
# Note that KwargSpec should only be allowed within function return type hints, and it can only affect kwargs (since Concantenate already handles args)
def example_wrapper(func: Callable[P], T)  -> Callable[P_Mutated, ReturnType]:
    ...
Trying to type-annotate decorator that adds default value