PEP 692: Using TypedDict for more precise **kwargs typing

Speaking of PEP 646, has that at all motivated __typing_unpack__ as well? There’s no mention of that being a concern previously.

I also now remember why PEP 646 feels a bit different compare to this PEP: the extra syntactic use of * in other indexing contexts. This PEP only affects syntax for type checking while PEP 646 opened things up for non-typing uses.

PEP 646 didn’t need a new dunder because there was an obvious runtime meaning for * in that context: iteration. So a *T annotation is approximately equivalent to next(iter(T)) (see PEP 646 – Variadic Generics | peps.python.org for details). For kwargs, there isn’t a similar dunder that works well, which is why we are suggesting a new one.

PEP 646 is certainly much broader than 692, although the non-typing benefits of 646 are fairly marginal.

(Speaking for myself, not the SC, although these questions did come out of discussions with the rest of the SC).

I think the use-case for TypedDict for **kwargs is sensible, and a logical extension of both PEP 646 and PEP 589, but I have strong reservations about the syntactic change. Specifically, the PEP introduces a new meaning of ** in an expression (but only in one specific context), something that’s in line with the use of ** for parameter lists but very different from ** elsewhere in expressions. So I have a few pointed questions:

  • How much does using ** actually win here? The alternative is to use Unpack, which doesn’t seem that bad, especially since it’s an established annotation. Is the use of Unpack really confusing enough, or common enough, or error-prone enough to warrant special syntax? Is the use of ** really obvious enough, and discoverable enough, to be less confusing or less error-prone?
  • How bad would be it be to invalidate type annotations on **kwargs that aren’t Unpack? Do people actually want the current behaviour of annotating **kwargs? Regardless of whether they do, is the current behaviour actually desirable?
  • If the current behaviour isn’t desirable, how bad would it be to transition to using TypedDict semantics by default on **kwargs annotations? How many annotations of **kwargs using the old semantics are in fact correct, rather than misunderstandings or overly broad annotations like Any?
  • I feel like at some point in the journey to type-annotate all Python code we will end up discovering mistakes of the past or better ways of doing things, so having transition processes is probably a good idea. Something similar in concept, if not design, as __future__ imports. Considering the simplicity of the Unpack alternative to the new syntax, and assuming a transition is warranted, would this not be a good testing ground for such a process?

Just so it’s clear, the SC hasn’t decided yet on the PEP, one way or another, but I would like some clarity on the questions above to help make the decision.

It is difficult to give an answer that is not subjective. To me, using ** is intuitive and more concise than Unpack. It’s been proposed both on the original GitHub issue as well as during one of the typing meetups and it seemed to get neutral/positive responses.

It is hard to come up with a motivating example for the current behavior off the top of my head. In general it might be useful for very specific cases where a function expects a variable amount of keyword arguments of the same type and doesn’t care about the keyword names. It doesn’t seem to be a very prevalent use case. At the same time, I remember playing with sourcegraph a couple of weeks back, and I recall that there were a lot of examples of, as you’ve put it, overly broad annotations like Any and misunderstandings like **kwargs: Movie (where Movie is a TypedDict - I assume the author did not mean for every keyword argument to be a TypedDict). At the same time I am wondering to what extent those mistakes stem from the fact that trying something like **kwargs: **Movie is illegal.

I think, given the assumptions you’ve mentioned, that it would be a good testing ground for such a process. That said, it would be a major change and definately would require a lot of effort.

Also, I think it is worth repeating what Guido has mentioned regarding *args and **kwargs

I think this is what makes the current proposal “intuitive”.

Personally I don’t think it’s extremely important, and I’d be OK with the SC approving the new type system feature proposed in the PEP while rejecting the syntax change. The biggest argument in favor of the syntax change is consistency: PEP 646 added new syntax for typing *args with *args: *tuple[int, ...], and it would be odd if the analogous construct for **kwargs didn’t have anologous syntax.

I believe it is uncommon for **kwargs to be annotated as anything other than Any or occasionally object. I looked in our internal codebase and found half a dozen annotations as dict[str, Any] or similar (probably incorrect) and three where the annotation was something meaningful like int. I definitely believe it would have been better if the annotation for **kwargs had worked like you suggest from the beginning.

But changing this now would be the worst kind of backward compatibility break, where currently working code would suddenly mean something else. So to change it, we’d need a careful plan.

Also, if we change this, we’d want to make an analogous change to the meaning of *args annotations.

That’s an interesting idea but we’d have to think more about how to approach it. I suppose it could just be a flag to typecheckers that initially defaults to off, then gets switched after a few releases.

At last week’s typing-sig meeting, we discussed the idea of changing the meaning of **kwargs annotations and the reception was negative. I am planning to add an entry about this to PEP 692’s “Rejected Ideas” section but haven’t gotten around to it yet.

Why not just use a __future__ import as the flag for typecheckers?

Something like

from __future__ import explicit_args_kwargs

def f(*args: tuple[int, ...], **kwargs: dict[str, float]) -> bool: ...

but the future import wouldn’t actually do anything (other than be a signal for a type checker).

The new syntax introduced in PEP 646 would still be needed for things like

from typing import TypeVarTuple

Shape = TypeVarTuple('Shape')
class Array(Generic[*Shape]): ...

but the **kwargs: **MyDict syntax wouldn’t be required anymore, because it would just be **kwargs: MyDict. (And for a TypeVarTuple for *args, it would simply be *args: Ts instead of *args: *Ts.)

EDIT: and if someone doesn’t like the __future__ import, they can always just use Unpack[...].

EDIT2: to make this into a concrete proposal:

  1. the future import is added to Python 3.12
    • type checkers switch to the new behavior in presence of the import
  2. after Python 3.11 has reached EOL, the next Python version (3.16?) makes the new behavior default
    • type checkers usually have a target_version that specifies which Python version they target (I found such a config option in mypy, pyright, pyre (undocumented though), pycharm and pytype)
    • if target_version is set to 3.16 (or whatever the version after 3.12 EOL is), then type checkers should use the new semantics; for target versions below that, like 3.12, type checkers should look for the future import

The most potential disruption would then happen when people start to target Python 3.16 with their type checkers.

4 Likes

I submitted PEP 692: Add changing the meaning of **kwargs annotations to rejected ideas by JelleZijlstra · Pull Request #2916 · python/peps · GitHub adding this to the Rejected Ideas section for PEP 692.