Thanks for the explanation. I guess this makes sense to me - it’s harder to run into issues with dependency conflicts that way. The proposal doesn’t encourage that though, that was just a question that made me curious.
I like what you proposed, I think it’s reasonable - it still says that the function was implemented using **kwargs and provides the information about what is actually expected.
Is the extra ** really necessary in Sphinx output?
def f(**kwargs: {"bar": int, "spam": str})
It seems perfectly comprehensible without (maybe even moreso, with less clutter/distraction).
For those who question why not use keyword or keyword-only arguments, here’s a motivating example with a matplotlib-like API:
class Line2DKwargs(TypedDict):
...
color: str | Color
linestyle: str
linewidth: float
...
class Line2D:
...
def __init__(self, x, y, **kwargs: **Line2DKwargs):
...
def plot(xdata, ydata, **kwargs: **Line2DKwargs):
return [Line2D(x, y **kwargs) for x, y in zip(xdata, ydata)]
Repeating the kwargs of Line2DKwargs in both plot() and Line2D.__init__() will be PITA especially for deeply nested calls. Most documentation would either repeat the kwargs info in the docstring OR make a reference to the receiving object.
Absolutely. That’s a perfect example of a case where being able to copy (and modify) the signature of another callable would be perfect.
Having to define a class Line2DKwargs that tracks the signature of Line2D is error-prone, duplicates work, and ties the code to a specific version of the Line2D class. Particularly if Line2D isn’t a class you wrote, but one from a 3rd party library.
So yes, we need something to handle the case of wrapping another callable, but PEP 692 isn’t a good solution.
For passing keyword arguments to a 3rd party callable,
If 3rd party callable has typing in its keyword arguments, then introspection of the signature would be optimal (similar to ParamSpec).
If 3rd party callable does not have typing in its keyword arguments, then you’ll have to maintain your own typed signature (e.g., via keyword-only arguments or this PEP). This necessarily tracks a version of the 3rd party signature.
Note: this is currently possible with TypedDict, however without unpacking:
def plot(x, y, plot_kwargs: Line2DKwargs):
return [Line2D(x, y **plot_kwargs) for x, y in zip(xdata, ydata)]
So, this PEP does not provide a solution for situation (1), but does provide another solution for situation (2). The cons you’ve listed is necessary for typing situation (2).
I think wrapping a callable function while related is fundamentally different and not what is needed for this use case. We already have a way to wrap callables for decorators using Paramspec. A simple example is you can do,
def log_decorator(func: Callable[P, R]) -> Callable[P, R]:
def _func(*args: P.args, **kwargs: P.kwargs) -> R:
logger.info('some debug stuff")
return _func(*args, **kwargs)
return _func
The Line2D example is fundamentally different. We are not wrapping all of arguments of Line2D, but only **kwargs portion. We do not wrap x, y arguments it has. Even if we had Signature[Line2D] type tool it would not capture right thing.
If we did follow path of Signature[Line2D] we’d have two options,
If we had this pep then 2 makes some sense as a way to refer to kwargs type without knowing name of annotation. But without this pep there’s no way to provide information about color/linestyle and have useful validation occur. You can’t wrap signature of Line2D in a useful manner without first having a way to type hint signature fully of Line2D.__init__. Paramspec pep is devoted to wrapping callables, but wrapping an untyped argument similarly has little meaning.
Also this pep fits in well with situation of *args. When type system was started in pep 484 you just did,
def foo(*args: int):
...
to mean all arguments are int. With PEP 646 (variadic generics pep) it is possible to do heterogenous arguments in similar fashion as this one,
def foo(*args: *tuple[int, str]):
...
This example with *args is already accepted for 3.11 and can be used right now in some type checkers to mean one int and one string argument. I think there’s less value of doing explicit expansion with *args, but we have better sense that this pattern fits and we have 1 type checker already supporting this PEP provisionally, with a draft implementation for mypy too.
Even if we were to go back in type, it’s unlikely dict[str, int] way of doing kwargs would make much sense. Back then typeddicts didn’t exist. And there are a bunch of functions where them being same type is fine (often object is type doesn’t matter much).
I’ll admit downside of confusion. I’ve seen a number of people try to write,
def foo(**kwargs: dict[str, object]):
...
or similar and not realize that’s probably not what they want (maybe linter could warn for this). But I consider that confusion to be orthogonal to this PEP given large backward compatibility break if we changed that.
The current version of the PEP is unclear on what changes will be made to typing.Unpack.
The 3.11 docs say that Unpack can be used interchangeably with * in the context of types. And you can do things like:
>>> Unpack[Animal]
*<class '__main__.Animal'>
>>> Unpack[123]
*123
Will this change?
It seems that the repr should change to **, so this would be relevant at runtime.
Good point. Unpack[] will be available as an alternative to both * and ** in different contexts. I think we should change its repr() to just Unpack[T].
Wouldn’t the type of the argument (TypesDict or not) be a good enough hint? Or rather the presence of the new dunder?
There is an example in the Grammar Changes section that, along with the new dunder description, implicitly introduce the change of Unpack’s repr to Unpack[T]:
>>> def foo(**kwargs: **Movie): ...
...
>>> foo.__annotations__
{'kwargs': Unpack[Movie]}
But I agree it would be worth to state this explicitly. Not sure though if I can make a quick fix now after the PEP has been submitted to the SC for review?
Submit a PR to the peps repo and add Petr as a reviewer.
The SC discussed this PEP in the last meeting and we had a couple of questions.
One, does anyone have any feeling or idea how much the current **kargs: <value type> shorthand is used? This leads into …
Two, the PEP feels like it is trying to use syntax to work around the **kwargs shorthand as if it’s been found insufficient. Now that dict make defining a dictionary type hint easier, the SC was wondering whether it would make more sense to explicitly type the **kwargs argument by transitioning away from the shorthand and instead to explicitly typing to the actual type of the parameter? We realize this would mean developing a way to tell type checkers which syntax was being used (effectively a __future__ solution for types), but perhaps that’s the best solution long-term? Mechanically changing code from **kwargs: str to **kwargs: dict[str, str] should be reasonably straightforward and we couldn’t think of any nasty transition hurdles to block such mechanical change. All of this would mean no new syntax would be needed and the type annotation for **kwargs would be expected to match what the parameter’s actual typing is. Or are we misinterpreting the PEP somehow and its intent?
I don’t know how often it’s used, but I don’t think it’s rare exactly. Someone who is good at searching GitHub can provide a better answer.
I don’t think it would be practical to just transition to **kwargs: dict[...] unless you also transition *args: T to *args: tuple[T, ...] at the same time, since the two syntax forms are supposed to be similar (“rhyming” is the word that comes to mind). But I think for *args we’ve gone a different way.
context:global **kwargs:… - Sourcegraph one sorta of search for how common. Looks like most of the time people type hint **kwargs: Any/**kwargs: object mostly leaving it as anything. There’s also some P.kwargs when using paramspecs. I only searched pyi files because .py files most hits for **kwargs: are docstrings.
If this was guarded under future import + deprecation warning for non dict/typeddict type it feels plausible migration. Maybe pyupgrade/similar tool could automate this. I think this would be first future import to impact pyi files as typeshed would likely need to do 1 time migration.
A more comprehensive search looking for kwargs in functions definitions, including .py files: context:global \*\*kwarg… - Sourcegraph
Good point.
Is there a reason to not go a similar way for **kwargs? A TypeVarDict would be equally as unambiguous as to what the type hint is meant to convey without requiring new syntax. And if there’s a desire to have *args and **kwargs “rhyme”, then would that make more sense?
Could you clarify why this wouldn’t require a new syntax? For *args as described in PEP 646 we still need new syntax, not only for *args but also in indexes.
Then, if I understand correctly, this could be the only reason not to go similar way for **kwargs, because then we wouldn’t have to allow for the ** in indexes. As a consequence we wouldn’t allow for things like:
D = TypeVarDict("D")
def foo(**kwargs: **D) -> TypedDict[**D]: ...
But to me it seems like allowing for things like that could be proposed in a seperate PEP (if there’s ever a need), especially since allowing for unpacking dicts like this would be quite complex, for example - how should the dicts be combined in case of TypedDict[**D, Movie]? It is not obvious how to handle duplicate keys and differences in totality.
PEP 646 set out to solve a different issue where introducing a TypeVarTuple was necessary. For the purposes of this PEP introducing something like TypeVarDict isn’t.
Given that and the the fact that we would like to make *args and **kwargs rhyme, I think it makes sense to introduce the new syntax, but only limit the unpacking to concrete TypedDicts.
The best analogy between PEPs 646 and 692 is not to TypeVarTuple, but to something like def f(*args: *tuple[int, str]), which allows precisely typing heterogeneous *args. This is one of the new features of PEP 646, but it is essentially independent from TypeVarTuple.
A “TypeVarDict” would be a far more complicated feature and no use case has been presented for it.
Sorry, I misread/misremembered the PEP and thought the fact that TypeVarTuple was specified as the type that the type checkers knew what to do w/o having * on the type hint itself.