PEP Idea: Extend spec of Callable to accept unpacked TypeDicts to specify keyword-only parameters

[blhsing (Ben Hsing)]

@guido

• lambda x, y: x*y remains a lambda function
• lambda(x: int, y: str) -> str is a callable signature; the -> ... part is optional; the part in parentheses is exactly as it is for def.

Sacrificing the possibility of a typed lambda function to achieve a clean and capable callable signature sounds like a good compromise to make. +1

@jorenham

I personally wouldn’t mind def(x: int) -> int, as I’ve always associated “def” to mean “function” (I’ve been reading it as “de functie” (“the function” in dutch) for some reason, but that’s probably just me haha).

I wouldn’t mind def(x: int) -> int myself either since it means that support for a typed lambda is then potentially possible with lambda(): ..., but def does read like “define” to me (who speaks zero Dutch) so I understand why Guido dislikes the use of def as a noun.

@jorenham

So unless def is still an option, then something like Rust’s fn might be something that’s worth considering instead.

Unfortunately a soft keyword that acts like a function can’t work because it would be ambiguous as to whether fn() should resolve to a keyword or a regular name.

[blhsing (Ben Hsing)]

@guido

I thought a bit about alternate keywords, but callable as a soft keyword can’t work because of the ambiguity of callable(x), and a brand new (even soft) keyword is too much of a pain

How about we make Callable a soft keyword that resolves into a keyword if and only if it is followed by parentheses and an arrow? The arrow is then mandatory for typing a signature:

Callable(a: int, b: str) -> Any # typing signature
Callable(a, b) # calling Callable

I prefer Callable as a soft keyword over callable because Callable is currently used with square brackets for typing a callable so there aren’t nearly as many existing codes that call Callable as a callable compared to callable.

This will require backtracking to fall back from Callable() -> ... to Callable() but since there aren’t many existing codes (only 10.6k matches on GitHub) that call Callable() I don’t think backtracking will occur often enough to cause performance concerns.

[ZeeD (Vito De Tullio)]

@jamesdow21

We’ve usually ended up just replacing int_checker = Checker(lambda x: isinstance(x, int)) with

def _int_check(value: object) -> TypeGuard[int]:
    return isinstance(value, int)

int_checker = Checker(_int_check)

FWIW you may use Checker as a decorator

@Checker
def int_checker(value: object) -> TypeGuard[int]:
    return isinstance(value, int)

[sayandipdutta]

@guido

• lambda(x: int, y: str) -> str is a callable signature; the -> ... part is optional; the part in parentheses is exactly as it is for def.
• lambda[T](...) should also be supported, allowing all the same syntactic forms as def
• async lambda ... looks great to me

This syntax works well IMO. And on further thoughts, I am beginning to dislike def as well; definition should not be an expression. Although it would be nice to find something shorter than lambda, I am ok with this compromise.

But these are the scenarios when I really wish we had PEP 638 macros. Hopefully it could enable something like: fn!(x: int, y: str -> str).

[Daraan]

@yangdanny97

@Daraan

Callable[[Unpack[TypedDict[{"a":int}]]], Any]

I’m worried about readability, that’s a lot of nested brackets for a simple example.

It is definitely no beauty. With InlinedDicts PEP it would be the just the natural consequence that any variable TD of a TypedDict can be replaced with TypedDict[{...}].

[Daraan]

@Jelle

@Daraan

If yes this could allow Callable[**TypedDict[{"a":int}], Any] in the future.

This is currently a syntax error, so “in the future” is doing a lot of work here. If we’re adding new syntax, we might want something more elegant.

I am indifferent towards a **TD unpacking and for the same reason as PEP 692 I do not really want to propose it.
It would add one more use-case to motivate a possible Unpack replaced with */** to the discussion.
Maybe to summarize, I think would be unexpected that type C[*Ts] = Callable[[int, *Ts, Unpack[TD]], Any] is accepted syntax but: type C[*Ts] = Callable[[int, *Ts, **TD], Any] isn’t.
Alternatively type C[*Ts] = Callable[[int, *Ts, *TD], Any] would be a middle ground that does not require changes to the syntax that should be doable without much overhead.

[Daraan]

@jpe

Has there been any discussion of using something like the following to specify callable types –

def example_func(a: int, b: int) -> bool: ...
type example_func_type = typing.Signature[example_func]

Yes there are some ideas that go in that direction, in fact I started a discussion about this as well here, a larger one was made before here as well.
I remember at least one more, likely linked in the later one. There are quite a few flavors that exist for that idea.

[Daraan]

@guido

How would people feel about renaming the thread to something like “Reviving PEP 677 – a syntax for callable types”? Or maybe splitting the discussion about that topic off into its own thread?

I think the discussion should be split as it is two different topics and goals
Unpack[TD] in Callable is something that should be doable more easily without much needed changes. Reviving PEP 677 would be the long run for something completely new.

[Daraan]

First of all thank your very for the comprehensive reply.

@erictraut

possible problematic interactions with ParamSpec. Currently, ParamSpecs cannot be used to define signatures that have keyword parameters (other than **kwargs). For example, (*args: P.args, x: int, **kwargs: P.kwargs)
isn’t allowed because x is a keyword parameter that appears between the *args and **kwargs. This limitation was specified as part of PEP 612 when ParamSpec was introduced. If Unpack[TD] can be used in combination with a ParamSpec, it would allow Callable to create this disallowed signature form — one that could not be expressed using a callback protocol. To make this work, the limitation would need to be eliminated for ParamSpec. That’s doable, but it raises additional questions that the original PEP 612 authors deftly avoided answering.

I figured that the keyword in between ParamSpec problem would come flying around again. Its again been a while since I read about it. I know you posted a great link somewhere, but I could not find it again. Maybe it lead to the respective PEP discussion here?
This point might best discussed separately?

@erictraut

If an extra_items isn’t specified, then there is an implied **kwargs: object. I think it’s fine for Callable to support **kwargs here, but it should be considered as part of the spec. It appears that mypy’s implementation of this feature has a bug in this regard.

Right, without closed=True, any extra items in kwargs are allowed.

Nice catch. It seems to be more typed like Any by mypy or rather Unknown when speaking in pyright terms. I took a glance on their PR and didn’t see any special handling, e.g. fixing it to Any.

@erictraut

It’s not clear to me from the above mini-spec whether positional-only parameters can be combined with keyword-only parameters, as in: Callable[[int, str, Unpack[TD]], Any].

Yes I think positional arguments should be allowed as well. I’ll do a revisit on my first post soon. A TypeVarTuple looks fine to me as well. Anything else afterwards should be prohibited, maybe expect further Unpack[TD2]s and maybe maybe a ParamSpec.

@erictraut

I presume that generic TypedDicts are supported for the Unpack? It looks like mypy’s implementation handles this fine.

You do not mean Unpack[T], just like in your snipper right? Unpacking a generic TypedDict should be allowed. Should I mention this, it feels like a given to me.

---

@Daraan

Are multiple TypedDicts allowed Callable[[Unpack[TD], Unpack[TD2]], Any], and what has to be considered? What when they are incompatible in some type, should they follow the dict unpacking rule {**td1, **td2} that TD2 overwrites TD or should type-checkers report an error?

For multiple TypedDicts [Unpack[TD1], Unpack[TD2]] it might be the most reasonable to treat it like an implicit subclass class TD(TD2, TD1) that is then unpacked - and raise errors like when creating that subclass. This is likely the safest options at the cost of intended overwriting in case of a key conflict.
However, as such a decision would lock us in, I might be wiser to not allow multiple Unpack (for now). At the cost of flexibility, users can make a subclass with positional adjustments themselves.

However, I figure some still might favor a quick inline injection of an additional keywords if PEP 764 comes to be: [Unpack[ResuableTD], Unpack[TypedDict[{"new_keyword": int}]].

Just as a side note: Unpack[TD1, TD2] is theoretically still left open for future-use and a a possible re-interpretation, that could also be used for **kwargs typing as.

I am still very interested in hearing thoughts and opinions on this matter, if [Unpack[TD1], Unpack[TD2]] should be allowed and which how it should be potentially treated by a type-checker.