Currently the parameters described by Callable are considered positional-only. For more precise signatures we need to write a callback Protocol.
Since PEP 692TypedDicts can be used to type keyword parameters: def foo(**kwargs: Unpack[TD]), PEP 692 does not specify whether Unpack[TD] is also allowed within Callable[[Unpack[TD]], Any] or not - afaik this is also not done somewhere else.
Mypy does support it since their PR by @ilevkivskyi and I agree with @Jelle’s comment on that PR that is quite obvious what is means: a Callable with keyword-only parameters. But such usage it is currently not specified, I would like to change it and fill that gap.
The natural extension of this idea is the behavior with ParamSpec which is not supported by mypy (tracked in this Issue):
from typing import Callable, TypedDict, Unpack, Any
class Signature(TypedDict):
a: int
type CallableP[**P] = Callable[P, Any]
# New supported types:
type SigCallable = Callable[[Unpack[Signature]], Any]
type SigCallable 2 = CallableP[Unpack[Signature]]
# The last two types are equivalent.
Further extensions that will align with this will be:
extra_items=T (PEP 728) that would allow to type kwargs: T from within Callable.
Inline TypedDicts (PEP 764) that would allow direct inline typing of a keyword-only Callable[[Unpack[TypedDict[{"a":int}]]], Any]. To be noted is that currently Unpack is required as Callable[[**TypedDict[{"a":int}]], Any] is no valid syntax.
Open Question:
Should a shorter syntax Callable[Unpack[TD], Any] be allowed similar to ParamSpecCallable[P, Any]? This needs an update in the typing module. If yes this could allow Callable[**TypedDict[{"a":int}], Any] in the future.
Are multiple TypedDicts allowed Callable[[Unpack[TD], Unpack[TD2]], Any], and what has to be considered? What when they are incompatible in some type, should they follow the dict unpacking rule {**td1, **td2} that TD2 overwrites TD or should type-checkers report an error?
Could there any problematic interactions when using it together with a ParamSpec, Callable[Concatenate[Unpack[TD], P], Any], keywords not covered by TD are covered by P?
I’ve created this mypy playground with most of the parts that should be accepted and rejected. I think some are quite obvious to be rejected when combined with existing spec. So far I did not see any unexpected rejections by mypy - so far as it currently covers this feature.
PS: I am still not sure if I should tag people here when I refer to them (and its not really important).
I agree this feels like a natural extension. The syntax is clunky when combined with inline TypedDicts, but you don’t have to use inline TypedDicts, and we can pursue syntax improvements later (only in new Python versions, while this proposal could help people on existing versions).
This is currently a syntax error, so “in the future” is doing a lot of work here. If we’re adding new syntax, we might want something more elegant.
While this would be nice, I’m also more interested in a more expressive callable type. New syntax rather than expanding Callable would also be my preference. Callable isn’t able to handle a lot of other things beyond kwargs. The limitations here on “how far can this go without new syntax” are more noticeable with decorator factories that return generic functions.
Thanks for putting together the proposal, and thanks to Ivan for prototyping the idea in mypy.
In addition to the open questions you posed, here are a few more that come to mind.
You asked about possible problematic interactions with ParamSpec. Currently, ParamSpecs cannot be used to define signatures that have keyword parameters (other than **kwargs). For example, (*args: P.args, x: int, **kwargs: P.kwargs)
isn’t allowed because x is a keyword parameter that appears between the *args and **kwargs. This limitation was specified as part of PEP 612 when ParamSpec was introduced. If Unpack[TD] can be used in combination with a ParamSpec, it would allow Callable to create this disallowed signature form — one that could not be expressed using a callback protocol. To make this work, the limitation would need to be eliminated for ParamSpec. That’s doable, but it raises additional questions that the original PEP 612 authors deftly avoided answering.
Today, Callable cannot be used to specify a **kwargs parameter. This new feature would effectively enable this capability because all non-closed TypedDicts introduce an implied **kwargs parameter when unpacked. If an extra_items isn’t specified, then there is an implied **kwargs: object. I think it’s fine for Callable to support **kwargs here, but it should be considered as part of the spec. It appears that mypy’s implementation of this feature has a bug in this regard.
class Signature(TypedDict):
a: int
# SigCallable1 and SigCallable2 should be considered equivalent
type SigCallable1 = Callable[[Unpack[Signature]], Any]
class SigCallable2(Protocol):
def __call__(self, *, a: int, **kwargs: object) -> None: ...
# s1 and s2 should be considered equivalent
def s1(x: SigCallable1) -> None: ...
def s2(x: SigCallable2) -> None: ...
def test1(*, a: int, **kwargs: object) -> None: ...
def test2(*, a: int, **kwargs: int) -> None: ...
s1(test1) # OK
s2(test1) # OK
s1(test2) # Should be error, but it is accepted
s2(test2) # Error
It’s not clear to me from the above mini-spec whether positional-only parameters can be combined with keyword-only parameters, as in: Callable[[int, str, Unpack[TD]], Any]. It looks like mypy’s implementation allows this. If they can be combined, do the positional-only parameters need to come first? It looks like mypy’s implementation doesn’t like positional-only parameters that come after the Unpack, which seems reasonable, but the error message is confusing and leads me to believe this wasn’t necessarily considered in the implementation.
I presume that generic TypedDicts are supported for the Unpack? It looks like mypy’s implementation handles this fine.
class Signature[T](TypedDict):
a: T
type SigCallable[T] = Callable[[Unpack[Signature[T]]], Any]
def s1[T](x: T, f: SigCallable[T]) -> T:
return f(a=x)
def test(*, a: float) -> None: ...
reveal_type(s1(1, test)) # float
Maybe if the current typing council is interested it can be revisited? The situation has changed, and it might be better positioned to be accepted this time.
This is certainly something I’ve wished for when trying to define clear callback signatures.
A clear syntactic form for future use and a more verbose but backwards compatible form to improve typing of existing APIs makes sense.
Skimming PEP 677, the main thing I noticed is that while it looks at how other languages define callable type specifications, it doesn’t appear to look at how they use the proposed “(…) → expr” syntax. Having that mean a callable type specification in Python when it’s an arrow function definition in JavaScript would be unfortunate. Flipping the proposed symbol usage (so “->” is reserved as a lambda shorthand while “=>” indicates a callable signature) would avoid that semantic conflict without otherwise increasing the complexity of the proposal.
I was trying to decide if the open question of deferred field evaluation in t-strings was a relevant context change, but I think that really only impacts the lambda shorthand use case, not the callable type one.
I think the typing community was generally very supportive of PEP 677, and many of us were disappointed when it was rejected. For reference, here are the steering council’s stated reasons for rejecting it. I would support revisiting it if there’s a chance it might be accepted.
I’m curious what you think has changed that might result in the SC coming to a different conclusion today. Do you have specific modifications in mind for the PEP that you think would help tilt the odds in favor of acceptance?
Tagging @stroxler, who is the co-author of PEP 677.
I wonder if it’d work better to not have the syntax change limited to typing. Instead, add a new spelling for lambda (or expand the existing one), to include parameter and return annotations. That would has the advantage of solving issues where type checkers can’t infer the right types for lambdas. Then it could also be used in type hints. Potentially syntax for the return annotation could be skipped, in typing contexts we could use the ‘body’/return value of the lambda as the type hint. Might be a little too cute though.
Personally, I like it, because it’s essentially the same as arrow functions in JavaScript/TypeScript[1], but I remember there being opposition to that in the past
which is what I use second most often after Python ↩︎
IMO the advantage is that the existing function def syntax is used, though I realize that there may be cases that this doesn’t cover. It’s also more verbose than some options, but that may be a good thing – I like any nontrivial type to be given a name so that I can refer to it by name.
There could also be a decorator version of this; something like –
I imagine the counterargument to a proposal like that is that it’s already possible to do that with Protocol
from typing import Protocol
class ExampleFuncType(Protocol):
def __call__(self, a: int, b: int) -> bool: ...
I wonder if the combination of the PEP 695 type statement and the other additions (both those already added and open proposals) in the original post here (plus the additional 3 years since the swap to the PEG parser) is enough of a change in the time since the PEP 677 rejection to “tilt the odds”
Let me go over the SC’s arguments at the time, summarizing each cryptically:
“Be cautious with new syntax.” While this looks like a fairly weak argument, it feels like an implicit criticism of the specific syntax chosen. I’ve now spent the past 8 months reading and writing copious amounts of TypeScript, and I have to agree that the TS syntax often feels cryptic, especially since there’s no distinctive token at the start. You can be quite far into a parenthesized list of things before you realize that it’s a lambda or callable type (they both use the => arrow in TS – which is easier because the TS parser always knows whether it’s parsing a type or an expression).
“Callable does work.” This was an easy argument to make because PEP 677 carefully chose not to offer features that re beyond Callable’s capabilities (such as optional args or keyword args). So maybe we should actually propose to support optional and keyword args from day one of the new proposal.
“PEP 677 locks us into a syntax choice.” This is a weird one. Of course new syntax locks you in. But what potential future alternative use of (a, b) -> x were they thinking of? Most likely lambdas, but I think TypeScript suggests that we could reuse the same syntax if we wanted to.
Uneasy with multiple -> tokens. This would be easily addressed by changing the syntax to require that a callable to the right of -> must be parenthesized. (It’s easy to drop that requirement in the future if we end up feeling too constrained by it.)
And their recommendation:
“Prefer low complexity syntax, not requiring PEG to parse.” A repeat of (1) above, so they must feel rather strongly about the syntax. And it matches my intuition after using TypeScript.
Now what shall we make of this? I worry that proposing the same syntax (with or without additional features beyond Callable’s current capabilities) would set us up for another failure. (Pinging SC members on how they might feel is not very effective, they won’t commit until the new or revised PEP is properly reviewed on Discourse and then submitted for SC review.)
Would it be completely insane if Python went its own way here rather than trying to conform to what a few other languages do? My strawman: extend lambda to e.g. support the following syntax (here given by example):
lambda a, b: a * b # Old lambda syntax still works (NOT deprecated)
lambda(a: int, b: str) -> str: a * b # A typed lambda expression -- a function
lambda(int, str) -> str: ... # A type signature (maybe we can make `...` explicitly fail when called)
lambda(a: int, b: str) -> int: a * b # Can be used as a type signature or a lambda function
Note that the 3rd example lambda(int, str) -> str: ... might seem ambiguous – is that a lambda function with arguments named int and str, returning the value Ellipsis, or a type signature? We can either let this be decided by the presence of ..., or make this the way lambda with parenthesized arguments is always parsed. (A parameter would basically be [NAME ':'] EXPR rather than NAME [':' EXPR].)
If people think this is all terrible, let’s forget about it, but the lambda prefix makes at least human parsing a lot simpler (you know what you’re looking at from the first token you see).
How would people feel about renaming the thread to something like “Reviving PEP 677 – a syntax for callable types”? Or maybe splitting the discussion about that topic off into its own thread?
I wonder if it’s reasonable to only allow this syntax in type statements as an alias. This would cut some of the concerns on mixing a function’s return -> with a callable description’s syntax’s -> It probably prevents any catastrophically hard to mentally parse cases too if people are pushed to write it this way, but it does reduce the usability for simple cases where locality of information might be more important (I would hope good alias names counterbalance that, but this is hard to intuit without trying to switch a real codebase to it)
