PEP Idea: Extend spec of Callable to accept unpacked TypeDicts to specify keyword-only parameters

[blhsing (Ben Hsing)]

@guido

lambda a, b: a * b  # Old lambda syntax still works (NOT deprecated)
lambda(a: int, b: str) -> str: a * b  # A typed lambda expression -- a function
lambda(int, str) -> str: ...  # A type signature (maybe we can make `...` explicitly fail when called)
lambda(a: int, b: str) -> int: a * b  # Can be used as a type signature or a lambda function

Thanks Guido for the insightful analysis of points made in the rejection of PEP-677. I like your proposed rectification a lot because like @TeamSpen210 suggested, by expanding on the existing lambda syntax the new syntax would not be limited to serving the sole purpose of type hinting a callable.

@guido

Note that the 3rd example lambda(int, str) -> str: ... might seem ambiguous – is that a lambda function with arguments named int and str, returning the value Ellipsis, or a type signature? We can either let this be decided by the presence of ..., or make this the way lambda with parenthesized arguments is always parsed. (A parameter would basically be [NAME ':'] EXPR rather than NAME [':' EXPR].)

I personally think it’s a good idea to let the absence of : ... to indicate that it’s a type signature rather than an actual lambda function. Even though such an indicator is at the end of the expression it shouldn’t confuse a human parser because by going through all the type hints of parameters along the way it would be already clear to a human that they are not parameters of a function before reaching the end to find no : ....

EDIT: Oops I spoke too soon. When I said it would not be confusing to a human parser when reading type hints along the way I was referring to the lambda(int, str) -> str example because int and str are clearly type hints of positional parameters.

But for a callable with keyword parameters it does look ambiguous until finding no : ...:

lambda(a: int, b: str) -> int        # a type signature
lambda(a: int, b: str) -> int: a * b # a lambda function

[guido (Guido van Rossum)]

I wonder if it’s reasonable to only allow this syntax in type statements as an alias.

I definitely would not want to lose the ability of inlining callable types. In my experience callable types occur most frequently in function definitions that take an ad-hoc predicate or callback function for which making up a name is a chore.

let the absence of : ... to indicate that it’s a type signature

There are plenty of places where a type is expected that are not special to the Python interpreter (the first argument to cast is just one example) and in general leaving out the ;........ part of the lambda might introduce ambiguities if we also allow lambda(........)->:...... in the same position. E.g.

if lambda: None:
    pass

is currently valid; support this implies that

if lambda(): None:
    pass

is also valid and means the same thing; then

if lambda():
    pass

is at least confusing, and would require invocation of the PEG backtracking machinery. I’d rather require the colon and use some other marker to distinguish them.

[blhsing (Ben Hsing)]

@guido

if lambda: None:
    pass

is currently valid; support this implies that

if lambda(): None:
    pass

is also valid and means the same thing; then

if lambda():
    pass

is at least confusing, and would require invocation of the PEG backtracking machinery. I’d rather require the colon and use some other marker to distinguish them.

Ah you’re right, especially considering the one-line form of an if statement:

if lambda(): lambda(): None

If a colon isn’t mandatory for lambda it would be unclear if the above is an if statement with a condition of lambda(): lambda() and a body of None or an if statement with a condition of lambda() and a body of lambda(): None.

[erictraut]

Thanks @guido for the analysis and ideas. I think this is a promising line of thinking.

I unearthed some slides that I presented at a typing meetup where we discussed options for PEP 677. Slide 3 is probably the most informative. It lists the various features supported by a callback protocol and compares / contrasts that with the features supported by each of the syntax options we were exploring at the time. Your lambda proposal is most similar to the variant we called “hybrid syntax” in my slides.

Here’s an updated table that compares the Callable and callback protocol mechanisms with PEP 677 and your proposed lambda syntax. I’ve also added some new rows for features that have been added to the type system in the intervening years.

Feature	Callable	Callback protocol	PEP 677	lambda Syntax
1. Positional-only params
2. Any input signature (…)
3. Positional + keyword params
4. Default args
5. Variadics: *args & **kwargs
6. Keyword-only params
7. async keyword
8. ParamSpec
9. Concatenate
10. TypeVarTuples
11. Overloads
12. Unpacked TypedDict (PEP 692)
13. Type parameters (PEP 695)				?

The lambda syntax is more flexible than the existing Callable mechanism and the PEP 677 proposal. There are still a few features that are possible only with a callback protocol, but I think that’s fine.

I wonder if we could (or should) make line 13 work. That would address a type system feature gap that is often raised in the typing forum. I think the syntax would be straightforward and natural, but the runtime scoping rules might be challenging to implement. I’d look to @Jelle (informed by the work he did on PEP 695) to say whether it’s feasible to implement.

lambda [T] (x: T) -> T: x

Here’s how it would look when used in a return type annotation.

def get_transform() -> (lambda [T] (x: T) -> T: ...): ...

[blhsing (Ben Hsing)]

@erictraut

There are still a few features that are possible only with a callback protocol, but I think that’s fine.

Thanks for the compilation of a feature comparison. The async keyword is actually included in PEP-677 and I think it can also be a possible feature of the lambda syntax.

I agree that type parameters should nicely fit with the lambda syntax.

[erictraut]

@blhsing

The async keyword is actually included in PEP-677

Good catch. I’ll update my table to reflect that.

[Jelle (Jelle Zijlstra)]

That sounds doable, yes. If we were to add typed lambda syntax, I think this would be a natural extension.

[blhsing (Ben Hsing)]

@erictraut

Feature	Callable	Callback protocol	PEP 677	lambda Syntax
5. Variadics: *args & **kwargs
8. ParamSpec

I do wonder how a lambda syntax can support both variadic keyword arguments and ParamSpec at the same time. Which of the two does the following refer to?

lambda(**P): ...

One possibility may be to default to ParamSpec unless it’s annotated:

lambda(**kwargs: Any): ... # variadic keyword arguments
lambda(**P): ... # ParamSpec

Another possibility may be to require an Unpack for a ParamSpec:

lambda(**kwargs): ... # variadic keyword arguments
lambda(Unpack[P]): ... # ParamSpec

I think the latter would be less ambiguous albeit less cute.

EDIT: Ditto for variadic arguments *args vs. TypeVarTuple.

[erictraut]

@blhsing

I do wonder how a lambda syntax can support both variadic keyword arguments and ParamSpec at the same time.

A ParamSpec requires the use of a variadic *args and **kwargs parameter, so I’m not sure what you mean “supporting them at the same time”.

I presume the “typed lambda” syntax would support a ParamSpec in the same way that it’s supported today in a def statement:

lambda [**P] (*a: P.args, **k: P.kwargs): ...

This is a natural extension to today’s untyped lambda syntax (lambda *a, **k: ...).

Similarly for TypeVarTuple:

lambda [*Ts] (*a: *Ts): ...

Are you proposing that the lambda syntax should (in addition to the above) also support a shorter form that deviates from the syntactical rules of the def statement? I guess that Guido’s proposal does already involve some deviation, since he’s proposing that the syntax allow lambda (int, str) -> str: ....

Perhaps it would be better if we just stuck to the same syntax rules used for def statements. It’s admittedly a little more verbose in some cases, but that tradeoff may be worth it. I think that consistency is really compelling (and I suspect the SC will also): the same syntax used to define a typed lambda can also be used to “spell” its type. Plus, this new syntax would be easy to teach because it’s the same syntax that every Python developer already knows for def statements.

[blhsing (Ben Hsing)]

@erictraut

Are you proposing that the lambda syntax should (in addition to the above) also support a shorter form that deviates from the syntactical rules of the def statement? I guess that Guido’s proposal does already involve some deviation, since he’s proposing that the syntax allow lambda (int, str) -> str: ....

Yes, Guido’s proposed syntax of lambda (int, str) -> str: ... is what I thought may cause ambiguity when it comes to supporting both variadics and ParamSpec/TypeVarTuple.

So yeah you’re right that we should stick to the def syntax and require names for positional parameters even if they are positional-only.

But then the next problem is how the compiler can tell whether lambda(a: int, b: str) -> int: ... is a callable type or a typed lambda function. To maximize compatibility maybe we can compile a parenthesized lambda that returns ... into a lambda function with a code object that has a special new flag of inspect.CO_ELLIPSIS to indicate that all the code does is to return ..., so that the resulting function can be called normally but can also be easily identified for callable typing as needed.

[sayandipdutta]

@guido

Note that the 3rd example lambda(int, str) -> str: ... might seem ambiguous – is that a lambda function with arguments named int and str, returning the value Ellipsis, or a type signature? We can either let this be decided by the presence of ..., or make this the way lambda with parenthesized arguments is always parsed. (A parameter would basically be [NAME ':'] EXPR rather than NAME [':' EXPR].)

@blhsing

I personally think it’s a good idea to let the absence of : ... to indicate that it’s a type signature rather than an actual lambda function.

Isn’t it simpler to disambiguate by using def instead of lambda here? I was trying to write things down and see how they look, here’s the result:

# proposed by @Guido
def foo(x: int, y: lambda(): ...) -> int: ...

def foo(x: int, y: lambda() -> None: ...) -> int: ...

def foo(x: int, y: lambda(int, str) -> bool: ...) -> int: ...

def foo(x: int, y: lambda[T1, T2](a: T1, b: T2) -> bool: ...) -> int: ...

def foo(x: int, y: lambda[T1, T2](a: T1, b: T2) -> bool: ...) -> (lambda[R](g: type[R]) -> R: ...): ...

# replace with `def`
def foo(x: int, y: def(): ...) -> int: ...

def foo(x: int, y: def() -> None: ...) -> int: ...

def foo(x: int, y: def(int, str) -> bool: ...) -> int: ...

def foo(x: int, y: def[T1, T2](a: T1, b: T2) -> bool: ...) -> int: ...

def foo(x: int, y: def[T1, T2](a: T1, b: T2) -> bool: ...) -> (def[R](g: type[R]) -> R: ...): ...

# drop `: ...`
def foo(x: int, y: def()) -> int: ...

def foo(x: int, y: def() -> None) -> int: ...

def foo(x: int, y: def(int, str) -> bool) -> int: ...

def foo(x: int, y: def[T1, T2](a: T1, b: T2) -> bool) -> int: ...

def foo(x: int, y: def[T1, T2](a: T1, b: T2) -> bool) -> (def[R](g: type[R]) -> R): ...

I think writing out lambda for anonymous functions have been a pain point for the users; taking this opportunity, if lambda can be shortened to def that could be a DX improvment as well.

Additionally, if dropping : ... is valuable, then if a def-expression does not contain a body, then it can only be a type annotation. But if it does contain a body, it can be used both as type annotation and proper anonymous function. Although I am personally not very interested in using it as both annotation and callable function.

def foo(x: def()): ...    # OK
def foo(x: def() -> None): ... # OK
def foo(x: def() -> None: None): ... # OK

if def(): None:    # OK (but not recommended, perhaps even not-OK?)
    ...

if (def(): None):    # OK
    ...

if def():    # Error (Not a complete expr, going into multi-line-lambda territory)
    ...

if def() -> None:  # Error (for same reason above)
    ...

if def(): ...    # syntax error (expected `:`)

If this is allowed, there would be two ways you could spell anonymous functions (lambda and def), but by the same token, in my view, using def is less ambiguous when juxtaposed with old lambda.

EDIT:

@mikeshardmind

I wonder if it’s reasonable to only allow this syntax in type statements as an alias.

Dropping :... should address this as well:

type IdentityFunction[T] = def[T](T) -> T

[blhsing (Ben Hsing)]

@blhsing

But then the next problem is how the compiler can tell whether lambda(a: int, b: str) -> int: ... is a callable type or a typed lambda function. To maximize compatibility maybe we can compile a parenthesized lambda that returns ... into a lambda function with a code object that has a special new flag of inspect.CO_ELLIPSIS to indicate that all the code does is to return ..., so that the resulting function can be called normally but can also be easily identified for callable typing as needed.

On second thought, why special case ... at all? Just like abstract methods and method declarations in protocols, lambda(): ... can be compiled into normal functions, and when used for typing, only its signature is considered, its body completely disregarded. The type checkers and linters can enforce ... in typing context when desired.

[blhsing (Ben Hsing)]

@sayandipdutta

def foo(x: def()): ...    # OK
if def():    # Error (Not a complete expr, going into multi-line-lambda territory)
    ...

If def() is valid in def foo(x: def()): ... then it means def() is a valid expression by itself, and therefore it has to be valid in if def(): as well because if EXPR: accepts any valid expression.

And if we allow def(): ... on top of def() then we run into the same ambiguity previously discussed for making : ... optional for lambda().

I do like the idea of dropping the noise of : ... by using a different keyword though. I think we can simply allow def() without a colon to be a shorthand for lambda(): ..., which is going to be a typed function whose signature can be used to type a callable in a typing context.

[sayandipdutta]

I was indicating towards if (def()):, i.e. if it appears in places other than type annotation, it must be parenthesized (something close to :=), but that might introduce some special-casing at the parser level. Since def is already a keyword, and currently cannot appear everywhere lambda can appear, I thought this special casing would not be very disruptive. In any case, if needed, def() can be disallowed in favor of def() -> None.

def() <-> lambda(): ... could also be acceptable.

One of the main reasons in favor of dropping :... is, what if I want to return Ellipsis?

[guido (Guido van Rossum)]

Synthesizing a bit…

• I really dislike reusing def – it stands for “define” after all
• I thought a bit about alternate keywords, but callable as a soft keyword can’t work because of the ambiguity of callable(x), and a brand new (even soft) keyword is too much of a pain
• I can let go of (int, int) -> int if it gets in the way; we could leave it for a future PEP
• I agree the : ... is an inelegant eyesore

Maybe we can do this:

• lambda x, y: x*y remains a lambda function
• lambda(x: int, y: str) -> str is a callable signature; the -> ... part is optional; the part in parentheses is exactly as it is for def.
• lambda[T](...) should also be supported, allowing all the same syntactic forms as def
• async lambda ... looks great to me

I’m also not a big fan of PEP 677’s position on ... -> ... -> ...; even though we have another right-associative operator (**), I’d prefer requiring parentheses here. I also expect this to be a rare case except among Haskell fans.

[bsdz (Blair Azzopardi)]

@guido

lambda(x: int, y: str) → str

For a type signature, how about?

lambda(: int, : str) -> str

Or perhaps too noisy with all those colons?

It would be great if we could use a Unicode character λ for lambda. Lean (prover) has some useful input macros/abbreviations; one can type \la and it will be converted to a λ symbol.

[MegaIng (Cornelius Krupp)]

@bsdz

It would be great if we could use a Unicode character λ for lambda.

Use a font that has this as a ligature; IMO python syntax constructs should stay pure ASCII so that most of the world has as little trouble as possible to type this, und adding alternative keywords feels even less clean to me.

[Tinche (Tin Tvrtković)]

@bsdz

It would be great if we could use a Unicode character λ for lambda.

I don’t really see this happening but I’d really love this.

@MegaIng

Use a font that has this as a ligature

One of the problems with the verboseness of the lambda statement is that code formatters break lines in ugly ways. A ligature won’t help here.

[guido (Guido van Rossum)]

@Tinche

One of the problems with the verboseness of the lambda statement is that code formatters break lines in ugly ways. A ligature won’t help here.

But the parentheses will!

The lambda character (which I have no idea how to type) will not happen in the language. I expect that many Python users don’t even know the Greek alphabet.

[jorenham (Joren Hammudoglu)]

For what it’s worth, in Rust (where all function params are positional-only), function (pointer) types look something like this:

fn do_twice(f: fn(i32) -> i32, arg: i32) -> i32 {
    f(arg) + f(arg)
}

I personally wouldn’t mind def(x: int) -> int, as I’ve always associated “def” to mean “function” (I’ve been reading it as “de functie” (“the function” in dutch) for some reason, but that’s probably just me haha).
Writing it as lambda(x: int) -> int doesn’t look much different at first glance, but after thinking about it a bit more, I’m worried that the line between lambda declarations and callable type signatures will become too blurry this way. So unless def is still an option, then something like Rust’s fn might be something that’s worth considering instead.