That seems like a reasonable list of requirements.
Might I also add to the list the ability to use the notation in-line in an annotation? In past discussions it has come up repeatedly that we need an inline form (like Callable is, but with all of the above options). Yes, if it becomes unreadable through nesting, by all means use a type alias or one of the more verbose alternatives (Protocol or a dummy function def), but if that’s the only way, we have not solved the problem.
Would it be reasonable to use the soft-keyword nature of type to allow type def (...) (Where ... is an expression of parameters) to remove ambiguity of certain symbols in a general context? If so, is that possible to do inline without creating issues of parsing? it seems so to me, but I think this increases the complexity of parsing, and I’m not sure if it’s the best way here, however this could allow:
without the concerns of < > also being operators.
I’m not sure I agree, but I would definitely prefer if the goal was inline use first, and only drop down to an improvement on ergonomics of an alias if we can’t come up with something unambiguous when inline.
or it could just mirror normal function declaration entirely like that, minus the name…
So, there seems to be some support here for a syntax with a keyword (which can be used to bind type vars to). I agree that Fn would be a nice keyword, but it would have to be a soft keyword, and as I understand it, soft keywords aren’t feasible in expressions. This only leaves def (or perhaps lambda, but that seems even more confusing).
The other question is what to do with other types which expect a signature (i.e., those with ParamSpec):
from typing import Callable
class CallableWithFlag[**P]:
def __init__(self, func: Callable[P, None]):
self.func = func
self.flag: bool = False
def __call__(self, *args: P.args, **kwargs: P.kwargs) -> None:
self.flag = True
return self.func(*args, **kwargs)
def f(c: CallableWithFlag[[int, int]]) -> None:
c(1, 2)
assert c.flag
It would be nice if we could write something like
CallableWithFlag[def(x: int, *, y: str = "foo")]
which currently cannot be expressed at all.
But it would be quite disappointing if Callable still had to be written like this:
type MyFunc = Callable[def(x: int, *, y: str = "foo"), None]
instead of something like this:
type MyFunc = def(x: int, *, y: str = "foo") -> None
But it’s also weird to have two different syntaxes?
Could we define a new special method that gets called if you write this?
from typing import Callable as Fn
Fn(...) -> ...
Fn.__<something>__(def(...) -> ...)
Or should we reuse __class_getitem__()?
from typing import Callable as Fn
Fn(...) -> ...
Fn[def(...) -> ...]
Fn.__class_getitem__(def(...) -> ...)
But as @MegaIng pointed out, this could get quite unreadable.
It’s feasible for the new PEG parser, because of the -> but imo it’s a bad idea for human parsers. Seeing Fn(...) in an expression will make people assume it’s a function call. It also further complicates highlighting of code (match and case can somewhat easily be identified by noticing the: at the end of the line for example)
I think angular brackets <> are our best bet for something visually distinctive from both function calls and scalar generic parameterization. It also should be fairly easy to parse[1], because < at the start of an expression should always be distinguishable from an operator, since a binary operator cannot start an expression[2], I don’t even think we need to leverage the PEG parser to make this happen.
The main argument against using angular brackets imho, would be that they’re a powerful tool we haven’t yet used and they could potentially be used for a generally more expressive typing grammar, rather than limiting ourselves to parameter types, so by using them prematurely we’re potentially closing off other promising avenues.
We could allow both a parameter list and a full signature, the latter of which would be equivalent to Callable.
foo: MyCallable[<x: int, *, y: str = "foo">, None]
bar: <(x: int, *, y: str = "foo") -> None> # same as Callable[<...>, None]
We potentially could prefix the angular bracket with something like the proposed “Fn”, although at that point it would once again be difficult to distinguish from the operator.
And Fn{}? That’s also unused.
AFAIK nowhere else in Python are < and > paired up – and in this case, not even paired in an intuitive way since there is going to be an intervening > that does not end the pair! I don’t think there’s any other case where a character is sometimes standalone and sometimes paired up based on surrounding context? I hope this isn’t chosen as it looks ugly to me.
This would solve a number of the problems that happen in complex terms by making the delimiters explicit, but my prior is that getting a PEP approved using < / > is pretty unlikely.
It doesn’t “feel” very pythonic at least to me even if it has clear advantages (which I agree it does), and my instinct is that it’s tough to get buy-in for syntax that doesn’t feel pythonic.
It also doesn’t give us a place to bind a type variable, which I would find a little disappointing (although that would be easy enough to add, e.g. something like <[T](x: T) -> T>).
rustc’s parser and Rust in general (maybe other languages too) suffer from this problem too. The workaround is not pretty.
Here’s a proposal that covers both the Callable case and also the “signature” case (explicit specification of ParamSpec values).
I agree that Fn (or fn) would be a better keyword, and it’s also true that Fn would work as a soft keyword in something like Fn (int) -> str because we can backtrack from the ->, but it wouldn’t work for the “signature” use case, where we don’t specify a return type (see below). So, I still think def is the only option.
I think the easiest way to demonstrate this use case is something like this:
from collections.abc import Callable
type CallableReturningInt[**P] = Callable[P, int]
This type alias can be used like this (works in pyright and mypy master):
def f(x: int) -> int:
return x
g: CallableReturningInt[[str]] = f # Error: "str" incompatible with "int"
h: CallableReturningInt[[int]] = f # OK
But we are severely limited in what kind of signatures we can specify for CallableReturningInt. We can only specify positional arguments: CallableReturningInt[[str, int, float]]. It would be nice to be able to specify more complex signatures.
def followed by parentheses (, ), now creates a new object called Signature:
sig = def (x: int, /, y: str, *, z: bool)
# Equivalent to
sig = Signature([
Parameter(name='x', kind=POSITIONAL_ONLY, annotation=int),
Parameter(name='y', kind=POSITIONAL_OR_KEYWORD, annotation=str),
Parameter(name='z', kind=KEYWORD_ONLY, annotation=bool),
])
These signatures can then be used in types as the value for a ParamSpec:
type IntToInt = CallableReturningInt[def (x: int)]
But with the benefit that static type checkers can understand this now, and can treat it as roughly equivalent to
type IntToInt = CallableReturningInt[[int]]
but more precise, because we were also able to give the name of the function parameter.
The most common use of signatures will be in Callables, so they get a special syntax:
def followed by parentheses (, ), and then followed by ->, is syntactic sugar for a Callable with the given signature and return type:
type MyFunc = def (x: str) -> int
# Equivalent to
type MyFunc = Callable[def (x: str), int]
# Which in turn is roughly equivalent to
type MyFunc = Callable[[str], int]
def to define not functions but signature objects and types might be confusing to peopledef might be even more confusingThis solves two problems at once: complex callables are currently awkward to specify (with Protocol), and ParamSpec values can only be specified in a very limited way.
A potential extension of the idea to solve that: if Callable instances defined __call__ based on the signature they described and returned their return expression, the proposed syntax would also work as a replacement for lambda expressions (you’d just put an actual calculation to the right of the arrow rather than declaring the resulting type).
Previous proposals for a new lambda syntax failed because they just changed the syntax, they didn’t increase language expressivity at all.
Getting the runtime performance to match existing lambda expressions wouldn’t be easy, but shouldn’t be impossible (at least for the syntactic form).
This wouldn’t even need to be part of the typing proposal, just noted as a possible future extension to counter this particular objection.
Scratch that idea, the scoping for the return expression doesn’t work. Left it here in case anyone else has the same idea.