PEP 677 with an easier-to-parse and more expressive syntax

Typing
21

lambda[x: int, y: int] -> int feels weird to me, because part of the annotation exists outside of the square brackets. It looks like the parameter signature is a separate “thing” from the return signature. As an amateur typing enthusiast, something like this feels more natural:

lambda[(x: int, y: int) -> int]

I like that this preserves the (args) -> return form which mirrors the syntax of a signature definition, whilst (arguably) sidestepping the issue of ambiguity/readability. Presumably this wouldn’t work without some under-the-hood parser changes to support -> inside square brackets, but reading through this thread syntax changes seem to be on the table for consideration.

Perhaps this is unrelated, but it would also be cool to consider how generic callables might be expressed using the new type parameter syntax, and have a fourth supported type parameter form:

class Example[T, *Ts, **P, Fn()]: ...

If Fn() in a type parameter were aliased to some new typing construct (e.g. typing.Signature) perhaps it could be an opportunity to revisit some of the ParamSpec semantics:

def my_decorator[Fn()](fn: Fn):
    @wraps(fn)
    def wrapper(*args: Fn.args, **kwargs: Fn.kwargs) -> Fn.return:
        print('Calling decorated function')
        return fn(*args, **kwargs)
    return wrapper

(edit: fixed an error in the example, thanks @JoBe )

4 Likes
22

I suppose for the example you meant return fn(*args, **kwargs).

Imo we could try and re-use inspect.Signature or add some annotationlib.Signature. I kinda feel like cluttering typing with more and more stuff would only make it harder to maintain.

Generally however, I love the idea of having lambda[(a: ..., b: ...) -> ...]. I’d would be easy for users to get all arguments inside the lambda hint; e.g. for making a function out of a lambda.

def make_fn(name, hints, executes):
    fn_hints = repr(hints).removeprefix("lambda[").removesuffix("]")
    return eval(f"def {name}{fn_hints}: {executes}")

(Assuming some repr lambda[(...) -> ...]).

1 Like
23

I do like this idea. Especially if we allow (args) -> return in any square bracket, so that you could specify the generic type as

my_decorator[(int, flag: bool) -> float]

(assuming some form of PEP 718 but it would of course also work for generic classes)

We could then redefine Callable as

class Callable[Fn()](Protocol):
    def __call__(self, *args: Fn.args, **kwargs: Fn.kwargs) -> Fn.return: ...

so that we can say

Callable[(x: int, y: int) -> int]

and then we don’t need the lambda keyword anymore.

But there are some problems with this, I think:

  1. It’s a bit strange to have both ParamSpec and also SignatureSpec (or whatever we want to call the new TypeVar-like that also has a return type). And the problem remains that ParamSpec is awkward to explicitly specify (by which I mean that if I have class C[**P]: ..., then how can I specify that C[...] has named arguments, arguments with default, etc.?)
  2. To me, Callable[(x: int, y: int) -> int] looks clunkier than lambda(x: int, y: int) -> int; and this is the most common case (I’m guessing).
24

I have though about some typing.Returns[fn] for some time now, and it could help solve a few little problems..The Fn.args and Fn.kwargs to access the types of *args and **kwargs should work for decorators that are implemented in the usual way (def inner(*args, **kwargs)). However:

  • Some arguments can be accessed by position or keyword. Where should their types reside?
  • Sometimes we need the type of specifically the n-th argument, or we need the type of specifically the argument with the name x.

Therefore I’d like to propose a new way to access these attributes. Some wrapper Signature[fn] will have a __getitem__ to index the positional only arguments with hints. It would also have a args and kwargs property, to only return types of arguments where we applied some * or ** before. Then some __getattr__ could be used to gain the keyword only arguments.

This way, we can hint functions who’s arguments match with those of some other function, e.g.:

def fn1(a: int, /, *, b: str) -> float: ...
def fn2(a: int, /, *, b: str) -> float: ...

Where both can be used to supply the types of a, b and return type.

I’ve implemented something similar before, where you could access the types by name. I’ve called it Copy although I think the proposed name of Signature fits better.

class Copy[T1, *T2]: #TODO doc; TEST
    def __class_getitem__(cls, attr: object, arg: str = "") -> Any:
        if not arg:
            arg = "return"
        return attr.__annotations__.get(arg, Any) # Assume Any if not annotated

Obviously this is just a quick demo, it’s missing name validation for the arg as an example, but it should work for a quick implementation.

25

Do you have an example use case for this?

26

Perhaps within functools.partial (and similar).

27

The discussion here about making FunctionType generic, like Callable is today, made me consider this idea again.

Here is how we could have a better Callable, and also a better FunctionType, and any other kind of callable that has properties.

There would be a new TypeVar-like called SignatureSpec (name can still be bikeshedded):

F = SignatureSpec("F")

It works similar to ParamSpec, except that it also includes the return type and has a specialized syntax for specifying it (as we will see). Its PEP 695 syntax is F() (can still be bikeshedded) and it’s used like this:

class FunctionType[F()]:
    __name__: str
    __qualname__: str
	def __call__(self, *args: F.args, **kwargs: F.kwargs) -> F.returns: ...

For specifying the SignatureSpec, the Extended Syntax Supporting Named and Optional Arguments from PEP 677 is used:

type MyFunc = FunctionType[(int, y: float, *, z: bool = ...) -> bool]

Note that, in contrast to PEP 677, the (...) -> ... syntax can now only appear within square brackets. This hopefully makes parsing (by humans and machines) less expensive.

Unfortunately, we cannot change the definition of Callable to use SignatureSpec without breaking tons of already existing typed code. But of course we would want to use the new syntax for Callables, so what to do?

One option is to introduce a sort of type alias for Callable, which uses SignatureSpec. It could be called typing.Fn (name not final), such that

from collections.abc import Callable

def reduce(
    func: Callable[[int, int], int],
    l: list[int],
) -> int: ...

becomes

from typing import Fn

def reduce(
    func: Fn[(int, int) -> int],
    l: list[int],
) -> int: ...

One of the nice things about this approach is that it allows the specification of “complex callables” — which are not uncommon in Python — with a natural-looking syntax. We have already seen FunctionType, but there is more.

Take for example nn.Module from PyTorch. What PyTorch calls modules are basically callables but they also carry state that is updated during gradient descent. The user of the library sub-classes nn.Module and implements the forward() method which is then called by __call__() (which also does other things needed for computing gradients and things like that).

With SignatureSpec, nn.Module could be defined like this:

class Module[F()]:
	@abstractmethod
    def forward(self, *args: F.args, **kwargs: F.kwargs) -> F.returns: ...
    def __call__(self, *args: F.args, **kwargs: F.kwargs) -> F.returns:
        # do other stuff here
        return self.forward(*args, **kwargs)

A precise signature of a PyTorch module can then be specified with:

layer: nn.Module[(Tensor, my_flag: bool) -> Tensor]
layer(torch.zeros(4), my_flag=False)

What is described here is to some degree already possible with ParamSpec, but note for example that one cannot specify the name of the boolean flag with ParamSpec.

nn.Module can be sub-classed like this (example from here):

type Activations = Literal["relu", "lrelu"]
class DynamicNet(nn.Module[(Tensor, act: Activations) -> Tensor]):
    def __init__(self, num_layers: int):
        super().__init__()
        self.linears = nn.ModuleList(
            [MyLinear(4, 4) for _ in range(num_layers)])
        self.activations = nn.ModuleDict({
            'relu': nn.ReLU(),
            'lrelu': nn.LeakyReLU(),
        })
    def forward(self, x: Tensor, act: Activations) -> Tensor:
        for linear in self.linears:
            x = self.activations[act](linear(x))
        return x

net = DynamicNet(2)
net(torch.ones(9), act="sigmoid")  # type checker: "sigmoid" is not allowed here

We have to specify the signature here twice (once in nn.Module[] and once in the definition of forward()), but the type checker will inform us when they get out of sync, so this doesn’t seem so bad.

One criticism of PEP 677 was that nested callables start to look very confusing. Consider this example from the PEP:

def f() -> (int) -> (str) -> bool: pass

With this proposal here, it would be

def f() -> Fn[(int) -> Fn[(str) -> bool]]: pass

which is less confusing, I would say.

There are some clear disadvantages to this approach:

  • It’s still required to import something (typing.Fn) to specify callables. One selling point of PEP 677 was that the import of Callable wasn’t necessary anymore, but the import will unfortunately still be necessary with this proposal.
  • I think it isn’t possible to specify generic callables (that is, callables that have their own bound TypeVar as in lambda[T](T) -> T), but that’s also not possible with today’s Callable and also wasn’t possible with PEP 677.
  • The lexer still needs to perform a look-ahead in order to recognize the syntax, but at least it’s only needed within square brackets.

But the advantage is that this approach can more accurately describe the breadth of callables that are used in Python.

1 Like
28

Perhaps we can ignore the new syntax for now, and add to Callable, so that something like Callable[[int, *str, b: float = 0.0], bool] is possible?!

Imo adding completely new syntax (like (…) -> … is an overcomplication at best. Extending the syntax of GenericAliases however could also help with more extensive features in the future.

29

That’s also new syntax. (The *str part is syntactically legal now, but making it work would require making types iterable. The b: float = 0.0 part is not legal syntax.)

30

Well, but that new syntax is possibly useful for other applications, whereas I can only see one way to use the (…) ->… syntax.

As an example, being able to do MyAlias[x: y] just slices the alias (still legal syntax, just that the __slice__ method has to be good to detect stuff like that), and that leaves us with only needing to add MyAlias[x=y] (and the combination of : and =) as new syntax.

As I already stated, imo, still using GenericAliases, but refining them instead of refining each kind of application where different GenericAliases can be used (e.g. Callable, Generator, …), is cleaner, and allows us to add more special features in the future, without having to add syntax for each feature.

(Basically having a small ‘sacrifice’ now would make it easier for us to use the gains later on.)

31

Most options for this are going to have to either create new syntax, or make some concession or other.

Consequence of reusing getitem semantics and type infomration being evaluated as python expressions.

Something that could be done today without adding new syntax to the language would be something like:

ExpressiveCallable["(x: int, y: int, /) -> int"]
# and
ExpressiveCallable.from_runtime(foo)  # copies the callable signature of `foo`

The good thing about not requiring new syntax is it means you can also test how this works in practice outside the standard library (if you are a type checker author, for a typechecker to support it, or work with a typechecker author). It might be good enough to then use to argue for the syntax you want on what it enabled + what was found missing.

I would expect that both of these would suitably convey intent within existing syntax. To better support runtime introspection, I would also expect that the string there can be turned into something like an inspect.Signature object (when requested for introspection, not eagerly)

5 Likes
32

Implementing it first as a string so it can be tested out is a good idea.

So I guess the steps would be

  1. Write a pre-PEP, get type checker opinions
  2. Get the necessary runtime objects into typing_extensions
  3. Type checkers implement it
  4. Experiment

I can see two levels of possible ambition here:

Go only for the Callable replacement

This would only require typing_extensions.ExpressiveCallable and would likely make writing the pre-PEP relatively easy because most could be copied from PEP 677. There will likely be unforeseen edge cases but that’s what the experiment is for.

Include SignatureSpec in the experiment as well

This would additionally require a typing_extensions.SignatureSpec runtime object. The implementation for it could probably be mostly copied from ParamSpec with only an additional .returns attribute. There would also need to be specification for SignatureSpec but there is actually not much one can do with a SignatureSpec object (for example, it can’t be used with Concatenate, I think), so this might not be a big problem. Though I may be underestimating this.

The .from_runtime(f) feature is something I definitely have wished for before, but I think this syntax won’t work. I seem to remember from somewhere that function calls in type annotations are problematic. I think this’d need to be a new type form, CallableLike[f].

1 Like
33

Function calls in type annotations are “okay” with a few caveats, but those caveats basically explain why they aren’t something in use currently

  1. Currently, the specification says that type alias statements shouldn’t use them. This isn’t enforced at runtime, so you can start using them today, and the language there can be changed to “only supported function calls…”

  2. They have to be something that has a well defined meaning at both annotation statically, and that when evaluated, resolves to an object that has a useul interpretation when introspected by runtime typechecking tools.

As it’s a function call, it will look different in the case of a forward reference than anything currently expected in a typing related forward reference, but that’s just a matter of tools that use typing needing to be updated, and that is going to be needed no matter what.

Most function calls fail at the second of these, because they aren’t actually meant for annotation use. Because this is proposing an addition to typing (typing_extensions first), this is already a case where typecheckers need to know what it means either way, and having it mean something meaningful at runtime is relatively simple. (Making this a class method that returns an instance of an ExpressiveCallable solves this)

1 Like