Passing anonymous functions as the last positional argument

Ideas
1

Passing closures as the last argument is supported by many modern languages including Swift, Groovy, Scala, Ruby, Kotlin, Javascript. Supporting this in Python will greatly simplify many boilerplates and significantly improve code flexibility and readability.

Proposal

Suppose we have a function that takes a Callable as the last positional-only argument:

def foo(fn: Callable[[int, int, int], None], /, x=1):
    fn(x, x*2, x*3)

Without anonymous function passing:

def bar(a, b, c):
    print(a, b, c)

foo(bar, x=1)
foo(bar) # with default value

With anonymous function passing:

foo(x=1) a, b, c:
    print(a, b, c)

# with default value
foo a, b, c:
    print(a, b, c)

Alternatives

Note that we can also pass lambda function in this case, but the difference is that lambda only accepts a single statement or expression:

foo(lambda a, b, c: print(a, b, c), x=1)

Another way to achieve it is to use decorator, but it’s usually pretty cumbersome and tricky to define a decorator that optionally accepts parameters:

import functools
def foo(x_or_fn: int | Callable):
    default_x = 1
    def _foo(x, fn: Callable):
        fn(x, x*2, x*3)
    if isinstance(x_or_fn, Callable):
        return _foo(default_x, x_or_fn)
    else:
        return functools.partial(_foo, x_or_fn)

Then we can call:

@foo(1)
def bar(a, b, c):
    print(a, b, c)

# use default value
@foo
def bar(a, b, c):
    print(a, b, c)

Use Case

One simple use case is a retry function:

def retry(fn, /, n=2, *, exception_type=Exception):
    for i in range(n):
        try:
            fn()
            break
        except exception_type as e:
            if i == n-1:
                raise e
            import sys
            print(e, file=sys.stderr)
        except Exception as e:
            raise e

Then we can use it as follows:

retry():
    do_some_stuff()
    do_some_other_stuff()

retry(3):
    do_some_stuff()
    do_some_other_stuff()

retry(n=3, exception_type=ValueError):
    do_some_stuff()
    do_some_other_stuff()
2 Likes
2

How? Could you provide analogies with the proposed Python syntax?

3

I was wrong about JavaScript, but here’re analogies for the rest:

Swift

func foo(x: Int = 1, fn: (Int, Int, Int) -> Void) {
    fn(x, x * 2, x * 3)
}

foo(x: 1) { a, b, c in
    print(a, b, c)
}

Groovy

def foo(int x = 1, Closure fn) {
    fn(x, x * 2, x * 3)
}

foo(1) { a, b, c ->
    println "$a $b $c"
}

Scala

def foo(x: Int = 1)(fn: (Int, Int, Int) => Unit): Unit = {
    fn(x, x * 2, x * 3)
}

foo(x = 1) { (a, b, c) =>
    println(s"$a $b $c")
}

Ruby

def foo(x = 1, &fn)
  fn.call(x, x * 2, x * 3)
end

foo(1) do |a, b, c|
  puts "#{a} #{b} #{c}"
end

Kotlin

fun foo(x: Int = 1, fn: (Int, Int, Int) -> Unit) {
    fn(x, x * 2, x * 3)
}

foo(1) { a, b, c ->
    println("$a $b $c")
}

Proposal for Python

def foo(fn: Callable[[int, int, int], None], /, x=1):
    fn(x, x*2, x*3)

foo(x=1) a, b, c:
    print(a, b, c)
2 Likes
4

By limiting this to only supporting the ability to pass an anonymous function as the last argument to a function call that is itself theonly thing on the line, the resulting syntax is probably the best I’ve seen for any of the many proposals to add multi-line anonymous functions.

However, I’m concerned that by limiting the use so severely, you may have disallowed all of the motivating use cases that people have for this feature. I think you need to look very closely at the history here, and report back on how this proposal helps the various use cases that have been brought up over the years. You should also survey some substantial bodies of real-world Python code (the stdlib, and the source of some of the more popular Python packages, are common choices) and determine how often this feature would be useful to them, and what the code would look like if it were used.

I’m a bit concerned about the precise syntax here, though. Your examples only show a function being called for its side effects - the return value of the function is ignored. You haven’t described what you expect to do if the function returns a value. Would you allow arbitrary expressions as long as the function call was the last thing on the line? So something like

x = fn(1,2,3) a, b:
   retrun a + b

would be allowed? What about more complex expressions?

x = 13 if some_test() else fn(1,2) a, b:
    return a + b

This rapidly becomes unreadable for the human reader, and quite possibly unparseable for the interpreter as well.

IMO, the only sensible option is to disallow everything except a function call for side effects, and I think that’s probably too limited to be worth it. But maybe I’m wrong - the only way to know is, as I said, to survey how useful this would be in real life.

10 Likes
5

Thanks for the advise, I will do some research on this. But just from my first intuition, I can see that all of the other languages impose similar level of constraints and still being very widely used. As far as I know, Homebrew formula heavily relies on this feature from Ruby, Jenkinsfile leverage this feature from Groovy as its basis, SwiftUI uses it extensively for UI interactions (though Swift allows passing multiple closures). These use cases in other languages can provide useful analogies and guidance for assessing its practicality in Python.

However, I understand that if this feature can be integrated easily and seamlessly into Python’s long-established history is indeed a question. Modern languages designed with this feature from the outset do not face the same compatibility concerns with legacy designs, unlike Python. For example, the callable parameter in map and filter is the first positional-only parameter rather than the last. How to maintain backward compatibility indeed requires some rumination.

6

Just a brainstorm. I think one way to loosen the constraint is to introduce a special symbol : for locating the callable parameter position. I’m not sure if introducing a new symbol in function arguments is too much of a cost, but this will definitely make it much more usable and backward compatible.

transformed = map(:, [1, 2, 3]) x:
    return x * 2

filtered = filter(:, transformed) x:
    return x > 1

result = functools.reduce(:, filtered) x, y:
    return x * y

Even for one liners, I feel it’s more readable than:

transformed = map(lambda x: x * 2, [1, 2, 3])
filtered = filter(lambda x: x > 1, transformed)
result = functools.reduce(lambda x, y: x * y, filtered)

Using my original example

def foo(fn: Callable[[int, int, int], None], /, x=1):
    fn(x, x*2, x*3)

foo(x=1) a, b, c:
    print(a, b, c)

This will be equivalent to:

foo(:, x=1) a, b, c:
    print(a, b, c)

If the callable is not positional only or it’s even kw only:

def foo(*, fn: Callable[[int, int, int], None], x=1):
    fn(x, x*2, x*3)

We can also do this:

foo(x=1, fn=:) a, b, c:
    print(a, b, c)

This imposes much less constraints by introducing a new symbol.

Choosing : over other symbols is because : roughly represents the start of a closure in python for most of the times. To me, it looks syntactically intuitive and consistent.

1 Like
7

In a past life I wrote a whole lot of code in Groovy, and I actually miss this feature quite dearly. I’d be thrilled to see something like it in Python.[1]

I don’t see the need to restrict trailing callable arguments to positional-only parameters. The callable could be bound to the called function’s first unbound positional-only or positional-or-keyword parameter (after binding all the other arguments), and that should still be unambiguous.

That would indeed be too limited to be worth it. But I’m not so sure that the ability to write confusing code with this feature means that it needs to be heavily restricted to prevent code from being written that way. You can make just about any language construct look pretty nasty if you try hard enough.[2]

At the very least you’d want to use this in assignments, return, await, and probably many other places. The suggestion to allow it in arbitrary expressions as long as the function call was the last thing on the line is probably sufficient to make this readable in general. (As long as the person writing the code cares about readability in the first place)

  1. We can leave the unqualified name lookup shenanigans at home, though. :slight_smile: ↩︎

  2. And you don’t even have to try all that hard with conditional expressions. :slight_smile: ↩︎

8

I think there’s a middle ground here. Remembering how decorators were originally quite restricted, but were subsequently relaxed, I’d be inclined to whitelist a few forms of expression that are allowed to have this happen, with a view to allowing that to be expanded in the future.

So here’s my ideas for what would be useful:

  • Function call for side effects: do_stuff(1, 2, 3) a, b: ...
  • Function call with assignment: TARGET = do_stuff(1, 2, 3) a, b: ...
  • Direct assignment: TARGET = a, b: ... (useful when the target is, say, a subscripting - this would thus be able to handle another thing that gets asked for periodically, def x[1](a, b): ... to stick something into somewhere). Open to syntax suggestions.
  • return statements
  • Any others?

I know for sure that people will want to be able to pass the lambda function as a kwarg, but I’m not comfortable with suggesting a syntax for that.

2 Likes
9

This IMO is drifting back towards the sort of things that previous proposals did to try to be “general”. I’d be surprised if you end up with a proposal that adds anything over the previous ones if you take this route.

To take this idea to its ultimate conclusion, let’s (1) allow the user to use any name they like rather than using a : - that helps readability. And (2) let’s put the indented block before the expression (for no particular reason, although “put the things you refer to before the things that refer to them” is a reasonable principle) - you end up with

def double(x):
    return x * 2
transformed = map(double, [1, 2, 3])

def gt1(x):
    return x > 1
filtered = filter(gt1, transformed)

def mul(x, y):
    return x * y
result = functools.reduce(mul, filtered)

At this point you have perfectly legal Python, and it’s barely any different in practice from your proposal.

4 Likes
10

To add to your prior art collection [1], here’s what Pike has. It’s a bracey language, and you can add an implicit lambda function after any function call.

//Passing a function by name
result = do_some_stuff(1, 2, 3, callback_function);

//Using the regular lambda syntax
result = do_some_stuff(1, 2, 3, lambda() {whatever();});

//Using an implicit lambda function
result = do_some_stuff(1, 2, 3) {whatever();};

The function receives an array of arguments in __ARGS__, equivalent to def func(*__ARGS__): in Python. This notation is quite handy when working with promises, since you can write blahblah()->then() {code; to; run; after; promise; resolves;}; which is pretty convenient. (Not as convenient as actual __async__ functions with an await keyword, though, which is what the latest alphas of Pike now support.)

Doesn’t much help with picking out a syntax for Python though, since it depends on the standard bracey syntax to delimit the function body.

  1. now I want to see a wing at a public gallery labelled “Prior Art” ↩︎

11

Yeah, there’s no ambiguity here. Recommendation to everyone who’s making proposals regarding functions: Be aware of the difference between function definitions and function invocations.

  • When you define a function, you specify the parameters it expects. These come in five flavours: positional-only, positional-or-keyword, keyword-only, and the collections *args and **kwargs.
  • When you call a function, you specify the arguments it receives. These come in two flavours: positional and keyword. Python maps them to their parameters.

Since this is a proposal regarding invocations, all it needs to specify is that the additional argument is passed positionally. There may need to be a small amount of special-casing in that this is the last positional argument even if there are keyword arguments, but that’s a relatively small matter and fairly obvious; I doubt anyone will be confused by it.

1 Like
12

Since I have never touched any of the mentioned languages, I need a specification to to understand your examples. I am partly confused also because the title says ‘anonymous functions’, while the text says ‘closures’. To me, in the context of Python, the two concepts are orthogonal: function definition bodies may refer to names in surrounding local namespaces whether defined with def or lambda syntax. Can you clarify your meanings?

2 Likes
13

I have, so I can elaborate a little. Not speaking for the OP of course and you’re most welcome to add to this or disagree with me entirely.

All Python functions are closures, but if you want a multi-statement function, you have to give it a name and create it with a statement. Technically the lack of name isn’t a particularly significant distinction, and it’s entirely possible to have a named function in an expression context (JavaScript is capable of this, not that people actually do it very often), so to most people, “anonymous function” is synonymous with “function definition expression” - that is to say, a function definition that can be embedded inside something else.

In this specific proposal, it’s not QUITE a function definition expression, but it serves broadly the same purpose, allowing one to be embedded in something else - a function call.

1 Like
14

Sorry for the confusion. The reason why I say it’s closure in other languages is because closure refers to a nameless function body (basically a multiline lambda) that can exist on it’s own (an independent expression). But my proposal defines an anonymous function by attaching it to a host function call, while the anonymous function itself can not independently exist as an expression. It’s only a syntax sugar to inject a disposable function body into another function call. The purpose is not to have pure closure expressions (or you can call it multiline lambda) as in other languages.

For example, in groovy, you can pass a closure to a function like this:

someFunc(1, 2, 3, {
    doSomething()
    doSomeOtherThings()
})

It’s the same as this syntax sugar:

someFunc(1, 2, 3) {
    doSomething()
    doSomeOtherThings()
}

note that this part by itself can be evaluated into a value without the existance of someFunc:

{
    doSomething()
    doSomeOtherThings()
}

you can even create a list of this stuff:

closureList = [
    {
        doSomething()
        doSomeOtherThings()
    }, 
    {
        doSomething()
        doSomeOtherThings()
    }
]

But in Python, the nature of indentation based syntax doesn’t allow such thing. and That’s why we can only have a single line in a lambda expression. So what I proposed is just a form of anonymous function, instead of a pure closure as in other languages.

16

BTW, there have been so many requests about adding anonymous functions in python over the years. Most of the proposals are not doable due to python’s unique indentation based syntax. My goal is to propose a trade off that covers most of the use cases of a anonymous closure while still honors python’s original syntax. It won’t be as flexible as a real closure though, which is most likely not possible in python.

17

-1. While the intention is good, I find the proposed syntax way too limiting with the restriction of the callable as only the last positional-only argument when we have so many good use cases for anonymous multi-statement function as non-last, non-positional-only or keyword argument, a few of which that quickly came to my mind include:

  1. A callable as the repl argument (which is the second, non-last, non-positional-only, possibly keyword argument) for re.sub that unpacks match.groups() into meaningful names before building a replacement string, which requires at least two statements.
  2. A callable as the target argument (which is the second, non-last, non-positional-only, possibly keyword argument) for multiprocessing.Process that performs an ad-hoc task requiring two or more statements.
  3. A callable as the type argument (which is a keyword-only argument extracted from kwargs) for argparse.ArgumentParser.add_argument that performs a simple but multi-statement validation just for one argument.

I would propose a new anonymous function syntax as a block-based expression, either as lambda or def, where an indented-and-then-detented block is expected when a colon is at a line end.

With lambda, the returning value is the last evaluated expression:

last_first_name = re.sub(
    r'(\w+) (\w+)',
    lambda match:
        first_name, last_name = match.groups()
        f'{last_name}, {first_name}'.upper()
    , first_last_name
)

With def, the returning value is returned explicitly:

last_first_name = re.sub(
    r'(\w+) (\w+)',
    def(match):
        first_name, last_name = match.groups()
        return f'{last_name}, {first_name}'.upper()
    , first_last_name
)

The only thing that I find to be possibly out of style in the proposed syntax above is that it requires a leading comma for the argument after the block, but that’s something PEP-8 apparently has nothing against.

18

Is the perfect to be the enemy of the good? Proposals that try to encompass the entire gamut of possible uses for inline functions inevitably fall down; this proposal specifically narrows the scope in a way that still includes a lot of valid uses.

I’ll cite decorators again: for a decade and a half, the syntax available for a decorator was severely restricted, and yet they were incredbly useful. Should the original proposal have been rejected due to not covering every possible use-case?

19

Can you respond to the common use cases I presented, and point out why a new syntax should not attempt to address them? And can you point out the fallacies of my proposed alternative syntax?

I simply find the proposal of the OP way too limiting to be useful for many of the use cases I yearn for.

20

Because not every proposal has to solve every problem. This can be a real benefit without having to be everything for everyone.

Fundamentally, it’s up against the same problem that these kinds of proposals always are: that you want significant indentation in a context where indentation isn’t significant. That’s a big challenge, and quite frankly, an unindent marked by a loose comma is quite ugly in my opinion.

But your proposal is completely separate from the OP’s. If you want feedback on it, post it as a separate proposal. Don’t use it as a reason to reject another proposal.

21

But by having a syntax that solves a narrow scope of a problem it makes having a better-rounded syntax that solves the problem in a more general way less likely to happen in the future because people will then say we already have this syntax that solves some of the problem.

You’re right that it will make the parser more complex by requiring to special-case lambda and/or def in a free-flowing context, though still technically doable. The question is whether the use cases that this enables is worth the additional complexity.

As I noted I agree that it is not up to the style Pythonists are accustomed to, though I think it’s a relatively small quirk like the trailing comma of a 1-tuple where the functionality gain outweighs the style loss.

I believe it is perfectly on topic to post alternatives to proposals in this forum when the goals are similar.

1 Like