Passing anonymous functions as the last positional argument

Ideas
42

The syntax might be made more readable by requiring some (soft) keywords, e.g., do and passing (or any other pair of keywords; the point is that the first would be the callback placeholder and the other would be a well visible indicator of the entire construct, preceding the callback’s parameters, if any):

foo(do, x=1) passing a, b, c:
   print(a, b, c)

retry(do, n=3) passing:
   do_something()

my_list.sort(key=do) passing item:
    return x**3 - 2x + 1

last_first_name = re.sub(r'(\w+) (\w+)', do, first_last_name) passing match:
    first_name, last_name = match.groups()
    return f'{last_name}, {first_name}'.upper()

parsed = map(do, unparsed) passing raw_item:
    head, sep, tail = raw_item.partition('-')
    if sep:
        return [head, tail]
    return [head]

x = 13 if some_test() else fn(1, 2, do) passing a, b:
    return a + b

x = (fn(1, 2, do) if some_test() else 13) passing a, b:
    return a + b

When it comes to non-operator syntax elements, Python seems to like keywords more than punctuation…

But, regardless of the syntax details, the main problem I see with this whole approach is that when we need to make our callback return some values (as in the most of the above examples), the return statement may look as it was returning from the enclosing scope, i.e., it is not clear enough (visually) that return belongs to the inner definition of the callback. Consider this:

def adjust_names(text):
    output = re.sub(r'(\w+)\s+(\w+)', do, text) passing match:
        first_name, last_name = match.groups()
        if first_name.lower() == 'johnny':
            return f'{last_name}, John Jr.'.upper()
        return f'{last_name}, {first_name}'.upper()
    if output == text:
        return 'Nothing interesting...'
    return output

It is not obvious at the first sight which return returns from the callback and which from the enclosing function.

However, have you read PEP 3150 and PEP 403 by Alyssa Coghlan? Each of those documents answers in its own way to that problem, and I found both of them very interesting. Even if they are not exactly the solution you seek for, maybe you can take some inspiration from them?

1 Like
43

Thanks for pointing out that similar ideas have been proposed and discussed before, but I honestly don’t see why old ideas that aim to solve a similar problem can’t be discussed here with the new context of this proposal in order to contrast and compare their pros and cons and to hopefully lead to an overall better-rounded solution.

With regards to the proposal as it is exactly posted by the OP, I find it not only too restrictive to be useful enough to be worth the effort (which you have already pointed out and which I have already commented on) but also unreadable because it is not clear at all from the syntax that it is the last positional argument that the anonymous function being defined is going to be passed as. Why not the first? The implicit position of the callable argument appears to have been chosen arbitrarily.

So again, -1 from me if we are to discuss strictly about the OP’s proposal as-is without improvements.

OTOH, one aspect I like about this proposal is that it extends a simple statement with a block instead of embedding a block within the simple statement so that the simple statement itself stays compact and simple. Where the callable that the block defines is placed within the simple statement though, is the problem.

I would like to know what potential downsides there may be to the improvement I suggested, by allowing a def expression as an indicator of where in the statement the callable is placed. A similar idea with a named forward reference has been discussed before, yes, but this proposal focuses on defining just one anonymous function, which I think simplifies the syntax while still satisfying most use cases, so it’s worth a further discussion.

That works too, but I feel that my suggestion with def(match) reads slightly better because it places arguments close together with the callable’s placement within the simple statement.

Agreed. How about reusing with instead?

last_first_name = re.sub(r'(\w+) (\w+)', def(match), first_last_name) with:
    first_name, last_name = match.groups()
    return f'{last_name}, {first_name}'.upper()

Reads like “define a function here with this block of code”.

1 Like
44

No matter if we use passing or with, it looks redundant when you want to do something like this:

fn_dict = {}
# implicit form
fn_dict['+'] = with a, b:
    return a + b
# explicit form
fn_dict['-'] = def(a, b) with:
    return a - b

I feel it less confusing without that keyword:

fn_dict = {}
# implicit form
fn_dict['+'] = a, b:
    return a + b
# explicit form
fn_dict['-'] = def(a, b):
    return a - b

Direct assignment is suggested by @Rosuav by the way,

I don’t think def x[1](a, b): ... is going to work though, as the syntax is already reserved for generics in Python 3.12.

45

slightly OT:
Can someone explain to me why anonymous functions are a recurring requirement even though Python has a very lightweight syntax for function definition. Furthermore nested functions can be defined locally.
What is the advantage compared to a simply enumerated named function like:

def f1():
    ...
    # as many statements as you like
    ...

I understand the argument that it’s difficult to come up with good names.
But “no name” and “such simple names” should be on the same level?
So there must be a very tempting advantage to avoiding the name altogether…?

3 Likes
46

The purpose of this proposal is slightly different from other proposal for anonymous functions. The true intention of this proposal is to enable more declarative style of programming. This is the same reason why we have decorators. It’s not about the one line of code it’s saving or less efforts on naming.

If there are no decorators in python, it’s definitely not the end of the world.

@decorator
def my_awesome_function():
    ...

Honestly, it does not even save a single line than this form:

def my_awesome_function():
    ...
my_awesome_function = decorator(my_awesome_function)

But you call tell the difference. It has a much more declarative style that fits more naturally into human’s logical thought process.

2 Likes
47

I really don’t like the syntax. Feels both cryptic and inflexible. I feel that the proper order when passing callbacks to function is what people want, so why not just give that?

foo(cb, x=1) with cb(a, b, c):
  print(a, b, c)

which is precisely equivalent to:

def cb(a, b, c):
  print(a, b, c)
foo(cb, x=1)

Simple semantics, simple syntax and quite flexible. The callback doesn’t even have to be anonymous, a simple name such as ‘cb’ is short enough and fits well. In javascript it is traditionaly used to denote callback parameters.

48

Well, the simple answer is that you can tell them apart by the placement of the def. An absolute beginner might not know what a function definition looks like, but at some point learning the language syntax is the beginner’s responsibility. Of course the language syntax should make that as frictionless as possible, but you have to draw a line somewhere. I’d put this one on the “beginner’s responsibility” side of that line, but there’s definitely room for reasonable disagreement there.

At any rate, the suggestion is still an improvement over the def-less one, because that one looks even more like a function definition. Really I meant the suggestion as “maybe let’s put a keyword in there somewhere to help tell them apart”, I’m not married to the idea of it specifically being def in that particular location.

One one potential downside is, you have to decide what to do if there’s more than one def expression in the call:

foo(def(), def()):
    do_something()

Either it’s an error to do that, or the same callable object gets passed to both parameters. Either one seems fairly reasonable, and any construct that can be interpreted in multiple reasonable ways the first time you see it has the potential to be confusing, even to non-beginners.

Also, while you can’t actually declare a function named def, without keyword syntax highlighting, def(a, b, c) in an expression context is indistinguishable from a function call.

(I hope I don’t come across as overly negative here - you seemed to be asking for downsides to your suggestion and these are things that came to mind. I actually like the suggestion better than some of the others in this thread.)

49

As I mentioned in the previous post:

Imagine if you want to define a Jenkins pipeline:

pipeine(node='some-node'):
    stage("Setup"):
        ... # do setup
    stage("Checkout"):
        steps = []
        steps.append():
            ... # checkout repo A 
        steps.append():
            ... # checkout repo B
        steps.append():
            ... # checkout repo C
        parallel(steps)
    stage("Build"):
        ... # build artifacts
    stage("Deploy"):
        ... # upload artifacts
    stage("Clean up"):
        ... # do cleaning

If you use with keyword:

pipeine(node='some-node', cb) with cb():
    stage("Setup", cb) with cb():
        ... # do setup
    stage("Checkout", cb) with cb():
        steps = []
        steps.append(cb) with cb():
            ... # checkout repo A 
        steps.append(cb) with cb():
            ... # checkout repo B
        steps.append(cb) with cb():
            ... # checkout repo C
        parallel(steps)
    stage("Build", cb) with cb():
        ... # build artifacts
    stage("Deploy", cb) with cb():
        ... # upload artifacts
    stage("Clean up", cb) with cb():
        ... # do cleaning

I don’t think it’s more readable.

2 Likes
50

It occurred to me that without some kind of trailing keyword (or something else), the syntax is ambiguous:

def match(arg):
    ...

def case():
    ...

value = ...

match (value):
    case ():
        'what does this do?'

This only works because match and case are soft keywords.

51

I don’t think it as ambiguous, it allows user to define their custom match-case.

52

This should just be prohibited. In the rare case where you need to pass the same callable twice, you just do:

def fn():
    ...
foo(fn, fn)
53

I have to disagree. Elegant, succint, maybe. Almost a DSL, really. But when it comes to actually understanding what’s going on the explicit version is more clear and not much more verbose. Crucially, it doesn’t add or rearrange lines.

1 Like
54

Ruby at least does something deeper with its block syntax, deliberately departing from semantics of a nested lambda or def:
Blocks use looks like a custom control structure wrapping a piece of code that still appears to be part of the outer function, so Ruby makes return and break inside a block do a non-local exit not just from that block but from the whole function. This post goes into details:

In most mentioned languages with “bracy” syntax, blocks are also nestable within expressions, allowing things like (Ruby)

ns = [1,2,3].map {|n| n*10}.tap do |n| 
  puts(n)
end.select {|n| n > 15}

So it’s not enough to just say “we want to enjoy it like Ruby” as motivation, unless you propose full-powered equivalent, including one where it’s not equivalent to a nested def. For more limited proposals like this one, it’s important to find example use cases that it hopes to address.

1 Like
55

If you want an explicit form, there you go:

pipeine(node='some-node', cb=def()):
    stage("Setup", cb=def()):
        ... # do setup
    stage("Checkout", cb=def()):
        steps = []
        steps.append(def()):
            ... # checkout repo A 
        steps.append(def()):
            ... # checkout repo B
        steps.append(def()):
            ... # checkout repo C
        parallel(steps)
    stage("Build", cb=def()):
        ... # build artifacts
    stage("Deploy", cb=def()):
        ... # upload artifacts
    stage("Clean up", cb=def()):
        ... # do cleaning

The user have the power to choose whether they just want to use it as a DSL or they need more clarity.

56

How about this:

ns = map(lambda n: n * 10, [1,2,3]).tap(def(n)).select(lambda n: n > 15):
    return n * 10

If it’s too long, then:

ns = (
    map(lambda n: n * 10, [1,2,3])
    .tap(def(n))
    .select(lambda n: n > 15)
):
    return n * 10
57

OK, that’s very much defining a DSL (domain specific language, for anyone who’s not familiar with the term) in Python. And that’s something that the language design principles have always been against (when Guido was BDFL, I’m fairly sure he explicitly said he didn’t want to enable people to write DSLs in Python, and as far as I’m aware the steering council has never said anything to change that principle).

So if this is your motivating example, I’m afraid it will likely not get much support.

Personally, I quite like the look of that example, and I could see it being an attractive way of writing certain types of code. But I’m not sufficiently interested in it to push for it, myself. What I would say is that if that’s the example that you were thinking of, you should reframe your proposal in terms of supporting this type of API/DSL design, rather than being about “anonymous functions”. I think you’ll have a better chance of persuading people that way. Still not good, but at least you’ll have a specific use case that people can understand and relate to.

1 Like
58

At this point the syntax is getting remarkably similar to what can be achieved with decorators (see below about the different semantics).
Imagine if we allowed @expr to preceed def on same line:

@pipeine(node='some-node') def _():
    @stage def Setup():
        ... # do setup
    @stage def Checkout():
        steps = []
        @steps.append def A():
            ... # checkout repo A 
        @steps.append def B():
            ... # checkout repo B
        @steps.append def C():
            ... # checkout repo C
        parallel(steps)
    @stage def Build():
        ... # build artifacts
    @stage def Deploy():
        ... # upload artifacts
    @stage def CleanUp():
        ... # do cleaning

Semantics: The way the function is passed differs.

  • The proposals here all inject it as an additional argument to an existing function call.
    Con: In some cases it’s not visually obvious where exactly? The more “explicit” variants address that.
  • Whereas a decorator may or may not be result of function call, but either way it gets called separately with the def’ined function as sole argument.
    Con: functions returning parametrized decorator have to written in special “curried” style; many existing functions taking a lambda like re.sub() are not suitable. (I was lucky with @steps.append that it only wants one arg)

The squished @expr def ...: ... syntax is still limited to being used for side effects (plus storing a value into the name that was def’d, which is not always sensible).
There is no reasonable way to extend it to appear in middle of expression.

But I gotta say, it’s so close to existing syntax and semantics that it might be promising. Next time I use or make a decorators-based DSL, I’ll try to see how it would look squished…

59

A better use-case might be GUI design, which is forever a mix of code and structure. Current Python libraries are all a bit of a pain in one way or another, and IMO there’s a lot of room to try out ideas like this. For example, you could have a function which defines a GUI element, and as its last argument, it takes a function which will provide children for that element - something like:

window(title="Example window"):
    button("Close")

For reference, here’s how Pike does implicit lambda:

window("Example window") {
    button("Close");
};

and as with this proposal, this passes a function as an additional (positional) parameter.

4 Likes
60

I wouldn’t say that’s the entire purpose, I’m just trying to show concrete examples to demonstrate the possibilities and why certain design choice is better than the other under different scenarios.

Regarding DSL, decorator also enables DSL as a side effect, but you can’t argue that we should ban decorators because it makes DSL possible.

61

Because in many use cases callable arguments passed to a call are one-time-use functions whose logics describe the details of only that very call.

Currently we have to not only come up with pointless names for these ad-hoc functions but also read them first before the actual calls that use them, forcing backwards scrolling when reading the calls.

With the proposed syntax it would be much clearer when reading the code that we’re making a call that needs an ad-hoc callable passed as an argument with its logics explained right under the call.

With this syntax only one ad-hoc callable argument can be defined still, so it’s somewhat redundant to name it IMHO, but it certainly solves the main problem of code reading order just as well.

Naming the callable argument would make sense if we find a way to allow multiple ad-hoc callable arguments to be defined, maybe by allowing additional and with blocks:

foo(on_click=click, on_hover=hover) with click(event):
   ...
and with hover(event):
   ...