Make lambdas proper closures

This post was flagged by the community and is temporarily hidden.

Sorry Maksym, your attempt to explain why I have misunderstood you has
just confused me more.

You seem to understand why no declarations are needed, that forward
references are perfectly legal in Python (regardless of whether they are
within a function or not) and yet you were surprised that code using a
forward reference didn’t fail.

And then after demonstrating that you understand perfectly why it is
legal for a function to have a forward reference to a name not yet
defined (otherwise we would need Pascal-like “forward” declarations,
how 1970s is that?) you make a statement like this:

“personally i think it’s a bug that x magically becomes visible inside
f after f has been defined. to me it’s scope leaking.”


Anyway, moving on…

Obviously there is no “one true answer” to how loop unrolling should
work. Likewise for scoping and early/late binding, where programming
languages are free to make their own choices.

I may have strong opinions on what I like, but that’s not to say that
other languages should not make other decisions.

But if the rolled and unrolled loops are not equivalent, then that is
going to be surprising, and it would likely rule out future compiler
optimizations to do with loop unrolling. CPython doesn’t currently do
that, but some day it might, and other implementations such as PyPy
might even do it now.

One major issue is that there is a ton of code that assumes that the
body of a for loop is the same as the surrounding scope.

x = 1
for item in seq:
    x = something()

If the x inside the loop and the x outside the loop were different
variables, that would break lots of code. So any change to the scoping
rules would have to be include an implicit nonlocal declaration inside
the body. And that would likely fail mysteriously in nested code like

def outer():
    x = 999
    seq = something()
    assert seq

    def inner():
        for item in seq:
            # implicit nonlocal x
            x = process(item)

    assert x == 999  # Fails!

The intent is for x to be local to inner(), but the implicit nonlocal
makes it local to outer() instead because that’s the first scope with
an existing name x. Ouch!

Now there are probably cunning ways to make it work so that x goes into
inner() rather than outer(), but the level of complexity is increasing
and the likelihood of bugs and gotchas and weird unexpected corner cases
is high. And even if we can get it perfectly right, it is still a
backwards-incompatible change due to putting the loop variable in its
own scope rather than the function scope.

Steven, I appreciate your patience!

you were surprised that code using a forward reference didn’t fail

this was a bit of a tongue in cheek and i’ve elaborated why i made that comment in my reply to Serhiy: Make lambdas proper closures - #13 by maksym

to reiterate - i would make distinction in name resolution between functions and variables, but i realize that in python it’s not possible because both are just names and functions just happen to be callable. not suggesting anything needs to change here.

Regarding loop rolling, we both agree that there is no one true answer on how to unroll, but in then whichever way to unroll we choose, we can pick the right scoping rules to satisfy rolled/unrolled equivalency. I don’t understand why you seem to insist that there wouldn’t be equivalency with iteration-scoped loop variables?

As for your examples with x variable, i think we’re getting sidetracked as my suggestion would be only to make loop variables iteration-scoped (only item in your examples) and leave everything else working as it did before. Also it seems completely doable for me to let the last iteration of loop variables to leak into outer scope to remain compatible with essentially all existing workflows.

Thanks for making that poll and linking lots of useful context!

it is an implementation detail of python that functions are callable

No it isn’t, it is a core part of the execution model of the language.
Functions are values just like strings and lists and floats, they are
no more or less privileged than any other value.

It is not just an accident of implementation that functions are objects,
or that the names of functions and the names of other variables are in
the same namespace. Functions are instances, they have a type, they have
attributes and a __dict__ so you can add your own attributes to them.

> def spam():
...     pass
> spam.eggs = 1
> spam.eggs

The compiler cannot assume that f is always bound to a function just
because it sees def f(): ... in a module. Not only can f be rebound or
unbound within the module, but any other module can reach in and change
the name binding at runtime.

So the compiler or interpreter could not distinguish between the two
cases of “undefined variable” and “undefined function” because functions
are variables.

And that’s part of the design of the language, and we like it that way.

Maksym is hardly the only one who has been surprised or bitten by this.
I’ve seen many discussions about it on comp.lang.python, I’m sure you
will find it on Stackoverflow, I daresay its been raised on
Python-Ideas, and there’s a FAQ about it. I would be shocked if there
haven’t been bug reports raised over it too.

Other languages have made the choice to put for loops in their own
scope, and they didn’t do it for no reason. We may or may not agree with
Maksym’s preference, but he’s not alone.

Ultimately though, each language chooses its behaviour, and programmers
have to learn to adapt. There’s no more use in complaining that Python
doesn’t behave just like Java as it would to complain that Java doesn’t
behave just like Python. If they were identical, they wouldn’t be
different languages.

Especially now that Python is a stable, mature language, one of the top
three or five most popular in the world (which means tens or hundreds of
thousands of users, and billions of lines of code), we are far more
conservative about making major changes that break backwards

I don’t wish to give Maksym false hope, but if he is willing to push
this, and gather support from at least one core developer, and write a
PEP, and win over the Steering Council, it’s not impossible that it
could be changed. But honestly at this point it’s probably more likely
that the USA swaps over to the metric system wink

FYI I have hidden a bunch of posts and put this thread into slow mode which restricts how often people can post for now in hopes that everyone participates respectively.

1 Like

I know this is another “workaround”, but it hasn’t been mentioned yet AFAICT: you can generally separate out a “constructor” for your callables:

# Example 1:
def number_squarer(i):
    return lambda: i ** 2

squarers = [number_squarer(i) for i in range(10)]
print([f() for f in squarers])
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

# Example 2
def printer(message, index):
    def do_print():
        print(f"Message at button {index}:", message)
    return do_print

# Hypothetical GUI buttons
buttons = [Button(label=str(i), callback=printer("clicked!", i)) for i in range(10)]

To my eye, these sorts of things are more self-explanatory and less hacky/workaround-y than the lambda i=i: i business, especially if you can find good names for things.

I’ll also add that in some situations (though not every situation), it can be more understandable to avoid a lot of the passing around of dangling verbs and stick to just making a list of nouns one at a time in a function, avoiding much of the issue altogether:

# Modified Example 2
def make_button(index):
    def do_print():
        print(f"Message at button {index}: clicked!")
    return Button(label=str(index), callback=do_print)

buttons = [make_button(i) for i in range(10)]
1 Like

I consider it a feature that lambda expression create the same sort of functions with the same behavior as def statements. I won’t repeat the explanation to demonstrate this that Steven D’Aprano already gave several posts ago.

I also consider it a feature that Python does not have block scoping. If I wanted that, I would not have switched to Python 24 years ago.

1 Like