Should loops be in their own scope? [poll]

Ideas
1

In Python today, for loops run in the same scope as their enclosing block. Some languages have for loops create their own scope, as functions and classes do.

Python’s current behaviour is discussed here.

Some people would like Python for each iteration of a loop to run in its own scope.

That would probably solve the issue described in the topic “Make lambdas proper closures” but there may be performance or other costs.

There is a FAQ about the behaviour of lambdas in loops.

Changing the behaviour of loops would break backwards compatibility and break any code that expects variables inside a for loop to be the visible outside of the loop, and so would need to initially be a __future__ import.

Given all that, what do you think future Python versions (probably no earlier than 3.12) should do with for loops?

  • No change, leave loops as they are.
  • Change loops to use their own scope.
  • No change for loops by default, but add an option to run them in a new scope.

0 voters

(This is an unofficial and non-binding poll. There may be technical reasons why any change to for loop scopes is impossible or too difficult.)

2

The current status is non-obvious enough to require checking what the scoping rules are every once in a while.

This would be a major change, requiring a 4.0 release.

Think about all the code that defines names for later use within a with statement.

1 Like
3

In principle I would be in favour of giving for-loops their own scope. However, it’s far too convenient for most Python code to be able to check what the last value of the loop variable was after the loop has finished.

(Eg what was the last item in the iterator.)

That kind of code is prone to errors, because you have to make sure that your loop body ran at least once. But it is nevertheless very popular and convenient.

Btw, if you want to fiddle with scoping, I’d suggest something simpler:

Have different operators for assigning to an existing variable (perhaps keep a = 1 syntax) and for creating a new variable (perhaps use a := 1 or so as a syntax). That way you can retire the global and nonlocal keywords.

4 Likes
4

Unless I misunderstand the proposal, I’m not sure how I see how making a massive, fundamental change to the syntax and semantics of assignment of objects to names in Python (which to note, doesn’t actually have “variables” in the strict sense, but rather names and objects), which would require rewriting an enormous fraction of existing code, is somehow simpler than changing the scoping rules for one specific construct, unless this is Poe’s law at work. I also am not sure how it is really solves the same problem. Finally, := is already used for assignment expressions, so it would have to be something else.

6

Instead of being black box/white box, I would prefer a gray box solution, consider following:
black box: loop has it’s own scope, it does not leak it’s internal variable outside the loop
white box: loop does not have it’s own scope, it just using it’s parent’s scope

With “gray box”, loop has it’s own scope, with the possibility to explicitly exporting it’s scope variable to parent scope.

7

In C++ you can either define a local loop variable or use an outer one. Would that easily be the case? If not, it seems to fix one thing, but not everything. While I lean in favor, this just seems like way too big of a breakage to make sense, possibly even in a 4.0 version.

But is there a poll choice missing? I read option as maybe a program or module-level option, but what about a new type, or keyword?

for i: LoopVar[int] in lst:
     x: LoopVar[int] = i+5
    ...

It has the declarative familiarity of the usage in most languages, avoids going un-noticed, and is backward compatible.

1 Like
8

Realistically, a new type seems to be the most feasible way to implement this, as it doesn’t touch the syntax or semantics of the language itself or its behavior at runtime, could be verified statically by type checkers, there’s related precedent with things like Final, and could progressively start as a per-type-checker extension, then if it is successful and a PEP is accepted, graduate to typing/typing-extensions. Type checkers could verify that any name defined with type LoopVar (or whatever name is used, as there would surely be bikeshedding on this point) were not used outside the scope of the loop without being redefined first, it would have to disallow redefining any name previously defined outside the loop, though that is consistent with the behavior of type checkers in general, and is not commonly used and is rather somewhat of an antipattern that could cause confusion.

1 Like
9

Well… sure if it’s only done using a type checker. But not of course if it’s actually a change in python. While a change in python is a change in python, it’s still not a change that breaks any existing code, and shouldn’t realistically cause any confusion either, or at least nobody would be confused without realizing they’re confused.

10

My mistake was probably obvious, but types aren’t allowed there on the loop variable, presumably because of the ambiguity of the colon. Several alternatives could be used and apparently are already used:

And I like this one:

for i in range(5):  
    i: LoopVar[int]
    ....

The downside to starting it in a type-checker [edit: the type itself still would presumably ship with the language], is that then if it is added to the language later, and then if shadowing is added to the language too, then you’ll get valid code written that will work and is expected to work on the latest version of the language, even without type checkers, but that code will be also be accepted on the intermediate versions (where they type was available but shadowing wasn’t implemented), but will silently bug on intermediate versions when not using the type checker. Of course if shadowing is never added to the language that code would still bug without using the type checker. The difference then is everyone would expect that it should bug.

If you add the type and the shadowing change at the same time, python either gives an error on versions where the type didn’t exist, or it runs it properly.

Edit: My vote would be add it once and add it right.

11

Why are we discussing this? More than a supermajority (75%) are in favour of the status quo according to this poll.

Unless there is a really strong argument for adding this complexity to the language, nothing is going to change.

It would be unprecedented for the scope of a variable to be determined by the type of the object bound to that variable.

It would also be difficult to implement efficiently: scoping rules in Python are determined by the interpreter at compile time. Type information of values is not available until runtime.

If we did this, the semantics would be weird. The scope of the variable could change according to the position of the loop:

values = [2, 'c', LoopVar(2), None]
for x in values:
    print(x)

As the loop proceeded, the variable x would be local, local, loop-only, local again.

I think it is clear that a new type is not a feasible way to implement this at all. Using type annotations as a declaration is a bit more feasible, but not much: type annotations have no meaning to the interpreter, and using one to declare scope would again be unprecedented. Types and scopes are unrelated entities.

Purely as a hypothetical exercise, if we were to introduce a change to scope, there are three feasible ways, none of them involving types:

(1) Use a future import to introduce the new behaviour for the entire module, with the expectation that after three or five new versions of Python, the new behaviour would become mandatory:

from __future__ import local_loop_variables

(2) Use a declaration, like nonlocal and global, perhaps inside the loop:

x = 0
for x in [1, 2, 3]:
    loopscope x
    print(x)
assert x == 0

(3) Use a new syntax in the for loop, say:

x = 0
for x in [1, 2, 3] as loopscope:
    print(x)
assert x == 0

I suppose there is a fourth alternative, one which is not an actual change of scope but would act almost the same as one: autodeletion.

x = 0
with autodelete('x'):
    for x in [1, 2, 3]:
        print(x)
assert x == 0  # This line will now raise NameError or UnboundLocalError.
2 Likes
12

Sorry to cause any offense.

I voted no on the poll, but what is being discussed in the comments is something else, mostly ideas that would not break code and would be noticed in-place in all new code even if a reader didn’t realize some new option was set somewhere. I don’t have a strong opinion about importance or implementation issues beyond syntax.

For what it’s worth, I agree that a new keyword would be a good way. Anyway, just thoughts from a random user. Feel free to ignore.

13

Given all that, the answer was clearly (and I would hope) no.

1 Like
14

To be clear, I wasn’t myself advocating for this, but rather simply responding to the user who advanced several proposals for it as to the most practically feasible (i.e., that wouldn’t require a core language change at all, and could be implemented first by third party tooling). Being one of the PEP Editors, I’m naturally well aware how unlikely it would be for a core language change (new syntax, new keyword, or changed semantics) to be accepted, given not only its lack of popularity but the other issues, concerns and lack of positive consensus involved, and the ability to address the issue with tooling instead, which I shy I urged the user toward the possible course of action that would not require any of these, while accomplishing most of what they are looking for.

I’m talking about a new static type annotation in typing evaluated by static type checkers, as the user I was replying to had proposed, not a new runtime type (in types, etc). Of course a new runtime type would be nonsensical for this, but that’s not what’s being discussed here. As mentioned there’s a number of relatively similar cases in static typing where a name’s type annotation changes its semantics for static typing tools beyond just type compatibility (including in what scopes it can be assigned to and where it can validly be found) but not at runtime, including Final, ClassVar, Never, NoReturn, TypeGuard, etc. It would be at least plausible that the typing community might at least consider something like this.

Couldn’t you more or less do something like that right now with, e.g.,

class autodelete:
    def __init__(self, name, scope):
        self.name = name
        self.scope = scope

    def __enter__(self):
        return self.scope

    def __exit__(self, exc_type, exc_val, exc_tb):
        try:
            print(self.scope)
            del self.scope[self.name]
        except KeyError:
            pass

would could be used like so:

with autodelete('x', vars()):
    for x in range(3):
        print(x)
assert x
15

Depending what runtime means, @dataclass makes very substantial “runtime” (it’s not a separate run of some code checker anyway) changes to scope based on type annotations. Of course that’s the decorator generating code so maybe beside the point, but one shipped with my python. Anyway, I’m definitely not advocating for annotations over keywords.

16

In what way?

17

No offense taken.

The original poll gives three options: keep the status quo, change all loops to always be in their own scope, and to support both behaviours. The poll was agnostic towards the mechanism and syntax used.

If we were to change the behaviour, either by adding a way to opt-in to loops with their own scope, or by changing all loops, there would need to be a PEP demonstrating very good reasons for the change.

People are free to invent and use features outside of the language and standard library itself, such as a linter which checks for use of loop variables outside the loop and complains.

1 Like
18

Well, this is what I was trying to communicate anyway, possibly incorrectly (maybe scope is the wrong word?):

from dataclasses import dataclass 

@dataclass
class MyClass:
    a : int = 0
    b = 0
    
x=MyClass()
MyClass.a=1
MyClass.b=1
print(x.a)
print(x.b)

ouputs:

0
1

@dataclass creates a dict from all attributes annotated at class level, and by default those all are initialized automatically in __init__ as instance variables. x.b though still refers to a class attribute. Again, by runtime here I just mean it happens when you run the python script, not just some linter/type-checker/etc.

Also it’s true the poll was agnostic about the “option” option, but there’s a choice of something good, something bad, and something not yet thought out or well defined, and not clearly possible to define, (edit:) and actually to me did not sound like it described things like this, so…

1 Like
19

Right, though if the feature was successful in one or more of the major type checkers, it could be submitted as a Typing PEP and added to the standard library typing module, like many others have. But in thinking about it its a little silly to need a type annotation in that spot for loop-scope specifically, since a type checker could just see that it was a loop variable from the AST and could have an option to automatically restrict such variables to that scope (perhaps if otherwise annotated). And of course, linters could too.

Whether a class-level name has an annotation controls whether it is treated as a field, but it otherwise has no special effect at runtime (of course, there are third-party packages like Pydantic where it can do much more).

If you want to change things at runtime, a type annotation is very much not the way to do it here, as opposed to what @steven.daprano suggested; on top of it being very unidiomatic and unexpected for this use case, it is doubly awkward as you can’t even annotate a loop variable without doing so within the loop body, at which point it wouldn’t make any sense to change the scope there.

I suggested that option you proposed because of it allows you to do the same thing statically with popular type checking tools and a path to standardization, but with much less friction than a language/runtime change, but you wouldn’t really need the annotation at all, just a type checker (or linter) option, so I suggest directing your efforts down that path.

20

I like Steven’s fourth alternative idea, which can be improved (I think so) as:

class Enclosure:
    def __enter__(self):
        locals = sys._getframe().f_back.f_locals
        self.__preset = set(locals.keys()) # copy keys as set

    def __exit__(self, *args):
        locals = sys._getframe().f_back.f_locals
        for k in (set(locals.keys()) - self.__preset):
            del locals[k]

would be used like:

s = 0
with Enclosure(): # similar to c encolosing with {...}
    for x in range(5):
        s += x
print("s =", s)
print("x =", x)

The last line will raise NameError: name 'x' is not defined.

1 Like
22

I feel like if you want to do something like this, you may as well just use a curly brace language (maybe Go?). You’re basically re-inventing curly braces here. Why use Python if you are throwing away the entire gimmick of a significant whitespace?

1 Like