Painful details of variable scope mixed with classes

I’m revisiting variable scope technicalities in Python for my personal interpreter project. Some time ago, I asked about that and got the tip that CPython has a multi-pass system that figures out variables, scopes, and bindings ahead of generating byte code. I started doing similar, and I was doing great with just functions, but I realized I wasn’t managing classes with it correctly.

It’s pretty natural to delve into some really obscure stuff when you start testing for this stuff, and I can say that I’ve lost myself.

So here’s something that kind of puzzles me:

a = 1


class Foo1:
    a += 1

    def __init__(self):
        global a
        self.local_a = a + 1

    class Foo2:
        a += 100

        def __init__(self):
            global a
            self.local_a = a + 1


f = Foo1()
print(a)
print(Foo1.a)
print(Foo1.Foo2.a)

g = Foo1.Foo2()
print(f.local_a)
print(g.local_a)

print(a)

Result:

1
2
101
2
2
1

It looks like Foo1 and Foo2 on first invocation will get a copy of the global a and do their own thing with their copy. The initializers grab the root level one.

If I qualify the class members with ‘global a’ then I’ll be playing with the global one at the root just fine. Otherwise, nothing happens to it. If I want Foo2 to touch the global ‘a’, I have to make it global in Foo1 as well:

class Foo1:
   global a     # Need this if I want Foo2 to see it.
   ...
   class Foo2:
      global a

If I had nested functions, I wouldn’t have to “carry” the global.

I hope somebody can explain how classes muddle with variable scopes. At first glance, it looked like a class declaration creates a jail blocking against upper scope and takes copies of globals inside of itself. Then a class method runs and can break right out of that to reach globals outside of the class anyways.

…work in a language for nearly 15 years and then break your brain on this kind of thing…

There’s a lot to cover here, so I’m going to break it down over multiple posts with simplified examples.

a = 1

class Spam:
   # Could also use a += 1
   a = a + 1

assert a == 1 and Spam.a == 2

In this snippet, the name lookup for a returns the global a but binds a local a.

This works all the way back to Python 1.5 so I guess it is intentional. The PyPy developers seem to believe it is intentional, as they have copied the behaviour.

(Disclaimer: I have only tested it in CPython 1.5 and 3.10, and PyPy 2.7.)

I believe what is happening here is the inside a class, the interpreter is using the full LEGB scoping rule without the function optimization.

That is, any name lookup (outside of a function):

  1. searches the local scope for that name (L);
  2. if not found it searches any enclosing (nonlocal) function scopes (E);
  3. if not found it searches the module level globals (G);
  4. if still not found it searches the builtins (B);
  5. and if still not found it raises NameError.

At the top level of a module, the local scope is the global scope, so the search path is just GB, or LB if you prefer.

Name bindings theoretically apply in the same order, however the local binding always succeeds, so this is effectively just an immediate local binding. (Unless declared global.)

If my model of the interpreter is correct, then this:

a = 1

class Eggs:
    a = a + 1
    a = a + 1

assert a == 1 and Eggs.a == 3

should pass. And sure enough, it does.

It is only functions which are special, and use an abbreviated lookup for locals (L only). So inside a function, you cannot replicate that Spam.a behaviour in CPython. Instead, you get a NameError subclass, UnboundLocalError:

a = 1
def spam():
    a = a + 1

spam()

This raises UnboundLocalError: local variable 'a' referenced before assignment.

(Other interpreters are permitted to allow this, and behave like a class. The fast locals trick is documented as an interpreter implementation, not a language feature.)

1 Like

Next we can look at nested classes. When a class is nested inside another class, the surrounding class is not part of the variable search path. The E in LEGB only refers to enclosing functions.

a = 'global'
class Spam:
    a = 'class local'
    class Eggs:
        b = a  # Picks up the *global* a
        a = 'local'  # Now we have a local a
        a = a + '!'  # And this uses the local a

assert a == 'global'
assert Spam.a == 'class local'
assert Spam.Eggs.a == 'local!'
assert Spam.Eggs.b == 'global'

So inside the nested class Eggs, the scopes are L (the Eggs class body), there is no E, G (module globals) and B (builtins).

If you try inserting a nonlocal declaration inside Eggs, it fails because there is no enclosing function.

2 Likes

Oh man I didn’t even think about trying the a + 1 statement twice in a row to see what happens. That’s, uh, wow. Huh. I’d write more now but I’m hosting a party tonight and I think I’ll need to sneak in a drink! =D I’ll give you a cheers haha.

I’m still digesting all of it. Somehow I got it in my head that the double a + 1 statements resulted in Eggs.a == 2 instead of 3, which was really hurting my head. You did help me scheme some ways to perform these different operations without it become a huge pile of paranoid-red-yarn-if-else-garbage so thanks. I may end up following up in a week or two of free time hacking if I tripped on something else.

How pedantic are we getting with functions? I had an interesting effect with methods:

a = 100

class Spam:
    a = 101

    def __init__(self, some_num):
        self.a = a + some_num

eggs = Spam(1000)
print(eggs.a)

This will give me 1100. This appears to mean that the initializer is grabbing a as a global as you might expect of a non-function LEGB binding order lookup.

I have to admit I didn’t really understand the fasts optimization. I implemented it in my little project too to make comparisons much easier, but I didn’t really understand the . . . scope of it haw haw haw.

Steven covered this here:

You can access class or object variables through the dot operator (not by their plain name): Spam.a or self.a.

Spam.__init__ is a function. Spam(1000).__init__ is a method. When a method is called, the __func__ attribute of the method in turn gets called with the method’s __self__ attribute inserted as the first positional argument (e.g. self). For example:

>>> Spam.__init__
<function Spam.__init__ at 0x7f4e33f6ad40>
>>> s = Spam(1000)
>>> s.__init__
<bound method Spam.__init__ of <__main__.Spam object at 0x7f4e33f5fd50>>
>>> s.__init__.__func__ is Spam.__init__
True
>>> s.__init__.__self__ is s
True

In your example, in the scope of the __init__ function call, variable a is the global variable defined by a = 100. The instance attribute self.a is thus assigned the value of the expression 100 + 1000.

The class itself can be referenced as the instance attribute self.__class__. One can also use the implicit __class__ closure. See creating the class object. For example:

>>> code = compile(r'''
... a = 100
... 
... class Spam:
...     a = 101
... 
...     def __init__(self, some_num):
...         self.a = __class__.a + some_num
... 
... eggs = Spam(1000)
... print(eggs.a)
... ''', '', 'exec')
>>> exec(code)
1101

In Spam.__init__, the variable __class__ is a free variable. It gets assigned the contents of the corresponding closure cell when the function is called:

>>> Spam.__init__.__code__.co_freevars
('__class__',)
>>> Spam.__init__.__closure__
(<cell at 0x7f4e3419faf0: type object at 0x555faaea4840>,)
>>> Spam.__init__.__closure__[0].cell_contents is Spam
True

In theory, the compiler could emit code to implicitly reference __class__.a for variable a, but that’s new behavior. We’d need a keyword to indicate that a is a class variable.

4 Likes

While refining the implementation, I found some more confusing things with built-ins. So let’s say I’m using print() or len(). I’m seeing them getting put on the stack using LOAD_GLOBAL. There isn’t a LOAD_BUILTIN opcode, so it’s not like I can expect that. I would have figured maybe LOAD_NAME because I need that LEGB/LGB resolution to hit the built-ins. Instead, LOAD_GLOBAL somehow kicks things in motion to resolve built-ins. Does this mean that a LOAD_GLOBAL opcode will search globals and then fall back to built-ins?

In code that’s compiled to use optimized (i.e. fast) locals, dynamic locals and import * are not supported, and if a variable is local, then it’s always local. Note that in this case, trying to reference a local variable before it’s assigned, or after it’s deleted, will raise UnboundLocalError. With this knowledge in hand, the compiler knows that a free variable that’s not from an enclosing function scope can be assumed to be in globals or builtins. It thus uses the opcode LOAD_GLOBAL, which checks f_globals and f_builtins.

The above is in contrast to code that’s compiled to use non-optimized locals (i.e. locals stored in the f_locals mapping), such as the top-level code of a class body, module, or exec(). In this case, the scope of a name can change fluidly between locals and globals, or locals and nonlocals. Also, dynamic locals are supported, as well as import * usually (except not in a class body). Unless a variable is explicitly declared global, or is from an enclosing function scope, the compiler references it via LOAD_NAME, which checks f_locals, f_globals, and f_builtins, in that order. If a variable is explicitly declared global, then LOAD_GLOBAL is used instead of LOAD_NAME.

In a class body, a variable from an enclosing function scope may be referenced by the opcode LOAD_CLASSDEREF, which is a hybrid lookup. If the name isn’t defined in the f_locals mapping, the interpreter falls back on loading the cell object from the enclosing scope and dereferencing its contents. Note that if the name is assigned to as a local variable, then the compiler will reference the variable using LOAD_NAME instead of LOAD_CLASSDEREF, except if the assigned variable is declared nonlocal. For example:

def f():
    x = 'nonlocal 1'
    class C:
        nonlocal x
        print(x)
        x = 'nonlocal 2'
        print(x)
        x = 'nonlocal 3' 
        locals()['x'] = 'local' # dynamic assign
        print(x)
        del locals()['x'] # dynamic delete
        print(x)
>>> f()
nonlocal 1
nonlocal 2
local
nonlocal 3
2 Likes

Thanks. I may look at the opcode dump in the dis package documentation and see if it could use a few caveats (if they’ll take them). I haven’t even looked up LOAD_CLASSDEREF and it sounds like I wish I never knew about them, so thanks for that lol. :stuck_out_tongue:

More a progress report than a question: I think I have gotten it mostly down, although I still find some gotchas. Something that regressed in all this was variable resolution in modules. If I have module foo:

def terminal_call():
   return 1

def outer_call():
   print(globals())
   return terminal_call()

And I run it as:

import foo
foo.outer_call()

I’ll see the globals when in outer_call are terminal_call and outer_call. This is what I’d expect. Currently, my implementation has broken this association in a “how did this ever work in the first place?” kind of way; I did have this working before lol. So now I have to (re)figure out how variable state is ferried when going across modules. What a Gordian knot.

Every module has its own storage for global variables. This storage, or “namespace”, is a dict attached to the module object.

The details don’t really matter, but for the record this dict is called __dict__ (two leading and trailing underscores).

To access that namespace of a module, you can use:

  • 'vars(module)` to access the namespace of the given module;
  • or more commonly, globals() to access the namespace of the current module.

When I say “current module”, I mean the module where the call is written, not the module where it is called from. That might be a bit confusing, so let me explain:

Let’s say we have a module called “mylibrary.py”, and in that module we define a function and some global variables:

# mylibrary
a = 10
b = 20

def func():
    return globals()

Then we have a second module, “myapp.py”, which imports that module, and also has some global variables:

# myapp
a = "Hello"
b = "Goodbye"

import mylibrary

print(globals()['a'])         # This will print "Hello"
print(mylibrary.func()['a'])  # This will print 10

Inside myapp, the direct call to globals() always returns myapp’s global namespace, so accessing the name ‘a’ will give you “Hello”.

Inside mylibrary, every call to globals() always returns mylibrary’s global namespace, even if the call is made from a different module.

So even though we are calling mylibrary.func() from mylibrary, the call to globals itself occurs inside mylibrary, not myapp, so it returns mylibrary’s global namespace, and accessing the name ‘a’ returns the value of mylibrary’s global variable, namely 10.

Until you wrap your head around it, this may be a bit complicated, but eventually it will seem absolutely normal.

The same rules apply even if you use from mylibrary import func.

The justification is this:

If you have a function in a module, the global variables that function sees should be the globals in the same module, not the globals in the module which merely happens to call that function.

To give an analogy, if modules are countries, then when somebody in the USA says “what’s the capital city of my country?” they should get the value Washington. If somebody in Italy asks the same question, they will get the value Rome.

If somebody in the USA invites a person from Italy to visit (importing a module), and then asks that Italian “What’s the capital city of your country?”, it doesn’t matter that the question was asked in the USA, the answer is still Rome.

I hope this helps.

A code object doesn’t have a reference to the builtins and globals scopes that are required to execute it. Thus exec() and eval() take the globals to use as an option, which defaults to the current globals. The builtins scope can be set as __builtins__ in globals, else it defaults to the current builtins.

A function object references its builtins and globals scopes, which are exposed as its __builtins__ and __globals__ read-only attributes. It doesn’t matter from where a function is called. It always uses the same builtins and globals. For example:

>>> b = vars(builtins).copy()
>>> g = {'__builtins__': b}
>>> exec('f = lambda: spam', g)

>>> g['f'].__builtins__ is b
True
>>> g['f'].__globals__ is g
True

>>> b['spam'] = 'eggs'
>>> g['f']()
'eggs'

Note that the __builtins__ reference in globals no longer matters to the function object because it has its own reference.

>>> del g['__builtins__']
>>> g['f']()
'eggs'
2 Likes

Thanks everybody. I was just kind of venting when I posted that because that particular resolution used to work. However, I found out the way it used to work was a raging tire fire so I having that information was very useful. While independently sifting through some stuff, I noticed the module gets __builtins__ but doesn’t get __globals__. I guess that can be derived through other means if it was necessary? Or rather anything callable inside of it has the globals anyways so it doesn’t matter. I still thought it was peculiar.

Is there a PyCon talk on this level of painful detail or y’all trying to suck me into it? :stuck_out_tongue:

When a code object is executed, the __builtins__ value in the globals scope, if it exists, is used as the builtins scope. If the globals scope doesn’t have __builtins__, then the current builtins scope is used and set as __builtins__.

The globals of the frame that’s executed to create the namespace of a module becomes the module’s __dict__, i.e. the attributes of the module. Referencing it again as __globals__ wouldn’t serve any purpose. It can be accessed as either vars(some_module) or some_module.__dict__.

Good news! I managed to make an implementation that passes all my previous tests and some new ones.

Bad news! I forgot about closures that retain variables! I hate myself!

Now I’m a new hell with free and cell variables, along with closures. It’s my understanding that both free and cell variables are enclosed variables and I would access them via LOAD/STORE_DEREF. I need to figure out more exact rules for the sake of implementation. It looks like a cell variable is the origination of an enclosed variable, and any other nested reference is a free variable. Is that right?

My claim comes from making this sandwich (current.py):

def top_bun(w):
   a = 1
   def spam_meat(y):      
      nonlocal a
      a = a + y
      def bottom_bun(x):
        return a + x
      return bottom_bun
   return spam_meat(w)

…and hence poking at the code objects:

>>> current.top_bun.__code__.co_cellvars
('a',)
>>> current.top_bun.__code__.co_freevars
()
>>> current.top_bun.__code__.co_consts[2].co_cellvars
()
>>> current.top_bun.__code__.co_consts[2].co_freevars
('a',)
>>> current.top_bun.__code__.co_consts[2].co_consts[1].co_cellvars
()
>>> current.top_bun.__code__.co_consts[2].co_consts[1].co_freevars
('a',)

And just for completeness, the function I would get from invoking this will have a non-None __closure__ dunder that will contain Cell Objects corresponding to its co_freevars.

>>> sandwich = current.top_bun(100)
>>> sandwich.__closure__
(<cell at 0x00000276065077F0: int object at 0x0000027605F755F0>,)
>>> sandwich.__code__.co_freevars
('a',)

Yep that’s correct. Cell and free variables just refer to where the cell originates from - cellvars means the function creates the cell objects when it’s called and creating its scope, while freevars receive the cell objects externally when the function object is created. If you disassemble top_bun you’ll see it assembling the closure tuple. In the body of the function, locals, cell and free variables are all stored in the same array - *_DEREF just then accesses the cell_contents attribute of the cell.

There is also the fun special case of LOAD_CLASSDEREF mentioned above, used if you have a class definition referring to a variable name defined in an enclosing function. It checks the class namespace first, then dereferences the free var if not found.

In computing, a free variable is a non-local variable. Technically, this includes global variables. That said, in code that’s compiled with non-optimized locals (e.g. compiled in “exec” mode or a class body), it isn’t generally possible to know whether a variable is local or non-local, and the status can change, unless the name is explicitly declared global or nonlocal (if supported) .

At least, that’s how I think it should work. However, the behavior of a nonlocal variable in a class body isn’t what I expect, and it isn’t consistent with the behavior of a global variable. If a variable is declared nonlocal in a class body, the compiler should switch to using LOAD_DEREF instead of LOAD_CLASSDEREF, i.e. the local scope should not be checked in this case. This would mirror how the compiler switches to using LOAD_GLOBAL instead of LOAD_NAME when a variable is declared global in a class body. LOAD_CLASSDEREF should only be used when a non-local variable is referenced without declaring the scope.