Guido and I were looking at an old bug this week. It’s about an inconsistency in how variables are resolved in class namespaces. There are several possible solutions, including closing the bug as “working as intended.” I drafted a PEP to describe the problem and propose three non-trivial solutions. I’d be interested to hear which of the approaches in the PEP seem reasonable to people.
2 Likes
Does not it conflict with my PEP 837?
1 Like
I didn’t see your PEP 837! I thought the process was to take the next available number. How can we fix it?
837-839 are all open PRs. The next available is 840, let’s switch this one to that: python/peps#5043.
For next time, see hugovk.dev/next-pep-number/ or:
❯ uvx pepotron next
Next available PEP: 840
2 Likes
Loooking a bit more about the various examples, I think this is the one that’s wrong (and has been since 3.4):
x = 1
def f(x):
class Test:
x = x
return Test
print(f(2).x)
This prints 1 because for a variable name that the static analysis sees is written in the class, the classic rule “try locals then globals” is used regardless of whether there’s a definition for the name in a containing function scope. But when the name is not assigned locally and there’s a containing function scope that defines the name, the rule “locals, then outer” is used. OTOH when there’s no definition in a containing function scope, the classic rule is also used.
My view is that for the first case (written locally), if there’s a definition in an outer function scope, the “locals, then outer” rule should also be used.
1 Like
What’s the rationale for classes and functions differing in this respect?
>>> x = 0
>>> def f(x):
... def g():
... x = x
... return g
...
>>> f(1)()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in g
UnboundLocalError: local variable 'x' referenced before assignment
>>>
I feel uncertainty about the two kinds of name resolution working differently. Why should the variable x in “x = x” mean different things on the LHS and RHS?
The best case I can make is backwards compatibility. You could refer to two different variables with “x = x” for so long that it’s too late to change it.
Jeremy
Name lookup in a class scope is just different. It’s normally dynamically scoped: local first, then fallback (global or containing function). This is different than name lookup in a function — that’s always statically defined (the decision to load a global is statically made based on scope analysis).
I looked into this behavior in some detail when I worked on PEP 695: see my post at Implementing PEP 695 | Jelle Zijlstra and some of the examples in my PyCon talk a few years ago (slides at https://jellezijlstra.github.io/talks/pycon2024-pep695.pdf from slide 34 onward).
The current class scoping behavior is indeed questionable, but the practical impact seems limited and changing a core language behavior is always risky. Therefore, I’d lean towards making no changes and documenting the existing behavior.
A useful discussion to look at would be PEP 709: one behavior change that was missed in the PEP, where we discussed another subtle change related to class scopes. There we gathered some evidence from open source code to see how common the affected kind of code was in the wild.
6 Likes
I acknowledge that the practical impact is limited and the risks a bit hard to ascertain. I hadn’t seen your talk, but it looks like the PEP recapitulates the point you were making about class namespaces being weird. You make a point that “You never know what’s actually in the class dict” and that seems fundamentally right; it’s hard to do much static analysis of variable use in class namespaces.
It does make sense to analyze a large collection of code to see what the impact might be. I think there are two approximate tests that would be useful. First, look for cases where a straight line analysis shows that a variable is used in a class before binding and also has a binding in an enclosing; these would be cases where we’d expect program behavior to change (but may not). Second, the more general issue is that any local could be deleted unexpectedly, so analyze how often you’re creating a “fallback cell” that isn’t obviously needed but exists in case the local disappears.
Jeremy
I was impressed by the thoroughness of the research done for the class scope issue in PEP 709 (linked by Jelle). But despite that nearly all occurrences found were either false positives or bugs where the proposed new semantics were mistakenly assumed, the old behavior was restored before the PEP was accepted in 3.12.
This suggests that no matter the outcome of similar research we could do for the current proposal, it’s likely that the status quo, inconsistent as it is, will prevail for backward compatibility reasons.
Therefore I don’t want to waste more of my time on this discussion.
3 Likes
I would suggest always requiring qualification, like class variables and instance variables, of anything non-local, i.e.:
global.x
nonlocal.x
nonlocal.nonlocal.x # 2 scopes out
class.x # Inside the class itself
class.class.x # Inside 2 nested classes themselves
ClassName.x # As is
ClassName0.ClassName1.x # As is
self.x # As is
x # Local only
So the example in PEP840 would become:
>>> x = 0
>>> def f():
... x = 1
... class A:
... x = 2
... del class.x
... a = global.x
... return A
Ultimately this is a breaking change. However it can be introduced softly over time by staging:
- Allow new qualifications but also allow old syntax.
- Warn about use of old syntax.
- Error on old syntax.
With this process staged over many years.
Edit: Corrected a typo and added another existing syntax example.
The current class scoping behavior is indeed questionable, but the practical impact seems limited and changing a core language behavior is always risky.
What if the current behavior is kept, but we could allow either “global” and “nonlocal” keywords to be used in the class body to disambiguate the behavior when needed, from here forward?
You already can!
I was involved in fixing a few subtle bugs that emerged when global and nonlocal are used within a class body when implementing PEPs 649/749.
It certainly highlighted how weird the current behaviour is, but there’s now been so much code written around it, I think it’s too big of a breaking change now.
1 Like
You already can!
Ooops - I see your answer, and go try it myself, only to release that (1) yes, one gets a reliable and previsible behavior with both global and nonlocal, and (2) such behavior consists in updating the variable outside the class body scope, and one gets no attribute assignment in the class itself, which is what the PEP is about. 