Revisiting PEP 505 – None-aware operators

[gcewing (Greg Ewing)]

“Adding support” is the wrong way to think about this. The point isn’t
to make something possible, it’s to decide on semantics that are easy to
document, understand and reason about.

As far as I can see, the only reason for having the (IMO unintuitive)
parentheses-escaping semantics is to avoid having to change the AST. But
that’s letting the implementation determine the spec, whereas it should
be the other way around. The spec should be designed based on what makes
the most sense to people, and the implementation should do whatever is
necessary to implement it. And all the evidence in this thread says that
most people expect the short-circuiting not to escape parentheses.

I don’t think that changing the AST is that much of a deal-breaker. Type
checkers etc. are already going to have to be changed to accommodate the
new ?. operator, and we’ve warned people all along that the AST can
change between Python versions.

Also keep in mind that the AST exposed to Python programmers doesn’t
need to have the same structure as the one used internally by the
compiler. There’s already a translation layer between Python types and
the internal data structures, and some rearrangements could be made to
happen there if desired.

[Nineteendo (Nice Zombies)]

@cdce8p

For a?.b.c this is the current AST

Module(
  body=[
    Expr(
      value=Attribute(
        value=NoneAwareAttribute(
          value=Name(id='a'),
          attr='b'),
        attr='c'))])

How about using flags? (like lazy imports)

Attribute(
  value=Attribute(
    value=Name(id='a'),
    attr='b',
    noneAware=1,
    shortCircuiting=1),
  attr='c',
  noneAware=0,
  shortCircuiting=1) # 0 for (a?.b).c

[JakobStadler]

How many times do all of you actually access an attribute or call a method of None?

I can’t recall a single time where I would have really needed this, so I suspect the discussed AST changes are mostly pointless anyways.

And in the rare case I would need it, I could use the other proposed tools to still make it fairly compact, i.e. (a?.b.c ?? None).__bool__()

[bwoodsend (Brénainn Woodsend)]

The method call short circuiting presumably makes more sense in languages that use get/set methods instead of properties/direct attribute access. i.e. a?.getB().getC() instead of a?.b.c

[elis.byberi]

@gcewing

The spec should be designed based on what makes
the most sense to people, and the implementation should do whatever is
necessary to implement it.

I completely agree with your reasoning.

---

For a chain of attribute accesses like obj.a.b.c, any extra parentheses around parts of the chain do not change the meaning or behavior.

• obj.a.b.c
• (obj.a).b.c
• ((obj.a).b).c

We all know how ?. works in almost all other languages, except Rust. The question is how we would teach it to beginners. Why do parentheses change its behavior now?

When I read obj?.a.b, I desugar it as:
(obj.a if obj is not None else None).b
Am I doing something wrong?

[JakobStadler]

@elis.byberi

When I read obj?.a.b, I desugar it as:
(obj.a if obj is not None else None).b
Am I doing something wrong?

The advantage of the ?. operator is short-circuiting. So result = obj?.a.b becomes result = None if obj is None else obj.a.b. Without short-circuiting, you would need to add ?. to every part of the chain, which i.m.o. is worse and increasing cognitive overhead.

[rsdenijs (Roderick de Nijs)]

If you were forced to add ? at every point in the chain you lose the ability to differentiate between only the first part being possibly None or any part being possibly None.

[elis.byberi]

Adding ?. to every attribute seems overkill, but do we really need to add it at all? Are there any particular reasons we are meticulously tracking the optionality of each attribute?

[cdce8p (Marc Mueller)]

Thanks for the discussion so far. It seems to me some feel quite strongly about (a?.b).c. I took some time to reflect on what I’ve written yesterday on why limiting the scope of short-circuiting should should be rejected.

One of my arguments against it was that type checkers should need to warn about accessing .c on an optional value again. On closer inspection this could also be an argument in favor though. type checkers and IDEs will warn you if it’s used incorrectly. Do I still believe it should almost always be used with a fallback? Yes! However, we’re all consenting adults here. If someone really once to use that pattern without fallback, why not let them? Of course, if it can be prevented on a language level that’s great but if not, linters could always emit a warning for it.

Thinking one step further, they might also be able to change (a?.b.c)?.d to just a?.b.c.d if they recognize it’s a safe transformation.

Where does that leave us? The whole idea for none-aware access operators is that it’s basically a transformation which is applied to an expression. a?.b is equivalent to

_t.b if ((_t := a) is not None) else None

where a and b can be replaced with any other “arbitrary” expression. It might get complicated to read for two or more ?. due to the left-recursive nature but it should still be possible. Applying the same logic to (a?.b).c just replacing the inner group part would then be equivalent to

(_t.b if ((_t := a) is not None) else None).c

which would indeed raise an AttributeError if a = None thus the short-circuiting would not escape the group itself. I believe that’s what all you guys were trying to tell me before.

That just leave the question how to implement it but as @gcewing rightly called out the implementation should not drive the design. As a maintainer of pylint and contributor to mypy I frequently work with the AST generated by Python, so I feel strongly that any change we make to existing nodes should be as limited as possible. Thankfully, there might be a solution which we haven’t discussed so far. We could add a new group attribute (maybe someone will come up with a better name) to the Call, Attribute, Subscript, NoneAwareAttribute and NoneAwareSubscript nodes which will be set to 1 if it’s the topmost node in a group. We can then use the attribute to start a new block with its own jump target during bytecode generation.

To summarize, I’ll be updating my draft again to point out that short-circuiting does not escape a group.

[BrenBarn (Brendan Barnwell)]

@gcewing

Conceptually, this is not very different from

a and b and c

If a is falsey, neither b nor c are evaluated.

But that is not an attribute access. I see this proposal as defining a new form of attribute access, and I think the proposed behavior is too different from existing attribute access. In particular my point is that currently a chain of dotted names results in a chain of separate attribute accesses; this proposal would turn a chain of ?-and-dotted names into a single short-circuiting chunk, changing the behavior not only of the ?. parts but also of the parts that just use ordinary dots. That’s too big of a change, in my opinion.

Another difference with your example is that (unless I’m mistaken) in a and b and c, in fact both ands are evaluated; it’s just that because a is falsey, the result of the first a and b is a, and since that is still falsey the second one still evaluates to a. In other words, the entire expression does not short-circuit; each sub-expression short-circuits. There is no action at a distance. There is no error condition possible here that is separable between the two ands, since in both cases the left operand winds up being a (so if its __bool__ raises an error it will fail at the first and). That is not the case for attribute accesses, where there is existing defined behavior for accessing a nonexistent attribute on None (namely an AttributeError).

[rsdenijs (Roderick de Nijs)]

changing the behavior not only of the ?. parts but also of the parts that just use ordinary dots.

the parts with the regular . do not change, they simply do not run if a previous ?. goes through the unhappy path.

@BrenBarn

Another difference with your example is that (unless I’m mistaken) in a and b and c, in fact both ands are evaluated;

the second and is not evaluated, e.g. you can try 0 and 0/0 and it will run just fine.

[gcewing (Greg Ewing)]

If you look at the bytecode, it actually skips the second and when a is
false.

You’re right that this could be seen as just an optimisation. But when I
read the code, I’m not thinking about it that way – I’m thinking “if a
is false, none of the rest will be executed”. And that’s the same way
that a?.b.c should be thought about.

[gcewing (Greg Ewing)]

I think he meant that the second sees which short-circuits to . It’s true that this would give the same result, although it’s not actually implemented that way.

[BrenBarn (Brendan Barnwell)]

@gcewing

But when I
read the code, I’m not thinking about it that way – I’m thinking “if a
is false, none of the rest will be executed”. And that’s the same way
that a?.b.c should be thought about.

I don’t think about it that way. I think about it as a sequence of operations that happen one at a time. I mean, obviously we disagree, but I’m just saying my perspective is that the language is easier to reason about when the evaluation rules are simpler and happen one step at a time, with less action at a distance.

@gcewing

If you look at the bytecode, it actually skips the second and when a is
false.

This actually makes me consider raising it as a bug. As far as I can tell that behavior is not consistent with the documentation.

[gcewing (Greg Ewing)]

I don’t think this is an improvement – if anything, it’s even more
confusing that it was before. If I didn’t already know how it worked,
this would leave me with no clear idea of how, or even whether,
parentheses affect the short-circuiting.

There’s a Zen principle that I think applies here: If it’s hard to
explain, it’s probably a bad idea.

[h-vetinari]

@gcewing

I don’t think this is an improvement – if anything, it’s even more
confusing that it was before.

This is slightly misleading in the UI

because it’s responding to a much older comment (~30h ago)

whereas @cdce8p has more or less walked this back in a more recent comment (16h ago) already.

[guido (Guido van Rossum)]

@gcewing

you look at the bytecode, it actually skips the second and when a is
false.

You’re right that this could be seen as just an optimisation.

It isn’t — it’s mandatory semantics. It determines how often __bool__ is called, and that may have side effects.

[BrenBarn (Brendan Barnwell)]

@guido

It isn’t — it’s mandatory semantics. It determines how often __bool__ is called, and that may have side effects.

Is it documented?

[guido (Guido van Rossum)]

@BrenBarn

@guido

It isn’t — it’s mandatory semantics. It determines how often __bool__ is called, and that may have side effects.

Is it documented?

Good question. It should be. I don’t know if we ever made this clear in the spec, but you could look. Maybe @markshannon remembers? (Mark, I’m pretty sure you had a strong opinion about this in some internal “Faster CPython” discussions.)

[mikeshardmind (Michael H)]

yes, see here: 6. Expressions — Python 3.14.2 documentation

The expression x and y first evaluates x; if x is false, its value is returned; otherwise, y is evaluated and the resulting value is returned.

The expression x or y first evaluates x; if x is true, its value is returned; otherwise, y is evaluated and the resulting value is returned.