I don’t buy that. The four are all closely related because they all check whether their left operand is None and change control flow when it is.
I was wrong about the following – see my later post about it.
Separately, IIRC there were some grumblings about whether to prescribe short-circuit behavior or not. This suggests there are some folks who are somewhat afraid of short-circuiting. Such a fear would be highly irrational, though. ?? is a better version of or, which short-circuits for excellent reasons and always has (not just in Python, but this holds true for || in every language derived or inspired by C, with the probably exception of languages for symbolic algebra).
My apologies! This is new info for me. (Apparently I never encountered this in TS or at least never thought about it. Or I just forgot this aspect since I last studied PEP 505?)
The bytecode would start to be generated from right to left, as it’s recursing towards a, before “going out” again from left to right.
We start with Attribute(..., attr='d'), before we perform the lookup for .c a jump target which skips the LOAD_ATTR bytecode is stored in the compiler_unit. This is marked as the Top target for later.
Next we do the first recursion / lookup for ?.c. This is actually a NoneAwareAttribute. Before doing anything else, we again push a new jump target which would skip any remaining actions. This one isn’t on top though.
Ok, let’s do this dance again for .b, store a new jump target and then actually lookup a with LOAD_NAME (a).
Now we need to “go out” again. First we pop the last jump target (the one for .b). Continue by adding LOAD_ATTR (b). Since the target for .b wasn’t actually the top one when it was added, we don’t add any USE_LABEL for it and instead skip it.
One step out again for ?.c. We again pop the next jump target but this time we also want to get a reference to the Top target we stored early, i.e. to the end of .d. Add COPY and POP_IF_NOT_NONE and set the jump target to the end of .d. If we don’t need to jump, we still need to add LOAD_ATTR (c) and check if USE_LABEL should be added which isn’t the case. ?.c wasn’t the Top target.
Lastly we are at .d again. Pop the next jump target, this is just cleanup now as it’s the last one, add LOAD_ATTR (d) and now also add USE_LABEL since this is the “end” of the expression the is not None check should skip to.
All done now. I needed a bit of additional magic to make the grouping work but the idea is still the same except that in reality I store multiple blocks with targets. Since we don’t get the group directly from the AST, I check the node kind to know weather were still in the same block. Anything other than Name, Attribute, Subscript, Call, NoneAwareAttribute and NoneAwareSubscript will result in a new block with a separate “end” jump target. E.g. (a?.b or c)?.d.e would have two different blocks which are evaluated independently: a?.b and (...)?.d.e
–
With my current implementation the TypeScript grouping for (a?.b).c would be roughly equivalent to (a?.b or None).c. I do wonder what the use case for it would actual be. This is almost certainly going to raise an AttributeError at some point, so it’s not really different from just doing a.b.c.
I hope I understand your question correctly. If X = None, X?.a.b would short-circuit to just None. There is no need to add an additional ?. for b just because X?.a can be None.
>>> X = None
>>> print(X?.a.b)
None
You only need to add ?. for attributes which itself can be None.
>>> class Y:
... a: int | None = None
...
>>> y = Y()
>>> y?.a.b
Traceback (most recent call last):
File "<python-input>", line 1, in <module>
x?.a.b
>>> print(y?.a?.b)
None
True, it’s pretty unlikely that this would cause a difficulty in practice. But the fact remains that you haven’t quite implemented the semantics described in the PEP, since the short-circuiting isn’t stopped by parentheses but by hitting some other operation.
Note that in this comment, when I say “equivalent”, I’m talking about conceptually/mathematically/logically.
I’m not talking about AST or byte code - because this is about how Python programmers think about the language they are using, and mostly they are not thinking about AST or byte code.
We should be able to replace <smaller expression> with something equivalent to <smaller expression> and the overall expression should remain equivalent.
example:
(1 + 2) + 5
We can replace the <smaller expression> like this:
(3) + 5
and the overall expression stays equivalent.
But with this proposal (and current demo implementation), if we have this:
(a?.b).c
If we change the <smaller expression> to something equivalent to the <smaller expression>, then the overall expression is not equivalent.
I’ve already added an apology/retraction, so there’s no need to continue refuting what I said.
I do note that we can’t define Python precisely by thinking how Python programmers think, because most programmers don’t think precisely – and when they do, they probably use suitable abstractions like ASTs.
Also, it looks like Marc Mueller’s current implementation already works the way you want it to – (a?.b).c is not equivalent to a?.b.c because in the latter case .b.c is skipped when a is None, while in the former case only .b is skipped. Unless Marc has a bug in his implementation – his spec spells out that this should happen. (I haven’t had the time to run his implementation.)
This surprised me (I agree with the rest of your post). Why introduce or None here? That would imply that if a?.b evaluates to an empty list, if .c were .sort(), it would fail, because ([] or None) evaluates to None and None.sort() raises AttributeError. What am I missing?
The purpose of that comment is not to refute what you said. I wasn’t thinking about you when writing that comment.
It was just to state more clearly the issue I was seeing, because I don’t think I had expressed it clearly before.
I had said that it didn’t match my intuition, but I was trying to figure out what was leading my intuition.
I’d like to start by saying that I truly appreciate the discussion. The details of short-circuiting is an area which, to my knowledge, hasn’t been discussed much if at all in this topic.
I’ll see how I can incorporate it into my draft.
I believe this is incorrect. Since my implementation does not create separate AST nodes for groups, (a?.b).c and a?.b.c will produce the same AST and thus the same bytecode. Both variants will short-circuit and return None if a = None. I’ll look to make this clearer when I update the draft.
>>> a = None
>>> print((a?.b).c)
None
>>> print(a?.b.c)
None
I should have been more precise. My example was flawed. The idea was to mimic TypeScripts behavior with the current way grouping is implemented in my demo. For the inner expression to be evaluate separately, I need some kind of node that “breaks” the outer expression. In my example that was a BoolOp. However other nodes will work too. Unless I’m missing something again, it should actually be
(_t1 if ((_t1 := a?.b) is not None) else None).c
# or with the coalesce op
(a?.b ?? None).c
I’ll note that Guido liked my original post, which I took to mean he agreed with my interpretation. Now you are saying I was wrong.
Regardless of the final decision over semantics, this feels to me like strong evidence that this feature is hard to reason about - in particular, the short circuiting. Maybe we should remove the short circuiting and simply require users to use a series of ?. operators if that is what they mean?
I agree that it can be hard to reason about - I also misunderstood how short-circuiting worked in parenthesis groups after reading the PEP.
However, I don’t believe that removing it is the best solution. For example, say that we had a chain as follows:
a.b.optional?.c.d.e
If we removed short-circuiting, it would have to be spelled as
a.b.optional?.c?.d?.e
Which could introduce bugs if c or d (which should never be None) ended up as None. Code should explicitly raise anAttributeError instead of silently ignoring the bug.
This is similar to the proposed maybe syntax which was rejected for reasons listed in the PEP.
Reading the recent chunk of the thread and thinking about short circuiting and handing of parentheses, I’m becoming way more sympathetic to the idea of a keyword.
Here’s the fun example which I have in mind
x = None
cls = x?.y.__class__
I don’t think there’s any set of rules which makes that expression not confusing. Of course, “don’t write it like that” is a possible answer.
But do we have a compelling case for needing these sorts of fine grained and subtle semantics? This is the point Barry was making last week and it’s hard to refute without some strong examples of real code which would need this. (I’m not aware of any such cases.)
I don’t think the semantics as described in the PEP are difficult to
reason about. They can be captured using a right-recursive grammar rule
in which the short-circuiting consists of simply skipping the right
branch of a ?. node.
The “parentheses-escaping” behaviour of the current implementation, on
the other hand, seems to defy being fitted into any kind of grammar.
Regardless of whether we expect anyone to actually write such code, at
some point we need to nail down the precise semantics of these corner
cases and describe them clearly in the documentation.
My conjecture is that it’s easier to do that with
non-parenthesis-escaping behaviour than parenthesis-escaping behaviour.
In my experience, ?? and ?. are less useful than ?[...] in Python.
JavaScript/TypeScript have them because they don’t separate objects and dicts (Map in ES6 isn’t as popular as dict in Python). Additionally, they have undefined, which is distinct from null. As a result, they need these operators to extract information from possibly unstructured objects.
However, Python has dict, so I usually store unstructured data in a dict (e.g. data from json.loads()), and then “deserialize” it into a structured dataclass with minimal use of Optional fields. In my opinion, using too many Optional fields in a class is an anti-pattern: every additional Optional field leads to the loss of 1 bit of information, which needs to be checked everywhere else, so it’s better to use them as few as necessary. I worry that introducing ?? and ?. would encourage people to add more and more Optional types to their APIs, but ?[...] is basically fine because dicts are naturally unstructured.
I’m lukewarm on the proposal as a whole, but this is a showstopper for me. Attribute access happens one at a time from left to right. If the proposal was for “none-aware attribute access happens one at a time from left to right with some extra logic at each step” that’s one thing[1], but if a single attribute access can short-cicruit the whole chain, that’s too big a change for my tastes.