Revisiting PEP 505

PEPs
1

All,

I would like to revive PEP 505, which proposes null-coalescing via ?., ??, and ??=. I believe this set of operators fills an ergonomic gap in the language where users are currently forced to choose between the footguns associated with or and the verbosity of if else expressions. I am hoping to solicit feedback on how I can get the processing moving again.

Some Thoughts on the Original PEP

  • In my opinion, PEP 505 does not sufficiently emphasize why None is special. While truthiness is a convenient measure of numbers and containers, the only semantically correct way to distinguish arbitrary values from None is by identity. This entails a verbose and oft-repeated boilerplate foo if foo is not None else bar when inferring defaults, operating on optional variables, and accessing optional attributes. Users must not reach for or in these scenarios because 0, "", and sometimes even (), [], and {} have intrinsic meanings that would be clobbered by their apparent falsiness (this is an aforementioned footgun).
  • In the next iteration of this PEP, I would weakly prefer to omit the ?[] operator. In my experience, its use case is much rarer than that of the other operators and does not merit the work to specify or teach it. I could be convinced to omit ??= as well, although I find this one to be much less invasive as it follows the well-trod path of the other binary operators.

Initial Address of Counterarguments

Skimming the related email thread, I found that the few cogent counterarguments fell into the following categories:

  1. It looks less Pythonic, it introduces line noise, it’s ugly: it is my strong opinion that the aesthetic changes introduced by this feature are no more invasive or alien than those of the walrus operator := from PEP 572 or the matrix multiplication operator @ from PEP 465. Of these three features, I would actually expect the null-coalescing operators to be the most easily-identified by the average user thanks to their prevalence in modern programming languages. I feel that complaints about the illegibility of a?.b?.c?.d are contrived and exaggerated, as I expect chaining two or more conditionals to be vanishingly rare. Moreover, I would like to suggest that these operators actually reduce line noise where if else expressions are necessary.
  2. These operators cannot be overridden with magic methods: the ability to override __bool__ (beyond the fact that 0, "", etc. are falsy) is why truthiness checks are not sufficient to distinguish arbitrary values from None. Consider x in the body of some def foo(x: Bar = None): ...: one cannot determine whether x was specified by testing for truthiness, e.g. if x:, because even if Bar does not override __bool__ itself, its subclasses might, and in doing so, may return False. Consequently, it is semantically incorrect to provide a default Bar via x = x or Bar(), whereas x ??= Bar() works as expected. The correctness of the latter statement hinges on null-coalescing operators being None-specific and un-overridable.
  3. Python is evolving too quickly: I am not involved enough with core development to know whether this is still the case, but it has been many years since PEP 505 was published and this discussion was held. I would appreciate feedback here.
  4. The grammatical implementation is flawed or has unintuitive behavior: I plan on digging into this once it’s clear that a re-proposal won’t be immediately shot down, as it will require more careful, academic work.

Going Forward

I am fairly new to the PEP ecosystem and its customs. Above all else, I am looking for guidance on how to best advocate for null coalescing operators. More concretely, I would like to use this thread to:

  • Get a read on the community’s current disposition towards ?., ??, and ??=
  • Work towards addressing any current counterarguments

If there is significant positive sentiment, I will begin working through the following incrementally:

  • Iterating on the grammatical specification of these operators
  • Updating the reference implementation
  • Redrafting the PEP

This PEP deals heavily with subjective notions like aesthetics, readability, and intuition. I implore thread participants to preface their opinions with phrases like “I think”, “I believe”, and “I feel that” to keep the discussion grounded in what is real and what is perceived. Demonstrating that a null-coalescing operator precedence introduces a bug in existing code is objective. Suggesting that the precedence is confusing is subjective and should be disclaimed because it should be justified and may be argued.

Amendment 1: Specification of Operators

As I see it, the new operators should serve purely as syntax sugar for the following expansions (ignoring precedence for now):

a?.b === a.b if a is not None else None
a ?? b === a if a is not None else b
a ??= b === if a is None: a = b

Please refer to this specification in the followup discussion, but note that it’s not final and that I’m happy to entertain alternative proposals (which I’ll continually append here).

Amendment 2: Example Usage

>>> a = None
>>> a?.foo  # `a` is None so attribute access is elided, returning None
>>> a ?? 0
0 
>>> a ??= 0  # `a` is now 0, equivalent to `a = a ?? 0`
>>> a ?? b  # `a` is not None, so evaluation of `b` is elided and the value of `a` is returned
0
>>> b ?? a  # `b` is not defined, so raise
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'b' is not defined
>>> a?.as_integer_ratio()
(0, 1)
>>> a?.foo  # `a is not None` so attribute access occurs normally
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute 'foo'
35 Likes
2

Having written a bunch of TypeScript in the last few months I agree. These operators are super handy when traversing JSON objects or other nested dicts and objects with optional fields.

We should probably decide whether a.b?.c returns None when a.b is None or only when b is missing from a, I vote for both, since None has no attributes.

Similar for ?? and ??=. Very handy. Though we need to decide what a ?? b does when a is not defined.

(And yeah, I know that TypeScript objects are different from Python objects, but I’ve missed this in Python too, and I’m sure I will miss it more when I go back to writing Python code.

33 Likes
3

This is something I’d love to see, and was in fact on my list of things to bring up! If you’re looking for a PEP sponsor or someone to collaborate with, I’d be happy to volunteer.

17 Likes
4

Just noting a couple recent discussions (the first has a number of links to other discussions near the top of the thread):

For myself, I’m fairly neutral on it. I am quite sure I would use it, if added, but it’s unclear how much and if the other impacts on the language would be justified by the utility, so I tend to stay out of these discussions.

4 Likes
5

Thanks, I wouldn’t have found these because I was unfamiliar with the phrase “safe navigation”. I will include anything relevant in an edit to my original post.

1 Like
6

I will strongly love to have ?. and ??!

I’m not a strong fan of ??=. I don’t recall I ever wrote data = data if data is not None else whatever, but maybe it’s only me.

A question: why the binary operator ?? and the assignment operator ??= need two question marks?

7

A question: why the binary operator ?? and the assignment operator ??= need two question marks?

I’m guessing the original PEP authors proposed ?? for parity with languages like Swift, C#, TypeScript, etc. (this is at least why I feel it should be this way). I assume those languages like use two question marks a ?? b to avoid grammatical ambiguity with the C-style ternary operator a ? b : c.

2 Likes
8

Sorry for the double posting, I forgot the BDFL post.

Both for me too.

IMHO if a ?? b will coalesce to b even if a is undefined can potentially mask a bug in your implementation. Furthermore I see very little use for it (again, maybe it’s only me)

3 Likes
9

I agree with your position on a ?? b. a ?? b should raise as a alone would if a is not defined. However, contrary to your and Guido’s idea, I feel that ?. should be symmetrical with this notion, and that a?.b should raise the same error as the access a.b if b is not a member of a (assuming a is defined and not None). I will add some examples in the original post to illustrate my proposed behavior.

11 Likes
10

Whenever I start reading PEP 505, I stop at:

a, b = None, None
def c(): return None
def ex(): raise Exception()

(a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
(a * b ?? c // d) == a * (b ?? c) // d
(a ?? True and b ?? False) == (a ?? True) and (b ?? False)
(c() ?? c() ?? True) == True
(True ?? ex()) == True
(c ?? ex)() == c()

Going further and giving it another chance, I encounter:
await a?.b(c).d?[e]
That’s when I give up.

3 Likes
11

These examples are intended to demonstrate the space of possible precedence interactions as concisely as possible. They are not representative of code people will write using these features, especially the last example (which I explicitly point out in my original post). Similar examples were required to fully specify await syntax in PEP 492 and the walrus operator := in PEP 572.

Going forward, please voice your complaints as explicit, constructive criticism of the proposal (ideally including justification for your position). Carefully consider whether your argument applies singularly to ??, ??=, and ?., or whether it could be just as easily contrived to argue against other operators.

11 Likes
12

I agree that all of these are terrible examples. But they are intended only to demonstrate a rather obscure (but important) detail of the semantics.

IMO the PEP would be vastly improved by focusing on real-world examples of how the feature would provide improved clarity in code, with the technical details like this clearly separated into an “implementation details” section.

I’m personally neutral on this PEP. I’ve never really been in a situation where the benefits have been obvious to me, or at least I haven’t noticed if I have, so a motivation section explaining why this is worthwhile to the unconvinced reader would be very useful.

15 Likes
13

Hm… It’s subtle. I checked JavaScript, and it acts if the Left Hand Side (LHS) is undefined, but it raises if the LHS doesn’t exist. So if you do a ?? b or a?.f and a doesn’t exist at all, it will raise, but if a exists with the value undefined, the operation will do its special thing.

However, the get-attr operation in JavaScript returns undefined, so that if we have an a that exists but doesn’t have an attribute f, a.f will return undefined, and you can write a.f?.g and it will return undefined. Only for top-level variables is there a difference between “does not exist” and “is undefined”. Also, function arguments that weren’t passed have the value undefined, and that’s another big use case.

Python works different – if you have a JSON dict a that may or may not have a key k, a["k"] raises, and if we want to have a["k"]?.f to return None, it looks like the? operator would have to catch the KeyError from a["k"]. If we don’t catch that exception, ?. would be much less useful than it is in JavaScript. The same reasoning applies to a.f?.g – we’d have to catch the AttributeError.

So you’ve half-convinced me – a NameError must pass through, but a KeyError or AttributeError on the previous operation (if it is respectively a[k] or a.f) must be suppressed. And this points to a key contention – there are a lot of people in the Python community (including core devs) who abhor operations that silence exceptions without an except clause (see e.g. the recent debate about finally: return suppressing errors “unexpectedly”). To be clear, I am not in that camp, and I believe that in this case practicality beats purity.

8 Likes
14

Maybe it’s just me, but I don’t like deciphering symbols. For example, I would be thankful if you gave me a full paragraph to read rather than just some emojis.

b = a ?? 2 ** b ?? 3 is read b equals a if a is not None else 2 to the power of b if b is not None else 2 to the power of 3:
b = a if a is not None else 2 ** b if b is not None else 2 ** 3

In the case of await a?.b(c).d?[e], it should not be allowed. There’s no way to make it foolproof without reverting to if/else statements. I didn’t read anything about this being disallowed in the PEP. Can we disallow it?

3 Likes
15

Here’s a use case for ??=.

Suppose we have a function whose argument is either a list or None, and if None, we replace it with []. This is a pretty common API pattern (needed to avoid the def f(x = []): return x problem). The current code looks like this:

def f(x=None):
    if x is None:
        x = []
    x.append(1)
    return x

I think it would be clearer if we could write it like this:

def f(x=None):
    x ??= []
    x.append(1)
    return x

Use cases for x ?? y are even more abundant – every time you write x or y you have to check that this returns the right thing (or you don’t care) if x is one of the many falsy values; x ?? y would remove your worries because it only triggers on None. (JavaScript has falsy values too, and it has the same problem when you write x || y, making x ?? y a better choice.)

9 Likes
16

Ah, OK. While I see what you’re saying, personally I’m ambivalent. The ??= version might be clearer once you’re used to it, but it’s something new to learn and get used to, and I don’t find the verbosity of the explicit is None check to be offputting[1]. I can imagine the 1-line version being better for a long sequence of checks, but I’d be worried about a function with that many optional parameters.

Similarly, I never use x or y as a None-check on x. But in that case the explicit version is annoyingly verbose, so I tend to just find other ways to express what I want. I guess that means I’m warping my code to cater for the lack of a ?? operator…

I guess I can see the benefits of these two.

I will say that I’ve never seen an example of ?. or ?[] which I didn’t find confusing, though. The questions about whether a non-existent attribute is trapped as well as an attribute with a value None leave me struggling to get an intuition for what the code does. And worse, the “right thing to do” feels like it depends on your data, so it’s not even a case where there’s an obvious best choice.

:person_shrugging: I’m not against the proposal, I’m just not sure how useful I’d find it. But there’s plenty of other things I’ve barely used (match statements, :=) so I can live with that.

  1. am I really quoting “explicit is better than implicit” to Guido? :blush: ↩︎

12 Likes
17

I see where you’re coming from with regard to object and dictionary traversal, I just don’t think I can bring myself to support suppressing exceptions. I worry doing so will introduce as many footguns as we’ve solved. Consider a contrived example:

class User:
    is_banned: bool

class Session:
    user: User | None

If we want to prevent a user from taking some action while they’re banned, we might test session.user?.is_banned ?? False. If we write a typo, e.g. session.use?.is_banned ?? False, this code will silently do the wrong thing for banned users. I suppose this is something linters like MyPy could address, though.

More broadly, I find it too incongruous for a binary operator to participate in exception handling. In JavaScript, ?. feels intuitive (to me) because undefined is a value, but in Python we’d have to introduce a new mechanism to intercept the exact AttributeError or KeyError. Is there any precedent for this? I can’t think of any examples. Also, I worry this suggests an implicit grouping of (a["k"]?).f rather than (a["k"])?.f.

Would you accept a.get("k")?.f? :woozy_face:

Addendum: a ?. operator that suppresses exceptions would be made more complicated by the fact that attribute and item accesses may raise completely unrelated AttributeError’s or KeyError’s that should still be surfaced. For example, even with exception suppression, I would want the following code to raise instead of returning None:

class Node:
    @property
    def parent(self) -> Node | None:
        return self._parent  # I forgot to implement this

>>> node = Node()
>>> grandparent = node.parent?.parent  # None because the `AttributeError` from the bug is suppressed by the `?.` on `node.parent`.

Another addendum: using foo.bar?.qux to simultaneously check if foo.bar is defined never occurred to me before this discussion, maybe because I come from a pretty well-typed world? This is something I haven’t ever consciously reached for, although I could see it being useful on unions of types.

12 Likes
18

As discussed in the previous thread, you would need to use a try/except block.

1 Like
19

Chalk it up to personal preference. I much prefer a ?? 2 ** b ?? 3, although personally I would probably write it as a ?? (2 ** b) ?? 3 so the precedence is clear at a glance, just as I would with e.g. a * (2 ** b) * 3 :smile: (future edit: this is incorrect, see note below)

Also, I wouldn’t worry about people writing await a?.b(c).d?[e] because a awaiting a coalesced None always raises. Since ?? precedence would be below await, you could only rectify this by writing paren around the entire thing, i.e. await (a?.b(c).d?[e] ?? f).

Addendum: whoops, I’ve gotten the precedence details wrong: a ?? 2 ** b ?? 3 actually evaluates to a ?? (2 ** (b ?? 3)). I’ve accidentally revealed my own a priori disagreement with the proposed precedence :joy:. In any event, I believe discussions about precedence and other implementation details should be sequenced wholly after it’s decided what shape null-coalescing operators will take in Python and whether they should be included at all, so I plan to set this aside for now.

4 Likes
20

That wouldn’t work if b is None, as it would raise a TypeError. The precedence is a ?? (2 ** (b ?? 3)). Also, 2 ** b cannot be None.

2 Likes