PEP 823: None-aware access operators

PEPs
42

Today, while working on a project of mine, I noticed this snippet in typeshed:

    # stdlib/subprocess.pyi

class CompletedProcess(Generic[_T]):
    # morally: _CMD
    args: Any
    returncode: int
    # These can both be None, but requiring checks for None would be tedious
    # and writing all the overloads would be horrific.
    stdout: _T
    stderr: _T

This seems like almost a perfect example of what I said earlier:

This also means even if you run typecheckers, you’ll only catch this when a runtime error happens.

With PEP 823/824, this kind of thing gets a lot less tedious and more correct:

error = result.stderr?.strip() ?? "command failed for unknown reasons"

And maybe in post PEP 823/4 world, typeshed may be updated to reflect reality (im not too aware of their backwards compatibility policy)

To be clear: I’m not saying typeshed did anything wrong, they made the pragmatic choice and one that’s likely less frustrating to users.

8 Likes
43

I don’t think this is a good example of the ?. operator. In theory, subprocess.run should have some overloads that return one of four different types, e.g. CompletedProcessNotCaptured, CompletedProcessWithStdout, CompletedProcessWithStderr, and CompletedProcessWithStdoutStderr. But for simplicity, they didn’t make this too complicated. When I explicitly pass capture_output=True, I know the result can’t be None, so I don’t want to use ?. to add noise.

3 Likes
44

For me a syntactic sugar that allows us writing:
target_population = continent?.largest_country?.capital?.population ?? 1.000.000

rather than:
target_population = (continent.largest_country.capital.population
if continent and continent.largest_country and continent.largest_country.capital and continent.largest_country.capital.population
else 1.000.000)

is pretty helpful.

5 Likes
45

I could not agree more with that sentence.

2 Likes
46

This is precisely why people like me are hesitant about this syntax. I have coworkers who would create data structures that work like this if it was this easy. But the code is much easier to grasp if these fields aren’t all nullable.[1]

Yes I know the ideal approach would be for people to be disciplined. But when management doesn’t see the value in discipline, and some coworkers don’t see the value in readable code, I do prefer working with a language (like python) where some degree of discipline is enforced by the language.

  1. I can see the appeal because in practice it is often the case that one of those 3 steps will have to be nullable for one reason or another. But still, 1 is much better than 3. ↩︎

20 Likes
47

Maybe this operator can exist as a function before it becomes syntax.

Something like:

target_population = maybe_getattr(continent, "largest_country.population")

Which behaves the same as:

if continent is not None:
    target_population = continent.largest_country.population
else:
    target_population = None

This function doesn’t seem that hard to implement and could live in the operator module before it becomes new syntax.

1 Like
48

Big +1 from me, it will make almost any API-related work simpler to read and write, and, in many cases (because of the more explicit None-checking Vs existing shorter lazier falsy-checking, or the given examples of just assuming non-nullness generally) more correct.

49

Not all continents have a country (Antarctica doesn’t), and not all countries have a capital (Switzerland doesn’t).

So I don’t see how you would fix the example to not have these nullable fields and be easier to understand.

3 Likes
50

Geographically true. Also Antarctic doesn’t have inhabitants (let alone 1,000,000), and Swiss cities are significantly smaller than the default 1,000,000.

Meaning, special cases (in geography) need special handling (in coding). Lumping together various missing attributes simplifies data beyond credibility.

4 Likes
51

I’d argue the opposite. Trivialness of the implementation aside, I think the pep does an adequate job of explaining the use case.

If anything I’d argue that PEP should add a function for this to operator.

import operator

def maybe_getattr(obj, attr):
    getter = operator.attrgetter(attr)
    try:
        attr = getter(obj)
    except AttributeError:
        return None
    return attr
1 Like
52

Isn’t that just getattr(obj, attr, None)?

It doesn’t look relevant to the PEP, since it doesn’t even need a check for ‘obj is None’.

And it isn’t a replacement for obj?.attr since the latter also suppresses chains like obj?.attr.another and more.

53

That’s a good point. I’m conflating a few things here. My function doesn’t support the initial check for the object being None. getattr does not support nested attribute access and operator.attrgetter supports nested access but lacks a default return for an attribute not being present.

54

+1
IMHO one of the most important PEP’s Python has ever had.

Python is all about duck typing / dictionary-based code…yet always lacked operators to make safe access ergonomic.

Before this PEP, safe access in Python has always come with the disadvantage of seriously mangling your access syntax. Example:

:white_check_mark: GOOD:

  • foo.bar safe access foo?.bar
  • foo.bar safe access foo.bar ?? null
  • :white_check_mark: Autocomplete on properties still work!
  • :white_check_mark: Just add a few characters to existing code.
  • foo?['bar'] wont autocomplete, but at least it won’t mangle your code (add 1 character!).

:cross_mark: BAD:

  • foo.bar safe access if 'bar' in foo:
  • foo.bar safe access foo.get('bar')
  • :cross_mark: Will never autocomplete!
  • :cross_mark: Mangles your access syntax. Requires restructuring code. :sob:

JS, PHP, Ruby etc has had this forever. (PHP has had non-mangling safe access in version 1.0 with @ !!! But now has all the standard operators too!)

Super happy about the PEP- natural feature for dynamic languages.

2 Likes
PEP 824: None-coalescing operators
55

I worry a little about code readability.

Having used the JS/TS equivalent and Rust equivalent, I find that sometimes it feels like sprinkling magic pixie dust over the code.

In JS case, it’s an acceptable cost… albeit there are some null/undefined/falsy quirks and IDE/language server/ESlint becomes almost essential for producing quality code: some automated tool is needed to remove redundant question marks, especially so when refactoring or changing library versions: some place else ceases to be nullable and now code here needs a fix.

In the cast of Rust, the mental overhead was just too much for me. Because of the type system, inference and other concise syntax, it just gets garbled in my head.

I wonder where on this spectrum Python may land.

For example, where would # ruff: ignore[...] go when the code works around imprecise types? That is the type system says you don’t need a question mark here, but at runtime, you actually do.

Today, I’m personally hesitant.

1 Like
56

I gave this more thought, and while I’m still generally positive on 824, I’m negative on 823.

The issues for me here is that I still think that this encourages APIs that are worse than they should be.

In my experience, it’s both rarely the case that I’ll have nested nullable data, and it’s practically always incorrect to short circuit when I do have it. In the very few cases I have it, I need to be able to handle the different cases for why, so short circuiting and getting None doesn’t accomplish that. That is to say, a?.b?.c is a mistake, because a being None is behaviorally significantly different to b being None. If these weren’t significantly different, then this shouldn’t be nested to begin with, and the data is modeled poorly.

So it seems to me that this further encourages API designs that I don’t think we should be encouraging, on the basis of “people get external data they don’t control the shape of”. Even if that’s the case, handling that yourself in one place or using a library that can structure it to a more reasonable shape for you (eg. cattrs), means that you do that once, and then the rest of your program no longer has to deal with that.

12 Likes
57

I feel you may be overreacting, or perhaps your use of Python is significantly more structured than many other Python users, or perhaps the types of data structures you have to deal with is different than the data structures that others encounter.

I’d also say that I expect a?.b?.c to be quite rare. We shouldn’t exclude it from the grammar, because it has some applications and its semantics are clear once you know about single ?., but I’d say that it’s pretty rare to encounter doubly-nested optionality.

To quantify “pretty rare”, I asked an AI to count various occurrences in one TypeScript project I co-own(ed), and it summarized the results as follows:

Across 1,238 tracked .ts files:

  • Standard property chains such as a.b or a.b.c.d: 61,796 uses.
  • Property chains containing optional chaining, such as a?.b: 1,602 uses. This is 2.59% as common as standard property chains, or roughly one optional chain for every 38.6 standard chains.
  • Optional property chains with two or more ?. occurrences in the same uninterrupted chain, such as a?.b?.c: 69 uses. This is 4.31% of optional chains and 0.11% of standard chains.

Comments and quoted string/template contents were excluded. Each maximal chain made exclusively of identifiers separated by . or ?. was counted once, so a.b.c.d counts as one use.

I have to admit that I don’t quite follow your reason for expecting this to encourage inferior API design. Can you show an example of how using cattrs helps structure the data to avoid None? Suppose a user may or may not have a subscripton. If user.subscription is None, what would you want change it to? What’s bad about having an optional subscription field (not possibly missing, but having a type of the form S | None). Or perhaps you can work through some other example where you consider nullability a bad design? (Or if you already gave such an example and I missed it, just link to it?)

7 Likes
58

If you only need the single, then isn’t ?? (824) reasonably sufficient?

It’s specifically the nested nullability where it becomes an issue. Given an earlier example of:

The answer here is that the short circuiting None handling is still not ideal here. In what way is a default of 1.00.000 useful? In what way is any default value useful?

If a user requested the population of the capital of the largest country on the continent, and the continent in question is antarctica, you should display an error that antarctica doesn’t have a largest country. That requires knowing that largest_country in that API is None.

(Now why that’s an attribute or property is another question)

None is certainly both useful and special, but it’s also something that should be explicitly handled where it is necessary to allow, because the reasons for it’s existence when looking at nested access aren’t typically the same for each level of access.

5 Likes
59

You mean you can write a?.b as a ?? a.b ? That still repeats a, which feels verbose, and if the thing before ?.is a bit longer, the repetition gets annoying:

account_name = user.account?.name
# vs.
account_name = user.account ?? user.account.name

Not to mention something like

account_name = get_account(user)?.name
# vs.
account_name = get_account(user) ?? get_account(user).name

Of course we should write the latter using a temp variable. But I would really much rather have an operation using the stack than having to explicitly create a temp variable – especially since, as it’s already a nuisance to invent, I’m not going to waste more code to GC that temp variable.

9 Likes
60

FWIW, I also worry about readability after this addition. Looks like Perl.

Second, looking on the TS statistics, this seems to be rather niche case:

Any chance that such patterns would be more common in Python projects? Probably something like pyastgrep could be used to find appropriate cases.

61

I don’t otherwise have a strong opinion on the topic, though I would like to propose to make ?. and ?[] not raise AttributeError / LookupError if the attribute or item is not present, rather they should return None.

The use of ? has made clear that the author intends to use None rather than exceptions as the absence mechanism. In situations such as incoming JSON where missing values aren’t marked by null (and thus the deserializer doesn’t produce None) but by omitting the key altogether, ?[] as proposed originally doesn’t help much as it still raises KeyError on missing keys.