Building on the ideas from PEP 505 and after many discussions over the years, I’m happy to present PEP 823: None-aware access operators. Please be sure to also check out the discussion for the companion PEP 824.
Abstract
This PEP proposes adding two new operators.
The “None-aware attribute access” operator ?.
The “None-aware indexing” operator ?[ ]
The general idea is to provide access operators which can traverse None values without raising exceptions.
Both operators evaluate the left-hand side, check if it is not None and only then evaluate the full expression. They are roughly equivalent to:
# a.b?.c.d
_t.c.d if ((_t := a.b) is not None) else None
# a.b?[c].d
_t[c].d if ((_t := a.b) is not None) else None
Just to clarify, the splitting of PEP 505 into these two PEPs is so that they can be considered independently? As in, we could accept one but not the other? (Sorry for the duplicate post, but I wanted this to be apparent in both PEPs for other readers.)
I like this PEP. I do get tired of writing so many ugly null checks.
I read the Specification section in the PEP carefully, and at first many of the examples confused me, especially the complex combinations of operators. After reading it a few more times, the reasoning behind the base/tail design suddenly clicked. Then the way various cases are handled felt entirely reasonable.
Thank you for the PEPs! These do look like the best version of the proposal. Best of luck with the PEP process!
(I’m -0; I don’t run into needing this often enough.)
The PEP mentions the projects glom, jmespath, nonesafe, pymaybe. Did contact their authors/maintainers for feedback or endorsement? It’s good to pull someone in the discussion if you mention them.
The No-Value Protocol rejected idea should probably mention PEP 661 sentinels – an official generalization of None that doesn’t have math.nan’s issues. (I still prefer the crisp “None only” rule, though: sentinels are only meant for cases where None isn’t enough; writing out None if x is MISSING else x.foo is nice and explicit.)
Note that IPython uses ? (and ??) for accessing help, and it’s very popular/useful there. CPython’s REPL has been getting non-Python syntax (like help without parentheses), and we might want ? for that purpose too.
IPython doesn’t allow ? with arbitrary expressions, but I’ve sometimes wanted to be able to use it that way. So, to me, this expression from the PEP’s Motivation reads as “I want to get help on data.customer”:
data.customer?.user.name.lower()
I don’t have a concrete proposal in this area, but PEP 823 does limit future options.
I’ll bring up an alternative to this PEP that maybe no-one will like but that I personally would find more useful:
Let x? mean x is not None. The is not None is the part that all of these scenarios always have in common and is the part that gets really annoying to type.
The examples above would become:
# a.b?.c.d
ab.c.d if (ab := a.b)? else None
# a.b?[c].d
ab[c].d if (ab := a.b)? else None
And from the other PEP:
# a ?? b
a if a? else b
But more importantly, I would be able to replace all these statements which litter my code:
if (x := d.get("x")) is not None:
# ...
with
if (x := d.get("x"))?:
# ...
Also asserts:
assert var?
All this said though, I’m not exactly against the PEP. It’s just that I don’t expect it to be that useful to me.
EDIT: hmm I see that this would directly conflict with ipython’s help syntax mentioned above. still, ipython could switch to a different syntax (maybe prefixing the ??)
I probably won’t use these all that much (824 is more likely to see use for me). I tend to structure data in a way to not have nested optional data, and frequently use msgspec to ensure by the time I’m handling data from external sources, that it is in the expected shape and type.
The PEP being limited to None, rather than generic to any special sentinels or also handling attribute errors is a positive in my view, and means in the few places where I do still have optional nesting, I’d probably be willing to use this.
I like these PEPs! I have a minor bikeshed that affects readability (and understanding of people outside the PEP process): the term “optional” is a bit confusing (it always was when it was used in the type system for the same purpose). “Nullable” is much more generally clear (even if Python calls the null value “None”). Just 2 cents, thanks for the work on both PEPs.
My 2 cents on this: As discussed in the previous thread, I personally prefer ?[ ] to ?. because objects should be structured data, which may have some Optional fields but not too many. I would prefer to see a single ?[ ] operator without ?..
If the left-hand side (the base) for ?. or ?[ ] evaluates to None, the remaining expression (the tail) is skipped and the result will be set to None instead. This comprises everything in the tail part, including the evaluation of function arguments or subscripts.
When the topic was brought up before, I felt strongly about both pushing people towards structural pattern matching and having as few optional fields as possible. The objections section doesn’t really convince me otherwise.
making sure the attribute names are correct since the class pattern can also match missing once and as such will not emit an error if it is misspelled
I agree that this is a footgun. Since we treat non-defined attributes as type checking errors everywhere else, I think we should consider having a linter rule for this.
anecdotal evidence suggests that developers still prefer other alternatives for optional attributes, at least for simple, straight-forward expression
This isn’t really surprising, since structural pattern matching is not a common language feature in imperative languages, at least not for matching complicated data structures.
I don’t think that means it’s bad for this use case though, and I’d rather encourage more use of it than providing an alternative. For the simple cases that are left, normal if statements are good enough to me.
However, just because None represents an anti-pattern for some, has not prevented the community as a whole from using it extensively.
I’m mostly concerned about it becoming worse, since I believe the shorter syntax will lead to people preferring returning None over using exceptions, changing the semantics for convenience.
Thank you for putting the PEP together. I am very glad to see this – both ?. and ?[ ] fill a gap that I need coming from TypeScript.
A couple notes about the PEP:
I am glad to see “Why not use .get?” in the Rejected ideas. This section is quite thorough!
The later “Just use …” section is very compelling too!
I think the example in Parsing structured data could use a comment about why we need data.get; it is a few lines down from the comment about NotRequired and the parallel was not immediately obvious to me.
I think Terminology could use code examples for dictionaries too; both examples there are classes.
I am strongly supportive of this PEP. I think you have done a great job of capturing + distilling years (!!) of discussion into a tight, concrete proposal, and I hope to see this accepted.
Big fan of both this and PEP 824, thank you for putting them both together!
The example in the “Nested objects with optional attributes” section is similar to where I’d use these the most. No amount of parsing the data first will change that it’s perfectly ok for a customer not to be logged in (customer.user is None). That’s not an error at all
I’d even go a step further and have the data model allow for the User.name attribute to be optional as well. Similarly as before, it’s not an error for someone to have made an account but not filled in any of the bio information after creating it[1]
I’ve written a lot of code dealing with data in entity-relationship models with plenty of optional attributes and relationships. While the status quo works, it ends up being a ton of extra of line noise and I’d much rather just be able to write name = data.customer?.user?.name?.lower()
I personally don’t really like this proposal - the semantics, while clearly defined, aren’t “obvious” to me, and I can’t think of any times when I would have found these operators useful.
I suspect the key difference is that I’ve never used TypeScript. The languages we’re familiar with shape how we think about problems, and I suspect that people used to TypeScript tend to design their solutions in a way that falls naturally out of the availability of ?. style operators. On the other hand, those of us who are more familiar with languages without those constructs design solutions that don’t need them.
Whether it’s worth adding operators to the language that help people familiar with other languages to express solutions that feel natural to them, rather than learn how to express a solution in idiomatic Python, I don’t know. That’s probably one for the SC. All I can say is that I personally probably won’t use this much if at all.
I like this proposal and think it would improve a bunch of code I’ve personally written! Thank you for working on it. (I do slightly sympathize with the objection that adding syntactic sugar for None handling could proliferate a style where Nones are freely littered into every variable which is typically bad, but I think that’s unlikely to become a substantially worse problem than it is now, and this PEP is pragmatically a good choice.)
I do have a couple of feedback points on the PEP document: mainly, expanding on @dmoisset’s point, I think it’s a bad idea to use the term optional for a name that can be None. This confusingly overloads the term that is already used for a very basic Python concept, i.e. optional parameters of a function. The same conflation happened when static typing was first introduced and used the typing.Optional[X] form for a union of X and None, but I think many in the typing community have concluded that the naming choice is problematic because despite the concepts often overlapping in usage they’re quite different.
The docs for typing.Optional now have to have a clarifier explaining that one can apply while the other doesn’t, and typing.python.org’s best practices guide recommends avoiding it since the | union syntax was added. I personally have encountered skilled Python devs who were tripped up by this misunderstanding. Please, let’s not perpetuate the confusion in permanent PEP doc again! One can always use an alternate description, like “nullable” in line with other languages or “Noneable / None-able” as the Python flavor.
Other, minor comment on the text: in the match statement section, I haven’t been able to figure out what “making sure the attribute names are correct since the class pattern can also match missing once and as such will not emit an error if it is misspelled” means.
For normal attribute access if you misspell the attribute name, you’ll get an AttributeError. For class patterns, it will just not match the expected case. If you don’t immediately see the spelling error, it can be quite a chore to find out why the match case isn’t working.
What’s so special about indexing? It’s just one of many binary operators in Python. If we add ?[ ], should we not then also add ?+, ?@, ?& and many more? I’m not saying I like them, but they make equally as much sense to me.