Introducing a Safe Navigation Operator in Python

Ideas
61

This post summarizes my thoughts on this operator.

As a Node developer, I really see the usefulness of this operator in my day-to-day work because of JavaScript’s particularities, but in the context of Python I have mixed feelings.

Python’s data model is stricter and the built-in attribute lookup and collection traversal make no assumptions. An attribute/item exists or not and any exceptions must be explicitly handled or suppressed with one of the safe “getters”, such as getattr() and dict.get() that you mentioned.

A separate comment I would like to make is that when I see repeated or long/deep optional chaining expressions in the middle of business logic, that portion of code is a potential block to be refactored and/or the data structures may need to be revised, or perhaps that it is worth including a parsing/validation/transformation step to isolate or reduce the need for later checks.
It’s too easy/tempting to keep adding ?. and performing nullish coalescing to save time, until these things start piling up and you notice that half of an object structure is being tested attribute by attribute. Of course, there are scenarios where there is no escape, but indiscriminate use can make code a burden to read and interpret.

62

The argument “we should avoid this feature because it can be misused” is a dangerous one. We already have MANY features in the language which can be misused to create horrific code. I’m more interested in how something can be used well than in how it can be used badly.

8 Likes
New syntax for safe attribute and safe subscript access
63

Very much +1 to that for this feature. Particularly since I don’t see how the equivalent code without the new operators could do anything functionally different – it would be the same but more verbose (and probably confusing).

I also don’t really buy into the “people will stop checking for None” narrative. If type checking has taught us anything, it’s that people aren’t nearly None-aware enough already. So an equally convincing narrative, to me, is that the new operators might actually make people more None-aware in their code, because it’s less of a bother to check. I don’t know that there’s data to back either story.

4 Likes
64

There’s plenty of examples in JavaScript which suggests a carelessness with regards to undefined; it’s very prevalent. I’m not sure the optional attributes is the cause or a symptom, but there’s a different mindset when developing JS.

I find None quite rare outside of mutable parameter defaults. None is almost always another invalid value and type (like passing an int when str is required). In JS, undefined is acceptable by default

2 Likes
65

Indeed, which why I am not terribly convinced by the argument that people might start throwing None around too carelessly if it becomes too easy to deal with None values. But perhaps I am biased here by the fact that I almost always write library code, so that there is an API barrier between me consuming maybe-None values and someone producing maybe-None values.

66

I somewhat agree with the sentiment that most of the perceived need for this is due to the lacking nature of other things. I think the (also deferred) lazy defaults may provide higher value towards solving real pain points and it might be worth working on that first to then see if this still feels needed, and then in what contexts it does.

Lazy defaults also clears up some annoyances with typing and sentinel values that aren’t meant to be passed by users in a better way than adding explicit support for sentinels to the type system, something else also under current consideration.

1 Like
67

None shows up all the time in data processing tasks. It greatly simplifies working with optional/nullable data from external systems, and/or nested data structures that can contain optional elements.

This is not a speculative use case. I encounter it in almost every application I write, and I’m sure there are many other programmers who experience the same or similar. If there weren’t, there wouldn’t be such a persistent demand across mailing lists, forums, and chatrooms for something like PEP 505.

Consider that Glom exists in no small part because PEP 505 does not exist.

Here’s some example code I wrote literally today.

I started here:

radius = user_geodata.get('radiusValue', user_geodata.get('RadiusValue', None))

Then I refactored it to use Glom, which I think reduces some visual clutter in this case:

radius_spec = Coalesce('radiusValue', 'RadiusValue', default=None)
radius = glom(user_geodata, radius_spec)

And now imagine if I could do this in the standard library:

radius = user_geodata.get('radiusValue')?.get('RadiusValue')

Maybe you think this ?. example is ugly, or that the behavior should remain in a 3rd-party library and out of the core language. Maybe you like the first version the best!

But at least let’s acknowledge that the need for handling None values is both ubiquitous and verbose in Python as it exists today. I don’t think it serves anyone to dismiss PEP 505 as addressing a “quite rare” use case.

(Side note) Another benefit of Glom (unrelated to PEP 505) is the ability to declaratively specify relatively sophisticated data processing pipelines and invoke them in a uniform manner (not unlike optics in functional programming). This particular example is a little contrived, but alongside 3 or 5 other cases of accessing deeply-nested attributes, the declarative style and uniform application really starts to feel good.

2 Likes
68

How about:

radius = safe(user_geodata.radiusValue).RadiusValue

where

class SafeAccessor:
    def __getattr__(self, x: Any) -> None:
        return None

def safe[T](x: T) -> T | SafeAccessor:
    if x is None:
        return SafeAccessor()
    return x

This is pretty close the ?. syntax y ou want and probably simpler than the Coalesce/glom solution?

Of if you wanted getitem instead of getattr, it’s pretty similar—I wasn’t sure if you wanted ?. or ?[.

2 Likes
69

That’s an interesting idea for sure. My point was less to emphasize any particular solution (PEP 505 or otherwise) as it was to emphasize that lots of people need to handle None all the time.

2 Likes
70

You’re right, I’d forgotten about dict.get, even though it’s something I’ve used at least thousands of times. For me, however, the None informs control flow: I’ll use if-statements to protect attribute access. I think this is what more readable due to the indentation, and better describes intent.

71

And now imagine if I could do this in the standard library:

radius = user_geodata.get('radiusValue')?.get('RadiusValue')

That doesn’t mean the same as your original (assuming you use the PEP-505 ?. operator), that’s doing a nested lookup. You may want to use ?? instead, which would look like this:

radius = user_geodata.get('radiusValue', user_geodata.get('RadiusValue'))  # before
radius = user_geodata.get('radiusValue') ?? user_geodata.get('RadiusValue')  # after

This is a nice example on how these operators are less intuitive than they seem. And also you can see that the final version is not significantly different than the one you started with (which by the way can be simplified removing the None argument to dict.get().

glom, roam and jmespath do a decent job at covering usual cases of this. What I miss most from PEP505 is not the navigation operators, but the coalescing one: that one is much harder to turn into a library, and if you look at the “survey” in PEP505 it’s usually the one that makes a bigger difference.

3 Likes
72

obj.attribute being None does not generally mean that you should not do something with the obj.attribute. That would have syntax more like obj?.attribute? by your thinking as I understand it. It is also quite common to do something to set obj if it is None before accessing attributes. That happens quite often in my own code. Since ‘?’ is not legal in python identifiers the proposal would cause rather substantial changes. It would also introduce some identifiers doing something else than being a label for some object. So it this is not a simple change.

1 Like
73

Maybe start with a standard library function and see if it gets traction?

def elvis(f):
  'elvis(lambda: foo.bar["baz"].quux)'
  try: return f()
  except (AttributeError, IndexError): return None
2 Likes
75

Can confirm what @kfdf mentioned.

As far as I’m aware, lambda: with try / except is the only decent polyfill I can think of that works in all situations (and doesn’t crap out if there’s a string in one of the child elements).

Been using a similar implementation for quite some time now to implement safe navigation operators in Python.

I really do wish we get this in core so I can stop writing foo(lambda: ...) everywhere.

2 Likes
76

Not excited about python reading more like regex.

and/or/getattr compositions do the trick usually. Maybe not enough people use and/or for their non-boolean behaviors and think python is deficient?

x and x.attr is sufficiently terse IMO; no need for x?.attr. Or how about a custom False-ish object which returns itself for absent attributes (if you care that much about not repeating an identifier): (x or Nullify).attr

Upgrade that to x and getattr(x, "attr", None) to protect against an AttributeError.

1 Like
77

That’s always the way with trivial examples, but it absolutely isn’t true once you start chaining them. Consider that you have a mapping of random miscellaneous information, which might contain in it a “session” if the current request contained a session cookie. The session information might contain a user dictionary, if someone’s logged in. How do you get the currently logged-in user’s display name, or “Anonymous” if nobody’s logged in?

(misc and misc.session and misc.session.user and misc.session.user.display_name) or "Anonymous"

Or:

misc.?session.?user.?display_name or "Anonymous"

Note that PEP 505 semantics might let you shortcut this even further in many common situations, but I deliberately worded this such that each and every level could be omitted independently.

This actually isn’t an artificial example. I lifted it directly from my Twitch channel bot, which is written in a language that DOES have this sort of “optional indexing” operator. The “misc” mapping is itself an attribute of the HTTP request object, so the sequence actually looks like req->misc->?session->?user->?display_name || "Anonymous" but the effect is the same. And this sort of thing happens all the time.

In Python, I would probably end up writing a helper function, but it’s still clunky.

Responding to your edit (and by the way, please don’t edit posts to make substantive changes - not everyone will see the edit - a followup is way better when you’re actually adding another real option): (x or Nullify) would somewhat work, but now you need a magic object that returns itself for every attribute, and yet collapses into being good ol’ None when you want it to. Otherwise, you end up stuck with this dangerous Nullify object at the end, when you thought you were done with it. The code would end up looking like (x or Nullify).attr.attr.attr.attr or None or something like that. This suggestion really feels like a “why don’t you just…” sentence, not well thought out and definitely not practical.

The helper function idea HAS been suggested a number of times, and I consider it the least bad option, but an actual None-aware operator set would be of material value.

8 Likes
78

Thanks for the edit suggestion (I was treating it like force --with-lease and didn’t consider that someone might be in the middle of a reply).

Something like Nullify gets close to the usecase here, though? We’re asking for a mechanism which let’s None act like NaN in the sense that we want it to propagate through getattrs.

I do think I’d end up using ?. if it was available. I remember being doubtful of the walrus operator but ended up using it all the time.

79

Not only that, but anyone who’s responding to the emails won’t ever see the edit. So it’s fine to edit for non-substantive reasons (fixing a typo, or especially fixing formatting), but I would avoid it for anything that might materially affect replies.

Kinda almost close, but since Nullify is not None, you end up with this residual object at the end. Instead of finishing up with either a value or None, you now finish up with either a value or Nullify, and that means that any attempt to work with it will silently do the wrong thing, which is the very worst kind of bug. This makes it an extremely dangerous idiom. At least with NaN, the only way it propagates is through arithmetic; if you attempt anything else with it, it’s obviously NaN.

80

Isn’t there the possibility to forget more easily the fact a variable is None and should not be None? In other words, couldn’t this allow to write bad code more easily?

81

The possibility is the same, but the failure mode is far safer. Any attempt to do anything with None, beyond simply printing it out and the like, will result in a quick exception. But a Nullify object can be used for any attribute, and will return itself. If it’s also able to be called and will return itself (a common semantic aspect and a useful one), ie that Nullify.any_attribute() is Nullify, then that means Nullify.__iter__() is Nullify, and Nullify.__next__() is Nullify, so it’s infinitely iterable and produces a stream of itself. Same goes for nearly every other protocol - they will succeed but give back more Nullify. Unless, at some point, you forcibly coalesce it to a real None value, this will cause PHP-level “just keep going, it’ll be fine” semantics, rather than actually telling you that there’s a problem.

PEP 505, and every implementation of “null-aware” operators in any language I’ve worked with, does not have this problem. At the end of the single expression that it’s in, you have a perfectly normal object with all its normal protections.

1 Like