Introducing a Safe Navigation Operator in Python

Ideas
41

(And I’d rather allow the default argument to actually be the list, ie def f(o=>[]): )

4 Likes
42

I would suggest providing complete examples.

In this particular case, I don’t understand the purpose of the variable client_category because it shouldn’t exist at all if the client doesn’t exist. It’s important to note that ‘null’ (or ‘None’ in Python) is a valid value. If client_category is null, it signifies that the client exists. I have no idea how the flow diagram will progress from this point.

43

This has got me thinking a lot. I was initially very pro ?. having used similar in other languages. But taking the example of Swift, one of the major reasons (I believe) it places such emphasis on nil-aware operators is for interoperability with existing Objective-C APIs where nil was pervasive.

It’s a lot less interesting of an operator without an existing ecosystem of APIs where null appears a lot (e.g. Java).

44

Summary (as I understood it):

  • A Python Core dev pointed out to him that adding feature to a language will affect how people use that language. Example:
    • Without the None coalescing operator (NCO), people check for None before appending a value to a list.
Example (as I understand it) 
mylist: list[object] = []

for ... in ...:
    element: object | None
    if element is not None:
        mylist.append(element)

# mylist does *not* contain None

for element in mylist:
    element.method()
  • With NCOs, people put None values into the list because they can handle the None values no problem using NCOs when they take the values out of the list.
Example (as I understand it) 
mylist: list[object | None] = []

for ... in ...:
    element: object | None
    mylist.append(element)

# mylist now contains None
# mylist = [object, None, None, object, ...]

for element in mylist:
    # using the NCO to handle None values
    element?.method()
  • In other words, the NCO encourages bad code quality by making it easier to handle None values when you take them out of a list, instead of when you put them in.

However, he is in favour of using the glom library to navigate safely in Python objects. Also, he mentioned using Structural Pattern Matching to achieve similar behavior.

Here is an example I created that uses a match statement: 
import dataclasses


@dataclasses.dataclass
class BankDetails:
    iban: str


@dataclasses.dataclass
class User:
    name: str
    bank_details: BankDetails | None


test_users: list[User | None] = [
    None,
    User("Name", None),
    User("Name", BankDetails("IBAN")),

    # These have incorrect types, therefore should raise an AttributeError, but don't in this example.
    False,
    "Hello",
    User("Name", "Hello")
]

for user in test_users:
    # iban = user?.bank_details?.iban
    match user:
        case User(name, bank_details=BankDetails() as bank_details):
            iban = bank_details.iban
        case _:
            iban = None

    print(iban)

Output:

None
None
IBAN
None
None
None

I think, Structural Pattern Matching can be an alternative to NCOs, but they do seem very verbose and not concise in comparison, which is the whole point of NCOs.

7 Likes
45

Not exactly. Protecting from a missing attribute/key already exists and is spelled getattr(obj, "attribute", None) or dictionary.get(key, None).

Broadly, in Ruby/JS/etc. such operators protect against the object/collection itself being None, e.g. same as the expression obj.attribute if obj is not None else None, or shorter but looser obj and obj.attribute (hence Ruby’s spelling obj&.attribute).

Moreover, x?.y should still fail if x is not None but lacks .y attribute!

Such expressions get much clumsier to build once you chain things e.g. writing x?.y?.z would become something like
None if x is None else None if x.y is None else x.y.z
or equivalently
x.y.z if x is not None and x.y is not None else None
or approximately
x and x.y and x.y.z
except it should only evaluate x.y once. That last quality is hard to achieve in an expression!

However it’s pretty easy to achieve with new helper functions e.g.

def andgetattr(maybe_obj, attr):
    if maybe_obj is None:
        return None
    return getattr(maybe_obj, attr)

andgetattr(andgetattr(x, 'y'), 'z')

Not pretty, especially after you define andgetitem, andcall helpers for collections & methods… Compare foo?.bar?[baz]?.quux?(arg1, arg2) with andcall(andgetattr(andgetitem(andgetattr(foo, 'bar'), baz), 'quux'), arg1, arg2).

Hmm that really begs for left-to-right version, something like dig(foo, 'bar', [baz], 'quux', (arg1, arg2,)) — using strings to denote x.attr, lists for x[indexing], tuples for x(calling)? :thinking:

I’m not convinced such operator would be great fit for Python. But that’s^ the clumsiness it tries to improve upon.

It tends to exist in languages where common APIs/operations return such values a lot. JS obj.no_such_attr aka obj['no_such_attr'] fails by returning undefined; Ruby hash[no_such_key] fails by returning nil…
Whereas Python raises exceptions — unless you specify explicit defaults to getattr() / dict.get().
For dictionaries, this pattern works great: (d or {}).get('e', {}).get('f', {})
I’d look for things that extend those…

9 Likes
46

Great clip and succinct summary. My history in Perl, and experience as a language teacher, shies me away from punctuation heavy language features.

2 Likes
47

As someone who teaches Python a lot, I dislike the idea of more syntax as a solution to this problem. Something like this a ??= b?[c]?.f() is just impenetrable to a newcomer. I have always loved that Python can be read by beginners and advanced alike, and more punctuation goes against this.

There is some (old) discussion around this at https://www.reddit.com/r/Python/comments/3lkaxc/pep_505_none_coalescing_operators/, which is mostly very against the idea.

I do understand the requirement when traversing non-rectangular data structures. I’ve also had times where it’s simpler in dealing with SQL engines that return NULL (as None). For the simple case, I just use a simple safe function, and for the more complex case, such as JSON, I either use a different parser, (or pandas which works well with this), or I’ve written a SafeNone class and associated safe function (which supports attribute access, slicing, etc). A SafeNone class solves some of the verboseness of the use of andgetattr function above, but retains the clear use of functions, and doesn’t add any new syntax to the language.
I’ve also seen a nice decorator on the Reddit link that could also work.

As @steve.dower well points out, there’s also the bigger question as to why there are None objects that need to be accessed - sometimes I’ve found the best solution is to stop the None’s getting into the list (or whatever structure). And that may encourage better coding practices.(I like what @rhettinger had to say about this.) It’s easier to get the length of a list that doesn’t have None’s, than to find out how many non-None objects a list has. Also a dict’s get method with a default is sometimes forgotten.

I like what wikipedia has to say about the Null object pattern (and by inference the null/None coalescing operator):
“This pattern should be used carefully as it can make errors/bugs appear as normal program execution. Care should be taken not to implement this pattern just to avoid null checks and make code more readable, since the harder-to-read code may just move to another place and be less standard—such as when different logic must execute in case the object provided is indeed the null object.”

5 Likes
48

That’s an important distinction. I’ve used short-circuiting in JS and besides brevity it has real benefit of communicating intent :+1:. b?.c?.d?.e communicates that I fear all of b, b.c, b.c.d being null/undefined; b?.c.d?.e communicates that b and b.c.d are nullable, but when b exists I do expect b.c will always be non-null.

But it comes with a big price which IMHO would be unacceptable in Python. Syntactically, it’s no longer a single operator :-1:; it eats up all consecutive attribute accesses/item indexing/method calls to form an “optional chain”. Simplifying the JS spec a bit:

OptionalExpression :
    MemberExpression OptionalChain
    CallExpression OptionalChain
    OptionalExpression OptionalChain

OptionalChain :
    ?. Arguments
    ?. [ Expression ]
    ?. IdentifierName
    ?. TemplateLiteral
    ?. PrivateIdentifier
    OptionalChain Arguments
    OptionalChain [ Expression ]
    OptionalChain . IdentifierName
    OptionalChain TemplateLiteral
    OptionalChain . PrivateIdentifier

what this means in concrete terms is there is no way to interpret foo?.bar[baz].quux(arg) as a sequence of independent steps ((foo?.bar)[baz]).quux(arg) ≡

var1 = foo?.bar
var2 = var1[baz]
var2.quux(arg)

But Python stands out from other languages in syntactically treating even method calls as two separate steps :clap: (var2.quux)(arg) ≡

var3 = var2.quux
var3(arg)

(whereas most OOP languages parse obj.method(arg) as a single “method call” construct)

So I don’t think there is any technical barrier to adding syntactic short-circuitable chains, but conceptually IMHO it’d be very confusing.

3 Likes
49

With regards to “should this be implemented or not”, putting aside the implementation details and whatnot; my thinking is that in the hierarchy of values, we should put readability and “guessability” (?) in a higher place than syntax minimalism.

I would actually argue that, both to a programmer that is unfamiliar to Python and a complete programming novice, the obj?.attribute syntax both more succinctly and language-agnostically communicates its logical function than the long-winded None-check. When I read if obj is not None: print(obj.attr) I mentally translate it to a singular logical unit: safely print attribute. That is a nontrivial task of mental chunking that a Python newb has to learn to do. The introduction of this operator would resolve that issue. I also think it helps when quickly skimming through code as well as grepping for stuff.

2 Likes
50

Before we can seriously discuss any proposal, it would need to be clear on what sort of short circuiting behaviour it supports.

For example, suppose something writes this:

found = entity?.get_location() is not None

I’d like the people arguing for None-aware operators to clarify what they believe this should mean.

Here are three possible interpretations:

found = (None if entity is None else entity.get_location)() is not None #1

found = (None if entity is None else entity.get_location()) is not None #2

found = None if entity is None else (entity.get_location() is not None) #3

The intended interpretation is #2. #1 would call None if entity is None, and #3 would mix types. Do we just accept that plausible looking code like this is wrong, or is there a reasonable language rule that would make interpretation #2 the right one?

5 Likes
51

I believe the opposite! Some english words are better than a showering of punctuation signs.

13 Likes
52

I was about to make the same argument. Is there a way to satisfy both camps? Can we have a succinct way of writing some of the coalesce operators using English words? For example, replace

self.opacity = 0 if default_opacity is None else default_opacity

with

self.opacity = default_opacity coalesce with 0

Can the parser deal with a binary operator soft keyword?

1 Like
53

Now that just reads like SQL :slight_smile:

2 Likes
54

Agreed! Coming from Perl 5, and seeing the negative effect more punctuation had on Perl 6 (and may have led to its demise), I definitely come from “Readability counts” camp.

1 Like
55

I have wondered about a naked else for that case:

self.opacity = default_opacity if default_opacity is not None else 0
# self.opacity = default_opacity else 0

But I still think the operators are friendlier.

2 Likes
56

I personally like this, because it’s quite so close to both the ternary operator and default_opacity or 0 and has a similar goal. It’s basically a “safer” alternative to the or operator.

4 Likes
57

I’ve also thought about a naked else, but apart from any considerations on the effect on complexity of learning/reading the language, the relative fixity of x else y and x if y else z would need to be settled. Which of the following would a = b if c else d else e be equivalent to?

  • a = (b if c else d) else e
  • a = b if (c else d) else e
  • a = b if c else (d else e)

The unparenthesized version is unreadable garbage (and it’s unpleasant if not unreadable with parentheses), but I’m sure somebody would write it.

3 Likes
58

How about default_opacity otherwise 0?

59

And as for the other safe navigation operators, we could add an object to the standard library (not sure where) like:

from somewhere import safe_navigation as safe

value = ((x otherwise safe).attribute otherwise safe)[12]

where safe[anything] and safe.anything is None. That’s also not as compact as ?. and ?[], but balances convenience with readability?

60

Well, that made me realize that there’s yet a third parse. With otherwise, we could still have the following choice:

  • a = (b if c else d) otherwise e
  • a = b if c else (d otherwise e)