Better exception localization

TLDR: One-line exception triage for better error handling with lambda-like syntax:

# initial proposed syntax; '^except' is a single token
t = father["smell"]
^except KeyError: return "elder berries"

In my 2 decades with Python across gaming, social media, and aerospace, a very common issue I’ve run into is exception-laziness caused most frequently by aesthetic aversions.

The “try first, ask later” use of exceptions tends to be unpopular because it produces so much boilerplate, reminding me of go’s

result, err := camelot()
if err != nil {
    runaway()
}

Back to Python, it is also a frequent, common cause of simple bad engineering where people are lazy about dealing with exceptions because they don’t want to write four lines of code - let someone else handle it.

Let’s take a most trivial example, index checking, to underline how this starts badly and scales worse:

# real-world preferred
if (h := data.get("hello")) and (w := data.get("world")):

# properly pythonically, while retaining ability to
# deal with which key was affected most reasonably
try:
  h := data["hello"]
except KeyError:
  ...
try:
  w := data["world"]
except KeyError:
  ...

The second aversion is nesting and displacement:

try:
    with open("coconuts.pickle", "rb") as jar:
        try:
            header = pickle.load(jar)
        except Pickle.UnpickleError as e:
            ...
        ... other operations allowing exceptions to fall to the top
except FileNotFoundError as e:
    ...
except PermissionError as e:
    ...
except ValueError as e:
    ... 
    hang on mr elipsis writer - what code is THIS responding to?

People are frequently averse to localizing exception handling because it’s just plain annoying to try and say “oh if I get a key error here”.

the above example would become:

with open("coconuts.pickle", "rb") as jar:
    ^except FileNotFoundError: return "Brave Sir Robin, run away"
    ^except PermissionError:   return "Go away you silly, English, kniggit"

    header = pickle.load(jar)
    ^except Pickle.UnpickleError as e: return "YOUR father smells of herring"

    try:  # the others are validly a single block with common handlers
        ...
    except ValueError as e:

I suspect initially it needs to be available in very limited contexts: single-statement lines

horse = (Coconut(), Coconut()); horse.ride()
^except  # <- error

and not inside compounds

x = {
  "x": 1,
  "y": getDatabase(),
  ^except   # <- error
  "z": getDatabase()   # <- user omitted comma thinking that was the issue
  ^except   # <- error
...

y = [
  getHorse("Coconut"),
  ^except   # <- error, with or without the comma
]

May be later we’ll want it available in [edit] comprehensions but that can be revisited as an improvement.

I suspect we should also prohibit it from use on multi-value with statements:

# ok
with open("nudge", "wb") as out:
    ^except PermissionErrora s e: return RuntimeError("can't write arthur's file")

# not ok
with Connect("db") as db, open("nudge", "wb") as out:
    ^except ...

# because the user could also have written that across multiple lines.
1 Like

Side point: “Localization” usually means translating something into a different language. You may wish to pick a different name; I came to this thread expecting a proposal along the lines of “let’s make it easier to get my error messages in French”.

The caret syntax is, quite frankly, ugly. It’s not going to read very well. Perhaps more importantly though, having a syntax that comes immediately after an otherwise-complete statement is going to cause a lot of confusion.

PEP 463 proposed similar functionality, but for a number of reasons, it didn’t get traction. One of those reasons was the difficulty of finding a really good syntax, but it wasn’t the only one, and I suspect that, even if someone does come up with the most gorgeous way to write this, it’s still not going to be accepted. You are, of course, most welcome to try, but I would at least recommend giving that document a read so you know what challenges you have to overcome here.

5 Likes

And will simply not work in REPL, in which each complete statement is executed immediately.

Thanks for the link to PEP-463, which I wasn’t aware of before. I think one problem with any attempt to generalize exception handling as an expression is that most exception names are pretty long to begin with, so including it and the default value on the same line as the expression that may produce the error is likely to make the line too long to fit the maximum line width of many style recommendations.

I think most use cases listed in PEP-463 can be more succinctly expressed with the operators proposed in PEP-505, which Guido, who rejected PEP-463, is an avid proponent of, and which I suggest the OP to take a look at as well.

1 Like

The rational for this is hard to understand and these examples don’t really help.

For the key lookups, the if (h := data.get("hello")) and (w := data.get("world")): form is perfectly fine. There’s nothing wrong with look before you leap if the criteria for leaping are as easily definable as that.

And for the pickle example, this kind of exhaustive exception catching is only appropriate for top level code. That needs to separate exceptions into expected errors that are treated as signals, errors that need translating into something else (PermissionError for example rarely has anything to do with permissions) before being presented to the user and unexpected exceptions that require additional information gathering and some “I’m a bug, report me at $url” presentation code – all much more work than one line returns of Monty Python themed strings or the perceived inconvenience of the word try: and an extra line break. Overly handling exceptions the rest of the time only obfuscates diagnostics.

So when you say…

… I’m inclined to think that this is only coming from an overuse of “try first, ask later” mixed with the Java-esque mindset that every exception must be handled where it originates.

2 Likes

What is stopping you from doing this instead?

if (h := data.get("hello", None)) and (w := data.get("world", None)): ...

If you want some code to be executed if certain keys are missing, you can also just do "hello" in data to check for it’s presence.

As for the proposed syntax, I think it is unlikely to get accepted. The ^ points to the line above, but if something is written in one line, that doesn’t make sense anymore. With your two decades of Python, you should have known that this is unlikely to be accepted, as it can already be done in various other ways. And a mix between an operator ^ which is used for logical xor (__xor__, …) and a keyword except is very unlikely. Something like that has never happened (except for import * I guess).

As you say with import *, that’s actually not too big a problem. A caret isn’t currently meaningful at the start of a statement, although I will say it’s amusing how many of them we can pile up:

>>> ^
  File "<python-input-0>", line 1
    ^
    ^
SyntaxError: invalid syntax

My brain wants to perceive this as a rocket on the launch pad, ready to lift off. Anyhow, there are other issues with this syntax, so it’s kinda moot, but having the same symbol be a binary operator and also having other meaning isn’t inherently a problem. Notably, an at sign could be matrix multiplication a@b or it could be a function decorator @deco. Wouldn’t help with this particular example though.

I had looked at it, and a few others, but there are various good reasons not to have it on the same line; it is a separate statement and I suspect that that in practice it’s going to be very unhelpful having hidden exception handling:

with sqlalchemy.engine.connect_to_database(getDatabaseUri(), getDatabaseAuth()) as db ... except ConnectionError: raise RuntimeError("Who ever looks this far to the right?")

Your point about it not working in the REPL is pretty fair, but I really want it to be more visible than Rust’s ? while being less kludgy than Go’s if err != nil {four liners. I had an impulse against needing something on the end of the proceeding line but I think “:” is so well ingrained in us that some probably even pronounce it half-way between “colon” and “continue” :slight_smile:

The REPL point is fairly terminal on this wording :pensive_face:

Some languages love to throw as much as they can onto a line, whereas I definitely feel that there’s a separation between action and reaction that should be visually distinct. I’ve seen 505 and Guido’s position, but - acknowledging I didn’t re-read the whole thing just now - I thought that was to do with coalescing which might be viewed as orthogonal if not a very different direction than the idea here; that is, it is typically part of buck-passing mechanisms like Rust’s ? operator.

1 Like

On a context manager? Yes, definitely, and PEP 463 never said anything about context managers. They’re already statements, and there’s no point trying to cram that much into a line.

The main goal of PEP 463 was to do the sort of job that dict.get() does, without requiring each such feature to be its own API. For example, we have stuff[x]stuff.get(x, dflt), but we have getattr(x, name) that has getattr(x, name, dflt), and next(it)next(it, dflt). And some don’t currently have any such alternative, so you have to make your own. It was always about short expressions, though, where a four-line try block is a huge expansion over a simple expression. Unfortunately, the requirement to name the exception leads to it still being quite verbose (and if you DON’T name the expression, we have the inverse problem that now we’re catching way too much).

1 Like

I noted the example was “real world preferred”, and the variable captures are there so you can perhaps write code telling the user which field was missing. The emphasis was intended to be on how you map such a basic, mundane operation into the try-first approach and how that expands out line wise, because unless you break it into individual try blocks, your exception handler can end up becoming more complex. In my example, it’s trivial, but let’s flesh it out a little

try:
  h, w = data["hello"], data["world"]
  with database.transaction() as t:
    uid, lid = t.query(User.id, User.lang_id).where(User.name == w)
    ...
    ...

    ...
    ...

    loc = locstr(f'$("GREET_{h.upper()}", "{w}")', w)
    greeting = locstrs[loc][lid]
    ...
    ...
      ...
      ...
        ...
          ...

       ...
         ...
           ...
         ...
           ...
             ...

    result = sendRpc(uuid=uid, type=RPC.Greeting, name=w, param=greeting)
    ...
      ...

    with ...
      ...
      ...
        with ...
          ...
          with ...
            ...

    log("greeted {} with '{}' [{}]", h, w, greeting)
except KeyError as e:
  if e.args == ("world",):
    raise ErrorBetweenKeyboardAndChair("The 'world' field is missing")

[edit: not sure when I deleted this:] this is paraphrased from code I recently had to help with

(* no ninja mutant hero turtles were harmed in the obfuscating of this code)

Come back to that in a moment.

That’s a good point, I’ll admit I partly had that Java/C++ exception-atomicity in mind - but not the way I took you to be suggesting.

There’s a point where between context managers and etc you already have a lot of left-space, and then you have a long horizontal distance (because your local style guide doesn’t prohibit 220 line functions) between h, w = … and the except handler(s) at the end of what becomes a very long block as the code grows (most of the code bases I’ve worked on tend to have long lifespans; my current employer’s Python codebase is old enough to drink/marry/buy a gun in almost every country)

I regularly see people coming to Python interpreting that as Pythonic: every function is a try block, and everything is raise. It ends up becoming goto like.

I tend to think that people get overly familiar with the pattern of just letting errors happen - because that’s actually kind of Pythonic, but they’re overlooking or avoiding the fact that when X first wrote this as a 9 line function, 4 of those lines were exception handling.

The offending line in the code was the (unobfuscated original of)

    greeting = locstrs[loc][lid]

Which was handled, just fine, it just wasn’t handled the way the author intended because they didn’t want to further clutter that function with

    try:
      greeting = locstrs[loc][lid]
    except KeyError as e:
      raise ConfigIssue(...)

I suspect on one level the author of that code didn’t even think to handle a specific error because on one hand that was inconsistent with the entire rest of that function and on the other hand it was noisy. Perhaps what they should have written is:

    try:
      greeting = locstrs[loc][lid]
    except KeyError as e:
      if e.args[0] == lid:
          raise ConfigIssue(... in {language} ...)
      raise ConfigIssue(... undefined localization ...)

but this is out of family in that codebase where everything is either carefree or one giant try block.

note1: locstrs doesn’t have a get operator; well, yet; note2: I’m giving this codebase a rather bad rep, functionally it’s incredibly robust and reliable, but the ux when things are off-nominal is bloody awful. In this actual case, due to minor recent change, the program ran silently for 9 hours and then output “KeyError: hello”.

I’m not trying to preach c-family Thou Shalt Have No Lines Between Thy Bug And Thy Except, but I suspect the baby was in that bath water…