Expressions to handle raising and catching exceptions, plus coalescion

Ideas
1

This idea isn’t new (PEP 463 – Exception-catching expressions | peps.python.org, which is where I’m pulling lots of my examples from, and PEP 505 – None-aware operators | peps.python.org), but I’d like to discuss it again in the context of dealing with both sides of “dealing with functions that decided to handle failure differently than I need them to”.

The reason why I’m proposing this is because adapting the behavior when calling such functions can usually be expressed as a single thought - “If this value can’t be found, I want to raise an exception” or “if this operation fails, just use this default value instead”, which should optimally translate to a single clear line of code. The current reality is that writing the code to achieve that will often take two to four lines instead, with cumbersome syntax, levels of indentation, and overall more surface area that you have to understand while reading the code, that could produce bugs.

When a function that’s supposed to return a value fails to achieve its goal, there are two common approaches to dealing with it:

  • raising an exception
  • returning None

Plus the cop-out:

  • provide either a parameter which lets the caller decide what happens (including setting a better default than None), or an alternative function with the other behavior

I personally like the cop-out best, but it puts strain both on library maintainers for writing, documenting, and caring for these additional parameters/functions, as well as users who’ll then have to learn one more keyword/function and how it’s used.

Some examples, in case you don’t know what I’m talking about.

  • standard library, likes to cop-out with a default-parameter:
    • next(iter, default)
    • min(sequence, default)
    • dict[key] vs dict.get(key, default), two “functions” plus a default-parameter in one
    • list.pop() has no None if raise option or alternative
    • bytes.decode(encoding, errors='strict' | 'ignore' | 'replace'), cop-out, but “default” didn’t apply
    • asyncio.current_task() has no raise if None option or alternative
  • sqlalchemy: Query.one vs Query.one_or_none
  • pandas:
    • pandas.to_datetime(arg, errors='ignore' | 'raise' | 'coerce', ...)
    • DataFrame.loc has no None if raise option or (straight-forward) alternative

I’d like to propose the following:

# raise if None
task: asyncio.Task = asyncio.current_task() ?? raise ValueError("Can't proceed if no task is present.")

# appropriate default instead of None
match: re.Match = re.search(my_pattern, my_string) ?? re.search("", "")

# None if raise
value: int | None = my_int_container.pop() ?! None

# appropriate default if raise
value: int = my_int_container.pop() ?! 0

# raise a different exception
value: str = my_str_dict["key"] ?! raise IndexError

That’s it. The syntax is inspired by null-coalescing operators that I know from C# and PHP.

Discussion points that I anticipate:

  • why not words instead of ?? and ?!, obscure operators aren’t pythonic → happy to hear suggestions, I simply couldn’t come up with anything good that didn’t look like a half-baked conditional expression.
  • the exception handling should be able to narrow down caught exceptions → I personally agree, but since I’m proposing a pattern with the purpose of simplification, I didn’t want to start off with any potential bells or whistles. Also, I don’t know what it should look like, foo = func() ?! ValueError: None? It would make it syntactically different from its ?? sibling as well, which might be confusing.
  • why not provide check-options to both operators then, and have their omission mean ?? None: ... and ?! Exception: ... respectively → I like the simplicity of just not having that, but maybe it’s not useful enough if it’s missing.
2

Related:

1 Like
3

These are all actually what you call “cop-out”, not just “default”. IMO this is the best API design, but the fact that it requires sentinels to correctly handle None] from pure python [1] is ofcourse a bit annoying.

  1. Or you could have even more confusing sigantures with *args and **kwargs, but that doesn’t make the handling any simpler ↩︎

4

Thanks for linking those, I’ll try to read them and incorporate their info if appropriate. Too bad PEP 505 doesn’t include a rationale on its deferral.

5

True, I’ll add that info. I also like it, but I also didn’t know that min has one too and only learned that while reading PEP 463. Yet another reason why I’d like a straight forward pattern that allows me to simulate e.g. the “default”-behavior no matter how the function behaves.

6

When you say ‘if raise’, do you mean if any exception is raised? What if it’s KeyboardInterrupt?

3 Likes
7

I tried to address this in the discussion points at the bottom of my post, maybe it’d be a good idea to write it down less prose-y. In short, I agree that KeyboardInterrupt (and SystemExit) shouldn’t be caught, and by default ?! should cap at Exception instead of BaseException. In long, it might make sense to extend the patterns to allow checking the coalesced value or handled exception explicitly:

# function that returns a custom singleton on failure instead of None
import pandas as pd
date: pd.Timestamp = pd.to_datetime(maybe_date, errors='coerce') ?? pd.NaT: pd.Timestamp(0)

# more fine-grained control to not capture `ValueError`s
from pandas.errors import ParserError
date: pd.Timestamp = pd.to_datetime(maybe_date, errors='raise') ?! ParserError: pd.Timestamp(0)

# default behavior, equivalent to omitting `Exception: `
val: int = my_int_dict["key"] ?! Exception: 0

# default behavior, equivalent to omitting `None: `
val: int = my_int_dict.get("key") ?? None: 0  
# I'm aware that this ^ would be bad code, but imo it doesn't look much worse

Something like this.

8

I’m not a big fan of the style personally, but I think the best possible syntax for it would be a soft keyword which is followed by a mandatory list of exception types (I don’t say “tuple” to be more like except), probably wrapped in parens.

Borrowing Ruby’s rescue keyword, which is the best name I’ve seen for this concept:

d["foo"][0]["bar"] rescue(LookupError) "baz"

IMO having an unqualified exception handler, even if it’s scoped down from BaseException, would be a non-starter.

This would allow for constructs like

run() rescue(KeyboardIntereupt) sys.exit(130)

Mostly I dislike this style in languages which have it[1] because they fail to specify which exception types are caught. If it were added as above, I might actually use it because it’s just a sugar for a larger, minimally correct, try-except block.

  1. Okay, admission of guilt here. I only really know it from Ruby, and am vaguely aware that some other languages have this (mis)feature. ↩︎

1 Like
9

@Rosuav As the original author of the referenced PEP proposing a similar feature in the past, you might have some particularly helpful comments here…

10

I don’t think PEP 463 died for lack of a good syntax. While a truly compelling syntax might very well benefit the idea, it’s not a clear case of “come up with a different spelling and it’ll be viable”.

(Note Guido’s rejection message as quoted at the top of the PEP - he wasn’t enthusiastic about the syntax, but called it “acceptable”, and had other considerations.)

This is one of those concepts that is fundamentally difficult to come up with syntax for. It needs to be, at best, a ternary expression: it has to contain a primary expression, an exceptioin list, and a replacement. Those are HARD to devise good syntax for - just look at PEP 308 and the array of syntax options that were debated.

So, rather than focusing on the syntax, it would be better to focus on other concerns, most notably Guido’s rejection relating to PEP 463’s motivation and rationale. What exactly is the benefit, and how compelling is it?

In the nearly ten years (it’s been THAT LONG??!?) since that PEP was written, how many times have I yearned for expression catching syntax? Actually, not many. There have definitely been other features that I am more likely to wish for (out of my own PEPs, late-bound argument defaults (PEP 671) is far more likely to be beneficial, and I don’t know if it’s ever had a PEP, but a “filtered iteration” syntax would frequently make my code look more elegant), so I think the motivation for PEP 463 isn’t strong enough to really push for.

If someone DOES come up with an absolutely spectacular syntax, I think it would be great! And the feature would certainly be of value, not hugely frequently but it’d be nice. But when all the available syntax options are weak, the use-cases are relatively rare, and the demand for it is sporadic and luke-warm, I don’t think the proposal’s going to go anywhere.

But I would be happy to be proven wrong on this.

5 Likes
11

This case is already a feature of the language:

match: re.Match = re.search(my_pattern, my_string) or re.search("", "")

The or operator returns its first truthy operand from left to right.

1 Like
12

I see that this could work but I question how useful it would be. The lines would get quite long from listing the exception classes, so it might be more readable to just use the existing syntax that spans multiple lines.

1 Like
13

Yep!
And that’s part of why I’m not a huge fan.

But IMO having specified error classes is pretty important, seeing as bare except is frequently (in my experience) a bug.

1 Like
14

The existing Python offers plenty of ways to write overly long lines of code. There’s not really anything that can be done to prevent sufficiently… motivated programmers from writing hideous code.

The consideration should be about whether good, responsible uses of a new syntax enable something elegant.

I see potential, but there are a lot of less-than-obvious design decisions to make for something like this.

15

Thanks for chiming in Chris, I wasn’t brave enough to ping you myself.

I read your and Stephen’s input and took a hard look at what I’m trying to propose, and came to the conclusion that I do think that there’s something of value here that I’d like to continue pursuing. To address the points you made and some more that I got from reading the other threads –

How does this proposal avoid the reasons why PEP 505 failed?

As I’ve understood it, PEP 505 failed because one of the things it tried to introduce, ?./?[], was deemed to be an overall / long term detriment for the python ecosystem. None such concerns were voiced for ??. I rest my case.
Well, actually people were a bit lukewarm on the ?? operator though, because (paraphrasing) “python already has powerful patterns to deal with None, so it doesn’t need something like ?? as much as other languages once you actually learn to write idiomatic python code”. I’ll try to address this in My value proposition.

How does this proposal avoid the reasons why PEP 463 failed?

I’m not confident that I can come up with amazing syntax, so the only angle I’m left with, again, is that a better value proposition would help.

In my opinion, a weak point of the pep was that it didn’t have a very tight focus in what exactly it was trying to achieve and why. While the example functions and why they shared a shortcoming was good and convincing, it then went on with a discussion about fundamentals (LBYL vs EAFP) and ended up concluding that it would be helpful if it were possible to use exception-handling syntax within expressions. Allowing this has the potential of serious ramifications outside of solving the initially outlined issue, which was just about how to handle functions that raise exceptions even though one would rather have a default value.

Also, the timing right now is better because soft keywords are an option.

My value proposition

Chris, you say

how many times have I yearned for expression catching syntax? Actually, not many.

I personally have a different experience. Every time I do either of

# fail with None
value = foo()
if value is not None:
    value = "default"

# fail with exception
try:
    value = bar()
except ValueError:
    value = "default"

I’d preferred to have a construct that better represents what I want to achieve. This isn’t only about lines of code that you can avoid, although 1 is less than both 3 and 4, but whether a common pattern (“provide a default value on failure”) should be solved with constrained/bespoke syntax that solves it cleanly.

To give a parallel, if statements and gotos are perfectly fine to write loops. Introducing while and for didn’t solve a problem that wasn’t solvable before – as long as you write code that doesn’t have bugs and is easy for others to maintain without messing things up.

I personally feel that this problem is common enough, and leads to enough issues that could be solved by having this kind of pattern, to make it worth it.

@sirosen did you mean that rescue should be used instead of both operators? I think that having a different word for non-exceptions would be better, but don’t have a strong opinion here, e.g.

# fail with None
value = foo() coerce "default"  # option 1.1: noone seemed to mind un-parametrized `??`
value = foo() coerce(None) "default"  # option 1.2: looks a bit weird to me
value = foo() coerce(None): "default"  # option 1.3: I like this one better

# fail with exception
value = bar() rescue(ValueError) "default"  # option 2.1: what stephen suggested
value = bar() rescue(ValueError): "default"  # option 2.2: also prefer this, for consistency with myself

# and just to throw something new into the ring, what about this option 3?
bar_errors = [ValueError, IndexError, RuntimeError]
value = bar() rescue(*bar_errors): "default"

I also still like both ?? and ?! though, mostly because they invoke a mental image that fits the operation, but no strong opinion there either.

16

or doesn’t get used much in this context though, because people expect it to “return” a bool. I know I was very confused the first time I saw that that wasn’t the case. Also, it can’t handle functions that return valid falsey values, so it can only cover part of the problem.

As for

[when listing many exceptions,] it might be more readable to just use the existing syntax

I thought a lot about this, and I think my answer is that reducing lines of code isn’t the main goal anyway, it’s about reducing mental load by having dedicated syntax. If you see a try...except, it could achieve all kinds of different things, with “provide a default value” just being one of them. You have to read it in order to understand it. If you see rescue, you know exactly what it does, and in that sense I’d argue that it’d be more readable. This is what my goto vs looping-syntax example was supposed to show as well.

So it really comes down to if the saved mental load from having rescue/coerce is more than the added mental load from having new syntax, and if there aren’t other lower hanging fruits anyway.

While I’m still very much in favor of my own post here, one thing that it can’t handle is adding logs:

try:
    value = foo(*args, **kwargs)
except ValueError:
    logging.warning(f"Couldn't call foo with {args=} and {kwargs=}, proceeding with {default=}.")
    value = default

So sometimes when I really want that info I’d have to go back to the long form anyway =/

17

For the record, I’m entirely pingable :slight_smile: No need to be afraid. Though there’s a good chance that a thread in the Ideas section with a subject line like this will catch my attention.

Pun intended? :smiley:

The truth is, that’s the case for me too… but the number of times when that actually comes up are fewer than you might think. In a lot of situations, it’s more complicated than the trivial and easy-to-justify examples, which can make it less worth transforming.

Oh yes, I absolutely agree here. This is exactly the benefit. Shorter code is its own advantage but being able to express intent better is the true goal.

18

I’d like to hear them =)

To give my current list:

def excape_hatch_lol_logs_here_I_come(exc) -> bool:
    log.warning(f"Couldn't call bar, got {exc!}, proceeding with {default=}")
    return isinstance(exc, (ValueError, IndexError, RuntimeError))

value = bar() rescue(excape_hatch_lol_logs_here_I_come): default

*edited the function because I completely messed up the interface in my first go

19

This is the big issue I have with proposals like this. They work when things are simple, but in real-world cases, you often need more. With exception handling, that “more” is very often some form of wanting to add logging, or otherwise record the fact that not everything went to plan. And as soon as you need that, you have to go back to the statement form. So maintainability suffers, and there’s a small, but definite pressure to skip the logging because it will stop you using the “simpler form”.

2 Likes
20

Maybe it’s time for the legwork then… sift through tons and tons of code and get an idea what kind of percentage of user / lib / app code actually involves a rescue. I’ll see if I can script and review for that in the somewhat soon-ish future.

Pun intended? :smiley:

actually no, it was tongue in cheek, but I don’t see the pun :frowning_with_open_mouth: