Better way to deal with exceptions

Python Help
45

okay. explain how we would be able to catch it, then.

source = OurDataSource([TheirDataSource()])

# what are we even supposed to catch here?
source.get_property_values("repos")
46

We don’t know, we did not design your API contract. But if you’re using exceptions as some kind of protocol you need to catch something. That’s how exceptions work.

47

then what was the point in us writing all this stuff:

or this stuff:

or all the other stuff we wrote in an attempt to explain what we think python should do better.

what was the point in writing all this just to get an “we don’t know. you’re the one who’s supposed to know.” as a response.

48

Mod here: I merged the split posts back into one as the topic didn’t seem to change significantly enough to require a separate conversation. Plus the original thread was not so long as to make it difficult to read and catch up on.

I have also turned on slow mode to try and reign in the tone as several people flagged the overall discussion as not going well and I agree it did not look to be heading in a good direction it things don’t change.

49

It doesn’t seem to me unpythonic to use return codes if they work and exceptions don’t. It would seem more unpythonic to have a construct to make exceptions feel like return codes. The fact that exceptions propagate and returned values don’t is a fundamental difference between the two. If you don’t want propagation, I don’t understand why exceptions need to be used.

50

but the language doesn’t provide anything to work with exceptions beyond the barest of try/except logic.

Python provides a lot of possibilities to extend its functionalities. Did you try to work with a decorator for you functions? This could be done with something along the following lines:

import dis

def wrap_expecptions(func):
    def result(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except BaseException as e:
            # Error happens in the wrapper if the arguments do not 
            # agree with parameters
            if e.__traceback__.tb_next is None:
                raise
            # Let exceptions through that were explicitly raised 
            # directly in func
            tb = e.__traceback__.tb_next
            if ( tb.tb_next is None 
              and tb.tb_frame.f_code.co_code[tb.tb_frame.f_lasti] 
              == dis.opmap['RAISE_VARARGS'] ):
                raise
            # Others are masked by a RuntimeError
            raise RuntimeError("Unhandled exception") from None
    return result

The tests behave as follows:

# Test cases
def g(): raise TypeError

@wrap_expecptions
def f(i):
    if i==0: raise ValueError
    if i==1: return 1/0
    if i==2: g()

# Exceptions from wrong parameters and explicitly raised ones are left trough
f()  # TypeError: f() missing 1 required positional argument: 'i'
f(0) # ValueError
# Exceptions raised in c code or subroutines are wrapped
f(1) # RuntimeError: Unhandled exception
f(2) # RuntimeError: Unhandled exception
51

what a fascinating and yet mildly cursed approach (dis???)

you missed a case tho:

    if i==3:
        try:
            g()
        except TypeError:
            raise
52

@SoniEx2 I (and others, I presume) need a code snippet from you to work off of to understand what you’re talking about. I don’t know what you mean by “data sources” or “data properties”. Can you please provide:

  • A minimal code snippet showing what you are able to achieve now, but what the problems are
  • A minimal code snippet showing what you wish you could do and describing how it would behave?

It doesn’t seem worth spending time on this topic without concrete code snippets to work from.

edit: you started to get at this with the start of your new topic but I think more detail was needed… Maybe give it another stab/fresh start?

53

This was only a proof of concept implementation and there are many test cases missing. But I have tried your test case and it behaves as expected - since the exception was thrown in g, it is replaced by a RuntimeError. This is what I would usually expect here: Catching and re-raising an exception is (like a context manager) a paradigm in Python for resource cleanup.

But the better choice would be to mark the exceptions that should propagate out of your library, like by including the library’s name in the text message of the exception. To query that, you would not need to do any disassembly. But in the end, it is your choice what you want to implement.

54

what we really want is some sort wrapper for all of this concept of “fallible” APIs.

some programming languages (icon, unicon) are built entirely around fallibility. however, they only have two… results, let’s say: an expression can succeed with any number of values, or it can fail.

python has typed exceptions. effectively, these represent multiple values an expression can fail with. so, in python, an expression can succeed with any number of values, or alternatively fail with any number of values.

our take is precisely that we could use functionality to finely control how failures propagate through our APIs. generators already do that by preventing StopIteration from propagating through them, while also producing StopIteration themselves.

in fact we would argue generators actually make things less debuggable sometimes, because if you want to propagate a StopIteration, you need to turn it into a return, like so:

try:
  next(whatever)
except StopIteration:
  return

this erases the old stack trace, so you can’t tell “my generator is stopping because this other iterator is stopping”. (side note: python could detect tail-position yield from and even tail-position for loops in generators and use the original StopIteration as a cause.)

imagine if we could use similar tech outside of generators. imagine if we could say, “this function raises MyFailure, but if and only if this specific part raises MyFailure”. imagine if your function has 3 callbacks but only one of them is allowed to raise MyFailure. today, you’d write it like so:

def foo(cb1, cb2, cb3):
  try:
    cb1()
  except MyFailure:
    raise RuntimeError
  cb2() # is allowed to raise MyFailure
  try:
    cb3()
  except MyFailure:
    raise RuntimeError

with expressive power (read: syntax) to finely control failure, you’d write something like this instead:

def foo(cb1, cb2, cb3):
  try MyFailure in:
    cb1()
    raise MyFailure in: cb2()
    cb3()

like, why not improve on the concept of fallibility?

55 
def foo(cb1, cb2, cb3):
  try:
    cb1()
  except MyFailure:
    raise RuntimeError
  cb2() # is allowed to raise MyFailure
  try:
    cb3()
  except MyFailure:
    raise RuntimeError

def foo(cb1, cb2, cb3):
  try MyFailure in:
    cb1()
    raise MyFailure in: cb2()
    cb3()

This is finally an understandable proposal. But I find your syntax confusing. If cb1() raises MyFailure will foo() raise RuntimeError? That would need to be encoded in the syntax I think? Something like

def foo(cb1, cb2, cb3):
  try MyFailure as RuntimeError in:
    cb1()
    allow MyFailure in: cb2()
    cb3()

The utility of this could be debated. I’ve never been close to having a need for something like this. But anyways, leaving aside that discussion for the moment, here is a pair of context managers that gives something like what you’re looking for. This is probably similar to Stefan’s implementation.

class ExceptionTransformer:
    def __init__(
            self,
            transforms: dict[Exception, Exception]
    ) -> None:
        self.suppressed = False
        self.transforms = transforms

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type in self.transforms:
            new_exc_type = self.transforms[exc_type]
            if not self.suppressed:
                print(f"replacing {exc_type} with {new_exc_type}")
                raise new_exc_type from exc_value
            else:
                print(f"Transform suppressed. Not replacing {exc_type} with {new_exc_type}")


class SuppressExceptionTransform:
    def __init__(self, exception_transformer: ExceptionTransformer):
        self.exception_transformer = exception_transformer

    def __enter__(self):
        self.exception_transformer.suppressed = True

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_value is None:
            self.exception_transformer.suppressed = False

class MyFailure(Exception):
    pass

def cb(success):
    if not success:
        raise MyFailure

def foo(cb1, cb2, cb3, successes):
    with ExceptionTransformer({MyFailure: RuntimeError}) as et:
        cb1(successes[0])
        with SuppressExceptionTransform(et):
            cb2(successes[1])
        cb3(successes[2])

print("test 1")
foo(cb, cb, cb, [True, True, True])

print("test 2")
try:
    foo(cb, cb, cb, [True, True, False])
except RuntimeError:
    pass

print("test 3")
try:
    foo(cb, cb, cb, [True, False, True])
except MyFailure:
    pass

print("test 4")
try:
    foo(cb, cb, cb, [False, True, True])
except RuntimeError:
    pass

With results:

# test 1
# test 2
# replacing <class '__main__.MyFailure'> with <class 'RuntimeError'>
# test 3
# Transform suppressed. Not replacing <class '__main__.MyFailure'> with <class 'RuntimeError'>
# test 4
# replacing <class '__main__.MyFailure'> with <class 'RuntimeError'>
56

why should RuntimeError be encoded in the syntax? it’s not encoded in the syntax of generators at least.

it’s kinda implied that an exception you shouldn’t catch (i.e. one that indicates a bug in your code) is raised as RuntimeError. this is what generators do. they didn’t make a GeneratorError or anything to separate StopIteration-caused exceptions from other RuntimeError, for a reason.

since we are trying to tell the developer “if the first callback you passed raises MyFailure, that’s a bug in your code, fix it”, it makes no sense for us to ever use something other than RuntimeError.

we can’t speak to how you write your code, but we ran into this issue with composable APIs (the data source stuff), where you can build transformation chains (or trees) by putting objects inside objects, making the code feel more declarative. as far as we know this is pretty unusual to do in python, but it can be useful sometimes.

compared to something like rust, where exceptions aren’t used but instead values are wrapped in a result type, you’re kinda forced to specify whether something should be an assertion (.expect() or .unwrap()) or if you want to propagate the failure (represented by a little ? at the end of the expression).

actually we can come up with a somewhat practical example. say you have configuration files represented as dicts. configuration files are made by the user, so they can have or not have arbitrary keys. however, you want to abstract over the configuration file so you can query a configuration file by specific properties, and you use another dict to do so:

my_config_keys = {}
my_config_keys[SITE_URI] = "site_uri"

then to retrieve the config:

def get_config(prop):
  key = my_config_keys[prop]
  return config[key]

and you can check if the config doesn’t exist by catching KeyError. in rust, you’d use unwrap() for the first indexing operation, and ? for the second (or something equivalent, if you know rust you know the “proper” way of doing this, we’re oversimplifying (also you probably wouldn’t use the equivalent of dicts they are a very python thing)). but, this python will also raise KeyError if you pass in an unknown value for prop.

so you could write it as:

def get_config(prop):
  key = my_config_keys[prop]
  try:
    return config[key]
  except KeyError:
    raise ConfigError

but why not instead

def get_config(prop):
  try KeyError in:
    key = my_config_keys[prop]
    raise KeyError in: return config[key]

such that passing in the wrong prop causes a proper RuntimeError!

57

I think you are thinking about exception in a way that is non-standard for python. Nonetheless, you can achieve what you want with context managers. So why is a (very costly) language change needed?

Who is “you” in this sentence? Also, I don’t think that implication is true in any circumstance. RuntimeError is not some special kind of exception. There are many types of exceptions, each with different semantics to allow python authors to encode a variety of types of failures. Any type of exception can be caught or not at any layer in the stack at the developers discretion.

It sounds like you’re worried about something like:

  • The user gives your library a bad configuration bad_user_config.
  • This bad configuration is then used in some of your libraries code some_code
  • The bad user config bad_user_config then causes some sort of exception to be raised in some_code

I think you are looking for a way to communicate to the user (using exceptions) that their code has caused a problem in their library, but that it’s not your fault, it’s their fault. This is a fine thing to communicate. However, if I’ve captured correctly your issue, I think it would make much more sense to do the validation and exception raising on the bad config as soon as possible, fail early. This way you don’t need to worry about the user planting mines that will only explode much later in runtime making it challenging to communicate what happened.

If you do that some_code can assume that user_config is always good and callers of some_code can assume that any exceptions raised during some_code are an issue with some_code and not an issue with some configuration they passed in at an earlier time.

Also, you seem to be implying that if a user gets a RuntimeError they will understand that it was an issue with their configuration or something. This just doesn’t make sense… can you explain more about how you are thinking about RuntimeError and why you think it is special? Why is RuntimeError “proper” here?

FYI I have zero familiarity with rust.

58

First of all, IMHO catching an exception and re-raising RuntimeError is a bad practice.

Secondly, why catching all the rest of the world and change the exception to RuntimeError, instead of emitting a custom exception only for the code I want to discriminate? This is quite more simple, and it was suggested to you quite often previously.

About the new syntax, frankly I found it really unreadable. It’s only my opinion of course.

59

if it’s so bad practice then why does python do it (see: generators)?

it’s pretty standard when you don’t want your caller to handle a given exception. nobody’s gonna catch RuntimeError except for logging.

the syntax, as usual with these things, is mostly a matter of getting used to it.

60

How many times do you have to be told that this is not the parallel you think it is?

Make your own case and don’t try to pretend that generators have anything to do with it.

Yes, and they also won’t catch your custom exception. So… why do you subclass a standard exception, and then act all shocked when it gets caught when that exception is being caught? That is the way exception handling works.

61

StopIteration is a particular exception, because it’s swallowed silently in many contexts. IMHO PEP 479 was a sane decision, and I don’t know in this particular case what exception could be used apart RuntimeError. Certainly not a custom exception, that instead you can happily use.

Define “standard”.

Maybe, but Python historically is very readable in plain English. What try exception in means in English? And why in, that is used for iterables?

Furthermore your syntax will use implicitly RuntimeError. Apart the fact that “explicit is better than implicit”, this will make RuntimeError special.

62

well they could have introduced a GeneratorError instead of using RuntimeError for it. but then ppl could catch a GeneratorError and handle it, while handling a RuntimeError is frowned upon.

ah we see. it’s mostly an issue of finding syntax/keywords that will fit nicely without breaking everything or conflicting with existing code.

so maybe:

try with StopIteration, ValueError: # try to run the following block with these exceptions as "protected" or "checked" exceptions
  ...
  raise StopIteration, ValueError from: # raise any of these otherwise-"protected" exceptions if they're raised from the following block
    ...
  ...

(but this might be confused with regular with so maybe not a great idea… but anyway)

63

Why introducing a new exception in the language for a very narrow case?

…I think you answered yourself :smile:

Nope. The syntax is a problem, but it’s not the main problem for me. The main problem for me here is that you’re assigning a special rule to RuntimeError, that’s not a so special exception.

64

ah.

… would adding an UncaughtError be too much?