Better way to deal with exceptions

Python Help
1

we would like to use exceptions in our code. that is, we would like to expose exceptions as a public API. that is, users are expected to be able to catch these exceptions.

however, sometimes we wanna use standard exception types, like KeyError and ValueError just to name a couple. unfortunately, when writing complex code, we are bound to use standard functions or language features that can also raise these exceptions, like tuple unpacking:

>>> (x, y, z) = (1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: not enough values to unpack (expected 3, got 2)

in theory these kinds of mistakes should never happen. unfortunately, if we have ValueError as a public API, then anyone who hits such an edge-case will be met with an undebuggable misbehaviour.

we’re currently doing our best to wrap anything that might raise ValueError and reraise it as a RuntimeError instead, just in case there’s a bug in our code we didn’t catch. then, only the parts where we explicitly raise ValueError will be caught by the caller. for example, let’s take this snippet from our code:

    def get_property_value(self, prop):
        """Returns the value associated with the given property.

        If duplicated, an earlier value should override a later value.

        Args:
            prop (DataProperty): The property.

        Returns:
            The value associated with the given property.

        Raises:
            PropertyError: If the property is not supported by this data
            source.
            LookupError: If the property is supported, but isn't available.
            ValueError: If the property doesn't have exactly one value.
        """
        iterator = self.get_property_values(prop)
        try:
            # note: unpacking
            ret, = iterator
        except LookupError as exc:
            # don't accidentally swallow bugs in the iterator
            raise RuntimeError from exc
        return ret

    @abc.abstractmethod
    def get_property_values(self, prop):
        """Returns the values associated with the given property as an iterable.

        If duplicated, earlier values should override later values.

        Args:
            prop (DataProperty): The property.

        Returns:
            The values associated with the given property.

        Raises:
            PropertyError: If the property is not supported by this data
            source.
            LookupError: If the property is supported, but isn't available.

        """
        raise PropertyError

here, we opted to let ValueError be raised by the unpacking and that propagates and all is well. but, we accidentally left the get_property_values call unprotected, so it could also raise its own, unrelated ValueError.

this is really frustrating, and we wish we had a better way of dealing with this issue. so, does anyone know if there is a better way?

2

Usually, if I don’t let the exception to bubble out, I log anyway the stack trace.

3

Hi,

for the particular case that you showed regarding this example:

(x, y, z) = (1, 2)

would you be open to incorporating code that checks the length of the term on the right side prior to assignment?

Assuming you are referencing the tuple on the right side with a variable:

r_term = (1, 2)

while True:

    if len(r_term) != 3:
        print('The tuple entered does not equal three items.  Please make correction.')
        r_term = input('Please enter a tuple with three items: ')
        r_term = tuple(map(int, r_term.split(",")))

        if len(r_term) == 3:
            break

print('Successfully entered the tuple: ', r_term)
4

If you want users of your code to be able to distinguish between your custom exceptions versus builtin exceptions, it seems to me that you must define your own base exception that all of your custom exceptions inherit, which can be as simple as:

class MyLibError(Exception): pass
class MyLibLookupError(MyLibError): pass
class MyLibPropertyError(MyLibError): pass

With this, users of your code can choose to catch any MyLibError or a specific subexception:

try:
    ...
except MyLibPropertyError:
    ...
except MyLibError:
    ...

In some cases, it can be helpful to allow a custom exception to also be handled as a builtin exception, for example if you need to maintain compatibility with some other API:

class MyLibValueError(MyLibException, ValueError): pass
2 Likes
5

did nobody notice the abstractmethod and the fact that the user is expected to bring their own bugs that will be promptly silently swallowed by our code because exceptions-as-API?

also, python is all about “it’s better to ask for forgiveness than permission”, so in theory we should be using exceptions. but the language doesn’t provide anything to work with exceptions beyond the barest of try/except logic.

anyway we would like to have a better time with “exception-oriented programming” than we’re currently having.

6

And context managers, and for loops, and generators, and coroutines, but yeah, no magical solution that stops you from needing to think about them. Do you have an actual concrete proposal?

7

wait what do for loops and those other constructs have to do with exceptions? (we are aware of context managers behaving effectively as a weird try-except tho)

8

Look up how they’re implemented. All of them make use of exceptions in some way or other. For example (pun intended), a for loop uses StopIteration to mark the end of the thing you’re iterating over (or possibly IndexError).

2 Likes
9

oh right, that.

now we remember. wasn’t there something about no longer being able to raise StopIteration in a comprehension since like 3.10 or so? edit: apparently it was 3.7, PEP 479 – Change StopIteration handling inside generators | peps.python.org

so yeah we mean, the issue we’re running into is basically equivalent to the one raised in PEP 479, but not in the context of generators/StopIteration. and none of the PEP 479 alternatives were ever generic enough to deal with our instance of the issue.

10

That’s because they automatically raise that exception. If you raised it manualy from inside the generator, it caused confusion. What you’re seeing isn’t related.

11

no it’s because they implicitly raised it and it made an undebuggable mess.

they’re the same problem, just with different framing. it becomes really obvious if you’re willing to contextualize it differently.

an exception was expected in an outer context, but an inner context raised the exception in a place the programmer didn’t expect. python learned this the hard way with generators and failed to generalize to non-generators.

12

It was the explicit raising of StopIteration that caused problems. Thus raising StopIteration inside a generator (or more specifically, leaking one out of a generator) was made into a RuntimeError.

This is completely incorrect.

13

the funny thing is the PEP even mentions e.g. __getitem__, yet still refuses to generalize.

both cases boil down to leaking exceptions into your API that are not meant to be in your API.

in fact, there was never a strong argument against allowing explicit StopIteration in generators, only against leaking it from inner function calls.

that is, this is bad:

def foo():
  x = next()
  yield x

this is fine:

def foo():
  for x in something:
    if x == bad:
      raise StopIteration
    yield x

but if you make up something about consistency (and refuse to generalize), as the PEP does, then you get to make this look bad, while also pretending that undebuggability issues with situations like the __getitem__ example aren’t worth helping programmers with.

14

I’m seconding @wylee on this: your library should create a base class that your library’s code raises. If your library ends up raising MyLibError, the user knows it’s an expected problem with how their code uses the library. If your library raises ValueError or some other Python exception, then it’s a bug in your library itself.

Brett Slatkin’s book Effective Python also endorses this view, I believe.

2 Likes
15

AttributeError isn’t a bug in __getattr__. KeyError isn’t a bug in __getitem__.

exceptions are control flow, not bugs. they should generally be caught and handled.

16

while we’re at it: StopIteration isn’t a bug in __next__. except when it is. but you’re never gonna catch that with regular python.

17

Looks like you just want to, not that you need to. If you’re going to use exceptions as API you need to use your own, not the standard ones.

2 Likes
18

I ran into similar cases before. My dirty solution looked like this:

x, y, z, *_ = tuple(arg) + (None,) * 3

This might not be the best practice but it’s a one-liner.

19

ugh we have itertools (chain and islice) for a reason

still ignoring the problem.

1 Like
20

Just being curious: how can they help in this case?