Try With Resources — Elegant Handling of Exceptions In Context Managers

Ideas
1

A common task in programming is accessing some resource that might or might not be available. If the resource is available, processing can happen as usual — but the resource should be safely released when it is no longer needed; however, if the resource is not available when requested, usually some exception is raised and should be handled or propagated. Exceptions can also occur during the processing of an open resource — in which case the resource should be closed.

This task is common enough that some languages like Java have implemented try-with-resources constructs that allow for this pattern. A resource can be accessed — but if any exceptions occur while acquiring the resource or during processing of the resource, they can be caught.

Python currently does not have a single construct to achieve this — resources can be acquired and closed with context managers, and exceptions can be caught with try statements. Combining these constructs, while a common enough task, is clunky.

The pattern of resource acquisition/processing with error handling is common enough that I feel Python should allow with statements to include except, else, and finally blocks to handle any errors that are raised during a context manager’s lifetime.

For example:

with open(file, "rb") as f:
    config = json.load(f)
except FileNotFoundError as e:
    panic_with_message(f"Cannot find file: `{file}`", e)
except JSONDecodeError as e:
    panic_with_message(f"Invalid JSON in: `{file}`", e)
else:
    run(config=config)
finally:
    global_cache.clear()

Context managers do have the ability to handle exceptions as they are being exited (For example, as Suppress does); however, there is currently no elegant way of providing custom handling logic to context managers.

5 Likes
2

This has been discussed before: `try with` Syntactic Sugar - #8 by oscarbenjamin — but the conversation there seemed to revolve mostly around aesthetics and code length rather than providing a codified implementation of this common resource handling pattern.

1 Like
3

While it could be made less verbose, I don’t see any technical issues with

try:
    with ...:
except:
    ...
else:
    ...
finally:
    ...

So I’m neutral on the idea atm.

2 Likes
4

The example block of code is quite appealing. But doesn’t this need to decide (or at least it should be documented) whether the context manager’s __exit__ method is called first, or the finally: block is executed first? A clue is in the name, but a combination of try: and with makes that unambiguous and explicit.

1 Like
5

I guess that would depend on the expected behaviour of examples such as:

class Echo:
    def __enter__(self):
        print("Entered Echo")

    def __exit__(self, exc_type, exc_val, exc_tb):
        print(f"Exiting Echo with: {exc_type}")
        return False


with Echo():
    print("Raising ValueError")
    raise ValueError()
except ValueError:
    print("Except ValueError")
finally:
    print("Finally")

Would this print:

Entered Echo
Raising ValueError
Exiting Echo with: <class 'ValueError'>
Except ValueError
Finally

Or:

Entered Echo
Raising ValueError
Except ValueError
Exiting Echo with: None
Finally

I’m in favour of the former — I feel the __exit__ code should run before any exceptions or other blocks are handled — which would close any resources as soon as the with block exits. This is also how Java’s try with resources behaves:

public class Main {
    public static void main(String[] args) {
        try (Closing closing = new Closing()) {
            System.out.println("In try block");
            throw new Exception();
        } catch (Exception e) {
            System.out.println("Catching Exception");
        } finally {
            System.out.println("Finally");
        }
    }
}

class Closing implements AutoCloseable {
    public Closing() {
        System.out.println("Creating Closing");
    }

    @Override
    public void close() {
        System.out.println("Closing now");
    }
}

prints:

Creating Closing
In try block
Closing now
Catching Exception
Finally

__exit__ running before any except blocks also removes confusion around the order of events for exceptions that happen in __enter__ or the context manager constructor (e.g. FileNotFoundErrors) which are not followed by __exit__ and exceptions that happen in the with block which are followed by __exit__

6 
class Echo:
    def __enter__(self):
        print("Entered Echo")

    def __exit__(self, exc_type, exc_val, exc_tb):
        print(f"Exiting Echo with: {exc_type}")
        return False

try:
    with Echo():
        print("Raising ValueError")
        raise ValueError()
except ValueError:
    print("Except ValueError")
finally:
    print("Finally")

Produces exactly the output that you say you prefer:

Entered Echo
Raising ValueError
Exiting Echo with: <class 'ValueError'>
Except ValueError
Finally

What’s the problem?

1 Like
7

You are conflating two different sources of errors: with calling open and with using the object that a successful call to open returns. They can and should be separated. The with statement is already a sort of try statement; any exceptions are passed to the __exit__ method of the context manager. In this case, you want to handle an exception that occurs before the file-like object even exists.

What you’ve shown should be written like this

try:
    f = open(file, "rb")
except FileNotFoundError:
    panic_with_message(f"Cannot find file: `{file}`", e)
else:
    # *If* the file was opened successfully, ensure it is closed
    # no matter what happens while we are using it
    with f:
        try:
            config = json.load(f)
        except JSONDecodeError as e:
            panic_with_message(f"Invalid JSON in: `{file}`", e)
finally:
    global_cache.clear()

It’s only 2 lines longer, and it’s more clear about when a file actually needs to be closed. It’s not important that the call to open appear at the start of the with statement, only that the value it can produce does.

10 Likes
8

While it is true that you can write all that. But it is not as elegant.

Although I understand the reluctance in considering this as a suggestion since the alternatives exist.

1 Like
9

Elegance is in the eye of the beholder. I find it elegant that with blocks and try blocks are orthogonal components that can be easily composed. I would find it less elegant to combine the two.

If the only benefit is saving 9 characters ("try:\n ") and a level of indentation, it seems like a clear loss to me.

1 Like
10

I ran into this issue with a locking mechanism.

I had old code like this

try:
  lock = obtain_lock()
except LockError:
  log_lock_error()
  return

do_thing_a()

if b:
   lock.release()
   return

do_thing_c()
lock.release()

I ended up cleaning it up like so

try:
  lock = obtain_lock()
except LockError:
  log_lock_error()

with lock:
  do_thing_a()
 
  if b:
    return
  
  do_thing_c()

What I would like to write is:

with obtain_lock():
  do_thing_a()
  if b:
    return
  do_thing_c()
except LockError:
  log_lock_error()

but here I’m actually only interested in exceptions thrown in __enter__, and wouldn’t expect except to capture errors within the code block.

In a way, reading everyone else’s example has convinced me it would be hard to have with/except work in a clear way.

I really don’t like double-indentation of try/with (think it harms readability, especially given how code with locks tends to be gnarly already), I don’t like having to two-phase it, but I don’t know how with/except can be clear unless there’s literally a new trywith keyword or something.

4 Likes
11

Knew I am not the only one with this problem.

Just recently I happened to need context managers, and are they unwieldy in the current way. The biggest problem I am encountering is that I can’t catch exceptions from the context manager itself and from the `with` block body itself. Okay, I can circumvent that in my own context managers by wrapping every exception in a specific exception that is only thrown by my context manager. Smelly, but ok, even though it is unwieldy.

But what if I need to use another library’s context manager? What if it doesn’t disclose properly what exception can be raised? What if I don’t care what exception is raised from that context manager, I just need to know and handle the exceptions from the context manager itself and from the `with` block body separately? Ok, it may work if you wrap the insides in a try-catch, but what if I need the reverse of it? I can’t wrap the context manager invocation specifically in a try-catch.

The fact that the with-block exceptions and context manager exceptions are mixed and there is no way to separate them naturally is rather bad. Perhaps this should be a different suggestion though.

UPD: Found about ExitStack, which is what I needed, however, it is still pretty unwieldy and reads wrong, albeit with much less code smell. I still wish there was some easier way to differentiate.

1 Like