Add `try for` and `try async for` syntax

I have been working with async iterators a lot recently and one aspect I have found awkward is handling exceptions which are raised in the iteration. For context my use case (fairly common I imagine) is using async iterators over IO streams, eg MQTT or Kafka messages. These systems can potentially throw errors when you receive a message, and sometimes you want to handle the exception and continue iterating. I would like to use the async for iterating over such async streams but it comes with the issue that it is cumbersome to catch an error in the iteration, then continue iterating. I would like to propose a new feature try async for (and for consistency try for to address this)

To illustrate my example imagine we have a mock IO function and I create a class to iterate over some fetched results:

import asyncio
import random

class IOError(Exception):
    pass

async def io(x):
    if random.random() > 0.5:
        raise IOError
    else:
        return x    

class MyIterator:
    
    def __init__(self):
        self.n = 0
        
    async def __anext__(self):
        try:
            result = await io(self.n)
        finally:
            self.n += 1
        if self.n > 10:
            raise StopAsyncIteration
        return result
        
    def __aiter__(self):
        return self

So the issue is what if the IOError happens, but I still want to continue the iteration?

One option is to abandon the for loop and use a while loop:

my_iter = MyIterator()
while True:
    try:
        result = await my_iter.__anext__()
        print(result)
    except IOError:
        pass
    except StopAsyncIteration:
        break

This works but it’s not very Pythonic, and removes the point of async for. Another option is to catch the exception externally then continue iterating.

my_iter = MyIterator()
while True:
    try:
        async for res in my_iter:
            print(res)
        else:
            break
    except IOError:
        pass    

This to me seems difficult to read and is nested excessively. To solve these problems we might consider changing the iterator to return a sentinel value instead of raising an exception:

class MyIterator:
    
    def __init__(self):
        self.n = 0
        
    async def __anext__(self):
        try:
            result = await io(self.n)
        except IOError:
            return None
        finally:
            self.n += 1
        if self.n > 10:
            raise StopAsyncIteration
        return result
        
    def __aiter__(self):
        return self

So we can iterate through like so:

async for result in MyIterator():
    if result is not None:
        print(result)

However this is also not very Pythonic, as we are meant to use exceptions, not sentinel values to show exceptional conditions.

To solve this I would propose a simple syntax extension, allow try before async for or for loops. So referring back to the first class which raises the exception in the __anext__ method my example would look like:

try async for result in MyIterator():
    print(result)
except:
    pass

The semantics here are if the __anext__ method raises an exception (aside from StopIteration of course), it triggers the except branch, and continues with the iteration. This to me seems much cleaner, and is fairly intuitive to grasp how this syntax works. finally would work in a similar fashion:

try async for result in MyIterator():
    print(result)
except:
    pass
finally:
   print("Always prints!")

What do you think about this new syntax addition? Grateful for any feedback.

Please no. Over the last 20 years, people have proposed at least several combinations of compound statements to save a line and an indent level for a combination that they personally used. `try with` Syntactic Sugar is the most recent previous example. There is really no end of possible compositions people would like added.

5 Likes

How about wrapping the asynchronous iterator with an asynchronous iterator that skips over the IOErrors for you?

class SupressExceptions:
    def __init__(self, it, exceptions):
        self.it = it
        self.exceptions = exceptions

    async def __anext__(self):
        while True:
            try:
                return await anext(self.it)
            except self.exceptions:
                continue

    def __aiter__(self):
        return self

Then you can use an ordinary async for loop to the same effect:

my_iter = SupressExceptions(MyIterator(), IOError)
async for res in my_iter:
    print(res)
3 Likes

I would argue that the try with isn’t really necessary because you can always put your with logic inside the try. With iteration however you often want to handle the exception without breaking out of the loop. It’s not a case of just saving indentation because a for nested in a try does something different.

Do you agree that the existing alternatives I put in my original post are ugly? If not I would like to know which one is best practise.

I think it’s a situation which comes up a lot because if you are iterating asynchronously, you are probably doing some IO operation that can fail.

I think that this too is not a great solution. You have to create a new class when you want different exception handling behaviour, so it’s not very reusable. You could tidy it up by passing in a callback in the init, but that also looks quite inelegant to me.

A new class is far more reusable than simply repeating the code. And it doesn’t involve a language change. You seem to be assuming that language changes to add new syntax are cheap - this is not the case, and any new syntax needs a very compelling case to be accepted. Avoiding the need for a reusable helper class frankly isn’t anywhere near enough of a benefit.

They are rather clumsy, but for an uncommon pattern[1] not impossibly so.

The best practice, IMO, would be to use a utility helper like the SuppressExceptions class @bschubert proposed.


  1. we’d need better evidence before concluding that this is a common pattern - as you’re the only person ever to have flagged this issue, it seems more likely that it’s a relatively uncommon problem ↩︎

4 Likes

OP is not the only person, in fact they are the second person in the last month: For ... except ... finally

I did see this thread but I didn’t think it expressed my issues with the current situation in the same way.

The issue isn’t AS important for regular iterators as opposed to async iterators in my view, as in general these are wrappers around custom collections and the possibility of raising an exception is smaller. And if it is generally in those cases you would want to stop the iteration.

However if you are using an async iterator then you are almost certainly doing some IO, as that is the main use case of async and IO can always error. And in the case of it being a stream representing events, sometimes the IO operation can fail but you still want to continue the iteration.

Eg

  • I have some async request coroutines hitting sequential endpoints on a webserver. If one of them fails, I want to keep going.
  • I have some message queue listening to some topic, and it gets disconnected. It raises exception, but I still want to process the stream if it reconnects again.
1 Like

I think that helper class is quite ugly and cumbersome for a user to use. That class must be customised for every type of behaviour you want, and even if you provide a callback you still can’t do some basic things (like break out of the loop).
For example sometimes if there is an exception I will want to continue, other times conditionally break out of the loop, return early etc.
If I was going to go that way I would just provide the sentinel value instead.
My async iterator is in the context of a library, and its much nicer for the user to do:

async for result in MyIterator():
    if result is not None:
        print(result)
    else:
      # Do whatever 
       ...

Than to create a separate class for each type of behaviour they want. Surely you would agree?

The reason that I made my original post is that I thought that using sentinel values like this are quite unpythonic, you would do something like this in Rust by setting the iteration item to a Result<>, but you would have to use sentinel objects in Python as it doesn’t have sum types.

However given that its pythonic to handle errors using exceptions its unfortunate you have to use this style here.

I don’t assume that adding the new syntax to the language is “cheap”, but in this case it would be warranted in my view as it removes the need to do such ugly workarounds I put in my post, and it would maintain backwards compatibility, be technically straightforward to implement, and not cause any grammar ambiguities. This would be in line with other syntactic changes which save similar (or sometimes less lines of code)

It is still the exact same suggestion, with almost the exact same syntax. It would have been better if you posted your usecase in that thread.

1 Like

I would argue the contrary.

try followed by with so often happens. And contextmanagers are really useful to perform automatic closing / cleaning up without you having to remember doing all the closing / cleaning up in finally. Which is why I proposed having a try with syntactic sugar. There is totally no change in how try and with works, so it’s pure syntactic sugar and the parsing involves only a small change to the try..except..else..finally EBNF.

On the other hand, I really can’t see myself using try outside of the for block if I expect an iteration is possible to raise an arrer but I still want to continue iteration. What’s wrong with doing it this way:

errs = []
async for i in async_iterable:
    try:
        do_things_with(i)
    except Exception as e:
        print(f"Whoopsie got <{e}> processing {i}")
        errs.append(e)

do_something_when_done()

if errs:
    do_something_with_errors(errs)

This sounds like you have leaked error handling concerns from your iterator into your calling code. If you want iteration to recover from errors, that should be the responsibility of the iterator.

This sounds like maybe you’ve coupled too much into your iterator? Split out “iterate through endpoints” and “contact server” into an iterator and a coroutine, and now it’s easy to try/catch/continue.

There is also a third refactoring option: return a coroutine from your iterator.

These are simple changes that obviate the need for new syntax and arguably make the code better (subjective but so is this discussion). They follow a simple rule: don’t raise errors mid-iteration that you expect the client to recover from and continue iterating. Most iterator errors leave the iterator dead, so this is a great rule to follow, as users can legitimately expect this to be true, and be very surprised and unhappy at an API that breaks the rule.

1 Like

Your try with proposal saves one extra line and one level of nesting. That is not worth extra syntax in my opinion. My syntax proposal is not the same as this because try for has different behaviour because it gives opportunity to continue the iteration. Your example won’t work because the error is uncatchable if it is raised in the __anext__ method of async_iterable. You could move the async code into the loop but then you are defeating the point of making an asynchronous iterator abstraction in the first place. It was probably async because it was doing IO operations which I want to catch the error for.

The reason I am using an async iterator is to provide an interface for my users so they don’t have to worry about the details, so forcing them to split out the logic is also undesirable to me.

My current approach and one which is still the best as of now, though not optimal is using sentinel values to indicate errors, I would consider my first while loop to be the second best way.

If you come up with something better (keeping in mind I want to encapsulate the asynchronous logic) I would be interested to see that.

You are not achieving that. You are forcing them to understand which errors your iterator can recover from and which it can’t, not to mention even knowing that you’re breaking a cardinal rule of iterators in the first place. Refactoring your API is the right solution, not trying to change the language to fit your model.

3 Likes

I don’t know what your source is for saying that iterators should not continue is a cardinal rule. I would agree that normal iterators shouldn’t generally do this, but when dealing with async iterators that’s often what you want to do, as in general they are dealing with io devices that can fail.

I want my users to be able to do whatever they want with the exception, log it, append it to a list or whatever, so “refactoring” my api like this would not work for me.

Two of the three refactoring suggestions I gave would let users do this, so I can only assume you didn’t read them.

Give me the code sample for my initial example in the original post, because I don’t know what you mean.

I disagree vehemently with the syntax, though.

For every Python statement, it’s a given that a loop happens inside an indented block. When code exits that indented block then the loop already finishes, the block won’t be revisited (unless the looping statement is itself part of a looping block of a higher-level statement)

Your syntax totally ran against that concept as the code exits the indented block, does something elsewhere, then reverts back to the the try async for line and reenters the previous indented block.

No other existing Python statement works like this.

The only way to maintain this principle is to make it like this:

while True:
    try async for result in MyIterator():
        print(result)
    else:
        break
    except:
        pass
    finally:
       print("Always prints!")

But that means try async for has been reduced to ‘merely’ a syntactic sugar of this:

while True:
    try:
        async for result in MyIterator():
            print(result)
        else:
            break
    except:
        pass
    finally:
       print("Always prints!")

And if you don’t like try with because it’s “mere syntactic sugar”, then you certainly won’t like try async for as yet another “mere syntactic sugar” :stuck_out_tongue:

The last time I encountered this problem, I implemented a solution which felt fairly pythonic and may suit your needs.


import asyncio
import random

class IOError(Exception):
    pass

async def io(x):
    if random.random() > 0.5:
        raise IOError
    else:
        return x    

class MyIterator:
    
    def __init__(self):
        self.n = 0
        
    async def __anext__(self):
        try:
            return await io(self.n)
        except Exception as e:
            return e
        finally:
            self.n += 1
        if self.n > 10:
            raise StopAsyncIteration
        
    def __aiter__(self):
        return self

Which can then be used like this:


async for result in MyIterator():
    if isinstance(result, IOError):
        # log or handle IO error
    elif isintance(result, Exception):
        # handle unexpected errors
    else:
        print(result)

Or more simply:


async for result in MyIterator():
    if not isintance(result, Exception):
        print(result)

P. S. apologies for any typos or formatting errors, I’m writing this on mobile

1 Like