Is an iterable that raises an exception "exhausted"?

Python Help
1

For context, I’ve read PEP 234, and a few questions here and on StackOverflow which are variants on this theme, but…

If calling next(iterable) raises an exception, is the iterator now exhausted? It clearly is if StopIteration (or StopAsyncIteration) is raised, but what about exceptions more generally?

I guess the question has two parts:

  1. Is it legal for an iterator that raises not to be exhausted?
  2. Is it pythonic for it not to be exhausted?

Obviously a generator will inevitably become exhausted if it raises, but not every iterator is a generator.

I looked at itertools to see how it behaves in the face of iterators that raise:

import itertools

def collect(iterator):
    result = []
    while True:
        try:
            for item in iterator:
                result.append(item)
            else:
                return result
        except Exception as e:
            result.append(e)

class RaisingIterator:
    def __init__(self):
        self.i = 0
    def __iter__(self):
        return self
    def __next__(self):
        try:
            if self.i == 5 or self.i == 15:
                raise RuntimeError
            elif self.i >= 20:
                raise StopIteration
            else:
                return self.i
        finally:
            self.i += 1

iterator1 = RaisingIterator()

grouped = itertools.groupby(iterator1, key = lambda i: i//10)
for k,g in grouped:
    print(f'group {k=!r}: g->{collect(g)!r}')

# group k=0: g->[0, 1, 2, 3, 4, RuntimeError(), 6, 7, 8, 9]
# group k=1: g->[10, 11, 12, 13, 14, RuntimeError(), 16, 17, 18, 19]

iterator2 = RaisingIterator()

for _ in range(4):
    s = itertools.islice(iterator2, 12)
    print(f'islice: s->{collect(s)!r}')

# islice: s->[0, 1, 2, 3, 4, RuntimeError()]
# islice: s->[6, 7, 8, 9, 10, 11, 12, 13, 14, RuntimeError()]
# islice: s->[16, 17, 18, 19]
# islice: s->[]

So if one uses groupby, the groups it generates are not exhausted by the source iterator raising an exception, but if one uses islice the slices are.

I’m writing some utilities that manipulate (asynchronous, as it happens) iterators, and how to handle an iterator throwing is a thorn in my side. Should I just relax and say any behaviour is reasonable provided I neither crash nor silently ignore the exception?

2

I’d tend to say that this is the right answer, but it depends on context. In a lot of cases I’d expect an error to stop processing.

One small note is that if your case requires that an iterator can produce an error and keep going, I’d strongly consider yielding errors rather raising them. That way, the user is very clearly intentionally giving your consumer an error to process as part of the data.

3

I’ve considered that. And, while it could make sense as an option, I’m very wary of it as a default for very much the same reason it was decided in the iteration protocol to raise StopIteration instead of returning a sentinel value.

Worse, if I decide that iteration can continue after an exception there’s the very grave risk of the exception being silently overlooked and passed out through several levels of code to something that has no clue how to handle it.

4

In many applications, I won’t say most but definitely many, that’s exactly the correct behavior. And high level orchestration code may catch all errors and do something sensible (like logging) with them.

For truly generic code, naive to it’s surrounding context, you should not catch and squash errors. I assume you have a more specialized case.

You’ll need to document whatever contract your utility has with the iterables/generators it consumes. The starting assumption should be that errors are not caught, it if they are that they are reraised. So just document deviations from that default – e.g. that the utility will log a warning but continue processing if a ValueError is raised.

5

IMO doing any implicit exception handling besides StopIteration in itertool-like functions is a bad practice. Exceptions should be handled by a generator/iterator wrapping an exception-raising iterator.