Exception Handling With Function Generators

Hello,

I am currently studying the topics of Generator Functions and Generator Expressions.
As some of you may know, generators yield a value once per call using either the next(x) or x.__next__() built-ins as opposed to returning all of the results all at once of an iterable.

The following is the original generator function test code (thereafter I attempt to add exception
handling to the test code to catch the StopIteration exception when attempting to step passed the final
iterable value - for test purposes):

def both(N):
    yield from range(N)
    yield from (x **2 for x in range(N))

result = both(4) # 
next(result) # Repeat this line until all values haven been yielded 

I want to include exception handling so that I ‘catch’ the StopIteration exception so that it does
not crash the script. So, I tried the following two methods:

def both(N):
    try:
        yield from range(N)
        yield from (x **2 for x in range(N))
    except StopIteration:
        print('All values have been processed.')

and

def both(N):
    try:
        yield from range(N)
    except StopIteration:
        print('All numbers have been processed.')
    try:
        yield from (x **2 for x in range(N))
    except StopIteration:
        print('All values have been processed.')

When the two attempts above did not work (exception handling did not catch the exception when stepping through passed the last iterable), I tried the following way:

while True:
    try:
        next(a)
    except StopIteration:
        print('All numbers have been processed.')
        break

This final method did catch the StopIteration exception. However, with this final attempt, all of the values are yielded all at once due to the while statement. How can I wrap exception handling to catch the StopIteration such that I step through the yield values one by one?

Any help with clarifying this would be welcome.

Your both returns a generator. The exception is raised by this object’s __next__ method.
The method is read-only. You could wrap that object in your own generator which defines its own __next__ method. This, can have the try-catch inside.

Do you mean having to wrap every next(x) with a try/except pair?
I am trying to avoid that.

I mean something like this

def both(N):
    yield from range(N)
    yield from (x ** 2 for x in range(N))


class Wrapper:
    def __init__(self, foo) -> None:
        self.my_foo = foo

    def __next__(self):
        try:
            _value = next(self.my_foo)
            return _value
        except StopIteration:
            return 'All numbers have been processed.'

# We want to call the __next__ from Wrapper,
#     instead of the __next__ from `both(4)`.
# The latter can raise the StopIteration exception.

new_result = Wrapper(both(4))
for i in range(10):
    print(next(new_result)) 

Perhaps you would like to, instead of return "All numbers have been processed" you want to print it and re-raise the StopIteration exception. That way someone using new_result as an interator knows when it has finised.

1 Like

Hi,

thank you. But towards the bottom of your code, you use a for loop statement whereby all of the values are printed automatically. In practical terms, this is not much different than my while loop example since all of the results are obtained in one instance. I am looking for a method whereby the user manually steps through each iteration without using any type of iterable loop.

For example (though this way does not work either):

new_result = Wrapper(both(4))
next(new_result)
next(new_result)
next(new_result)
next(new_result)

I put the for there for illustration only. I put an iteration from 0 to 9 to go beyond the eight values that both(4) would normally yield.

Your

new_result = Wrapper(both(4))
next(new_result)
next(new_result)
next(new_result)
next(new_result)

works as well. Nothing is being done with the return values of each next(new_result).
So, running this as a script doesn’t print anything. On a Python interpreter maybe you see their outputs in the terminal. Otherwise do print(next(new_result)).
Four of those would print

0
1
2
3
1 Like

Aahhh, that pesky print statement strikes again! :blush:

Yes, it does work as advertised. I had copied your code on to a module and ran it.
I was expecting the same result as if running it on the IDLE shell interpreter whereby
a next(x) statement call would suffice and the yield value would be displayed.

Well, thank you for your time and help.

p.s.
This is a lot more complicated than I had anticipated just for being able to catch
a StopIteration exception on a function generator.

Your class seems complicated / slow, could just use another generator:

def wrapper(foo):
    yield from foo
    while True:
        yield 'All numbers have been processed.'

Or with itertools:

def wrapper(foo):
    return chain(foo, repeat('All numbers have been processed.'))
2 Likes

Assumption: the objective is to be able to process a sequence of values, produced by a generator - where the generator is actually creating two sequences. ie (using the Python-REPL):

>>> def both(N):
...     yield from range(N)
...     yield from (x **2 for x in range(N))
... 
>>> for one in both( 3 ): print( one, end=" ", )
... 
0 1 2 0 1 4 

Per experimentation, yield from will handle its own end-of-data scenario. So, there is no need to manually try-except within the generator.

The place to handle EOD is where the generator is “consumed”.

The next thing to note is that the word “generator” can be confusing! In this case, both() is a “generator-function”. Before it can be consumed (in this scenario!), a fresh(!) generator-instance is required. Thus:

>>> generator_instance = both( 3 )
>>> while True:
...     print( next( generator_instance ), end=" ", )
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
StopIteration
0 1 2 0 1 4 

and the need to ‘wrap’ the next if the StopIteration exception is to be handled constructively! (per own solution, above)

As you can see, per @Stefan2 , there is no need for something more complicated.

Ref the two ‘time to think’ exclamation marks (above):

1 Because for-statements will happily accept both iterators and iterables, and will silence the StopIteration there is no need to manually ‘convert’ the generator-function into an instance (see debug-prints above, serving to prove that the appropriate sequence is being produced)

2 A generator-instance may only be consumed once. Once StopIteration occurs the instance is done/gone/finished. Should the sequence be required again, a new instance must be created. (will leave that to your experimentation, if required/of-interest…)

How do you make the double underscores appear? I tried but they were truncated in my original post. Here, when I selected your dunder, the double underscores were also truncated.

Try quoting that again but select not just the __next__ but include the word before and the word after, then you’ll see.

See earlier answer comparing and contrasting the use of a generator-function within a for-statement, and the use of a generator-instance and next().

So, it all depends on the individual use-case - and how whether it is important to catch the end-case and recognise same somehow.

As an alternative you could use next(a, None) form to return None (or anything else) when the iterator is exhausted, instead of raising an exception.

1 Like

These posts are formatted using the “Markdown” mark-up ‘language’ (more research for you!)

You’ve already figured-out how to enclose (“fence”) code. The rest is similar. For now it might be easier to highlight the text (click and drag) and use the formatting bar (above, when composing) and the " (quotation marks) or the </> (pre-formatted text) functions.

To write code inline enclose it in backticks `.

That’s the ticket! Much obliged sir.

Hello,

thank you for your time in responding to my query. Yes, I am aware that the for and while statements automatically ‘take care’ of the StopIteration exception as per theory. However, my objective was to capture the generated exception when performing the next(x) and/or x__next()__ calls manually.

Thank you! Tested the code.

Works like a charm. :grinning:

def both(N):
    yield from range(N)
    yield from (x ** 2 for x in range(N))

def wrapper(foo):
    yield from foo
    while True:
        yield 'All numbers have been processed.'

q = both(3)
a = wrapper(q)
next(a)  # Repeat until end of iterables and string is printed
for identifier in generator:
    ... do something with identifier
print( "all processed" )