Generators and Yield

Hi community

I have run this code in the python visualizer to understand the code better, and find that in step 6 & 7, it skips over the multiplication right after yield, moves back to line 7, then in this sequence, executes the multiplication.
My question is, what is the reason for it skipping over multiplication the first time, but not the rest of the code till the iteration completes?

def powersOf2(n):
    pow = 1
    for i in range(n):
        yield pow
        pow *= 2

t = list(powersOf2(3))

print(t)

It doesn’t skip it, it just hasn’t reached line 7 yet. There are no special rules for generators, so at the first time going into the for loop, it yields the current version of pow. As it has not hit the pow *= 2 yet, that is still 1. After the list function has consumed the value, it “returns” into the generator. The next statement in the generator is pow *= 2, so only then it executes that for the first time, loops and yields the next value, being 2.

3 Likes

I think it is better to see what we are talking about. Here are the lines printed as they are executed. I used the standard trace module:

$ python3 -m trace -t powers_gen.py
 --- modulename: powers_gen, funcname: <module>
powers_gen.py(1): def powersOf2(n):
powers_gen.py(7): t = list(powersOf2(3))
 --- modulename: powers_gen, funcname: powersOf2
powers_gen.py(2):     pow = 1
powers_gen.py(3):     for i in range(n):
powers_gen.py(4):         yield pow
 --- modulename: powers_gen, funcname: powersOf2
powers_gen.py(5):         pow *= 2
powers_gen.py(3):     for i in range(n):
powers_gen.py(4):         yield pow
 --- modulename: powers_gen, funcname: powersOf2
powers_gen.py(5):         pow *= 2
powers_gen.py(3):     for i in range(n):
powers_gen.py(4):         yield pow
 --- modulename: powers_gen, funcname: powersOf2
powers_gen.py(5):         pow *= 2
powers_gen.py(3):     for i in range(n):
powers_gen.py(9): print(t)
[1, 2, 4]

When you call a generator function you get an iterator. From that iterator you can request individual values using next() calls. Calling list(iterator) calls next(iterator) repeatedly for you. The iterator keeps the executional frame (the state of execution) of the generator function. So the executional state of the generator function between the individual next() calls is not forgotten.

The command yield provides a single value from the iterator and the execution of the program goes out of the frame of the generator function powersOf2() back to the frame which called next(). It is similar to the return statement with a big difference that after yield the executional frame of the generator function is not destroyed. When next value is requested from the iterator, the execution continues in the existing frame from the statement following the yield statement.

That is the reason why in the trace you always see --- modulename: powers_gen, funcname: powersOf2 between the statements yield pow and pow *= 2. This shows that the frame of powersOf2 is being entered repeatedly.

3 Likes

We can see the executional flow better when we unroll the list creation:

def powersOf2(n):
    pow = 1
    for i in range(n):
        yield pow
        pow *= 2

iterator = powersOf2(3)
t = []
t.append(next(iterator))    # 1st value
t.append(next(iterator))    # 2nd value
t.append(next(iterator))    # 3rd value

print(t)
$ python3 -m trace -t powers_gen_unrolled.py
 --- modulename: powers_gen_unrolled, funcname: <module>
powers_gen_unrolled.py(1): def powersOf2(n):
powers_gen_unrolled.py(7): iterator = powersOf2(3)
powers_gen_unrolled.py(8): t = []
powers_gen_unrolled.py(9): t.append(next(iterator))    # 1st value
 --- modulename: powers_gen_unrolled, funcname: powersOf2
powers_gen_unrolled.py(2):     pow = 1
powers_gen_unrolled.py(3):     for i in range(n):
powers_gen_unrolled.py(4):         yield pow
powers_gen_unrolled.py(10): t.append(next(iterator))    # 2nd value
 --- modulename: powers_gen_unrolled, funcname: powersOf2
powers_gen_unrolled.py(5):         pow *= 2
powers_gen_unrolled.py(3):     for i in range(n):
powers_gen_unrolled.py(4):         yield pow
powers_gen_unrolled.py(11): t.append(next(iterator))    # 3rd value
 --- modulename: powers_gen_unrolled, funcname: powersOf2
powers_gen_unrolled.py(5):         pow *= 2
powers_gen_unrolled.py(3):     for i in range(n):
powers_gen_unrolled.py(4):         yield pow
powers_gen_unrolled.py(13): print(t)
[1, 2, 4]

Notice that calling a generator function (powersOf2(3)) does not enter its frame because it just creates an iterator.

1 Like