Currently, if we have a return value in a generator definition, it’s cumbersome to be able to retrieve that value. We won’t be able to directly use for loop, because the StopIteration exception is getting swallowed.
For example:
def generator():
for i in range(3):
yield i
return -1
To retrieve the -1, we need to do this:
itr = generator()
while True:
try:
value = next(itr)
print(value)
except StopIteration as e:
return_value = e.value
print(return_value)
break
# This will print:
# 0
# 1
# 2
# -1
This approach looks very unnatural, but I don’t think there is a better way without boilerplate codes.
Can we extend the syntax for for-else loop to achieve the same thing like this:
for value in generator():
print(value)
else as return_value:
print(return_value)
This is much more pythonic and fully backward-compatible, and it doesn’t introduce new builtin keywords. This is a very simple extension to the existing for-else loop, which swallows the StopIteration under the hood. Instead of swallowing it, it makes more sense to have a mechanism to expose the value.
This is one way that you could do it by making an iterator that wraps the other one and stores the value:
class LoopVal:
def __init__(self, iterable):
self.iterable = iterable
def __iter__(self):
self.value = yield from self.iterable
def generator():
for i in range(3):
yield i
return -1
vals = LoopVal(generator())
for val in vals:
print(val)
print(vals.value)
# prints 0, 1, 2, -1
def loopval(iterable):
yield lambda: result
result = yield from iterable
def generator():
for i in range(3):
yield i
return -1
vals = loopval(generator())
result = next(vals)
for val in vals:
print(val)
print(result())
# prints 0, 1, 2, -1
This is useful in many scenarios. For example, I’m using generator function to process incoming data stream. When the data stream terminates, the generator will return a metadata dict that contains the special events happened during the process. This metadata dict will be used by the main program to perform subsequent actions.
Regardless of the actual use case, the point is that things should exist for a reason. The thing is Python does support return statement in a generator function. Then it doesn’t make sense that the return value is not treated as a first-class citizen, and being hidden from the surface. If we figured that it’s pointless to use return statement in a generator function, then it should be well removed completely from the language.
Concrete example from real code where I would expose this if it were convenient in any real way: A parser with a function that yields out the indivdual tokens that have been parsed, for e.g. progress reports or debugging functionality (or more special cased invasive operations), which also returning the proper parsing result. Currently it’s just an attribute on the parser object, but if there was a good way to access the return value, I would use that as well.
def loopval(iterable, retrn=None):
(retrn or [None])[0] = yield from iterable
def generator():
for i in range(3):
yield i
return -1
for val in loopval(generator(), retrn := [None]):
print(val)
print(retrn[0])
# prints 0, 1, 2, -1
2 line function
1 line less for using compared to proposal
So re-using this more than 2 times ends up being less boilerplate than proposal.
I can see how it would be used, although it’s not obvious to me that this makes for better code than an alternative design for that use case[1].
To me, a big issue for this syntax is that it makes some kinds of for-loops different from others. If I refactor returns_a_generator() to returns_an_iterable() then my loop no longer has valid syntax (or if it does, what should it do?), but I don’t think there would be a way for the parser to figure that out, so it’s an error at runtime.
maybe a real code example would be more convincing though ↩︎
No, it’s fully backward compatible, because python already supports for-else loop. The problem is that the existing for-else loop does’t capture the value of StopIteration but throw that value away. Your concern won’t be a problem because every iterable in Python also has a return value in it’s StopIteration exception. The only difference is that the return value captured by StopIteration is default to be None.
The point is exactly that I’m doing everything progressively, because the input data stream can be infinitly long, but I have to yield the processed item immediately to save them to the database. The return value will only be used when the data stream finishes.
I don’t think there is a more natural alternative design than this. The essence of data processing and ETL is just “map” and “reduce”. If we consider normal function as a “reduce” process, a generator as a “map” process, what if we need to do both at the same time? The generator in python fits naturally to this purpose as it supports both yield and return.
What does else as do when it’s not handling a generator?
for value in returns_an_iterable():
print(value)
else as return_value:
print(return_value) # what is return_value?
This is the proposed new syntax. Either this use is valid syntax that hasn’t been addressed in the proposal thus far, or it’s invalid.
I didn’t say it’s not backwards compatible, rather it’s new syntax. You’re introducing for / else as for the special case of a for loop over a generator. I’m asking about all the other for loops.
Yes, the parsing happens progressively, and can in fact be feed in by arbitrary code, most notably the code inside the for loop.
The same thing yield from returns when passed in a normal iterator: None. This doesn’t have to be called out explicitly, since it is the obvious extension of already existing behavior.
As I already mentioned, that “as” captures the value from StopIteration. If you loop through a normal iterable with next() it also gives you a StopIteration exception in the end. If you want to know what’s inside the e.value, please give it a try by yourself.
