Expected behavior:
>>> def pack_b():
while True:
L = [(yield) for i in range(2)]
print(L)
>>> pb = pack_b()
>>> next(pb)
>>> pb.send(1)
>>> pb.send(2)
[1, 2]
>>>
2 Likes
I’ve made a demo: GitHub - yunline/cpython at comp-yield
If this idea is good, I’m glad to open a pull request.
It was possible to properly support yield inside comprehension when it was not inlined. It is easy, the compiler only needs to add yield from when call the implicit internal function:
def pack_b():
while True:
def _compr(_arg):
_result = []
for i in _arg:
_result.append(yield)
return _result
L = yield from _compr(iter(range(2)))
print(L)
But it was said not to do this.
It’s not that it was impossible or difficult to implement. It was decided not to implement it for other reasons. You must show that these reasons are no longer valid or that the other reasons are more important.
Thanks for your reply.
I’ve read the issue, but I still don’t know why “raising a SyntaxError” is a better solution than some fix like “add an implicit yield from”.
I’m sorry, could you please point the reason out?
The reason I would give is that there is an expectation that comprehensions are pure and work in “one go”. I feel like adding a yield breaks both expectations, but I don’t feel particularly strongly about it.
I am rather sympathetic to this idea (I implemented similar idea for await and async for in nested comprehensions). But, it seems, there were no good reasons for its implementation. And it is difficult to do this in generator expressions. What does ((yield) for i in range(2)) even mean?
I think it make sence to forbid yield inside a genexpr.
It’s mentioned in docs:
To avoid interfering with the expected operation of the generator expression itself,
yieldandyield fromexpressions are prohibited in the implicitly defined generator.
In my demo branch, “yield in generator expressions” still raises a SyntaxError.
About reasons:
Expressions like [(yield) for i in range(2)] are not ambiguous. They are straightforward, intuitive and make the comprehension expressions more flexible. Why not add them back? Especially when there is few difficulty on implementing this.
(Btw, I’ve been writing a tool to convert python scripts into oneliner expressions. The implementation of “oneliner yield” is largely determined by whether python allows “yield inside comprehension”. With “yield in comp”, the output code can be largely simplified. This is the direct reason why I want “yield in comp” XD.)
But my intuition is that [(yield) for i in range(2)] should mean exactly the same as list((yield) for i in range(2)). So if yield in a genexpr is forbidden, I’d expect yield in a listexpr to be as well.
To give a concrete example, if I have code that does something like [x*2 for x in range(10)] and I realise I need an immutable value, I’d simply turn that into tuple(x*2 for x in range(10)). If that transformation didn’t always work, I’d find it annoying and confusing. Conceded, I can’t imagine using yield in that situation[1] but consistency and reliable invariants are important.
TBH, I don’t think I’ve ever actually used a yield expression ↩︎
3 Likes
This intuition does not work with asynchronous comprehensions and generators.
[await expr for x in iterable]
and
[expr async for x in iterable]
are working and pretty common code. But
list(await expr for x in iterable)
or
list(expr async for x in iterable)
unfortunately do not work and cannot work in the current Python execution model.
The difference is that it was not yet shown that there is a significant demand of yield in comprehensions. It it was so, there should be many examples of generator functions that create, populate in a loop and return lists, sets or dicts, and which could be easily transformed to comprehensions with yield. But I do not know any of such examples.
And there would need to be some reason that yield from was insufficient for these examples.
I can’t really think of why one would want to create a list comprehension of yields when yield from (genexpr) is possible.
If I’m not wrong, it’s not possible to get the send() value from a genexpr.
So, for most of cases that “yield in comp” used, they couldn’t be rewritten in yield from (genexpr) version because they used the value returned from yield
Here is one example:
def gen0():
l = []
for i in range(10):
l.append((yield i))
print(f"gen0: {l}")
def gen1():
l = [(yield i) for i in range(10)]
print(f"gen1: {l}")
def run(gen):
it = iter(gen)
last_value = next(it)
while 1:
try:
last_value = it.send(str(last_value))
except StopIteration:
break
>>> run(gen0())
gen0: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
>>> run(gen1())
gen1: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
>>>
Okay, but why would you want to write such code? gen0 seems much more reasonable if you actually want do something with the sent values rather than just collect them–and if you just want to collect them, why send them?
Maybe there’s a realistic example where this would be useful?
One specific example:
def non_recursion(gen):
def _non_recursion(*args):
stack = [gen(*args)]
converted = None
while stack:
try:
sub_args = stack[-1].send(converted)
stack.append(gen(*sub_args))
converted = None
except StopIteration as err:
converted = err.value
stack.pop()
return converted
return _non_recursion
I wrote a non_recursion decorator. Add the decorator to the recursive function, repace the self-calls with yield expressions and then we got a non-recursion function with the same functionality as before.
def reverse_list_1(_list):
return [reverse_list_1(sub) if isinstance(sub, list) else sub for sub in reversed(_list)]
@non_recursion
def reverse_list_2(_list):
return [(yield [sub]) if isinstance(sub, list) else sub for sub in reversed(_list)]
test_list = [1, 2, [6, 5, 4, [9, 8, [10, 11, 12, 13], 6]], 4]
print(reverse_list_1(test_list))
print(reverse_list_2(test_list))
# both print [4, [[6, [13, 12, 11, 10], 8, 9], 4, 5, 6], 2, 1]
In this specific case (reverse_list), without “yield in comp”, it’s not possible to convert the function in such “low cost” way. We have to rewrite the comprehension as a for loop since “yield in comp” is forbidden.
Don’t show us the code which you could write if “yield” be supported in comprehensions. Show an existing code in the form:
def gen(...):
result = []
for ...:
result.append(... yield ...)
return result
If there is enough number of such examples in the real world code, it can be an argument for implementing this feature.
Okay, I’m trying to search on github and get a list of links of code.
Btw, I think pattern like for ...: result.append(... yield from ...) should be included too.
Cases of append(...(yield)...) (59 in total)
https://github.com/saghul/asyncio-redis/blob/fd0ac6f267c1884dd8d43517d0aa2bc85dcb11f6/tests.py#L832
Cases of append(...(yield from)...) (19 in total)
https://github.com/rshorey/moxie/blob/0fac956c04d7054af4a435a0d383a99e158be8ec/moxie/app.py#L54
https://github.com/sixstars/xcat/blob/1e568d36bb9540069601459abaaa95a6cdcfce08/xcat/xcat.py#L384
https://github.com/wijnen/supernovel/blob/4a38ec9cc6d67397322a19f8471936e09927df04/supernovel#L189
I got these data by manually searching on github. Forks and duplicates are not counted. Because of the limitation of manual search, the real numbe of cases should be way larger than the number I got.
yield_cases.txt
↑ Here I’ve got another 504 cases.
Regular expression pattern:
r"for .* in .*:(([\n].*){0,10}).*\.(add|append)\(.*yield (from )*.*\)"