Yield comprehension

I would like to propose a small change about the definition (but not call) of a function.

Consider the yield expression

yield from (<expression> for object_ in iterable)

for instance, where <expression> is any expression. It’s such that the yield statement consisting of it is equivalent to

for object_ in iterable:
    yield <expression>

It feels natural to expect that the yield expression could more simply be written as

yield <expression> for object_ in iterable

so making this indeed a valid expression inside the definition of a function seems helpful. (It raises SyntaxError as of Python 3.9.)

What do you think?

The simplified syntax would in general be for an expression to yield from a generator displayed by a generator expression (comprehension), e.g.,

yield <expression>\
    for object_ in iterable if <condition> if <condition'> ...\
        for object_1 in iterable1 if <condition1> if ...\

where iterable1 can instead be something that depends on object_, and so on.

I think it would be bad to have yield x for x in [] be different from yield (x for x in []). The latter already has a meaning.


Thank you for your comments.

I understand your concern is that the parentheses can be forgotten from

yield (x for x in [])

Actually, I had already thought about that. I thought that should be extremely rare since the parentheses in this case must be chosen from among brackets, braces and parentheses.

Indeed, forgotten parentheses around comprehension are already accepted to be undetectable. Note that the expression

{x for x in [] for y in iterable}

is allowed even though it’s meaning is different from

{(x for x in []) for y in iterable}

The difference of the meanings for those parentheses doesn’t seem to have been enough to exclude the former expression.


yield <expression> for object_ in iterable

is an expression, what is the value of this expression?

If it is a statement, how does it differ from the following statement and what are its large advantages?

for object_ in iterable: yield <expression>

Thank you for your interests!

yield <expression> for object_ in iterable

should be an expression, and it can just be equivalent to

yield from (<expression> for object_ in iterable)

I don’t think this can wait for a sent value, and its value seems to be None. Please note I’m only proposing a new syntax, and not a new behaviour of a generator function/iterator. I’m not proposing something like

(yield <expression>) for object_ in iterable


Thus, the expression doesn’t always have to be a statement by itself, but I’m suggesting the syntax not because the statement consisting of the expression has a large or even any advantage over

for object_ in iterable: yield <expression>

The reason for which I suggest it is that I think

yield <expression> for object_ in iterable

can look reasonably like a valid expression, and does make good sense (at least to me), so I think a programmer may easily get believing it is valid. This won’t happen if she firmly knows it isn’t valid and will never be, but it would be annoying if you sometimes happen to be believing it’s valid and write such expressions at places in your code to later get SyntaxError at every one of those places. So, if the expression just could be valid, then I think the language would be more generous to the user if the expression actually was valid.

The validity of the expression feels like a natural consequence for the language already having comprehensions and yield expressions, and from a user’s point of view, its current invalidity seems to be an unnecessary restriction of the language for no user’s benefit.

What’s the advantage over the yield from version?

As a side note, I would be curious how this interacts with the changes in PEP 709 to inline comprehensions.

Thank you for your interests!

I’m not sure if it has an advantage over the yield from version either, at least if I assume that the meanings will be the same. My motivation was as described in my previous post.

Since that expression is a little simpler than the yield from version, I’m afraid it might be what you keep writing when you’re not sure that it’s invalid, until you come up with the yield from version. If this may happen to some people, then I don’t see a good reason to leave it invalid and keep raising SyntaxError for that.

This may sound more like a complaint than a proposal, but I don’t know if it’s a bug.

Interactions with the inlined comprehensions sounds interesting.

  1. If the expression is treated just as identical to the yield from version of it, then I don’t expect any interaction.

  2. If some other object than the generator (<expression> for object_ in iterable) is to be constructed at some point instead, then there may be going to be a function to construct that object, which could be inlined.

However, if execution of the statement for object_ in iterable: yield <expression> doesn’t have to create any object…

  1. If no object is to be constructed there, then no function therefore to inline?

I now think the advantage of

yield <expression> for object_ in iterable

over the yield from version of it is that it describes the programmer’s intention more directly. You just want to have something yielded for each thing in iterable, and the generator (<expression> for object_ in iterable) is only an auxiliary thing in just one means to achieve that purpose. If the yield from version is really the intention (and for what possible purpose?), then you can write it, but normally, that doesn’t sound like what the programmer is concerned about.

If the value of the expression is always None, why not use literal None? If the value of the expression is not used, it have not be an expression.

Thank you for your comments again!

Your claim sounded like that an expression had no more use than use of it as a statement and use of its value. Thus, we may need to discuss something quite general. An expression expresses a process of computation of its value. It’s difficult to tell if I understood you correctly, but you sounded to me as if saying that, as long as the value would be the same, the process was unimportant unless the expression was used as a statement.

I don’t think I know anything you don’t know about this, but an expression can also be used as a part to form an expression. In that case, the process expressed by a part is important as a part of the process expressed by the expression containing the part.

How important is this use? Well, a lot of programming can be done as construction of expressions. This is a transparent way to construct processes as combinations of subprocesses. It is not always the best way of programming, but having it as an option seems to help sometimes in a concrete and practical manner. Do you see what I mean?

Let me add a little remark that an expression which forms a part of another expression cannot necessarily be replaced by mere its value (supplied in any way). Indeed, the evaluation of an expression may evaluate any part of it any not predetermined number of times (including 0), and any evaluation of any part must be done as a subprocess of the evalution of the whole expression. Therefore, a part may have to be there really as an actual exression of a process, and its value may not be enough.

I can be more concrete on this whole discussion, but I hope things made sense to you.

I don’t think you weren’t familiar with what I wrote, but I will give a simple example.

Let us first note that the value always being None isn’t important. Thus, I’m going to use the expression

(yield <expression> for object_ in iterable)\
    or get_value()

and illustrate the advantage of this to the statement for object_ in iterable: yield <expression> (plus the expression get_value()).

Now an example of a thing you can do with it is to form an expression like

get_some_value() if some_check()\
    else ((yield <expression> for object_ in iterable)
          or get_value())\
          if another_check()\
             else get_other_value() if ...\
	          else ...\

for use in any other expression or a statement. How close (if not better) a thing can you do with the statement for object_ in iterable: yield <expression>?

I can discuss this, but let me just tell my answer. The best thing I know of you can do with it is to do

def f(iterable):
    for object_ in iterable:
        yield <expression>
    return get_value()

and form the expression

get_some_value() if some_check()\
    else (yield from f(iterable))\
         if another_check()\
            else get_other_value() if ...\
	         else ...\

equivalent to the one I gave. Any other thing I know of has some decisive disadvantage to this. In other words, if you don’t have the expression yield <expression> for object_ in iterable, then creating an expression equivalent to it anyway is the best thing you can do, so it would be very clearly easier if you had the expression from the beginning, except in the case where you want to reuse f elsewhere (which won’t always be the case).

Did I miss some better way? Is there an alternative thing which you think will be better?

The most important reason why I think

yield <expression> for object_ in iterable

should be an expression is similar to the reason why I think it would be better to be valid, which I explained in my reply to your first post was that it looks reasonably like a valid and meaningful expression, but is currently invalid, which can lead to hindering the programmer for no good reason. This is also a reason why I think it should be an expression. Namely, an important reason is that it looks like an expression (and I have shown that the usefulness of its value is not an important factor to determine the usefulness of this expression).

‘yield from’ is an expression, so you could replace this yield comprehension with (yield from (<expression for object_ in iterable)) and it would still be able to be daisy-chained. That said, though, I’m hard-pressed to imagine where I’d want this. Do you have a concrete use-case in mind?

To me, that looks rather convoluted and I wouldn’t want to write or read that…


[Edit: an example removed (for misunderstanding it) and put back in previous two editings, third and fourth to add these remarks.]

Sure. I only needed to say that you could’t do as good a thing with a statement there, but I don’t give yield from (<expression> for object_ in iterable) priority over the proposed version for the points I made in my replies to your previous post anyway. I forgot to mention that there. Thank you for pointing that out!

That said, can you imagine one may want to use an expression version of the statement for object_ in iterable: yield <expression>? I thought the arguments in my replies to Serhiy’s previous post had essentially shown that an expression version of almost any idiom for a statement could be an attractive option for use. It’s not difficult to find more examples similar to the one I gave there. (I’d like the both expression versions of for object_ in iterable: yield <expression> to be valid, and would usually prefer the currently invalid one.)

A typical place where you may also want an expression version is within a lambda expression.

In addition to parts of an expression, the syntax of some statements require some parts of the statement to be an expression. For instance,

while (yield from (<expression> for object_ in list_))\
      or test():

allows you to have things yielded before every test.

[quote added in edition]

Thank you Stefan for your interests!

What makes it look so to you? The structure of the expression is very simple. Is it the layout? It looked fine on my editor with highlighting and alignment.

If it’s the structure, what may be your alternative for an equivalent code?

I think it’s both the structure and the layout. And the or hack.

I remember reading it and having to change my understanding when I arrived at your if another_check()\, which appears after the two things it leads to (the yielding loop and the hacked-on get_value()) and is in my opinion too far indented (I’d rather expect the if at the same indentation as the previous else, not at the same indentation as the or, which makes it looks like it’s even inside the parentheses along with the yielding loop and the or-added expression).

This appended if two lines down and that far indented made me realize only at that point that the previous else is really rather an elif. With the if part “hidden” later/deeper. This reminded me of “Sentences should be readable from left to right without ambiguity” from Knuth et al.'s “Mathematical Writing” - I don’t like when I have to rethink the first half of a long sentence/expression because of something in the middle, I prefer to read and understand it “from left to right”.

So with something this long, I’d prefer the if/elifs/else structure to be clearly visible at first sight. I’d write it like this:

if some_check():
elif another_check():
    for object_ in iterable:
        yield <expression>
elif ...:
else ...:

Thank you so much for your detailed comments. They were extremely helpful!

I just noticed what you wrote as a replacement was not an expression but a statement. You are throwing away all the values you may get with get_some_value(), get_value() etc. (My expression gets the last one computed of those values as its own value.) It would be great if you could show me how you may deal with this problem, since there seems to me to be something non-trivial about it.