Why is yield not in the operator precedence table? To me it seems similar to lambda, which is included. For example, lambda is shown as having lower precedence than and, so you know that lambda: 3 and 4 means lambda: (3 and 4), not (lambda: 3) and 4. Wouldn’t it equally make sense to include yield, so you know that yield 3 and 4 means yield (3 and 4), not (yield 3) and 4? What’s the difference why lambda is included and yield isn’t?
Might have simply been overlooked.
Looking at the atom definition earlier on, it seems to me that yield expressions belong in the same precedence bucket as “parenthesized expression, list display, dictionary display, set display”, i.e. the top tightest-binding row in the current table.
That’s yield_atom, which is a yield expression in parentheses.
yield isn’t really an operator in any significant sense. yield expressions can’t be nested within a larger expression; they are defined as expressions so that they can be used as the left-hand side of an assignment statement (so that the argument to a send method can be captured inside a generator function).
I guess this this means that in the parsing sense, they bind very weakly? In some sense, syntactically they do seem to behave like expressions (and operators): For x = yield 1 + 1, it seems sensible to ask if it parses as yield (1 + 1) or (yield 1) + 1, even if the latter one is perhaps not allowed.
1 Like
From the relevant PEP:
A yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment.
Due to that, they were never explicitly listed with their own precedence.
I do think it makes sense to add them, though, as within those required parentheses (or when appearing on the right hand side of an assignment statement) they bind more weakly than everything else (so the expression being yielded never needs its own set of parentheses unless it’s something like a generator expression or another yield expression).
1 Like
But we don’t ask if return 1 + 1 is the same as (return 1) + 1 or return (1 + 1), because we don’t think of return as an operator. In this sense, yield still has more in common with return than any operator.
If you look at the low end of the precedence table, you see lambda and :=, which despite being very loosely binding, they still have one thing in common that yield does not share: you can embed lambda and := arbitrarily deep in an “ordinary” expression (I also almost said “regular” expression, but man, would that open a can of worms 
). A yield “expression” is just fundamentally different from other kinds of expressions.
3 Likes
Note that the only way yield forms part of a larger expression (in which precedence would be needed) as a primary expression, in the form of a yield_atom. In this sense, it is already included in the table, the first row, the parentheses evaluate first. Lambdas do not have this requirement when forming part of larger expressions.
The other way that yield appears in expressions is as part of the yield_expression, in which it is already defined to be atop the expression or expression_list. The primary expressions that might appear inside it got evaluated first.
In particular yield 3 and 4 would only be read as being a yield_expression, in which therefore 3 and 4 is part of an expression_list. It cannot be anything else, since the other option would have to be a yield_atom and parenthesis would need to be there somewhere.
Although the order is implicitly defined, being explicit and saying something in the table wouldn’t hurt.
2 Likes
I believe that’s because return does not evaluate to a value (i.e. it’s a statement) while yield (in some contexts) does (it’s an expression). The extension to the language where it is usable in a more complex expression is also at least naively imaginable.
1 Like
The other expression level keywords missing from the table are the component parts of comprehensions (for, in, if) . I think they’re just above lambda, but they might be below assignment expressions (they’re definitely not below yield).
I think they are, implicitly, since when they appear infix in expressions, they do so as list_display, set_display, generator_expression. which are enclosed in brackets, braces or parentheses. These are in the first row of the table.
This isn’t the direction where we’re suggesting precedence table entries might be useful. Instead, it would be to address the questions below.
While these aren’t technically about operator precedence (since you get compilation errors when you omit the parentheses rather than things happening in the wrong order), it’s still a useful way to document the syntactic restrictions.
(yield EXPR_TO_YIELD)
When does EXPR_TO_YIELD need to be parenthesesised to make the following yield expression work properly?
The answer turns out to be “When EXPR_TO_YIELD is an assignment expression”, since yield and return follow the prohibition on disallowing assignment expressions outside parentheses when they’re on the right hand side of another assignment statement. lambda is allowed without parentheses:
>>> def g():
... return (yield lambda: 1)
...
>>> def g():
... return (yield x := 1)
File "<stdin>", line 2
return (yield x := 1)
^^
SyntaxError: invalid syntax
>>> def g():
... return (yield (x := 1))
...
This means yield expressions would not in fact be the last row in the table. Instead, they would go above assignment expressions (but below lambda expressions).
(EXPR_ITEM for NAME in EXPR_ITERABLE if EXPR_CONDITION)
When do EXPR_ITEM, EXPR_ITERABLE, and EXPR_CONDITION need to be parenthesised?
The rules here are slightly complicated by the fact that assignment expressions are prohibited outright as part of EXPR_ITERABLE:
>>> [a:=1 for x in b:=[2] if c:=3]
File "<stdin>", line 1
[a:=1 for x in b:=[2] if c:=3]
^^
SyntaxError: invalid syntax
>>> [a:=1 for x in (b:=[2]) if c:=3]
File "<stdin>", line 1
[a:=1 for x in (b:=[2]) if c:=3]
^^
SyntaxError: invalid syntax
>>> [a:=1 for x in (b:=[2]) if (c:=3)]
File "<stdin>", line 1
SyntaxError: assignment expression cannot be used in a comprehension iterable expression
Correcting for that shows that EXPR_ITEM doesn’t need parentheses (so the leading for has lower precedence than assignment expressions), but EXPR_CONDITIONAL does (suggesting it is considered higher precedence):
>>> [a:=1 for x in [2] if c:=3]
File "<stdin>", line 1
[a:=1 for x in [2] if c:=3]
^^
SyntaxError: invalid syntax
>>> [a:=1 for x in [2] if (c:=3)]
[1]
lambda is similarly higher precedence than the for (no parentheses needed in EXPR_ITEM), but lower precedence than the other two keywords:
>>> [lambda:1 for x in lambda:[2] if lambda:3]
File "<stdin>", line 1
[lambda:1 for x in lambda:[2] if lambda:3]
^^^^^^
SyntaxError: invalid syntax
>>> [lambda:1 for x in (lambda:[2]) if lambda:3]
File "<stdin>", line 1
[lambda:1 for x in (lambda:[2]) if lambda:3]
^^^^^^
SyntaxError: invalid syntax
>>> [lambda:1 for x in (lambda:[2]) if (lambda:3)]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'function' object is not iterable
>>> [lambda:1 for x in [2] if (lambda:3)]
[<function <lambda> at 0x7f534dce4cc0>]
The pattern repeats on the next step up the table with conditional expressions (allowed without parentheses as EXPR_ITEM, but otherwise parentheses are needed):
>>> [1 if 1 else 0 for x in [2] if 2 else [] if 3 if 3 else 0]
File "<stdin>", line 1
[1 if 1 else 0 for x in [2] if 2 else [] if 3 if 3 else 0]
^^^^
SyntaxError: invalid syntax
>>> [1 if 1 else 0 for x in ([2] if 2 else []) if 3 if 3 else 0]
File "<stdin>", line 1
[1 if 1 else 0 for x in ([2] if 2 else []) if 3 if 3 else 0]
^^^^
SyntaxError: invalid syntax
>>> [1 if 1 else 0 for x in ([2] if 2 else []) if (3 if 3 else 0)]
[1]
It’s only when we reach or that we find the parentheses are optional for all three expressions:
>>> [1 or 0 for x in [2] or [] if 3 or 0]
[1]
Accordingly, the comprehension related expression keywords could usefully have two entries on the operator precedence table:
- one between
orandif-elsefor the iterable and condition expressions - one below assignment expressions for the item expressions
1 Like
Seems to me like a lot of responsibility for a little table on operator precedence to also have to decide valid syntax. It is even possible to cover all of valid syntax as a set of precedence rules?
It might be better job for some notes and examples in the section, but outside of the table. They could cover some examples and tell the reader “these get decided by these other rules, that made them invalid syntax” and point to the relevant sections.
When the question to be answered is of the form “When do these expressions require parentheses?”, an operator precedence table is a pretty decent format to use. Specifying whether omitting the parentheses gives you a compilation error or a wrong result isn’t the table’s job, but it can still capture the point where the parentheses are required rather than making folks figure that out for themselves.
(Specifying when even parenthesized forms are disallowed would also be out of scope for this table)