Don’t you mean
new_list.extend([1], [2])?
I think he explicitly means new_list.extend(1, 2), so a signature of
...
def extend(self, *args: T) -> None: ...
Its obvious that 1 and 2 are not lsits, and they should form one. But for simplicity i think using some new_list.append(*args) would work better, as extend is used to merge lists, whilst append adds to lists.
Very well written PEP. Maybe a little too verbose, but that’s not necessarily a bad thing. This feature doesn’t let you do anything that wasn’t possible before, it just lets you express old things in a slightly more compact and efficient form. I’m not thrilled about Python’s syntax getting bigger, but at least this feature organically complements the existing syntax.
I am concerned about the semantic difference between synchronous and asynchronous generator expressions. The difference is evident when using close()/aclose(), send()/asend(), throw()/athrow() on the resulting generator. Do you have a specific use case for this? Wouldn’t it be better not to use yield from in both cases for consistency? Or simulate yield from in asynchronous generators (by using send() instead of __next__ for iterating the innermost loop)?
Thanks for the feedback, and sorry for the confusion! The intention for ...x... there was indeed any valid expression (starred or not). For example, when we say that [...x... for x in it] is equivalent to:
out = []
for x in it:
out.extend([...x...])
What we are trying to say is that these two are equivalent no matter what you replace ...x... with, regardless of whether it is a regular expression or a starred expression. I’m going to be making another pass through that section based on some earlier feedback anyway, so I’ll see if I can improve the wording of that part as well.
Interesting point! My gut feeling is that I would expect it to be rare that the distinction would be impactful, but I do think it’s worth thinking through carefully. I can see several options being reasonable, depending on one’s viewpoint. I’ll briefly share my thoughts on the few options that come to my mind:
Leave the proposal as-is, accepting a difference between synchronous and asynchronous generators. There is precedent from PEP 525 (specifically the section on Asynchronous yield from) for things being different there already, so I’m not sure how jarring it would be for them also to be different (in the same way) in comprehensions. I lean this way mostly because I feel like it preserves equivalence with what are probably the most common ways to phrase these things currently.
Adjust the proposal to avoid yield from in both. In addition to making unpacking in regular and async generators more similar, this has the benefit of matching the itertools.chain examples more closely, since (iirc) chain doesn’t delegate to subgenerators.
Adjust the proposal to require (something like) yield from in both cases so that both delegate to their subgenerators. I expect that this would be substantially more difficult, and I also feel like going this route would want to be part of a bigger change actually allowing yield from in async generators more generally. So I do not think I would want to go this way.
I suppose we could also leave that question as an implementation detail rather than part of the proposal; but I’m not a big fan of that because if we do go that way, someone will start to depend on that implementation detail anyway and then be reasonably upset when it goes away.
I still personally lean toward option 1 (leaving the proposal as-is), but I’m not really opposed to option 2 at all (neither form delegates); my own feelings on this point are not strong right now. I’ll think on this some more, but I’d certainly be curious to hear what other people think!
I made some small changes to the “How to Teach This” section to try to improve it based on the feedback here (if you feel like your concerns weren’t addressed, let me know!).
Updated PEP is still at the same location: PEP 798 – Unpacking in Comprehensions | peps.python.org
Diff is available here: PEP 798: Fixes and Clarifications from Discourse Thread (#4518) · python/peps@b1facf0 · GitHub
I didn’t yet make any substantial changes related to the discussion about semantics for async generator expressions because I’d like to gather some more feedback/thoughts/suggestions on that front if possible.
Very supportive of this and it’s definitely something that feels like it should be a part of how you’d expect the language to work.
My only complaint is that I’d like to see it go further. Unless I’ve missed something, this seems to introduce an ability to apply an if condition or a function to the iterables and unpack them in a more concise way. But doesn’t help as much with the conciseness of applying an if condition or function to the underlying elements of the iterable. i.e., I can now do the following replacements:
old: [x for y in z for x in y if f(y)]
new: [*y for y in z if f(y)]
old: [x for y in z for x in y*2]
new:[*(y*2) for y in z] (are the brackets necessary here?)
but there’s not much more succinctness when operating on the sub-elements:
old: [x for y in z for x in y if f(x)]
new: [*[x for x in y if f(x)] for y in z]
old: [x*2 for y in z for x in y]
new: [*[x*2 for x in y] for y in z]
I’d definitely argue that the new versions are more readable, and that in itself is a reason to support this PEP; however while we’re making these changes, would it make sense to extend the unpacking syntax so that following an in, it results in un-nesting? i.e.:
old: [x for y in z for x in y if f(x)]
new: [x for x in *z if f(x)]
old: [x*2 for y in z for x in y]
new: [x*2 for x in *z]
IMO, no. I would read for x in *z as equivalent to for x in (*z,) which is exactly wrong. What you are describing a bit too magically. If it were added, it also should be added to for loops outside of comprehensions, and is therefore out-of-scope for this PEP. If you want to argue for that, make a seperate thread.
A slightly uninformed opinion: Since currently yield from doesn’t exists for async generator function we shouldn’t use it for async generator comprehensions either. However, if it gets added we should be using the same semantics. That leaves two (three) options:
yield from is added. Yes, that might be a breaking change but it’s IMO fine if it’s well documented.* syntax for async generator comprehensions unless/till yield from is implemented. It’s annoying to not keep parity with sync generator comprehensions, but IMO this difference in semantics can be enough of a justification.yield from in async generator in a separate PEP with the same target python version. This topic has come up before IIRC and there is some interest in it)Of these (A) is my preference.
I agree with the sentiment here, and I think it’s worth trying to think about ways to extend this idea a little bit to make this easier.
That said, I don’t think the specific syntax you’re suggesting here quite hits the mark for me, for the same reasons that @MegaIng mentions. The *z in [x for x in *z if f(x)] reads to me like unpacking z itself, with no indication that it’s actually intended to unpack the elements within, so I don’t think this extension quite works without fundamentally changing the meaning of *.
I’m not sure that there’s going to be a natural extension of the current syntax that gives us this functionality without that downside. One alternative, though, could be to add something like Javascript’s flatMap to, e.g., functools. That wouldn’t give us a nice Pythonic syntax, but it would give us the functionality.
The brackets are not necessary here, to stay consistent with the fact that parentheses are not necessary in [*y*2] currently. But I personally think they should be used (and I agree with @NeilGirdhar’s proposed Ruff rule to that effect).
Ah, good! I agree completely that if we do go this route, we should add an explicit note that, in the event that async generators do get support for yield from in the future, this structure’s semantics should change to use it.
All of your suggested extensions seem to go too far, to me. It’s hard to be sure with constructs like this (nothing is obvious until you’ve learned it and got used to it) but I really struggle to see why the “new” versions are better than the “old” ones. Conciseness is far from the most important consideration here, IMO.
I think the biggest remaining question here is about how * unpacking should behave for synchronous versus async generator expressions (current proposal is here in the draft). I wanted to open this back up since there’s some nuance to it and I’m still curious for more opinions if I can gather them, so here’s a poll that I think reflects the main options (but feel free to follow up if I’ve missed something).
Regardless of the outcome here, I’m planning on adding a note stating that this question should be revisited if and when async generator expressions gain support for yield from; so the question is really about what the behavior should be in the meantime).
yield from, async uses an explicit loop)yield from (both sync and async use explicit loops)yield from (sync uses yield from, we implement an equivalent to yield from for the async case)yield fromAside from this question (and probably adding something to “Rejected Alternative Proposals” depending on where this ends up), I’m not anticipating a lot of change in the PEP from its current form based on the feedback so far; but if there are other points that people want to push back on, more feedback is still welcome about other aspects of the PEP as well 
.
Although I voted for “Make both sync and async use something like yield from (sync uses yield from, we implement an equivalent to yield from for the async case)”, I’d be happy with the option of leaving with existing semantics and waiting for a future generic implementation of yield from in async generator functions and update this PEP’s implementation later once that change comes about.
Sorry for a long silence here! Just wanted to post a quick update to say that I’ve edited PEP 798 to include some discussion about the yield from-versus-explicit-loop question for generator expressions that use unpacking, in addition to a few wording/grammar tweaks throughout the PEP (and thanks to @Jelle for feedback and suggestions along the way!).
The main changes since the last draft are the addition of two new sections: Alternative Generator Expression Semantics and Appendix: Semantics of Generator Delegation.
Additional feedback is still very much welcome if anyone has more thoughts
Just popping in with another quick update for anyone interested: as of a few minutes ago, PEP 798 has been submitted to the Steering Council
A few comments.
* not only unpacks, but also transforms the map call into an implicit flat map call.map built-in. Breaking the parity between map(f, xs) and (f(x) for x in xs)) shouldn’t be done lightly.While the PEP outlines the overlap with existing methods, this example is to punctuate the entropy.
import itertools
f = lambda n: list(range(n))
x0 = f(3)
x1 = f(4)
x2 = f(2)
xs = [x0, x1, x2]
# Many, many different ways to do the same thing.
# Several of these have generator versions and can be
# used with the walrus operator.
y0 = [*x0, *x1, *x2] # I don't really see this usage in the wild
y1 = list(itertools.chain(*xs))
y2 = list(itertools.chain.from_iterable(xs))
y3 = [x_ for x in xs for x_ in x]
y4 = [*x for x in xs] # new way of doing it
y5 = []
for x in xs:
y5.extend(x)
Regardless of the contrived aspects of this example, perhaps the grammar/language shouldn’t have evolved to enable this?
xs = [(1, ("a", "b")), (2, ("c", "d"))]
z0 = [(k, v) for k, *v in xs]
z1 = [(k, *v) for k, v in xs]
z2 = [(k, *v) for k, *v in xs]
z3 = [*(k, v) for k, *v in xs] # new
z4 = [*(k, *v) for k, *v in xs] # new
I get the spirit and motivation of the PEP, however, it’s hard to reconcile with the entropy that’s being added and the disconnect between map and (*x for x in xs) comprehensions.
Just my two cents: In my view PEP 798 fails one of the most fundamental Python principles: “Explicit is better than implicit.” The proposed syntax [*it for it in its] isn’t actually clear to me and in my opinion it’s cryptically terse, replacing straightforward iteration with punctuation that obscures what’s actually happening. When you read [x for it in its for x in it], the logic is completely explicit: “for each container in my collection, iterate through that container’s elements.” But [*it for it in its] hides that two-level iteration behind a star operator, forcing readers to mentally unpack what the unpacking means. This is precisely the kind of “clever” terseness that makes code harder to maintain.
The real problem is that this PEP prioritizes brevity over clarity. Yes, itertools.chain.from_iterable(its) is verbose, but it’s also utterly unambiguous: you know exactly what operation is being performed. The double-loop comprehension might be longer, but it’s pedagogically superior: beginners can see the nested iteration structure explicitly rather than having to understand that * in a comprehension context means “flatten one level.”
Python’s strength has always been readability, not conciseness. This proposal trades explicit, understandable code for terse symbols that must be decoded. It asks every Python programmer to learn yet another context-dependent meaning for *, creating more mental overhead every time someone reads code using this syntax. The few characters saved simply aren’t worth sacrificing the explicitness that makes Python accessible and maintainable.
Rationale
[…]
This proposal was motivated in part by a written exam in a Python programming class, where several students used the notation (specifically thesetversion) in their solutions, assuming that it already existed in Python. This suggests that the notation is intuitive, even to beginners. By contrast, the existing syntax[x for it in its for x in it]is one that students often get wrong, the natural impulse for many students being to reverse the order of theforclauses.
Hi,
The Python Steering Council has begun deliberating on PEP 798 today, and we expect to decide by next week. We’re still evaluating several aspects of the proposal and gathering additional information.
One point we’d like to raise is the framing of the new syntax. The PEP currently presents unpacking in comprehensions as offering “much clearer” code compared to existing alternatives (such as itertools.chain.from_iterable, explicit loops, or nested comprehensions). We view readability claims like this as inherently subjective: what reads clearly to one programmer may not to another. We would recommend not to position this syntax as “more readable” or suggest people should adopt it for clarity reasons. We believe the stronger, more objective argument is syntactic consistency as extending Python’s existing unpacking patterns naturally into comprehensions.
We’re still considering other aspects of the proposal and will have more information to share next week, but we wanted to update you as soon as possible, as we know waiting with no news can be frustrating.
Thanks for your patience and for the thoughtful work on this PEP.
The Python Steering Council
Thanks for the update, @pablogsal!
This makes good sense; thanks for the feedback. I agree that that’s the stronger argument.
Forgive my ignorance of the process here: should I start thinking about ways to adjust the PEP text, or would it be better to leave the current draft alone for now while it’s still being reviewed?