Hi friends! Just wanted to say that the Steering Council has appointed @yselivanov as PEP delegate for PEP 828 and he has accepted. Given that Yury authored the original async/await and async generator PEPs, he’s well-positioned to make the call here. He will make a recommendation to the SC as soon as he finish deliberating.
Thanks you all for your patience!
If return values aren’t added then pluggy will be stuck using clutches for async
I think it’s a little more complicated than that, because to ensure timely cleanup you generally need to write with closing(gen()) as g: yield from g (or async variant). Unfortunately the [async] yield from expression in particular makes it very attractive to write the more concise but often-incorrect inline version instead.
As usual, I want to ensure that Trio users and flake8-async can ensure that every async function transitively yields-or-awaits back into the framework, so that timeouts can fire etc. Either the current proposal of distinguished async yield from and yield from expressions, or banning yield from <sync gen>-in-async-gen (lightly preferred, as I also have not seen uses for this), would work for me; but a yield from accepting either sync or async generators would not.
I’ve tried the reference implementation but I’m not sure the exception handling (athrow) is working right. Or maybe it is working but that indicates a mismatch between sync and async generators. Or maybe I’m missing something?
This version (all async) works:
"""async-test"""
import asyncio
async def get_nums():
try:
yield 1
except EOFError:
print("Don't care about EOF")
yield 2
yield 3
print("get_nums finished")
async def my_gen():
async yield from get_nums()
print("my_gen done")
async def amain():
g = my_gen()
print(await anext(g))
print(await g.athrow(EOFError()))
print(await anext(g))
try:
print(await anext(g))
except StopAsyncIteration:
print("fin")
asyncio.run(amain())
This version (mixed sync async) doesn’t work:
"""sync-test"""
import asyncio
def get_nums():
try:
yield 1
except EOFError:
print("Don't care about EOF")
yield 2
yield 3
print("get_nums finished")
async def my_gen():
yield from get_nums()
print("my_gen done")
async def amain():
g = my_gen()
print(await anext(g))
print(await g.athrow(EOFError())) # RuntimeError: Task got bad yield: 2
print(await anext(g))
try:
print(await anext(g))
except StopAsyncIteration:
print("fin")
asyncio.run(amain())
I agree with
I’m a big fan of PEP 380 (yield from) in synchronous contexts. I’m glad it pushed back against the criticisms of allowing return values. I discovered yield from in the process of writing an incremental parser a couple of years ago. I was revisiting it a couple of days ago, and long story short I ended up discovering this PEP.
A very stripped down (and for a different task) version of the parser is below.
from typing import Generator, TypeAlias, TypeVar
T = TypeVar("T")
Returner: TypeAlias = Generator[None, None, T]
class IncrementalChunker:
def __init__(self) -> None:
self._results: list[int] = list()
self._buffer: bytes = bytes()
self._generator: Returner[int] = self._parse_forever()
self._wakeup()
def __iter__(self) -> Generator[int]:
while self._results:
yield self._results.pop(0)
def feed(self, more_bytes: bytes) -> None:
self._buffer += more_bytes
self._wakeup()
def _wakeup(self) -> None:
self._generator.send(None)
def _await_n_bytes(self, n: int) -> Returner[bytes]:
while len(self._buffer) < n:
yield
ret = self._buffer[:n]
self._buffer = self._buffer[n:]
return ret
def _parse_forever(self) -> Returner[None]:
while True:
parsed = yield from self._parse_single()
self._results.append(parsed)
def _parse_single(self) -> Returner[bytes]:
b_1 = yield from self._await_n_bytes(1)
b_2 = yield from self._await_n_bytes(1)
# could've used `_await_n_bytes(2)` but I'm
# demonstrating a point of how we can wait
return b_1 + b_2
decoder = IncrementalChunker()
decoder.feed(b"Hello")
for t in decoder:
print("Round 1", t)
# Round 1 b'He'
# Round 1 b'll'
decoder.feed(b"")
for t in decoder:
print("Round 2", t)
decoder.feed(b" ")
for t in decoder:
print("Round 3", t)
# Round 3 b'o '
decoder.feed(b" World!")
for t in decoder:
print("Round 4", t)
# Round 4 b' W'
# Round 4 b'or'
# Round 4 b'ld'
I’ll post later ( 
sorry I’m really busy with a project rn ) about trying to mimic asyncio (at-least the basics of PEP-3156 based on PEP 380 but pre PEP-492) using yield from, and trying to implement async generators as a subsequence of the yields. Just to see if it’s possible to spell out the yield from and/or async yield from formal semantics similar to PEP 380 in py code if possible, rather than browsing any intense reference implementation in C, and comparing yield from (sync) vs await (async).
It looks like this case isn’t covered by our test suite, so I must have missed it. The bug is likely just a missing _PyAsyncGenValueWrapperNew call somewhere.
I have a branch where I started this, but I haven’t got around to finishing it yet. I think it’s less important for this PEP because the semantics are supposed to be exactly the same as PEP 380, so any differences should be immediately flagged as bugs.
For the async iteration case, it’s important to spell out exactly where the yields are versus the awaits, and that’s not something PEP 380 covers.
Synchronous iteration is literally identical, so that can refer back to the existing behaviour (and highlight that as part of the rationale for requiring different syntax to request async iteration).
Just an update: I discussed this heavily at PyCon US and there’s a few changes that need to be made here.
In particular, Yury was able to convince me that delegation to synchronous subgenerators was a bad idea, because there’s a weird translation layer that doesn’t work well in practice. For example:
async def agen():
async with asyncio.timeout(1):
yield from syncgen() # asyncio.TimeoutError can be injected into a *synchronous* generator!
That leaves the question of what syntax should be used for delegating to an async generator: should it be yield from or async yield from? I’m currently torn on this point.
Here’s a summary of the arguments:
yield from to mean delegation to async from async would mean that we couldn’t ever add synchronous delegation in the future, though some argue that it will never happen anyway.yield from would break the convention that asynchronous context switches are explicitly prefixed with async.async yield from is verbose. As Pablo puts it, this would be the first “triple keyword trifecta” in Python.async yield from would break symmetry with yield in async generators, though yield doesn’t implicitly invoke any awaits.I’m currently leaning towards async yield from, but I have also accepted that I will be unhappy with either decision.
IMO the first two points are strong reasons to use async yield from, despite the verbosity. It’s wordy, but it’s built logically.
My brain is happy with this symmetry and would like it to be preserved; a theoretical async yield X would mean that yielding that thing could wait (which it can’t, but it’s at least logically sane).
TBH I think this is the strongest argument:
This is the biggest benefit of async/await over threads: you can see syntactically where every context switch can happen. It should be possible to quickly search for them all by eyeball or grep. The special case of a three-part word isn’t special enough to break the rule “if there’s a context switch, it has async”.
I’m not sure I find the “sync iterators aren’t expecting async timeout and cancellation exceptions” argument compelling (since resource cleanup doesn’t care about exception details), but if we do go with async-only delegation, perhaps it might make sense to spell it await from?
It’s not as explicit as async yield from, but I think the brevity is a genuine improvement.
That spelling also emphasises the connection between the new delegation expression and the return values of async generators.
Edit: on further reflection, I think that spelling works even if synchronous delegation is added later.
I’ll have to mull this one over. When discussing this in person, async from and some variations of it were also suggested and weren’t received too well.
The main issue is that it seems pretty important to have yield somewhere in the syntax, and it’s hard to spell that out without being verbose. We might just have to live with the verbosity, or go with plain yield from.
The thought was mostly born from looking at async yield from and thinking “of those keywords, which feels the most redundant”, and concluding it was the yield.
The from implies the iteration, the async indicates that we expect a coroutine rather than a synchronous iterable. The yield does emphasise that the iteration body yields values, but that is already weakly implied by the from.
The step from async from to await from was then a matter of mentally reframing the delegation structure as “an await that can also yield additional values during its processing” rather than “an async for loop that can also return a value”.
It’s definitely not a perfect resolution, and I’d personally be equally fine with async yield from, but as a more concise spelling I’d vastly prefer it over making the meaning of a plain yield from context dependent.
Te feature needs to be possible*, not necessarily convenient.
I sprefer the explanation that “async yield from” is to “ yield from” as “async yield” is to “yield”, and shortening breaks that rhyme.
I agree with Guido on this. In the days of autocomplete and snippets (‘ayf’ expanding to ‘async yield from’), I wouldn’t worry too much about shortening. Clarity and understanding are more important to usage and teaching.
Thanks @ZeroIntensity for thoughtfully moving this forward.
This is pretty much what I’d expect the yield from to do. The synchronous generator should deal with the exception in the same way it deals with any unknown exception.
I’ll probably want synchronous delegation at some point (if the PEP is accepted).
A sync gen → async gen transformation should be doable in user space, so I’m fine with disallowing sync delegation this for now. But I’d like if the door was kept open here.
So, I’m for async yield from. It also plays well with recent SC opinion:
Today every line performing asynchronous work [uses]
asyncorawait[…], which is a property readers and static analysis tools often rely on.
I might need that if I’ll be writing async yield from ensure_asyncgen(syncgen) :)
But I don’t agree with the general idea. Each special tool or setup that we take for granted makes the language harder to use.
At the risk of bike shed painting, I’m curious if yield async from was considered? It associates the keyword more closely with the implicit loop of the from .
The SC isn’t too fond of that, as far as I can tell:
I updated the PEP to only specify async yield from. I also added a lot of commentary about the decision to use async yield from and why to disallow sync subgen delegation in async generators.
Perhaps we also want to update the PEP title now?
A few months ago the Steering Council asked me to be the PEP-delegate for this PEP and I accepted. Since then I’ve been thinking about the proposed semantics & the syntax, talking to more than a few Python core developers including Peter, and to a few Python developers whose opinions I trust.
I propose to accept the PEP with the caveat of changing the syntax to yield from (away from async yield from).
The initial version of the PEP proposed allowing both yield from and async yield from. While it seemed attractive on the surface, I convinced Peter that it’s not something we should pursue. If we were to allow that, it would be possible for an asynchronous call await generator.asend(...) to suspend in a synchronous frame of a synchronous generator, which would make things quite confusing to the user. While implementing this is possible in principle, the costs are going to be quite high and our generators machinery is already one of the gnarliest parts of the interpreter.
With the option of delegating to synchronous generators gone we’re left with just one key decision to make: yield from async_gen() or async yield from async_gen(). Guido loosely compared this to deciding what 0^0 should be: it really depends on the point of view.
People who advocate for async yield from want explicitness:
yield from has just one meaning and only one base type for rhsasync with as a counterpart for with, async for for for, async if for ifasync yield from as a counterpart for yield fromI’d like to challenge the above, but bear in mind, that this isn’t precise science. It ultimately boils down to taste.
Regarding 1 – lack of clear signal that this is asynchronous yield from. We already have a loud signal to the user: the code is either nested inside def or async def. One might then argue “but what if the function body is long and you only see part of it” – well, then the reader still has an opportunity to be confused: the visible part of the code might not contain a single await or yield, so there could be no indication that this piece of code is inside a coroutine function or inside a generator or inside an asynchronous generator. Since a single yield can change the meaning of a multi-hundred-long function completely, I argue that yield from does not need to have async prefix to be more explicit in asynchronous generators.
Regarding 2 – one meaning for yield from. We already compile and work with yield very differently depending on when it’s inside a synchronous or an asynchronous generator. I don’t see why yield from is much more special than yield.
As for different accepted type for the operand of yield from – we’ll have different code paths for async vs. sync implementation anyway. And for the user – with the help of IDEs, type checkers, linters, and increasingly more capable LLMs this argument is moot.
Regarding 3 – ‘async yield from’ is a counterpart for ‘yield from’. I get this one (and all of the above, fwiw!). But but the verbosity of async yield from, the inconvenience of typing it on a keyboard, the space it will take in a language where we don’t encourage soft line breaks, all of these arguments make yield from much more compelling.
We do not have async yield, we have just yield. It’s the signature of the function that signals what this body of code is and how it can be called (nobody is surprised that you can’t list(agen()).
I argue that the same exact reasoning can be applied to yield from. For the user it will be just as clear, that inside async def the argument of yield from must be an asynchronous generator (or iterator, more generally).
I strongly recommend we go with yield from. After years of writing async/await code myself and observing others, I’m confident that the extra verbosity of async prefix is not needed.
Sad.
I would have liked to, in some future, be able to turn:
def foo():
yield from range(5)
sleep(1)
yield from range(5)
into
async def foo():
yield from range(5)
await aiomodule.sleep(1)
yield from range(5)
The PEP has 13- and 20-line wrappers for this that it calls trivial. (I beg to disagree with that characterization.) It then says:
In addition, there seemed to be much less demand for [delegating to synchronous generators] compared to support for asynchronous delegation, so solving these issues is less of a priority for now.
That doesn’t make sense. Using the obvious syntax for this feature for something else means we effectively can’t add it in the future. We’re closing the door here.
And ironically, the main stated reason for this limitation is implementation complexity – the same thing that prevented delegation in PEP 525 a decade ago.
Hi Petr,
I understand the use case you want, but it’s just not possible to introduce it without iteration semantics falling apart. If an asynchronous generator is suspended inside a synchronous generator, then await agen.asend() could be driving a synchronous generator, and then that synchronous generator can get errors like asyncio.CancelledError etc thrown into it. Cancellation of tasks and closing generators becomes much trickier to reason about, the boundary between sync and async becomes blurry. This is not a bridge we want to build.
And yes, implementing support for this bridge to work in all edge cases will be extremely tricky. The implementation complexity argument is valid.