Supporting 'yield from' in asynchronous generators

Ideas
1

Hi,

I’m proposing a few new changes to support yield from in asynchronous generators. Namely:

  1. Allowing an asynchronous generator to use yield from to delegate to a synchronous subgenerator.
  2. Adding an async yield from statement to delegate to an asynchronous subgenerator.
  3. Allowing non-None return values in asynchronous generators for when async yield from is an expression.

Here’s an example to visualize:

def sync_generator():
    yield 1

async def async_generator():
    yield 2
    return 3

async def main():
    yield from sync_generator()
    result = async yield from async_generator()
    assert result == 3

Variations of this idea have come up on here a few times before. The main pushback I’ve seen is that the implementation would be too complex. Following some of my own frustration with a lack of support for yield from in async generators, I was able to come up with a reference implementation that isn’t particularly maintenance-heavy. (The majority of the diff comes from the regenerated parser and interpreter.)

I have a PEP draft on this; I suggest reading that for more details.

Thanks!

12 Likes
2

Anything that brings async and sync syntax close together has my support. The lack of yield from for async code has always been one of those little annoyances that forces me to stop and think for a second.

If this does get added, I’m sure there will be a lot of people who just end up using it accidentally without knowing about this pep since yield from is such a common pattern.

2 Likes
3

I’m not fully sold on a specific need for capturing return values with async generators[1], but behaviorally, this is what I would expect out of full parity here, and there’s a hard to quantify benefit to improving consistency when it comes to people learning and teaching, as well as writing, reading, and maintaining code to not have to spend an extra moment thinking “is this allowed here?”

This may be a good time for those interested in the type system to re-evaluate how aync def is handled in type stubs, as the preexisting issue here where typecheckers rely on being able to see the function body to determine the type of functions defined with async def is going to become a larger problem. (Not something I would expect as a blocker to for this)

  1. I’m sure someone can find a reasonable use for it, but I’ve found it generally better to handle code that looks like it would want that with a context manager + an async iterator, or just with synchronous generators. ↩︎

4

Yes please. Everytime I’ve encountered this I’ve been annoyed. The spelling is also as natural as it gets and something I’ve reached out to accidentally only to be disappointed. Thank you for working on this.

5

Why do you need async yield from? Do you really write the code like the following (this is a very simplified example) all the time?

while True:
    try:
        y = yield x
    except StopAsyncIteration:
        break
    except Exception as e:
        await a.athrow(e)
        break
    else:
        x = await a.asend(y)

Or you simply want to write

async for x in a: yield x

in one line?

1 Like
6

I’ve never come across this in actual usage, but orthogonality does have its appeal. +0.

7

Yes, that’s the primary use case, and I think the same is true for yield from.

The issue is that a loop isn’t equivalent to async yield from, because subgenerator delegation semantics are lost. That’s the whole reason yield from was added in the first place.

2 Likes
8

I’ve run into it a few times when batching things like requests:

import asyncio
from itertools import batched

# Something that handles a number of request futures
async def dispatch_requests(*requests):
    ...

# The thing that I always reach for before remembering you can't
# yield from an async generator 
async def batch_requests(*requests, batch_size):
    for batch in batched(requests, batch_size):
        yield from dispatch_requests(*batch)

# The way I end up writing it
async def old_batch_requests(*requests, batch_size):
    for batch in batched(requests, batch_size):
        async for resp in dispatch_requests(*batch):
            yield await resp

async def main():
    return await asyncio.gather(
        *batch_requests(..., 10), 
        return_exceptions=True
    )

I also do not use async code that often, so this toy example probably has some problems in it, but that sort of pattern is something I’ve run into a lot. Especially since a lot of my sync code is reliant on chains of generators that all yield from one another to form data pipelines.

4 Likes
9

PEP discussion thread is now open: PEP 828: Supporting 'yield from' in asynchronous generators