PEP 806: Mixed sync/async context managers with precise async marking

PEPs
1

Hi all - after some earlier discussion, I’m pleased to present PEP 806 for discussion. It’s in most ways a very small proposal, which would make a surprisingly large difference for my code!

Feedback, questions, requests for clarification, etc. all most welcome.

Abstract

Python allows the with and async with statements to handle multiple context managers in a single statement, so long as they are all respectively synchronous or asynchronous. When mixing synchronous and asynchronous context managers, developers must use deeply nested statements or use risky workarounds such as overuse of AsyncExitStack.

We therefore propose to allow with statements to accept both synchronous and asynchronous context managers in a single statement by prefixing individual async context managers with the async keyword.

This change eliminates unnecessary nesting, improves code readability, and improves ergonomics without making async code any less explicit.

For example:

async def process_data():
    async with acquire_lock() as lock:
        with temp_directory() as tmpdir:
            async with connect_to_db(cache=tmpdir) as db:
                with open('config.json', encoding='utf-8') as f:
                    # We're now 16 spaces deep before any actual logic
                    config = json.load(f)
                    await db.execute(config['query'])
                    # ... more processing

becomes

async def process_data():
    with (
        async acquire_lock() as lock,
        temp_directory() as tmpdir,
        async connect_to_db(cache=tmpdir) as db,
        open('config.json', encoding='utf-8') as f,
    ):
        config = json.load(f)
        await db.execute(config['query'])
        # ... more processing
35 Likes
2

I’m a big +1 on this, as I think this would help to improve the readability l. Are the parenthesis necessary however, or could work t be written as a one-liner?

3

The PEP proposes allowing the one-line form, including a simple with async ctx(): without parentheses.

That said, I think it will remain better style to write async with ctx(): for this case, and would only recommend using the new feature for multi-line parenthesized cases.

1 Like
4

The PEP only gives examples of

with (
    async aquire_lock() as l,
    open("file") as f)
):
    ...

Or

with async aquire_lock() as l:
    ...

So, to clarify, would

with async aquire_lock() as l, open("file") as f:
    ...

Be allowed?

1 Like
5

I have a related question: What happens with with async followed by parentheses. Is the code
with async (
foo_context() as foo,
bar_context() as bar ):
legal, and so are both foo and bar asynchronous or only foo.

Joe Gottman

1 Like
6

Thanks for asking! The async keyword is specified in relation to the ast.withitem, and so this would be a SyntaxError.

If you play around with adding parens in the current with statement, you’ll find that the only valid placements are (1) around the entire expression, with ( …. ):, or around individual context-manager-returning expressions like with ctx1() as a, (foo().ctx_attr) as b:. The PEP doesn’t propose changing this, only allowing you to prepend async to each ast.withitem.

1 Like
7

I’m +1 on the PEP!

It’s maybe a little unfortunate the motivating examples use file operations as examples of sync operations. I’ve already seen comments on Reddit to the tune of “hurr durr they should be using async wrappers for files instead“ (while a complex topic, also isn’t incorrect in the general sense). I imagine the main motivating cases are actually the Trio cancel scope managers and their ilk?

Some examples I’ve encountered where I’ve needed to mix sync and async context managers is using our internal library for tracing (with TRACER.span(): ) and the timeout managers in my Quattro library (which are modeled after the Trio ones). But I understand you can’t stick those into a PEP.

Edit: maybe examples using pytest.raises(…) and contextlib.suppress(…) would be clearer.

1 Like
8

I’m a bit skeptical of this proposal because it weakens Python’s principle that async code should be visually explicit. One of the best design decisions with async/await was making asynchronous code unmistakably distinct: when you see async with, you immediately know “this can suspend execution.” The proposal blurs this by allowing with async ctx() which looks like a sync statement with an async detail buried inside. At a glance, you lose the clear signal that this block involves async operations. The current forced nesting actually serves a purpose: it makes you think about the async/sync boundary and makes it visually obvious which scopes are async.

The motivating example showing “16 spaces deep” is already a code smell that should be refactored regardless of async: you shouldn’t have four nested context managers in the first place. If deep nesting is genuinely necessary, the explicitness of separate async with blocks is worth the verbosity because it maintains clarity about where suspension points exist. In my view the ergonomic improvement isn’t worth compromising the visual distinction that makes async code safe and maintainable.

4 Likes
9

I had to give this some significant thought and look at my own use, personal and professional.

I don’t personally see allowing mixing these as a positive after doing so.

I couldn’t find any examples in my own use that even came close to paralleling this, and I do think that this compromises on the primary benefit of async as a keyword in the first place (the high visibility and clear ordering of context switches, note that these context switches also happen on exiting the async context managers)

I’m going to critique the example, heavily, but the same critiques likely apply to any other code that would seem to benefit from this, as I believe there’s a reason I can’t find any code in my own use that would even remotely benefit from this.

The example given seems pretty weak, I’d like to see a realworld example where you actually need to aquire these, in this particular order, all at once, using a mix of sync and async context managers to do it. If the ordering is unimportant, just that all are aquired before continuing, then released afterward, or even if only the aquisition of the lock happening at the outermost level matters, you can already group this into two groups (those that are async, and those that are not).

What’s the lock even for? temp directories should always be sufficiently unique, file access can be in exclusive mode, and dbs typically provide their own locking as needed. If multiple things would be frequently updating the same config file, and this is intentional, you should be using a write queue rather than locked access, or even storing configuration in the database.

Edit: I think that this is essentially already acting as a visual indication of code that should be refactored, not of code that the language needs to accomodate better.

2 Likes
10

A simple question: are you using asyncio.timeout()? If you value the benefit of explicit suspension points (like I do, and you imply you do) you should be using something else instead (either a home-grown solution, or quattro.fail_after()/quattro.move_on_after() and friends) since asyncio.timeout() is an async context manager but without any possible suspension points inside. (Fun fact, I raised this point while working on it with Andrew but we were ultimately overruled.)

And I personally have had need to combine quattro.fail_after() with taskgroups etc, so that would be my concrete example.

1 Like
11

No. If I need a timeout on a coroutine that doesn’t provide it, I don’t use asyncio.timeout, but asyncio.wait_for or asyncio.wait (depending on the cancellation semantics needed in case of timeout) with a timeout parameter. While structured in code differently than the [edit: closest] context manager equivalent, the semantics are in general better.

12

On behalf of the Python Steering Council: We’ve chosen to reject PEP 806, “Mixed sync/async context managers with precise async marking”. We thank Zac for the careful survey and write-up, and we acknowledge that the nesting pattern the PEP describes is a real annoyance for code that mixes sync and async context managers. However, the cost of adding new syntax to Python is very high: a permanent addition to the grammar is paid for by every parser, AST consumer, linter, formatter, type checker, and teaching resource in the ecosystem, and it is carried forward by every Python programmer from here on. The problem this PEP addresses, while genuine, does not in our view meet that bar. The PEP’s own survey suggests adoption would be limited to a specific subset of async code that also uses synchronous context managers, and that narrow payoff is not enough on its own to justify new syntax.

In addition to that background concern, we have several more specific reservations:

  • The proposal introduces two valid spellings with different rules. Under this PEP, async with ctx(): and with async ctx(): would both be valid spellings of an async context manager entry, with different compositional rules: the former disallows mixing, the latter permits it. The PEP explicitly defers style enforcement to linters and formatters, but we are reluctant to ship syntax whose usability depends on external tooling to decide which of two legal forms is idiomatic.

  • We have some concerns about how with async interacts with the existing async marking convention. Today every line performing asynchronous work begins with async or await as its first non-whitespace token, which is a property readers and static analysis tools often rely on. Placing async as an item-level prefix inside a parenthesised with block changes that, and the PEP’s own recommendation to keep async with for single-line statements suggests the authors share some of this concern. On balance we would prefer to preserve the existing convention rather than loosen it for a syntactic convenience.

  • Existing workarounds are adequate for the cases the PEP addresses. contextlib.AsyncExitStack and the as_acm() helper pattern already cover the mixed sync/async case. The PEP’s objection to as_acm() rests on the await sleep(0) cancellation checkpoint breaking the property that syntactic await/async for/async with always yields to the scheduler, but the PEP itself acknowledges that few codebases enforce this property today. We would be receptive to a proposal for a stdlib version of as_acm(), with or without the checkpoint, as a smaller and more targeted change.

None of this is to dismiss the underlying pain point. We would encourage interested parties to continue exploring library-level solutions, and we think the survey data gathered for this PEP would be valuable input into such a discussion. Thanks to Zac for the work, and to everyone who contributed to the discussion.

11 Likes
13

Thank you, Pablo and Steering Council, for your careful stewardship of Python! I also appreciate the writeup of your reservations.

I don’t think an as_acm() proposal is yet mature enough to propose for the stdlib or even third-party frameworks, but I’ll now spend some time experimenting with library-level solutions, and aim to write up any generally-useful observations or suggestions.

1 Like
14

Just a nitpick since there are plenty of other reasons given, but this isn’t true. await can be used in expressions like blah = await foo.

3 Likes