PEP 707: A simplified signature for __exit__ and __aexit__

Are you able to put together an implementation of this and make an alternative proposal? There might still be time for 3.12.

We can add a builtin helper function that takes the exit function and an exception and does the introspection and calls exit as the interpreter would.

1 Like

I’m personally more in the “live with the redundancy indefinitely” camp, at least until someone designs and implements an updated context management protocol that meets the goals Serhiy mentioned above:

  • passes the return value from the CM entry function to the exit function (to make re-entrant CMs easier to write without having to rely on closures to store state), with the active exception as an optional second argument
  • allows the CM exit function to indicate that exceptions should be suppressed and return a value (e.g. by returning a non-empty tuple)
  • updates the grammar to allow a 2-tuple of names in the “as” clause of with statements

(Technically, the latter two points could be handled independently from the first, since they don’t affect the method signature - they just tweak the meaning of non-empty tuples when the updated name binding syntax is used)

As Mark pointed out above, we even have __leave__ available as an entirely viable name for the new method.

The derived wrapper function technique can then be used at type construction time to ensure that implementing either version of the CM termination protocol gets you both methods on the resulting class. (at which point, it probably wouldn’t be worth it to deprecate the old method - it could just live on indefinitely, like some of the pickle and copy compatibility APIs)

However, even for proposals that I don’t personally favour, I prefer to see them evaluated in their best possible form, rather than making final decisions based on aspects that are specific to an initial proof of concept implementation (in this case, the fact the introspection could be moved to type definition time means that the runtime impact of the introspection could be minimised even without a specialising interpreter implementation)

1 Like

I don’t think it can. See this comment.

The fact late-binding __exit__ on a class doesn’t have the desired effect is what I was referring to when noting that the PEP as implemented isn’t taking full advantage of the descriptor machinery.

It also wouldn’t be a new limitation, even if the PEP decided not to resolve it.

I don’t follow. It seems to me that late binding does have the desired effect.

First of all, thank you for your great work on getting rid of the exception triples.
(I have a habit of only speaking up when I see issues, which is a fine CLI design principle but not that great with people. I hope I don’t seem too negative!)


I meant changing the semantics between Python code and the compiled version: __exit(arg)__ would behave differently based on whether the compiled version uses METH_O, METH_FASTCALL, or a custom callable type.


While I won’t have time to drive it for 3.12, I spent a few hours on a proof of concept for __leave__: all the ways of setting a type’s __exit__ also set __leave__ to a wrapper that calls that __exit__, and vice versa. The interpreter then only calls __leave__.
I guess the piece that was missing is capturing the callable at __setattr__ time, rather than trying to look it up dynamically.
As expected, it’s probably harder to implement than PEP 707 (touching typeobject.c is never pleasant), but I do think keeping heuristics out of a core protocol is worth it.
A very rough proof of concept is here. It only passes most tests, and leaks a lot, but the concept is there.

2 Likes

The SC rejected this PEP. For simplicity, here’s the message I posted on the issue asking for pronouncement:

We discussed the PEP and have decided to reject it. Our thinking was the magic and risk of potential breakage didn’t warrant the benefits. We are totally supportive, though, of exploring a potential context manager v2 API or __leave__.

1 Like

I will update the PEP.

I hope that one of the other alternatives will make it into 3.13. Since the status quo received only 3% of the vote, it would be a pity if this is the option we will end up choosing.

As far as I can tell, nobody so far has indicated that they are planning to work on the alternative they proposed. If you are non-core-dev looking for a cpython project to take on, perhaps you could help make that happen. You can use this thread to initiate such a collaboration.

The SC does agree that this bears fixing, and if neither __leave__ nor a more thorough redesigned of context managers ends up being practical, we could reconsider this PEP… but the objections to __leave__ so far are not strong enough to choose this alternative.

From my point of view, the problem with this proposal is that it muddies the water just as much as adding a new method, in that the same method can get called multiple ways. It makes the desired end state (a single argument) the special case, and provides no way to migrate to that desired end state (we can’t add deprecation warnings because we don’t know what will or won’t keep working). The fact that it causes changes in how the __exit__ method is called when it, for instance, is wrapped in a seeming innocuous lambda self, *args: f(self, *args) makes it particularly uneasy for me.

2 Likes

I’m not following. What would change here? A signature of (self, *args) would be interpreted as the 3-arg version.

Could we not add a deprecation warning just before calling a 3-arg __exit__?

Understood. TBH, I expected the SC to defer decision on this PEP until we’ve had time to evaluate the other options. I was not aware that a PEP can be reconsidered after being rejected. That’s good to know.

Yes, but the original method might take just one argument. The method would be expecting “new” semantics, and work fine, and then fail to work when it’s wrapped by something else. Or, when the wrapper used to wrap it is changed for other reasons.

Not without a lot of false positives, the same you’re looking to avoid by introducing the magic switching in the first place :slight_smile: All the places that ignore the args or merely pass them along would then still have to change just to silence the deprecation warning.

Everything can always be reconsidered :slight_smile:

1 Like

My main concern is actually not so much to save code changes, but to avoid having to teach every python developer out there that __leave__ is kind of the same thing as __exit__. I think this can create a lot of confusion.

But maybe we can take the idea that @markshannon and @encukou proposed and apply it to the introspection solution, in a way that will avoid the breakage you described with the wrapper.

They proposed that every type that has __leave__ or __exit__ is automatically given the other. We could, instead, wrap the given __exit__ with a new __exit__ that takes *args and does the right thing for all cases: checks whether args is an exception or an exc_info, and then calls the wrapped __exit__ with one or 3 args, depending on introspection result.

If context managers generally don’t use their arguments at all, would it not be easier to offer a zero-argument __exit__() instead, if the exception can still be retrieved from sys.exception?

Look in contextlib. Sometimes you want a different exception than sys.exception().

1 Like

Yep, unfortunately I can’t promise I’ll have time. But I’d like to work on this, especially if someone new joins in.

FWIW, I actually think __leave__ is more teachable. There are two things, one old and one new, but each has a clear name so it’s pretty easy to tell them apart & talk about them.
It’s definitely more discoverable. If you see an unfamiliar name you know that you need to look it up. If you see __exit__ behaving weirdly, that might not be the first thing to come to mind.

This’ll break code that assumes that after setting C.__exit__ = foo will mean C.__exit__ is foo.
We could get away with breaking that assumption. But the downside is that either we’re stuck with a wrapper forever, or such details will need to change again when the three-arg version is removed.

2 Likes

Interesting. I’m thinking it would be explained as “there is one thing, and it has two names because of some business with the args that you are probably not using”.

Hello. I would like to take up this task because I’ve dealt in my work with the points @storchaka mentioned.

I wrote a pure Python gist which simulates the operation of a potential v2 context protocol - Python context manager protocol v2 · GitHub.

I don’t see how this can simplify writing re-entrant managers, because a re-entrant __enter__ call still needs to somehow know if it’s re-entrant or first call. But it still helps that we do not need to store __enter__ result in cases it is not just self.

I also have a few questions that will need to be resolved:

  1. Currently, if the manager returns an iterable object then with expr as (arg1, arg2) unpacks the result of expr.__enter__ immediately (e.g. with open(__file__) as (first_line, *lines). If the current use case is deprecated, then we need to consider that the existing code, after updating if exactly 2 values are used, will give not a syntax error, but very wrong behaviour. So, what alternatives to the syntax with expr as (enter_result, leave_result) could be?
  2. To indicate that the exception is not handled, in that gist __leave__ must raise it again, because it cannot just return an exception (as shown in the assertRaises example), and ‘any truthy value’ obviously may be the expected result. What were the reasons when creating the current context protocol to choose return False instead of raise exc_val?
  3. Currently, all uses of the expr as name idiom immediately bind the name, whereas leave_result binds only after leaving the block. Is this acceptable?

And the main question. Do I need to create a new discussion for this topic, and if so Ideas or core-dev?

It might be a good idea to have a separate thread for redesigning context managers. You can create it in core-dev because there is clearly interest already. Make sure to ping @storchaka and @ncoghlan on it.

Hello folks. I have created a post in which I described a specification for an extended context manager protocol.

1 Like