PEP 707: A simplified signature for __exit__ and __aexit__

[iritkatriel]

@ncoghlan

the fact the introspection could be moved to type definition time

I don’t think it can. See this comment.

[ncoghlan (Alyssa Coghlan)]

The fact late-binding __exit__ on a class doesn’t have the desired effect is what I was referring to when noting that the PEP as implemented isn’t taking full advantage of the descriptor machinery.

It also wouldn’t be a new limitation, even if the PEP decided not to resolve it.

[iritkatriel]

I don’t follow. It seems to me that late binding does have the desired effect.

[encukou (Petr Viktorin)]

First of all, thank you for your great work on getting rid of the exception triples.
(I have a habit of only speaking up when I see issues, which is a fine CLI design principle but not that great with people. I hope I don’t seem too negative!)

---

@iritkatriel

@encukou

Such tools will need to generate METH_O functions (rather than e.g. METH_FASTCALL) to avoid changing semantics, right?

Not sure what you mean by “avoid changing semantics”. The current code would not be interpreted as single-arg, it would continue to work as it does now. If they want to generate exits which are interpreted as single-arg, they will need to refer to the PEP to see how to do that.

I meant changing the semantics between Python code and the compiled version: __exit(arg)__ would behave differently based on whether the compiled version uses METH_O, METH_FASTCALL, or a custom callable type.

---

While I won’t have time to drive it for 3.12, I spent a few hours on a proof of concept for __leave__: all the ways of setting a type’s __exit__ also set __leave__ to a wrapper that calls that __exit__, and vice versa. The interpreter then only calls __leave__.
I guess the piece that was missing is capturing the callable at __setattr__ time, rather than trying to look it up dynamically.
As expected, it’s probably harder to implement than PEP 707 (touching typeobject.c is never pleasant), but I do think keeping heuristics out of a core protocol is worth it.
A very rough proof of concept is here. It only passes most tests, and leaks a lot, but the concept is there.

[brettcannon]

The SC rejected this PEP. For simplicity, here’s the message I posted on the issue asking for pronouncement:

We discussed the PEP and have decided to reject it. Our thinking was the magic and risk of potential breakage didn’t warrant the benefits. We are totally supportive, though, of exploring a potential context manager v2 API or __leave__.

[iritkatriel]

I will update the PEP.

I hope that one of the other alternatives will make it into 3.13. Since the status quo received only 3% of the vote, it would be a pity if this is the option we will end up choosing.

As far as I can tell, nobody so far has indicated that they are planning to work on the alternative they proposed. If you are non-core-dev looking for a cpython project to take on, perhaps you could help make that happen. You can use this thread to initiate such a collaboration.

[thomas (Thomas Wouters)]

The SC does agree that this bears fixing, and if neither __leave__ nor a more thorough redesigned of context managers ends up being practical, we could reconsider this PEP… but the objections to __leave__ so far are not strong enough to choose this alternative.

From my point of view, the problem with this proposal is that it muddies the water just as much as adding a new method, in that the same method can get called multiple ways. It makes the desired end state (a single argument) the special case, and provides no way to migrate to that desired end state (we can’t add deprecation warnings because we don’t know what will or won’t keep working). The fact that it causes changes in how the __exit__ method is called when it, for instance, is wrapped in a seeming innocuous lambda self, *args: f(self, *args) makes it particularly uneasy for me.

[iritkatriel]

@thomas

The fact that it causes changes in how the __exit__ method is called when it, for instance, is wrapped in a seeming innocuous lambda self, *args: f(self, *args) makes it particularly uneasy for me.

I’m not following. What would change here? A signature of (self, *args) would be interpreted as the 3-arg version.

@thomas

we can’t add deprecation warnings because we don’t know what will or won’t keep working

Could we not add a deprecation warning just before calling a 3-arg __exit__?

@thomas

we could reconsider this PEP… but the objections to __leave__ so far are not strong enough to choose this alternative.

Understood. TBH, I expected the SC to defer decision on this PEP until we’ve had time to evaluate the other options. I was not aware that a PEP can be reconsidered after being rejected. That’s good to know.

[thomas (Thomas Wouters)]

@iritkatriel

@thomas

The fact that it causes changes in how the __exit__ method is called when it, for instance, is wrapped in a seeming innocuous lambda self, *args: f(self, *args) makes it particularly uneasy for me.

I’m not following. What would change here? A signature of (self, *args) would be interpreted as the 3-arg version.

Yes, but the original method might take just one argument. The method would be expecting “new” semantics, and work fine, and then fail to work when it’s wrapped by something else. Or, when the wrapper used to wrap it is changed for other reasons.

@iritkatriel

@thomas

we can’t add deprecation warnings because we don’t know what will or won’t keep working

Could we not add a deprecation warning just before calling a 3-arg __exit__?

Not without a lot of false positives, the same you’re looking to avoid by introducing the magic switching in the first place All the places that ignore the args or merely pass them along would then still have to change just to silence the deprecation warning.

@iritkatriel

@thomas

we could reconsider this PEP… but the objections to __leave__ so far are not strong enough to choose this alternative.

Understood. TBH, I expected the SC to defer decision on this PEP until we’ve had time to evaluate the other options. I was not aware that a PEP can be reconsidered after being rejected. That’s good to know.

Everything can always be reconsidered

[iritkatriel]

@thomas

the same you’re looking to avoid by introducing the magic switching in the first place

My main concern is actually not so much to save code changes, but to avoid having to teach every python developer out there that __leave__ is kind of the same thing as __exit__. I think this can create a lot of confusion.

But maybe we can take the idea that @markshannon and @encukou proposed and apply it to the introspection solution, in a way that will avoid the breakage you described with the wrapper.

They proposed that every type that has __leave__ or __exit__ is automatically given the other. We could, instead, wrap the given __exit__ with a new __exit__ that takes *args and does the right thing for all cases: checks whether args is an exception or an exc_info, and then calls the wrapped __exit__ with one or 3 args, depending on introspection result.

[ntessore (Nicolas Tessore)]

If context managers generally don’t use their arguments at all, would it not be easier to offer a zero-argument __exit__() instead, if the exception can still be retrieved from sys.exception?

[iritkatriel]

Look in contextlib. Sometimes you want a different exception than sys.exception().

[encukou (Petr Viktorin)]

@iritkatriel

As far as I can tell, nobody so far has indicated that they are planning to work on the alternative they proposed. If you are non-core-dev looking for a cpython project to take on, perhaps you could help make that happen. You can use this thread to initiate such a collaboration.

Yep, unfortunately I can’t promise I’ll have time. But I’d like to work on this, especially if someone new joins in.

@iritkatriel

My main concern is actually not so much to save code changes, but to avoid having to teach every python developer out there that __leave__ is kind of the same thing as __exit__. I think this can create a lot of confusion.

FWIW, I actually think __leave__ is more teachable. There are two things, one old and one new, but each has a clear name so it’s pretty easy to tell them apart & talk about them.
It’s definitely more discoverable. If you see an unfamiliar name you know that you need to look it up. If you see __exit__ behaving weirdly, that might not be the first thing to come to mind.

@iritkatriel

They proposed that every type that has __leave__ or __exit__ is automatically given the other. We could, instead, wrap the given __exit__ with a new __exit__ that takes *args and does the right thing for all cases: checks whether args is an exception or an exc_info, and then calls the wrapped __exit__ with one or 3 args, depending on introspection result.

This’ll break code that assumes that after setting C.__exit__ = foo will mean C.__exit__ is foo.
We could get away with breaking that assumption. But the downside is that either we’re stuck with a wrapper forever, or such details will need to change again when the three-arg version is removed.

[iritkatriel]

@encukou

FWIW, I actually think __leave__ is more teachable. There are two things, one old and one new, but each has a clear name so it’s pretty easy to tell them apart & talk about them.

Interesting. I’m thinking it would be explained as “there is one thing, and it has two names because of some business with the args that you are probably not using”.

[Andy_kl (Andrej Klychin)]

Hello. I would like to take up this task because I’ve dealt in my work with the points @storchaka mentioned.

I wrote a pure Python gist which simulates the operation of a potential v2 context protocol - Python context manager protocol v2 · GitHub.

@ncoghlan

passes the return value from the CM entry function to the exit function (to make re-entrant CMs easier to write without having to rely on closures to store state), with the active exception as an optional second argument

I don’t see how this can simplify writing re-entrant managers, because a re-entrant __enter__ call still needs to somehow know if it’s re-entrant or first call. But it still helps that we do not need to store __enter__ result in cases it is not just self.

I also have a few questions that will need to be resolved:

1. Currently, if the manager returns an iterable object then with expr as (arg1, arg2) unpacks the result of expr.__enter__ immediately (e.g. with open(__file__) as (first_line, *lines). If the current use case is deprecated, then we need to consider that the existing code, after updating if exactly 2 values are used, will give not a syntax error, but very wrong behaviour. So, what alternatives to the syntax with expr as (enter_result, leave_result) could be?
2. To indicate that the exception is not handled, in that gist __leave__ must raise it again, because it cannot just return an exception (as shown in the assertRaises example), and ‘any truthy value’ obviously may be the expected result. What were the reasons when creating the current context protocol to choose return False instead of raise exc_val?
3. Currently, all uses of the expr as name idiom immediately bind the name, whereas leave_result binds only after leaving the block. Is this acceptable?

And the main question. Do I need to create a new discussion for this topic, and if so Ideas or core-dev?

[iritkatriel]

@Andy_kl

Do I need to create a new discussion for this topic, and if so Ideas or core-dev?

It might be a good idea to have a separate thread for redesigning context managers. You can create it in core-dev because there is clearly interest already. Make sure to ping @storchaka and @ncoghlan on it.

[Andy_kl (Andrej Klychin)]

Hello folks. I have created a post in which I described a specification for an extended context manager protocol.