PEP 707: A simplified signature for __exit__ and __aexit__

PEPs
1

The legacy representation of exceptions as a (type, value, traceback) triplet was useful in the past, but now it is redundant because the type and traceback can always be deduced from the value. Over the past few years we have been working to remove this redundancy from the language and its implementation.

One of the harder problems is how to evolve the signature of context managers’ __exit__/__aexit__ functions. A number of us have been bouncing ideas about this, and feel that it’s time to bring them forward so that they can be discussed. I have written PEP 707, with my preferred option, to kick this off.

I am expecting at least one alternative proposal to be presented, hopefully even more. As I wrote in the PEP: This imperfect solution was chosen among several imperfect alternatives in the spirit of practicality. It is my hope that the discussion about this PEP will explore the other options and lead us to the best way forward, which may well be to remain with our imperfect status quo.

13 Likes
2

Thank you for this PEP! One slightly confusing thing for me: in the “Motivations” section, sys.exc_info is listed in a table under a “Deprecated” column, implying that it is already deprecated. But I can’t see a note in the docs for sys.exc_info saying that it’s deprecated. And slightly lower down in the PEP, you state:

It will take multiple releases to get to a point where we can think of deprecating sys.exc_info() .

Perhaps it would be better to name the first column of the table in the “Motivations” section “Old-style APIs” or “Legacy APIs”?

3 Likes
3

I think relying on introspection is a very wrong way.

Look at precedence of solving similar problems in the past: the pickle protocol, rich comparison, the buffer protocol. Introducing a new dunder has underwater rocks, but it is how it always was done.

5 Likes
5

Thanks. Would be great if you (or others) could elaborate how that would be applied in this case.

6

As I have understood it from its description in PEP 707, it is a Mark’s idea about __leave__. Although I did not read a discussion about it, only your summary.

7

I feel that the rationale presented in the PEP for rejecting __leave__ (with automatic addition of the counterpart trampoline to every type that defines __leave__ or __exit__) isn’t convincing/sufficient to reject it. The only downside mentioned is that the subset of existing __exit__ methods that take the form def __exit__(*args): and don’t use *args would need to be converted to __leave__, whereas in the introspection version they wouldn’t. But a) absent data I’m not sure how high a percentage of existing __exit__ methods this really is, and b) renaming such a method to def __leave__(exc) isn’t an onerous or risky update: it’s safe and easily automatable.

The advantages of the __leave__ proposal in clarity and performance seem to me to outweigh this downside.

2 Likes
8

If a new name is used, there should probably be a deprecation warning when __exit__ is encountered, recommending switching to the new method.

9

__cmp__ and the old buffer protocol were deprecated and no longer supported, but __reduce__ and __getnewargs__ are still supported, alongside with __reduce_ex__ and __getnewargs_ex__. Deprecating __exit__ will harm a lot of people.

Replacing a triple of arguments with a single argument is a weak reason for introducing a new dunder. The first and the third arguments can be derived from the second argument, so it is only a matter of convenience. I do not think that it justifies all drawbacks.

But we can return to it if decide to extend the context manager semantic.

  1. The result of __enter__ can be provided to a user, but it is not accessible in __exit__. For example, one of problems related to implementation of the buffer protocol from Python side is that __enter__ can return an acquired memory view, but __exit__ does not get it back to release it. It may be worth to pass the result of __enter__ as yet one argument to __leave__.

  2. The result of __enter__ can be provided to a user, but in many cases the useful result of the context manager is only available after exiting the with block. Examples are assertRaises() and assertWarns(). The only solution is to return a mutable object, keep the reference to it in the context manager (see (1)), and updating it in __exit__. You cannot return an immutable object. You need to keep a reference. You cannot implement the timeit context manager which return just a time spent to execute the block as a float.

    What if make the result of __leave__ available to a user? Maybe overriding the variable in as, or introducing a new syntax.

I think it should be a separate topic for discussing an extended context manager protocol.

8 Likes
10

I think this is on topic here actually. Redesigning a new context manager from scratch with other improvements instead of iterating over the existing one is an option we should consider.

4 Likes
11

In the stdlib (excluding tests) there are 85 __exit__ functions, of which 33 take a *vararg (two also **kwargs) and don’t use it at all, like:

        def __exit__(self, *args):
            self.close()

Then there are 5 which take a *vararg and forward it to another __exit__ function, like:

    def __exit__(self, *args):
        return self._lock.__exit__(*args)

The only __exit__ I saw which takes *vararg and would need to change is in contextlib._GeneratorContextManager.

There are 46 that take 3 positional args (and may or may not use them).

Edit: I added counts for __exit__ that don’t use *vararg because they are also relevant.

1 Like
12

For another data point, I just checked in instagram server codebase: there are 87 occurrences of def __exit__, and 10 of them have a * in the signature. I suspect many of the other 77 don’t use the exception information either, but they copied the signature from another context manager, or from Python docs, and didn’t think to replace the three args with a vararg.

1 Like
13

I’m strongly in favour of the __leave__ approach.

I feel that __leave__ is more robust, without the edge cases that having two versions of __exit__ has. While introspection is viable for Python and builtin functions, it breaks down for other callables, like classes.
E.g. for a class is it the __new__ or the __init__ method that determines which variant to call?

14

Whichever approach we take there will be two changes to the language:

  1. From what we have now, to the one-argument-first implementation (2023)
  2. Removal of support for the three argument form (~2028)

With the one-argument form of __exit__, nothing need be done to handle the first step.
But things get complicated for the second step.

Any library that needs to support versions straddling the second transition need to dynamically handle either one or three arguments. Something like:

def __exit__(self, arg0, arg1=SENTINEL, arg2=SENTINEL):
    if arg1 is SENTINEL:
        exc = arg0
    else:
        exc = arg1
    ...

With __leave__, libraries need to add a __leave__ method, and remove the __exit__ sometime between steps 1 and 2. More work, yes, but safer and more easily understood.

1 Like
15

This is correct, in the uncommon case where the function does something with the args, otherwise *args works for both.

16

I generally support the initial proposal to alter __exit__'s signature:

  • As an existing Python user who is familiar with the current form of __exit__, I support the approach of changing the signature of the existing __exit__. The proposed new signature (with just the exception value) is what I’ve always thought it should be anyway.

  • I expect a new Python user writing code that has to interface with older library code (that uses __exit__) could easily be confused that they may need to override an “obscure” __exit__ function rather than define a (now recommended) __leave__ function.

  • I expect a new Python user writing new code wouldn’t care whether __exit__ or __leave__ was the recommended form, only that the recommended form was clearly documented.

Small nitpick: The link from word “specialization” in the PEP goes to PEP 564, which doesn’t appear to have anything to do with specialization. Perhaps it was intended to link to a different PEP?

2 Likes
17

Or, use __leave__ if it exists, ignoring __exit__; if only __exit__ exists, either issue a deprecation warning, or an error, as appropriate for that version of Python. That way __exit__ doesn’t have to be removed until the library in question decides to stop supporting Python versions that only use __exit__.

2 Likes
18

Is there a performance hit for the proposed change?

If not (or if insignificant) I’d greatly prefer just modifying __exit__ in the proposed way. It seems weird to think about adding a __leave__ considering we’ve had exit all this time. We can even mark the old signature of __exit__ as pending deprecation (then deprecated) for a few versions if desired to notify and give time for libraries to update.

I’m not for or against that deprecation. Though I am for the proposal in the PEP.

If we did have __leave__ what would we do if both __exit__ and __leave__ were defined? I’m sure we can come up with a precedence, but I think we’re adding undo complexity and confusion for folks.

19

This is addressed in the PEP - only the straightforward cases are interpreted as single-arg. I don’t think this is would be a problem in practice.

As mentioned in the PEP, any performance impact can be mitigated with specialisation

The interpreter can do that, but other calls to __exit__, from libraries and subclasses, need to continue working through the transition.

As I wrote in the intro, I am not sure myself that introspection is the right approach. I also agree with you that just removing two redundant args is weak justification for a new dunder. I think your suggestion to redesign context managers from scratch should be explored. However, I would like better justification to reject the introspection approach than “that’s not what we did in the past”. What are the problems with this approach that make it unacceptable in your view?

20

Does it seem better or worse if the compiler checks the arguments and somehow marks the code object in a way that can be checked specifically at runtime (e.g. a ONE_ARG_EXIT flag or attribute)? Technically I guess it’s still introspection, but at least specified all the way through rather than trying to interpret other metadata after the fact.

Seems like potentially a bad precedent, but otherwise I’m not totally convinced it’s the worst option.

Also, if we get 13% perf improvement from dropping two arguments (presumably that’s mostly calling convention overhead?), why not also allow dropping all arguments for cases where you don’t care about what the exception is at all, or even whether one occurred? Most of my __exit__s are in this category.

1 Like
21

The compiler may not know that it’s an exit function:

>>> def myexit(self, *args): print(args)
...
>>> class CM:
...     def __enter__(self): return self
...
>>> CM.__exit__ = myexit
>>>
>>> with CM(): 1/0
...
(<class 'ZeroDivisionError'>, ZeroDivisionError('division by zero'), <traceback object at 0x102582d00>)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero