The legacy representation of exceptions as a (type, value, traceback) triplet was useful in the past, but now it is redundant because the type and traceback can always be deduced from the value. Over the past few years we have been working to remove this redundancy from the language and its implementation.
One of the harder problems is how to evolve the signature of context managers’ __exit__/__aexit__ functions. A number of us have been bouncing ideas about this, and feel that it’s time to bring them forward so that they can be discussed. I have written PEP 707, with my preferred option, to kick this off.
I am expecting at least one alternative proposal to be presented, hopefully even more. As I wrote in the PEP: This imperfect solution was chosen among several imperfect alternatives in the spirit of practicality. It is my hope that the discussion about this PEP will explore the other options and lead us to the best way forward, which may well be to remain with our imperfect status quo.
I feel that the rationale presented in the PEP for rejecting __leave__ (with automatic addition of the counterpart trampoline to every type that defines __leave__ or __exit__) isn’t convincing/sufficient to reject it. The only downside mentioned is that the subset of existing __exit__ methods that take the form def __exit__(*args): and don’t use *args would need to be converted to __leave__, whereas in the introspection version they wouldn’t. But a) absent data I’m not sure how high a percentage of existing __exit__ methods this really is, and b) renaming such a method to def __leave__(exc) isn’t an onerous or risky update: it’s safe and easily automatable.
The advantages of the __leave__ proposal in clarity and performance seem to me to outweigh this downside.
__cmp__ and the old buffer protocol were deprecated and no longer supported, but __reduce__ and __getnewargs__ are still supported, alongside with __reduce_ex__ and __getnewargs_ex__. Deprecating __exit__ will harm a lot of people.
Replacing a triple of arguments with a single argument is a weak reason for introducing a new dunder. The first and the third arguments can be derived from the second argument, so it is only a matter of convenience. I do not think that it justifies all drawbacks.
But we can return to it if decide to extend the context manager semantic.
The result of __enter__ can be provided to a user, but it is not accessible in __exit__. For example, one of problems related to implementation of the buffer protocol from Python side is that __enter__ can return an acquired memory view, but __exit__ does not get it back to release it. It may be worth to pass the result of __enter__ as yet one argument to __leave__.
The result of __enter__ can be provided to a user, but in many cases the useful result of the context manager is only available after exiting the with block. Examples are assertRaises() and assertWarns(). The only solution is to return a mutable object, keep the reference to it in the context manager (see (1)), and updating it in __exit__. You cannot return an immutable object. You need to keep a reference. You cannot implement the timeit context manager which return just a time spent to execute the block as a float.
What if make the result of __leave__ available to a user? Maybe overriding the variable in as, or introducing a new syntax.
I think it should be a separate topic for discussing an extended context manager protocol.
For another data point, I just checked in instagram server codebase: there are 87 occurrences of def __exit__, and 10 of them have a * in the signature. I suspect many of the other 77 don’t use the exception information either, but they copied the signature from another context manager, or from Python docs, and didn’t think to replace the three args with a vararg.
I feel that __leave__ is more robust, without the edge cases that having two versions of __exit__ has. While introspection is viable for Python and builtin functions, it breaks down for other callables, like classes.
E.g. for a class is it the __new__ or the __init__ method that determines which variant to call?
I generally support the initial proposal to alter __exit__'s signature:
As an existing Python user who is familiar with the current form of __exit__, I support the approach of changing the signature of the existing __exit__. The proposed new signature (with just the exception value) is what I’ve always thought it should be anyway.
I expect a new Python user writing code that has to interface with older library code (that uses __exit__) could easily be confused that they may need to override an “obscure” __exit__ function rather than define a (now recommended) __leave__ function.
I expect a new Python user writing new code wouldn’t care whether __exit__ or __leave__ was the recommended form, only that the recommended form was clearly documented.
Small nitpick: The link from word “specialization” in the PEP goes to PEP 564, which doesn’t appear to have anything to do with specialization. Perhaps it was intended to link to a different PEP?
Or, use __leave__ if it exists, ignoring __exit__; if only __exit__ exists, either issue a deprecation warning, or an error, as appropriate for that version of Python. That way __exit__ doesn’t have to be removed until the library in question decides to stop supporting Python versions that only use __exit__.
Is there a performance hit for the proposed change?
If not (or if insignificant) I’d greatly prefer just modifying __exit__ in the proposed way. It seems weird to think about adding a __leave__ considering we’ve had exit all this time. We can even mark the old signature of __exit__ as pending deprecation (then deprecated) for a few versions if desired to notify and give time for libraries to update.
I’m not for or against that deprecation. Though I am for the proposal in the PEP.
If we did have __leave__ what would we do if both __exit__ and __leave__ were defined? I’m sure we can come up with a precedence, but I think we’re adding undo complexity and confusion for folks.
This is addressed in the PEP - only the straightforward cases are interpreted as single-arg. I don’t think this is would be a problem in practice.
As mentioned in the PEP, any performance impact can be mitigated with specialisation
The interpreter can do that, but other calls to __exit__, from libraries and subclasses, need to continue working through the transition.
As I wrote in the intro, I am not sure myself that introspection is the right approach. I also agree with you that just removing two redundant args is weak justification for a new dunder. I think your suggestion to redesign context managers from scratch should be explored. However, I would like better justification to reject the introspection approach than “that’s not what we did in the past”. What are the problems with this approach that make it unacceptable in your view?
Does it seem better or worse if the compiler checks the arguments and somehow marks the code object in a way that can be checked specifically at runtime (e.g. a ONE_ARG_EXIT flag or attribute)? Technically I guess it’s still introspection, but at least specified all the way through rather than trying to interpret other metadata after the fact.
Seems like potentially a bad precedent, but otherwise I’m not totally convinced it’s the worst option.
Also, if we get 13% perf improvement from dropping two arguments (presumably that’s mostly calling convention overhead?), why not also allow dropping all arguments for cases where you don’t care about what the exception is at all, or even whether one occurred? Most of my __exit__s are in this category.