PEP 707: A simplified signature for __exit__ and __aexit__

I think relying on introspection is a very wrong way.

Look at precedence of solving similar problems in the past: the pickle protocol, rich comparison, the buffer protocol. Introducing a new dunder has underwater rocks, but it is how it always was done.

5 Likes

Thanks. Would be great if you (or others) could elaborate how that would be applied in this case.

As I have understood it from its description in PEP 707, it is a Mark’s idea about __leave__. Although I did not read a discussion about it, only your summary.

I feel that the rationale presented in the PEP for rejecting __leave__ (with automatic addition of the counterpart trampoline to every type that defines __leave__ or __exit__) isn’t convincing/sufficient to reject it. The only downside mentioned is that the subset of existing __exit__ methods that take the form def __exit__(*args): and don’t use *args would need to be converted to __leave__, whereas in the introspection version they wouldn’t. But a) absent data I’m not sure how high a percentage of existing __exit__ methods this really is, and b) renaming such a method to def __leave__(exc) isn’t an onerous or risky update: it’s safe and easily automatable.

The advantages of the __leave__ proposal in clarity and performance seem to me to outweigh this downside.

2 Likes

If a new name is used, there should probably be a deprecation warning when __exit__ is encountered, recommending switching to the new method.

__cmp__ and the old buffer protocol were deprecated and no longer supported, but __reduce__ and __getnewargs__ are still supported, alongside with __reduce_ex__ and __getnewargs_ex__. Deprecating __exit__ will harm a lot of people.

Replacing a triple of arguments with a single argument is a weak reason for introducing a new dunder. The first and the third arguments can be derived from the second argument, so it is only a matter of convenience. I do not think that it justifies all drawbacks.

But we can return to it if decide to extend the context manager semantic.

  1. The result of __enter__ can be provided to a user, but it is not accessible in __exit__. For example, one of problems related to implementation of the buffer protocol from Python side is that __enter__ can return an acquired memory view, but __exit__ does not get it back to release it. It may be worth to pass the result of __enter__ as yet one argument to __leave__.

  2. The result of __enter__ can be provided to a user, but in many cases the useful result of the context manager is only available after exiting the with block. Examples are assertRaises() and assertWarns(). The only solution is to return a mutable object, keep the reference to it in the context manager (see (1)), and updating it in __exit__. You cannot return an immutable object. You need to keep a reference. You cannot implement the timeit context manager which return just a time spent to execute the block as a float.

    What if make the result of __leave__ available to a user? Maybe overriding the variable in as, or introducing a new syntax.

I think it should be a separate topic for discussing an extended context manager protocol.

8 Likes

I think this is on topic here actually. Redesigning a new context manager from scratch with other improvements instead of iterating over the existing one is an option we should consider.

4 Likes

In the stdlib (excluding tests) there are 85 __exit__ functions, of which 33 take a *vararg (two also **kwargs) and don’t use it at all, like:

        def __exit__(self, *args):
            self.close()

Then there are 5 which take a *vararg and forward it to another __exit__ function, like:

    def __exit__(self, *args):
        return self._lock.__exit__(*args)

The only __exit__ I saw which takes *vararg and would need to change is in contextlib._GeneratorContextManager.

There are 46 that take 3 positional args (and may or may not use them).

Edit: I added counts for __exit__ that don’t use *vararg because they are also relevant.

1 Like

For another data point, I just checked in instagram server codebase: there are 87 occurrences of def __exit__, and 10 of them have a * in the signature. I suspect many of the other 77 don’t use the exception information either, but they copied the signature from another context manager, or from Python docs, and didn’t think to replace the three args with a vararg.

1 Like

I’m strongly in favour of the __leave__ approach.

I feel that __leave__ is more robust, without the edge cases that having two versions of __exit__ has. While introspection is viable for Python and builtin functions, it breaks down for other callables, like classes.
E.g. for a class is it the __new__ or the __init__ method that determines which variant to call?

Whichever approach we take there will be two changes to the language:

  1. From what we have now, to the one-argument-first implementation (2023)
  2. Removal of support for the three argument form (~2028)

With the one-argument form of __exit__, nothing need be done to handle the first step.
But things get complicated for the second step.

Any library that needs to support versions straddling the second transition need to dynamically handle either one or three arguments. Something like:

def __exit__(self, arg0, arg1=SENTINEL, arg2=SENTINEL):
    if arg1 is SENTINEL:
        exc = arg0
    else:
        exc = arg1
    ...

With __leave__, libraries need to add a __leave__ method, and remove the __exit__ sometime between steps 1 and 2. More work, yes, but safer and more easily understood.

1 Like

This is correct, in the uncommon case where the function does something with the args, otherwise *args works for both.

I generally support the initial proposal to alter __exit__'s signature:

  • As an existing Python user who is familiar with the current form of __exit__, I support the approach of changing the signature of the existing __exit__. The proposed new signature (with just the exception value) is what I’ve always thought it should be anyway.

  • I expect a new Python user writing code that has to interface with older library code (that uses __exit__) could easily be confused that they may need to override an “obscure” __exit__ function rather than define a (now recommended) __leave__ function.

  • I expect a new Python user writing new code wouldn’t care whether __exit__ or __leave__ was the recommended form, only that the recommended form was clearly documented.


Small nitpick: The link from word “specialization” in the PEP goes to PEP 564, which doesn’t appear to have anything to do with specialization. Perhaps it was intended to link to a different PEP?

2 Likes

Or, use __leave__ if it exists, ignoring __exit__; if only __exit__ exists, either issue a deprecation warning, or an error, as appropriate for that version of Python. That way __exit__ doesn’t have to be removed until the library in question decides to stop supporting Python versions that only use __exit__.

2 Likes

Is there a performance hit for the proposed change?

If not (or if insignificant) I’d greatly prefer just modifying __exit__ in the proposed way. It seems weird to think about adding a __leave__ considering we’ve had exit all this time. We can even mark the old signature of __exit__ as pending deprecation (then deprecated) for a few versions if desired to notify and give time for libraries to update.

I’m not for or against that deprecation. Though I am for the proposal in the PEP.

If we did have __leave__ what would we do if both __exit__ and __leave__ were defined? I’m sure we can come up with a precedence, but I think we’re adding undo complexity and confusion for folks.

This is addressed in the PEP - only the straightforward cases are interpreted as single-arg. I don’t think this is would be a problem in practice.

As mentioned in the PEP, any performance impact can be mitigated with specialisation

The interpreter can do that, but other calls to __exit__, from libraries and subclasses, need to continue working through the transition.

As I wrote in the intro, I am not sure myself that introspection is the right approach. I also agree with you that just removing two redundant args is weak justification for a new dunder. I think your suggestion to redesign context managers from scratch should be explored. However, I would like better justification to reject the introspection approach than “that’s not what we did in the past”. What are the problems with this approach that make it unacceptable in your view?

Does it seem better or worse if the compiler checks the arguments and somehow marks the code object in a way that can be checked specifically at runtime (e.g. a ONE_ARG_EXIT flag or attribute)? Technically I guess it’s still introspection, but at least specified all the way through rather than trying to interpret other metadata after the fact.

Seems like potentially a bad precedent, but otherwise I’m not totally convinced it’s the worst option.

Also, if we get 13% perf improvement from dropping two arguments (presumably that’s mostly calling convention overhead?), why not also allow dropping all arguments for cases where you don’t care about what the exception is at all, or even whether one occurred? Most of my __exit__s are in this category.

1 Like

The compiler may not know that it’s an exit function:

>>> def myexit(self, *args): print(args)
...
>>> class CM:
...     def __enter__(self): return self
...
>>> CM.__exit__ = myexit
>>>
>>> with CM(): 1/0
...
(<class 'ZeroDivisionError'>, ZeroDivisionError('division by zero'), <traceback object at 0x102582d00>)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

Yeah, there’s 100 ways to break introspection no matter how you do it. That’s why adding the new name is the only truly safe option.

3 Likes

If we do it in the compiler it’s static analysis rather than introspection. That’s a lot more limited.

I’ve seen real code with functions defined outside the class and assigned to __exit__. I haven’t yet seen any which are non-function callables.