Yeah, there’s 100 ways to break introspection no matter how you do it. That’s why adding the new name is the only truly safe option.
3 Likes
If we do it in the compiler it’s static analysis rather than introspection. That’s a lot more limited.
I’ve seen real code with functions defined outside the class and assigned to __exit__. I haven’t yet seen any which are non-function callables.
Both introspection or static analysis are suboptimal because people will have to support two use cases. Then all the benefits of “a single argument” are gone. You end up writing a lot more for years instead!
By the way, this PEP doesn’t even need to be implemented for that to be achieved
def exit(f):
@wraps(f)
def decorated_exit(self, typ, exc, tb):
return f(self, typ(exc).with_traceback(tb))
return decorated_exit
class Foo:
def __enter__(self):
print('enter')
@exit
def __exit__(self, exc):
print('exit', exc)
3 Likes
I guess the most common way to get non-function callables here would be compiling with something like Cython or mypyc.
Such tools will need to generate METH_O functions (rather than e.g. METH_FASTCALL) to avoid changing semantics, right?
Some people will, but most won’t because their __exit__ is only ever called by the interpreter. It’s only in cases where the context manager is in a library, designed to be subclassed or called by framework code, where you need to worry about the legacy signature.
The decorator you propose is cute, but it doesn’t give us the speedup that we get by making the interpreter call the single-arg version whenever possible. It also doesn’t give us a 5-10 year plan to remove the legacy signature from the language.
Not sure what you mean by “avoid changing semantics”. The current code would not be interpreted as single-arg, it would continue to work as it does now. If they want to generate exits which are interpreted as single-arg, they will need to refer to the PEP to see how to do that.
Oh so you’re Irit? Love the work you did with exceptions in the recent python versions 
I would LOVE to find a way to implement this PEP. If you wish to reduce the runtime cost, perhaps the introspection can be achieved at the class definition level? Whenever a function called __exit__ is being set on a class object, the introspection can happen and set a private flag like __is_legacy_exit__ on the function object? That way the majority of the runtime cost will happen once upon definition.
On most days.
Thank you!
That’s too early:
>>> class C:
... def __enter__(self): return self
...
>>> C.__exit__ = lambda *arg : print('exit with', arg)
>>>
>>> with C(): 1/0
...
exit with (<__main__.C object at 0x109946fe0>, <class 'ZeroDivisionError'>, ZeroDivisionError('division by zero'), <traceback object at 0x109947070>)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero
>>>
But we can achieve the same performance optimization with specialization/caching in the bytecode.
If we redesign context managers from scratch as @storchaka suggested, then perhaps the exception can be attached to the context manager instead of being passed to __exit__ as an arg.
We would need to measure, but I’m not sure the 13% speedup going from 3 args (METH_VARARGS) to 1 (METH_O) would translate to another 6.5% for removing the last arg.
Finally, introspection is limited. It might be just about good enough to get us from 3 args to 1, but I wouldn’t use it to implement a forever feature.
Is specialization/caching in the bytecode a preferable thing to do or more of a last resort? Is it a complicated thing to add compared to older approaches?
Perhaps an alternative way that accounts for your example would be a descriptor on type objects so the dunder methods could be introspected even when retroactively setting them on an existing class?
We would have to put the descriptor on every type object, whether it is a context manager or not. Currently type objects don’t have anything allocated for __exit__ by default.
The discussion seems to have quieted down for now, but there isn’t a clear consensus so I’d like to have a poll about the options raised so far. Please select what you think is best (and feel free to continue discussing the options below).
- Do nothing, exit takes (typ, exc, tb) and sys.exc_info() is not deprecated.
- Submit PEP 707 to the steering council for consideration for 3.12.
- Add a leave(exc) api instead of PEP 707 instrospection (needs a PEP, may still be possible in 3.12).
- Design context managers from scratch along the lines of Serhiy’s proposal above (probably for 3.13).
0 voters
I feel like we end up regretting most rushed new APIs, so would definitely rather take the time to design new CM API than use up a good name like __leave__ in a hurry. (I’d settle for __exit1__ or something ugly if it’s urgent.)
Introspection as a transitional tool is worth exploring though, so I wouldn’t be totally opposed to it. If we do it here though, I’d definitely want our experience to be documented somewhere. Otherwise it might turn out horribly but in a few years we do it again anyway “because we did it for 707”…
5 Likes
Keep voting and discussing, but so far the vote reflects my view that it’s a choice between PEP 707 and a more comprehensive redesign of context managers. I have handed this over to the SC for a decision.
I think if I was on the SC I would like to know how likely it is that a PEP for redesigning context managers would be presented for 3.13. If you (@storchaka or anyone else) are serious about working on this, it may help to indicate so here.
A datapoint for support of the idea to revamp the context manager protocol is the discussion in Yield-based contextmanager for classes, and especially the quote from @njs
So new users constantly try to write their own
__enter__/__exit__methods […] and I think literally every person who’s ever tried this has gotten it wrong (mostly around exception handling details). At this point we don’t even try to debug; we just tell users to always use@contextmanager.
1 Like
Within PEP 707 itself, I don’t think the proposed implementation is taking sufficient advantage of the descriptor machinery and is pushing too much complexity into the main interpreter loop as a result. (the descriptor machinery allows post-creation updates to the special methods to be intercepted and processed as they would have been at type instance creation time)
Specifically, if this change were to be pursued, then at the very least a new C level special method slot should be introduced.
This new slot would hold a new function pointer that accepted a single exception instance rather than the traditional exception triple.
The main interpreter loop would unconditionally call the new slot.
Any callable introspection would occur in the type machinery when it is fixing up the special method definitions rather than happening every time the context manager is used.
For types declared in C with only the old slot populated, the new slot would always get a shared wrapper function that decomposes back to the old triple format and calls the method in the existing slot. Similarly, types with only the new slot populated would get a wrapper in the old slot that forwards only the exception instance to the new slot. Types with both slots already populated would be left alone.
For types declared in Python, both slots would always be populated. The pair of wrappers used would pass either the exception triple or the exception instance through to the Python exit method based on introspection of the target callable.
(This implementation strategy also applies if the Python level “new or old?” API decision is unambiguously indicated via the use of a different method name, but at that point the suggestion is fundamentally a different PEP, rather than a different way of implementing PEP 707)
There is no old slot. I agree if there was we would do something like this.
With specialisation we can do the introspection once and cache the result, we don’t need to do it every time.
True, I forgot we decided we didn’t need the acceleration for the with statement slots back when they were first added.
While I think the core of the alternative implementation proposal still holds even with only the new signature getting a slot, the observation highlights the more general concern with the overall concept behind PEP 707: the expected signature of type(x).__exit__ will be inherently ambiguous for any code that calls it. There’s no descriptor magic that can hide that - the only way to avoid it is to give the new signature a new magic method name.
An implicitly added __exit__ wrapper could be generated for a new name, so code that relied on the __exit__ API would still work.
Are you able to put together an implementation of this and make an alternative proposal? There might still be time for 3.12.
We can add a builtin helper function that takes the exit function and an exception and does the introspection and calls exit as the interpreter would.
1 Like
I’m personally more in the “live with the redundancy indefinitely” camp, at least until someone designs and implements an updated context management protocol that meets the goals Serhiy mentioned above:
- passes the return value from the CM entry function to the exit function (to make re-entrant CMs easier to write without having to rely on closures to store state), with the active exception as an optional second argument
- allows the CM exit function to indicate that exceptions should be suppressed and return a value (e.g. by returning a non-empty tuple)
- updates the grammar to allow a 2-tuple of names in the “as” clause of with statements
(Technically, the latter two points could be handled independently from the first, since they don’t affect the method signature - they just tweak the meaning of non-empty tuples when the updated name binding syntax is used)
As Mark pointed out above, we even have __leave__ available as an entirely viable name for the new method.
The derived wrapper function technique can then be used at type construction time to ensure that implementing either version of the CM termination protocol gets you both methods on the resulting class. (at which point, it probably wouldn’t be worth it to deprecate the old method - it could just live on indefinitely, like some of the pickle and copy compatibility APIs)
However, even for proposals that I don’t personally favour, I prefer to see them evaluated in their best possible form, rather than making final decisions based on aspects that are specific to an initial proof of concept implementation (in this case, the fact the introspection could be moved to type definition time means that the runtime impact of the introspection could be minimised even without a specialising interpreter implementation)
1 Like