manager = (EXPRESSION)
enter = manager.__enter__
exit = manager.__exit__
value = enter()
hit_except = False
try:
TARGET = value
SUITE
except:
hit_except = True
if not exit(*sys.exc_info()):
raise
finally:
if not hit_except:
exit(None, None, None)
Now I wonder, what if the program gets an exception right after the value = enter() (e.g. MemoryError or a signal handler causes sys.exit())? That might already require clean-ups by a exit(), but the code is still outside the try block so the exception shouldn’t be caught?
Similarly, with contextlib.ExitStack:
It first does result = _enter() but before it registers the _exit it might already be kicked out.
Or if people use schemes like: self.resource = get_resource() exit_stack.callback(self.resource.close)
That’s quite a bummer, ain’t it? I mean the core promise of context managers is actually not even valid (regardless of whether one uses signalor not?
Shouldn’t the warning then rather be in the context manager documentation?
When a signal handler causes an exception, like when it calls e.g. sys.exit()… where does that exception continue after the signal handler is done? At the place where the code was previously interrupted?
Just read about your defer_interrupts()idea… that would obviously be neat…
But I guess with blocks themselves (or at least some parts of it) would need such a deferring.
Do you think anything will actually move forward with that discussion? It seems to have died off in December.
It is not so painful for applications that can simply register a signal handler and avoid KeyboardInterrupt and friends completely, but for libraries this liberty can’t be taken so lightly.
I don’t see a problem with its location, it’s definitely got more to do with signals rather than context managers, as the same caveats apply to finally blocks as well.
I think it’s got more to do with asynchronous exceptions, and hence signals, is what I’m trying to say.
At a purposely undefined place, to avoid constraining the VM or the language. That’s where the deferred interrupts idea fits in.
I can’t really say when, but I’ll try to push for it.
I would certainly dislike having the exit callback get called if enter raised an exception because of a bug or something that was not an interrupt.
Well my main point was, that many users will probably never read the documentation of signal and simply think that a context manager protects them in every case… that’s at least what they’re often advertised as.
Also, from what’s written in signal it’s IMO not really all too obvious what many consequences this has.
But you are right, that the root “problem” are of course the async exceptions.
I guess that would be POSIX only, but couldn’t one simply:
btw: Would it make sense to open some bug for this? I mean, as far as I understand you, this should be fixed in Python and the libs, not by users manually doing tricks like the SIG_BLOCK above, so that eventually ExiStacketc. work safely out of the box… and it would be nice one had an issue that one could track.
But even then, I guess ExitStack.callback could never be fixed (without the user manually doing stuff)… so maybe at least there it might make sense to add documentation to contextlib? Like if the callback function can deal with it gracefully, it might even already be enough to simply register it before the actual resource has been acquired.
This is a known and discussed problem that unfortunately is very hard to fix. Nathaniel Smith @njs did some very thorough blog posts on the subject during his work on trio which led to some fixes and I think a solution that allowed ContextManagers in C to be able to behave atomically but for those in python I it was substantially harder since they involve multiple op-codes. Sorry I don’t have time to provide direct references right now but do a search for those as well as related reports before opening a new one.
But just to be on the sure that I understand correctly…
with-blocks (and thus obviously also users of that like contextlib.ExitStack and try/finally are all actually not really safe in the commonly assumed way, i.e. not whenever signals come into play (that is: even if there are no custom signal handlers, but merely a KeyboardInterrupt), right?
Or is it only if the signals cause exceptions?
And should this eventually be fixed on the Python side, or should people work around this (like by using SIG_BLOCK… though I’m not even sure whether that would actually help in all possible cases)?
I’ll happily be corrected by someone more up to date since I know there was some discussions of fixing some aspects of it but in the presence of signals or other sources of asynchronous exceptions (for example using ctypes and `PyThreadState_SetAsyncExc`) I think the concensus was that there was a small risk for those happening at the absolutely wrong time breaking invariants.
This is the blogpost I remembered and while it’s old I don’t think everything was completely resolved.
If you really want to avoid it I’ve seen examples of running all the real program code in a thread since only the main thread is interrupted by signals which can then signal the background thread to abort in a program specific way.
One workaround I can think of is to use a wrapper context manager to set flags that keep track of whether __enter__ and __exit__ have been called for a given context manager. The flags should be set in finally clauses to ensure atomicity. An outer context manager would then help the cleanup by calling the exit method if the flags indicate that the context manager exited with __enter__ called but not __exit__:
class SafeCMWrapper:
def __init__(self, cm):
self.cm = cm
self.entered = False
self.exited = False
def __enter__(self):
try:
return self.cm.__enter__()
finally:
self.entered = True
def __exit__(self, *exc_info):
try:
return self.cm.__exit__(*exc_info)
finally:
self.exited = True
class SafeCM:
def __init__(self, cm):
self.wrapper = SafeCMWrapper(cm)
def __enter__(self):
return self.wrapper
def __exit__(self, *exc_info):
if self.wrapper.entered and not self.wrapper.exited:
return self.wrapper.cm.__exit__(*exc_info)
Usage (given a context manager cm):
with SafeCM(cm) as safe_cm, safe_cm:
...
The usage isn’t very pretty, but it should solve the issue.
Can this be safe, if - as I understood it from the above comments - finally is not respectively async exceptions may pop up anywhere in the main thread?
Unless there is something that prevents an exception between entering the finally block and the instruction setting the flag there’s still a tiny window for race conditions there.
If there’s an exception during the inner enter the flag will be set but the exit method will still not run and even if you would try to catch that in enter and run the inner cm’s exit in that case it has no idea of what parts of enter was actually completed.
Doesn’t the same problem even exist if the async exception comes “out of thin air”, when one already is in the finally block? I mean it would interrupt that (and thereby and cleanups), wouldn’t it?
stack = ExitStack()
try:
x = stack.enter_context(cm)
except Exception:
# handle __enter__ exception
else:
with stack:
# Handle normal case
Maybe I miss something, but even if with blocks and finally were safe, that seems fundamentally flawed.
If any async exception comes out of thin air after the assignment of x (either in the except clause or before having actually entered the with block, I’d say there’s nothing that ever cleans up the exit stack, or is there?
You’re correct in that it wouldn’t get cleaned up but it wouldn’t really matter since exit stacks don’t really have anything to clean up except their contained contexts.
It would mean that cm was entered fully though and won’t be exited unless the except block handles that case manually (it may clean up idempotently for example)
Short of blocking signals or running in a background thread I think you’ll always have this problem. The reason ot still mostly works is that normally you don’t recover and continue from asynchronous and the regions there’s a risk are generally very small and fast.
