from etils import epy
class A(epy.ContextManager):
def __contextmanager__(self) -> Iterable[A]:
yield self
with A() as a:
pass
But my implementation is likely not thread safe, nor support slots. Having a implementation be part of the language or in contextmanager would be a huge usability win.
A generator function __with__ can be convenient when implement in Python, but how do you implement it in C?
For every __with__ method you need add a new class for its result. It should implement methods __iter__, __next__, close and throw, not counting constructor, destructor and auxilary slots like tp_traverse and tp_clear. It is much more complex than implementing just two methods __enter__ and __exit__.
C does not support closures and coroutines, so all should be implemented as a state mashine with several plain methods called in corresponding states.
I don’t think it is that an issue. Yes, from C perspective two methods are more convenient, I thought about something between those lines but in C
type EnterFunc = Callable[[object], PyObjectWithContext]
type LeaveFunc = Callable[PyObjectWithContext], None] # clear exception if want to suppress
class PyTypeObject:
tp_enter: EnterFunc | NULL
tp_leave: LeaveFunc | NULL
# Both are implemented or neither
class PyObjectWithContext:
leave_data: ( # maybe kind + obj
GeneratorType | # generator returned by __with__()
MethodType | # bound __exit__ method
*void) # Data for a C-defined leave, maybe this class be a subclass of PyTuple?
# What else do we need? Something for interpreter/thread/asunc safety?
PyObjectWithContext PyObjectEnterContext(obj):
if type(obj).tp_enter is not NULL:
return type(obj).tp_enter(obj)
if hasattr(obj, "__with__"):
try:
gen = obj.__with__()
except:
return NULL
try:
enter_result = gen.send(None)
except Stopiteration:
raise RuntimeError("__with__ must yield")
return NULL
return PyObjectWithContext(gen, enter_result)
# Old protocol
if hasattr(obj, "__enter__") and hasattr(obj, "__exit__"):
try:
enter_result = obj.__enter__()
except:
return NULL
return PyObjectWithContext(obj.__exit__, enter_result)
raise TypeError(f"{type(obj)} object does not support the context manager protocol")
return NULL
int PyOjbectLeaveContext(context: PyObjectWithContext):
# return 1 on error, probably need a 3-state return.
if type(obj).tp_leave is not NULL:
return type(obj).tp_leave(context)
exception = PyErr_GetRaisedException()
if isinstance(context.leave_data, GeneratorType):
if exception is None:
try:
context.leave_data.send(None)
except StopIteration:
return 0
else:
raise RuntimeError("__with__ must not yield a second time")
return 1
else:
context.leave_data.throw(exception)
return 0 # Exception can be set to old exception or new exception or be cleaned
else: # __exit__ case
try:
if exception is None:
context.leave_data(None, None, None)
else:
suppress = context.leave_data(type(exception), exception, exception.__traceback__)
if suppress:
PyErr_Clear()
return 0
except:
return 1
So, if extension type want to support the protocol they can
Define __with__ returning correct generator object, which is hard to do from C
Define __enter__/__exit__ as they do now.
Define tp_enter that createPyObjectWithContext, fill up it with data they need in leave and return it, AND define tp_leave that takes PyObjectWithContext that handles curretly raised exception.
If we don’t want to add 2 new slots to a PyTypeObject, extentions still has an option 2 to implement the protocol, but this way they can’t have an access to enter result or other data that enter would like to pass to leave.
As folks have noted, __with__ was actually there in the original context management implementation, but we took it out prior to release.
It wasn’t anything technical that killed it, but a terminology problem: what do you call an object that can produce a context manager for itself, but is not itself a context manager? (the removal decision for the alternate method happened after I raised concerns about the launch documentation for context managers getting the distinction between the two confused)
For the iterator protocol, the terms are clear: iterators handle the actual iteration, iterables can produce an iterator via a standard protocol when asked to do so.
For context management, context managers are the iterator equivalent, but we’ve still never come up with a good counterpart to the “iterable” term. One plausible option (if a bit CS jargony) would be “context manager factory”, but I can’t say I love that one.
I’m generally +1 on the overall idea, though. It would let us get rid of a bit of nasty hackery in contextlib when it comes to making reusable and re-entrant context managers (allowing the public CM factory protocol to be used instead of the existing private protocol)
I don’t like it much either. The term “factory” has become the butt of
jokes in recent times, due to its overuse in certain circles. Although
the same could probably be said about “manager”.
If an object that can be used with await is called an “awaitable
object”, maybe an object that can be used in a with-statement should be
called a “withable object”?
I don’t think the analogy holds, as await is used as a verb expression, so awaiting/awaited/awaitable all work as regular meaningful words in a way that withing/withed/withable don’t (since with is a preposition rather than a verb).
If “managable” wasn’t such a pain to spell and pronounce and “managed” wasn’t so spectacularly non-specific, the terminology problem wouldn’t be as challenging as it is (if I recall correctly, the terminology I was proposing to fix the ambiguity in the docs before Guido killed the __with__ method prior to release was “context managers” and “managed objects”).
(I did just make myself laugh by thinking “Since they’re managed by context managers, maybe they could be called ‘context minions’”?)
Come to think of it, isn’t the name “context manager” wrong in the first place?
To me, that sounds like the purpose is to maintain some invariant of the code before and after the with block (the context in which that block appears). But that isn’t it at all. Neither is it maintaining an invariant of the code within the block (if we call that block the “context”) - because e.g. contextlib.closing doesn’t prevent the code inside the block from calling .close on the object; rather it ensures that .close is called as the block exits.
What it’s really doing is maintaining an invariant of the object mentioned in the with statement: something will reliably happen to it to clean it up, comparable to __del__ but without waiting for garbage collection (and thereby creating a pseudo-scope).
That sounds to me more like a “resource manager” instead.
Yeah, that case can definitely be made. The difference I see between this idea and iterators is that for iterators, “implement __iter__ to return self” was part of the iterator protocol from the start, whereas here it would be a later (optional) addition to the context management protocol, so checking it second makes it clearer that the method is redundant for context managers.
While resource management is a common context management use case, it’s far from the only use case as context managers are a general purpose tool for extracting any recurring code pattern previously handled via try/except/finally. Even just in the standard library, there are the subtest CMs in unit test, exception handling CMs in contextlib, block timing CMs, and more.
PEP 343 links to [Python-Dev] 'With' context documentation draft (was Re: Terminology for PEP 343 as the point where we decided that we liked the context management terminology well enough to settle on it as the path forward: narrow enough to be meaningful, but not so narrow as to sometimes be misleading when considering the potential use cases.
My objection is that I don’t find the term meaningful regardless of how narrow or broad it is. The existence of a “context manager” implies the existence of a “context” to manage; but I can’t tell what section of the code, object, or other concept within the code is supposed to be described by that term (and indeed, I can’t tell which category of thing the term “context” is intended to refer to in… well, this context).
It’s genuinely and intentionally vague: all the “context management” terminology tells you is that there is some aspect of behaviour which is being manipulated before and/or after the suite in the with statement body is executed.
Since that manipulation is literally executing arbitrary code, attempting to use more precise terminology than that is intrinsically misleading as to the scope of what context managers make possible (hence the extended discussions before settling on context management).
I do not care that we lack a nice term for this kind of class.
On the existing prior art front, like @Conchylicultor I also rolled a
mixin for this kind of thing (ContextManagerMixin from the cs.context module) which provides a __enter_exit__ method, which is
a generator; mine allows the usual single yield and also an optional
second yield to provide a return value for __exit__ (I almost
never use the second yield).
All because keeping state between enter and exit is a ridiculous PITA
when they’re two methods, whereas having a single method lets you do all
the usual nested context managers for using/freeing resources around the yield, and/or keeping info in local variables across the yield.
Open questions for me is how this would work with inheritance. Ideally
I think this should do expected thing
[...]
class ParentHandlesFirst(Parent): # Handle if parent reraised
def __with__(self):
try:
with super() as result:
print("ParentHandlesFirst enter")
yield result
except:
print("ParentHandlesFirst exception")
else:
[...]
Currently super does not support context manager.
We can not use yield from because it is a SyntaxError in async def.
My own kludge for this is an as_contextmanager class method whose
example use from the docstring looks like:
class RunState(HasThreadState):
.....
def __enter_exit__(self):
with HasThreadState.as_contextmanager(self):
... RunState context manager stuff ...
Here we’re using the parent class HasThreadState.as_contextmanager(self)
to make use of the parent’s __enter_exit__ (aka __with__). It is
cumbersome but does work. You do need to nest them if you inherit from
two such classes:
def __enter_exit__(self):
with HasThreadState.as_contextmanager(self):
with MultiOpenMixin.as_contextmanager(self):
yield
