Correct usage of contextlib.ContextDecorator?

Python Help
1

I wonder if anyone with more experience in contextlib can comment on this?

I have to enter a stack of unknown depth to which requires a file object to be open, so I recycle the SQL concept of a transaction as I only want to call File.open once.

An classic minimal viable example of a problem would be this:

# <--- begin transaction
for table in tables_to_be_deleted:
	for column in table:
		for page in column:
			reduce ref count
			if ref count is zero:
				delete page
		delete column
	delete table
# ---> end transaction

where I reuse the functions (DRY) delete_column, delete_page from elsewhere:

@transaction
def delete_column(name):
    ...

@transaction
def delete_page(name):
    ...

My implementation of the “transaction” is this, where I keep track of file-handles and sessions, to avoid re-entry on file objects. I also track the stack depth, so that I know for sure when I enter an already open transaction.

class Transaction(ContextDecorator):
    sessions = {}
    handles = {}
    session_depth = {}

    def __init__(self, name, mode='r+', **kwargs) -> None:
        self.key = name, mode
        if not isinstance(name, str):
            raise TypeError
        self.key = name, mode
        if self.key not in self.sessions:
            if not pathlib.Path(name).exists():  # make it.
                h5py.File(name, "w").close()

            self.handles[self.key] = fh = h5py.File(name, mode, **kwargs)
            self.sessions[self.key] = fh.__enter__()
            self.session_depth[self.key] = 0

    def __enter__(self):
        self.session_depth[self.key] += 1
        return self.sessions[self.key]

    def __exit__(self, *args):
        self.session_depth[self.key] -= 1
        if self.session_depth[self.key] == 0:
            fh = self.handles[self.key]
            fh.__exit__(*args)
            fh.close()
            del self.session_depth[self.key]
            del self.sessions[self.key]
            del self.handles[self.key]

Questions:

  1. Is this abuse of the decorator? Is there any way that the decorator can see the name arg?
  2. Is there a better practice that this?
  3. Should I used contextlib.ExitStack instead of tracking the sessions?

Thanks again for all your help.

2

If you really want to properly borrow the idea of an SQL transaction, I’d recommend doing it with a context manager instead:

with transaction():
    for table in tables_to_be_deleted:
	for column in table:
		for page in column:
			reduce ref count
			if ref count is zero:
				delete page
		delete column
	delete table

If possible, the context manager should be designed so that, if you leave it via an exception, it rolls back the transaction instead of committing it. (I suspect that that wouldn’t work here, but at very least, document that thoroughly.)

Since you’re using ContextDecorator, your decorator code probably doesn’t even need any changes, although the depth/recursion testing might become unnecessary.

3

@Rosuav - I think your intuition was good.

I have chosen to settle with a context manager with memory, as this supports stacked re-entry whenever the initialisation arguments are repeated (in my case the HDF5 file):


class HDF5(object):
    """A context manager tracking h5py.File object sessions."""

    sessions = {}
    handles = {}
    session_depth = {}

    def __init__(self, name, mode="r+", **kwargs) -> None:
        self.key = name, mode
        if not isinstance(name, str):
            raise TypeError
        self.key = name, mode
        if self.key not in self.sessions:
            if not pathlib.Path(name).exists():  # make it.
                h5py.File(name, "w").close()

            self.handles[self.key] = fh = h5py.File(name, mode, **kwargs)
            self.sessions[self.key] = fh.__enter__()
            self.session_depth[self.key] = 0

    def __enter__(self):
        self.session_depth[self.key] += 1
        return self.sessions[self.key]

    def __exit__(self, *args):
        self.session_depth[self.key] -= 1
        if self.session_depth[self.key] == 0:
            fh = self.handles[self.key]
            fh.__exit__(*args)
            fh.close()
            del self.session_depth[self.key]
            del self.sessions[self.key]
            del self.handles[self.key]

This approach avoids numerous opening and closing of the file handle whilst the target of the transaction remains the same.

def test_h5_context_mgr():
    x = pathlib.Path(tempfile.gettempdir()) / "decafbad.h5"
    if x.exists():
        os.remove(x)
    x = str(x)

    with HDF5(x) as h5:
        h5.create_group("1")  # enter level 1.
        with HDF5(x) as h5:
            h5.create_group("1.1")  # enter level 2.
        h5.create_group("2")  # this should fail if level 2 exit closes the h5 session.

        with HDF5(x) as h5:
            del h5["1.1"]  # testing if a nested del can close all sessions
        del h5["1"]  # this should fail if previous del closed all sessions
        with HDF5(x) as h5:
            print(h5["2"])
            with HDF5(x) as h5:
                h5.create_group("3")
            print(h5["3"])

I hope this snippet can be inspiration for others.