Wrapping all methods of builtin collections

As you can see here: https://github.com/Korijn/observ/blob/master/observ/init.py#L146

I’m trying to implement observable containers. That is, list, set and dict types that emit a signal when they are read from, and when they are mutated. I can simply overload all the methods I can think of as you can see in the code linked above, but I’d rather use something like a metaclass or a class decorator to dynamically wrap all relevant methods.

However, I am clueless on the following:

(1) how to dynamically select the “proper” set of methods to wrap? dir also yields things like __getattribute__ which I don’t think are appropriate to wrap
(2) how to determine if the method I’m wrapping is either “read” or “write” (since that determines which signal to emit)

Do you have any advice for me?

1 Like

It’s part of the protocol spec whether each function reads/updates the underlying data. The methods themselves don’t have any attribute or other inspectable which will tell you whether they read, write or both, so you’ll have to declare each function in “reads” and “writes” lists, which will need to be updated as Python is updated.

Other than that, you can construct the methods in the metaclass, as you suggest. Remember that special methods (eg __iter__, __delitem__) are not instance attributes (ie the instance’s class’ method is called with the instance, not the instance’s bound method), whereas normal methods (eg get, setdefault) are instance attributes.

PS: I guess technically you could look at the method’s code’s AST, and inspect that in __getattribute__ after calling object.__getattribute__, but that doesn’t sound fun

PPS: \_\_init\_\_ -> __init__, while __init__ -> init

1 Like

Korijn van Golen asked:

“Do you have any advice for me?”

Read the docs? Every method will tell you what it does. If all else
fails, experiment in the REPL.

There’s no programmable way to know what a method does. It could read,
or write, or do both (e.g. list.pop both mutates the list and returns
something). There is no API for telling which methods do what. You have
to find out by experiment or reading the documentation. If you want a
list, you have to make it yourself :slight_smile:

I suppose it might be possible to analyse the function object looking
for specific byte-codes, but then you need a list of what byte-codes do
what, and writing the byte-code analyser will not be a simple job.

I agree that a class decorator is the right approach. Here’s an untested
sketch of the approach I would take:

# Untested...
def signalling(cls):

    def signal_read(func):
        @functools.wraps(func):
        def inner(self, *args, **kwargs):
            if Dep.stack:
                get_dep(self).depend()
            return func(*args, **kwargs)
        return inner

    for name in cls._READERS:
        func = getattr(cls, name)
        setattr(cls, name, signal_read(func))

    # And similar for the writers.


@signalling
class ObservableList(list):
    _READERS = ['count', 'index', ...]
    _WRITERS = ['append', 'extend', ...]
    # That's it! Nothing else to add in the subclass,
    # all the logic is in the decorator.

Good luck!

1 Like

Thanks for the great advice, both! I will be working on an update soon.

Remember that special methods (eg __iter__ , __delitem__ ) are not instance attributes (ie the instance’s class’ method is called with the instance, not the instance’s bound method), whereas normal methods (eg get , setdefault ) are instance attributes.

I had actually noticed this before while debugging. Is there anything I can read about this in the docs? I am not sure I fully understand why this is the way it is and what it means.

There’s a summary in the data model docs. The main two reasons are to allow special method calling on classes, and for performance by bypassing __getattribute__.

1 Like

Got it done, if you’re interested, you can take a look here: https://github.com/Korijn/observ/blob/master/observ/__init__.py#L116

Still need to do some rigorous testing but the basics are there :slight_smile:

Thanks a lot!

By way of dynamic unit tests I was able to prove that the wrappers are complete and working!

I also specialized the wrappers some more to discern read/write/delete and if a method is by key or not (for some dict methods), and the end result is pretty awesome.

First release is on PyPI!

Thanks!