Subclassing via decorators

Python Help
1

Hi there,

I use a certain base class. Then, every subclass of this base inherits all the functionalities without problem.

My issue now comes from the fact that I tried turning this into a decorator for cleaner syntax and got a weird problem with super().

After many tries I wasn’t able to debug so I give you a minimal example here:

class BaseClass: ...  # The base class

def baseclass(cls):  # The decorator
    """To emulate subclassing BaseClass."""
    return type(cls.__name__, (BaseClass,) + cls.__bases__, dict(cls.__dict__))

class SubClass(BaseClass):
    def __init__(self):
        super().__init__()

@baseclass
class Decorated:
    def __init__(self):
        super().__init__()

@baseclass
class DecoratedWithArguments:
    def __init__(self):
        super(type(self), self).__init__()  # Helping `super()`

This approach works perfectly for everything I needed except… the use of super(). Indeed, I get the following:

>>> SubClass()
<__main__.SubClass object at 0x00000241942C2D50>
>>> DecoratedWithArguments()  # Works when `super()` gets help
<__main__.DecoratedWithArguments object at 0x000002419416E210>
>>> Decorated()  # Doesn't work without arguments specification
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 6, in __init__
TypeError: super(type, obj): obj must be an instance or subtype of type

I’ve looked a both the MRO and __bases__ inside corresponding all three __init__:

SubClass:
MRO: [<class '__main__.SubClass'>, <class '__main__.BaseClass'>, <class 'object'>]
bases: (<class '__main__.BaseClass'>,)

DecoratedWithArguments
MRO: [<class '__main__.DecoratedWithArguments'>, <class '__main__.BaseClass'>, <class 'object'>]
bases: (<class '__main__.BaseClass'>, <class 'object'>)

Decorated
MRO: [<class '__main__.Decorated'>, <class '__main__.BaseClass'>, <class 'object'>]
bases: (<class '__main__.BaseClass'>, <class 'object'>)

I’m completely desperate, the only difference I could find is there, with __bases__ containing object when I manually use type (decorator approach). I naively tried to remove it indice baseclass definition, to no avail.

If anyone has any insight, it’d be greatly appreciated!!

In advance, thank you!

2

super() without arguments relies on a decent amount of magic (e.g. if you do mysuper = super you can’t use mysuper() because the name super is specialcased)

My advice: don’t use a decorator. Just use an explicit baseclass, it will save you a lot of headaches.

IIRC to get super working you need to inject a correct class cell somewhere, I believe in the class dict.

2 Likes
3

I think this should work, but didn’t test it:

def baseclass(cls):
    cls.__bases__ = (Baseclass,) + cls.__bases__
    return cls

But yeah, just inherit it normally, this one won’t call the correct __init_subclass__, for example.

4

The reason Decorated() doesn’t work is because you’re creating a new class, but its __init__ method gets the __class__ cell from the old one, so the super() call uses the old class and an instance of the new class, triggering the error.

5

Thanks everyone for your answers.

I think this should work, but didn’t test it

I don’t see why your solution would work honestly, since there’s no type machinery involved in creating the “new” class. For example, keep in mind that this is a MWE, but in reality I use my own MetaClass instead of type, since my BaseClass is actually defined using this MetaClass that provides a lot of specific functionalities.

@MegaIng @TIGirardi Do you have any good reference on class cells and their relation to super()? I’m down to reading a bit more about it and how it works, because this is very hard to debug (for example, I have no idea how to “catch” which arguments super() uses when I don’t provide them). I did try using pdb, but when I do and call super() myself, oddly enough… everything works fine haha.

In any case, thank you both for your answers!

6

Read up on the datamodel of python: 3. Data model — Python 3.13.1 documentation, especially the class creation sections. I am not sure if class cells are properly explained in there. Especially if you want to do more metaprogramming in python, most of this is IMO required reading.

Especially if you are using metaclasses, my answer continues to be: Don’t. Just use normal class creation mechanisms instead of doing a weird hybrid style where you first create a class, then disassemble it and it reassemble it in different pattern. Even if you manage to get this specific case working, you will eternally run into issues (Others I can see coming without thinking too hard: <metaclass>.__prepare__, __init_subclass__, class creation arguments, all three exhibit unavoidable changes in behavior with what you are doing)

3 Likes
7

My point is precisely not creating another class: the __class__ cell won’t refer to what you’d expect otherwise.

I also believe @MegaIng has better advice then me.

8

Thanks very much for the pieces of advice and the references. I also found this post giving a glimpse of how messy things can get, so in the end, I do think I will drop the decorator approach altogether. Nevertheless, I would be very happy ifI could make it work as a toy example to understand exactly what happens under the hood so I will definitely read about it and try a bit more.I might update the thread if anything remotely informative comes out.

Cheers!

9

Also, I see that @dataclass goes through the process of adding methods (__init__, __repr__, …) a posteriori using exec. For example at one point:

func_builder.add_fn('__init__',
                    [self_name] + _init_params,
                    body_lines,
                    locals=locals,
                    return_type=None)

However it is a bit easier in this case since they control exactly what goes into these methods but they still need to handle locals I see.

Do you think it could be a first step towards an acceptable solution? I know that it is possble to retrieve the source code in most cases using inspect.getsource and I wouldn’t mind resorting to a degraded mode (with warning) if inspect.getsource failed.

I was thinking about this:

def baseclass(cls):  # The decorator
    try:
        source = inspect.getsource(cls)
    except OSError:
        # Degraded mode where `super()` fails. This should not happen cases of interest.
        return type(cls.__name__, (BaseClass,) + cls.__bases__, dict(cls.__dict__))

    name = cls.__name__
    del cls  # Useless or not?
    new_source = # Add `BaseClass` as first parent class
    exec(new_source, <globals ?>, <locals ?>, closure=<closure ?>)
    return locals()[name]

Of course the class is created twice which can feel dirty, but that’s kind inherent to subclassing via decorating.

10

Yes, this is the part I am advising against.

You don’t actually need to call exec here, you just would need to modify all methods closures. But this is an implementation detail of CPython and might change.

Also, I think you are misunderstanding the purpose of locals in the call to exec? It is not related to the locals within the generated function, it’s primarily to capture the produced function.

11

Yes, this is the part I am advising against.

I understand. The fact is that my metaclass currently also creates a dummy class to retrieve the source code, analyze its AST and in consequence adapt the creation of the final class. So the fact that the class is alreayd created once is not a problem since I can pass it as “precomputed” and it saves me recreating a dummy one.

Also, I think you are misunderstanding the purpose of locals in the call to exec?

You’re right, I don’t really understand how to fill the arguments of exec, so I edited my snippet with “?” and modified a bit what it does to be closer to reality I think.