Implicit initialisation of inherited attributes

In Python, I have noticed that the __init__ constructor of a class does not implicitly call the __init__ constructor of its base class. Indeed, this program:

class A:
    def __init__(self): print("A")

class B(A):
    def __init__(self): print("B")

b = B()

outputs:

B

This contrasts with C++ for which the constructor of a class implicitly calls the default constructor of its base class. Indeed, this program:

#include <iostream>

struct A {
    A() { std::cout << "A" << std::endl; }
};

struct B: A {
    B() { std::cout << "B" << std::endl; }
};

int main() {
    B b {};
    return 0;
}

outputs:

A
B

I see two benefits with the behaviour of Python: __init__ behaves like any other methods and everything is explicit.

But I see also one drawback: we can get runtime errors (AttributeError) when a class calls inherited methods that use uninitialised attributes, because the constructor of the class did not explicitly call the constructor of its base class. Indeed, this program:

class A:
    def __init__(self): self.x = 0
    def f(self): return self.x

class B(A):
    def __init__(self): pass

b = B()
b.f()

outputs:

[…]
AttributeError: 'B' object has no attribute 'x'

The problem is obvious in this program since we can see the implementation of the base class, but when we subclass a base class defined in another module I imagine it could be more pernicious.

For explicitly calling the __init__ constructor of a base class, instead of using the base class directly we can use the super class in order to allow cooperative multiple inheritance at the same time. Like explained in Raymond Hettinger’s excellent article Python’s super() considered super!, this requires:

  • calling super().__init__ in each __init__ constructor of the inheritance graph;
  • matching caller arguments and callee parameters (by having each __init__ constructor take the keyword parameters that it uses and a variadic keyword argument that it forwards to the super().__init__ base constructor).

Indeed, this program:

class A:
    def __init__(self, a, **kwargs):
        print("A", a, kwargs)
        super().__init__(**kwargs)

class B(A):
    def __init__(self, b, **kwargs):
        print("B", b, kwargs)
        super().__init__(**kwargs)

class C(A):
    def __init__(self, c, **kwargs):
        print("C", c, kwargs)
        super().__init__(**kwargs)

class D(B, C):
    def __init__(self, d, **kwargs):
        print("D", d, kwargs)
        super().__init__(**kwargs)

D(a=1, b=2, c=3, d=4)

outputs:

D 4 {'a': 1, 'b': 2, 'c': 3}
B 2 {'a': 1, 'c': 3}
C 3 {'a': 1}
A 1 {}

If Python was designed in such a way that the __init__ constructor of a class implicitly called the __init__ constructor of its base class, the previous program would look like this:

class A:
    def __init__(self, a):
        print("A", a)

class B(A):
    def __init__(self, b):
        print("B", b)

class C(A):
    def __init__(self, c):
        print("C", c)

class D(B, C):
    def __init__(self, d):
        print("D", d)

D(a=1, b=2, c=3, d=4)

Finally here are my questions:

  1. Would such a language design be possible?
  2. Would such a language design be desirable?

Not only is it possible, but you can do it right now in your own classes: you could create your own meta-class which builds the instantiation parameters (in __new__) by inspecting and passing them to each base classes’ __init__. To stay explicit, you might want to call the meta-class ImplicitParentInit, and have it only pass to base classes which have this meta-class (or a specific attribute). I’ve implemented an example below.

This wouldn’t be incorporated into standard Python, firstly because it’s breaking (duh), but also because it’s not explicit; try import this.

The way I would implement your example above is to use dataclasses and inheritance (although optical parameters will get you: I have a PR fixing that). You can still put methods on dataclasses, but you get __init__ and __repr__ for free.

Hi GĂ©ry,

As you point out, the same applies for every method. __init__ and

__new__ are not special.

You give some example code using super():


class A:

    def __init__(self, a, **kwargs):

        print("A", a, kwargs)

        super().__init__(**kwargs)

 

class B(A):

    def __init__(self, b, **kwargs):

        print("B", b, kwargs)

        super().__init__(**kwargs)

(For brevity I have ignored C and D classes.)

You say:

"If Python was designed in such a way that the __init__ constructor

of a class implicitly called the __init__ constructor of its base

class, the previous program would look like this:"


class A:

    def __init__(self, a):

        print("A", a)



class B(A):

    def __init__(self, b):

        print("B", b)

and then ask if such a design was possible and desirable.

Possible, I guess so, but it would be expensive.

Desirable, no.

Even in this simplified, shallow class hierarchy, we can see why this

would be undesirable.

Where should the implicit call to super() go? Before or after running

the current method? In the above example, that makes a difference

between printing “B” first or “A” first.

Whichever choice you choose, some subclasses will want the opposite.

And what about subclasses that intentionally don’t call super, because

they want to completely override the superclass constructor? Failing to

call super is not necessarily a bug, sometimes it is intentional. You

will break them. Likewise for classes that already call super,

explicitly.

This would also be a useability nightmare. As the caller of this class,

how am I supposed to know that I cannot call B(b=2) but have to supply

an undocumented parameter that doesn’t show up in the constructor

signature?

"I am trying to instantiate a class that takes a single parameter, but

whenever I call it I get a TypeError about missing arguments."

What happens if the caller passes arguments positionally rather than as

keyword? B(1, 2) rather than B(a=1, b=2)?

It would also slow down class instantiation. The interpreter would have

to try to allocate arguments in the normal way:

B(a=1, b=2)

which would fail. It would then have to search the MRO of B looking for

a class that takes a parameter a. Having found one, it then has to

remove a from the arguments, and try calling B.__init__ again.

Repeat each time it fails until there are no extra parameters.

Once all the parameters are allocated for B, then it would have

to call B.__init__, implicitly calling super() with those extra

arguments that had been removed, and repeat the process for every

superclass in the MRO.

And this whole process has to be repeated on every call to B() because

the constructor methods are resolved at runtime, not compile time. It’s

not like statically typed languages where the compiler can resolve the

calls once, at compile time. It has to be done on every call.

What about alternative constructors? It would be very odd, and

confusing, if this magical behaviour applied to B() but not alternate

constructors like B.from_items().

1 Like

I’ve implemented a metaclass that achieves your syntax:

import inspect

class ImplicitParentInitMeta(type):
    def __call__(cls, **kwargs):
        self = object.__new__(cls)
        for cls_ in cls.__mro__:
            if isinstance(cls_, ImplicitParentInitMeta):
                sig = inspect.signature(cls_.__init__)
                kw = {n: kwargs[n] for n in sig.parameters if n in kwargs}
                cls_.__init__(self, **kw)
        return self

class A(metaclass=ImplicitParentInitMeta):
    def __init__(self, a):
        print("A", a)

class B(A, metaclass=ImplicitParentInitMeta):
    def __init__(self, b):
        print("B", b)

...
1 Like

Thanks a lot for this neat implementation @EpicWink! Metaclasses are indeed very powerful.

Your implementation follows the C++ design which allows each constructor to forward the arguments that it uses, and therefore allows multiple classes in the inheritance graph to use the same arguments. Indeed, this C++ program:

#include <iostream>

struct A {
    A(int x) { std::cout << "A(" << x << ")" << std::endl; }
};

struct B: A {
    // Constructor B forwards the argument x that it uses
    B(int x): A(x) { std::cout << "B(" << x << ")" << std::endl; }
};

int main() {
    B b {3};
    return 0;
}

outputs:

A(3)
B(3)

On the other hand, in @rhettinger’s implementation of cooperative multiple inheritance, each __init__ constructor strips off the arguments that it uses from the keyword argument dictionary to forward, and therefore prevents multiple classes in the inheritance graph from using the same arguments.

Here is a slight variation on your implementation @EpicWink, but following @rhettinger’s design:

import inspect

class BaseInitializer(type):
    def __call__(cls, **kwargs):
        instance = cls.__new__(cls, **kwargs)
        if isinstance(instance, cls):
            for cls_ in cls.__mro__:
                if isinstance(cls_, BaseInitializer):
                    sig = inspect.signature(cls_.__init__)
                    kwargs_ = {
                        p: kwargs.pop(p)
                        for p in sig.parameters if p in kwargs
                    }
                    cls_.__init__(instance, **kwargs_)
        return instance

class A(metaclass=BaseInitializer):
    def __init__(self, a):
        print("A", a)

class B(A, metaclass=BaseInitializer):
    def __init__(self, b):
        print("B", b)

class C(A, metaclass=BaseInitializer):
    def __init__(self, c):
        print("C", c)

class D(B, C, metaclass=BaseInitializer):
    def __init__(self, d):
        print("D", d)

D(a=1, b=2, c=3, d=4)

It would be interesting to have @rhettinger’s motivations for preventing multiple classes in the inheritance graph from using the same arguments. I guess that most of the time we do not want an object to have different copies of the same attribute (the one owned and the ones inherited), but should it be the rule?

Thank you @steven.daprano for the very sensible remarks. I think @EpicWink’s metaclass approach is a very nice solution to implement implicit base initialization now while remaining optional.

Remember that assignment doesn’t copy in Python, rather just creates a new reference to the assigned object. Seeing as the majority of the time you assign the argument to the same name in the unit, you’ll likely just overwrite with the same thing.

def __init__(self, a):
    self.a = a

You are right @EpicWink, it is an assignment. Which makes things worse, because now you get aliasing issues with inherited attributes sharing values. For instance, your ImplicitParentInitMeta metaclass allows this kind of Python program:

class A(metaclass=ImplicitParentInitMeta):
    def __init__(self, a):
        self.__x = a
    def f(self):
        self.__x[0] += 1

class B(A, metaclass=ImplicitParentInitMeta):
    def __init__(self, a):
        self.__y = a
    def g(self):
        return self.__y

b = B(a=[1])
print(b.g())
b.f()
print(b.g())

which outputs:

[1]
[2]

This might be surprising. But C++ allows the same kind of vertical attribute interaction anyway. Indeed this C++ program:

#include <iostream>

struct A {
    A(int& a): x{a} {}
    void f() { ++x; }
  private:
    int& x;
};

struct B: A {
    B(int& a): y{a}, A{a} {}
    int g() { return y; }
  private:
    int& y;
};

int main() {
    int a{1};
    B b{a};
    std::cout << b.g();
    b.f();
    std::cout << b.g();
    return 0;
}

outputs:

12

Was thinking in my example you have the same name for the instance attribute as the instantiation argument:

class A(metaclass=ImplicitParentInitMeta):
    def __init__(self, x):
        self.x = x

class B(A, metaclass=ImplicitParentInitMeta):
    def __init__(self, x):
        self.x = x

b = B(3)
print(b.x)  # 3

Oh I see, we were talking about two different issues that could result from passing the same argument list to each class’s __init__ in the inheritance graph: I was talking about instance attribute aliasing, while you were talking about instance attribute overriding. But contrary to the former, instance attribute overriding can occur even when you don’t pass the same argument list to each class’s __init__ in the inheritance graph, like prescribed by @rhettinger’s collaborative multiple inheritance (so that cannot be his motivation):

class A:
    def __init__(self):
        self.a = 1
        self._b = 2
        self.__c = 3

class B(A):
    def __init__(self):
        self.d = 1
        self._e = 2
        self.__f = 3

class C(A):
    def __init__(self):
        super().__init__()
        self.d = 1
        self._e = 2
        self.__f = 3

class D(A):
    def __init__(self):
        self.d = 1
        self._e = 2
        self.__f = 3
        super().__init__()

class E(A):
    def __init__(self):
        self.a = 4
        self._b = 5
        self.__c = 6

class F(A):
    def __init__(self):
        super().__init__()
        self.a = 4
        self._b = 5
        self.__c = 6

class G(A):
    def __init__(self):
        self.a = 4
        self._b = 5
        self.__c = 6
        super().__init__()

b = B()
c = C()
d = D()
e = E()
f = F()
g = G()
assert vars(b) == {'d': 1, '_e': 2, '_B__f': 3}
assert vars(c) == {'a': 1, '_b': 2, '_A__c': 3, 'd': 1, '_e': 2, '_C__f': 3}
assert vars(d) == {'a': 1, '_b': 2, '_A__c': 3, 'd': 1, '_e': 2, '_D__f': 3}
assert vars(e) == {'a': 4, '_b': 5, '_E__c': 6}
assert vars(f) == {'a': 4, '_b': 5, '_F__c': 6, '_A__c': 3}
assert vars(g) == {'a': 1, '_b': 2, '_G__c': 6, '_A__c': 3}

As it feels more natural to me that a derived class overrides an inherited instance attribute than the contrary (in other words that super().__init__ be called before rather than after), here is a new implementation of your metaclass (the only difference is the reversed call):

class ImplicitParentInitMeta(type):

    def __call__(cls, **kwargs):
        instance = cls.__new__(cls, **kwargs)
        if isinstance(instance, cls):
            for cls_ in reversed(cls.__mro__):
                if isinstance(cls_, ImplicitParentInitMeta):
                    sig = inspect.signature(cls_.__init__)
                    kwargs_ = {
                        p: kwargs[p]
                        for p in sig.parameters if p in kwargs
                    }
                    cls_.__init__(instance, **kwargs_)
        return instance

I have to wonder why you’d want something like this to be in the language. Although you can make this kinda thing happen with metaclasses and such like mentioned here, this is a complex behavior that would be difficult to explain to someone that wasn’t intimately familiar with the details.

I have to wonder…why not just try to simplify how you do your __init__ methods and make them more independent? This isn’t C++; you don’t have to allocate a ton of resources just to make an object. And as mentioned above, Python style typically prefers a more explicit way of handing those things when you need it. What problem are you trying to solve (at a higher level than “make the language act like I want it to”)? Perhaps there’s a more idiomatic way of approaching it

Like explained in the original post, my motivation for exploring implicit initialisation of inherited attributes was:

  1. Avoiding runtime errors ( AttributeError ) when a class calls inherited methods that use uninitialised attributes.
  2. Making cooperative multiple inheritance automatic.

But I did not face any real problem. I just wanted to better understand the Python inheritance model (its strengths and weaknesses) by discussing this idea with other developers to see where it leads and at least learn a few things on the way. @steven.daprano did the intellectual exercise very well and @EpicWink even provided an implementation.

With @EpicWink’s metaclass, we have the choice to use implicit initialisation or not, and it could support a keyword boolean argument init_after in the base class list to change the order (I think before should be the default as it is more natural to override inherited attributes in derived classes than the opposite).

That is a real issue, I agree.

Yes that is an issue. Since cooperative multiple inheritance is not practical with positional arguments, they should be forbidden for instantiating classes. This is the case with @EpicWink’s metaclass.

The MRO is actually traversed once in @EpicWink’s metaclass, with __init__ called for each class in the MRO. The popping or not of keyword arguments is independent of the traversal.

I am not sure that I follow you here since B.from_items() delegates to B(), right?

GĂ©ry Ogam replied:

"Yes that is an issue. Since cooperative multiple inheritance is not

practical with positional arguments, they should be forbidden for

instantiating classes. This is the case with @EpicWink’s metaclass."

That’s a bold claim. I’ve written MI classes that accept positional

arguments. Perhaps that’s true in the most general sense of MI with

arbitrary keyword only arguments being added by subclasses, but I don’t

think its true of any use of positional arguments.

GĂ©ry:

"I am not sure that I follow you here since B.from_items() delegates

to B(), right?"

You can’t make that assumption. B.from_items() may be completely

independent, or perhaps B() delegates to from_items, or both

delegate to some other method. I’ve written classes which do all of

those things.

Remember that in Python there’s very little special about either

__new__ or __init__, and you can construct a new instance in any

method. So while it might be common for alternative constructors to

delegate to __new__, it isn’t necessary or mandatory.