Class-level private property isolation

This proposal is the abridgment of private_attribute_cpp:

Abstract

Python now use name-magging on the name with pre-double underline to create the namespace for this class. However, when two classes have same name (or the difference is just the number of prefer underline), if they appear in one inheritance chain, there will be still conflict. You can use these codes to reqppear this problem (this example has been changed so some comments may seems to be strange):

One module export PublicClass and public_method in its document and pyi file:

class PublicClass:
    def __init__(self): ...
    def start(self) -> None: ...

def public_function(obj: PublicClass): ...

The document says that the user should inherit the PublicClass and define its custom start method. Finally the public_function will call it.

For simple maintenance, you use the Core and MainClass to split different logic in different way:

from module import PublicClass, public_function

class Core(PublicClass):
    def __init__(self):
        super().__init__()
        self.__state = 0

    def start(self):
        super().start()
        self.__state += 1


class MainClass(Core):
    # some other codes
    ...

a = MainClass()
public_function(a)

However, you get:

Traceback (most recent call last):
  File "C:\Users\hh180\OneDrive\Desktop\fake_projects\conflict_by_private_class\script.py", line 18, in <module>
    public_function(a)
  File "C:\Users\hh180\OneDrive\Desktop\fake_projects\conflict_by_private_class\module.py", line 19, in public_function
    obj.start()
  File "C:\Users\hh180\OneDrive\Desktop\fake_projects\conflict_by_private_class\script.py", line 10, in start
    self.__state += 1
TypeError: can only concatenate str (not "int") to str

The reason is that the author also did this split:

class _Core:
    def __init__(self):
        self.__state = "uninit"

    def start(self):
        self.__state = "init"


class _BaseClass(_Core):
    # some code
    ...


class PublicClass(_BaseClass):
    # some code
    ...

def public_function(obj: PublicClass):
    obj.start()

The conflict is: under the class _Core and Core, the name magging for one attribute name still gets same name.

Proposal

This proposal use __private_attributes__ declaration with type-level namespace to fix it.

All heaptypes have the attribute __private_attributes_dict__ to store the private attribute for different instances.

Usage

In compile time, the interpreter will bind the class-level codes to this class. For example:

class Example:
    class InExample:
        def some_function(self): ... # This code will be binded to only "InExample"

    def __init__(self): ... # This code will be binded to "Example"

    def some_function(self): # This code will be binded to "Example"
        a = lambda: ...  # This code will be binded to "Example"
        def b():   # This code will be binded to "Example"
            def in_b(): ...  # This code will be binded to "Example"
        class C:
            def d(self): ...  # This code will be binded only to "C"

        e = (i for i in something)  # This code will be binded to "Example"

When define the class, you can define “__private_attributes__” to declare which attributes will be stored to private dict:

class MyClass:
    __private_attributes__ = ("_secret",)
    def __init__(self, secret):
        self._secret = secret

    @property
    def secret(self):
        return self._secret

Now you can get:

>>> a = MyClass(1)
>>> a._secret = 2
>>> a._secret
2
>>> a.secret
1

Outside of the class it will visit __dict__ to store or load. But in the code of the class it will visit cls.__private_attributes_dict__[id(self)][name].

Subclasses will inherit this attribute. If subclasses define this too, they will be merged:

class SubClass(MyClass):
    __private_attributes__ = ("_other_secret",)
    def __init__(self, secret1, secret2, secret3):
        super().__init__(secret1)
        self._secret = secret2
        self._other_secret = secret3

    def get_both_secret(self):
        return self.secret, self._secret, self._other_secret

Then you will get:

>>> b = SubClass(1, 2, 3)
>>> b.secret
1
>>> b.get_both_secret()
(1, 2, 3)

You can use SubClass.__private_attributes_dict__[id(b)][name] to change the private attribute of b outside of the SubClass and MyClass.__private_attributes_dict__[id(b)][name] to change the private attribute of b outside of the MyClass.

Compare

To be compared with the name magging, this way will be safer because the namespace isolation is related on the class itself instead of the name.

Notes

  • Defining the name in __private_attribute_dict__ that stards and ends with “__” may cause the problems. Don’t do that.
  • The custom __getattribute__, __getattr__, __setattr__, __delattr__ will not work if the name for the code of this frame is the private attribute of the class. This will protect the parent class.
  • When the object is collected, all classes on this object type’s inheritance chain will remove the key of this object’s id.

I know this doesn’t solve your problem, but you may want to consider that this may be a misuse of inheritance. Often when you have multiple parent classes with stored data, you should probably be using composition rather than inheritance.

5 Likes

This may not the solutions for the next situation:

One module declared

# module.pyi
class PublicClass:
    def __init__(self): ...
    def start(self) -> None: ...

Sometimes you need to inherit this class, because one function in this module declared that it will call custom start method for PublicClass (or more). For ease of maintenace, you used different subclass to implement different requirements:

from module import PublicClass

class Core(PublicClass):
    def __init__(self):
        super().__init__()
        self.__state = 0

    def start(self):
        super().start()
        self.__state += 1


class MainClass(Core):
    # some other codes
    ...

a = MainClass()
a.start()

It seems to be okay (tools reported 0 error and 0 warning), but it failed:

Traceback (most recent call last):
  File "C:\Users\hh180\OneDrive\Desktop\fake_projects\conflict_by_private_class\script.py", line 18, in <module>
    a.start()
  File "C:\Users\hh180\OneDrive\Desktop\fake_projects\conflict_by_private_class\script.py", line 10, in start
    self.__state += 1
TypeError: can only concatenate str (not "int") to str

The reason is that in real implemention of the module, it depends on _Core which is defined by module author, which also thought that spliting different level codes in different classes is friendly for maintenace.

For users, “known that I need to reconstruct the code to avoid the conflict with module private realization” is difficult because users will read the document first. It is wrong for a way that need users to understand the details of module implementation.

Where’s the multiple inheritance?

Is there a reason that the behavior of name mangling can’t be adjusted to eliminate the naming conflict issue? I’m not super familiar with name mangling so I’m not sure whether that would be an implementation detail that can be safely changed, or something that would break backwards compatibility.

Just need a long chain.

The full chain is: module._Coremodule._BaseClassmodule.PublicClass__main__.Core__main__.MainClass.

For module._Core and __main__.Core, they defined __status as the private attribute name. Python does name magging and let them become same name _Core__status

Oh, I see you’re reusing the name, but there’s no branching like in your top level idea. Okay, then never mind!

But I think you should change your top level idea to get rid of the multiple inheritance if you don’t need it to justify your idea. Presumably, all you need is the name mangling to use the class name and file?

This change will break millions of projects. So it still needs a new way to solve it.

Use composition instead of inheritance :wink:

You can see this example:)

The combination is not suitable for every situation

How does it do that?

In this proposal, the namespaces are splited by class itself instead of name, which can avoid this problem.

Yes, but why isn’t this problem fixed by just enriching the mangling algorithm with more information?

Now it is too late. Many projects related this behavior. Changing this feature will break them. In fact, just add file name is not enough (maybe the classes are in different core.py).

How does it break them? I don’t undersand. Can you give a MWE of something that would be broken by changing the name mangling algorithm?

Many projects have do it:

attr.ibs that will be __name-mangled should be attrs-init-mangled to <name> not ClassName_<name> · Issue #619 · python-attrs/attrs · GitHub (use the name without prefix “_”, maybe the feature of the module)

Using /\._[A-Za-z][A-Za-z0-9_]*__[A-Za-z_][A-Za-z0-9_]*/ language:Python to search you can search many.

1 Like

It’s called “name mangling”, not “name magging”.

I think there are two separate issues here.

A) Classes _Spam and Spam mangle their identifiers with the same prefix, thus causing unintended collisions:

class _Spam: __ham = 1
class Spam: __ham = 1
dir(_Spam)  # ['_Spam__ham', ...]  why not __Spam__ham ?
dir(Spam)   # ['_Spam__ham', ...]

B) If multiple classes in an inheritance tree share the same name (or similar name, see A), name mangling will create collisions.

The automatic name mangling is rather primitive. Perhaps it should use the (fully qualified?) module name instead/in-addition-to the classname for prefixing. I don’t know if changing this would be considered backwards incompatible, because I don’t know what guarantees Python makes about mangled names.


I think it’s worth remembering that name mangling is just a compsci technique that you can apply manually. You don’t have to rely on Python’s automatic name mangling.

Generally, namespaces are preferred, e.g. via composition. However, if you want to stick with inheritance, you could also simply use an internal namespace:

class _Core:
    def __init__(self):
        super().__init__()
        self._internal = types.SimpleNamespace(state="uninit")

    def start(self):
        self._internal.state = "init"

class Core(_Core):
    def __init__(self):
        super().__init__()
        self.state = 0

    def start(self):
        super().start()
        self.state += 1

a = Core()
a.start()
assert a.state == 1
assert a._internal.state == "init"

I like this more than your proposal with __private_attributes__. If there was anything I would change, then that is how name mangling works or were made customizable.

This way is great if these codes are written by yourself. However, my example is “the module is written by another author”, which means that you don’t know what will happen if change its module’s code.

Well in that case things are clear. If the class is not designed for inheritance, you shouldn’t inherit from it, but use composition.

2 Likes

Inheritance is extremely close coupling. You do not generally want to inherit from someone else’s object unless (a) you have a close connection to it, usually because you’re in control of both; or (b) inheritance is the documented API, with very clear usage patterns and strong backward compatibility guarantees.

There is basically no use case for “I want to inherit someone else’s thing and pretend that we’re loosely coupled”. That’s what composition is for.