How does getattribute with set descriptor work?

Python Help
1

My question is loosely related to Why `self.__setattr__(attr, value)` trigger `__getattribute__`?
I’m struggling to understand the relationship between getattribute and write-descriptors

in the code below I add a descriptor to the Room class - skip over the descriptor it self, it’s not relevant

but at the point where self.desc = desc, the system obviously finds out that desc is a descriptor instance; so how does it find out, since it does not appear to call my __getattribute__ to this end ?

and on a similarly disconcerting note:

  • I take it that __getattribute__ itself is not searched through __getattribute__ either, so how exactly does that work ?
  • and the fact that __dict__ on the other hand gets searched by __getattribute__ also blows my mind; I was not expecting that; I guess the descriptor should probably use object.__getattr__ and object.__setattr__ instead of messing directly into instance.__dict__, what exactly are the differences, which one is better ?
# a standard attribute descriptor
class InstanceAttributeDescriptor:

    def __set_name__(self, owner, name):
        self.public_name = name
        self.private_name = "_" + name

    def __get__(self, instance, owner):
        return instance.__dict__[self.private_name]

    def __set__(self, instance, new_value):
        instance.__dict__[self.private_name] = new_value


class Room:
    desc = InstanceAttributeDescriptor()

    def __init__(self, name, desc):
        self.name = name
        self.desc = desc

    def __getattribute__(self, attr):
        print("in __getattribute__", attr)
        return object.__getattribute__(self, attr)

    def __setattr__(self, attr, value):
        print("in __setattr__", attr, value)
        # time.sleep(3)
        object.__setattr__(self, attr, value)

    def __repr__(self):
        return f"{self.name} ({self.desc})"


# just to make sure the descriptor indeed gets found along the mro    
class SubRoom(Room):
    pass


r = SubRoom("kitchen", "where to do the cooking")
print("--")
print(r)

which yields

 in __setattr__ name kitchen
in __setattr__ desc where to do the cooking
in __getattribute__ __dict__
--
in __getattribute__ name
in __getattribute__ desc
in __getattribute__ __dict__
kitchen (where to do the cooking)

any insight would be much appreciated

2

Hi,

by coincidence, I am also on the same general area of my Pythonic studies. So, disclaimer, I am not an expert.

Because you told it so here:

 desc = InstanceAttributeDescriptor()  # Assign descriptor instance

During this assignment, the descriptor is automatically called to initialize it via the __set_name__ method.

From theory:

  1. Descriptors provide an alternative way to intercept attribute access.

    • since you have defined desc = InstanceAttributeDescriptor(), when we are accessing it via the assignment, it causes the descriptor to be called.

  2. The __getattribute__ operator overloading method allows a class to intercept ALL attribute references.

    • If you follow along the code (I have added print statements so that you may follow the trajectory of the code as it is being executed), you will note that the following line induces the __getattribute__ intercept method to get called (inside the __set__ method):
    instance.__dict__[self.private_name] = new_value

Thus, it is an automatic action that occurs without direct instructions (it is implied). It is called every time that there is an attribute fetch by the code (instance.attribute).

Here, I have stripped the code to the bare minimum to get it to call the __getattribute__ intercept method.

class InstanceAttributeDescriptor:

    # owner = <class '__main__.Room'>
    # name  = desc
    def __set_name__(self, owner, name):
        print('Inside __set_name__ method.')

        self.public_name = name
        self.private_name = "_" + name

    def __set__(self, instance, new_value):
        print('instance = ', instance)
        print('new_value = ', new_value)
        print('Inside __set__ method.')
        instance.__dict__[self.private_name] = new_value

class Room:
    print('Inside the Room class')
    desc = InstanceAttributeDescriptor()

    def __init__(self, name, desc):
        print('Inside __init__ constructor.')
        self.name = name
        self.desc = desc

    def __getattribute__(self, attr):
        print("Inside __getattribute__", attr)
        return object.__getattribute__(self, attr)

class SubRoom(Room):
    pass

r = SubRoom("kitchen", "where to do the cooking")

By the way, the output that I got was:

Inside the Room class
Inside __set_name__ method.
Inside __init__ constructor.
instance =  <__main__.SubRoom object at 0x0000028A705DCC50>
new_value =  where to do the cooking
Inside __set__ method.
Inside __getattribute__ __dict__

From:

Hopefully after running the test code and following its trajectory, it will answer your question.

3

to rephrase my question in the context of your code, why do we not see __getattribute__ being called in the constructor since we are referring to 2 attributes in there, namely self.name and self.desc

this is particularly blatant in my code, where on purpose I have the descriptor instance inserted in a superclass, and so there is a need to “go up the inheritance tree” to locate it
and I thought this was typically the job of __getattribute__ to do this kind of upwards search, but I can’t seem to intercept that…

4

So, here you are asking why we don’t see the __getattribute__ getting called twice, once for each attribute assignment: name, and desc. Well, that is because the __getattribute__ is not designed to get called during attribute initial assignments (when they’re first created). It wouldn’t make much sense since you would expect it to be called during an actual event in your code or process and NOT during its initial design phase.

Note that for the desc attribute, it is fetched, as per my previous post during this line of code AFTER having already been initialized (conversely, you can comment this line out to confirm that it is not the assignment that induces the __getattribute__ method to be called):

instance.__dict__[self.private_name] = new_value

So, any fetch after the initial assignment, is fair game. If you add the following two print statements to the constructor, fetching the two attributes, you will notice that the __getattribute__ does get called twice as per your concern.

    def __init__(self, name, desc):
        print('Inside __init__ constructor.')
        self.name = name
        self.desc = desc
        print(self.name)   # fetch attribute - induces __getattribute__ call
        print(self.desc)    # fetch attribute - induces __getattribute__ call

Modify the test code that I previously provided to have these two additional lines and observe the results.

Here is the code trajectory from the previous code that I provided:

code_path
code_path523×573 20.5 KB

5

I appreciate your help in trying to solve this issue, but it feels like you’re missing the point that I’m trying to make
Like I was trying to imply in the title of this post, my question is about understanding the role of __getattribute__ when a set descriptor is at play
that is to say, before this line
instance.__dict__[self.private_name] = new_value
is even triggered, one must first find this code which is part of the descriptor instance - which is stored inside the superclass

in fact I’m trying to reconcile

  • the necessity of a ‘searching up the tree’ in the context of an attribute being written, for locating the decsriptor
  • with the ‘simple’ notion that we teach to regular users, i.e. that ‘the attribute goes into the object itself’, which implies that there is, precisely, no ‘searching up the tree’ at play in this case

Obviously, my mental model, where I kind of associate ‘searching up the tree’ with __getattribute__, is broken somehow, and I’d like to understand how to reconcile these apparent inconsistency

as an aside, I don’t think there is any notion of

attribute initial assignments (when they’re first created).

at least I don’t believe there is any mention of that anywhere in the docs, and imho writing attributes does not depend on whether or not it has been created already; but again this is beside the point entirely as far as I can tell

6

The role is whatever you want it to be. The __getattribute__ is optional - it is NOT a requirement - that is, your descriptor code and the __getattribute__ method are not tied at the hip. Remember, you as the developer gets to decide how you’re going to approach and implement your application. Being that you are at this level of Python, I am sure you are well aware that there are more than one way of implementing things in your code. If you’re following along in your studies as I am, they are trying to show you how you can make use of the __getattribute__ intercept method within the descriptor class code. Note that in the example that you have provided, all they are doing is assigning the desc attribute a new value. What they are implying is that whenever there is a reference to that attribute, they want this additional task to be executed automatically by way of the __getattribute__ method. However, in your code, you can add whatever it is that you want (maybe a specific task, i.e., writing to a file, getting a status update, turning on equipment, etc). Of course, there are perhaps an infinite ways that they could have showed you how to make use of the intercept method along with the descriptor. Due to time and budget constraints, they have showed you one.

From theory:

The __getattribute__ operator overloading method provides still other ways to intercept attribute fetches for class instances. Like properties and descriptors, they allow us to insert code to be run automatically when attributes are accessed.

So, a descriptor assignment designates a specific attribute (in your case desc) to call a descriptor every time that there is a reference to that attribute. You can then further compound that by adding a __getattribute__ to it when there is a change or reference to that same attribute within the descriptor class code. As they say, added sugar.

If you want more theory on the matter, I am using Learning Python, 5E, by Mark Lutz. Reference discussion starting on page 1226 (section Descriptors) and page 1237 (section __getattr__ and __getattributes__). It gives a relatively good explanation how the two intercept methods are used within descriptors.

7

object.__setattr__, which is getting called when you do self.name = ... does something like this:

def PyObject_GenericSetAttr(self, name, value):
    tp = type(self)
    descr = mro_lookup(tp, name)
    if descr and descr.__set__:
        descr.__set__(self, value)
        return
    d = get_dict(self)
    if d:
        d[name] = value
    else:
       raise AttributeError

(not covering error handling, translation of the C code)

I am not entirely sure which path calls __getattribute__ for __dict__ tbh, the many different ways __dict__ might exists now are slightly confusing in modern python versions. get_dict is a shorthand for various different paths that might be taken.

8

I was slowing getting to that, thanks to this very good explainer here where I learned a lot:

you’re mentioning object.__setattribute__ ; is that a thing or are you making it up ?
I mean, I don’t think I can provide my own __setattribute__, can I ?

9

Argh, right, I got confused by the asymmetry between __getattr__, __getattribute__ and __setattr__. I meant __setattr__, which corresponds to __getattribute__ but for setting instead of reading.

10

We do see it called for self.desc = desc, since the descriptor retrieves the __dict__ of the instance, like instance.__dict__[self.private_name] = new_value. Even though it’s looking for something as special as __dict__, the code still says to check for any custom lookup logic in instance. (This is why it’s often difficult to implement __getattribute__ without an unintended unbounded recursion.)

For self.name = name, the instance’s __setattr__ is called, which delegates to object.__setattr__, which checks for (and fails to find) a name descriptor in the class (SubRoom, and then bases along the MRO). After failing to find a descriptor, it directly modifies the __dict__ of the object, which it finds at the C level without going through any custom logic defined by the class.

11

Aha, right, I completely missed looking at the descriptor implementation.

12

oh, right, I guess that’s where my own confusion comes from in the first place…

so I can see now how I could - not that I need to, of course - bypass the descriptor altogether by providing my own __setattr__ that does not resort to object.__setattr__

thanks for your help !

# keep it bare simple: hard-wired descriptor for 'desc' attribute
class Descriptor:
    def __get__(self, instance, owner):
        print("in desc. __get__")
        return instance.__dict__["desc"]
    def __set__(self, instance, value):
        print("in desc. __set__")
        instance.__dict__["desc"] = value


class Room:
    desc = Descriptor()

    def __init__(self, name, desc):
        self.name = name
        self.desc = desc

    def __getattribute__(self, attr):
        print("in __getattribute__", attr)
        return object.__getattribute__(self, attr)

    def __setattr__(self, attr, value):
        print("in __setattr__", attr, value)
        # a purely rethorical attempt: can we bypass descriptors ?
        self.__dict__[attr] = "I caught you"

    def __repr__(self):
        return f"{self.name} ({self.desc})"

print("-- class done")

# just to make sure the descriptor indeed gets found along the mro
class SubRoom(Room):
    pass

print("-- subclass done")

r = SubRoom("kitchen", "where to do the cooking")

print("-- istantiated done")

print(r)

which gives

-- class done
-- subclass done
in __setattr__ name kitchen
in __getattribute__ __dict__
in __setattr__ desc where to do the cooking
in __getattribute__ __dict__
-- istantiated done
in __getattribute__ name
in __getattribute__ desc
in desc. __get__
in __getattribute__ __dict__
I caught you (I caught you)