Better doc support for slotted descriptors

Melendowski · July 24, 2022, 1:29pm

Using the descriptor protocol a developer can implement their own property like class which operates similarly with the output of help() where the instance __doc__ shows up under the name of the custom property and under the data descriptors section of help(). An issue occurs when you want to slot your custom property. You cannot slot your custom property, have a class docstring, and still be able to have a __doc__ on an instance of the custom property.

Examples:

class Foo:
    """Docstring on class, no slots, can override __doc__ on instance and get
    desired help() documentation.
    """

    def __init__(self, doc):
        self.__doc__ = doc

    def __get__(self, instance, owner):
        ...

    def __set__(self, instance, value):
        ...

class WithFoo:
    custom_property = Foo("Property doc string")

Now looking at the help of both

help(Foo)

class Foo(builtins.object)
 |  Foo(doc)
 |
 |  Docstring on class, no slots, can override __doc__ on instance and get
 |  desired help() documentation.
 |
 |  Methods defined here:
 |
 |  __get__(self, instance, owner)
 |
 |  __init__(self, doc)
 |      Initialize self.  See help(type(self)) for accurate signature.
 |
 |  __set__(self, instance, value)
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)

help(WithFoo)

Help on class WithFoo in module __main__:

class WithFoo(builtins.object)
 |  Data descriptors defined here:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)
 |
 |  custom_property
 |      Property doc string

Now if we try to __slot__ our custom property things start to get hairy.

class Foo:
    """Docstring on class, with slots, __doc__ not in slots, cannot override
    __doc__ on  instance and get desired help() documentation. Raises an
    AttributeError on instantiation.
    """
    __slots__ = ()

    def __init__(self, doc):
        self.__doc__ = doc

    def __get__(self, instance, owner):
        ...

    def __set__(self, instance, value):
        ...

class WithFoo:
    custom_property = Foo("Property doc string")

This raises an error

AttributeError                            Traceback (most recent call last)
Input In [41], in <cell line: 20>()
     17     def __set__(self, instance, value):
     18         ...
---> 20 class WithFoo:
     21     custom_property = Foo("Property doc string")

Input In [41], in WithFoo()
     20 class WithFoo:
---> 21     custom_property = Foo("Property doc string")

Input In [41], in Foo.__init__(self, doc)
      8 def __init__(self, doc):
----> 9     self.__doc__ = doc

AttributeError: 'Foo' object attribute '__doc__' is read-only

So naturally add __doc__ to __slots__ but then we cannot have a class docstring as this situation raises an error too.

class Foo:
    """Docstring on class, with slots, __doc__ in slots, cannot define class at
    all. Raises attribute error.
    """

    __slots__ = ("__doc__",)

    def __init__(self, doc):
        self.__doc__ = doc

    def __get__(self, instance, owner):
        ...

    def __set__(self, instance, value):
        ...

class WithFoo:
    custom_property = Foo("Property doc string")

ValueError                                Traceback (most recent call last)
Input In [43], in <cell line: 1>()
----> 1 class Foo:
      2     """Docstring on class, with slots, __doc__ in slots, cannot define class at
      3     all. Raises attribute error.
      4     """
      6     __slots__ = ("__doc__",)

ValueError: '__doc__' in __slots__ conflicts with class variable

So in closing if we want to slot a descriptor which is to act like a custom version of property then we are stuck with either defining a class docstring or an instance docstring. I think it would be a good idea to allow some sort of special casing, overriding __doc__ on slotted classes that also define a __get__ and __set__ so that there can exist a class docstring while simultaneously overriding it on the instance and obtaining the desired help() when the descriptor is instantiated.

steven.daprano · July 31, 2022, 4:31am

I think that all the descriptor protocol and property-like details are a red-herring. The same issue occurs with any class that tries to make __doc__ into a slot.

We can simplify your example to just a single, straight-forward example:

class A(object):
    """A doc string."""
    __slots__ = ('__doc__',)
    def __init__(self):
        self.__doc__ = 'Instance doc.'

And the output is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: '__doc__' in __slots__ conflicts with class variable

Right. Because it does. So how does property manage it?

To know for sure, we would need to read the source code in C, but we can get some hints by realising that property docstrings are member objects, that is, they are descriptors themselves.

Given this definition:

p = property(None, None, None, 'documentation')

we can see that p has no instance dict (vars(p) raises), nor does it have a visible __slots__ attribute, so there is no visible storage for a per-instance docstring. Nevertheless, there must be hidden storage somewhere, it is just not exposed directly from C to Python.

If we next check the property attribute, we see it is a descriptor:

d = vars(type(p))['__doc__']
print(d, '__get__' in dir(d))

and the output is something like:

<member '__doc__' of 'property' objects> True

So I think that this is enough to solve the problem of giving descriptors with slots a docstring.

Give your custom property-like class a slot _ds, say, to hold the per-instance docstring.
Write a small utility descriptor which gets and sets from the _ds slot.
In your customer property-like class, save the class docstring into a class attribute _ds, and set __doc__ to your utility descriptor.

Something like this:

class helpmember(object):
    # __get__ and __set__ a custom docstring _ds attribute.
    pass

class MyProperty(object):
    """Doc string"""
    _ds = __doc__
    __doc__ = helpmember()

    def __get__(self, ...):
       pass

Obviously this is untested and just a sketch. It’s been a long time since I’ve written a custom descriptor, so some of the details may be off.

I can’t be sure that is precisely what property does, but I expect that it will be rather similar.

Melendowski · July 31, 2022, 1:20pm

Hi Steve,

I think that all the descriptor protocol and property-like details are a red-herring. The same issue occurs with any class that tries to make __doc__ into a slot.

We can simplify your example to just a single, straight-forward example:

Yes, the bigger issue is defining a class attribute that conflicts with a slot name. I used the descriptor/property example because in the case of __doc__ I don’t see why a developer would ever set an explicit __doc__ on an instance of a class, which is not a descriptor , because it provides no additional documentation purposes unlike in the case of a descriptor.

Right. Because it does. So how does property manage it?

This was a very enlightening explanation and the use of a another descriptor for retrieving the correct __doc__ is an idea I hadn’t thought of. With that being said I still think this should be more streamlined. I could do it myself right now using a metaclass I believe, but that feels wrong. Additionally, the use of another to class just to make __doc__ and __slots__ play nicely together seems counter intuitive…I want to utilize __slots__ for memory saving purposes to begin with - this is probably moot point however because if this support was to be added it would be done like this anyway.

I remember reading on one of the mailing lists, as of a week or two ago, that descriptors don’t seem to be widely used, and that may be true for day to day programming but there are some major libraries that utilize them heavily. To name a few of the top of my head

params under holoviz project
sqlalchemy
openpyxl
python-pptx

I myself at my work use descriptors heavily, in a manner inspired by sqlalchemy, for developing a framework for defining tables to reside in H5s.

I do not know what kind of benefits the listed libraries would gain, or if they even care. I just think in the ways that these libraries and myself are using descriptors warrants the use __slots__ to save on memory, which comes with some negatives for documentation purposes.

This doesn’t seem to be a large concern for a lot people, as seen by the traffic on this thread so far. I’m not a subject matter expert on these things and cannot argue any further for this feature.

Thanks for your time so far.

Topic		Replies	Views
Alternative to __slots__ Ideas	4	865	July 7, 2022
Add a supported read-only classproperty decorator in the stdlib Ideas	16	4258	April 10, 2024
3.12 slot classes wishlist Ideas	3	558	November 5, 2022
Difference between slot wrapper, method wrapper and wrapper descriptor Python Help	6	3094	April 20, 2022
Slots for class objects? (i.e. avoiding creating __dict__ for classes themselves) Python Help	9	1426	December 9, 2022

Better __doc__ support for slotted descriptors

Related Topics

Better doc support for slotted descriptors