Slots for class objects? (i.e. avoiding creating __dict__ for classes themselves)

Python Help
1

Hi there,

I am working on a data-oriented library that uses classes much like any other object instance, and can generate large numbers (tens of thousands, if not low hundreds of thousands) of classes at runtime, by calling the invoking (i.e. effectively by invoking the 3-arg form of type()).

I’m looking for ways of cutting down the memory consumption of the library in general. As part of that I have the dynamically generated classes (and their parent classes) set up with __slots__, so that class instances avoid the overhead of having a __dict__ attribute and associated dict instance, and that has helped. However since the number of classes can be the same as or more than the number of instances, I figure it’s probably worth the classes themselves having slots, for the same reason. However, I can’t figure out how to do that.

Adding __slots__ to the metaclass doesn’t help:

mjog@sable:~$ python3
Python 3.10.7 (main, Nov  2 2022, 18:49:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Meta(type): __slots__ = ()
... 
>>> # Doesn't work for a static class
>>> class Foo(metaclass=Meta): __slots__ = ()
... 
>>> getattr(Foo, '__dict__')
mappingproxy({'__module__': '__main__', '__slots__': (), '__doc__': None})
>>> # Doesn't work for a dynamic class either
>>> Bar = Meta('Bar', (), {})
>>> getattr(Bar, '__dict__')
mappingproxy({'__module__': '__main__', '__dict__': <attribute '__dict__' of 'Bar' objec

And looking at the docs about metaclasses, it seems like there will always be a __dict__ assigned:

When a new class is created by type.__new__, the object provided as the namespace parameter is copied to a new ordered mapping and the original object is discarded. The new copy is wrapped in a read-only proxy, which becomes the __dict__ attribute of the class object.

I guess the fact that the 3-arg form of type() requires a dict to be passed in as the third arg implies classes will always have a __dict__ attribute?

Am I out of luck here?

2

No one has any suggestions?

3

I haven’t explored this in the slightest.

But two things come to mind:

  1. Maybe don’t allow your classes to proliferate so?
  2. Maybe try passing a dict-like object instead of a dict?

HTH

4

Unfortunately, my experience with asking these level of questions, you seldom get replies. You are probably better served going to Stack Overflow.

FWIW, you attempted what I would have attempted, metaprogramming.

5

Are your certain that replacing class.__dict__ with class.__slots__ would actually provide any benefits? Class dicts, unlike instance dicts, are already read-only and only allow str keys, much like __slots__.

6

If __slots__ were supported, you’d have to create a metaclass for each class, for which __slots__ would have to list all of the defined attributes of the class, including special methods. The dict of the metaclass would have a data descriptor for each slot. The combination of the metaclass and class would be bigger than just using a class with a __dict__.

1 Like
7

How similar are the classes? Normally I’d assume they’re all different, but if you’re just using them as namespaces, then maybe not. That is, can some of the metadata be shared between them?

8

Thanks for the suggestions everyone.

I guess I could just replace the use of dynamically generated classes by creating a callable Type class and calling that to create instances of Instance class instead or similar, but then I’d also have to (re-)create all of the existing Python type machinery like isinstance()/issubclass(), the ability to derive sub-types, attribute inheritance, and so on. That all seems like a pretty big waste of time considering Python already has a decent type system that allows me to do all of this already. Why reinvent another one?

I am sharing state between class objects where possible, but it really is the number of them that’s at issue here. I don’t think passing a non-dict mapping object as the third arg to the type()/__new__() call will help since per the docs above, it gets copied into a new mapping object and discarded, but also it would be a Python object rather than C based descriptors, so potentially still comparatively large and slow anyway.

To my mind that’s the key issue here since afaik the memory efficiency & performance improvements of using __slots__ comes from the fact that no mapping object gets created, and C descriptor slots at the native level are used instead. To quote the docs again:

The space saved over using __dict__ can be significant. Attribute lookup speed can be significantly improved as well.

This is an interesting point:

Not sure about the part about the special methods though - you don’t need to do that for instances of classes that both e.g. define __slots__ and __eq__?

But I don’t think it’s actually an issue anyway - the metaclass would need to include __slots__ = ('__slots__', [...]), then all class objects constructed using it could have their own __slots__ attribute also, each with different contents, couldn’t they? Just like regular instances of classes with slots.

So like:

class Meta(type): __slots__ = ('__slots__',)

FooClass = Meta('FooClass', (), {'__slots__': ('foo',)})
BarClass = Meta('BarClass', (), {'__slots__': ('bar',)})
BazClass = Meta('BazClass', (), {'baz': 'b'}) # AttributeError since the metaclass's slots doesn't include `bar` 

fooInstance = FooClass()
fooInstance.foo = 'a' # No error
fooInstance.bar = 'b' # AttributeError since the class's slots doesn't include `bar` 

barInstance = BarClass()
barInstance.foo = 'a' # AttributeError since the class's slots doesn't include `foo`
barInstance.bar = 'b' # No error
9

The __slots__ and __eq__ attributes are stored in the dict of the class. You’re asking about how to create a class that has no dict, which means that the class would need to have slots for these attributes instead of storing them in a dict. If this were supported, the slots would have to be declared in the metaclass, which has a dict, and, like any class, is is a fairly large object (about 1 KiB), and will grow considerably larger considering each slot is associated with a data descriptor in the metaclass. If you’re creating thousands of classes, all with a different set of attributes, having to create a metaclass for each class would defeat the point of using slots.

10

Sorry, still not sure how that’s any different from how slots currently work on regular instances.

If you define __slots__ on a class, instances already lose certain features if you don’t define special slots - e.g. not being able to set arbitrary attrs if you don’t include '__dict__', and not being able to refer to the instance with a weakref if you don’t include '__weak_ref__'. If you want those features, you need to define those attrs as slots.

If you mean there’s nowhere to store references to methods like __eq__, then they aren’t actually stored in the class dictionary, so it’s moot anyway?

>>> class Foo: pass
... 
>>> Foo.__dict__.keys()
dict_keys(['__module__', '__dict__', '__weakref__', '__doc__'])

Note there’s no __eq__ in there --^

If you’re creating thousands of classes, all with a different set of attributes, having to create a metaclass for each class would defeat the point of using slots.

Can you give an example of what this would look like? Because I believe the example I gave in my previous post pretty plainly demonstrates why having a metaclass per class is not necessary.