Currently functools.cached_property works only for objects with __dict__ and stores the cached value there. Would it make sense to make cached_property extensible to support more scenarios? My actual use case is a thread local property (currently implemented as a custom descriptor that caches the value in a threading.local attribute) but more could be enabled, such as classes with __slots__ or caching to a different object than the one with the cached_property.
2 Likes
I agree that it would be nice to extend cached_property for more use cases.
But it seems the scope could easily blow out, especially since we can’t just provide a pointer to the preferred storage variable; how simple was it to write the custom descriptor you mention?
Have you considered the documented alternative of stacking property() on top of lru_cache()?
from functools import lru_cache
class Foo:
__slots__ = ()
@property
@lru_cache(maxsize=1)
def a(self):
print('called')
return 1
foo = Foo()
print(foo.a) # called, 1
print(foo.a) # 1
1 Like
Doesn’t wrapping a method with the cache/lru_cache decorator result in instances sticking around when they “ought to be” GCed? Because the instance (self) goes into the cache, which lives on the function definition rather than on the bound method.
Anyway, storing state somewhere else, like in the OP’s case, seems like a perfectly good solution for any specific implementation. But not something I’d want a library, stdlib or otherwise, to do on the sly. If the cache itself is not accessible, that seems like a problem – how would I later clear it?
Speaking of which, it’s already fiddly enough to clear a cached_property. So if we’re looking at enhancing it, maybe that’s something to improve?
I think this thread was about how this library enables use of cached_property with __slots__, although I didn’t really follow the details.
1 Like
Ah, sorry, I’m not maintaining reslot anymore. It was just redundant with @attrs.define(init=False).
1 Like
Though, more pertinent to this thread, the attrs library will rewrite __slots__ if present, and keep @cached_property results there.
I haven’t, that’s an interesting option, even if not the most efficient; plus it doesn’t allow for set/delete.
I don’t have a use case for caching properties for class with slots but if I did I would store it as a “private” attribute:
class Foo:
__slots__ = ("a", "b", "_s")
def __init__(self, a, b):
self.a = a
self.b = b
@slotproperty
def s(self):
return self.a + self.b
def test_foo():
x = Foo(2, 3)
assert x.s == 5
x.s = 12
assert x.s == 12
del x.s
assert x.s == 5
1 Like
It wasn’t that hard but could be even simpler/shorter if cached_property exposed 3 “protected” methods/hooks: _get_cached, _set_cached, _del_cached. The default implementation would try to access obj.__dict__ in these methods (and only in these) but they could be overriden by subclasses. For example a slot_property that caches the value to a “private” attribute could be implemented as:
class slot_property(cached_property):
def _get_cached(self, obj):
return getattr(obj, "_" + self.attrname)
def _set_cached(self, obj, value):
setattr(obj, "_" + self.attrname, value)
def _del_cached(self, obj):
delattr(obj, "_" + self.attrname)
1 Like
One workaround to a lack of hooks to obj.__dict__ is to make the __dict__ attribute a property that returns your desired alternative cache:
from threading import local
from functools import cached_property
cache = local()
class Foo:
@property
def __dict__(self):
return cache.__dict__.setdefault(self, {})
@cached_property
def a(self):
print('called')
return 1
foo = Foo()
print(foo.a) # called, 1
print(foo.a) # 1
foo.a = 2
print(foo.a) # 2
del foo.a
print(foo.a) # 1
print(cache.__dict__) # {<__main__.Foo object at 0x0000018A4DD46660>: {'a': 1}}
Could you point me to where this is documented? Sounds really nice, but I haven’t been able to find anything about it, despite looking quite a bit
The init=False being broken seemed connected to the annotations.
No, sorry, I got entirely the wrong idea from browsing the source code.
You can do this with attrs for now, so I’ll edit my reply.
2 Likes
I think it would be useful (certainly to me) to have a decorator where
@dataclass(slots=True, frozen=True)
class C:
@slot_property
def p(self):
return f(self)
is a shorthand for
class C:
slots = ("__p")
@property
def p(self):
try: return self.__p
except AttributeError:
self.__p = f(self)
return self.__p
Attrs must do something in that direction, but I’ve never been able to fully understand how the attrs @cached_property code works.
Any solution involving dicts is likely not to be worth using, because you lose the whole performance benefit of using slots when you create a dict instance for each class instance. At least in the tests I did.
@cached_property is super cute (in its implementation) but it works by setting the instance attribute in such a way that the cached_poperty class instance gets hidden from getattr. I don’t think it’s worth subclassing from there.
The neatest solution would be if slots allowed falling back to the class attributes/properties when the slot is not initialised (like it does when you don’t have slots) but there’s got to be some technical reason why that’s hard to implement in CPython. Otherwise it would have been done already. (I expect)
1 Like
You may want to take a look at Context Variables:
1 Like
As documented in How Does It Work? - attrs 23.2.0 documentation, attrs stashes methods decorated with @cached_property and recreates the decorated class with slots with those method names, and a __getattr__ that delegates the lookup to the cached_property, which calls the stashed method and sets the attribute with the returning value so that the next lookup of this name will be resolved by the slot instead.
I haven’t looked at attrs’ code, but here’s what I think is a roughly equivalent implementation:
class slot_property:
def __init__(self, func):
self.func = func
self.name = None
def __set_name__(self, owner, name):
self.name = name
def __get__(self, obj, owner=None):
if obj is None:
return self
setattr(obj, self.name, value := self.func(obj))
return value
def slotify(cls):
def __getattr__(self, name):
try:
return slot_properties[name].__get__(self)
except KeyError:
raise AttributeError
slot_properties = {
name: obj for name, obj in vars(cls).items()
if isinstance(obj, slot_property)
}
return type(cls)(
cls.__name__,
cls.__bases__,
{
**{
name: obj for name, obj in vars(cls).items()
if name not in slot_properties
},
'__getattr__': __getattr__,
'__slots__': slot_properties
}
)
so that:
@slotify
class Foo:
@slot_property
def a(self):
print('called')
return 1
foo = Foo()
print(foo.a) # called, 1
print(foo.a) # 1
foo.b = 2 # AttributeError: 'Foo' object has no attribute 'b' and no __dict__ for setting new attributes
1 Like
from attrs import define
@define(slots=True)
class Matrix:
@cached_property
def the_one(self):
return "Neo"
On second thought there’s no need for slot_property to be a descriptor if __getattr__ is the only code that’s going to reference the stashed method. slot_property can be just a marker decorator instead:
def slot_property(func):
func._is_slot_property = True
return func
def slotify(cls):
def __getattr__(self, name):
try:
setattr(self, name, value := slot_properties[name](self))
except KeyError:
raise AttributeError
return value
slot_properties = {
name: obj for name, obj in vars(cls).items()
if getattr(obj, '_is_slot_property', False)
}
return type(cls)(
cls.__name__,
cls.__bases__,
{
**{
name: obj for name, obj in vars(cls).items()
if name not in slot_properties
},
'__getattr__': __getattr__,
'__slots__': slot_properties
}
)
That would be neat, but the technical reason why an uninitialized slot cannot fall back to a descriptor is that slots themselves are implemented as descriptors, so assigning a custom descriptor to a class attribute with the same name as a slot would simply overwrite the slot descriptor.
However, you can implement such a fallback behavior yourself by patching the class with slot descriptors replaced with wrapper properties that would fall back to a different value:
class Foo:
__slots__ = 'a'
def fallback(self):
try:
return slot.__get__(self)
except AttributeError:
return 1
slot = Foo.a
Foo.a = property(fallback, slot.__set__, slot.__delete__)
f = Foo()
print(f.a) # 1
f.a = 2
print(f.a) # 2
del f.a
print(f.a) # 1
The patching logics can be further generalized into a meta class and a marker decorator that effectively implements a @cached_property for slots:
class SlotPropertyMeta(type):
def __new__(metacls, name, bases, ns, **kwargs):
slot_properties = {
name: ns.pop(name) for name, obj in list(ns.items())
if getattr(obj, '_is_slot_property', False)
}
ns['__slots__'] = *ns.get('__slots__', ()), *slot_properties
cls = super().__new__(metacls, name, bases, ns, **kwargs)
for name, func in slot_properties.items():
slot = getattr(cls, name)
@property
def fallback(self, getter=slot.__get__, name=name, func=func):
try:
return getter(self)
except AttributeError:
setattr(self, name, value := func(self))
return value
setattr(cls, name,
fallback.setter(slot.__set__).deleter(slot.__delete__))
return cls
def slot_property(func):
func._is_slot_property = True
return func
so that:
class Foo(metaclass=SlotPropertyMeta):
@slot_property
def a(self):
print('called')
return 1
f = Foo()
print(f.a) # called, 1
print(f.a) # 1
f.a = 2
print(f.a) # 2
del f.a
print(f.a) # called, 1
1 Like
The source code for cached_property is only ~40 lines of relatively easy to follow code. Whenever I’ve wanted something similar but different to cached_property, I’ve found it fine to just to write my own using the original as a loose reference.
3 Likes
That’s true if you just want to store the cached values in a different namespace or mapping. But things are quite a bit trickier if the attributes are slots because you can’t normally have a custom descriptor with the same name as a slot.