I have been using Pure Python implementation of this for a while now, and it is the first concept in the area of “Deferred Evaluation” that I have identified to provide manageable & useful component that is worth attempting to implement in CPython, and is also a good complement to the rest of Python’s toolkit.
OP has picked up on some of my thoughts, combined it with his own ideas and proposed it.
Thus, it has taken some turns in the process as me and OP had to align.
By now I think this proposal is aligned to the concept that I had in mind to the degree that I am happy about. I.e. It closely adheres to it in aspects that I have taken fair amount of time to think about.
So the concept the way I see it is fairly simple and can be described as follows:
It is an object (currently its type is named defer), which:
- Is a proxy of
return value of wrapped callable.
- It has optional
args/kwargs, thus has similar constructor signature as functools.partial
- It evaluates on any
operation (or only on 1st operation in its cached version).
operation is defined as complete set of all Python’s operations, with exception of assignment. Thus, it’s a complete proxy object for underlying return value of wrapped callable.- Implementation of proxy features is largely similar to:
weakref.proxy or proxy objects of wrapt, but also addresses couple of extra operations.
- Couple of accompanying utilities are provided for direct access to
defer object, instead of return value of wrapped callable.
The reason this thread has turned out to be this way is because there was (still is) uncertainty whether “addressing couple of extra operations”, namely is,type is possible without making a big mess. It was (still is) important to make sure that complexity is justified by the benefit this brings. And complexity in question is largely limited to addressing the behaviour of is and type.
So to put it in code:
a = 1
d = defer(lambda: a + 1)
# Whatever you do with `d` (except assignment and special utilities)
# evaluates underlying callable and acts on it's result
print(type(d)) # int (calculates)
print(d + 1) # 2 (calculates)
a = 2
print(d + 1) # 3 (calculates)
# To only evaluate once and store result
a = 1
d = cached_defer(lambda: a + 1)
print(d + 1) # 2 (calculates)
a = 2
print(d + 1) # 2 (retrieves cached value)
# To bind arguments on definition:
a = 1
d = defer(lambda arg: arg + 1, a)
print(d + 1) # 2 (calculates)
a = 2
print(d + 1) # 2 (calculates)
What concerns expressions, it does not introduce any new logic itself.
Instead behaviour is fully dependent on its inputs.
The main problem that this solves can be summarised as follows:
“It allows lazy values without the need of implementing additional logic for it and it would introduce lazy default functionality to all existing defaults were lazy functionality is not implemented”
The simplest example is:
d = dict({'a': 1})
result = d.get('a', defer(math.factorial, 1_000_000));
# defer(math.factorial, 1_000_000) is not evaluated
The way that things are now, the only way to robustly implement lazy defaults is by implementing the logic manually with extra flag:
class dict:
def get(self, key, default=None, lazy=False):
if key not in self:
return default() if lazy else default
return self[key]
There is no robust way to implement it without extra keyword flag, because there is no robust check on the default value that could signal that it is lazy. I.e. “What if default value is meant to be unevaluated lambda?”
There are some ways around it, but all of them would result in additional complexity and discrepancies between different implementations. And many packages or even objects in CPython stdlib simply don’t implement it.
So this solution offers lazily evaluated arguments/defaults/values without changing any of the existing code structure and localising this problem to one concept.
There are things that the user will need to be mindful of, but those are nothing out of the ordinary. So far, I have not identified any ambiguities in expected behaviour of defer object which adheres to the concept above.
