Backquotes for deferred expression

Ideas
61

That looks like a good start.

I think DeferExpr needs *args and **kwds.

This way:

result = list()
for a in range(10):
    b = DeferExpr(operator.add, a, 1)
    result.append(DeferExpr(operator.add, b, 1))
lazy_result_sum = DeferExpr(sum, result)

Also, I think DeferExpr needs a more convenient name.
Personally, I probably would use it more than potential syntactic convenience for no-arg version.

After all, it seems that b = d"a + 1" is just syntactic convenience for special case b = DeferExpr(lambda: a + 1).

DeferExpr is essentially functools.lazy concept from before, maybe naming it lazy or deferred would be appropriate.

Its signature is similar to partial(func, *args, **kwds), while this also potentially has lazy(func, *args, **kwds), so this naming pattern would make things consistent.

Also, not sure what would be best location for such. From functionality POV functools seems good place.
However, given that this would most likely not have its pure Python version, it might be better to create a new module for it, e.g. proxies.

All in all, I would strongly suggest splitting this into 2 parts:

  1. Nail down DeferExpr object
  2. Then figure out syntactic conveniences and possible additions to builtins

All the important details to figure out are in DeferExpr concept.
And there is a high risk that syntax changes will be rejected.
And if DeferExpr is bundled with such, the whole thing will be rejected.

62

eval is too common attribute, that might need to act on evaluated return.

I would suggest dunder methods for all back-door/DeferExpr specific access.

__eval__() or might be even better to follow dask - __compute__ (FYI I was using this in my own prototypes), or lookup any other similar/popular 3rd party packages for naming ideas.

I don’t think eval is best option here. eval in Python most commonly refers to evaluation of string / AST, here it is simply calling wrapped function, thus it might be good to avoid possible confusions.

63

Also, it would be good to limit specific attributes to a bare minimum. Maybe one for explicit evaluation and one for attributes:

a = lazy(foo, *args, **kwds)
a.__compute__()    # foo(*args, **kwds)
a.__lazyattrs__    # tuple: (foo, args, kwds)

And one more thing to consider is caching.
If instead, the signature was lazy(func, args: tuple | Optional, kwds: dict | Optional), then could add more keyword arguments. In this case cache: bool = False.

This is how my latest latest version, which I was working on, roughly looks:

class lazy:
    def __init__(self, func, args=(), kwds={}, cache=True):
        """
        Args:
            cache: False, True, tuple[context, name]
                False   - will not cache and every usage will be new call
                True    - will cache locally for lazy instance
                tuple   - (namespace, name) will cache locally and set namespace variable
        """
        ...

    def __compute__(self):
        ...
64

Aha, a lot of interesting things to reply to! Let me get to them one by one.

I think it the other way: DeferExpr has the potential to evolve into a more efficient way to do lambda functions in python.

typedef struct {
    PyObject_HEAD
    // In current implementation,
    // PyDeferExprObject only has one internal ref
    PyFunctionObject *lambda;
    // Your request involves adding 2 more
    // and check them on each evaluation:
    PyObject *args;
    PyObject *kwargs;
} PyDeferExprObject;

Currently it cannot beat lambdas because it’s actually calling a lambda function under the hood. But if we adhere to allowing no arguments, we’ll have the choice to optimize away the entire argument lookup stage in the bytecode, if necessary.

I am not sure how much performance is at stake though. @dg-pb are you interested to run the performance tests you wrote earlier on this? That will give us solid metrics on its current performance, and help us understand the performance impact of argument lookup.

Currently working syntax can cover this (I know it might not be your favorite…):

result = list()
for a in range(10):
    b => operator.add(a, 1)
    r => operator.add(b, 1)
    result.append(r)

lazy_result_sum = DeferExpr(sum, result)

As stated above, the performance cost of 2 argument lookups might not be automatically better than 1 lookup plus 2 function calls (especially if the 1st function call has no argument to lookup).

BTW this piece of code code evaluates to 110 (i.e. sum([11] * 10) on the demo implementation.

Agreed, that’s an arbitrary choice of name (took me only 10 seconds). A interesting misunderstanding I noticed in your statement is that you assumed this backdoor exists on all DeferExpr instances - it does not.

# Tested on local binary
x => 1
x.eval # AttributeError: 'int' object has no attribute 'eval'

The only way to access it is DeferExpr.eval(). Similar backdoor methods can be developed to retrieve other attributes (e.g. closure) of the enclosed lambda function - and they are only available from DeferExpr, not its instances.

This is because I chose to implement option 2 - a fully undistinguishable proxy on the python side.

65

I am working the tricky ones: type() and is - they don’t use hook methods. I have a solution in my mind but I need a bit more time to make it work.

After that is done, it will be near impossible to distinguish a defer_expr from normal variables on the python side.

66

@MegaIng - Here are the actual results:

d_object => object()

str(d_object)  # <object object at 0x100ef8260>
repr(d_object) # <object object at 0x100ef8330>

d_int => 1

type(d_int)                   # <class 'DeferExpr'> -- working on this
d_int.__class__               # <class 'int'>
isinstance(d_int, int)        # True
issubclass(type(d_int), int)  # False -- same as above, working on this

from math import sin, factorial

d_float => 1.0

sin(d_float)     # 0.8414709848078965
factorial(d_int) # 1

from decimal import Decimal

d_decimal => Decimal(1)

sin(d_decimal) # 0.8414709848078965
factorial(d_decimal) # TypeError: index not supported

# Note it will also error out with vanilla decimals
factorial(Decimal(1))
# TypeError: 'decimal.Decimal' object cannot be interpreted as an integer

# I'm investigating why the errors are different though ...

I am working on fixing the few problems as I commented.

67

I think the focal point is the desired functionality. For lambda functionality there is lambda already. I think it would be much more useful to have argument binding on definition.

This is a good idea, given this is going to be what you currently think it should be. However, I have my doubts.

I don’t think no-argument-intrinsically-evaluated-lambda is going to provide enough coverage for different needs of “deferred evaluation” and the cost of introducing such construct might be too great for what it offers.

Performance is not as important here as you might think. Getting the concept right is the priority.

This is the issue. It should be 65:

sum(range(10)) + 2 * 10

Not having ANY backdoors on the object itself is a good idea. Either by classmethod or might be even better - separate utility function.

68

Just want to elaborate on this.

This is not suitable for constructing evaluation graphs, but is rather an endpoint deferred value.

It is not a substitute to framework such as dask, but should ideally be complement.

Why? Because it does not build a workable evaluation graph. And it shouldn’t.

So just to give an example and wider considerations of this:

Step 1. Build a graph with appropriate framework:

import dask
@dask.delayed
def later(x):
    return x

output = []
data = [23, 45, 62]
for x in data:
    x = later(x)
    a = x * 3
    b = 2**x
    c = a + b
    output.append(c)

total = sum(output)

Step 2. If lazy evaluation is needed, use “deferred expression”

deferred = LazyExpr(lambda: total.compute())
# Or to eliminate the risk of value change of variable "total"
deferred = LazyExpr(operator.call, total.compute)

The aim here, IMO, should be extending Python’s toolkit with components that talk with each other and make sense in the background of what is in Python stdlib and also 3rd party packages.

As opposed to building independent monolithic concept, which from what I have observed is the path that this proposal is at high risk of taking.

69

One thing I am worrying is people tends to request more features on a new proposal. That usually breaks the entire deal.

As @dg-pb suggested (not sure if I understood correctly), we should really narrow down the scope of problem to be solved by DeferExpr. And I would summarize it as follows:

it allows you to convert a small piece of statement into a “variable”, it fits into all existing code that expects a value, not a function.

This will allow us to tell if a problem is deemed “relevant” to this specific proposal.

Take the following as an example:

l = list[int]()
for x in range(10):
    y => x + 1
    l.append(y)

sum(l) # sum([1, ..., 11]) or sum([11] * 10) ?

Should not be considered a problem to be covered by DeferExpr. DeferExpr is supposed to behave the same as a zero-argument-lambda. And it does nothing more than a lambda function.

Immutable snapshots of values should be done in another proposal (IMO this will not really be feasible because it requires each loop iteration to have their own closure, which is too costly).

The same applies to computing graphs - they are too heavy to be covered by this proposal. A powerful 3rd party wheel will do the job better. And it will be great if DeferExpr can extend their capabilities.

70

Then I am -1 on this. Extending Python with new object, new syntax, parser adjustments, etc, only to get implicit evaluation of no-arg-lambda is not worth it.

In other words, if it is unable to bind input values on definition, this is a dealbreaker to me.

On the other hand, what I am suggesting would be able to handle both. But the concept needs to be oriented towards: lazy(func, *args, **kwds) and lambda input is a special case. This way, it would make sense.

71

I’m sorry, but I have no interest in trying to work out what the behaviour of this new construct is in edge cases by experimentation. The prototype is currently incomplete because it lacks documentation, and that’s a problem with the proposal which has been repeatedly pointed out here. You must describe the intended behaviour in detail at some point before this will even be viable as a PEP, and it’s already been pointed out to you that every single previous proposal has failed because the behaviour in edge cases isn’t workable (or simply wasn’t described).

For example, can you point me to the documentation that describes what’s expected to happen with the following code?

>>> a, b = 1, 2
>>> x = `[a, b]`
>>> y = x
>>> id(y) == id(x) # Is this True or False?
>>> z = `print(a)`
>>> tmp = z # Does this print 1 or not?
>>> tmp = z # Does *this* print 1 or not?
>>> tmp is None # Is this True? Dis it print 1?
>>> z is None # Is this True? Did it print 1?

I’m not asking someone to run this in the prototype and report the results. I literally don’t care what the prototype does. What I want to know is what the code is intended to do, in other words what the documentation claims it does. Without that, how do I know the prototype is working as intended?

For every one of the questions I asked in the code above, I want to understand why the answer is as it is. Not just what the answer is, which I can find by experiment, but what’s the reasoning that leads to that answer?

4 Likes
72

How about this little helper?

def defer(func, *args, **kwargs):
    return DeferExpr(lambda: func(*args, **kwargs))

Your request is by all means feasible. It can be done in pure python. We can make it a builtin in the proposal.

73

That would work too.

But overall performance would be better if this was a default. DeferExpr(func, *args, **kwds) would be able to handle all cases with 1 call, while 1 extra call would be needed for what you are suggesting.

To put it simply, why not have *args and **kwds if you can?

This is the reason why partial(func, *args, **kwds) has *args and **kwds. According to your logic, partial should look like:

class partial:
    def __init__(self, func):
        self.func = func
    def __call__(self):
        return self.func()

# and used like
p = partial(lambda: func(1, 2, a=1))

This is an exaggeration, but just trying to point to rationale which I think needs more weight here.

74

I just want to keep it possible to optimize further (eliminate arglist evaluation) - that will be the ultimate deal that works best for either case.

75

For now, allowing DeferExpr to accept args and kwargs will not eliminate such possibility - we can modify the internal behavior to make it optimal for both cases.

The reason this is not done in C is simply because I have limited time… Doing this in python saves me a few hours.

76

actually, most other operators, save for * (and +, - as they work as unary operators) could just be doubled. - [] , :: ::, etc…

This is a minor issue, but I also am -1 on trying to use something as hard to distinguish as " ` " .

Anyway, going back to your main proposal - I had missed something along these lines a couple of times - like having a “transparent future proxy”, for example (btw, had you checked weakref.proxy? )

Anyway - I think at this moment, I’d suggest you could watch the discussions around PEP 750, and see if that allows you what you have in mind with this - and, if it is missing something, maybe propose smaller tweaks there.

Otherwise, lambdas seem to do a lot of what you want, and already have well defined syntax and rules, about which scopes to use for outside variables - and it being even possible to choose if they should be used eagerly or lazily, by creating default arguments on the lambda declaration.

77

Here is what it currently does plus what it is supposed to:

a, b, c = 1, 2, "hello"
x => [a, b]
y = x
id(y) == id(x) # True (I am working on "is" statement, it should be False)
z => print(c)
tmp = z # Nothing printed, they reference to the same DeferExpr object
tmp = z # Does *this* print 1 or not? No
print(tmp is None) # False (should be True, working on this)
print(z is None) # False (should be True, working on this)
Disassembly view 
  --           MAKE_CELL                4 (a)
               MAKE_CELL                5 (b)
               MAKE_CELL                6 (c)

   1           RESUME                   0

   2           LOAD_CONST               0 ((1, 2, 'hello'))
               UNPACK_SEQUENCE          3
               STORE_DEREF              4 (a)
               STORE_DEREF              5 (b)
               STORE_DEREF              6 (c)

   3           LOAD_FAST                4 (a)
               LOAD_FAST                5 (b)
               BUILD_TUPLE              2
               LOAD_CONST               1 (<code object <lambda> at 0x101435610, file "/Users/Yuxuan/Lab/cpython/paul.py", line 3>)
               MAKE_FUNCTION
               SET_FUNCTION_ATTRIBUTE   8 (closure)
               MAKE_DEFER_EXPR
               STORE_FAST               0 (x)

   4           LOAD_FAST                0 (x)
               STORE_FAST               1 (y)

   5           LOAD_GLOBAL              1 (id + NULL)
               LOAD_FAST                1 (y)
               CALL                     1
               LOAD_GLOBAL              1 (id + NULL)
               LOAD_FAST                0 (x)
               CALL                     1
               COMPARE_OP              72 (==)
               POP_TOP

   6           LOAD_FAST                6 (c)
               BUILD_TUPLE              1
               LOAD_CONST               2 (<code object <lambda> at 0x10139ba50, file "/Users/Yuxuan/Lab/cpython/paul.py", line 6>)
               MAKE_FUNCTION
               SET_FUNCTION_ATTRIBUTE   8 (closure)
               MAKE_DEFER_EXPR
               STORE_FAST               2 (z)

   7           LOAD_FAST                2 (z)
               STORE_FAST               3 (tmp)

   8           LOAD_FAST                2 (z)
               STORE_FAST               3 (tmp)

   9           LOAD_GLOBAL              3 (print + NULL)
               LOAD_FAST                3 (tmp)
               LOAD_CONST               3 (None)
               IS_OP                    0 (is)
               CALL                     1
               POP_TOP

  10           LOAD_GLOBAL              3 (print + NULL)
               LOAD_FAST                2 (z)
               LOAD_CONST               3 (None)
               IS_OP                    0 (is)
               CALL                     1
               POP_TOP
               LOAD_CONST               3 (None)
               RETURN_VALUE

Disassembly of <code object <lambda> at 0x101435610, file "/Users/Yuxuan/Lab/cpython/paul.py", line 3>:
  --           COPY_FREE_VARS           2

   3           RESUME                   0
               LOAD_DEREF               0 (a)
               LOAD_DEREF               1 (b)
               BUILD_LIST               2
               RETURN_VALUE

Disassembly of <code object <lambda> at 0x10139ba50, file "/Users/Yuxuan/Lab/cpython/paul.py", line 6>:
  --           COPY_FREE_VARS           1

   6           RESUME                   0
               LOAD_GLOBAL              1 (print + NULL)
               LOAD_DEREF               0 (c)
               CALL                     1
               RETURN_VALUE
78

I think getting functionality right is more important here. With what you are proposing:
Benefits:

  1. Optimization by eliminating arglist evaluation

Drawbacks:

  1. Strictly non-serializable
  2. Does not support *args and **kwds

I don’t think the main concept should be primarily based on 1 specific optimization opportunity.
And you would have other opportunities to optimize anyways (see partial optimizations).

If your proposal is going to stand on the pillar of having lambda behaviour, then most of the feedback of community will be just “extra convenience is not worth it, because we already have lambda for that”.

And if you start with lambda, do some specific optimizations and then try to branch out to inclusion of args and kwds then it will be a messy design which no-one will want to maintain.

79

@pf_moore BTW I am struggling on the following two items:

  1. Add an additional logic to type(x) and is statement so they automatically performs an observation on any DeferExpr

    For type() I tried the following:

    • Modify PyType_Tpye.tp_call() in typeobject.c
    • Modify PyObject_Type() in abstract.c.
      However, debug prints show none of them actually executed when I invoke type on a builtin object (DeferExpr). I am not sure where to look next.

  2. Troubleshoot a segfault triggered by gc.collect().

    import gc
x => 1
del x # OK
gc.collect() # segfault

    I do not have much experience using gdb/lldb with a project at this scale.

Can you offer some advice or point me to related resources? Really appreciate it!

80

As I said, performance is not of paramount importance here, because this should not be used to build evaluation graphs, but rather provide a “lazy endpoint” which is constructed once. Thus “instantiation cost” is not an issue and __call__ can be optimized in the same manner as partial is.

If this was used as a backend to sympy, where graphs are constructed and manipulated continuously and instantiation time in comparison to additive workload is pretty high, then micro-optimizations would have a higher weight here.