Backquotes for deferred expression

Ideas
41

My suggestion would to just use the defer expr syntax I implemented in that commit for now and just try to change the behavior of the defer object. A few things to consider, where d_<type> is an defer object that evaluates to an instance of type:

  • str(d_object), repr(d_object)
  • type(d_int), d_int.__class__, isinstance(d_int, int), issubclass(type(d_int), int)
  • math.sin(d_float), math.sin(d_Decimal), math.factorial(d_int), math.factorial(d_Decimal)

Don’t tell me how you think these should behave, implement the behavior you think is best. And don’t forget, requiring the math module to be aware of defer objects is an unacceptable cost since it would also require modifying third party libraries.

2 Likes
42

I had already thought about two things:

  1. Shells use backticks for immediate rather than deferred evaluation - I can live with that
  2. Nested backticks - what fun would that bring?

Some thoughts on nested backticks, extending a previous example

x = 2
s = `x * x`
def fn(x: int, y: int, z: int = `x + y + s`):  # s evaluated so z= `x + y + 4`
    return z

print(fn(3, 4))  # prints 11

# Escaped deferred expression

x = 2
s = `x * x`
def fn(x: int, y: int, z: int = `x + y + \`s\``):  # s not evaluated so z= `x + y + s`
    return z  # s evaluated here

print(fn(3, 4))  # prints 3 + 4 + 3 * 3 = 16


# Escaped deferred expression: implied braces?

x = 2
s = `x + x`
def fn(x: int, y: int, z: int = `x * y * \`s\``):  # s not evaluated so z= `x * y * s`
    return z  # s evaluated here

print(fn(3, 4))  

# Does this print 3 * 4 * 3 + 3 OR 3 * 4 * (3 + 3)  ?
```
43

So I think there are 2 main points to consider.

1. Observation (option 1 partial proxy) vs (option 2 full proxy).

And I think if this is going to be implemented to maximise usefulness, it might be better to go with option 2.

Let’s take an example (which is not new, but bear with me).

from external_library import get_something_with_default

result = get_something_with_default(
    'a', 
    default=`expensive_function()`
)

# What if "black-box" function uses `is` or `type`?

So ideally one would want to be able to use this functionality in any code and be at least 99.99% sure that all is going to work as expected.

Otherwise, this feature will not be compatible with most of the code. And if external libraries need extra banner “deferred evaluation compatible”, to me this sounds like a very bad deal.

So 2 main ones as I see (there might be more of them) are is and type.

Although (option 1) is much simpler as one can actually make such object in pure python, but if implementing this in standard library, where capabilities are greater than that, I think this needs to be made use of.

So I suggest that ideally the proxy deferred object needs to be a 99.99% substitute, as opposed to 98% substitute that one gets if is and type do not act on the evaluated value.

In this case, there would need to be special cases to type and is and some convenient back-door access to ProxyObject needs to be devised.

2. Variable binding (at definition vs at evaluation)

I am not sure about this, but I might have a good starting point to think about this.

Serialisation.

Say:

class A:
    def __init__(self, a):
        self.a = a

from external_library import factorial
b = 100_000
inst = A(a=`factorial(b)`)
inst_des = pickle.loads(pickle.dumps(inst))

What would make most sense here to happen?

If this was lambda, i.e. lambda: factorial(b), it would fail. And it should. Because the values within are subject to change.

But if this was NOT to behave as lambda, and we are binding variables at definition, this could have a well defined and fixed behaviour.

Advantages of binding at definition would be a much more robust object that can be serialised and used in loops:

expr = 1
for a in [1, 2, 3, 4]:
    expr = `expr + a`

Disadvantages are that one could not use it for cases such as “A non-backdoor approach for typing forward references”.

However, if one wants lambda behaviour even if “binding on definition” approach is taken:

c = `(lambda: a + b)()`

Then one gets lambda behaviour and serialisation fails - as expected.

Personally, I would favour “binding at definition”.
Mainly because lambda already does “binding on evaluation”

And this would bring more to the table if this was different and not having same “issues” as something that already exists so that it can cover different cases.

Especially if there still was a way (although slightly more verbose) to use it with lambda within.

1 Like
44

I think the notation proposed in the OP is very pretty :heart_eyes:.

I also agree that deferred evaluation and late-bound function arguments should not be mixed together, because they have different needs. This was a conclusion of a previous discussion on this subject, and I don’t quite remember all the details. One of the points was that
f(a, b=`a`)
should work properly (as already mentioned in this thread).
One might also want deferred expressions to be evaluated explicitly by calling them.

I noticed as I’m writing this that using the ` symbol in python makes it significantly harder to write markdown, so that’s a disadvantage.

To me late-bound function arguments are more important, I’d really like to have that.

45

Agreed, if there are good alternatives, I also think that backquotes are not the best option given possible confusions with markdown and logical inconsistency with bash.

Just to add.

Of course serialisation can be done simply by evaluating and serialising result.

What is lambda and why does it behave the way it does is clear.
“Binding on definition” is also quite unambiguous concept both in implementation details and in reasoning.

While if deferred object would behave as lambda, but would serialise as return value, then this would be completely new breed. And personally, I don’t have a mental model of what would this be and what would be the mental model to think about it.

E.g.:

a, b, c = 1, 2, `a + b`
print(c) # 3
a, b = 3, 4
print(c) # 7
c = pickle.loads(pickle.dumps(c))
a, b = 1, 2
print(c)    # ????

This is also directly related to considerations of what happens after 1st evaluation. There are few options:

  1. It replaces the object with evaluated value
  2. It doesn’t replace the object, but the return value is cached, thus fixed
  3. As in the above example, it behaves like unevaluated lambda, where it re-evaluates every time (with possibly new values).

I don’t have answers to these, but I think the reasoning of “why” for all of these things should be well thought out and put in perspective of all alternatives.

46

Writing this

z = x + y + `s`

Will cause the DeferExpr to be evaluated immediately - it’s observed by + operator. Hence the behavior will be identical to z = x + y + s.

Again, take the zero-arg-lambda analogy:

s = lambda: x * x
z = lambda: x + y + s()
# each time you obersive z, i.e. z()
# s be re-evaluated

For your first case where you want it to be immediately dereferenced:

# Option1: collapse (overwrite) s
z = x + y + (s:=s)
# Option2: create a temporary container for s
z = x + y + (t:=s)
47

Good point. For anyone that does not like the backquote, does any of these alternatives work better for you?

# Option1: a new operator, extending PEP671
x => a + b
def fn(x => a + b): ...

# Option2: f-string style d""
x = d"a + b"
def fn(x = d"a + b"):...

For the demo, I will follow @MegaIng 's advice and focus on the behavior side (as long as the defer keyword syntax still works for me).

48

I see an endless recursion in this statement.

x = lambda: x() + 1
49

I think it will be clearer of optimal syntax once other things are solved. Main question is whether it is eval or exec type. If it is multi-line, then something more complex might be needed. But as of now, I lack information to have an opinion.

d"expr" is I think good one to prototype using it for the time being. Maybe following subinterpreters, it can also be:

x = d"""
expr
expr
...
"""

And any indentation conveniences of this then could be addressed in parallel with subinterpreters.

Apologies, my bad - faulty example:

result = list()
for a in range(10):
    b = d"a + 1"
    result.append(d"b + 1")
lazy_result_sum = d"sum(result)"
1 Like
50

It worked!!!

BTW I used the PEP671-like syntax for it.

Python 3.13.0a3+ (heads/add-defer-dirty:f56d132, Nov  9 2024, 19:57:20) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a => b + c
compiler_defer_stmt
MAKE_DEFER_EXPR
Creating DeferExpr Object
>>> b, c = 1, 2
>>> a
3
>>> b, c = 3, 4
>>> a
7
>>> type(a)
<class 'DeferExpr'>

I am cleaning up my code. I will post a link to a commit ASAP.

51

This really has nothing whatsoever to do with PEP 671. Please, can you describe it instead in some other way? Your syntax is much more like JavaScript’s arrow function syntax.

52

Well the next step is to allow this => operator to work as a special kwarg.

But you’re right, it should have a name.

53

The core problem with a deferred expression is really about when to evaluate the expression.

In your implementation a deferred expression is evaluated upon the first usage, but that behavior would be undesirable if the deferred expression is to be passed through a decorator/wrapper function.

But if you try to solve that problem by coming up with a syntax that indicates when to preserve or to evaluate a deferred expression, you are basically requiring that all existing code to be rewritten with deferred expressions in mind, which isn’t going to happen.

This is the paradox that needs to be solved before deferred expressions can make any progress. Until then, a lambda function with an explicit call for evaluation is still the most viable solution.

54

There are two ways to come up with such a syntax. Either you have syntax to indicate that it should be evaluated, or you have syntax to indicate that it should be preserved. Adorning it to show that it should now be evaluated can already be done with the () operator on a function, so the proposal has no benefit. Adorning it to show that it should be preserved violates numerous expectations regarding wrappers, and would force all kinds of code to now be deferred-aware.

55

I’m rebasing my changes to the main branch. I will have something for everyone to play with very soon. It will save a lot of unnecessary speculations.

56

To all who are interested:

Please try out this working prototype !

I am exploring an wasi build so I can serve a webassembly REPL on github pages. I’ll update this post to add the link when it’s ready.

The core functionality is done in Objects/deferexprobject.c. Check it out if you have questions on implementation details!

57
Is this what you expected? 
a => b + c

def factory(var):
    def add_one():
        return var + 1
    return add_one

add_one = factory(a)

b, c = 1, 2
print(add_one())

b, c = 3, 4
print(add_one())
# === OUTPUT ===
4
8
58

FYI - I pushed a tiny commit to expose DeferExpr as a builtin. It can be used as a function decorator that converts a vanilla function into a deferred variable! (so multiline deferred evaluation is really easy to write).

Here are some starter examples if you are not sure how to play with it:

a => b + 1

b = 1
print(a) # 2

b = 2
print(a) # 3

@DeferExpr
def c():
    return a * 2 # Notice `a` is also a DeferExpr

print(c) # 6
b = 3
print(c) # 8

# Snapshot of a variable - frozen reference
d = 4

@DeferExpr
def e(var = d):
    return var + 1

print(e) # 5
d = 999
print(e) # 5

# Snapshot of a defer expr
b = 1 # a = 2 for now

@DeferExpr
def e(var = DeferExpr.eval(a)):
    return var + 1

print(e) # 3
b = 999 # a = 1000
print(e) # 3
59

Another commit was pushed to fix the following bug:

x => 1
x + 1 # 2
1 + x # SystemError

Please update your local copy if you’ve already cloned one.

Also:

A new API DeferExpr.eval() is added to take a snapshot of a defer_expr at arbitrary point of time. It handles any other object by returning as-is. I’ve added the example to the starter’s list above.

60

P.S. I do not know how to install numpy for this home-brewed version of python. I really appreciate it if someone can test it for me or tell me how to do it.