Backquotes for deferred expression

zhangyx · November 8, 2024, 6:41pm

Back quotes have been freed from a syntactic sugar for repr since the first version of python3, I think it will be perfect to be repurposed for deferred expressions.

Use cases:

1. Late-bound function arguments

def fn(x: int, y: int, z: int = `x + y`):
    print(z) # expr will be evaluated here

fn(1, 2) # 3

def fn(el: object, z: list = `[]`):
    z.append(el)
    print(z)

fn(1) # [1]
fn(2) # [2]

def task1(next_task=`task2`): ...
def task2(next_task=`task1`): ...
# No NameError!

2. A new approach of declaring zero-argument lambdas

P.S. This is fundamentally how it behaves: a lambda function with zero argument and auto-evaluates itself upon observation (i.e. no empty parenthesis).

a, b, c = 1, 2, `a + b`

print(c) # 3

a, b = 3, 4
print(c) # 7

3. A non-backdoor approach for typing forward references

type X = `A | B`

class A: ...
class B: ...

Previous Attempts

PEP671 proposed a new operator => that works identical in use case 1. Unfortunately, this proposal have received too many objections and is currently stalled (thanks @Rosuav for sharing the information!).
A working prototype of defer keyword has been proposed in related topics 2 (see below). The defer keyword works analogous to the proposed back quotes. But it (1) takes a longer form and (2) may cause precedence problems for operators and separators (I might be wrong on this one).
My own proposal of “deferred arg list” - this does not raise much interest in the community. And the keyword decorator I proposed lacks flexibility and will cause confusions on its behavior.
functools.lazy and other 3rd party wheels for lazy evaluation (thanks @dg-pb for brining them up!) - they covers most of the cases above. But lack of interpreter support exposed some operations that cannot be properly proxied (such as is and type). In addition, lack of syntax sugar makes them cumbersome to spell:
```
# Proposed
def task1(next_task = `task2`): ...
# Existing: functools.lazy
def task1(next_task = functools.lazy(lambda: task2)): ...
# Shortest possible form: from functools import lazy as L
# Not friendly to code reviewers or readers
def task1(next_task = L(lambda: task2)): ...
```

Implementation Details:

Collapsible?

It remains to be discussed if the DeferExpr variable should collapse upon its first evaluation. (i.e. replace itself with the evaluation result). This might be favorable for performance considerations. However, it will invalidate the first two use cases presented above.

IMO the answer would be not to collapse. In some corner cases where dynamic values causes problems user can explicitly collapse a deferred expr by stating x = x in their code. For functions that do not like dynamic values as their arguments, a syntax of fn(x:=x) will be sufficient.

Observation - namely the is and type keyword:

Option 1: logically distinguishable

x = `True`
type(x)   # DeferExpr
x is x    # DeferExpr(...) is DeferExpr(...)? => True
x is True # DeferExpr(...) is True? => False
x.__expr__ # "True"
x.__locals__ # locals() at where it is declared
x.__globals__ # globals() at the module that it is declared

Option 2: completely undistinguishable - secret to the interpreter

x = `object()`
type(x)   # object (an observation of the value)
x is x    # object() is object()? => False
# In case you REALLY need to tell...
inspect.is_defer_expr(x) # True

This remains to be discussed - and it’s hard to tell which will work better among all use cases.

Typing

A deferred expression needs to be typed differently than a normal value, e.g. DeferExpr[T]. It seamlessly fits into any slots that expects type T. However, as stated above, there should be a way to hint that DeferExpr is not allowed - and it needs to be discussed.

Related Contents:

PEP671 - Syntax for late-bound function argument defaults
Defer Expressions - A working prototype of defer keyword.
Another idea on deferred arg list - Dynamic evaluation of function argument list initializer
TypeExpr-specific constructors - Where this idea originated.
functools.lazy - A discussion post bringing up proxy tools provided by both stdlib and 3rd party.

P.S.

I know this topic have been revisited many times. I myself stated in my original post “it might be too late to even bring this up for discussion”. However upon second thought I think this idea deserves to be presented to the community - even if it will very likely stall again.

Nineteendo · November 8, 2024, 6:56pm

How do we write an expression that spans multiple lines?

def fn(el: object, z: list = `[1,
                               2,
                               3]`):
    z.append(el)
    print(z)

OR

def fn(el: object, z: list = ```[1,
                                 2,
                                 3]```):
    z.append(el)
    print(z)

zhangyx · November 8, 2024, 6:57pm

I see no problem in these examples. Does it break any existing feature?

Nineteendo · November 8, 2024, 6:58pm

No, but given Python’s convention of 79 character lines, I frequently need to wrap lines.
So, if the deferred expression is too long, it won’t fit on the line and my IDE will complain.

zhangyx · November 8, 2024, 6:59pm

I guess it’s at least shorter than functools.lazy ?

Nineteendo · November 8, 2024, 7:04pm

My question is simply: “if it would be supported or not?” ^[1] Regardless of how long something is, you’ll have to wrap at some point. ^[2]

“no” is an acceptable answer ↩︎
that’s why I import names from modules as often as possible instead of importing the module ↩︎

zhangyx · November 8, 2024, 7:12pm

Got it. I am not familiar to AST related stuff. I will leave it as an open question.

dg-pb · November 8, 2024, 7:16pm

I have been thinking about this for a fair while now. 1.5 years to be precise.

See my early attempt to start (to be more precise to continue on previous discussions) discussion on this: Deferred Evaluation Initial Proof of Concept

My current take on this is that convenience syntax decisions can be left for the very end.

The major issues of this direction is not in syntax, but in concept and implementation.

Same as with many other syntax affecting proposals, my take is that it is best to explore whether good enough solution can be devised via implementing and combining a set of tools that use existing capabilities.

And question “whether syntax conveniences are needed” will be much easier to answer once there is a clear idea of what the functionality exactly is and its usefulness.

zhangyx · November 8, 2024, 7:27pm

Can you name the obstacles on the implementation side, or point me to a post on them?

The behavior of the backquote syntax has been mostly defined in the OP. I can even write you a python pre-processor that replaces the back quote syntax (back-qoute<expr>) into lazy + lambda (functools.lazy(lambda: <expr>)) and it will cover a subset of what’s supported by the new syntax.

dg-pb · November 8, 2024, 7:33pm

The link in that thread doesn’t work for some reason.

gist.github.com

https://gist.github.com/dg-pb/17005959e487556961ba92e586f8a467#file-pep_deferred_eval_concept-md

pep_deferred_eval_concept.md

**Contents:**

{{TOC}}

# 0. Backstory

See [From late-bound arguments to deferred computation, part 1](https://lwn.net/Articles/904777/) if you need to remember what was happening back then. (Personally I missed that and picked this up much later.)

# 1. Introduction
I spent a bit of time on proof of concept and I have a simple working version of `lazy` object.

This file has been truncated. show original

I have briefly explored syntax there too (including backticks). Although I was also initially pro-backticks, but there was feedback of some non-trivial historical complications for such. Also, if multiline expressions are supported, then backticks are awkward and it would be much more pythonic to have a statement-like pythonic-indentation syntax. Such as:

local_var = 1

a = lazy local_var:
    b = local_var + 1
    return b

But regardless, this syntax could just be syntactic convenience for:

local_var = 1

def foo(lazy a):
    b = a + 1
    return b

a = functools.lazy(foo, (local_var,))

But not necessarily of course. Alternatively implementation could be designed with more weight on parser, but I just don’t think it is a good idea.

dg-pb · November 8, 2024, 7:39pm

These are hard to give without at least 70% baked concept.

Also, read-up the thread after reading the document. By now it is quite clear that “late bound argument defaults” and “deferred evaluation” are orthogonal concepts and are best not to be mixed up.

dg-pb · November 8, 2024, 7:44pm

Agreed. It would be good to see robust functools.lazy implementation first.

I have done some initial drafts in Python and think it is possible, but the complexity of implementing such in C is greater by the order of magnitude. Have a look at proxy objects in wrapt package. And higher level of robustness, consideration for various nuances, etc, would be needed when implementing this in CPython stdlib.

zhangyx · November 8, 2024, 7:47pm

A decorator can convert a vanilla function into a deferred variable for you:

def defer_expr(fn):
    return `fn()`

local_var = 1

@defer_expr
def a():
    b = local_var + 1
    return b

This eliminates the need of a keyword. I don’t think the C++ style “capture clause” will help with performance nor memory footprint.

zhangyx · November 8, 2024, 7:51pm

I need a bit more time to finish reading, there are a lot of contents and posts that you pointed to. I might comeback with more questions when I am finished. (Somehow I missed the thread completely when doing my research writing this post)

Form what I’ve seen, you have already done a ton of impressive work on this. Really appreciate that you brought them up here!

zhangyx · November 8, 2024, 8:05pm

If you really need a snapshot reference to a variable:

local_var = 1

@defer_expr
def a(x=local_var):
    b = x + 1
    return b

Early-bound function defaults will work for you.

zhangyx · November 8, 2024, 8:20pm

I am super interested. Unfortunately I do not have the expertise to do this alone. I would really appreciate if someone with enough knowledge can guide me through this and make it happen.

dg-pb · November 8, 2024, 9:07pm

Lets take your use cases:

As I said, late-bound-defaults is not part of this.

def foo(a, b, c=`a+b`):
    ...

would not work, because a and b are not visible when c default is assigned. This is a separate concept as per PEP671.

A new approach of declaring zero-argument lambdas

This is a weak motivation as zero-argument lambdas can be done using lambda with no arguments. In other words, effort required for this is much greater than this benefit.

Or to expose its usefulness, it would be good to see real life applications of this that show the benefit.

A non-backdoor approach for typing forward references

I don’t use typing, but this one seems useful to me.

Personally, I am going in a slightly different direction than your proposal.

However, a possible common denominator is functools.lazy, which from my POV would be a good starting point.

And it is much easier sell than full proposal with new syntax.

And given it proves to be useful, combined with the fact that implementation exists and is functioning well, selling syntactic convenience would be much easier.

zhangyx · November 8, 2024, 9:15pm

Good catch. I did not elaborate on this since it will open up too much complexity. I probably shouldn’t use it to sell this proposal at all.

I agree. Instead of a selling point, this section is more about how to “describe” it - a “mental model” to help explain the new syntax to people who knows python but does not know this syntax.

In addition, if this proposal is lucky enough to get a chance for prototyping, I bet it will reuse a lot of infrastructures made for lambda functions.

Rosuav · November 8, 2024, 10:13pm

They weren’t really. They were listed in PEP 3099 as something that wasn’t going to be used. So you’ll have to explain what’s changed since then that means that they are now available for use.

But as with every proposal for magical deferred expressions, the key is semantics. Let’s look at yours:

Does it simply insert the text of the expression^[1] at that point? What variable scopes are available to it?

Does it capture variables or doesn’t it? If it doesn’t, this is so completely different from a lambda function that it’s only going to cause confusion to talk about them in parallel. If it does, it won’t work for function defaults.

And the biggest problem with every “magical deferred” proposal: WHEN does it get evaluated? What are the semantics of this “deferred object”? For example, if I do this:

x = `a + b`
print(x)
func(x)

what happens? Does it evaluate more than once? Does the first one print out the DeferExpr but the second one evaluate it? Does it capture the values of a and b from the caller’s scope, or does it reach into the callee’s scope to fetch them? This is a HUGE area that you will need to explain in detail. The biggest downfall of every “deferred object” proposal I’ve seen has been the inability to precisely define these semantics.

This is why PEP 671 never said anything of the sort. There is no deferred object. It is a syntactic structure that looks at the lack of an argument and executes some code.

or more precisely, its syntax tree - we don’t want C-style macro insanity ↩︎

zhangyx · November 8, 2024, 10:23pm

The expression is immediately parsed and should be converted to byte code just like any other code. Suppose you have syntax error inside it, then it should be raised during syntax evaluation time, not runtime.

The expression will capture the locals and globals at where it’s declared. It’s covered in the OP under Implementation Details 2.1.

This has been addressed in Implementation Details 1 (collapsible). I made it clear that it remains to be discussed, but favored no-collapsing and provided solutions for corner cases.