Defer Expressions

Ideas
1

This is an actual attempt at specifying the defer expression alternative to PEP-671. Note that I am not entirely sure if this is actually a good idea, and the syntax I am suggesting for sure can’t work as-is (it isn’t backwards compatible at all).

A new kind of expressions would be added, with temporary syntax defer <expr>. This would be almost equivalent to DeferType(lambda: <expr>) and would be valid anywhere the left form would also be valid. Preferably, DeferType would also store a string representation of <expr> for documentation purposes. [1]

The other half of the proposal is that when such a Defer instance is encountered in the default arguments of a function while those are copied over into the locals of the function, the callable contained within the Defer is executed and it’s value is used instead.

dataclasses.dataclass also special cases defer and treats it somewhat similar to specifying a default_factory.

DeferType instances would otherwise not have any special behavior. The underlying function can be read with .callable, it can be constructed with types.DeferType and the defer object can be called directly, i.e. defer_instance() is the exact same as defer_instance.callable().

This means that the following code would work as one might expect:

from dataclasses import dataclass, field


def test_defer(val, a=defer [1], /, b=defer [2], *, c=defer [3]):
    a.append(val)
    b.append(val)
    c.append(val)
    return a, b, c


print(test_defer('', [], b=[], c=[]))
print(test_defer('a'))
print(test_defer('b'))


@dataclass
class TestDefer:
    values1: list[int] = field(default_factory=lambda: [1])
    values2: list[int] = field(default_factory=lambda: [2], init=False)
    values3: list[int] = field(default=defer [3])
    values4: list[int] = field(default=defer [4], init=False)
    values5: list[int] = defer [5]


a = TestDefer()
a.values1.append('a')
a.values2.append('a')
a.values3.append('a')
a.values4.append('a')
a.values5.append('a')
b = TestDefer()
b.values1.append('b')
b.values2.append('b')
b.values3.append('b')
b.values4.append('b')
b.values5.append('b')

print(a, b)

A Proof-of-Concept capable of running the above code correctly is here: GitHub - MegaIng/cpython at add-defer

I am sure I made quite a few mistakes while implementing it, but that isn’t the point.

The point is whether or not this feature could be useful with these semantics, just maybe not this exact syntax or name: defer is the name used within the context of the other thread, but tbh it doesn’t quite make sense for the general feature.

IMO the benefit of this approach over 671 is that it allows for a more general usage of this feature, especially within dataclasses, and it allows an from what I can tell simpler implementation.

But I do admit that beyond dataclass-likes I don’t quite see many good uses for this feature. The primary point would be that it’s a callable that is treated specially within certain contexts.

  1. Not part of the PoC ↩︎

1 Like
2

Okay. Let’s try this.

import random

def initialize(n):
    global q
    q = defer n
initialize(random.randrange(10))
w = defer random.randrange(10)
def callme(q=q, w=w, e=defer random.randrange(10)):
    print(q, w, e)
callme()
callme(q)
callme(q, w)

Explain to me the behaviour and how you would implement this.

3

Correction: you already have something of an implementation. Alright then.

Please explain to me how this proposal can allow “more general use” of this feature, given that no more general use seems to work.

4

I still think that changing defaults to be late bound in all cases (with an appropriate future import and deprecation time) is the best option. I can’t come up with a non-contrived use case for early bound defaults, and any explicit syntax for switching between the two (with the error-prone early binding being the default) doesn’t improve the status quo, it just complicates the language.

2 Likes
5

Please explain me how your proposal is going to allow the dataclass usage shown in the OP.

And to your example, it would be a name error because q isn’t bound, but the defaults for w and e would work as expected.

6

Oh, q is bound. It would just be a constant though.

7

I don’t see how it wouldn’t be applicable, he would just need to change the default argument to be late-bound (just like you needed to change the implementation of dataclass to support DeferType), and then you get that behavior always, not just when you use defer at the call-site. I don’t feel like default=defer ... is a big improvement over default_factory=lambda: ..., in either case you need to know to use it.

Either way I don’t think dataclass is particularly compelling use-case, because the existing default_factory argument works just fine.

Your proposal just reads to me like “I want to have a margnally shorter syntax for lambda: ... that maybe signals intent a bit better and automagically gets undeferred in function defaults and one other arbitrary place in the standard library and otherwise it’s just a lambda without arguments.” That doesn’t seem very useful to me.

1 Like
8

What? Did I miss the point where the suggestion was for dataclasses to somehow become macros that are able to somehow turn the normal arguments they are getting into functions to be used later?

Note that one alternative in terms of syntax would be to only allow this directly after an assignment operator (covering all cases in OP) and use the exact same syntax as the pure late bound arguments proposal, i.e. name => expr, but with this syntax also working in normal assignment statements and keyword arguments.

9

Please elaborate. What would it mean if you used it in normal assignment?

I feel a strong sense of deja vu… proponents of “generic deferred expression” features wanting them to magically work but never elaborate enough on the semantics to explain that magic.

10

See OP or your code, what I mean is that instead of q = defer n, it would instead be written as q => n. Nothing more, nothing less. And no magic invocations, just a few places where defer objects are special cased instead of being used directly.

11

Yes, and what would the semantics be? What happens if you pass one of those defer objects as a function parameter? What happens if you put one into a dictionary? What happens if you assign it to another name? Does it always just remain as a defer object, and if so, why not lambda? Come on. Explain the proposal. I don’t want to have to pull teeth here.

12

Yes, it does remain a defer object. (No magic, just a few places where it’s special cased. I already explained my proposal. It’s just way simpler than you appear to imagine)

lambdas are obviously not an alternative inside of function defaults or for dataclasses.

13

Sorry, you are correct, I misremembered the scope of PEP671.

I still think dataclass is not a very compelling use-case for the reasons I outlined, whether or not it’s supported by PEP671 doesn’t even really play into it. If you have to use a special syntax either way, it’s only really an aesthetic improvement, not one in functionality. The final example of values: list[int] = defer [5] is slightly more compelling, but only slightly.

The only marginal benefit I see is that DeferType would not be a Callable compared to lambda, so it won’t look like a method in class definitions, so it’s easier to special case in meta classes. But you could just as well define your own DeferType that takes a single lambda, sure it’s a lot more verbose, but it’s functionally equivalent.

14

Agreed. It’s primary benefit IMO is that it mirrors function signature defaults, which is impossible with the other proposal.

It currently is a callable, but has no descriptor behaviour.

Yep. Reducing verbosity and integrating this behaviour into the function defaults are the only reasons why this would need to be a language feature.

15

So how is your proposal materially different from PEP 671? You’re proposing a thing that is exactly equivalent to a lambda function everywhere else, and exactly equivalent to PEP 671 semantics in the one and only place where PEP 671 makes a proposal. Yet you claim that this is somehow different. Dataclasses would have to be written to explicitly unpack the defer object, which is exactly how they already work with the default_factory. Your proposal is not in any way more general. It gives the deceptive impression that it does something, but it never actually does. It’s just a lambda function in disguise.

16

According to your own comment, it exactly contains PEP 671 [1], and adds behavior in other places. By definition, this is more general, no?

Which is very verbose, and doesn’t mirror the late binding abilities of function defaults despite conceptually being very similar.

Assuming instead of defer we use =>, this would be the syntax:

@dataclass
class TestDefer:
    values1: list[int] = field(default=>[4])
    values2: list[int] => [5]

Which mirrors the syntax proposed by PEP 671:

def test_defer(val, a=>[1], /, b=>[2], *, c=>[3]):
    a.append(val)
    return a, b, c

Note that I am not strictly in favor of this proposal, I am just rebutting some of your arguments I don’t view as fair.

This proposal has intentional limitations to not add too much magic. So please don’t say it not having magic makes it useless, this limitation to a useful subset of behavior is exactly the point.

  1. it doesn’t actually because it doesn’t allow references to other arguments. But that didn’t seem like a well liked feature anyway IIRC, and you haven’t even noticed that. ↩︎

2 Likes
17

No, it adds nothing that you can’t do with lambda expressions. They still have to be manually unpacked. Which means that the syntax you propose for a dataclass has to be special-cased within the dataclass, which leaves the question: what if you actually wanted the default to be a defer object? How would you represent that? Now you need a DIFFERENT way to indicate whether it’s a factory or an object. And we’re back where we started. You might just as well have said “if the default is a function, call it to get the actual default” and not bothered with the new type of object.

The defer object is never better than a lambda function. It is, in many ways, worse than one.

18

This is why defer objects are thin wrappers around callables and themself callable. The point is that you never should need to pass around defer objects unless you to have these very specific semantics. If you want to have something as a default that can just be called, use lambda: expr

I am sorry, but now you are just arguing in bad faith. Obviously, calling any function used as a default is backwards incompatible (and we already talked about this). If you don’t like the proposal, that’s fine. But don’t just claim arbitrary things to try and support your position.

Except ofcourse that it can be special cased in function defaults without problems. But apparently solving some of issues PEP 671 talks about is not a factor you are considering.

There are a few arguments you could make about how this doesn’t solve some of the problems PEP 671 solves. You have made none of those. Instead you criticizes an extra ability this syntax does allow because… it exists? You havn’t actually manged to point out why it’s bad that we have a short, cleared alternative to default_factory and similar constructs. You have just pointed out that it is a duplicate. [1]

  1. also, the special casing in dataclasses is very, very limited, primarily related to init=False. Otherwise it just gets put into the function defaults of __init__ where the default mechanism takes over the work ↩︎

1 Like
19

A short document (a pre PEP) explaining the intended semantics would be useful. In particular, in previous discussions there’s been frequent contention about the meaning of code like:

x = defer [5]
y = x
y.append(6)
z = x
print(x, y, z)

I agree that proposing a solution different from PEP671 is not a crime, less so when a draft implementation is provided.

The implementation branch is pending updates from upstream (though I don’t think I will be attempting a build).

1 Like
20

I thought the OP would fulfill that. It appears to me that you and the other readers are imaging extra semantics not mentioned in there and then complain that those semantics are unclear.

The answer is still: No magic. This is an attribute error because defer objects don’t have an append method. defer objects are only special cased in function defaults (only defaults, passing a defer object works like normal and results in a defer object being bound to the name) and some libraries should special case them to provide features like default_factory. Them having as little behavior as possible is the point. The implementation is basically a copy of cellobject, without the ability to set it and the extra ability to call it.

In what sense? Just that it isn’t perfectly up-to-date? I created and rebased it just last night, I don’t think it’s reaonsable to expect hourly updates from an implementation branch… Or did I do something wrong?

3 Likes