Is there any code that relies on the current default for constructors in argument defaults? I mean the behavior that leads to:
def append_to(element, to=None):
if to is None:
to = []
to.append(element)
return to
With typing annotations, the above has to be something like the noisy:
def append_to(element: Any, to: list[Any] | None=None):
if to is None:
to = []
to.append(element)
return to
Isn’t it time to consider evaluating defaults at call time? I thought about it a while, and I don’t think any existing code would break if defaults were evaluated at call time, because a declaration like this one already raises a warning in most Python linters:
def append_to(element, to=[]):
A solution might also apply to dataclass, which requires a field when an attribute requires a constructor.
As things are, everything is logical considering the Python design history. It is not logical for people coming to Python (“Hey! See this? Well, it doesn’t work as you’d expect!”).
I’m not sure there’s a way to change that behavior without introducing backwards-compatibility headaches and/or inconsistencies with other defaults.
For this to work I think you need to evaluate the defaults at the time of the function call[1]. This might be a problem if a) your default was captured via closure (is it still in scope?), or b) a default value was complex, and you don’t want to evaluate it a bunch of times.
An alternative approach would be to create some kind of default-factory mechanism, à ladefaultdict but for an arbitrary argument. I’m not sure what syntax for that might look like but I wonder if it could be implemented via decorator? That’d be a neat trick.
I don’t think it makes sense to special-case “all mutable objects”, since that’s nearly everything ↩︎
I think even ignoring backwards-compatibility issues for a moment, the runtime cost that would add to every function call is just way too big. Function calls are already pretty expensive in Python and happen implicitly all over the place (think descriptors and dunder methods).
__defaults__/__kwdefaults__ on function objects are mutable, so you can’t even really optimize this statically in the byte code, unless you had a way to distinguish between static defaults and defaults that need to be computed at every call-site, but then you add an additional branch for every default value.
Personally I wish Python had late-bound defaults as that generally leads to more intuitive behavior, but it’s too late to change that now. Backward compatibility means we can’t make current defaults late-bound. PEP 671 proposes adding a separate syntax for late-bound defaults. That could work but it’s likely not worth the extra complication.
I got nerd-sniped and messed around with decorators to come up with this:
from functools import wraps
# original function but to is kw-only
def append_to(element, *, to = None):
to.append(element)
return to
# a wrapper that converts None to a fresh object
def default_arg(func, none_to=list):
# as far as I could see I can't get this dictionary for other arguments
none_kwargs = {kw for kw, v in func.__kwdefaults__.items() if v is None}
@wraps(func)
def new_func(*args, **kwargs):
none_kw = none_kwargs.difference(kwargs)
kwargs.update((kw, none_to()) for kw in none_kw)
return func(*args, **kwargs)
return new_func
I have many tens (or maybe hundreds) of thousands of lines of code that have never seen a linter. Changing to call time evaluation of defaults would absolutely break some things.
Which kinda languished in the face of overwhelming opposition. Every once in a while I contemplate dusting it off and giving it another try, but it’s hard to bring myself to face the gauntlet of another round of everyone hating on it.
I agree with new syntax in that it would not break backwards compatibility.
I don’t like the => in PEP-671 because it messes with current, already complex, definitions of argument defaults. I’d rather the annotation was on the value, with either a symbol or a new keyword.
Note that dataclass already makes an exception for attributes with value field(), so we’ve been here before.
I think I like the ... elipsis for the new syntax:
def append_to(element: Any, to: list[Any] = ...[]):
@dataclass
class Something:
items : list[Item] = ...[]
The syntax could be restricted to literals as to not have to deal with what ...SomeObject() means.
Well, it would have to be something that isn’t actually a valid expression, otherwise you’re right back where you started. So how would this be different?
That’s the other thing that was so exhausting about the PEP 671 debates - everyone who insisted that I should drop this proposal in favour of a generic “deferred evaluation” proposal, but wouldn’t actually put forward such a proposal. Deferred evaluation is simply not the same thing.
I remember that thread and all of the opposition you weathered!
Maybe it’s time to try to figure out deferred evaluation? —if someone has the energy to do it.
Besides late-bound parameters and dataclass field defaults, what other problems could deferred evaluation solve? Could it simplify SPEC 1’s lazy loading?
The trouble is that it has to be somehow different from a lambda function, which means it has to magically “undefer” itself on usage. But then you have the problem of: what is usage? Nobody has yet been able to pin down any sort of sane and consistent semantics for when something undefers, and I certainly don’t believe that I could achieve that.
But even if someone does, it won’t solve mutable defaults. Consider this:
x = defer some_calculation()
print(x)
print(x)
Obviously this should only perform the calculation once, right? Now consider this:
x = defer []
print(id(x))
print(id(x))
One evaluation, one object, right? In other words, both IDs should be the same value.
But that’s exactly the same as mutable defaults!
def f(x=[]): return x
print(id(f()))
print(id(f()))
If this were replaced with a deferred evaluation, it would STILL need some sort of magic that says “oh, and also evaluate this once for every function call, instead of once at function definition”. And if you had that magic, you wouldn’t need deferred evaluation.
This is why I assert repeatedly that deferred evaluation has nothing whatsoever to do with PEP 671.
PEP 690 already proposed to build a mechanism into Python that would support generic deferred evaluation. In that specific PEP, creation of deferred objects would have been limited to imports, but there is nothing import-specific about the dict-based deferred evaluation technique.
But allowing any name reference / dict access to potentially result in arbitrary code evaluation is a can of worms, which is (at least in part) why that PEP was rejected.
Okay, but I think that PEP 671 could be made a lot more convincing in my opinion if you were to:
Simplify it by removing the ability of the deferred expression to see the other parameters. It seems overcomplicated and unnecessary. I think that if someone needs to see the other parameters, they should use a sentinel and code within the function.
Change the syntax from punctuation to a soft keyword. You’re doing something complicated enough that it could be a little more obvious to a reader than an extra character.
Come up with a syntax that works for dataclass default factories too. If the same soft keyword can work in both cases, then great. The default factories illustrate the same problem you just described about “magic”.
Edit: Just thinking a bit more about this, if you an expression defer [] that is assigned as the default value of a parameter and is “unboxed” every time the function is called and that parameter is chosen, then it seems to me that dataclasses could take some expression defer [] that is assigned to a field and set that to the parameter’s default when they’re constructing __init__, and the desired unboxing would happen automatically.
Thus, I think a syntax like defer some_expression that creates a box, and the magic you were talking about for functions to unbox defaults containing such boxes when they are used would be enough to solve the problem for both dataclasses default factories and functions with mutable defaults. This is more convincing to me than PEP 671 since it solves two related problems. What do you think?
My question is could such an expression be made any more useful by broadening the situations in which it is unboxed?
Doesn’t that immediately lead to the question of what happens if you assign defer some_expression to a local variable? The “box” you’re talking about has to be a value, and therefore it can be treated as one. It’s very hard not to then end up back to all of the questions around deferred objects.
What I’m suggesting is that defer x where x is an expression having type T could create an ordinary Python value like Defer(lambda: x) where Defer is just a regular class:
@dataclass
class Defer:
__x: Callable[[], T]
The only thing that makes Defer special is that the interpreter would unbox it only when both:
it is the default value of a function’s parameter, and
that parameter was not provided.
Every other use of it would be pointless (unless someone can come up with more uses by broadening the situations in which unboxing happens).
So I can evaluate a deferred value on demand by creating a function with one argument, defaulting to this value, which just returns its argument. Call that function and you have evaluate-on-demand. As I say, it’s very easy to “break out” of limitations like you’re imagining, and have to deal with the sematics of full deferred expressions (like what to do with local variables referenced in the expression).
That’s the exact problem of “when does it undefer”. Why should it undefer on function call, and not other times? What happens if you inspect the function’s __defaults__? What if you iterate over it and print them out?
You’re asking deferreds to magically behave the way that you want them to, but that can’t happen without rigorous definition. Whereas PEP 671 defines them, not as objects, but as expressions. There simply is no object that carries the “deferred state” there. (There’s a sentinel in __defaults__ but this sentinel is not a deferred-evaluation object of any kind.)
Maybe, but that restricts the valid use-cases. Why should they NOT be able to see previously-defined parameters?
Such as what? Would it be a word that goes to the right of the equals sign? If so, it immediately become invalid for use in any context where an expression would be valid, since otherwise it would become ambiguous. How is = defer [] better than =>[]? IMO it is worse, as it implies falsely that defer [] is somehow meaningful.