Future of `functools.partial`

Ideas
1

I couldn’t come to final decision on gh-125028: Prohibit placeholders in partial keywords by dg-pb · Pull Request #126062 · python/cpython · GitHub
Thus, I explored possibilities and finally convinced myself that it is best to prohibit keyword Placeholders as there is a real possibility for 1 more extension.

I am not planning on doing it any time soon, but thought it would be worth sharing findings for future reference and potential feedback.

1. Let’s define current functools.partial as V1.

It allows Placeholders in positional arguments.

2. Version V2 - kwpartial. Allowing Placeholder in keyword arguments.

Motivation.
Say there is an external function, which either has:
a) keyword-only argument of interest
b) positional-or-keyword, but very far down, say 10th argument. In this case it is impossible not to overwrite defaults in order to bring it to the front.

from itertools import starmap
_ = functools.Placeholder

def foo(a=1, b=1, c=1, *, d=1):
    return a + b + c - d

p = kwpartial(foo, _, d=_)
list_of_args = [(1, 2), (2, 3), (3, 4)]
print(list(starmap(p, list_of_args)))     # ???
# [1, 1, 1, 1] # V2 variation 1. signature(p) == foo(a, d, b=1, c=1)
# [3, 3, 3, 3] # V2 variation 2. signature(p) == foo(d, a, b=1, c=1)

So here is the first dilemma.
“Variation 1” is more natural.
However, with variation 1 it is impossible to retain coherent *args/**kwds internal representation when doing optimizations when using partial recursivley.
Thus, “variation 1” has a drastic complexity increase as it requires additional containers to store the order.
This would lead to 30-40% complexity increase (if not more).
While “variation 2” is trivial in comparison and does not require drastic changes.
It would lead to 10-15% complexity increase, which is much more sensible.

Here is a simple Python wrapper, which makes use of current partial to achieve this:
CPython issue comment

3. Version 3. - impartial (Indexed mode partial).

class IPlaceholder:
     __slots__ = ('idx',)
     ....

def foo(a=1, b=1, c=1, d=1, e=1):
    return a + b + c - d - e

_0, _1, _2 = map(IPlaceholder, range(2))
p = impartial(foo, _2, _0, _0, e=_1)
# signature(p)       # foo(b_and_c, e, a, d=1)
print(p(1, 2, 3))    # 4

This one is pretty complicated.
Overhead on top of “V2 variation 1” is smaller than it might seem.
However, it involves some inconveniences, such as merging arguments in signature, when 1 positional input fills more than one argument.
It becomes even more ambiguous with negative indices.

However, the good new is that it is possible to have a performant version of this for those rare cases where this can be useful.
Given kwpartial exists:

p = kwpartial(foo, _, _, _, e=_)
IG = opr.itemgetter(2, 0, 0, 1)
args = (1, 2, 3)
new_args = IG(*args)
print(p(*new_args))    # 4 (same as above)

This also allows negative indices.

Summary and Current conclusion

So far I think that the effort required for “V2 variation 2” might be justified.
It is fairly useful extension which does not add much complexity while providing convenient path to ad-hoc V3 - impartial.

Making kwpartial from current partial using pure python is expensive.
It results in > 100% performance overhead.

While making impartial from kwpartial is a much better deal, given performant itemgetter.
It is only ~50% (my best guess) overhead on top of kwpartial implemented in CPython, which is quite good given flexibility gain.

So my current take is that despite awkwardness of V2 variation 2, where keyword Placeholders get pushed in front of positional Placeholders, it could be a good compromise.

If say, previous partial covered 65% of cases.
V1 - positional Placeholders added 15% coverage.
V2 - keyword Placeholders would add extra 10%.
Extra 5% then has a convenient trick to get V3 - Indexed mode with attrgetter.
Which results in reasonably performant coverage for say 95% of cases.

Of course, this is speculation.
Percentage of remaining cases is hard to know.
But at least proportions of relative coverage should be somewhat sensible.

So this is mostly to document for the future.

However, if PRs on the back of current partial are merged and this is deemed worthwhile (not only by me), I might implement this.
I have worked out the logic in the process (apart from signature logic) anyways.

2

Coming at this with no particular background (I don’t know to what extent this has been discussed elsewhere) I find all of the proposed examples basically impossible to understand without some sort of comment.

I think they’d all be better expressed as either lambda expressions, or probably named functions. If this is about performance, and the overhead of a named function is an issue, we should optimise named functions (which would have a much more general impact) rather than trying to make more and more convoluted ways to use functools.partial.

But my concern here isn’t performance, it’s readability. And none of the proposed changes improve readability, IMO.

12 Likes
3

It should obviously be performant.
And performance was one of the driving factors for adding Placeholders (at least for me), but not the only one.

But now this is mostly about functionality (and completeness).

Although some effort has been made to bring mental model of partial closer to the one of lambda/def, but these are 2 are very different objects.

E.g. ad-hoc lambda is not serializable, thus lambda created at runtime has limited lifetime and is not suitable to use in various cases.

partial as opposed to lambda/def can be useful for performance, for its features and sometimes for convenience and even readability.

To put it simply, slightly different objects for different cases. To me, both very useful.

And no, it does not increase readability.
And might not be worth it at all.

4

I think the issue here is that this might be a bit too soon. I think prerequisite to easily understand this is confidence with current Placeholder functionality of partial.

5

I think partial never reads well compared to a lambda regardless of any of the extensions that are discussed here. Other languages where the equivalent of partial is more widely used have better syntax for it like currying.

1 Like
6

Agree. This is one of the things that I was thinking.

If partial had a certain level of completeness, it might be possible to consider syntactic convenience for it.

There seemed to be a fair amount of desire for similar syntax in: Introduce funnel operator i,e '|>' to allow for generator pipelines - #67 by Nodd

And this could be a relatively lightweight and very convenient solution (when combined with functools.pipe) as opposed to introducing new statement with complex syntax, while maintaining almost same level of brevity and generally would look very similar.

After all, partial is kinda curry, just lacks convenience for it. E.g.:

class curry:
    def __init__(self, func, *args, **kwds):
        if args or kwds:
            func = partial(func, *args, **kwds)
        self.func = self._ = func

    def __call__(self, *args, **kwds):
        return type(self)(self.func, *args, **kwds)

    def __ror__(self, other):
        return self.func(other)

g = curry(lambda a, b, c, d=4: a - b - c - d)
g(1)(2)(3)._(d=0)    # -4
7

I don’t think using partial recursively or even just partial(some_partial(…)) is a common pattern. There’s almost no reason to write it, as you can merge the effects of both partials by structuring whatever code here differently, so optimizing that case at the cost of single partial use seems unlikely to be an overall benefit.

I know I’ve frequently been an advocate for people using the functional programming parts of functions, but I don’t include trying to bring currying into python in an ad-hoc way with repeated use of partial in the list of things I’d advocate for.

The proposed variations don’t read as an improvement to me either, when things stop being simple partial function applications, write it explicitly as a function returning a function with a closure. This works better for anyone reviewing, and plays nicer than partial does with language servers and static analysis too.

3 Likes
8

You don’t need partial for that:

def curry(func, /, *args, **kwds):
    def newfunc(*fargs, **fkwds):
        if not (fargs or fkwds):
            return func(*args, **kwds)
        return curry(func, *args, *fargs, **(kwds | fkwds))
    return newfunc

@curry
def f(a, b, c, d=4):
    print(a - b - c - d)

f(1)(2)(3)(d=0)()  # -4
1 Like
9

I don’t get why there’s a need to allow Placeholder in keyword arguments. The whole point of Placeholder is to allow skipping positions. Keyword arguments don’t need skipping since it is directly specified by keywords.

Any index-based solution would be ugly and unmaintainable.

Placeholder/_ on the other hand provides visual cues to the relative positions to the other arguments, which is why it is accepted

2 Likes
10

Maybe there is a use case which makes sense: making arguments required (e.g encoding for open()):

If any Placeholder sentinels are present, all must be filled at call time:

>>> say_to_world = partial(print, Placeholder, Placeholder, "world!")
>>> say_to_world('Hello', 'dear')
Hello dear world!

Calling say_to_world('Hello') raises a TypeError, because only one positional argument is provided, but there are two placeholders that must be filled in.

So, this would be a logical extension (currently raises a TypeError) and could also be applied to keyword arguments:

>>> from functools import partial, Placeholder as _
>>> print3 = partial(print, _, _, _)
>>> print3() # TypeError: expected at least 3 arguments, got 0
>>> print3(1, 2, 3, 4) # OK

This is nicer than the alternative:

def print3(a, b, c, *args, **kwds):
    print(a, b, c, *args, **kwds)
1 Like
11

FTR, C++ also has an index-based solution: std::placeholders::_1, std::placeholders::_2, ..., std::placeholders::_N - cppreference.com (which was inspired by Chapter 1. Boost.Bind - 1.86.0 (boost being more or less equivalent to “lodash” in Javascript or “guava” in Java)). So it has some precedent and could make sense (note that due to the nature of the language, binding in C++ can only be done up to an implementation-defined number of parameters).

I wouldn’t call it necessarily “ugly” since it’s a matter of personal taste nor “unmaintainable” since changing the order of positional arguments should be considered as a breaking change in general. Nonetheless, I don’t think index-based placeholders would be accepted by the Python community, unless we want to make the placeholder interface as generic as possible.

3 Likes
12

Ah good to know about the precedent in C++. I agree that index-placed placeholders aren’t necessarily unmaintainable but it just isn’t very readable to me.

13

I agree. However, partial has been around for a fairly long time and removing recursive optimizations would be a breaking change.

Of course, there is an option to optimize conditional on non-presence of new features, but that results in different type of complexity.

curry made from partial with Placeholder allows skipping positional arguments so provides more flexible/complete version:

def foo(new, consumed, /, units):
    return (new - consumed) / units

a = curry(foo)
a(_, 2)(units=10)(1)()    # 0.1

To transform keyword-only argument to positional, which prepares function for certain applications, such as functools.starmap.

Or to simply transform the signature into desired form.

partial is better in a sense, that it can be used ad-hoc and remains serializable (as long as underlying func is). So it does return a new function without introduction of the inconvenient nuances that come with creating a new function. But naturally it has limitations, which I am trying to stretch a bit.

I don’t think making partial to cover all possible signature transformations is good idea. But this is what I am trying to figure out. I.e. “What is optimal cutoff point here, beyond which one should just use lambda/def (or something else)?”

I don’t know.
My current best guess is V2 variation 2 - kwpartial (with keyword Placeholders put in front of positional arguments).
It extends Placeholder to keywords and generally keeps things as they currently are without introduction of new complexities that come from issues that arise from V2 variation 1.

If V3 is ever needed, current partial then could be left as it is and new object created (say impartial) with indexed-Placeholders.

It is fairly easy to implement cross-nested-optimization between 2 different partial objects.
Much easier than to allow both non-indexed and indexed placeholders in the same object.

1 Like
14

Also, strict keyword has been implemented to map, which has closed the path for keyword-iterable-argument extension (at least in its natural form): gh-119793: Prefer `map(..., strict=True)` over starmap/zip in examples by lgeiger · Pull Request #126407 · python/cpython · GitHub

Thus, partial-keyword-Placeholder extension would provide an alternative to it by signature transformation.