Functools.partial extension to support specific positional arguments

I saw in slack a very interesting proposal to extend function.partial to support specific positional arguments, see How to fill specific positional arguments with partial in python? - Stack Overflow. Just wondering if this idea has been commented in python ideas. Has it?

I know that a lambda could be used or that I can define my own partial function that implements it. But to me it seems to be an idea worth considering to add to python.

There is a related topic but it does not mention the idea that I pointed to Support for using partial with positional only argument functions - #4 by steven.daprano

1 Like

Forgive me if I am wrong, but problems like these would be trivial to resolve with incremental binding if functools had support for curried functions, no? If so, I think functools would be better off implementing general functional programming concepts found in any functional programming language like currying or functional composition, although as we know Python developers aren’t particularly keen on that idea.

Problems like these are already trivial to resolve. Take the example from the linked thread:

from functools import partial
p = partial(
    datetime.datetime.strptime,
    partial.PLACEHOLDER,
    "%d %B, %Y"
)

This is easily achieved without even having to import functools:

p = lambda s: datetime.datetime.strptime(s, "%d %B, %Y")
1 Like

OT: it’s pretty obnoxious to link to YouTube videos.

1 Like

Sorry, I couldn’t immediately find a better resource that would indicate your position on this question, will try to dig around.

As I mentioned, I know this can be done with a lambda. Maybe better if I explain why I looked that up. Even that extension is not enough for what I would like. Essentially I would like the following to work and that static type checkers validate the types.

from functools import partial
from typing import Callable

class SomeClass:
    def __init__(self, a: str, b: int, c: float = 1.0):
        self.a = a
        self.b = b
        self.c = c

class OtherClass:
    def __init__(self, some: Callable[[str], SomeClass]):
        self.some_instance = some("value for 'a' parameter")
        assert isinstance(self.some_instance, SomeClass)

other = OtherClass(some=partial(SomeClass, ..., 2, c=3.4))

This assumes some hypothetical future partial that supports a placeholder for positionals ... and the respective types can be validated. I know that adding things to Python is difficult and a very long process. Still would be interesting to hear thoughts on this. For a shorter term I would also like to know if this is possible to implement with current python versions, including the type checking.

1 Like

Ignoring the type checking for the moment, why do you want to do this, and why isn’t lambda sufficient?

from functools import partial
from typing import Callable

class SomeClass:
    def __init__(self, a: str, b: int, c: float = 1.0):
        self.a = a
        self.b = b
        self.c = c

class OtherClass:
    def __init__(self, some: Callable[[str], SomeClass]):
        self.some_instance = some("value for 'a' parameter")
        assert isinstance(self.some_instance, SomeClass)

other = OtherClass(some=lambda a: SomeClass(a, 2, c=3.4))

As regards typing, I’d rather see a way of correctly annotating lambdas than some sort of “magic” partial that had typing support.

I have been developing jsonargparse which makes things configurable based on type hints. This already used to make dependency injection configurable, see pytorch-lightning/cli#multiple-models-and-or-datasets. However, this does not work for torch.optim.Optimizer because the first init positional must be the parameters of the model, so an optimizer instance can’t be given to the model’s init.

Currently we have a solution for this case, but I am thinking if something better can be done. Note that I am thinking on how this would be specified in a config file. Having a lambda seems not appropriate to be in a config. For certain type hints should not be in a config. A way to annotate lambdas would not be helpful.

But to me it seems to be an idea worth considering to add to python.

I disagree.

It’s true that when people learn about partial(), it is common to wonder about breaking out its left-to-right rule. However, when experimenters go down this path, they create a new problem – how do you specify which argument positions get the frozen values? There are many creative solutions; however, they are all worse than just using a lambda or def to write a wrapper function.

The best attempt I’ve seen is the better_partial project. It has many features, but the core syntax is g = partial(f)(10, _, 20, _, 30) to create the equivalent of g = lambda b, d: f(10, b, 20, d, 30).

Every variant I’ve seen is more complex, less speedy, and harder to debug than a simple def or lambda. Every variant required something extra to learn and remember, unlike def or lambda which are Python basics.

In addition, the use case is questionable. It is mostly an anti-pattern to create functions that take many positional arguments:

>>> twitter_search('#python', 20, False, True, {'US', 'UK', 'FR'}, 'json')

Another problem is that all the proposals I’ve seen restrict you from changing the argument order, from specifying default values, from writing type annotations, and from writing a better docstring. The proposals preclude options that are easily possible with def and lambda, for example:

def python_search(retweets: bool, num_tweets: int, format: str='json'):
    'Search #python hash tag in the US, France, and UK for tweets without unicode.'
    return twitter_search('#python', num_tweets, False, retweets, {'US', 'UK', 'FR'}, format)

IMO, a more advanced partial() is an attractive nuisance that steers people away from the better and obvious solution that we already have today.

5 Likes

Agreed. But nor should a partial().

Looking at the linked docs, it seems like you’re trying to infer a config file schema from the code. That sounds rather complex and fragile. It might be pretty neat if it works, but I only skimmed the linked example, so I didn’t see the advantage in that brief look.

But even so, none of this is saying that you can’t do what you’re suggesting. Writing your own variant of functools.partial isn’t particularly hard (beyond the fact that writing higher-order functions is always a bit more advanced than writing plain functions). So I’d suggest you do that. There’s no reason that you’ve mentioned which means what you want has to be in the stdlib. And the use case is niche enough that the stdlib quite probably isn’t the place for it either (especially given @rhettinger’s point that such extra functionality is often an attractive nuisance).

Thank you both for all the feedback. I agree with all your points. Anyway it did help me to think about how to do this better. I don’t need lambda, def or even my own variant of partial.

Inferring the config schema indeed is a bit complex, but I can say it does work very well, it has been tested quite enough to not be fragile and it is already used by lots of people and projects. Not easy for me to explain here the big advantages that this has.

1 Like

Thanks for the shoutout to better_partial. For what it’s worth, I initially created the library to make my life easier when using JAX to code up machine learning models. It turns out there are a lot of situations where you end up partially applying functions of many arguments and passing them around. Additionally in this context, you don’t end up paying any significant performance penalty because the better_partial function applications end up getting jitted.

That being said, I tend to be a little bullish on the better_partial project. I think it would make a nice addition to Python as a language feature. I can’t really think of a situation where I wouldn’t want a function to support the better_partial function application syntax (or something equivalent) out-of-the-box.

1 Like

Your package does look like a brilliant proposal, this is very similar to Lodash placeholder partials. However, I think that trying to use a meaningful underscore conflicts with a very well-established convention where an underscore indicates a throwaway variable, which isn’t the case in JavaScript. Something in your gist that doesn’t conflict with pre-established convention would probably make for a nice PEP.