Functools.partial extension to support specific positional arguments

Ideas
1

I saw in slack a very interesting proposal to extend function.partial to support specific positional arguments, see How to fill specific positional arguments with partial in python? - Stack Overflow. Just wondering if this idea has been commented in python ideas. Has it?

I know that a lambda could be used or that I can define my own partial function that implements it. But to me it seems to be an idea worth considering to add to python.

There is a related topic but it does not mention the idea that I pointed to Support for using partial with positional only argument functions - #4 by steven.daprano

1 Like
2

Forgive me if I am wrong, but problems like these would be trivial to resolve with incremental binding if functools had support for curried functions, no? If so, I think functools would be better off implementing general functional programming concepts found in any functional programming language like currying or functional composition, although as we know Python developers arenā€™t particularly keen on that idea.

3

Problems like these are already trivial to resolve. Take the example from the linked thread:

from functools import partial
p = partial(
    datetime.datetime.strptime,
    partial.PLACEHOLDER,
    "%d %B, %Y"
)

This is easily achieved without even having to import functools:

p = lambda s: datetime.datetime.strptime(s, "%d %B, %Y")
1 Like
4

OT: itā€™s pretty obnoxious to link to YouTube videos.

1 Like
5

Sorry, I couldnā€™t immediately find a better resource that would indicate your position on this question, will try to dig around.

6

As I mentioned, I know this can be done with a lambda. Maybe better if I explain why I looked that up. Even that extension is not enough for what I would like. Essentially I would like the following to work and that static type checkers validate the types.

from functools import partial
from typing import Callable

class SomeClass:
    def __init__(self, a: str, b: int, c: float = 1.0):
        self.a = a
        self.b = b
        self.c = c

class OtherClass:
    def __init__(self, some: Callable[[str], SomeClass]):
        self.some_instance = some("value for 'a' parameter")
        assert isinstance(self.some_instance, SomeClass)

other = OtherClass(some=partial(SomeClass, ..., 2, c=3.4))

This assumes some hypothetical future partial that supports a placeholder for positionals ... and the respective types can be validated. I know that adding things to Python is difficult and a very long process. Still would be interesting to hear thoughts on this. For a shorter term I would also like to know if this is possible to implement with current python versions, including the type checking.

1 Like
7

Ignoring the type checking for the moment, why do you want to do this, and why isnā€™t lambda sufficient?

from functools import partial
from typing import Callable

class SomeClass:
    def __init__(self, a: str, b: int, c: float = 1.0):
        self.a = a
        self.b = b
        self.c = c

class OtherClass:
    def __init__(self, some: Callable[[str], SomeClass]):
        self.some_instance = some("value for 'a' parameter")
        assert isinstance(self.some_instance, SomeClass)

other = OtherClass(some=lambda a: SomeClass(a, 2, c=3.4))

As regards typing, Iā€™d rather see a way of correctly annotating lambdas than some sort of ā€œmagicā€ partial that had typing support.

8

I have been developing jsonargparse which makes things configurable based on type hints. This already used to make dependency injection configurable, see pytorch-lightning/cli#multiple-models-and-or-datasets. However, this does not work for torch.optim.Optimizer because the first init positional must be the parameters of the model, so an optimizer instance canā€™t be given to the modelā€™s init.

Currently we have a solution for this case, but I am thinking if something better can be done. Note that I am thinking on how this would be specified in a config file. Having a lambda seems not appropriate to be in a config. For certain type hints should not be in a config. A way to annotate lambdas would not be helpful.

9

But to me it seems to be an idea worth considering to add to python.

I disagree.

Itā€™s true that when people learn about partial(), it is common to wonder about breaking out its left-to-right rule. However, when experimenters go down this path, they create a new problem ā€“ how do you specify which argument positions get the frozen values? There are many creative solutions; however, they are all worse than just using a lambda or def to write a wrapper function.

The best attempt Iā€™ve seen is the better_partial project. It has many features, but the core syntax is g = partial(f)(10, _, 20, _, 30) to create the equivalent of g = lambda b, d: f(10, b, 20, d, 30).

Every variant Iā€™ve seen is more complex, less speedy, and harder to debug than a simple def or lambda. Every variant required something extra to learn and remember, unlike def or lambda which are Python basics.

In addition, the use case is questionable. It is mostly an anti-pattern to create functions that take many positional arguments:

>>> twitter_search('#python', 20, False, True, {'US', 'UK', 'FR'}, 'json')

Another problem is that all the proposals Iā€™ve seen restrict you from changing the argument order, from specifying default values, from writing type annotations, and from writing a better docstring. The proposals preclude options that are easily possible with def and lambda, for example:

def python_search(retweets: bool, num_tweets: int, format: str='json'):
    'Search #python hash tag in the US, France, and UK for tweets without unicode.'
    return twitter_search('#python', num_tweets, False, retweets, {'US', 'UK', 'FR'}, format)

IMO, a more advanced partial() is an attractive nuisance that steers people away from the better and obvious solution that we already have today.

7 Likes
10

Agreed. But nor should a partial().

Looking at the linked docs, it seems like youā€™re trying to infer a config file schema from the code. That sounds rather complex and fragile. It might be pretty neat if it works, but I only skimmed the linked example, so I didnā€™t see the advantage in that brief look.

But even so, none of this is saying that you canā€™t do what youā€™re suggesting. Writing your own variant of functools.partial isnā€™t particularly hard (beyond the fact that writing higher-order functions is always a bit more advanced than writing plain functions). So Iā€™d suggest you do that. Thereā€™s no reason that youā€™ve mentioned which means what you want has to be in the stdlib. And the use case is niche enough that the stdlib quite probably isnā€™t the place for it either (especially given @rhettingerā€™s point that such extra functionality is often an attractive nuisance).

1 Like
11

Thank you both for all the feedback. I agree with all your points. Anyway it did help me to think about how to do this better. I donā€™t need lambda, def or even my own variant of partial.

Inferring the config schema indeed is a bit complex, but I can say it does work very well, it has been tested quite enough to not be fragile and it is already used by lots of people and projects. Not easy for me to explain here the big advantages that this has.

1 Like
12

Thanks for the shoutout to better_partial. For what itā€™s worth, I initially created the library to make my life easier when using JAX to code up machine learning models. It turns out there are a lot of situations where you end up partially applying functions of many arguments and passing them around. Additionally in this context, you donā€™t end up paying any significant performance penalty because the better_partial function applications end up getting jitted.

That being said, I tend to be a little bullish on the better_partial project. I think it would make a nice addition to Python as a language feature. I canā€™t really think of a situation where I wouldnā€™t want a function to support the better_partial function application syntax (or something equivalent) out-of-the-box.

2 Likes
14

Your package does look like a brilliant proposal, this is very similar to Lodash placeholder partials. However, I think that trying to use a meaningful underscore conflicts with a very well-established convention where an underscore indicates a throwaway variable, which isnā€™t the case in JavaScript. Something in your gist that doesnā€™t conflict with pre-established convention would probably make for a nice PEP.

15

Just a heads-up to revive this discussion a bit, here is a scenario where I think better-partial show to be a nice addition to the functools module.

Suppose you have a situation where within your function you want to divide a datetime.timedelta objects into minutes and seconds. You opt to use divmod(seconds, 60) for this. But to make your code more Pythonic and ready to be pulled into production, you face some dilemmas:

  1. Keep it as divmod(seconds, 60), which introduces a magic number in your otherwise well-structured, clean codebase.

  2. Declare a constant SECONDS_IN_A_MINUTE and turn your code into divmod(seconds, SECONDS_IN_A_MINUTE), which solves the issue above, but it is kind of inelegant to introduce a constant if you only use divmod() once in your script, and that constant is not used elsewhere.

  3. Define a function using seconds_to_minutes = lambda s: divmod(s, 60), which solves the issue above but breaks the PEP 8 and PyLint unless you add in # NOQA, which again is inelegant - codebases that spam NOQA left and right tend to raise eyebrows.

  4. Define a function using a def statement, which solves the issue above but is cumbersome and takes a lot of space, especially if you want it to play nicely with MyPy and PyLint and write a whole new docstring, long enough argument name and a bunch of type hints for what should have been a dead-simple one-liner that doesnā€™t visually bloat your module:

def seconds_to_minutes(seconds: float) -> tuple[float, float]:
    """Convert seconds into a tuple of total minutes and remaining seconds."""
    return divmod(seconds, 60)
  1. With better-partial-style partials, you would solve the issues with 1, 2, 3 and 4 with seconds_to_minutes = partial(divmod, _, 60) or seconds_to_minutes = partial(divmod, ..., remainder=60), which in my opinionated view is the most professional-looking of all the options out there. I donā€™t see any obvious downsides here. But, you wouldnā€™t want to introduce a whole new dependency package just to make your codebase look more professional, so that is not an option for you with the way functools.partial work at the moment.

The way I see this, partials have nothing to lose from getting this kind of functionality natively. Itā€™s a package with 100+ stars, so the public interest clearly is there, and it looks pretty much feature-complete. What do you all think?

17

Oh, so a lambda is wrong because it violates PEP 8, but some other bit of magic syntax is better? Iā€™m not buying that.

8 Likes
19

What would you feel that a better placeholder than magic dash syntax is possible? I donā€™t feel like the idea of a placeholder itself is grossly un-Pythonic.

20

Iā€™m 100% opposed to including better-partial or anything like it in the standard library. At best, it would be an attractive nuisance. The built-in def keyword is better in almost every way.

Writing a simple wrapper with def lets you easily replace any arguments you want, reorder the arguments, rename the arguments, support positional and keyword arguments, provide default values, add type annotations, put in / positional-only and * keyword-only restrictions, add a docstring, work easily with a debugger, insert prints, insert logging, adjust argument values, change datatype types, work with MyPy, work with lint tools, etc. Using def is 100% backwards compatible, requires no new learning, and is constantly being optimized to be ever faster. For simpler, in-line cases, a lambda would usually suffice.

FWIW, the seconds_to_minutes() example reflects a misunderstanding of how better-partial works. The fix requires a double call: seconds_to_minutes(divmod)(_, 60). Once "fixed", the docstring is None and the annotations are empty. Tooltips show (x, /) rather than the well named parameter seconds in the def-version. The 60 is still a "magic constant". Disassembly expands to over 200 lines with the partial-version versus 8 lines with the def-version. Calling pdb.run(ā€˜seconds_to_minutes(123)ā€™) takes you through an enormous and distracting number of steps. The running time goes up two orders of magnitude to 3.38 usec per loop for the partial-verison versus the 46.3 nsec per loop` in the def-version. IMO, the partial-version is a small catastrophe and I would not allow it to pass code review.

Using better-partial also costs you in terms of flexibility and maintenance. With the def-version, it is trivially easy to make changes such as:

Change the signature to tuple[int, int]:

def seconds_to_minutes(seconds: float) -> tuple[int, int]:
    """Convert from nearest second to a tuple of total minutes and remaining seconds."""
    return divmod(round(seconds), 60)

Accept a timedelta input:

def seconds_to_minutes(difference: timedelta) -> tuple[float, float]:
    """Convert a timedelta into a tuple of total minutes and remaining seconds."""
    return divmod(difference.seconds, 60)

Every way I look at it, the better-partial version makes us worse off.

One last thought. You seem to be fighting your tooling. PyLint and PEP 8 were trying to tell you that the docstrings, annotations, and parameter names were missing in your lambda version. The tools had a good point. If you must have a one-liner, then learn to embrace lambda and explicitly add # NOQA to acknowledge the price you paid for having one line. Donā€™t use better_partial.partial() to sneak past those tools.

11 Likes
21

I see your point, Raymond, and it is a good one.
Letā€™s imagine that I was a much more talented coder, and I were to bring to the table a Cython-based pull request for functools.partial to add placeholders. It has a syntax that GvR finds acceptable, and in the performance/debugging department compares as nicely as Lodash fancy partials compare to JavaScript arrow functions. My justification for this pull request would be that, while it may not be a deal-breaker due to the existence of lambdas, it doesnā€™t consistently nudge users to fight against the best and the most ingrained tooling the Python ecosystem has, such as PEP 8 inspection linters, PyLint, MyPy and pre-commit. Would you consider this justification to be weak, and be inclined to reject the pull request?

22

You donā€™t need a new DSL written in Cython to find happiness. If your tooling tells you that your function needs a docstring, then either write one or silence the suggestion.

1 Like
23

Thanks for the analysis Raymond, but I am surprised, even shocked, at the overhead of better_partial. (Perhaps it should be called ā€œworse partialā€ :slight_smile:

According to my quick and dirty test, the stdlib partial is much faster than calling a regular function. Is my test broken?

[steve ~]$ python3.10 -m timeit -s "def f(): return len([])" "f()"
5000000 loops, best of 5: 93.2 nsec per loop

[steve ~]$ python3.10 -m timeit -s "from functools import partial; f = partial(len, [])" "f()"
10000000 loops, best of 5: 30.1 nsec per loop

I agree with your other comments about documentation, etc, and fighting the tooling, etc. But the impression I have got is that functools.partial is faster than a regular def/lambda wrapper function. What am I missing?

2 Likes