Allow more than two arguments in certain functions in operator.py

Ideas
1

Introduction

Right now if you want to have the Python infix operators as functions, for example for usage as HOFs, the easiest way is to import operator to get the desired result:

# Using infix notation 
1 + 1
# Using operator.py
import operator
operator.add(1, 1)

Why do I care? I am playing around with Hissp, a Lisp that compiles into Python, and the regular operators do not work there. So we spam (operator..add) to do additions and (operator..sub) to do subtractions. And it is in this use-case that I run into the problem very quickly.

My problem

One major flaw of the functions is that they are hardcoded to allow only two arguments, which makes them less flexible and more clunky:

# Using infix notation
1 + 2 + 3 + 4
# Using operator.py
from operator import add
add(1, add(2, add(3, 4)))

where it would probably easier to just have

add(1, 2, 3, 4)

Why this behaviour?

Well a quick search of operator.py shows this:

def add(a, b):
    "Same as a + b."
    return a + b

How do we fix this?

We just need to be able to handle an arbitrary number of arguments.

Ideally, I would use functools.reduce() but in order to reduce (no pun intended) the number of modules needed to be imported, a different approach must be taken, a couple of which are possible.

The best way I can see is with a for loop:

def add(*args):
    subtotal = args[0]
    for i in args[1:]:
         subtotal += i
    return subtotal

Anyway, this should only be half an hour and a PR away. Please tell me any suggestions on my proposal.

EDIT: Probably should make this a separate library instead

2

I’m a little confused what you’re trying to do that you can’t achieve by using sum?

Sure it would work slightly differently than what you want add for, since add(*args)=sum(args), but is there a specific thing you want that doesn’t work with sum?

3

If the operator module functions were to be allowed to support more arguments, then the operators themselves should as well. (This is true of operator.call for example.) While it might not seem particularly difficult to extend addition to more arguments, the obvious question then becomes: which ones, and why, and are there semantic differences?

For example, which of these would be equivalent?

a + b + c + d
operator.add(a, b, c, d)

a - b - c - d
operator.sub(a, b, c, d)

a ^ b ^ c ^ d
operator.xor(a, b, c, d)

There are languages in which this is actually the case; for example, Pike’s equivalent of the operator module is a set of functions whose names start with a backtick, such as `+() and `*(). Several of these DO support multiple arguments, which allows - for example - the easy summation of any number of array elements using the equivalent of operator.add(*items). But this has semantic meaning, and is baked into the language accordingly. It’s not just the function that accepts more args - the operator itself does.

So - are you interested in putting together a proposal for multi-arg operators (as a generalization of existing operators)? It’ll be more complicated and harder to push for, but it wouldn’t desynchronize the operator module from the language.

(Side note: The only operator I know of in Python that truly does support arbitrary numbers of arguments, other than function calling which is a bit a special case, is assignment. You won’t find that in the operator module, but a = b = c = d is implemented as a single operation; it has three targets and one value.)

4

The problem is that it does not apply to all the operators, but of course sum() would probably be the best in every other measure.

5

The model for my behaviour is the one used in certain dialects of Lisp.

For example,

(- 1 2 3)
-4

which seems to be equivalent to

1 - 2 - 3
-4

in Python, that is, interpreted right to left

About the second question, yes, I think that would be a good idea. So would that go in a PEP? As you can probably tell I am pretty new to Python contribution.

6

Numpy has this via: np.add.reduce([1, 2, 3])

7

IMO, operator should stick to close matching up with, well, the python operators, i.e. the symbols + and the methods __add__, __radd__.

You should write a module of helper functions and use that.

2 Likes
8

I’m not familiar enough with Lisp to be able to walk through that one, but I can certainly explain the Python part for you. It is interpreted thus:

1 - 2 - 3
# is equivalent to
(1 - 2) - 3
# and is effectively
sub(sub(1, 2), 3)

You can tinker with this sort of thing in Python using the ast module:

>>> ast.dump(ast.parse("1 - 2 - 3"))
'Module(body=[Expr(value=BinOp(left=BinOp(left=Constant(value=1), op=Sub(), right=Constant(value=2)), op=Sub(), right=Constant(value=3)))])'

It’s fairly dense, but you can see a pair of BinOps (binary operations) for the subtraction - not a single operation with three operands. That’s the distinction I’m talking about; you can make a function that takes any number of arguments, but the operator module tries to stay in sync with the actual operators themselves, and in Python, they currently don’t work that way.

(Side note: These are interpreted left to right, not right to left as you described. The 1 - 2 part happens first, otherwise your result would be zero. I don’t think there are any programming languages that would evaluate this example right-to-left, but there certainly are cases where languages disagree on evaluation order, and it can be extremely confusing to those of us who work in multiple languages at once!)

Eventually. But first, write up a proposal here on Discourse. Figure out exactly what the semantics will be - don’t worry about implementation for now, but make sure you know what the expected results should be. This will need to be consistent, and will also need to retain backward compatibility. Here are a few other questions to think about:

  • What happens with obj.__add__(self, other) ? Would it now also accept multiple arguments? Or would there be a new __multiadd__ method that can take more, and if it’s absent, the interpreter calls __add__? What will this do in strange edge cases? Will this be reasonable?
  • Exactly which operators will this proposal cover? All? Just a couple? Something in between?
  • Will there be expected/mandated operational complexity here? For example, would there be an expectation that "spam" + "ham" + "eggs" operates in O(n) time rather than O(n²) time?
  • How much benefit is there, really?
  • What have other languages done in this area? Especially, are there any languages that used to have it the way you’re proposing, but now don’t? Find the reasoning behind such a change.
  • Will this break anyone’s code?

That last one is incredibly important, but sometimes hard to pin down. At the very least, you’ll need to show that “normal” code won’t be broken by this (say, something that runs 1 + 2 + 3.5 + 4 + 5.5 should still give the same result as it used to), and be prepared to find weird edge cases to discuss.

Expect a lot of pushback from people who don’t see it as valuable. Don’t get defensive; remain objective and explain the value of it. Good luck!

1 Like
9

functools is a fairly low-level standard library module, I don’t really follow the rationale for changing operator to accomodate not wanting to use functools.reduce.

The reduce() function has a Python implementation in functools.py, so if you don’t want the whole of functools, you could simply vendor that into your Hisp project, even exposing it as a builtin.

A

2 Likes
10

Why do you want to reduce the number of modules you import? If functools has what you need, you should use it.

1 Like
11

OK thanks! I will admit this is my first real venture into lower level so do correct any mistakes I make

Anyway, so something like this

def sub(*args):
    return(reduce(lambda x y: x - y, args))
12

Could I for example, test by running lots of unit-tested programs on the replacement runtime?

13

If you’re focusing only on the module, perhaps; but my recommendation is to keep the operator module in sync with the actual operators, which means that you need to look at the impact on the operators themselves.

14

I will do that first

15

Right. Thanks for all your input, and I will go study how all this works.

16

Would it be easier if I create a seperate library with multiargument variants of operator? As it seems currently no one else has an issue with how the operators work and would save a lot of hassle integrating all of this in.

17

At that point, you’re just creating a personal library of functions that do what you want, which has already been suggested as an alternative. That is most definitely an option, and a much easier one (since it doesn’t require the approval of a single other person!), but that’s no longer a proposal :slight_smile:

1 Like
18

Yeah that is probably what I should do considering it is not nearly an issue in regular Python.