There are several builtin functions that can accept a sequence and produce a scalar result:
all. Of these,
len has some special support: it calls a
__len__ dunder method belonging to the argument to implement the core logic. The others implement their own hard-coded algorithm.
My proposal is to implement corresponding
__max__ etc. that the argument can implement, with the existing logic as a fallback. This would both allow for substituting more efficient algorithms that take advantage of the class’ self-knowledge, as well as making certain operations possible that make sense but otherwise would not be supported.
Getting the maximum/minimum values of a
range should be O(1) - they’re necessarily the greater/lesser of the first and last element, both of which are O(1) to access. Similarly, the sum can be calculated in O(1) - a little German boy figured out the trick about 240 years ago. For a range
all(r) is equivalent to
0 not in r, and
any(r) is equivalent to
len(r) > 1 or (len(r) == 1 and r). However, currently all of these are O(N), because the builtins don’t have any idea about the pattern to the sequence.
bytes could bail out early on a
min calculation if a
0 respectively is found, and memory hacks could potentially be used to test several elements in parallel.
Suppose we have a user-defined class representing a closed range of real numbers. That can’t be iterated over or summed, but it clearly has a maximum and a minimum. Conceptually,
all also make sense.
An object representing a geometric sequence has a meaningful
sum whenever the absolute value of the ratio of terms is less than 1.
An object representing the factors of a large integer (by storing its prime factorization) can efficiently compute the
sum of those factors; the
min are trivial; and
all elements are known to be at least 1 and thus nonzero.
How often do you call
max() on the range object?
Note that adding a special dunder method will make the corresponding function slower in all other (more common) cases, because of the cost of an attribute lookup.
Aside from “Wouldn’t it be cool if …”, do you have a use-case for effectively special casing these objects? It is kinda cool, but we need more practical reasons for this sort of change.
For example, 99.99% of
sum() calls will be summing something other than a range object or a geometric series, where the summation will be O(N). What advantages do we gain by making the other 0.01% of calls faster, if the cost is that the other 99.99% of sums are slowed down by having to try to call a missing
__sum__ dunder first?
(By the way, I think that I’m being optimistic to suggest that as many as 1 in ten thousand sums are of a range object.)
Adding five new dunders (what about a sixth, for products?) has many costs. There is the one-off cost of adding them to the language in the first place. The on-going maintenance and testing cost. The extra memory usage. The runtime costs of testing for their existence. The extra complexity for people to learn.
All, or some, of those costs might be justified if there are good, strong use-cases where the benefits are significant. But you would need to cover each proposed dunder separately, and justify it on its own merits, not just as a batch.
Also, if you want this functionality, and are willing to pay the cost of the extra dispatching, why not use something like
functools.singledispatch to create a wrapper that calls the underlying builtin unless there is a specialised version? Then you can have your optimisations without needing a change to the language.
len is special because Python’s builtin strings and concrete collection objects have to know their length (or easily calculate it from 2 pointers) in order to operate.
.__len__ accesses that internal information that would otherwise be inaccessible. The defining variables for ranges (‘virtual’ sequences) also define their length.
None of the other collection measures have the property of being generally required and updated but inaccessible without a dunder. For sorted sequences, s and s[-1] are the min and max or vice versa. For semi-sorted heapified sequences, s is the min or max, and special entry and exit methods are required for updates to maintain that. In general, special purposes collections that maintain an invariant make that invariant available. An example would be a sliding window sum. People working with such instances usually know what they are working with and would not call the inefficient iteration method.
I think the best argument for these reduction dunders are numpy, etc.
Large performance difference on calling
numpy.sum on an numpy array and I think it’s a common pitfall.
Of course all of these reductions have an axis keyword (and many more which are less important) that is very import.
Lastly all these reduction dunders would have to be able to return non scalar values e.g. I call
sum on a Nx3 numpy array with
axis=1 and I would get back an array of shape 1x3
As you point out.
numpy is a lot more specialized. I think the prevalence of this issue suggests a need for better/cleared documentation / tutorials for newcomers to the pydata stack, not a new dunder method that only makes sense for a third-party use case.
I could imagine a world in which Python had a dispatch hook that allowed
numpy to register
sum(a: np.ndarray), or something like that. That seems like a much bigger but potentially useful language feature. edit: that is, akin to @pf_moore’s post but something configured by the package and avoiding additional overhead.