There are several builtin functions that can accept a sequence and produce a scalar result: len, sum, max, min, any, all. Of these, len has some special support: it calls a __len__ dunder method belonging to the argument to implement the core logic. The others implement their own hard-coded algorithm.
My proposal is to implement corresponding __sum__, __max__ etc. that the argument can implement, with the existing logic as a fallback. This would both allow for substituting more efficient algorithms that take advantage of the classâ self-knowledge, as well as making certain operations possible that make sense but otherwise would not be supported.
Examples:
Getting the maximum/minimum values of a range should be O(1) - theyâre necessarily the greater/lesser of the first and last element, both of which are O(1) to access. Similarly, the sum can be calculated in O(1) - a little German boy figured out the trick about 240 years ago. For a range r, all(r) is equivalent to 0 not in r, and any(r) is equivalent to len(r) > 1 or (len(r) == 1 and r[0]). However, currently all of these are O(N), because the builtins donât have any idea about the pattern to the sequence.
bytes could bail out early on a max or min calculation if a 255 or 0 respectively is found, and memory hacks could potentially be used to test several elements in parallel.
Suppose we have a user-defined class representing a closed range of real numbers. That canât be iterated over or summed, but it clearly has a maximum and a minimum. Conceptually, any and all also make sense.
An object representing a geometric sequence has a meaningful sum whenever the absolute value of the ratio of terms is less than 1.
An object representing the factors of a large integer (by storing its prime factorization) can efficiently compute the sum of those factors; the max and min are trivial; and all elements are known to be at least 1 and thus nonzero.
How often do you call sum() or max() on the range object?
Note that adding a special dunder method will make the corresponding function slower in all other (more common) cases, because of the cost of an attribute lookup.
Aside from âWouldnât it be cool if âŚâ, do you have a use-case for effectively special casing these objects? It is kinda cool, but we need more practical reasons for this sort of change.
For example, 99.99% of sum() calls will be summing something other than a range object or a geometric series, where the summation will be O(N). What advantages do we gain by making the other 0.01% of calls faster, if the cost is that the other 99.99% of sums are slowed down by having to try to call a missing __sum__ dunder first?
(By the way, I think that Iâm being optimistic to suggest that as many as 1 in ten thousand sums are of a range object.)
Adding five new dunders (what about a sixth, for products?) has many costs. There is the one-off cost of adding them to the language in the first place. The on-going maintenance and testing cost. The extra memory usage. The runtime costs of testing for their existence. The extra complexity for people to learn.
All, or some, of those costs might be justified if there are good, strong use-cases where the benefits are significant. But you would need to cover each proposed dunder separately, and justify it on its own merits, not just as a batch.
Also, if you want this functionality, and are willing to pay the cost of the extra dispatching, why not use something like functools.singledispatch to create a wrapper that calls the underlying builtin unless there is a specialised version? Then you can have your optimisations without needing a change to the language.
len is special because Pythonâs builtin strings and concrete collection objects have to know their length (or easily calculate it from 2 pointers) in order to operate. .__len__ accesses that internal information that would otherwise be inaccessible. The defining variables for ranges (âvirtualâ sequences) also define their length.
None of the other collection measures have the property of being generally required and updated but inaccessible without a dunder. For sorted sequences, s[0] and s[-1] are the min and max or vice versa. For semi-sorted heapified sequences, s[0] is the min or max, and special entry and exit methods are required for updates to maintain that. In general, special purposes collections that maintain an invariant make that invariant available. An example would be a sliding window sum. People working with such instances usually know what they are working with and would not call the inefficient iteration method.
Lastly all these reduction dunders would have to be able to return non scalar values e.g. I call sum on a Nx3 numpy array with axis=1 and I would get back an array of shape 1x3
As you point out. numpy is a lot more specialized. I think the prevalence of this issue suggests a need for better/cleared documentation / tutorials for newcomers to the pydata stack, not a new dunder method that only makes sense for a third-party use case.
I could imagine a world in which Python had a dispatch hook that allowed numpy to register numpy.sum for sum(a: np.ndarray), or something like that. That seems like a much bigger but potentially useful language feature. edit: that is, akin to @pf_mooreâs post but something configured by the package and avoiding additional overhead.