For any & every

Create a thread in Help, explaining the use case, the repository from which you’re attempting to call C code, what you’ve tried so far, and any other relevant details.

@dgrigonis, is any of these notations more readable? So far, I haven’t received feedback on that yet.

I feel like the first one reads like this, which it’s not equivalent to:

result = [any(value < 0 for value in range(stop))]
result = [all(value >= 0 for value in range(stop))]

The second one is just a faster version of the one with brackets.

Correction: again parentheses.

With all the extra setup costs and interaction costs, nobody should expect that.

Isn’t it reasonable to expect if [... for ... in ...] is fast, that all(... for ... in ...) is fast too? How are you supposed to know that this only works for keywords, which doesn’t include all() & any().

I mean the compiler can only optimise when using keywords.

OK, I’m going for the notation without brackets. If “all” or “any” are followed by a comprehension without brackets, it will be optimised. (With the exception that filtering with “if” isn’t allowed).

To put this in perspective:

Proposed functionality in comparison to current list comprehensions can be equated to:

more_itertools.ilen vs builtins.map
for any & every vs list comprehension

However, more_itertools.ilen is 20 lines of C code, while builtins.map is 300 lines of code. It is fundamentally different - 1 is a function, another is a class.

So ilen is 15% shorter implementation than map
And ilen's functionality is probably 5% of what map is able to do.

So its 1-3 ratio against map, map having better functionality to implementation & maintenance ratio.

Now your proposal against list comprehension.
Let’s assume functionality is the same as in the previous case. for any & every is 5% of what list comprehension can do.

While implementation of your idea could be up to 30%, maybe even more compared to the one of comprehensions. You still need to define syntax and all the same parts as for the list comprehension.

So functionality to implementation & maintenance ratio is 1 - 6 at minimum against your proposal.

For such idea to go through as it is it should offer much more functionality.

So this is a POV from 2 dimensions only, but the way I see it - it just doesn’t cut it.

For such cost to be justified, it would either need to be highly desired feature or to offer some magically extensive and useful functionality. Benefit of being a bit faster for short-size iterables is by a magnitude insufficient to justify such idea.

It does cover 95% of use cases of all() & 91% of any():

any() & all() are simply not used for what they’re good at: existing iterables. Instead they’re used with comprehensions and map().


I updated the syntax to allow filtering (see the GitHub repository):

result = (
    any value < 0
    for stop in range(stops) if stop > 5
    for value in range(stop)
)

Which is equivalent to:

def func(stops):
    for stop in range(stops):
        if stop > 5:
            for value in range(stop):
                if value < 0:
                    return True
    return False

result = func(stops)

Well, shouldn’t we put a note in the docs that they’re only efficient for existing iterables? And that you should use a for loop if performance is desired.

What they’re good at is making code simpler and clearer. And they are used for that.

How would you write this without them, “only with loops”?

if all(f(x) for x in xs):
    foo()
elif all(g(y) for y in ys):
    bar()
else:
    qux()

Like this, with all_map() being re-usable. Might even be clearer:

def all_map(fn, iterable):
    for item in iterable:
        if not fn(item):
            return False
    return True

if all_map(f, xs):
    foo()
elif all_map(g, ys):
    bar()
else:
    qux()

And it should be faster than calling all(map()). You can provide a lambda if you need something more complex. Maybe all_map() & any_map() could be useful builtins.

Yes, but I am comparing these to comprehensions, which not only returns 1 boolean value, but the whole iterable is transformed into another one, which can be reused again, and hundreds of different functions can be used on it again. The resulting value of your proposal is 1 bit for 2 different logics.

It would be helpful if these had accompanying examples on real-life problems.

Another thing to consider, is there a way to come up with something that nicely incorporates into existing language with minimal changes without inventing a completely new language construct?

I guess I shouldn’t have used an unrealistic minimalistic example. Didn’t anticipate you’d exploit that. Something like this would’ve been better.

if all(x**2 < 100 for x in xs if x % 2):
    foo()
elif all(3*y > z**2 for y in ys for z in zs):
    bar()
else:
    qux()

That specific example is just theoretical to show how it handles if conditions and multiple for clauses.
Should we search an example in a popular repository for the others?


Maybe any_map() like you mentioned earlier?


Yeah, in that case it’s more readable:

if all x**2 < 100 for x in xs if x % 2:
if all(x**2 < 100 for x in xs if x % 2):
if all_map(lambda x: not x % 2 or x**2 < 100, xs):
if all(False for x in xs if x % 2 and x**2 >= 100):

I just had an idea that doesn’t require any (no pun intended) new syntax:

result = (value >= 0 for value in range(stop)).all()
result = (value < 0 for value in range(stop)).any()

How feasible is this? I updated the GitHub page with the new suggestion.

I, personally, like it. Definitely in the right direction. This wouldn’t require any syntax changes, but would rather be a method of a generator object. So implementation complexity has reduced by a significant factor.

I am not 100% sure, but this might be not too difficult to implement by yourself to test it out.

The main barrier here is convincing that such methods for generator object is a good idea.

1 Like

If going in this direction, then I think eliminating counting example from your examples would make things simpler for now.

It could be a separate method, e.g. (...).ntrue(n=1, pred=None), which counts up to and including n.

Within the current semantic framework of python, can a method on the generator object have a fundamental benefit over the all(... for ... in ..) syntax? Or will this syntax be special cased so much that the generator object isn’t even created? Or will this rely on JIT to have a performance benefit?

Also, this breaks the Zen of Python: There should be one-- and preferably only one --obvious way to do it.
I see a lot of confusion being caused by any(...) vs (...).any() in the future. It also reads even worse, and hides a potential important aspect of the expression at the end of it.

And if we are willing to bend the semantics of python, I would much prefer introducing the ability for the bytecode compiler to special case builtins in general, so that all(...) can be just as efficient as a for loop, for example by introducing a rule that python assumes the builtin namespaces will stay intact in all scopes (that can be turned off somehow if necessary).

3 Likes