For any & every

[Nineteendo (Nice Zombies)]

NOTE: Check GitHub - nineteendo/for-any-each for the latest version.

The problem

Recently, #118946 was merged, replacing any() with a for loop:

-return any(key in m for m in self.maps)
+for mapping in self.maps:
+    if key in mapping:
+        return True
+return False

Why? Because any() is slower in this case:

script

# for_all_any.py
def all_base(stop):
    return all(value >= 0 for value in range(stop))

def any_base(stop):
    return any(value < 0 for value in range(stop))

def all_true(stop):
    return all(False for value in range(stop) if value < 0)

def any_true(stop):
    return any(True for value in range(stop) if value < 0)

def all_loop(stop):
    for value in range(stop):
        if value < 0:
            return False
    return True

def any_loop(stop):
    for value in range(stop):
        if value < 0:
            return True
    return False

# for_all_any.sh
echo 1 item
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_base(1)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_base(1)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_true(1)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_true(1)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_loop(1)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_loop(1)"
echo 10 items
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_base(10)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_base(10)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_true(10)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_true(10)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_loop(10)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_loop(10)"
echo 100 items
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_base(100)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_base(100)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_true(100)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_true(100)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_loop(100)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_loop(100)"
echo 1000 items
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_base(1000)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_base(1000)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_true(1000)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_true(1000)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.all_loop(1000)"
main/python.exe -m timeit -s "import for_all_any" "for_all_any.any_loop(1000)"

2.56x slower for 1 item

1000000 loops, best of 5: 361 nsec per loop  # all_base
1000000 loops, best of 5: 358 nsec per loop  # any_base
1000000 loops, best of 5: 333 nsec per loop  # all_true
1000000 loops, best of 5: 337 nsec per loop  # any_true
2000000 loops, best of 5: 142 nsec per loop  # all_loop
2000000 loops, best of 5: 141 nsec per loop  # any_loop

2.30x slower for 10 items

10 items
500000 loops, best of 5: 650 nsec per loop  # all_base
500000 loops, best of 5: 667 nsec per loop  # any_base
500000 loops, best of 5: 480 nsec per loop  # all_true
500000 loops, best of 5: 483 nsec per loop  # any_true
1000000 loops, best of 5: 283 nsec per loop  # all_loop
1000000 loops, best of 5: 283 nsec per loop  # any_loop

2.12x slower for 100 items

100 items
100000 loops, best of 5: 3.41 usec per loop  # all_base
100000 loops, best of 5: 3.43 usec per loop  # any_base
200000 loops, best of 5: 1.77 usec per loop  # all_true
200000 loops, best of 5: 1.78 usec per loop  # any_true
200000 loops, best of 5: 1.61 usec per loop  # all_loop
200000 loops, best of 5: 1.61 usec per loop  # any_loop

1.68x slower for 1000 items

1000 items
5000 loops, best of 5: 40.5 usec per loop  # all_base
5000 loops, best of 5: 40.9 usec per loop  # any_base
10000 loops, best of 5: 23.8 usec per loop  # all_true
10000 loops, best of 5: 24 usec per loop  # any_true
10000 loops, best of 5: 24.3 usec per loop  # all_loop
10000 loops, best of 5: 24.1 usec per loop  # any_loop

Quoting the Zen of Python:

There should be one-- and preferably only one --obvious way to do it.

This is clearly not the case here, the more code you use the faster it gets.
Therefore, I propose a syntax for all() & any() with a loop, which the compiler can optimize:

result = value < 0 for any value in range(stop)
result = value >= 0 for every value in range(stop)

Syntax

comprehension         ::= assignment_expression comp_for
comp_for              ::= comp_unquantified_for | comp_quantified_for
comp_unquantified_for ::= ["async"] "for" target_list "in" or_test
                          [comp_iter]
comp_quantified_for   ::= ["async"] "for" quantifier target_list "in"
                          or_test [comp_quantified_for]
quantifier            ::= any | every
comp_iter             ::= comp_for | comp_if
comp_if               ::= "if" or_test [comp_iter]

Can always be extended later, but only useful expressions should be allowed.

Examples

expand

Example 1

result = value < 0 for any value in range(stop)

Would be roughly equivalent to:

def func(stop):
    for value in range(stop):
        if value < 0:
            return True
    return False

result = func(stop)

Example 2

result = value >= 0 for every value in range(stop)

Would be roughly equivalent to:

def func(stop):
    for value in range(stop):
        if value < 0:
            return False
    return True

result = func(stop)

Example 3

result = (
    value < 0
    for any stop in range(stops)
    for any value in range(stop)
)

Would be roughly equivalent to:

def func(stops):
    for stop in range(stops):
        for value in range(stop):
            if value < 0:
                return True
    return False

result = func(stops)

Example 4

result = [
    value < 0
    for stop in range(stops)
    for any value in range(stop)
]

Would be roughly equivalent to:

def func(stop):
    for value in range(stop):
        if value < 0:
            return True
    return False

result = []
for stop in range(stops)
    result.append(func(stop))

Other discussions about comprehensions

expand

Itertools.takewhile but in a list comprehension

result = [value for value in range(stop) while value < 10]

Would be roughly equivalent to:

result = []
for value in range(stop):
    if value >= 10:
        break
    result.append(value)

Feedback:

• Cognitive overload
• Uncommon
• Intuitive
• Breaks duality with for loops

Why no tuple comprehension?

• (... for ... in ...) is already used for generator expressions
• Uncommon
• Tuples are immutable

Exception handling syntax in comprehensions

result = [1 / value for value in range(stop) except ZeroDivisionError]

Would be roughly equivalent to:

result = []
for value in range(stop):
    try:
        result.append(1 / value)
    except ZeroDivisionError:
        pass

Feedback:

• Less expressive than PEP 463
• Uncommon

Since comprehension is now inline-ed, shall python allow “yield inside comprehension” again?

def func(stop):
    while True:
        L = [(yield) for _ in range(stop)]
        print(L)

Would be roughly equivalent to:

def func(stop):
    while True:
        L = []
        for _ in range(stop):
            L.append((yield))
        print(L)

Feedback:

• Breaks expectation that comprehensions work in one go
• If (yield) is forbidden in generator expression it should be forbidden in other comprehensions too
• Uncommon

Allow comprehension syntax in loop header of for loop

for value in range(stop) if value % 3:
    print(value)

Would be roughly equivalent to:

for value in range(stop):
    if value % 3:
        print(value)

Feedback:

• Not readable
• You can use a generator expression
• There should be one-- and preferably only one --obvious way to do it

Proposal: for in match within list comprehension

result = [name for city in cities match City(country='BR', name=name)]

Would be roughly equivalent to:

result = []
for city in cities:
    match city:
        case City(country='BR', name=name):
            results.append(name)

Feedback:

• Hard to read
• You can use duck typing
• Doesn’t offer many optimisation opportunities

Using unpacking to generalize comprehensions with multiple elements

result = [*(value, value + 1) for value in range(1, stop, 3)]

Would be roughly equivalent to:

result = []
for value in range(1, stop, 3):
    result.extend((value, value + 1))

Feedback:

• You can use nested for loops
• Not intuitive

Yield comprehension

def func1(stop):
    yield value ** 2 for value in range(stop)

Would be roughly equivalent to:

def func1(stop):
    for value in range(stop):
        value ** 2

Feedback:

• Acts differently with parentheses
• You can use yield from
• Where does send(...) go?

where clauses in List comprehension

result = [string for string in strings if number % 2 where number = int(string)]

Would be roughly equivalent to:

result = []
for string in strings:
    number = int(string)
    if number % 2:
        result.append(string)

Feedback:

• You can use :=
• There should be one-- and preferably only one --obvious way to do it

GitHub searches

1. Roughly 2.7M files with all() & for
2. Roughly 1.8M files with any() & for
3. >35.8K files with all() & map()
4. >22.1K files with any() & map()

[Nineteendo (Nice Zombies)]

I replaced “every” with “each” as that’s 1 character shorter and still grammatically correct.

[barneygale]

Please re-read this post from 2 days ago. You’re still throwing out half-baked ideas and it’s becoming quite annoying.

Alternative call syntax

That’s not entirely true. Yes, library changes will get pushback of the form “why can’t this be on PyPI”, but that doesn’t mean they never get accepted. It just means the bar for acceptance is high (a lot higher than simply “I just had a good idea, what do you think?”). But the barrier for acceptance of syntax changes is also high - probably even higher than library changes, notwithstanding the previous point. The truth is, getting any sort of change into a widely used, mature language like Py…

[Nineteendo (Nice Zombies)]

@pf_moore

All you are doing, as has already been mentioned, is using up people’s patience with you to the point where you’re going to start getting ignored and your ideas dismissed.

Sorry, it feels like I already used up your patience with me in my previous posts. So, you just dismissed this one without reading it. That’s fine, focus on improving pathlib instead of wasting your time with my ideas.

---

@pf_moore

Once you’ve established a reputation like that, then you might find that people are receptive to proposals from you - especially if you take on board all the experience you’ve gained watching how other people’s proposals have fared, and make your own proposal as good as you possibly can.

1. Unlike some other suggestions for improving comprehensions, I’m proposing a faster alternative to constructs which are widely used.
2. The notation is natural, corresponding to logical expressions:
   • \forall x \in A: x >= 0
   • \exists x \in A: x < 0.
3. There isn’t an obvious way to write these checks that’s both fast and concise, this proposal tries to address that.

[MegaIng (Cornelius Krupp)]

@Nineteendo

@barneygale

You’re still throwing out half-baked ideas and it’s becoming quite annoying.

Do you think that because I replaced “every” with “each”, indicating that the details might not be finalised yet? Or because I don’t have an implementation for this idea?

I am not Barney, but I would also say it’s half baked. You haven’t actually thought this through. Modifying the comprehension syntax is fundamentally the wrong idea. You want to add a new syntax that looks like Comprehensions. The existing comprehensions result in multiple values and can be used to construct generators, sets, lists or dictionaries. This is fundamentally not what your proposal does. I don’t see you address this anywhere. This for example means that your “syntax” section is just incomplete: It doesn’t actually allow the examples you want to write down.

I am going to make a slightly stronger suggestion than has been made up until now by others: Please don’t make further suggestion unless you have a working prototype. Doesn’t have to pretty code, it just needs to show that you considered this enough to get something that actually can work.

[Nineteendo (Nice Zombies)]

@MegaIng

Modifying the comprehension syntax is fundamentally the wrong idea. You want to add a new syntax that looks like Comprehensions.

Do you have a better idea? Implementing this without using any new keywords is hard.
I’m happy to drop this in case we can do this without using the comprehension syntax though.

---

@MegaIng

This for example means that your “syntax” section is just incomplete: It doesn’t actually allow the examples you want to write down.

You mean that comprehension should always result in multiple values and quantified_comprehension in a single value? That’s should be an easy fix.

---

@MegaIng

Please don’t make further suggestion unless you have a working prototype.

I think it would be a huge waste of my time if I spent weeks implementing something that’s rejected with one word: “NO!”. Writing this up already took long enough, because I did some research for once. Also, I’m far more experienced with Python than C as C isn’t as user friendly.

[MegaIng (Cornelius Krupp)]

@Nineteendo

Do you have a better idea? Implementing this without using any new keywords is hard.
I’m happy to drop this in case we can do this without using the comprehension syntax though.

That’s not my point. I do somewhat like the syntax, although I don’t think it’s all that clear. The point is that you don’t seem to understand what comprehensions currently are and what exactly the implications of your suggested change are.

@Nineteendo

You mean that comprehension should always result in multiple values and quantified_comprehension in a single value? That’s should be an easy fix.

Surely quantified_comprehensions can only ever result in one value? Otherwise you need to describe the semantics better, since I don’t know how you expected that to potentially result in multiple values.

But what I mean is that currently, a = x for x in range(5) is not valid syntax. Nothing in your proposed syntax changes enables a = x for any x in range(5) to be valid. And you have not discussed the implications of allowing unbracketed comprehension-like syntax, which would be completely new syntax.

[Nineteendo (Nice Zombies)]

@MegaIng

Surely quantified_comprehensions can only ever result in one value? Otherwise you need to describe the semantics better, since I don’t know how you expected that to potentially result in multiple values.

Yes, that’s why they need to be split up from unquantified_comprehensions. I updated the syntax above.

---

@MegaIng

But what I mean is that currently, a = x for x in range(5) is not valid syntax.

Oh, I thought that worked for generators. I haven’t used them very often. I added parentheses.

[JamesParrott]

I got similar timing benchmarks from 1000 items. There’s absolutely no need for new syntax, but perhaps it’s food for thought about the implementation of any and all. But if 1000 iterations in 90 microseconds isn’t fast enough for some application, perhaps Python is not the best tool for it in the first place.

[Nineteendo (Nice Zombies)]

@JamesParrott

perhaps it’s food for thought about the implementation of any and all

That’s sadly not possible, they are regular builtin functions (no keyword), all they see is a generator.

---

@JamesParrott

if 1000 iterations in 90 microseconds isn’t fast enough for some application

I think it’s still bad that you need to write out the for loop if you want the best performance.
I don’t think it’s productive to replace all occurences in the stdlib with for loops, which is why I think we need this.

[Rosuav (Chris Angelico)]

@Nineteendo

Sorry, it feels like I already used up your patience with me in my previous posts.

You have used up a LOT of people’s patience. Probably more people than you realise - you won’t know how many people just see your name and move on. Or maybe have even blocked you, so they don’t even see your posts.

@Nineteendo

Do you think that because I replaced “every” with “each”, indicating that the details might not be finalised yet? Or because I don’t have an implementation for this idea?

Neither. It’s because the idea gives the impression that it hasn’t had a lot of effort put into figuring out how well it fits into Python.

I’m not going to explain that further, because you have already been told to go and read more of the archives instead of continuing to burn what patience people have left.

[Rosuav (Chris Angelico)]

@Nineteendo

I think it would be a huge waste of my time if I spent weeks implementing something that’s rejected with one word: “NO!”. Writing this up already took long enough, because I did some research for once.

How many hours did you spend on it?

How many people have read this thread? How much of their time have you consumed?

How much additional time of everyone’s have you wasted by, once again, making substantive edits to your posts, despite repeated requests to not do this?

Try to get a very very rough idea of how much of other people’s time you’re wasting with half-baked ideas, and then maybe the “huge waste of time” will be put into better perspective.

[Nineteendo (Nice Zombies)]

@Rosuav

I’m not going to explain that further, because you have already been told to go and read more of the archives instead of continuing to burn what patience people have left.

I think I went through enough discussions about comprehensions. Expand this: For any & every

---

@Rosuav

How many hours did you spend on it?

Frankly, I don’t know. I started writing this 2 days ago: GitHub - nineteendo/for-any-each

---

@Rosuav

How many people have read this thread? How much of their time have you consumed?

128 people have viewed (not necessary read) this thread. I don’t know how much time they spent on it.

---

@Rosuav

How much additional time of everyone’s have you wasted by, once again, making substantive edits to your posts, despite repeated requests to not do this?

I’m not the only person that does this, but you only complain about me. And I have mentioned every change I made, do I need to provide a diff as well? What’s the downside of cleaning them up?

[Nineteendo (Nice Zombies)]

I updated the examples and the syntax:

@Nineteendo

result = (value < 0 for any value in range(stop))
result = (value >= 0 for each value in range(stop))

@Nineteendo

unquantified_comprehension ::= assignment_expression comp_unquantified_for
quantified_comprehension   ::= assignment_expression comp_quantified_for
comp_for                   ::= comp_unquantified_for | comp_quantified_for
comp_unquantified_for      ::= ["async"] "for" target_list "in" or_test
                               [comp_iter]
comp_quantified_for        ::= ["async"] "for" quantifier target_list "in"
                               or_test [comp_quantified_for]
quantifier                 ::= any | each
comp_iter                  ::= comp_for | comp_if
comp_if                    ::= "if" or_test [comp_iter]

[elis.byberi]

(This is off-topic.)

For reference, I have over 20 years of experience in programming in general, and over 10 years in Python. However, I still wouldn’t call myself proficient in Python; I tend to forget things I don’t use frequently and often learn new things the hard way. I haven’t read the full Python reference from top to bottom because many concepts are outside my interest and understanding, making little sense to me.

In recent years, I’ve been implementing a Domain Specific Language (DSL), which has sparked my keen interest in Python ideas. When you see me leaving a comment, you should know that I’ve been studying the topic for weeks or months across multiple programming languages.

Also, you might have noticed that I don’t have a single comment in typing-related threads. It’s beyond my expertise!

[Nineteendo (Nice Zombies)]

In case you missed this edit:

@Nineteendo

@barneygale

You’re still throwing out half-baked ideas and it’s becoming quite annoying.

Do you think that because I replaced “every” with “each”, indicating that the details might not be finalised yet? Or because I don’t have an implementation for this idea?

I only recently realised for all ... in ... wasn’t gramatically correct, so I replaced this with for every ... in .... Because this was a bit too long for my liking I looked for a synonym replacing it with for each ... in ....

I think the rest is well thought out. Although there might be a better syntax for this. Not sure.

[Kxnr (Connor Keane)]

Another, hopefully gentle, perspective from someone new to this community ^[1] is to consider the audience here. On the whole, folks are here because they like Python. It may not be perfect, but that’s true of every language ^[2]. Making a change to Python means understanding what challenges other folks here face and getting the community ^[3] to agree that there’s a problem worth solving. Building that kind of consensus is hard, especially in a community as broad as Python’s, and that’s before trying to get the community to agree on a solution.

As Paul mentioned in the linked thread ^[4], this means listening to other people here and understanding what they want solved. Perhaps more importantly, understanding why whatever that is isn’t solved yet. An idea being under baked doesn’t only mean that the syntax is underspecified or some similar technicality. It also means understanding who needs this problem solved and what they need out of the solution.

edit: whitespace

---

1. longtime lurker, recently first time caller ↩︎

2. English included ↩︎

3. TM ↩︎

4. on the off chance you’re reading this, thanks. I appreciate the advice, even if I’m not directly part of the conversation ↩︎

[fungi]

128 people have viewed (not necessary read) this thread. I don’t
know how much time they spent on it.

Those of us participating in “mailing list mode” don’t show up in
that count.

I’m not the only person that does this, but you only complain
about me. And I have mentioned every change I made, do I need to
provide a diff as well? What’s the downside of cleaning them up?

Those of us participating in “mailing list mode” will never see your
edits, because they’re only to the historical copy kept on the web
forum and aren’t sent as updates to subscribers.

[pf_moore (Paul Moore)]

@Nineteendo

In case you missed this edit:

Please stop editing posts. Instead of thinking of your posts as documents that can be revised, think of them as part of a conversation. When you talk to someone, what you say can’t be changed or unsaid. Instead you have to make a new statement correcting yourself and incorporating whay others have said into your subsequent statements. That’s how people read forum posts, and interact with others on forums. Edits disrupt that flow, and are frankly considered impolite.

If you want to be part of this community, you need to work with the establish models of communication, not continually fight against them.

@Nineteendo

I’m not the only person that does this, but you only complain about me.

Not particularly true. Generally, everyone who doesn’t follow community norms gets corrected. Some accept the correction and get better. Others refuse to listen and get ignored, ultimately leaving the community. From my point of view, you’re somewhere in the middle (and that might be why you’re seeing so much pushback). You appear willing to learn and try to fit in with the community, so we are happy to help you do that^[1]. But as time goes on, your actions don’t seem to match your words - you engage with the advice you’ve been given and discuss it, but your behaviour isn’t changing. The willingness to help at that point turns into frustration, and finally people simply start giving up as the effort we’re putting into trying to help you work with the community is starting to feel wasted.

At this point, I honestly don’t know if the comment above will help you. I’m not quite willing to give up trying yet, but I’m no longer optimistic.

@Nineteendo

Sorry, it feels like I already used up your patience with me in my previous posts. So, you just dismissed this one without reading it.

… and it’s comments like this that make me think I’m getting nowhere with you.

Why do you think I didn’t read your suggestion? I’m borderline insulted that you think I’d comment without trying to understand what I’m responding to. The truth is, I do read your suggestions (this and the others you have made) but they are so vague and unclear that I barely understand them most of the time.

In this post, you proposed a syntax. You even wrote out the BNF. Great. But the syntax you were proposing was straightforward - allow “for any” or “for each” in place of “for”. But you haven’t provided any explanation of the semantics. You’ve not said what the new constructs would do. Without that, your proposal has no value. And there’s nothing substantive to respond to.

And even if you do provide semantics, you’ve still only barely scratched the surface. Just having syntax and semantics doesn’t mean a proposal is viable. For example, I could propose that we allow a new statement variable++ which is semantically equivalent to variable += 1. That’s far more clearly defined than any of your proposals have been so far, and yet it’s not going to happen (read the archives for why, it’s not relevant here and now).

I’m sorry, but I genuinely don’t know how to make it any clearer than this. If you still don’t understand what’s wrong with the way you’re engaging with this forum, I don’t think I can help you. I’ll still read your proposals, and maybe even respond to them (although I suspect I’ll just be saying “you haven’t explained your proposal well enough” again ), but I’ve got nothing more I can offer on how to improve the feedback you get…

---

1. certainly I am ↩︎

[FelixFourcolor]

@pf_moore

But you haven’t provided any explanation of the semantics. You’ve not said what the new constructs would do.

To be fair, they did

@Nineteendo

result = (value < 0 for any value in range(stop))

Would be roughly equivalent to:

def func(stop):
    for value in range(stop):
        if value < 0:
            return True
    return False

result = func(stop)