itertools.count is one of the few atomic constructs available (at least to the extent that the GIL’s protection can be considered atomic), which makes it useful as a shared sequence or counter, however this is hampered by the inability for an observer to access the value without incrementing, at least without tricks: it’s possible to retrieve the current value by either parsing the repr or using the pickling protocol:
A property to access the current value would make for much simpler e.g. task or work counting, without needing to adjust for retrieval updates or the like.
A straight addition would have to be protected by a mutex as it’s not atomic, a += ? compiling to (more or less)
LOAD a
LOAD ?
INPLACE_ADD
STORE a
with 3 suspention points between the loading and storage of a.
Possible alternative
Add basic atomic constructs (CAS, possibly load and store, maybe even operators like inc/dec/add/sub) to cpython, however it’s not entirely clear to me how that would work without language support, since the atomic constructs would need access to the storage location itself in order to atomically load/store while holding the GIL (or a finer lock if the GIL ends up getting removed).
Why not mutex
Mutexes are a lot more expensive in all dimensions: they’re pretty costly, they require more code, and they have a higher semantic overhead as the lock and what it protects are not technically directly linked. Though possibly an other alternative would be to make locks into (optional) containers à la Rust, and maybe add convenience / utility pseudo-atomics.
Nice recipe, personally very happy with it because it ended up faster than my version. However, logic is a bit involved, a bit too involved for what it does I would say.
Can I suggest that discussions about squeezing microsecond-level performance improvements out of Python code probably belong in the help topic? That sort of improvement is either something that can simply be submitted as a PR (if there are no backward compatibility issues) or something that will never happen (because that sort of benefit is way below the level to justify a language change). Either way, it’s not a feasible language change.
The ideas topic is specifically for language changes and this sort of micro-optimisation isn’t really a language design level discussion.
I love optimization. All kinds of it: macro, micro, downhill, cuckoos nest. One of my favourite parts. So you might see an increase in my comments about it when I am on “optimization break”. If it bothers many people I can try to refrain myself form commenting about it.
And I think optimization for python is very important. At least for the the level of applicability, functionality and modularity that I think software written in python could achieve.
Having that said, I appreciate that certain changes (such as JIT) might have a big impact on the status quo and some micro optimizations that I do might not be worth an effort.
Is there a way to dynamically replace __next__ method?
I.e. iterator.__next__ = other_iterator.__next__
%timeit bti.consume(more_itertools.countable(a)) # 12.6 ms
# What constitutes this time?
1.4 ms - consume
2.2 ms - counting integers
7 ms - __next__ method
2 ms - actual iteration
from itertools import count, compress
from operator import sub
def countable(iterable):
counter = count(1)
it = compress(iterable, counter)
cnt = map(sub, counter, count(1))
class IterCounter:
__iter__ = it.__iter__
__next__ = it.__next__
@property
def items_seen(self):
for c in cnt:
return c
return IterCounter()
