If I have this kind of generator:
def special_gen(d):
for n in count(1):
if is_special(d, n):
yield n
The call to is_special() takes time and I need to generate the values in increasing order.
I thought that a part of the solution would involve creating multiple generators and combining their results. Something starting like this (not in parallel):
PROCESSES = 4
def special_gen_mod(d, mod=1, offset=0):
for n in count(1 + offset, mod):
if is_special(d, n):
yield n
def special_gen(d):
sub_generators = [special_gen_mod(d, PROCESSES, off)
for off in range(PROCESSES)]
yield from heapq.merge(*sub_generators)
But I am stuck on how to add the parallelism. I’ve read and reread the docs and googled but couldn’t come up with a neat way of creating a “parallelised generator”.
My question is also on Stack Overflow, but I posted here in the hope that someone already had ideas on
“parallelised generators”.
Thanks in advance