Is there a way to create a generator function with internal paralellism?

If I have this kind of generator:

def special_gen(d):
    for n in count(1):
        if is_special(d, n):
            yield n

The call to is_special() takes time and I need to generate the values in increasing order.

I thought that a part of the solution would involve creating multiple generators and combining their results. Something starting like this (not in parallel):


def special_gen_mod(d, mod=1, offset=0):
    for n in count(1 + offset, mod):
        if is_special(d, n):
            yield n

def special_gen(d):
    sub_generators = [special_gen_mod(d, PROCESSES, off)
                      for off in range(PROCESSES)]
    yield from heapq.merge(*sub_generators)

But I am stuck on how to add the parallelism. I’ve read and reread the docs and googled but couldn’t come up with a neat way of creating a “parallelised generator”.

My question is also on Stack Overflow, but I posted here in the hope that someone already had ideas on
“parallelised generators”.

Thanks in advance :slight_smile: