Are 2 sequencial for loops faster itertools.zip_longest?

Python Help
1

I’m trying to benchmark itertools.zip_longest against 2 sequencial for loops. All the time, itertools.zip_longest seems to be the least effective for performance. But that does not sound right. What am I missing?

I’m checking it with 2 ranges of different size.

def test_without_zip_longest():
    for ele in range(1000000):
        pass
    for ele in range(100000):
        pass

test_without_zip_longest() takes around 0.01922622078504719 on average for 1000 repetitions.

from itertools import zip_longest

def test_with_zip_longest():
    for ele_1, ele_2 in zip_longest(range(1000000), range(100000)):
        pass

test_with_zip_longest() takes around 0.03109439081502569 on average for 1000 repetitions.

2

I think this is an expected result. All your code does is setting the loop variable. Without zip_longest this set a single variable for the total length of both ranges. With zip_longest, a sequence has to be unpacked into the two loop variables ele_1 and ele_2 for the longest range. Using zip_longest only makes sense if both variables have to be treated at the same time.

3

If you don’t actually need to pair up elements and fill in dummy values for the shorter iterable, then of course itertools.izip_longest will add overhead - because it does those things, and then the loop over that iterable has to unpack the pairs.

But if you do need those things, itertools.izip_longest will be much faster - and easier - than doing it yourself.

from itertools import repeat, zip_longest

def test_separate_iterations():
    for ele in range(1000000):
        pass
    for ele in range(100000):
        pass

def test_with_zip_longest():
    for ele_1, ele_2 in zip_longest(range(1000000), range(100000)):
        pass

def test_manual():
    # first, the matching elements up to the shorter length
    for ele_1, ele_2 in zip(range(100000), range(100000)):
        pass
    # then the extra elements paired with None
    for ele_1, ele_2 in zip(range(100000, 1000000), repeat(None, 900000)):
        pass

And if we “manually” do the work of the built-in zip, or to simulate itertools.repeat, it will get even worse:

def simulate_zip_longest(i1, i2):
    i1, i2 = iter(i1), iter(i2)
    while True:
        try:
            e1 = next(i1)
        except StopIteration:
            # ran out of elements from i1; output from i2 and exit
            for e in i2:
                yield (None, e)
            return
        try:
            e2 = next(i2)
        except StopIteration:
            # ran out of elements from i2; output from i1 and exit
            for e in i1:
                yield (e, None)
            return
        # Otherwise we have an element from both iterators still
        yield (e1, e2)

def test_with_simulated_zip_longest():
    for ele_1, ele_2 in simulate_zip_longest(range(1000000), range(100000)):
        pass

My results look like:

>>> timeit(test_separate_iterations, number=100)
1.9616155847907066
>>> timeit(test_with_zip_longest, number=100)
3.220167404972017
>>> timeit(test_manual, number=100)
3.2905724085867405
>>> timeit(test_with_simulated_zip_longest, number=100)
7.614517252892256