Joining strings is faster than adding but only if concatenating 3+strings

I was working on a project that would require some very large string concatenation. Because of this, I wanted to make sure that I was using the most efficient way to join the string. I narrowed it down to either summing the strings up (i.e. str1+str2) or using the join method (i.e. "".join([str1, str2])).
However, I noticed something strange when testing these two methods. Simply summing the strings is faster but only if summing just two strings if I want to concatenate three strings, it is faster to use the join method.

Can anyone explain why this is?

For reference, I am running the 64 bit version of Python 3.8.5 (downloaded from python.org) on a Windows 10 Computer with 16GB of RAM

As an example, here is a simple script that (on my computer at least) demonstrates this phenomena:


def add2fcn():
    x='a'
    new = 'b'
    for _ in range(100000):
        x = x + new

def join2fcn(): 
    x='a'
    new = 'b'
    for _ in range(100000):
        x = "".join([x, new])

def add3fcn(): 
    x='a'
    sep = '\\n'
    new = 'b'
    for _ in range(100000):
        x = x + sep + new

def join3fcn(): 
    x='a'
    sep = '\\n'
    new = 'b'
    for _ in range(100000):
        x = "".join([x, sep,new])

import timeit
import re
number = 100


add2 = timeit.timeit(add2fcn, number=number)
print(f"Adding two strings takes {add2 / number} seconds")

join2 = timeit.timeit(join2fcn, number=number)
print(f"Joining two strings takes {join2 / number} seconds")

print("\n")
add3 = timeit.timeit(add3fcn, number=number)
print(f"Adding three strings takes {add3 / number} seconds")

join3 = timeit.timeit(join3fcn, number=number)
print(f"Joining three strings takes {join3 / number} seconds")

This results in the following output:

Adding two strings takes 0.020516059000000003 seconds
Joining two strings takes 0.210043305 seconds


Adding three strings takes 0.7843044419999999 seconds
Joining three strings takes 0.665828113 seconds

You are timing a lot of extraneous code unrelated to string
concatenation: calling a function, defining named variables, building a
range object, iterating over the range object, plus the concatenation
you are interested in. The result is that there will be even more
opportunity for adding random noise to your measurements and your
timings don’t necessarily demonstrate what you think they demonstrate.

If you are purely interested in the speed of string concatenation, it is
best to try to keep all those other factors to the bare minimum, when
possible.

Timings are also heavily influenced by everything else happening on the
computer at the time, so it is best to run the operation many times and
average it, and also to run multiple trials and use the lowest result.
By default, that’s what timeit will do.

So here are some tests more narrowly focused on just the concatenation
alone. You run them from the OS’s command line (the prompt is usually a
dollar sign $ or percent sign % not the Python prompt).

# test join with two strings only
python3 -m timeit -s "L = ['abc', '123']" "''.join(L)"

# test string concatenation with two strings only
python3 -m timeit -s "a, b = ['abc', '123']" "a + b"

By default, timeit in Python 3.7 will work out the best number of loops
to run, and then do five trials, returning the best result of those five
trials. The setup code (the -s option) doesn’t get included in the
timing.

On my machine I get:

5000000 loops, best of 5: 76.9 nsec per loop

5000000 loops, best of 5: 45.1 nsec per loop

which is more or less the result I would expect. With just two strings,
the overhead of calling a method will be more than the overhead of the
plus operator.

If I do a similar exercise with three strings, I get:

python3 -m timeit -s "L = ['abc', '123', 'xyz']" "''.join(L)"
5000000 loops, best of 5: 83.8 nsec per loop

python3 -m timeit -s "a, b, c = ['abc', '123', 'xyz']" "a+b+c"
5000000 loops, best of 5: 81.4 nsec per loop

which again, is more or less what I would expect. With three substrings,
the overhead of calling join is pretty much the same as for two, so
there’s hardly any difference in time. But for the plus operator, it has
to be called twice, so you have twice as much overhead.

With four substrings, I would expect join to be faster, and sure enough:

python3 -m timeit -s "L = ['abc', '123', 'xyz', 'abc']" "''.join(L)"
5000000 loops, best of 5: 91.6 nsec per loop

python3 -m timeit -s "a,b,c,d = ['abc', '123', 'xyz', 'abc']" "a+b+c+d"
2000000 loops, best of 5: 129 nsec per loop

Bottom line: for a single concatenation, the plus operator is the
simplest thing that works, with the least overhead, and there’s no
reason to use anything else:

plural = word + 's'

is both more understandable and likely to be faster than any
alternative such as:

plural = ''.join([word, 's'])

but as you get more substrings, it becomes more efficient to accumulate
them in a list and join them all at once.