# String concatenation with a for loop

**URL:** <https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480>\
**Category:** Python Help\
**Created:** [November 26, 2024, 6:45pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480 "2024-11-26T18:45:57Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![StrongAlly](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@StrongAlly](https://discuss.python.org/u/StrongAlly)\
**Post date:** [November 26, 2024, 6:45pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/1 "2024-11-26T18:45:57Z")

</div>

Hello everyone,

When I run this program on Python 3.12.6, debian trixie:

```python
# Part 1
def test():
    s = 'a'*1_000_000
    t = ''
    for c in s:
        print(len(t), id(t))
        t += c

test()

##############################################
# Part 2

s = 'a'*1_000_000
t = ''
for c in s:
    print(len(t), id(t))
    t += c

```

part 1 is much faster than part 2. It seems that in part 2, Python copies the string at each step.  
What is the reason for this phenomenon?

Thanks !

---

<div class="post-metadata">

**Author:** ![barry-scott](https://avatars.discourse-cdn.com/v4/letter/b/e9c0ed/32.png) [@barry-scott](https://discuss.python.org/u/barry-scott)\
**Post date:** [November 26, 2024, 6:53pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/2 "2024-11-26T18:53:40Z")

</div>

My guess is that its the cost of access to local variables, part 1, compared to the cost of access to global variables in part 2.

In both cases the string manipulation of `t += c` is the same.  
Note that each time you call `t += c` python actually does `t = t + c` that creates a new string to replace `t` every time around the loop.

---

<div class="post-metadata">

**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\
**Post date:** [November 26, 2024, 7:07pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/3 "2024-11-26T19:07:15Z")

</div>

CPython actually does specific optimizations for string concatenation that means it tries to reuse the original object and avoid copying. I suspect this doesn’t always work, but even if it only works sometimes it can be a huge difference.

This can only work if the reference count of the object is exactly 1. I suspect that with the changed behavior of global variable accesses compared function locals this optimization is also prevented, in addition to what @barry-scott already mentioned of global variables being slower to access than function locals.

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [November 26, 2024, 7:26pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/4 "2024-11-26T19:26:55Z")

</div>

Please don’t just say “much faster” but say **how much** faster. And don’t `print` in there.

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [November 26, 2024, 7:29pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/5 "2024-11-26T19:29:04Z")

</div>

Explanation:

> <https://github.com/python/cpython/issues/99862>
>
> Hello,
> 
> We have found a regression between CPython 3.10.8 and CPython 3.11 res…ulting in string concatenation to be significantly slower in loops on Windows 10. This is described in details in this \[StackOverflow post\](https://stackoverflow.com/questions/74605279/python-3-11-worse-optimized-than-3-10/74607850).
> 
> Here is a minimal, reproducible example of benchmarking code:
> 
> \`\`\`python
> import time
> a = 'a'
> 
> start = time.time()
> for \_ in range(1000000):
> a += 'a'
> end = time.time()
> 
> print(a\[:5\], (end-start) \* 1000)
> \`\`\`
> 
> CPython 3.11.0 is about 100 times slower than CPython 3.10.8 due to a quadratic running time (as opposed to a linear running time for CPython 3.10.8).
> 
> The analysis shows that CPython 3.10.8 was generating an INPLACE\_ADD instruction so \`PyUnicode\_Append\` is called at runtime, while CPython 3.11.0 new generates a BINARY\_OP instruction so \`PyUnicode\_Concat\` is actually called. The later function creates a new bigger string reducing drastically the performance of the string appending loop in the provided code. This appears to be related to the issue #89799 . I think if we want to replace INPLACE\_ADD with a BINARY\_OP, then an optimization checking the number of references (so to eventually do an in-place operation) is missing in the code of CPython 3.11.0. What do you think about it?
> 
> My environment is an embedded CPython 3.10.8 and an embedded CPython 3.11.0, both running on Windows 10 (22H2) with a x86-64 processor (i5-9600KF).

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 26, 2024, 7:41pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/6 "2024-11-26T19:41:13Z")

</div>

It is not related to strings, but rather to the speed of accessing a local variable versus a global variable.

```python
import time

def bench():
    a = 0
    for i in range(1000000):
        a += i

    return a

start = time.time()
result = bench()

print(time.time() - start)
print(result)

start = time.time()
a = 0
for i in range(1000000):
    a += i

print(time.time() - start)
print(result)

```

```python
0.06985592842102051
499999500000
0.1436767578125
499999500000

```

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [November 26, 2024, 8:35pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/7 "2024-11-26T20:35:37Z")

</div>

@elis.byberi Did did you have a look at the GitHub issue with the explanation? And why did you measure ints instead of strings? That doesn’t show the huge difference.

---

<div class="post-metadata">

**Author:** ![StrongAlly](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@StrongAlly](https://discuss.python.org/u/StrongAlly)\
**Post date:** [November 26, 2024, 9:00pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/8 "2024-11-26T21:00:00Z")

</div>

0.03 seconds for part 1 versus 15 seconds for part 2.  
The prints highlight the fact that the string `t` is copied at each step in part 2, while this is not the case in part 1.

The link given by Stephan answers my question, thanks !

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 26, 2024, 9:17pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/10 "2024-11-26T21:17:14Z")

</div>

String: **global** vs **local**

```python
import time

def bench_local():
    s = 'a' * 100_000
    t = ''
    for c in s:
        t += c

    return len(t)

start = time.time()
result = bench_local()

print(time.time() - start)
print(result)

s = 'a' * 100_000
t = ''

def bench_global():
    global s
    global t
    for c in s:
        t += c

    return len(t)

start = time.time()
result = bench_global()

print(time.time() - start)
print(result)

```

```python
Python 3.10.12:
0.009384393692016602
100000
0.36257052421569824
100000

Python 3.12.7:
0.009612798690795898
100000
0.3437345027923584
100000

```

Local: **string** vs **integer**

```python
import time

def bench_string():
    s = 'a' * 1000000
    t = ''
    for c in s:
        t += c

    return len(t)

start = time.time()
result = bench_string()

print(time.time() - start)
print(result)

def bench_integer():
    a = 0
    for i in range(1000000):
        a += i

    return a

start = time.time()
result = bench_integer()

print(time.time() - start)
print(result)

```

```python
Python 3.10.12:
0.08484053611755371
1000000
0.06541633605957031
499999500000

Python 3.12.7:
0.06548666954040527
1000000
0.05298948287963867
499999500000

```

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [November 26, 2024, 11:52pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/11 "2024-11-26T23:52:18Z")

</div>

> [@StrongAlly](#):
>
> The prints highlight the fact that the string `t` is copied

Ah right, for that it’s good. Just not for timing.

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 27, 2024, 12:06am UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/12 "2024-11-27T00:06:09Z")

</div>

> [@StrongAlly](#):
>
> Python copies the string at each step.

That doesn’t seem to cause a significant slowdown. In Python 3.10, there are no copies, but the slowdown remains the [same](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/10):

```python
# Part 1
def test():
    s = 'a' * 1_000_000
    t = 'b' * 1_000_000
    for c in s[:10]:
        print(len(t), id(t))
        t += c

test()

##############################################
# Part 2

s = 'a' * 1_000_000
t = 'b' * 1_000_000
for c in s[:10]:
    print(len(t), id(t))
    t += c

```

```python
Python 3.10.12:
1000000 126645204770832
1000001 126645204770832
1000002 126645204770832
1000003 126645204770832
1000004 126645204770832
1000005 126645204770832
1000006 126645204770832
1000007 126645204770832
1000008 126645204770832
1000009 126645204770832
1000000 98116411510800
1000001 98116411510800
1000002 98116411510800
1000003 98116411510800
1000004 98116411510800
1000005 98116411510800
1000006 98116411510800
1000007 98116411510800
1000008 98116411510800
1000009 98116411510800

Python 3.12.7:
1000000 129595043692560
1000001 129595042689040
1000002 129595042689040
1000003 129595042689040
1000004 129595042689040
1000005 129595042689040
1000006 129595042689040
1000007 129595042689040
1000008 129595042689040
1000009 129595042689040
1000000 37507376
1000001 38507440
1000002 37507376
1000003 38507440
1000004 37507376
1000005 38507440
1000006 37507376
1000007 38507440
1000008 37507376
1000009 38507440

```

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [November 27, 2024, 12:29am UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/13 "2024-11-27T00:29:16Z")

</div>

@elis.byberi In your benchmark, you used a function with `global`. In your `id` logger, you didn’t.

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 27, 2024, 12:36am UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/14 "2024-11-27T00:36:02Z")

</div>

> [@Stefan2](#):
>
> you used a function with `global`

I don’t quite understand what you mean. Could you elaborate?

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [November 27, 2024, 12:43am UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/15 "2024-11-27T00:43:26Z")

</div>

This:

```python
def bench_global():
    global s
    global t
    for c in s:
        t += c

```

Vs this:

```python
s = 'a' * 1_000_000
t = 'b' * 1_000_000
for c in s[:10]:
    print(len(t), id(t))
    t += c

```

Try the latter with `global t` in a function like in the benchmark. What output do you get?

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 27, 2024, 12:58am UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/16 "2024-11-27T00:58:21Z")

</div>

It shows a similar output for both Python 3.10 and Python 3.12, but does not affect timing.

---

<div class="post-metadata">

**Author:** ![chepner](https://avatars.discourse-cdn.com/v4/letter/c/22d042/32.png) [@chepner](https://discuss.python.org/u/chepner)\
**Post date:** [November 27, 2024, 12:57pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/17 "2024-11-27T12:57:46Z")

</div>

> [@StrongAlly](#):
>
> What is the reason for this phenomenon?

The reason is implementation-specific, and of primarily academic interest. Python provides no guarantee that something like

```python
for c in s:
    print(len(t), id(t))
    t += c

```

won’t take quadratic time. It _does_ guarantee that

```python
for c in s:
    print(len(t), id(t))
t = ''.join(s)

```

will take linear time, so avoid repeated concatenation.

---

<div class="post-metadata">

**Author:** ![da-dada](https://avatars.discourse-cdn.com/v4/letter/d/8491ac/32.png) [@da-dada](https://discuss.python.org/u/da-dada)\
**Post date:** [November 27, 2024, 8:57pm UTC](https://discuss.python.org/t/string-concatenation-with-a-for-loop/72480/18 "2024-11-27T20:57:27Z")

</div>

> [@elis.byberi](#):
>
> speed of accessing a local variable versus a global variable.

better locals versus non locals

```python
import time
s = 'a'*1_000_000

def testo():
    t = ''
    def testi():
        start = time.time()
        nonlocal t
        for c in s:
            t += c
        print(str(time.time() - start))
    testi()

testo()

```

I think 🫠

 ![image](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/9/b/9b08a75670c173d68044e3fbcba03cfad2b61fc6.png)