Replacing one string inside another but without interfering with a second replace

I need to change "{{" & "}}" to "{" & "}", while at the same time also replacing "{" &"}" with "{{" & "}}".

Whichever way round I do this, one replacement will always interfere with the other. At the moment my best solution is to use replace with intermediate delimiters. However it feels like there should be a better/quicker solution.

I looked at str.translate(), but that is for single character replacements only. Any suggestions of an alternative way of doing this?

Also whatever I use as intermediate delimiters could potentially interfere with the incoming HTML if they happen to be contained within that (unless anyone can suggest any delimiters which could be used in Python but would be illegal in HTML/JavaScript).

Also tried a couple of different regex methods but they are both four times slower than .replace(). See my test code below:

import re
import datetime

x = "here is {javascript}, but here is a pythonExpression {{var1}}"
print("1 begin:", x)
start = datetime.datetime.now()
for _ in range(1000001):
    x = x.replace("{{", "<<").replace("}}", ">>")
    x = x.replace("{", "{{").replace("}", "}}")
    x = x.replace("<<", "{").replace(">>", "}")
finish = datetime.datetime.now()
print("1 end:", x) # -> here is {{javascript}}, but here is a pythonExpression {var1}, hmm how to deal with that.
print("Runtime (1000001): ", finish-start) # <- 3.7s

x = "here is {javascript}, but here is a pythonExpression {{var1}}"
print("2 begin:", x)
d = {"{{": "{", "}}": "}", "{": "{{", "}": "}}"}
start = datetime.datetime.now()
for _ in range(1000001):
    x = re.sub("|".join(d), lambda x: d[x.group(0)], x)
finish = datetime.datetime.now()
print("2 end:", x) # -> here is {{javascript}}, but here is a pythonExpression {var1}, hmm how to deal with that.
print("Runtime (1000001): ", finish-start) # <- 13s

def replacer(val):
    match val.group(0):
        case "{{":
            return "{"
        case "}}":
            return "}"
        case "{":
            return "{{"
        case "}":
            return "}}"
    return ""

x = "here is {javascript}, but here is a pythonExpression {{var1}}"
print("3 begin:", x)
# x = re.sub("'{{'  '}}' | '{' | '}'", myfunc(), x)
start = datetime.datetime.now()
for _ in range(1000001):
    x = re.sub("({{|}}|{|})", replacer, x, flags=re.MULTILINE)
finish = datetime.datetime.now()
print("3 end:", x) # -> here is {{javascript}}, but here is a pythonExpression {var1}, hmm how to deal with that.
print("Runtime (1000001): ", finish-start) # <- 12.5s

What was wrong with the suggestions at https://stackoverflow.com/q/73948326?

OP has already said the regex approach is slow, and the other approach using count occurrences requires the string to be in a specific form.

I doubt that HTML will contain U+FFFF (“\uFFFF”).

Is that speed difference actually relevant, though? “It’s slower” is not the universal knock-down unless it actually makes a real difference. IMO a regex is the right tool for this job.

That’s because that isn’t a valid character. So, yes, you should be safe-ish using that as a delimiter. However, there’s basically no way to 100% guarantee that something won’t show up. A one-pass replacement is safer.

Have you tried moving "|".join(d) outside the loop or compiling the regex ahead of time (re.compile)?

The intermediate token can be of any length, so you could increase the probability it won’t be present in the input by adding characters. But I’m not quite sure I understand the objective. If the input text is some HTML/JS block, can’t {/{{ appear in other contexts that you wouldn’t want to replace, such as inside a string? This reminds me of Jinja templates which I imagine is implemented as a custom parser.

This seems ~30% faster:

x = x.replace("{", "{{").replace("{{{", "").replace("}", "}}").replace("}}}", "")

Another regex solution, short but even slower:

x = re.sub(r'({|})\1|({|})', r'\1\2\2', x)

Also, if you care about performance, you should definitely be compiling your regex.

And also create the replacer function only once. Like:

d = {"{{": "{", "}}": "}", "{": "{{", "}": "}}"}
sub = re.compile("|".join(d)).sub
rep = lambda m: d[m[0]]
start = datetime.datetime.now()
for _ in range(1000001):
    x = sub(rep, x)
finish = datetime.datetime.now()

Or optimize rep further (at least for me in 3.13, this makes the benchmark ~10% faster):

def rep(d):
    m = yield
    while True:
        m = yield d[m[0]]
rep = rep(d)
next(rep)
rep = rep.send

Less relevant than you might think, since method lookups have some optimization.

rosuav@sikorsky:~$ python3 -m timeit -s 'import re; num = re.compile("[0-9]+").search' 'num("spam spam spam spam 42000 spam")'
2000000 loops, best of 5: 184 nsec per loop
rosuav@sikorsky:~$ python3 -m timeit -s 'import re; num = re.compile("[0-9]+")' 'num.search("spam spam spam spam 42000 spam")'
2000000 loops, best of 5: 186 nsec per loop

But ALL of this is completely irrelevant unless the OP is actually having real issues with performance. Just because a microbenchmark shows that one number’s bigger than another, that doesn’t mean that you have to obey it. We don’t HAVE to worship the little tin god of efficiency.

You misunderstood. I was talking about rep, not sub.

(The .sub was just an additional optimization I didn’t point out, just like using m[0] instead of m.group(0).)

You mentioned both.

Here’s a hardcore loop method I wrote

The replace method is still faster. I guess it’s because it’s implemented in C.


If the temporary strings (<<, >>) being in the string is an issue, you can do something lie this:

import random
while (sentinel := str(random.randrange(10000))) in x:
    pass

Also, every iteration in the loops of the first posts code is doing work ({{{ back and forth) (just saying this gotcha because I misunderstood this)

That’s bad. Demo:

x = '{{123456}}'

sentinel = '666'  # this could happen
print(sentinel in x)

x = x.replace("}}", sentinel)
x = x.replace("}", "}}")
x = x.replace(sentinel, "}")

print(x)

Output is {{12345}6 instead of {{123456}. Attempt This Online!

Nice catch! :sweat_smile:

I think there’s a more solid method: Use a single Unicode character that’s not in the original text. Provided such character exist, no prefix/suffix amalgamating should be able to happen.

You could use a code point from the unicode private use area.

From Private Use Areas - Wikipedia

Three Private Use Areas are defined: one in the Basic Multilingual Plane (U+E000–U+F8FF), and one each in, and nearly covering, planes 15 and 16 (U+F0000–U+FFFFD, U+100000–U+10FFFD). They are intentionally left undefined so that third parties may assign their own characters without conflicting with Unicode Standard assignments.

Post was closed! There seems to be an awful lot of destructive interference from a few individuals over there which prevents constructive discussion. Rather a shame as in the past I have found Stack Overflow to be a good forum but seemingly no more!

As I said in my post and gave examples the Regex methods, I tried two, are both too slow.