There’s a suggestion, timeout parameter to the matching functions in the re module, as a mitigation for catastrophic backtracking (ReDoS).
The Problem
time. When the pattern or the input comes from outside the program, user-submitted search strings, scraped HTML, log lines, config values, a single pathological case can pin a CPU core for minutes or hours. And since the match runs in C while holding the GIL, every other thread in the process stalls with it: health checks stop responding, and the worker usually has to be killed externally.
This isn’t hypothetical. I work at Arbisoft, a software services company, and across several Python services we maintain that process user-supplied text, we’ve had production incidents where one crafted input froze an entire worker. The current workarounds are all unsatisfying in their own way:
-
“Just write safe patterns” works until a pattern is user-supplied, generated, or inherited from a dependency. Auditing every regex in a large codebase plus its dependencies is not realistic.
-
Running matches in a subprocess with a kill timer is the only reliable containment today, but it adds IPC overhead and operational complexity that feels disproportionate for calling a stdlib function.
-
signal.alarm is Unix-only, main-thread-only, and fragile.
-
Switching to the third-party regex module, which already supports a timeout argument, works, but many teams can’t add a C-extension dependency just to make re.search interruptible.
The Proposed Solution
Add an optional, keyword-only timeout parameter (seconds, float) to the module-level functions and Pattern methods that execute a match (match, search, fullmatch, findall, finditer, sub, subn, split):
import re
try:
m = re.search(user_pattern, user_text, timeout=0.5)
except re.TimeoutError:
log.warning("pattern exceeded time budget, rejecting")
-
Default is None, meaning current behavior, zero overhead, full backward compatibility.
-
On expiry, raise a new re.TimeoutError (subclassing the builtin TimeoutError) so callers can distinguish it from a non-match.
-
Implementation-wise, the engine already counts steps internally; the check could piggyback on existing periodic bookkeeping (e.g., test a deadline every N opcodes) so the no-timeout path pays nothing and the timeout path pays a near-constant cost.
Prior art
-
The third-party regex module has had a timeout parameter for years.
-
.NET’s Regex accepts a matchTimeout and throws RegexMatchTimeoutException; Microsoft added it specifically as a ReDoS mitigation.
-
Java doesn’t have one, and “how do I time out a Pattern match” is a perennial question there, which I’d argue is evidence for building it in rather than against.
Anticipated objections
“The real fix is a linear-time engine (RE2-style) or memoization.” Agreed that would be strictly better, but it’s a far larger project with API-compatibility questions (backreferences, lookarounds), and previous attempts to rewrite the engine haven’t landed. A timeout is a small, contained mitigation that doesn’t preclude that work later.
“Per-call deadline checks will slow down matching.” The check only needs to run when timeout is passed, and only every N steps. Happy to help benchmark this if there’s interest.
“Use the regex module.” That’s a fine answer for some projects, but re is the tool the vast majority of code reaches for, including stdlib modules and frameworks that accept user patterns. A safety valve in the default tool benefits the whole ecosystem.
I’d appreciate feedback on whether this seems worth pursuing, and especially on whether the deadline-check approach is feasible in sre without measurable cost to the default path.
Thanks for reading.