In [6]: import re
In [7]: pattern = re.compile('')
In [8]: bool(pattern.search('1234'))
Out[8]: True
and I do not believe it should be. In my case, by accident an empty string made it to the compilation of the regex and the underlying code let all the inputs through instead of filtering them.
Mathematically speaking, an empty set is a subset of any set. But from a pragmatic point of view, this is dangerous. I would raise an exception when the input argument is an empty string. If the user wants to match anything, they can just use “.*”, which is very unlikely to be passed by mistake.
Strongly -1. Adding a check for this would deviate from standard regex behavior, as well as imposing a runtime penalty. This is the developer’s problem to avoid, not the language’s.
Actually, slightly weaker -1. It would deviate from behavior in non-regex contexts, like containment tests. It’s not that unstandard among regex flavors.
How is this dangerous? An empty string IS contained within any other string, whether you’re looking at simple "" in "1234" or using a regex. If this had unexpectedly abysmal performance, I could see that as being dangerous, but it simply matches at the start of the string and stops.
Not just Python - all regexp packages work this way, and none of them will change .
>>> import re
>>> m = re.search('', 'abc')
>>> m
<re.Match object; span=(0, 0), match=''>
>>> m.span()
(0, 0)
As it says, the pattern (which happens to be an empty string) matches the slice (which is also empty) 'abc'[0:0]
It may help to understand that regexps weren’t designed for human usability. They started as a mathematical notation in format language theory. Earlier languages like SNOBOL (and its successor Icon) were designed for humans to use, but only regexps caught on widely.
In language theory, an empty string is usually denoted by the Greek letter epsilon (\epsilon), but that convention wasn’t adopted by computer languages - lacking on most keyboards.
This isn’t unique to entire patterns either - an empty string can be used in any regexp context that wants a string. For example, the existential quantifier ? (e? matches 0 or 1 occurrences of e) isn’t usually part of the underlying theory either, but is instead a shortcut for saying
(e|\epsilon)
which is actually spelled
(e|)
in computer languages. That’s not a syntax error. It’s saying “match e or an empty string”. They have no explicit way to “empry string”, barring convolutions like a{0} Instead an empty string is implied by not seeing anything where you expect to see a string. No, that’s no a stellar design
>>> m = re.match('(e|)d', 'd')
>>> m
<re.Match object; span=(0, 1), match='d'>
>>> m.span(1) # the capturing group at the start matched an empty string
(0, 0)
Regexps are full of surprises, alas. Good advice: use them sparingly .
It is extremely dangerous, an empty string can get into your code in a million ways. For instance you could have an environment variable that was not set, by accident, made it into the regex expression and allowed something to happen that should not happen. If the regex is .*, it can only be like that because someone decided to write .*
There is no future for a language that doesn’t play along with universally implemented behviors, and this is no exception.
You can claim it’s “dangerous”, but my experience (spanning over 5 decades) says otherwise: I’ve never passed an empty string by mistake as a regexp, and never heard of that happening to anyone else either. You may as well object that i+j is “dangerous” because j may be 0, which probably isn’t what the programmer intended. One difference: that one actually happens to people in real life
Serious regexp users typically precompile them at module level, and never employ user-supplied input “as a regexp”, and an empty string is extremely visible in that context. It just wouldn’t escape notice, not even by the most casual review. Not to mention that a regexp that matches everything couldn’t pass the simplest of testing.
What is the danger though? What could go wrong? It matches. When you pass an empty search term, it always matches. This is, in fact, exactly what most people expect. There’s no unexpected memory or CPU load caused by this, there’s no sudden crash or exception (which you would actually be worsening), there’s no way for a remote attacker to run code in your process… what’s so dangerous about this?
.* is telling you, do something for every element. What is every element? How often do you do something for every element? Delete every element? Copy every element? Equating .* to what an empty string makes no sense. Maybe that makes sense from a math point of view. But not for me. If I want a program to do something for every element, I need to make it hard, I need a couple of very specific characters that cannot possibly be passed by mistake. Does the design of regular expressions not follow that? Then regular expressions are wrong. If python or other language will accept it it’s a different matter.
You’re right that it makes no sense; an empty string matches an empty string, but .* matches everything. [1] So they are absolutely NOT equivalent, and I don’t understand your point. It still doesn’t answer the question about how this is “dangerous”.
Maybe we’re talking past each other and you’re actually asking for something completely different, but if you want to match an empty string, .* is not the regular expression you want. (It will happen to match an empty string, but it certainly isn’t the best way to express that.)
modulo some rules like dot not matching newline, but near enough ↩︎
Not really. If I’m passing a regex pattern to a program as a CLI argument I may pass '^' or just '' to trivially match without capturing anything. If you’re remotely receiving strings from an untrusted source, which you want to use as regex patterns, then perform input validation / sanitizing properly eg with a library.
.* is telling you, do something for every element. What is every element? How often do you do something for every element? Delete every element? Copy every element? Equating .* to what an empty string makes no sense.
Here’s a straightforward demonstration of how they’re not being equated by re:
>>> import re
>>> empty = re.compile("")
>>> full = re.compile(".*")
>>> empty.search("foo").group(0)
''
>>> full.search("foo").group(0)
'foo'
So as previously stated, the empty pattern matches an empty subset of the start of the string while a wildcard matches elements within the string.
Then you should check the env variable is set to some expected value. If you’re that worried about "” regexes, then you could add a guard like
def assert_not_empty(val: str) -> str:
if val == "":
raise ValueError("String is empty")
return val
re.compile(assert_not_empty(os.environ["FOO"]))
For Literal empty strings, then you could review and test your code. If you don’t really want to do that, then you could petition a linter to add a check that you aren’t doing re.compile(““). You’ll probably have better buy in/a quicker turnaround if ruff or flake8 makes this an optional rule.
This is actually very worrying. The fact that you have never heard of anyone making this mistake might actually mean that there are a lot of people making it but no one has ever noticed because the mistake is so counterintuitive than people do not even know they might be making it. I only found out about this because I was persistent enough to not just move on, but I wanted to actually know what was happening.
It is a HUGE problem when bad design gets introduced, because it will cause a lot of mistakes downstream. I agree with your point that this bad design has been around for so long that the ammount of code relying on it might make it imposible to change, we now have to live with it.
Another example of bad design are the muttable defaults. They are completely counterintuitive, no one needs the behavior they have and we only know about them because of the problems they cause.
I am not sure how you do not understand my point. An empty string and .* are not the same, and should not be the same. However they are treated as if they were the same in places like:
import re
pattern = re.compile('')
x = bool(pattern.match('1234'))
print(x) # This is True
How is 1234 matching an empty string intuitive? That makes no sense. I could get an empty strting in a million ways. For sure if I wanted to match anything It would make more sense to use .*. If I pass an empty string 80% it might just be a bug somewhere and I would need an exception to be raised.
Tell me, are "" and "1234" the same thing? No, of course they’re not. Yet they match:
>>> "1234".startswith("")
True
This is what you are doing with the regular expression. Perhaps you are not expecting this behaviour. Perhaps pattern.fullmatch() would suit you better. But I have no idea, since all you have done is complain about something, rather than explaining where the misunderstanding is actually taking place.
I don’t understand why you are expecting an empty string to not match, and at this stage, all I can do is guess.
It isn’t, but that’s not what you asked for,.You asked whether '1234'starts (match()) with an empty string. And it does:
>>> '1234'.index('')
0
Thoroughly consistent. If you want to know whether all of '1234' matches an empty string, use fullmatch() instead:
>>> import re
>>> print(re.fullmatch('', '1234'))
None
Of course it doesn’t.
I’m losing my presumption of good will There are many options because many use cases want different behaviors in different cases. The defaults picked overwhelmingly match industry-wide behaviors for regexps, except that Python seems unusual in using match() to mean “restrict matches to start at the beginning” (in many other implementations, match() is used to ask for what Python means by search()).. The defaults are most convenient for what seems to be the most common cases; do case-sensitive matching against single lines of text (so, e.g., a lone period does not match a newline character by default).
In fact, the only option I routinely use is re.VERBOSE.
The worst aspect of regexps is that they do backtracking, which is rarely what people actually want, and is the cause of the overwhelmingly most likely real-life problems: regexps that don’t match what you actually want them to match, and that are prone to talking exponential time to fail to match inputs you didn’t anticipate (“catastrophe[hic backtracking”).
More modern implementations (like Python’s) include more magical constructs, like “possessive quantifiers” and “atomic groups” that can be used to cut off possibilities for unwanted backtracking, but in all they make the already-cryptic regexp “language” even more cryptic.
The best use for regexps is to do simple lexing (“identify the next token”) in a loop (like the old Unix™ lex tool), and put higher-level parsing logic in code (like the old yacc tool). Even for all their arcane complications, regexps aren’t even adequate for “simple” parsing tasks like “match the longest substring balanced with respect to parentheses”). In fact, few people seem able to write a correct and “always fast, even in failing cases” regexp to match a Python triple-quoted string literal.
Those are problems people actually wrestle with. You’re the only one to date who owned up to being burned by pausing an empty string as the pattern, and despite that it does exactly what you asked it to do .
.
.
There are many options because many use cases want different behaviors in different cases. The defaults picked overwhelmingly match industry-wide behaviors for regexps, except that Python seems unusual in using