`fnmatch.translate` and anchoring

Ideas
1

fnmatch.translate can be used for converting glob patterns to regular expressions. It’s especially convenient if you want your tool to support pattern matching, but don’t want to confuse users with regular expressions.

A problem with the function is that, because it is designed to be used with re.match (soon to be also known as re.prefixmatch), it doesn’t work well with re.search. The reason is that it adds anchoring to the end:

>>> fnmatch.translate("Py*n")
'(?s:Py.*n)\\Z'
>>> reobj = re.compile(_)
>>> reobj.search("Monty Python")
<re.Match object; span=(6, 12), match='Python'>
>>> reobj.search("Python Discussions")
>>>

The pattern matching if there’s something before the matched part, but not if there’s something after it, is pretty confusing. I can see it causing similar issues that re.match has caused and why the re.prefixmatch alias was added after a discussion here.

If you know about this behavior, you can add * at the end of the glob pattern, but that slows down pattern matching compared to not having anchors at all:

>>> fnmatch.translate("Py*n*")
'(?s:Py(?>.*?n).*)\\Z'
>>> reobj = re.compile(_)
>>> reobj.search("Monty Python")
<re.Match object; span=(6, 12), match='Python'>
>>> reobj.search("Python Discussions")
<re.Match object; span=(0, 18), match='Python Discussions'>
>>> 
>>> timeit(lambda: reobj.search("Python Discussions"))
0.4795150120044127
>>> 
>>> no_anchor = re.compile("(?s:Py.*n)")
>>> timeit(lambda: no_anchor.search("Python Discussions"))
0.23413758003152907

I propose enhancing the function so that we make anchoring configurable. I see two possibilities:

  1. Add new parameter anchoring: bool | None = None that can be used like anchoring=True to anchor both ends and anchoring=False to disable anchoring altogether. The default behavior would be preserving the current functionality for backwards compatibility. I believe I’d typically use anchoring=False and then use either re.search or re.fullmatch depending on the context.
  2. Another alternative would be adding anchor_start: bool = False, anchor_end: bool = True. This would be even more flexible than the earlier alternative, but due to backwards compatibility defaults would need to be inconsistent. It would be somewhat strange that if you use anchor_end=False, neither end is anchored.

How do others feel about this? Should I submit an issue?

5 Likes
2

To suggest another bikeshed colour, I wonder if using prefix/suffix language, linking it to regex, might be more understandable? I.e., go with number 2 but name them is_prefix or prefixmatch or similar, and equivently for suffix.

The issue then from my end is that a word is not usually both a suffix and a prefix, which might imply mutual exclusivity of the keyword arguments.

3

I don’t particularly care about the bikeshed color, but I believe the typical use case is disabling anchoring altogether and then using re.search, re.fullmatch or re.(prefix)match depending on the context. With that in mind, fnmatch.translate(pattern, anchoring=False) looks good to me. Full anchoring with anchoring=True looks fine too, but in most cases passing the pattern re.fullmatch would be simpler. anchoring=None would look strange, but there are no real needs to use None other than as the backwards-compatibility preserving default value.

1 Like
4

The reason I’m looking at this is that our tool has a grep-like utility with this code:

pattern = fnmatch.translate(f"{pattern}*")

The pattern is then a bit later used with re.search for matching lines. I wanted to make it explicit that we are not looking for an anchored match and changed the code to:

pattern = fnmatch.translate(f"*{pattern}*")

To my surprise, a user reported a huge performance regression. The numbers apparently vary depending on the pattern and the file size, but in one case the old code took 9ms while the new one took 55s!

I was curious why the change could be so big and decided to look at the fnmatch.translate source code. From there I learned that the generated regexp gets quite a bit more complicated if the pattern has * wildcards and especially if it has more than one of them. My explicit pattern f"*{pattern}*" guaranteed that there’s always two or more wildcards and even our original version f"{pattern}*" always has one.

I’m now planning to change our code to

pattern = fnmatch.translate(pattern)[:-2]

It ought to work exactly the same way as

pattern = fnmatch.translate(pattern, anchoring=False)

that I have proposed here, but the latter would be cleaner and wouldn’t break if someone changed the end of the pattern returned by fnmatch.translate. Very importantly, having this in the standard library would make the functionality easy to discover and avoid others making the same mistake I made.

1 Like