# Command line interface for the \`re\` module **URL:** \ **Category:** Ideas\ **Created:** [August 17, 2023, 2:20pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819 "2023-08-17T14:20:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\ **Post date:** [August 17, 2023, 2:20pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/1 "2023-08-17T14:20:12Z")
Some modules in the stdlib have also a simple CLI, for example the gzip module allows to compress and decompress files, the `uuid` module generates the UUID, and the `sqlite3` module provides the SQL REPL. And a number of modules (`tokenize`, `ast`, `symtable`, `dis`) allow to inspect Python sources at different levels. Using such commands as `grep` and `sed` every day I often regret their limited regular expression syntax. I propose to add the CLI for the `re` module which would allow to search and replace in files using the syntax of Python regular expressions. For searching the obvious candidate is implementing the `grep` command. For replacing, I’m still thinking about the interface. It could be possible to implement the `sed` command, but I think that it may be too complex for this. The benefits: - Windows users will have access to powerful tools from the box. - All users will be able to use more powerful regular expression syntax in these tools. So I have few questions to community: 1. Is it a good idea at all? I already implemented `grep`, with almost full set of the GNU `grep` options, and it all in just 250 lines. 2. How to invoke it? 3. Is there existing CLI tool similar to `grep` for replacing text, so we can borrow its interface instead of designing something new? ---
**Author:** ![erlendaasland](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/erlendaasland/32/4378_2.png) [@erlendaasland](https://discuss.python.org/u/erlendaasland)\ **Post date:** [August 17, 2023, 2:32pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/2 "2023-08-17T14:32:24Z")
It is a nice idea. How about simply `python -m grep`? If not, I would prefer `python -m re.grep` over the tool variant. (My favourite stdlib script is `python -m calendar`) ---
**Author:** ![jeanas](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jeanas/32/8115_2.png) [@jeanas](https://discuss.python.org/u/jeanas)\ **Post date:** [August 17, 2023, 2:53pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/3 "2023-08-17T14:53:23Z")
> [@storchaka](#): > > Is there existing CLI tool similar to `grep` for replacing text, so we can borrow its interface instead of designing something new? ripgrep (`rg` command) is quite popular these days. There are also `pcregrep`, `ack` and `ag`, off the top. And, of course, `git grep`. Note that `rg --engine=pcre2` or `git grep -P` or `ag` or `ack` gives you Perl-compatible regular expressions, which is about what Python supports, so I’m not sure this is worth doing. All of these support Windows. ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [August 17, 2023, 3:36pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/4 "2023-08-17T15:36:01Z")
Since the trend for this kind of tool seems to be towards short command names, can’t we just use ```auto python -m re PATTERN FILE ... ``` as the basic incantation? It could import `re. __main__ ` so it oughtn’t weigh down programmatic imports of the `re` module. I don’t think we should put any effort into perf or functionality to match tools like `ripgrep` or my personal standby, good old `ag` (Silver Searcher). It would be nice if it could search directories though. Maybe once we have no-GIL implemented we can add a `-j` flag. ---
**Author:** ![khs](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@khs](https://discuss.python.org/u/khs)\ **Post date:** [August 17, 2023, 6:09pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/6 "2023-08-17T18:09:30Z")
To iterate earlier response, what are the benefits compared with rg and other tools that are used everywhere today? ripgrep is one of the first tools I install if I’m getting a new dev system. ---
**Author:** ![stoneleaf](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stoneleaf/32/88_2.png) [@stoneleaf](https://discuss.python.org/u/stoneleaf)\ **Post date:** [August 17, 2023, 6:32pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/7 "2023-08-17T18:32:53Z")
> [@khs](#): > > what are the benefits compared with rg and other tools that are used everywhere today? They are not used everywhere today, as not everyone is a professional developer; but even casual Python programmers can benefit from an easy-to-use and built-in grep tool. ---
**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\ **Post date:** [August 17, 2023, 6:39pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/8 "2023-08-17T18:39:29Z")
This seems like it could be a package? Adding a slow version of `grep` to the stdlib seems like a lot of added maintenance, documentation, and support for a small convenience \[1\] Beginners who don’t know to install a better tool would be better served with a tutorial on how to write the appropriate python script for their task. 🙂 * * * 1. honestly this doesn’t even seem that convenient to me? ---
**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\ **Post date:** [August 17, 2023, 7:38pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/9 "2023-08-17T19:38:17Z")
> [@jamestwebber](#): > > lot of added maintenance, documentation, and support for a small convenience Serhiy noted that this was implemented in 250 lines, which doesn’t seem a massive burden, and often you do just have standard-library Python on a computer, so I can see the benefits. (c.f. batteries-included). A ---
**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\ **Post date:** [August 17, 2023, 7:45pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/10 "2023-08-17T19:45:19Z")
> [@jeanas](#): > > ripgrep (`rg` command) is quite popular these days. There are also `pcregrep`, `ack` and `ag`, off the top. Thank you, it is interesting. But they only support search (`ripgrep` also allows replacement in the output), not search & replace. > [@guido](#): > > `python -m re PATTERN FILE ...` Yes, it was also my initial idea. But I want to support two different operations: “search” and “search & replace”. They need two different commands. > [@khs](#): > > To iterate earlier response, what are the benefits compared with rg and other tools that are used everywhere today? 1. Better discoverability and accessibility for Python users. 2. More familiar regex syntax for Python users. 3. And the main reason – a CLI is a living test bench. Even for core devs it may be easier to use CLI for simple test than use REPL or write a script. It helps to support both the module to which the CLI belongs and `argparse` in healthy conditions. I have created an issue and a draft PR (only code, without docs and tests): [Command line interface for the `re` module · Issue #108095 · python/cpython · GitHub](https://github.com/python/cpython/issues/108095) ---
**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\ **Post date:** [August 17, 2023, 8:08pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/11 "2023-08-17T20:08:02Z")
> [@AA-Turner](#): > > Serhiy noted that this was implemented in 250 lines, which doesn’t seem a massive burden, and often you do just have standard-library Python on a computer, so I can see the benefits. (c.f. batteries-included). 250 lines might be how it _starts_, but the [scope creep](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/4) has already begun. 😉 Even more than maintaining the code, I was thinking that “how do I use grep” is suddenly a relevant topic for Python Help and similar places. This requires documentation and continuous support to be a useful feature for beginners, and it’s a redundant feature for non-beginners. Is there an OS that comes with python pre-installed, but _doesn’t_ give you grep \[1\]? In any case, I thought the idea was to discourage using system-managed python installations. I use a bunch of different environments in my work, and I really don’t want to worry that the activated environment will change how my file-searching tool works. I feel like the intersection of people who a) strongly prefer python regular expressions to other tools and b) **can’t install a PyPI package** is roughly no one. This feels like a perfect candidate for a package that people can install in their path. There’s no need to add the documentation and support burden to the stdlib, and it would delay availability by many years (what’s the version on that “os-provided python” you’ve got, anyway?). Personally I lean more on the side PEP 594, and I don’t think python needs any more miscellaneous batteries than it has. * * * 1. i.e. does Windows? ---
**Author:** ![smontanaro](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/smontanaro/32/1389_2.png) [@smontanaro](https://discuss.python.org/u/smontanaro)\ **Post date:** [August 17, 2023, 8:09pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/12 "2023-08-17T20:09:10Z")
> [@storchaka](#): > > Is it a good idea at all? I already implemented `grep`, with almost full set of the GNU `grep` options, and it all in just 250 lines. Isn’t a regular expression search command already widely available? `grep` (or `egrep` or `grep -E`, depending upon when you started with a Unix CLI) in Unix-like environments is widely available, even on Windows (WSL), right? > [@storchaka](#): > > Is there existing CLI tool similar to `grep` for replacing text, so we can borrow its interface instead of designing something new? I use `sed`, though only in its most elementary form (e.g., `sed -e s/pattern/replacement/[g]`, sometimes `sed -e '/pattern/d'` or its complement, `sed -n '/pattern/p'`). I never understood all the complexity of the rest of it (hold spaces and such). I also use `awk` where it’s simpler (input lines conveniently broken up into words, making `{print $3, $5, $7, ...}` trivial). In short, while providing a grep-like CLI for `re` might be an interesting programming exercise, I suspect other than for sussing out the preferred general approach to such interfaces for Python-based command line tools, I imagine you might find CLI-ifying other modules/packages would fill more of a niche. For example, years ago I did a lot of work with CSV files. It made sense (to me) to use Unix pipelines for quick-n-dirty transmogrification of such data (moving averages, Sharpe ratios, simple plotting, etc). I found it useful enough that I coaxed my employer at the time to let me take this [little toolkit](https://github.com/smontanaro/csvprogs) with me when I moved on to my next job. I don’t claim that it’s a world-beating CLI (it could hardly be called “properly designed”), but it worked for me, and I’ve enhanced it in one way or another over the years. Providing command line access to this sort of functionality (or at least helper packages to create command pipelines) might be more useful. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [August 17, 2023, 9:01pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/13 "2023-08-17T21:01:46Z")
> [@smontanaro](#): > > Isn’t a regular expression search command already widely available? Not on Windows. Yes, tools like ripgrep exist, and there are various ways of getting Unix-like commands. But first of all, we can’t assume these tools are available - in a locked down corporate environment, it’s not impossible that Python is the _only_ tool allowed on servers, for example. And secondly, there’s no way of writing instructions that will work for everyone - “run `grep ...`, or if you have ripgrep, `rg ...`, or if you have neither of those but you have git, find out where git is installed and run `C:\Path\To\Git\bin\grep.exe ...`, or…” Whereas being able to say “run `python -m re ...`”\[1\] works for everyone. Maybe that’s not important. But “everyone has tools like grep these days” just isn’t true, in my experience. It’s a lot closer than it used to be, and there’s usually _something_ that’s no more than a download away, but having something that will always work is a big advantage. For comparison, I’ve never actually used `py -m zipfile` or `py -m tarfile`, but knowing they are always available is an extremely valuable safety net when working in unknown environments. (Actually, even more useful would be if `difflib` had a CLI that handled the basic features of `diff`, because cross-platform implementations of that _are_ still uncommon). * * * 1. Yes, “how do I run Python” is still an issue, there’s the launcher, or store Python, etc. But that’s an issue that’s in our control to solve, if we want to 😉 ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [August 17, 2023, 9:45pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/14 "2023-08-17T21:45:20Z")
> [@storchaka](#): > > Yes, it was also my initial idea. But I want to support two different operations: “search” and “search & replace”. They need two different commands. I would use a command-line flag for replace. In my experience it is much less common (maybe 1% of all uses). ---
**Author:** ![smontanaro](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/smontanaro/32/1389_2.png) [@smontanaro](https://discuss.python.org/u/smontanaro)\ **Post date:** [August 17, 2023, 11:23pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/15 "2023-08-17T23:23:00Z")
> [@pf\_moore](#): > > Not on Windows. I’m not a Windows person, but at my last job (3+ years ago) our Windows systems had WSL. Isn’t that widely available? ---
**Author:** ![Rosuav](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rosuav/32/3429_2.png) [@Rosuav](https://discuss.python.org/u/Rosuav)\ **Post date:** [August 17, 2023, 11:35pm UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/16 "2023-08-17T23:35:52Z")
> [@pf\_moore](#): > > Maybe that’s not important. But “everyone has tools like grep these days” just isn’t true, in my experience. It’s a lot closer than it used to be, and there’s usually _something_ that’s no more than a download away, but having something that will always work is a big advantage. When assisting people on Mac OS, I’ve often run into the problem of “the tool exists, but THIS flag doesn’t” (because it’s not the GNU utility, it’s another with the same name and somewhat similar features). Having something I could depend on would be good, as long as I can dictate a Python command to someone. ---
**Author:** ![tjreedy](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tjreedy/32/137_2.png) [@tjreedy](https://discuss.python.org/u/tjreedy)\ **Post date:** [August 18, 2023, 2:05am UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/17 "2023-08-18T02:05:33Z")
IDLE has a grep facility called ‘find in files’ that uses os.walk to fine directories, fnmatch.fnmatch to filter files, and str or re functions to search lines. I cannot remember any questions about using this feature and found essentially nothing on Stackoverflow searching [python-idle] for ‘grep’ or ‘“find in files”’. If the grep options were sufficiently well documented, I would not expect a support burden. Searching installed 3.12 /Lib/\*.py for a no-match string takes over 10 seconds on my machine. A repeat with caches full takes under 2 seconds. (I am thinking of having IDLE report the search time in the future.) Would a C-coded grep really be much faster? Finding that a no non-idlelib stdlib file does ‘import idlelib’ but one 3rd-parth project I have loaded does (both expected) barely took longer. Edit: I work on Windows so this is my working grep. ---
**Author:** ![flyinghyrax](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/flyinghyrax/32/12394_2.png) [@flyinghyrax](https://discuss.python.org/u/flyinghyrax)\ **Post date:** [August 18, 2023, 2:06am UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/18 "2023-08-18T02:06:15Z")
> [@Rosuav](#): > > When assisting people on Mac OS, I’ve often run into the problem of “the tool exists, but THIS flag doesn’t” (because it’s not the GNU utility, it’s another with the same name and somewhat similar features). Oh fun times! My previous work machine was macOS and I found this out the hard way. Turns out that `grep` on macOS is BSD grep, and a _very old version_ of BSD grep. Which is manageable, if it weren’t also multiple orders of magnitude slower than GNU grep. Literally profiled it once. We had an old automation bash script that used grep a lot internally that I ended up writing a Dockerfile for\[1\] because running the script in a container was faster than natively with the macOS version of grep. Anyway. If there’s a point there I guess it’s that it can be nice to have options. * * * 1. not _solely_ because of grep, but it was definitely a factor… ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [August 18, 2023, 7:27am UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/19 "2023-08-18T07:27:03Z")
It’s sometimes available, but using the tools from WSL in a native shell isn’t easy. They aren’t on `PATH` by default, and they almost certainly handle filenames in a non-native manner. ---
**Author:** ![ncoghlan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ncoghlan/32/14266_2.png) [@ncoghlan](https://discuss.python.org/u/ncoghlan)\ **Post date:** [August 21, 2023, 1:45am UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/20 "2023-08-21T01:45:13Z")
> [@guido](#): > > I would use a command-line flag for replace. In my experience it is much less common (maybe 1% of all uses). An option like “python -m re --sub=‘replacement text’ ‘match pattern’ files…” would also be substantially less cryptic than sed’s “pattern separators with trailing command” approach. Other ideas seemed more ambiguous to me: - allowing two args to imply replacement would conflict with the convention of grep accepting the files to search at the end of the command - same rationale for associating the replacement text directly with the CLI option rather than changing the meaning of the second positional command line argument - “–sub” (rather than “–replace”) matches the name of the module function and re pattern method - a submodule would have to be called “re.replace” to avoid conflicting with the “re.sub” function ---
**Author:** ![gwerbin](https://avatars.discourse-cdn.com/v4/letter/g/76d3ee/32.png) [@gwerbin](https://discuss.python.org/u/gwerbin)\ **Post date:** [August 21, 2023, 4:21am UTC](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819/21 "2023-08-21T04:21:21Z")
I really like this proposal. It would also be nice if it had some flags to control output format like grep. An optional JSON output mode would be especially interesting for downstream processing. [Next page](https://discuss.python.org/t/command-line-interface-for-the-re-module/31819.md?page=2)