Getopt and optparse vs argparse

dg-pb · October 31, 2024, 3:38am

Github search:

import getopt - 100K files
import optparse - 124k files
import argparse - 3.5M files

ncoghlan · October 31, 2024, 4:15am

Adding to the Python GitHub search results:

import click - 327k files

(for how new it is, even Typer manages a respectable 45k)

ofek · October 31, 2024, 6:26am

This is basically what I was going to come here to write. As someone who maintains a significant number of CLIs, I feel very strongly that we should maintain a single argument parser in the standard library and it should be based on actual usage. I’ve been in this community for over a decade and I’ve never seen code in the wild (other than pip) using those other two options nor tutorials.

We should recommend third-party libraries like Click in the documentation as we do for HTTP functionality, and TOML as Alyssa mentioned.

The shadow you refer to pales in comparison to the potential one produced by having three suboptimal options in the standard library. We should recommend third-party libraries and continue our reduction of standard library modules, that’s the solution.

Barney Gale:

I know it will take longer and be more difficult, but IMHO it’s better to execute your plan without marking argparse as provisional or un-deprecating optparse. Deprecating and un-deprecating optparse sends a confusing message to users, especially as these bugs have existed the entire time that argparse has been in the standard library. I agree the bugs are bad - and only get more terrifying when you debug them and realise how argparse really works - but they’re edge cases that don’t affect most users, and often have simple workarounds, like re-arranging or quoting arguments. Just my 2c.

(Admittedly I quite like argparse and I’ve been using it since before it was in the standard library, because its interface is better than getopt, and it parses positional arguments unlike optparse.)

I agree with this completely, both regarding the bugs being edge cases and the general favorability of argparse. It’s easy to use, supports positional arguments, has fantastic documentation, and is familiar to most people because that’s just what everyone uses and writes tutorials about.

Why is there a desire for something niche in the standard library? Seems like a fantastic use case for a third-party library.

I want to be clear that I very much appreciate your work on this and more broadly as a core developer! At the same time, that statement gives me pause. I feel very much like we are in two different worlds. To me that sentence is exactly the same as saying that the re module is insufficient because it doesn’t support POSIX character classes like [[:digit:]], without acknowledging that the vast majority of users are content.

If you feel very strongly that those other two should be brought back then I would say a better option is to deprecate the remaining one. I don’t agree with that approach of course, as I prefer that we simply recommend third-party libraries as we usually do.

ncoghlan · October 31, 2024, 7:06am

@storchaka and I have a couple of alternative PRs up considering potential options for reversing the deprecation of optparse (and bringing the status of getopt back into line with the PEP 389 recommendation to leave it undeprecated and just note that it serves a very specific purpose in emulating C application behaviour). (Both PRs are flagged as DO-NOT-MERGE for now, since the discussion of whether or not the deprecations even should be reversed is still ongoing)

Serhiy’s is a relative minimalist change, mostly just restoring the relevant docs structure that existed in Python 3.12: gh-126180: Undeprecate getopt and optparse by serhiy-storchaka · Pull Request #126186 · python/cpython · GitHub

Mine takes a different perspective, explicitly positioning the 2 libraries as “argparse is for writing command line applications, while optparse is for implementing command line argument processing libraries (and potentially for working around certain argparse limitations if they affect your application)”: gh-126180: Remove getopt and optparse deprecation notices by ncoghlan · Pull Request #126227 · python/cpython · GitHub

For those advocating that the formal deprecation of optparse should stand:

if you’re writing your own command line argument processing library, you probably should be using optparse (not argparse), for the same reasons click does so
if you’re accepting command line options with parameter values where - is likely to be present as the first character, the way argparse works is probably going to cause you problems, and you’ll likely be better off using either optparse or one of the third party libraries based on it (like click or Typer)

The situation with getopt is different. While it wouldn’t pass muster as a new standard library addition these days, it also isn’t causing any maintenance hassles, so there’s no reason to disrupt the people that are using it, and if your goal is to emulate the behaviour of what C getopt can do, it’s a better option than even optparse. (I may move it back to the “Superseded” section of the docs in my draft PR, though. The use cases for it really are far more obscure than the use cases for the other two modules).

storchaka · October 31, 2024, 7:57am

This may be partially true, but the main reason is that they have less features and simpler internal logic, so it was faster to sweep out bugs. The last real bug report for getopt was in 2008 (fixed in 2010). All following bug reports were invalid – just misunderstanding from the user side. On other hand, argparse has design flaws. Its bugs are so long living because it is not easy to fix them while keeping most of other features working.

In foresight, argparse should not be added to the stdlib. It would be better to develop existing optparse than add a new module with similar but different API and complete different logic. getopt and optparse do not concur – one provides a low level API similar to APIs in other programming languages, and other provides an object-oriented extensible API.

What done is done, and I do not want to remove any of the existing module from the stdlib. This will only distrupt the use code. But argparse needs a lot of work. It was mistake to recomend it. We should not downplay other modules. Migrating from getopt or optparse to argparse will now cause more headaches than benefits.

No, this is like having an alternative to the re module in the stdlib, which supports POSIX character classes and many other advanced features, but with caveats – all quantifiers are non-greedy, backslash cannot be used in the character set, backtracking works only at one level, and capturing groups do not work in alternations. It may work identical to other regular expressions in other programming languages, but will shoot you in the foot more often than you want it.

hugovk · October 31, 2024, 10:36am

Searching the top 8k PyPI projects (the text below wrapped in regex \b) gives similar proportions:

import getopt - Found 256 matching lines in 117 projects
import optparse - Found 369 matching lines in 190 projects
import argparse - Found 5308 matching lines in 1273 projects

Third-party:

import click - Found 3018 matching lines in 332 projects
import typer - Found 996 matching lines in 38 projects

ofek · October 31, 2024, 12:42pm

I would like to ask this question again because I think it’s incredibly important:

DavidCEllis · October 31, 2024, 1:01pm

There have been a few suggestions that the documentation should recommend third-party solutions but I want to point out that outside of simply being included in the stdlib, one of the major advantages argparse has is that it has a significantly lower import time than the alternatives^[1].

This is to the point where a simple argparse based tool can import, parse arguments, execute and exit before many of the alternatives have finished their initial imports.

With “pass” as a baseline.

Summary
  python -c "pass" ran
    1.23 ± 0.06 times faster than python -c "import argparse"
    2.03 ± 0.10 times faster than python -c "import datargs"
    2.05 ± 0.10 times faster than python -c "import click"
    2.22 ± 0.11 times faster than python -c "import pyrallis"
    2.40 ± 0.12 times faster than python -c "import tap"
    3.20 ± 0.16 times faster than python -c "import simple_parsing"
    3.89 ± 0.18 times faster than python -c "import tyro"
    5.10 ± 0.24 times faster than python -c "import cappa"
    5.21 ± 0.25 times faster than python -c "import typer"

Full timings

Benchmark 1: python -c "pass"
  Time (mean ± σ):      27.4 ms ±   1.2 ms    [User: 11.7 ms, System: 12.6 ms]
  Range (min … max):    25.5 ms …  30.9 ms    20 runs

Benchmark 2: python -c "import argparse"
  Time (mean ± σ):      33.5 ms ±   0.6 ms    [User: 21.5 ms, System: 13.8 ms]
  Range (min … max):    32.4 ms …  34.7 ms    20 runs

Benchmark 3: python -c "import click"
  Time (mean ± σ):      56.0 ms ±   1.0 ms    [User: 29.3 ms, System: 21.6 ms]
  Range (min … max):    54.5 ms …  58.0 ms    20 runs

Benchmark 4: python -c "import typer"
  Time (mean ± σ):     142.7 ms ±   2.2 ms    [User: 99.5 ms, System: 36.6 ms]
  Range (min … max):   139.1 ms … 145.9 ms    20 runs

Benchmark 5: python -c "import cappa"
  Time (mean ± σ):     139.5 ms ±   2.1 ms    [User: 98.0 ms, System: 40.5 ms]
  Range (min … max):   137.5 ms … 146.5 ms    20 runs

Benchmark 6: python -c "import datargs"
  Time (mean ± σ):      55.5 ms ±   1.0 ms    [User: 34.7 ms, System: 19.0 ms]
  Range (min … max):    54.0 ms …  58.7 ms    20 runs

Benchmark 7: python -c "import pyrallis"
  Time (mean ± σ):      60.8 ms ±   0.8 ms    [User: 37.1 ms, System: 20.5 ms]
  Range (min … max):    59.6 ms …  63.0 ms    20 runs

Benchmark 8: python -c "import simple_parsing"
  Time (mean ± σ):      87.6 ms ±   2.1 ms    [User: 53.4 ms, System: 29.4 ms]
  Range (min … max):    85.5 ms …  91.9 ms    20 runs

Benchmark 9: python -c "import tap"
  Time (mean ± σ):      65.7 ms ±   1.3 ms    [User: 45.6 ms, System: 19.3 ms]
  Range (min … max):    64.3 ms …  68.1 ms    20 runs

Benchmark 10: python -c "import tyro"
  Time (mean ± σ):     106.6 ms ±   1.3 ms    [User: 73.0 ms, System: 30.5 ms]
  Range (min … max):   104.0 ms … 108.8 ms    20 runs

This is before doing any parsing or any other work for your application.

At least all of those that I’m aware of. ↩︎

ofek · October 31, 2024, 1:06pm

Very true, that’s often why I use it! Also it seems like it’s been optimized for import time versus the proposed alternative:

flying-sheep · October 31, 2024, 1:16pm

Why is there a desire for something niche in the standard library? Seems like a fantastic use case for a third-party library.

I fully agree.

If there are bugs or missing features in argparse, they should be fixed (unless these features revolve about ambiguity like nargs="*" for options, because ambiguity should be avoided in CLI design and not encouraged)

If there are niche use cases that argparse will never be able to deal with, people should gather around a 3rd party module that implements an API for that niche.

mcepl · October 31, 2024, 1:25pm

Why in the world we should do it, when for 99.9% (OK, 90% to be tolerant) of use cases of argparse are just fine? Removal of getopt and optparse, that’s a different thing, one I could happily stand behind.

DavidCEllis · October 31, 2024, 1:33pm

I think optparse has a couple of extra imports, it also depends on locale and textwrap. This makes the import a little slower than argparse, but not on the same level as some of the other packages.

argparse/optparse comparison

Benchmark 1: python -c "pass"
  Time (mean ± σ):      26.2 ms ±   0.7 ms    [User: 15.4 ms, System: 11.2 ms]
  Range (min … max):    24.9 ms …  27.4 ms    20 runs

Benchmark 2: python -c "import argparse"
  Time (mean ± σ):      33.7 ms ±   0.8 ms    [User: 18.7 ms, System: 15.0 ms]
  Range (min … max):    32.8 ms …  35.5 ms    20 runs

Benchmark 3: python -c "import optparse"
  Time (mean ± σ):      35.0 ms ±   0.9 ms    [User: 17.7 ms, System: 17.4 ms]
  Range (min … max):    34.0 ms …  37.2 ms    20 runs

Summary
  python -c "pass" ran
    1.29 ± 0.05 times faster than python -c "import argparse"
    1.34 ± 0.05 times faster than python -c "import optparse"

dg-pb · October 31, 2024, 1:45pm

I think it might be useful to give some time for more people to look at argparse issues. Maybe someone can come up with new solutions to address its shortcomings? Even if it would break things, it might be worth it given then benefit of not having 2 modules that are very similar.

storchaka · October 31, 2024, 1:46pm

“Niche” here does not mean something used by a handful of users. This means that getopt firmly occupies a place that cannot be occupied by something else. This is a part of a niche larger than Python. getopt is a good entry point for users already familiar with getopt in other programming languages. It helps to port programs to Python. I do not understand desire to exclude it from the stdlib.

All this shows that we did a great job with promoting argparse.

Yes, it was optimized. See Reduce the number of imports for argparse · Issue #74338 · python/cpython · GitHub.

storchaka · October 31, 2024, 1:52pm

You are welcome.

If you break enough things to fix all argparse issues, you will get optparse, just with different name. This is my plan actually, but it will take many years to do this right.

dg-pb · October 31, 2024, 2:14pm

As long as this is the goal and direction is clear, I like your proposal (regarding optparse/argparse). After all, if to bring argparse to certain completeness breaking changes are inevitable, then it is very useful to make it provisional.

jamestwebber · October 31, 2024, 2:17pm

I wonder what proportion of new Python programmers are familiar with getopt from another language (mostly C?). These days I suspect that’s pretty rare!

It also shows that the issues with argparse are not deal-breakers for the majority of users–at least, they are outweighed by the ease of use.

Wombat · October 31, 2024, 4:35pm

I agree 100%. These modules are stable and still have their uses.

brettcannon · October 31, 2024, 5:15pm

…

Since we now live in a post-packaging world where you can install things easily and we have all come to accept that the stdlib doesn’t need to have everything in it and it won’t always be the best for everyone’s use-case, then trying to cater to every CLI API use-case in the stdlib seems unnecessary.

To me, that suggests we choose a “winner” CLI module and do what we need to do to make maintaining it easy.

Unfortunately, with the install numbers people are posting, it seems to clearly be that dropping argparse will break too much. But it might not be the end of the world to drop getopt and optparse (but I’m known for wanting to slim down the stdlib, so I have a bias ).

So with that in mind, I like Serhiy’s plan of cleaning up argparse and eventually making it the sole CLI module:

notatallshaw · October 31, 2024, 5:37pm

Taking pip as an example of a popular tool that depends on optparse and assuming that you don’t want to have a tool like that vendor it’s own copy of optparse, with all the subtle issues that might introduce. Then the timeline would have to look like this:

Improve argparse across multiple Python versions
Have an announcement that argparse is a viable alternative for tools like pip
Wait until all versions of Python where argparse isn’t good enough to become EoL
Allow tools like pip to transition to new argparse and hope they don’t find some show stopping issue
Concurrently, deprecate optparse with a target that allows tools to transition

Optimistically this would be ~8 years? To me, it seems more realistically 10 to 15 years.