Getopt and optparse vs argparse

Core Development
103

I have never said that optparse or even getopt should be removed from the face of the Earth. Just what should be in the standard library, what would satisfy those 80% of users for whom the standard library is intended [1]. Of course, optparse (to be honest, getopt is so tiny, that it could be pretty much left be) would be on PyPI and it could be very well used by anybody who actually needs it.

[1] https://dreamsongs.com/RiseOfWorseIsBetter.html

104

I’m looking at adding a feature to my wrapper library and rediscovered a limitation of argparse that frustrates me: lack of nested subparsers.

In my tracking bug, I use the command line tool (popular at least on Linux), nmcli.

A small sample:

nmcli [options] {help | general | networking | radio | etc }
nmcli general {status | hostname | permissions | logging | reload }
nmcli networking {on | off | connectivity}
nmcli radio {all | wifi | wwan}
nmcli radio all {on | off}

And many of those have additional sub-commands with mixed positional and flagged parameters.

For now, I use something like:

toy feature1-cmd1-list
toy feature1-cmd1-set
toy feature1-cmd1-del
toy feature1-cmd2-...

At least it is the same number of characters… but does make the --help output really long.

I guess what I’m suggesting is:
Don’t have a 10-year plan for argparse. Create argparse2 from scratch, addresses all known issues (creating new ones, of course), and let people migrate as they wish (just like was done with optparse). After all, we’ve migrated from python → python2 → python3. As experience is gained, lessons learned, and the world changes, sometimes it really is easier/better to set a boundary and what can/will be done.

105

This is how we ended up having both optparse and argparse in the stdlib with both needing to remain in perpetuity. In retrospect it would have been better if commonly used features had been added to optparse as needed or a clear decision had been taken not to add the features either in optparse or in any new stdlib module.

Third party libraries will provide for all of your nested subparser needs. It was a mistake to add argparse as a separate stdlib library to optparse. Some justification mentioned above is that installing third party packages was harder back then: the same argument definitely does not work today. If PEP 389 was on the table today then the response would be unambiguous: it doesn’t need to be in the stdlib, just put it on PyPI. You can of course create argparse2 from scratch but it would be a PyPI package and would never be added to the stdlib.

8 Likes
106

This is possible and quite easy, we just need to document the pattern: hatch/backend/src/hatchling/cli/__init__.py at hatchling-v1.26.3 ¡ pypa/hatch ¡ GitHub

image
image1066×313 7.15 KB

2 Likes
107

Sorry, I was not clear.

I consider those “top level commands” and when I said “nested parsers”, I mean one more layer below that.

The command line tool I often use as an example is nmcli(1) (common on Linux, unsure of other platforms).

It roughly goes like this:

nmcli {help | general | device | etc }
nmcli general { status | hostname | etc }
nmcli device { status | show | set | up | down | etc }

And some of those are nested even more, and each leaf command can have its own set of options/flags

For my projects, I ended up doing something like:

address-update 
address-type-list
address-type-set
address-type-delete
address-type-value-list
address-type-value-set
address-type-value-delete

Which works (- is just as easy to type as ), but really clutters up the --help output.

1 Like
108

From my experience argparse handles this.
ArgumentParser.add_subparsers

I have cli based on argparse, which does:

cmd -h
cmd subcmd -h
cmd subcmd subsubcmd -h

Or isn’t this what you are talking about?

1 Like
109

I was about to say the same thing, that is supported and if you look at the code I linked you can apply the pattern in a nested fashion arbitrarily deep.

1 Like
110

Huh.

I think this entire time (years) I kept trying to add a subparser directly to a subparser rather than a parser.

I did look at the hatchling stuff, just not enough and missed that the dep command has the synced subcommand.

So, wow, thanks!

4 Likes
111

A (slightly) early Christmas present for happy optparse and getopt users: the PR reversing their formal deprecation notices has been merged to main (backport to 3.13 is in progress, and it will show up on docs.python.org in the next day or so).

The optparse docs now begin with specific guidance on Choosing an argument parsing library (and the docs for the other two libraries reference that guidance).

(see the commit to main for additional details on exactly what has changed)

Edit: The new docs are live: optparse — Parser for command line options — Python 3.13.1 documentation

20 Likes
112

(Edit… Removed my original comment)…

I completed missed the blocks at the top of argparse and optparse. All good. Sorry for that.

1 Like
113

I concur with this, or perhaps this is too long for the library documentation and it would be better just to link to Argparse Tutorial — Python 3.13.1 documentation? However, yes, the link should go to manuals for all three libraries. Also, it would be nice to have in the top of that HOWTO some short summary in the style “If you don’t know any better, just use argparse and you will be well served.”.

114

That is how the intro to the argparse tutorial is already framed: the first paragraph states that argparse is the recommended default choice, and then the note block acknowledges the existence of the other two libraries and explains the tutorial doesn’t cover those, but may still be useful for getting started with optparse since many of the concepts are similar.

The docs markup could potentially be tweaked to make the note block regular paragraphs, but there’s no equivocation in the text itself on the recommended default choice when folks don’t already have a specific reason for choosing one of the other options.

1 Like
115

Yes, I was not so much concerned about the wording of the text as by having two versions of essentially the same document in the library.

116

I’m confused, which documents are duplicating each other? We have five:

  • the respective API docs for the three modules
  • the argparse tutorial
  • the optparse to argparse migration guide

The largest block of duplicated text is shared between the end of the getopt docs and the start of the optparse docs (since they each include the same chunks of example optparse and argparse code).

Edit: oh, I see the source of confusion. The full comparison of the three libraries isn’t duplicated. It only lives in the optparse docs, since that’s the first alternative we want people to evaluate when argparse isn’t serving them well, and then the other 4 docs all reference that section of the optparse docs early in their own text.

117

Perhaps somewhat off-topic, but the argparse docs are regularly mentioned as being confusing and difficult to use for beginners and experienced users alike (I usually prefer to cargo-cult some argparse from another project instead of trying to find help in the docs; ChatGPT also works great :slight_smile: ).

The Editorial Board has an intent to do something about this in the upcoming months (it was on the top three modules we found needed a serious docs upgrade, in addition to asyncio and all things threading).

5 Likes
118

I meant Argparse Tutorial — Python 3.13.1 documentation and optparse — Parser for command line options — Python 3.13.1 documentation

119

I suspect some overlap is to be expected. Argparse was derived from optparse I think. I’m sure doc patches would be appreciated.

120

One thing I just noticed is the link for getopt in the Library Reference index is still at the bottom in a “Superceded” section. It’s not present in the “Command Line Interface Libraries” section. I created an issue and a trivial PR, then pinged @ncoghlan.

121

Leaving getopt as superseded was intentional. We reverted the formal deprecation because we didn’t want existing users to feel they needed to switch to something else, but the cases where it’s a better choice than optparse for new code are sufficiently niche that it doesn’t make sense to present it at the same level as the more declarative alternatives.

122

Ah, yeah, I wasn’t looking to address any pre-existing duplication in the details of the optparse docs.

I did mention looking at the idea, and I do agree it would be worth doing (they have never been seriously revised to account for argparse and its tutorial being added), but the deprecation reversal PR was already big enough so I didn’t end up pursuing that task myself.