Argparse: Add `subnamespace` flag to allow nested `Namespace`s

Ideas
1

Summary

We add the keyword-only flag subnamespace to _SubParsersAction.add_parser(), which when set to True tells the parent parser to store the subparser’s parsed arguments contained in their own Namespace, nested within the parent’s Namespace.

This allows hierarchical nested Namespaces to form when calling parser.parse_args(), which mirror the hierarchical nature of nested subparsers.

This also solves the problem of parsers and their subparsers having conflicting dest parameters which are subject to shadowing / conflict resolution.

By default subnamespace is False for backwards compatibility, making this feature opt-in.

Mini Example

Taken from the implementation tests

import argparse

inet = argparse.ArgumentParser(add_help=False)
inet.add_argument("address")
inet.add_argument("port", type=int)
inet.add_argument("--use-proxy", action="store_true")

unix = argparse.ArgumentParser(add_help=False)
unix.add_argument("path")

parser = argparse.ArgumentParser(prog="my-socat")
parser.add_argument("--key-file")
action = parser.add_subparsers(required=True, dest="action")

parser_bind = action.add_parser("bind", subnamespace=True)
parser_bind.add_argument("--fork", action="store_true")
bind_family = parser_bind.add_subparsers(required=True, dest="family")

parser_bind_inet = bind_family.add_parser("inet", subnamespace=True, parents=[inet])
parser_bind_unix = bind_family.add_parser("unix", subnamespace=True, parents=[unix])

parser_connect = action.add_parser("connect", subnamespace=True)
connect_family = parser_connect.add_subparsers(required=True, dest="family")

parser_connect_inet = connect_family.add_parser("inet", subnamespace=True, parents=[inet])
parser_connect_unix = connect_family.add_parser("unix", subnamespace=True, parents=[unix])

args = parser.parse_args(["bind", "unix", "/foo/bar/socket"])

assert args == argparse.Namespace(
    key_file=None,
    action="bind",
    bind=argparse.Namespace(
        fork=False,
        family="unix",
        unix=argparse.Namespace(
            path="/foo/bar/socket"
        )
    )
)

assert args.bind.unix.path == "/foo/bar/socket"

Pros

  • Having arguments ordered hierarchically makes the programmer’s life easier. Instead of all arguments being squashed into one noisy Namespace we have a tree-like structure. This also allows one to pick out a subNamespace, eg addr = args.bind.unix or addr = args.bind.inet, with subsequent attribute accesses not being unnecessarily longly prefixed, ie addr.path or addr.address and addr.port being nice and short.

  • The tree-like structure of the Namespaces mirrors the tree-like structure of the parsers. It’s easier to follow along in one’s head or in code where the arguments are stored. Where the subparsers go, the namespaces go.

  • We solve the issue of conflicting dest parameters between parent parser and child parser. I have purposely added a test to show this off. A parent parser and child parser can both have different eg -f options, or --dirty flags. Closes gh-59633. This has been a long-standing issue.

Cons

  • Not exactly a con, but I have refrained from encouraging this in the docs: One may try to come up with a programming style of passing only a subNamespace to a ‘subcommand’ of a program, like the docs talk about svn checkout and svn update etc. There may however be ‘global’ options in the root namespace that one wishes to pass to all subcommands, eg --dry-run. Thus I would still encourage passing around the entire root namespace, albeit much nicer organised into hierarchical subnamespaces, rather than picking out subsections of the root namespace manually and passing only those to a subcommand.

Reference Implementation

Implementation, documentation, and tests on my branch. Diff here.

The implementation itself was rather easy to write. The subparsers are already recursively called and the root namespace built up. Instead of always setattr-ing onto the parent namespace the arguments from the transient namespace of the subparser, we setattr the ‘transient’ namespace itself onto the parent namespace, with the attribute name being the subparser’s name.

Rejected Alternatives

  • Do nothing.

  • Add a dest_prefix argument to _SubParsersAction.add_parser() instead of subnamespace, which causes arguments from subparsers to have their attribute name prefixed with the subparser name during the copying stage of building up the flat root Namespace from subnamespaces. The Namespace’s attribute names will be the same name when accessing them, just with _s instead of .s.

Modifying the mini example:

assert args == argparse.Namespace(
    key_file=None,
    action="bind",
    bind_fork=False,
    bind_family="unix",
    bind_unix_path="/foo/bar/socket"
)

This probably solves the subparser shadowing issue like subnamespace does, but is flat and ugly. Nested subnamespaces make it clear which parsed arguments belong to what subparsers, and allow passing only a subtree of the root namespace if desired, eg via a simple args.bind.

Open issues

  • The same subnamespace-ing behavior could possibly be implemented for _ArgumentGroup (and its subclass _MutuallyExclusiveGroup). I haven’t thought too much about that yet, but on first thought that seems like a good idea.

History

I had this idea back in 2023 and opened issue gh-103639, but I didn’t discuss it properly and thus PR gh-103640 grew stale. This has turned out to be advantageous however as the implementation proposed here is simpler: subnamespace is just a bool, with the attribute name at which the child Namespace is stored being deduced from the child parser name, rather than being specified manually.

What did we I do before this forum? :sweat_smile:

2 Likes
2

Github Issue and PR with proposed implementation (up for discussion here or there).