Stop ArgumentParser from forcing arguments containing spaces to be positional

Ideas
1

ArgumentParser currently forces arguments that contain spaces to be treated as positional, even if they look like an option (start with --). This is different to the Unix convention and getopt behaviour in particular, so it might be unexpected.

I raised this as a bug, but it was closed because it appears to be deliberate, and thus a discussion (here) is required for anyone to consider changing it.

Take for example this simple setup:

>>> from argparse import ArgumentParser
>>> parser = ArgumentParser(exit_on_error=False)
>>> parser.add_argument('tags', nargs='*')

Note that the --timeout option that I will use below is not defined here, so it should be rejected as an unrecognised argument, and it normally is:

>>> parser.parse_args(['--timeout'])
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    parser.parse_args(['--timeout'])
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/argparse.py", line 1904, in parse_args
    raise ArgumentError(None, msg)
argparse.ArgumentError: unrecognized arguments: --timeout

However if we accidentally included a space in the argument, then it would not be rejected, but treated as a valid positional argument instead:

>>> parser.parse_args(['--timeout 1'])
Namespace(tags=['--timeout 1'])

If I do the same with getopt I get the more expected result, that --timeout 1 is treated as an unrecognised option:

>>> from getopt import getopt
>>> getopt(['--timeout', '1'], '', ['timeout='])
([('--timeout', '1')], [])
>>> getopt(['--timeout 1'], '', ['timeout='])
Traceback (most recent call last):
...
getopt.GetoptError: option --timeout 1 not recognized

The code in argparse.py which deliberately treats the unrecognised option --timeout 1 as a positional argument is here:

    def _parse_optional(self, arg_string):
        ...
        # if it contains a space, it was meant to be a positional
        if ' ' in arg_string:
            return None

I must say I donā€™t understand why this was done. Itā€™s been that way since the argparse module was created. I donā€™t see a specific test for such arguments either, so I donā€™t think any existing test cases would break if this was changed as suggested below, although I havenā€™t tried it.

Iā€™d like to suggest removing these lines:

        # if it contains a space, it was meant to be a positional
        if ' ' in arg_string:
            return None
2

Can you explain why this should change, and why the result would be better? I donā€™t consider matching getopt to be, in itself, an argument for change.

The behavior seems pretty quirky to me, but Iā€™m not sure what we should expect a CLI parser to do with ["--timeout 1"] other than potentially to error.

In theory, someone could be relying on this behavior to forward arguments from one tool to another. Iā€™d consider this a minor backwards incompatible change. Which means it can happen but there needs to be strong motivation.

1 Like
3

This looks like a misguided attempt to correct some kind of upstream
error in splitting a command line into arguments. But the correction
doesnā€™t make much sense. It might make some sense to treat it as two
arguments , ā€œā€“timeoutā€ and ā€œ1ā€, but I canā€™t see any logic behind
turning it into a positional argument.

A possible justification for removing it is that it could mask errors.
It would be preferable to get an error message such as ā€œUnrecognised
option ā€˜ā€“timeout 1ā€™ā€ than to, e.g., go on to erroneously create a file
called ā€˜ā€“timeout 1ā€™.

1 Like
4

There are 12 errors:

Summary 
======================================================================
ERROR: test_successes_many_groups_listargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_many_groups_listargs) (args=['-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 253, in listargs
    return parser.parse_args(args)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35munrecognized arguments: -a badger\x1b[0m\n')

======================================================================
ERROR: test_successes_many_groups_listargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_many_groups_listargs) (args=['-y', '-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 253, in listargs
    return parser.parse_args(args)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35margument -y/--yyy: expected one argument\x1b[0m\n')

======================================================================
ERROR: test_successes_many_groups_sysargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_many_groups_sysargs) (args=['-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 260, in sysargs
    return parser.parse_args()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35munrecognized arguments: -a badger\x1b[0m\n')

======================================================================
ERROR: test_successes_many_groups_sysargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_many_groups_sysargs) (args=['-y', '-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 260, in sysargs
    return parser.parse_args()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35margument -y/--yyy: expected one argument\x1b[0m\n')

ERROR: test_successes_no_groups_listargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_no_groups_listargs) (args=['-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 253, in listargs
    return parser.parse_args(args)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35munrecognized arguments: -a badger\x1b[0m\n')

======================================================================
ERROR: test_successes_no_groups_listargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_no_groups_listargs) (args=['-y', '-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 253, in listargs
    return parser.parse_args(args)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35margument -y/--yyy: expected one argument\x1b[0m\n')

======================================================================
ERROR: test_successes_no_groups_sysargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_no_groups_sysargs) (args=['-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 260, in sysargs
    return parser.parse_args()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35munrecognized arguments: -a badger\x1b[0m\n')

======================================================================
ERROR: test_successes_no_groups_sysargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_no_groups_sysargs) (args=['-y', '-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 260, in sysargs
    return parser.parse_args()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35margument -y/--yyy: expected one argument\x1b[0m\n')

======================================================================
ERROR: test_successes_one_group_listargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_one_group_listargs) (args=['-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 253, in listargs
    return parser.parse_args(args)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35munrecognized arguments: -a badger\x1b[0m\n')

======================================================================
ERROR: test_successes_one_group_listargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_one_group_listargs) (args=['-y', '-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 253, in listargs
    return parser.parse_args(args)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35margument -y/--yyy: expected one argument\x1b[0m\n')

======================================================================
ERROR: test_successes_one_group_sysargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_one_group_sysargs) (args=['-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 260, in sysargs
    return parser.parse_args()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35munrecognized arguments: -a badger\x1b[0m\n')

======================================================================
ERROR: test_successes_one_group_sysargs (test.test_argparse.TestEmptyAndSpaceContainingArguments.test_successes_one_group_sysargs) (args=['-y', '-a badger'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 308, in test_successes
    result_ns = self._parse_args(parser, args)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 260, in sysargs
    return parser.parse_args()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 191, in parse_args
    return stderr_to_parser_error(parse_args, *args, **kwargs)
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_argparse.py", line 180, in stderr_to_parser_error
    raise ArgumentParserError(
        "SystemExit", stdout, stderr, code) from None
test.test_argparse.ArgumentParserError: ('SystemExit', '', '\x1b[1;34musage: \x1b[0m\x1b[1;35mpython -m test.libregrtest.worker\x1b[0m [\x1b[32m-h\x1b[0m] [\x1b[32m-y \x1b[33mY\x1b[0m] \x1b[32m[x]\x1b[0m\npython -m test.libregrtest.worker: \x1b[1;35merror:\x1b[0m \x1b[35margument -y/--yyy: expected one argument\x1b[0m\n')

----------------------------------------------------------------------
Ran 1945 tests in 6.024s

FAILED (errors=12)
test test_argparse failed
1 Like
5

Perhaps a better fix would be

...
if " " in arg_string:
    if not "=" in arg_string:
        return None
    else:
        raise ArgumentError(None, f"unrecognized arguments {arg_string}")

We have a check of the positional not being positional, and being defined before, but no check of it not being defined, at least not in the function itself.

6

Usually we get complains that argparse does allow to pass arguments starting with ā€œ-ā€ for options with a required argument. I.e. --foo --bar does not mean option --foo with value --bar, even if --foo requires an argument. This is the first case when we get a complain that argparse does not interprets argument starting with ā€œ-ā€ as option.

When you pass an argument containing spaces in the CLI, you have either quote the whole argument ("--timeour 1") or escape spaces (--timeour\ 1). In any waycase you have to do this intentionally, and this does not look like an option, So the argparse behavior is reasonable. But it differs from the behavior of most of other programs (which most likely use getopt). We may change this if we find enough good reasons, but this can break user code,

4 Likes
7

Too much of a corner case to risk breaking existing code.

1 Like