Mypy vs passing along kwargs (special appearance by argparse.ArgumentParser)

Python Help
1

I have code where I accept kwargs that I augment and pass along to another function (in this case. ArgumentParser()).

I start with the simple module, t.py

import argparse

def foo(use_bar, **kwargs):
    if use_bar:
        pass

    opts = {
        'formatter_class': argparse.RawDescriptionHelpFormatter,
        'add_help': False,
    }

    opts.update(kwargs)

    return argparse.ArgumentParser(**opts)

I want to start using typing with this, so I check with mypy and all is good:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

$ python --version
Python 3.11.2

$ mypy --version
mypy 1.0.1 (compiled: yes)

$ mypy t.py
Success: no issues found in 1 source file

I start by adding type information to just use_bar, and mypy gets upset about ArgumentParser():

$ diff -u0 t.py u.py
--- t.py	2024-01-06 18:00:17.395737928 -0800
+++ u.py	2024-01-06 18:00:17.395737928 -0800
@@ -3 +3 @@
-def foo(use_bar, **kwargs):
+def foo(use_bar: bool, **kwargs):

$ mypy u.py
u.py:14: error: Argument 1 to "ArgumentParser" has incompatible type "**Dict[str, object]"; expected "Optional[str]"  [arg-type]
u.py:14: error: Argument 1 to "ArgumentParser" has incompatible type "**Dict[str, object]"; expected "Sequence[ArgumentParser]"  [arg-type]
u.py:14: error: Argument 1 to "ArgumentParser" has incompatible type "**Dict[str, object]"; expected "_FormatterClass"  [arg-type]
u.py:14: note: "dict" is missing following "_FormatterClass" protocol member:
u.py:14: note:     __call__
u.py:14: error: Argument 1 to "ArgumentParser" has incompatible type "**Dict[str, object]"; expected "str"  [arg-type]
u.py:14: error: Argument 1 to "ArgumentParser" has incompatible type "**Dict[str, object]"; expected "bool"  [arg-type]
Found 5 errors in 1 file (checked 1 source file)

After a lot of experimentation, I finally get this working solution:

$ diff -u0 u.py v.py 
--- u.py	2024-01-06 18:00:17.395737928 -0800
+++ v.py	2024-01-06 18:00:17.395737928 -0800
@@ -1,0 +2 @@
+import typing
@@ -3 +4,12 @@
-def foo(use_bar: bool, **kwargs):
+
+class ArgparseKwargs(typing.TypedDict, total=False):
+    prog: str
+    usage: str | None
+    epilog: str | None
+    formatter_class: argparse._FormatterClass
+    fromfile_prefix_chars: str | None
+    add_help: bool
+    allow_abbrev: bool
+
+
+def foo(use_bar: bool, **kwargs: typing.Unpack[ArgparseKwargs]):
@@ -7 +19 @@
-    opts = {
+    opts: ArgparseKwargs = {

$ mypy --enable-incomplete-feature=Unpack v.py
Success: no issues found in 1 source file

And, of course, pylint is now unhappy:

$ pylint --disable=all --enable=no-member v.py
************* Module v
v.py:9:21: E1101: Module 'argparse' has no '_FormatterClass' member (no-member)

------------------------------------------------------------------
Your code has been rated at 6.88/10 (previous run: 6.88/10, +0.00)

And in the end I wind up with this:


import argparse
import typing


class ArgparseKwargs(typing.TypedDict, total=False):
    prog: str
    usage: str | None
    epilog: str | None
    formatter_class: 'argparse._FormatterClass'
    fromfile_prefix_chars: str | None
    add_help: bool
    allow_abbrev: bool


def foo(use_bar: bool, **kwargs: typing.Unpack[ArgparseKwargs]):
    if use_bar:
        pass

    opts: ArgparseKwargs = {
        'formatter_class': argparse.RawDescriptionHelpFormatter,
        'add_help': False,
    }

    opts.update(kwargs)

    return argparse.ArgumentParser(**opts)

Which seems like quite a bit for wanting to add a wafer-thin : bool.

For this ArgparseKwargs class I had to write, is there a better way than copying it from typeshed/stdlib/argparse.pyi with a transformation (i.e., _FormatterClass)? What have I missed reading?

Note: I’d prefer to keep with whatever Debian ships, but if absolutely necessary, I might use pip.

Thanks,
mrc

2

The untyped version worked because by default mypy won’t check untyped functions. As soon as you write the first type annotation in your function, mypy will start looking at it.

There are two problems here. The first is that you want to forward arguments from your function’s parameters to another callable. As you discovered by yourself, the only way of doing this in a fully-typed manner is by re-writing the full signature of the target callable as a TypedDict. IMO, this is one of the places in which you should take advantage of the fact that typing in Python is gradual: just declare opts as Any. Or, if you prefer, hide the mypy error with a #type: ignore comment.

If you want to go ahead with the TypedDict solution, you will get to the second problem: one of the types of your target callable is a private module member. Argh. Again, this is not easy to solve, and again IMO you should just type that as Any.

You might want to take a look at Taking the argument signature from a different function

1 Like
3

Would it make sense for libraries to provide official version of such TypedDicts? Or should that be the domain of typeshed? Or neither?

It sort of feels like replacing kwargs with some sort of formal Config dataclass.