Python AST module, missing and cannot find keyword arguments for a ast.FunctionDef?

Python Help
1

Given a trivial example like


class Widget:
    
      def doThing(self, foo:bool=True):
          pass

The doThing ast.FunctionDef.arguments has defaults at 0 holding a ast.Constant but kw_defaults == [] and kwarg is None. Is there a flag or setting I am missing so that ast.Parse’s tree will show keyword arguments and their default values? I am currently using python 3.11.2.

I used pdb and JetBrain’s debugger to inspect the AST tree but still couldn’t find what I was looking for.

I am getting the AST Module via module = ast.parse(src_file.read_text(), src_file.name, mode="exec") where src_file is a pathlib.Path object.

edit: The AST looks like this

FunctionDef(
    name='doThing',
    args=arguments(
        posonlyargs=[],
        args=[
            arg(arg='self'),
            arg(
                arg='foo',
                annotation=Name(id='bool', ctx=Load())),
            arg(arg='bar')],
        kwonlyargs=[],
        kw_defaults=[],
        defaults=[
            Constant(value=True),
            Constant(value='123')]),
    body=[
        Pass()],
    decorator_list=[])
2

kw_defaults is used for keyword only arguments.

eg:

import ast

funcdef = "def f(a, /, b, c=2, *args, d=3, **kwargs): pass"

ast_funcdef = ast.parse(funcdef).body[0]

print(ast.dump(ast_funcdef))

Output

FunctionDef(
    name='f', 
    args=arguments(
        posonlyargs=[arg(arg='a')], 
        args=[arg(arg='b'), arg(arg='c')], 
        vararg=arg(arg='args'), 
        kwonlyargs=[arg(arg='d')], 
        kw_defaults=[Constant(value=3)], 
        kwarg=arg(arg='kwargs'), 
        defaults=[Constant(value=2)]
    ), 
    body=[Pass()], 
    decorator_list=[]
)

As ‘c’ can be a positional argument it’s under ‘args’ with its default value under ‘defaults’. Keyword only arguments have their own list of defaults and are handled slightly differently. vararg and kwarg are for the * and ** style arguments.

1 Like
3

The fundamental issue here is that “keyword” is not an actual kind of parameter that Python functions and methods have. Instead, the options are as described in the documentation. Being pedantic for a second here: formally, “parameter” means the thing written in the function signature, and “argument” means the thing written when calling the function. (From what I’ve seen, even the documentation is not 100% correct on this distinction! But being careful about this makes it easier to understand everything properly and speak precisely.) So in fact, Python functions and methods don’t have “arguments” at all, let alone “keyword arguments” - they have parameters.

But more importantly: the phrasing “show keyword arguments and their default values?” makes it sound like you expect that parameters with a default value are “keyword” parameters. However, as far as Python is concerned, self and foo are the same kind of parameter - “positional-or-keyword”. We can more easily see this with the inspect standard library module:

>>> class Widget:
...     def doThing(self, foo:bool=True):
...         pass
... 
>>> import inspect
>>> inspect.signature(Widget.doThing).parameters['self'].kind
<_ParameterKind.POSITIONAL_OR_KEYWORD: 1>
>>> inspect.signature(Widget.doThing).parameters['foo'].kind
<_ParameterKind.POSITIONAL_OR_KEYWORD: 1>

Notwithstanding that self will ordinarily be passed automatically via method lookup - both arguments can be passed either positionally or by keyword:

>>> Widget.doThing(1, 2)
>>> Widget.doThing(foo=2, self=1)

No errors occur here because doThing was looked up directly in the class rather than via an object (so it is a plain function) and because both parameters have the positional-or-keyword kind. This is the “default” sort of parameter kind - the kind they will have unless you use / or * or ** syntax.

Default values for parameters have nothing to do with this. You can pass foo positionally even though it has a default value, and you can pass self by keyword even though it does not.

To give a more complete overview: “positional” and “keyword” are the two kinds of arguments. But parameters have five kinds. Here is an example, without any default values, to show them all:

>>> def example(a, /, b, *c, d, **e):
...     return a, b, c, d, e
... 
>>> inspect.signature(example).parameters['a'].kind
<_ParameterKind.POSITIONAL_ONLY: 0>
>>> inspect.signature(example).parameters['b'].kind
<_ParameterKind.POSITIONAL_OR_KEYWORD: 1>
>>> inspect.signature(example).parameters['c'].kind
<_ParameterKind.VAR_POSITIONAL: 2>
>>> inspect.signature(example).parameters['d'].kind
<_ParameterKind.KEYWORD_ONLY: 3>
>>> inspect.signature(example).parameters['e'].kind
<_ParameterKind.VAR_KEYWORD: 4>

When we call this function:
a must be passed positionally.
b can be passed either positionally or via keyword.
c is created from zero or more values that were passed positionally.
d must be passed by keyword (otherwise it would get folded into c).
e is created from zero or more values that were passed by keyword.
Additionally, we must follow the normal rules for call syntax: we put all the positional arguments first, then all the keyword arguments.

We must pass at least one positional argument (to satisfy a) and at least one keyword argument (to satisfy d). Additionally, we must specify b in exactly one way: positionally or by keyword, not both. Thus:

  • with one positional argument, it fills a. b and d must be provided by keyword, and any other keywords are collected into e. c will necessarily be an empty tuple.
  • with two or more positional arguments, they fill a and b, and any additional positional arguments are collected into c. d must be provided by keyword, and b must not be provided by keyword. Any other keyword arguments will be collected in e (it cannot have either 'b' or 'd' as keys).

Similarly, the different kinds of parameters have to appear in that order: positional-only, positional-or-keyword, var-positional, keyword-only, and var-keyword. We use /, * and ** to distinguish parameter kinds:

  • / separates positional-only parameters from positional-or-keyword parameters. (We need some kind of marker because there could be multiple of each.) However, there must be at least one positional-only parameter in order to use it (this is not strictly necessary, but Python checks it to avoid mistakes. It’s similar to how there is a pass statement and empty blocks are not allowed.)
  • There can only be one var-positional parameter (otherwise, there would be no way to know which arguments to collect for each). If there’s a parameter name prefixed with *, that marks the var-positional parameter, which therefore automatically separates the positional-or-keyword and keyword-only parameters. We can also use * by itself to separate those two groups without having a var-positional parameter; in this case, there must be at least one keyword-only parameter (this is the same kind of safeguard as above).
  • There can only be one var-keyword parameter (for the same reason). If there’s a parameter name prefixed with **, that marks the var-keyword parameter (which of course is separate from any keyword-only parameters). It must be last in the list, and we cannot use ** by itself (because there is nothing to separate).

With default values:

  • c and e cannot have default values.
  • If a has a default value, then b must also - but there is no restriction on d. There is a group of parameters that can be passed individually and positionally; within this group, the ones with default values have to come after the ones that don’t. This allows Python to determine the assignment of arguments to parameters.

Here is a straightforward conceptual model for how the function call works:

  1. *sequence and **mapping constructs are expanded to create multiple separate positional and keyword arguments.
  2. An error occurs if any positional argument appears after the first (if any) keyword argument. (Actually, this can be checked before step 1.)
  3. Positional arguments are assigned, left to right, to the combined group of positional-only and positional-or-keyword parameters. If there is a var-positional parameter (there may only be one, and it is not necessitated by keyword-only or var-keyword parameters), it receives a tuple of any excess positional arguments.
  4. Keyword arguments are assigned according to the keyword: anything that matches the name of a positional-or-keyword or keyword-only parameter gets assigned correspondingly. If the argument name doesn’t match any explicit parameter name, it goes into the dict for the unique var-keyword parameter if present, and otherwise is an error.
  5. If any positional-only, positional-or-keyword, or keyword-only parameter hasn’t been assigned yet, it takes its value from the default if there is one; otherwise an error occurs. (Assigning to var-positional and var-keyword parameters can’t fail; if there are no arguments that belong in the corresponding tuple or dict, they just end up as empty.)
1 Like
4

The distinction would be nice, but the AST uses arguments in its function definition. That probably doesn’t help things.

I feel like the defaults confusion comes partly from having kwonlyargs and kw_defaults instead of kwonlydefaults. The ordering when you dump the arguments is also a little unintuitive (it does bug me that you get kwonlyargs and kw_defaults get sandwiched between args and defaults).

5

That’s part of what I meant about the difficulty in making the distinction properly. Evidently, even the standard library API gets it wrong here :slight_smile: (The API provided by inspect, OTOH, gets it right.)

Agreed.

Conceptually, it’s just a namespace, so the order isn’t supposed to matter.
(Speaking of which: it comes across that the internal “AST node” type is reinventing the “plain namespace that just collects some attributes” wheel yet again…)

6

Perhaps conceptually, but practically once you’re calling ast.dump for some kind of debugging or other investigation it would be easier to follow if the related parts were together.

1 Like
7

Sure, that’s fair. After all, inspect uses an enumeration to describe the names of parameter kinds, and puts them in order.