Signatures, a call to action

Sorry for the thread exhumation, I only just now saw this. What follows is a little historical context.

The initial motivation for Argument Clinic was to make it easy to add Python-visible signatures to a function. I figured, adding them by hand would take forever and be error-prone. And they’d go out of date, as core devs modified the argument parsing for a function but forgot (yet again) to update the “signature” to match. Making Argument Clinic generate the argument-parsing code itself seemed like a win; the signatures it generated were guaranteed to be accurate and up-to-date. (Argument Clinic hides the signature at the end of the docstring, in a place inspect.signature can find it, which meant built-in functions would finally have signatures, and pydoc signature information would be automatically up-to-date.)

Also, automatically generating the argument parsing code seemed like a boon to development and maintenance. It’d make it easier to add new extension functions in the future. I figured Clinic would initially be a puzzle to new core devs; instead of having to figure out how to use the somewhat-awkward Python argument-parsing functions, they’d have to figure out how to use the also-somewhat-awkward Argument Clinic. But they’d have plenty of examples, letting them copy-paste-modify their way to early success. And of course it’s a boon of convenience for the core dev programmer, having their function prototype magically generated for them, with its arguments magically appearing in the right types at runtime.

But that’s why Clinic was only designed to accommodate functions with signatures you could express in Python–generally, functions already using PyArg_ParseTuple or PyArgParseTupleAndKeywords. (Maybe even the ancient PyArg_Parse.) It really was all about smuggling the signatures for extension functions into the runtime so inspect.signature could read them. And it just wasn’t designed to permit expressing functions with signatures you couldn’t write in Python.

There are loads of functions in Python that have signatures you can’t really express in Python. IIRC the most common of these are functions with a parameter whose default value can’t be represented in Python; internally the C variable the argument is slotted into is pre-initialized with NULL, and there’s no way in Python to pass in a C NULL. But there are other functions that play games, e.g. using other internal default values that couldn’t be represented in Python, or by counting their parameters and only allowing discrete numbers of values. By the time I shipped Clinic I’d tried to support the latter with “argument groups”, which weren’t directly expressible in Python, but could be rendered in a readable fashion by pydoc using square brackets. I dimly recall Clinic also handles functions with these internal NULL defaults by similarly marking them as optional using square brackets. These aren’t proper Python signatures, though; IIRC I had to teach inspect.signature to ignore those square brackets. (The only way a inspect.Parameter object can represent the state “this parameter is optional” is by setting the default attribute to the parameter’s default value.)

After Argument Clinic shipped, the community responded by trying to force a lot of square pegs into round holes, adding Argument Clinic support to functions that weren’t really good candidates for it. Initially it seemed like a weird game, like contributors were trying to figure out ways to trick Clinic into generating code that parsed a reasonable simulacrum of the original function’s behavior, if you squinted your eyes just right. I found this low-key irritating, but I didn’t have the energy to fight about it. And I couldn’t really get mad that they seemed to want to make CPython better, make it easier to maintain, etc.

Eventually the effort to use Argument Clinic for as many functions as possible became a performance win, because it let us quickly innovate in faster argument parsing. You could write a new argument passing convention and only have to modify one spot–once you updated the Argument Clinic code generator to understand your new calling convention, you got the rest of the entire CPython source tree for free. I believe Victor’s “fast call” implementation was far easier to implement because of the work the community did in converting a significant portion of CPython extension functions to Clinic.

I’d say I’ve given any thought to how to extend Argument Clinic to handle more types of Python functions. But it’s also fair to say I’ve done nothing about it in a very long time, and I have no plans to return to the project. Finally, it’s not accurate to say that nothing has been done about it for more than a decade, as Argument Clinic didn’t land in the Python repo until October 2013.

9 Likes