Propose that Argument Clinic only uses public Python APIs to make it easier to use outside CPython

Python Help
1

Argument Clinic is an internal tool in CPython that handle argument parsing and calling convention for builtin modules.
Even “Its use is not supported for files outside CPython” (Argument Clinic How-To — Python 3.10.5 documentation), I found it very helpful when working with third-party modules outside CPython, e.g. code-data-share-for-python/_cdsmodule.c.h at main · alibaba/code-data-share-for-python · GitHub. This file works perfectly for 3.8 <= python <= 3.10, while the initial setup is also quite simple.

While Argument Clinic itself is changing with CPython, it’s fine for CPython as the generated code can always access the latest codebase of CPython. But for outside users, when Argument Clinic uses symbols in a new version of CPython, these change needs additional effort, which could be saved if AC only uses public Python APIs, by putting those new public APIs in GitHub - python/pythoncapi-compat: The pythoncapi-compat project can be used to write a C extension supporting a wide range of Python versions with a single code base..

cc @vstinner

PS: I’m not sure how this works for packages that support PyPy, or it’s totally irrevelant?

1 Like
2

See also Victor’s comment in gh-85283:

Argument Clinic could be modified to have an option to only use C API of the limited C API. Cython is working on a similar option (restraint emitted code to the limited C API).

1 Like
3

IMO we should first modify AC to avoid the internal C API and only use the public C API. Targeting the limited C API would be even better.

Recently, I added _PyCFunction_CAST() to fix a C++ compiler warning, but right now the function is not part of the public C API. AC should either avoid it or we should make this macro public: Consider adding `_PyCFunction_CAST` · Issue #43 · python/pythoncapi-compat · GitHub

I hesitated to make the macro public since… it’s a macro and I am trying to move the Python C API avoid from macro, I mean, I wrote PEP 670 with Erlend :smiley: PEP 670 – Convert macros to functions in the Python C API | peps.python.org But well, maybe such macro is better than the previous cast which emitted a C++ compiler warning (“old-style cast”).

4

No, we should not. AC is internal and CPython-only, so it can use CPython-specific API.
If AC is needed outside CPython, IMO it should be forked (and probably improved), like pegen.

2 Likes
5

It can be modified to have two build modes: one using the internal API, one using public or limited API (or we can have a 3rd mode for the limited API). CPython itself would benefit, I would like to be able to build a stdlib extension module with the limited API and use AC (since AC is great for docstring and inspect.signature()). See: https://github.com/python/cpython/issues/85283

Maintaining a fork outside Python sounds counter productive to me.

6

So, seems like this sort of went in the other direction, huh?

With the Argument Clinic implementation in current CPython’s Tools/clinic/, even if I call clinic.py --limited it will emit includes of internal headers, which unless I’m missing something makes its code unusable outside of the CPython tree?

(Partial results of a clinic.py --limited run just now, using the main branch of GitHub - python/cpython: The Python programming language):

/*[clinic input]
preserve
[clinic start generated code]*/

#include "pycore_long.h"          // _PyLong_UnsignedInt_Converter()

PyDoc_STRVAR(search_create_index__doc__,
"create_index($module, files, /, n_threads, search_dir, fragments_dir,\n"
"             html_dir, private_dir, stopwords_path)\n"
// ...
7

The limited C API is not fully supported. Some kind of arguments still use the internal C API. It’s a work in progress. Which arguments use the internal C API for you?

8

This option is experimental. It helps us to see what happens when try to use the limited C API globally, what should be done to make it usable. If some private C API is often used, we should consider adding the corresponding public C API. If some private C API is rarely used, we can consider making Argument Clinic not using it with the limited C API.

If you want to use Argument Clinic in your project, consider writing custom converters.

9

Well, integers, for one! That pycore_long.h header being included is located at /usr/include/python3.12/internal/pycore_long.h on my system, so I was taking it to be an internal header.