Using Private API in a C Extension

Suppose I am writing an extension that needs to print the floating-point number cache pool (freelist).
To do this, I need to access the interpreter state reference, which requires the definition of struct _is, namely PyInterpreterState. However, this belongs to the private API and requires pycore_interop.h.
Generally speaking, when writing a C program that embeds Python, or, a Python C extension, you don’t need to use these private APIs, but in my case, is this usage correct? To access struct _is, I had to #define Py_BUILD_CORE.

#define Py_BUILD_CORE
#include "Python.h"
#include "pytypedefs.h"
#include "internal/pycore_interp.h"
#include <stdio.h>


void print_list(PyFloatObject *list, int size) {
	if (list == NULL) {
		printf("Free List is NULL\n");
	} else {
		printf("%d elements in Free List\n", size);
		puts("The 1st one is:");
		PyObject_Print((PyObject *)&list[0], stdout, Py_PRINT_RAW);

	}
	while (list->ob_base.ob_type) {
		list = list->ob_base.ob_type;
		PyObject_Print((PyObject *)list, stdout, Py_PRINT_RAW);
		puts("");
		printf("(%f)\n", list->ob_fval);
	}
}


int main() {

	Py_Initialize();

	// Creating some random dummy floats
	//
	PyObject *fobj_1 = PyFloat_FromDouble(233.233);
	PyObject *fobj_2 = PyFloat_FromDouble(233.234);
	PyObject *fobj_3 = PyNumber_Add(fobj_1, fobj_2);

	Py_DECREF(fobj_1);
	Py_DECREF(fobj_2);
	Py_DECREF(fobj_3);

	PyThreadState *ts = PyThreadState_Get();
	PyInterpreterState *is = ts->interp;
	printf("Free List address: %p\n", is->float_state.free_list);
	print_list(is->float_state.free_list, is->float_state.numfree);
	Py_Finalize();
	return 0;
}

My question is, is using private APIs not recommended under any circumstances? How should they be used correctly, and what are the best practices?

You can use private APIs, but they might change at any time for no reason :smile:

2 Likes

Yeah, the way to use them correctly is to make sure you know who is using your code, make sure you can provide them updates quicker than they want/need to adopt a new CPython release, and generally take responsibility for the fact that you did it (rather than complaining to the core team when they change).

Otherwise, “we’re all consenting adults here,”[1] so go ahead and do whatever you need to get the job done.


  1. One of the guiding principles of Python, though I can’t find a linkable reference easily enough to post it. ↩︎

1 Like

In particular, the layout of the internal structs can (and does) change in patch releases. You’ll need separate builds for Python 3.12.3 and 3.12.4, for example.
Managing the updates will be up to you; wheels & PyPI generally only use 3.x for the version.

But for debugging or profiling locally, you’re fine! Happy hacking.

1 Like