(pssst) Let's treat all API in public headers as public

[pitrou (Antoine Pitrou)]

@malemburg

You can already see this happening in the data science space where more and more tools are using Arrow for data storage and Rust for all the processing work. Python is only used as a high level glue language to fit things together.

That is true, but it also doesn’t seem very related to the Python C API, is it? Most of NumPy (excluding tests) is written in C with some Cython, just like most of PyArrow (excluding tests) is written in C++ with some Cython.

I’ve worked on Python scientific packages for a number of years (most of all Numba, Dask, PyArrow), and the cases where the Python C API is important for performance are rather rare.

The main item is quick type testing and boxing and unboxing of Python objects. And, actually, it may be useful to have C APIs to do that on several elements at a time. For example, unboxing a C array of int32_t from a Python sequence of int objects, and being able to do it fast.

[vstinner (Victor Stinner)]

@pitrou

The main item is quick type testing and boxing and unboxing of Python objects. And, actually, it may be useful to have C APIs to do that on several elements at a time. For example, unboxing a C array of int32_t from a Python sequence of int objects, and being able to do it fast.

A first step towards such use case is a first API to get a read-only view of an array of object: PyObject_AsObjectArray(), C API: Add PyObject_AsObjectArray() function: get tuple/list items as PyObject** array · Issue #106593 · python/cpython · GitHub I also propose there a generic concept of “resource” to have an explicit lifetime: generic API to “release a resource” (in short, it’s just a callback function hidden through a public API).

And yeah, it would be nice to have a similar API for integer types. Maybe Py_buffer somehow already fit into this use case? I’m not sure.

[pitrou (Antoine Pitrou)]

@vstinner

And yeah, it would be nice to have a similar API for integer types. Maybe Py_buffer somehow already fit into this use case? I’m not sure.

It doesn’t. Py_buffer gives you a view over existing memory, it doesn’t convert Python objects to raw arrays.

[vstinner (Victor Stinner)]

@pitrou

It doesn’t. Py_buffer gives you a view over existing memory, it doesn’t convert Python objects to raw arrays.

Oh sorry, I understood that you wanted to export an array of integers as… an array of integers, of the same size Here you’re asking to unbox a sequence of Python objects as integers, ok.

[vstinner (Victor Stinner)]

I documented these private C API removals in What’s New In Python 3.13:

• Remove many APIs (functions, macros, variables) with names prefixed by
  _Py or _PY (considered as private API). If your project is affected
  by one of these removals and you consider that the removed API should remain
  available, please open a new issue to request a public C API and
  add cc @vstinner to the issue to notify Victor Stinner.
  (Contributed by Victor Stinner in :gh:`106320`.)

Maybe I will regret later suggesting people to ping me