Generalized ptr_wise_atomic_{memmove,memcpy} as part of public API

Copied from Generalized ptr_wise_atomic_{memmove,memcpy} as part of public API · Issue #106 · capi-workgroup/decisions

Inspired by cpython/Objects/listobject.c at main · kumaraditya303/cpython · GitHub, I propose to extend ptr_wise_atomic_memmove beyond PyListObjects. Since the proposed function ought to accept any Python C object type convertible from PyObject, here’s a sketch:

void
PyMem_PtrWiseAtomicMemmove(PyObject *a, PyObject **dest, PyObject **src, Py_ssize_t n)
{
#ifndef Py_GIL_DISABLED
    memmove(dest, src, n * sizeof(PyObject *));
#else
    // List built-in types with internal locks. May be more
    if (PyDict_Check(a) || PyList_Check(a) || PyDict_Check(a))
        _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(a);
    if (_Py_IsOwnedByCurrentThread((PyObject *)a) && !_PyObject_GC_IS_SHARED(a)) {
        // No other threads can read this list concurrently
        memmove(dest, src, n * sizeof(PyObject *));
        return;
    }
    if (dest < src) {
        for (Py_ssize_t i = 0; i != n; i++) {
            _Py_atomic_store_ptr_release(&dest[i], src[i]);
        }
    }
    else {
        // copy backwards to avoid overwriting src before it's read
        for (Py_ssize_t i = n; i != 0; i--) {
            _Py_atomic_store_ptr_release(&dest[i - 1], src[i - 1]);
        }
    }
#endif
}

void
PyMem_PtrWiseAtomicMemcpy(PyObject *a, PyObject **dest, PyObject **src, Py_ssize_t n)
{
#ifndef Py_GIL_DISABLED
    memcpy(dest, src, n * sizeof(PyObject *));
#else
    // List built-in types with internal locks. May be more
    if (PyDict_Check(a) || PyList_Check(a) || PyDict_Check(a))
        _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(a);
    if (_Py_IsOwnedByCurrentThread((PyObject *)a) && !_PyObject_GC_IS_SHARED(a)) {
        // No other threads can read this list concurrently
        memcpy(dest, src, n * sizeof(PyObject *));
        return;
    }
    if (dest < src) {
        for (Py_ssize_t i = 0; i != n; i++) {
            _Py_atomic_store_ptr_release(&dest[i], src[i]);
        }
    }
    else {
        // copy backwards to avoid overwriting src before it's read
        for (Py_ssize_t i = n; i != 0; i--) {
            _Py_atomic_store_ptr_release(&dest[i - 1], src[i - 1]);
        }
    }
#endif
}

This function seems to be very specific. I’m not sure that it’s a good idea to propose a general API for that.

Which projects are supposed to use such function? For what purpose?

1 Like

This function seems to be very specific. I’m not sure that it’s a good idea to propose a general API for that.

Which projects are supposed to use such function? For what purpose?

Maybe a public API is too broad of an proposal…
Perhaps prefix it with PyUnstable_ for now. As for the purpose, I took inspiration from gh-149816: fix thread safety of deletion of list slice by kumaraditya303 · Pull Request #149936 · python/cpython · GitHub, where I wanted to improve thread safety of memmoves and memsets within the free-threaded interpreter