Can Py_buffer format be const qualified?

Core Development C API
1

The char *format of a Py_buffer is often set to a string literal such as "B" or "d". For example, see get_native_fmtstr() in Objects/memoryobject.c, which is used to set view->format. These strings cannot be modified in memory.

Since C++11, string literals are not convertible or assignable to non-const char*, which makes setting format unpleasant for those programming in C++.

Would it be possible to change to const char *format?

By the way, PEP 3118 had the const qualifier…

2

I think that’s an API break, in the sense that C++ extensions that use the Py_buffer struct will probably see new compiler errors.

For example, this code in Cython, which is used by e.g. Pandas:

Did you try actually making the change to CPython and then building a few downstream libraries that use Py_buffer from C/C++ or Rust to see if they break? I see, on a manually patched CPython:

$ g++ -std=c++17 -c mvtest.cpp -I/usr/include/python3.13        # pristine: OK
$ g++ -std=c++17 -c mvtest.cpp -Ipyinc-patched                  # const char *format:
mvtest.cpp:8429:36: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]
 8429 |     __pyx_t_4 = __pyx_v_self->view.format;
mvtest.cpp:24507:67: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]
24507 |       array_obj = __pyx_array_new(shape_tuple, sizeof_dtype, buf->format, mode, NULL);

Py_buffer is in the limited API which allows making changes like that and this isn’t an ABI break, but it does lead to real downstream pain if you make a change to add a const qualifier. What does it buy CPython and downstream consumers of the buffer protocol API?

3

In C, if a program attempts to modify a string literal (through a char*), the behavior is undefined. So, I was hoping to buy some safety.

For C++, in addition to safety, I was hoping to avoid format = const_cast<char*>("d");, which is kinda a red flag in a code review.

In C, using gcc, the following assignment works without warnings with -Wall -Wpedantic:

    char *myformat = "d";
    const char *format = myformat;

so I was hoping the change I suggested wouldn’t be too disruptive downstream. But, I guess we can’t have nice things. :winking_face_with_tongue:

2 Likes
4

Hmm you’re right. It may be the case that just Cython runs into this. Maybe open a Cython issue? Of course that doesn’t fix old generated Cython code or old Cython versions, but if, say, VTK or other C++ libraries already either cast away or just assume the pointer is const it may not be so bad. Either way seems worth a Cython issue.

5

Cython is relatively easily changed here (in the sense that it is maintained and we do know how to fix this stuff). So I don’t think that you shouldn’t propose this change because Cython uses the current definition. But I do think it might be an indicator that other people do too.

The issue in the specific C code you show is that it’s going through a temp/function argument that is typed as char*. Obviously we could type that temp/function argument as const char* instead but that would break in current versions of Python when we tried to assign it back to the unqualified char* attribute. So it’d need either explicit casting or some version-specific typedef.

So I don’t think there’s specifically a Cython bug that’s worth reporting here - we’re generating code that works with the current definition. If the definition changed then we’d update to make it work. But right now there’s nothing to fix.

My personal opinion is that it’d have been better as const char* but probably isn’t worth the disruptive change right now.

6

PyMethodDef.ml_name and PyMethod.ml_doc had const added in Python
v3.7. It just happened - there wasn’t even a mention in the changelog.

It allowed me to remove some ugly casts.

7

Thanks, Phil. I also found the following changes in Python 3.7:

The result of PyUnicode_AsUTF8AndSize() and PyUnicode_AsUTF8() is now of
type const char * rather of char *.

and

The fields name and doc of structures PyMemberDef, PyGetSetDef,
PyStructSequence_Field, PyStructSequence_Desc, and wrapperbase are now of
type const char * rather of char *.

and in Python 3.8:

:c:func:PyExceptionClass_Name will now return const char * instead of
char *.

and

c:expr:PyDoc_VAR(name) and :c:expr:PyDoc_STRVAR(name,str) now create
static const char name[] instead of static char name[].

So, there does seem to be some interest in const-qualifying a char * where appropriate.

I tried building NumPy with this suggested change and got only two warnings–exactly the two warnings Nathan found above–since NumPy uses Cython for random/_generator.pyx.c. Other than having to update Cython, NumPy appears to need no other changes.

Nanobind does not need any changes. It casts a local variable having type const char* to char* when assigning to format. This will continue to be necessary in order to support older Python versions. The cast is harmless if format becomes const char* format because the compiler allows the assignment (effectively casting it back again to const char*). Eventually, when older Python versions are no longer supported by nanobind, the cast can be removed.

I recognize, of course, that other downstream consumers of the buffer protocol API may be affected, and their developers may not all be as gracious as Da Woods. It’s a judgement call whether to accept disruption in the short term. Uneasy lies the head that wears a crown.

I submitted: Add `const` qualifier to `format` member of `Py_buffer` · Issue #155819 · python/cpython · GitHub