Correct use of buffer protocol for dynamic array

Python Help
1

I first asked this in StackOverflow but didn’t get a satisfying answer, so I’ll give it a try here.

I have a dynamic array type in C++ that I would like to expose through the buffer protocol. The type is already exposed as a sequence in Python, but I want to build NumPy arrays with the C++ array data, and I am hoping that a buffer will be faster than the element-wise construction from a sequence.

Reading through the protocol description, I am not sure what is the correct way to do this. The problem is that, being a dynamic array, its memory may be rellocated, and the bounds of valid memory may change. But, from what I understand, the buffer protocol assumes that the exposed buffer will remain intact on the native side, at least as long as one Python buffer object is alive.

The only solution I can think of is copying the array contents into a new memory area when a buffer is requested and delete that memory after the buffer is no longer needed. But I am not sure if this complies with the buffer protocol, i.e. returning a buffer that may not represent the current state of the corresponding Python object.

The documentation on the obj field of the Py_buffer struct says:

As a special case, for temporary buffers that are wrapped by PyMemoryView_FromBuffer() or PyBuffer_FillInfo() this field is NULL. In general, exporting objects MUST NOT use this scheme.

If I did make a copy of the data on each buffer request, would it qualify as such a “temporary” buffer?

Obviously, making a copy of the data somewhat misses the point of the buffer protocol, but as I said I’m hoping NumPy array construction will be faster this way (that is, with just a memcpy copy instead of a loop over sequence items).

EDIT: Just to be clear, this is not a Python type that is internally implemented with a C++ dynamic array. Rather, this is a C++ application which offers scripting through an embedded Python interpreter and that has a Python interface for its dynamic arrays. So I cannot just forbid modifications to the object while a buffer view is alive, because in general these dynamic arrays can be directly modified by C++ code (unless I make deeper changes to the dynamic array type, and not just its Python interface).

2

One thing you could do is to add a small header to the allocated array, with a reference count. When fetching the buffer, you’d increment this reference count in the allocated block. The C++ object’s resizing code would then first check the count, and if it’s greater than 1 perform a copy of the array instead of reallocating it. Releasing the buffer would then decrement the refcount and finally deallocate it.

If you’re wanting to perform a copy, you don’t want to use the obj=null thing - that’s likely to make assumptions about the memory allocator etc. Instead probably the simplest method is to just make a bytearray/bytes object out of your data, then call its getbuffer method, let that deal with everything.

Alternatively, I know Numpy itself has a C API - you could add a method that constructs the array directly from the pointer, and orders it to perform the copy before passing it back to Python code.

3

Thanks for the suggestions, those are good ideas. I don’t think I can change the C++ dynamic array type in this case, though. But the solution of simply providing a method to construct a bytes object with a copy of the data might be the best option. The NumPy C API is a good point, too, although I’d rather have the buffer option as there are some other use cases that might benefit from this.

4

I don’t really know much about what Numpy does here, but back in the day we exposed C-style arrays via ctypes and pointers. That way it was the same memory in both Python and in C.

Just make sure if doing something like that: have one side own the raw array and be responsible for it. We had a Python wrapper that could resize it, deallocate it, etc. the wrappers would call things like calloc, free, etc. via ctypes.

That honestly was silly since in theory we could have just kept a reference and used a static allocation in our case, but alas I no longer work on the project… and didn’t write the original memory allocation stuff. :wink:

If going far beyond C arrays, you’d probably wind up needing a copy of some sort to bounce back and forth.

5

Thank you Charles, yes, you’re right, ctypes is another route. The problem here is I cannot have Python “own” the array (it can be modified by the C++ code of the application on a function call, without Python knowing it), and I cannot modify the array data structure either (it is “third party” code, let’s say). So I think something like this could work, and it would be simple and efficient, I’d have to be “careful” when I use it (i.e. get the pointer, or memoryview or whatever, use it and discard it, without calling any function that could modify the array in between, and assuming/hoping there are no other threads tripping you up). So that’s why I’m leaning towards a copy.