If anyone wants to take this on I’m happy to also provide support alongside @encukou.
Once upon a time I started drafting a PEP on this, but eventually it became too messy (and I ran out of town) trying to deal with semantics around memoryview, since they allow you to create many slices and sub-slices in Python.
I’m interested in taking this on after PEP 688. I think I’ll drop any attempt to detect mutability in PEP 688, so that the PEP only provides the basis for identifying buffers in Python. Then later we can use Petr’s proposed change as a vehicle for richer buffer typing (including mutability, and potentially also other concerns like contiguousness).
Just to add a missing piece, this is my understanding of how Petr’s proposal would fix the problem identified in Alex’s post: pyo3 would inspect the flags field to figure out whether the buffer is safe for shared use, and generate different Rust bindings based on the answer. For example, this would allow it to produce &[u8] Rust buffers from a bytes object. Is that correct?
I’m not sure how pyo3 works, actually
It could also have 3 fallible operations – asking for an immutable, exclusive, or unsafe raw buffer – for any PyObject, and fail if the requested one isn’t available.
Well, bytes is easy – it’s built-in so there can be a special case for it. This proposal is for third-party libraries, which need a way to announce they’re immutable. Currently you’d need to convert those to bytes (with a memory copy) to get a safe immutable buffer.
I think that’s exactly the API you’d end up with, and it’d totally solve our problems! Currently you only have a choice of the raw buffer (which you can either treat as “unsafe” or “safe, but totally unergonomic”): PyBuffer in pyo3::buffer - Rust