Hi python-ideas,
I’m working on LibrePy / WriterAgent, a FOSS project bringing native scientific Python (=PY()) and NumPy into LibreOffice. Because of binary compatibility constraints in the runtime, the host environment assumes only standard Python and must offload heavy calculations to an isolated worker process running NumPy from a specified venv.
While optimizing data transfer between the host and worker, I ran into a performance bottleneck in how Python serializes 2D grid/table data, and I’d like to propose a stdlib enhancement to solve it.
When sending a spreadsheet range (like a 20,000 × 5 grid of numbers, labels, and blank cells) across a process boundary, standard library Python typically represents this as a nested list: list[list[float | int | str | None]].
-
Standard
pickle.dumps()is slow on 2D lists: Serializing a nested list forcespickleto handle 100,000 individual heap objects and pointers, taking ~12 ms on dump and load (on my modern laptop). -
Pickle 5 binary buffers are near-instant: If the data is packed into a flat
float64binary buffer (e.g.,array.array('d')), Pickle Protocol 5 can transmit it in under 0.015 ms, and the worker process can materialize it in NumPy instantly usingnp.frombufferin 0.002 ms. -
The bottleneck is the packing loop: Converting a
list[list]into a flat binary array using pure Python on the host takes ~8.3 ms—accounting for 99% of total transfer time. Even with aggressive micro-optimizations (bound methods, direct type checks, sparse string dictionaries), bytecode execution over nested loops remains the wall. -
Cython proof of concept: Moving just that flattening loop into Cython dropped the time from 8.3 ms down to ~3.0 ms.
Could CPython provide a native C-speed helper—either inside _pickle or as a constructor method on array (e.g., array.array.from_grid())—to flatten 2D rectangular sequences directly into contiguous memory buffers?
This would allow any application passing tabular or matrix data between Python processes to achieve C-speed serialization using only the standard library.
I already have working version in LibrePy, but writing a custom 2D-to-buffer codec in userland involves many subtle complexities: handling jagged row validation, mapping missing values (None) and NaN, detecting column types, and maintaining a sparse index table for non-numeric strings.
The code is pure Python plus an optional Cython accelerator, however, it is complicated now with all the normal cases and edge cases. Implementing this natively inside CPython’s C layer would formalize this “useful” but tricky feature, and make a standard 2D extraction engine fast without needing any C/Cython.
The design doc is here (update: fixed link).
LibrePy / WriterAgent repo is here.
I’d appreciate any feedback. Thank you for reading,
-Keith