File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get raise self._value File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks put(task) File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes header = struct.pack("!i", n) struct.error: 'i' format requires -2147483648 <= number <= 2147483647
I ran a python 3.6 program that uses multiprocessing and I got this error.
As I know, it’s a problem of pickle size limit during multiprocessing.
I am also aware of pickle protocol 4 which lets you pass bigger data.
but the problem is that the program uses multiprocessing inside a Cython written compiled SO file module andI can’t change the SO file since I don’t have the source code for the pyx.
It’s also not possible to shrink the input size before it is used in the SO file module function cause I don’t know what exactly goes on inside the SO file module and it gets big while running the SO module function.
The only way I can think of is to fix the python multiprocessing package to use pickle protocol 4 by default.
If it’s possible could someone please tell me how? Or if there is another possible way to fix txme