Problem with pickle in multiprocessing

File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
put(task)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

I ran a python 3.6 program that uses multiprocessing and I got this error.

As I know, it’s a problem of pickle size limit during multiprocessing.

I am also aware of pickle protocol 4 which lets you pass bigger data.

but the problem is that the program uses multiprocessing inside a Cython written compiled SO file module andI can’t change the SO file since I don’t have the source code for the pyx.

It’s also not possible to shrink the input size before it is used in the SO file module function cause I don’t know what exactly goes on inside the SO file module and it gets big while running the SO module function.

The only way I can think of is to fix the python multiprocessing package to use pickle protocol 4 by default.

If it’s possible could someone please tell me how? Or if there is another possible way to fix txme

Try setting pickle.DEFAULT_PROTOCOL. The default has been 4 since Python 3.8.