Is multiprocessing broken on macOS in Python 3.8?

I have code that used to work prior to 3.8

it basically does:

multiprocessing.Process(
    target=function, args=args, kwargs=kwargs
)

the tests hitting this code pass when run on CI (a linux system), but fail when run locally on my macbook with:

_pickle.PicklingError: Can't pickle <function myfunc at 0x10c86b1f0>: it's not the same object as myproj.tests.integration.mymodule.test_utils.myfunc

I believe this must be due to the change from default “fork” mode to “spawn” when running under macOS, changed in Python 3.8:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
https://chrissardegna.com/blog/multiprocessing-changes-python-3-8/

But if you can’t pickle functions, and using spawn mode somehow means the target function has to be pickled while fork doesn’t… does this mean that multiprocessing.Process is not usable on macOS?

That can’t be right, what am I missing? Is there a workaround?

1 Like

Following up…

The pickling problem seems to be specifically with decorated functions, even when using functools.wraps

Other functions can be pickled and used with multiprocessing.