Complex multiprocessing bugs


@ambv told me about a multiprocessing issue and I told him that I tried but failed to fix a bunch of multiprocessing issues. I dislike implicit resource management. In multiprocessing, it’s not just a few MB of memory, but whole processes which have open files, shared memory, and many resources attached. I proposed some changes to make the resource management more explicit, but they got rejected. Well, at least I managed to add ResourceWarning to Pool (but not for all cases) and I added SimpleQueue.close() method :slight_smile:

In June 2022, I sent to python-dev: Looking for volunteers to maintain the multiprocessing and concurrent.futures modules.


Complex bugs (still exist in 2022, even if their issues got closed):

I just share my findings in case if someone else is volunteer to dig into these issues. Sadly, I’m not a multiprocessing expert and I cannot help much :frowning: It’s really hard to “get it right” and handle all corner cases (and avoid race conditions) when multiple threads, processes and signals are involved…


I’m not sure I understand what this posting is about. I don’t doubt that there are complex bugs in multiprocessing, given that multiprocessing itself is quite complex and reliant on low-level platform primitives. However, you’re also mentioning some ResourceWarning that don’t look related at all. What is the goal here?

I just wanted to share my notes about multiprocessing.

Honestly, that was a few years ago, and I forgot the whole context. But if I recall correctly, my problem was that in multiprocessing tests, there are many “resources” like processes which were not cleared explicitly, and so were still running “in the background”. My concern was that multiprocessing doesn’t provide any simple way to see which resources are “leaked”: not cleared explicitly.

Multiprocessing uses “finalizers” which are called when an object is deleted, but sometimes objects can be kept alive longer than expected because of reference cycles. For example, an exception can keep many objects alive if the exception stays alive, since it contains a traceback which contains references to local variables. But there are many things creating reference cycles.

Maybe calling them “enhancements” would be more fair, since multiprocessing just works in most cases :wink: