Complex multiprocessing bugs

[vstinner (Victor Stinner)]

Hi,

@ambv told me about a multiprocessing issue and I told him that I tried but failed to fix a bunch of multiprocessing issues. I dislike implicit resource management. In multiprocessing, it’s not just a few MB of memory, but whole processes which have open files, shared memory, and many resources attached. I proposed some changes to make the resource management more explicit, but they got rejected. Well, at least I managed to add ResourceWarning to Pool (but not for all cases) and I added SimpleQueue.close() method

In June 2022, I sent to python-dev: Looking for volunteers to maintain the multiprocessing and concurrent.futures modules.

ResourceWarning:

• python-dev thread: Usage of the multiprocessing API and object lifetime
• Pool: commit. TODO: consider emitting a warning if .join() hasn’t been called.
• TODO: Queue. Antoine Pitrou: “I don’t think a ResourceWarning is appropriate here.”
• DONE: Add SimpleQueue.close(): commit

Complex bugs (still exist in 2022, even if their issues got closed):

• Pool.join() always takes at least 100 ms
• Windows: DupHandle.detach() race condition on DuplicateHandle(DUPLICATE_CLOSE_SOURCE)

I just share my findings in case if someone else is volunteer to dig into these issues. Sadly, I’m not a multiprocessing expert and I cannot help much It’s really hard to “get it right” and handle all corner cases (and avoid race conditions) when multiple threads, processes and signals are involved…

[pitrou (Antoine Pitrou)]

I’m not sure I understand what this posting is about. I don’t doubt that there are complex bugs in multiprocessing, given that multiprocessing itself is quite complex and reliant on low-level platform primitives. However, you’re also mentioning some ResourceWarning that don’t look related at all. What is the goal here?

[vstinner (Victor Stinner)]

@pitrou

What is the goal here?

I just wanted to share my notes about multiprocessing.

@pitrou

you’re also mentioning some ResourceWarning that don’t look related at all

Honestly, that was a few years ago, and I forgot the whole context. But if I recall correctly, my problem was that in multiprocessing tests, there are many “resources” like processes which were not cleared explicitly, and so were still running “in the background”. My concern was that multiprocessing doesn’t provide any simple way to see which resources are “leaked”: not cleared explicitly.

Multiprocessing uses “finalizers” which are called when an object is deleted, but sometimes objects can be kept alive longer than expected because of reference cycles. For example, an exception can keep many objects alive if the exception stays alive, since it contains a traceback which contains references to local variables. But there are many things creating reference cycles.

@pitrou

I don’t doubt that there are complex bugs in multiprocessing, given that multiprocessing itself is quite complex and reliant on low-level platform primitives.

Maybe calling them “enhancements” would be more fair, since multiprocessing just works in most cases