@ambv told me about a multiprocessing issue and I told him that I tried but failed to fix a bunch of multiprocessing issues. I dislike implicit resource management. In multiprocessing, it’s not just a few MB of memory, but whole processes which have open files, shared memory, and many resources attached. I proposed some changes to make the resource management more explicit, but they got rejected. Well, at least I managed to add ResourceWarning to Pool (but not for all cases) and I added SimpleQueue.close() method
In June 2022, I sent to python-dev: Looking for volunteers to maintain the multiprocessing and concurrent.futures modules.
- python-dev thread: Usage of the multiprocessing API and object lifetime
- Pool: commit. TODO: consider emitting a warning if .join() hasn’t been called.
- TODO: Queue. Antoine Pitrou: “I don’t think a ResourceWarning is appropriate here.”
- DONE: Add SimpleQueue.close(): commit
Complex bugs (still exist in 2022, even if their issues got closed):
- Pool.join() always takes at least 100 ms
- Windows: DupHandle.detach() race condition on DuplicateHandle(DUPLICATE_CLOSE_SOURCE)
I just share my findings in case if someone else is volunteer to dig into these issues. Sadly, I’m not a multiprocessing expert and I cannot help much It’s really hard to “get it right” and handle all corner cases (and avoid race conditions) when multiple threads, processes and signals are involved…