I’m attempting to implement a Jupyter notebook like kernel with websockets
9.1 and multiprocessing, but have the problem that the asyncio loop in websockets doesn’t exit:
The parts are best illustrated like this:
main.py
launches 3 sub processes using multiprocessing:
- the notebook server (
multiprocessing.Process
) - the file service (
multiprocessing.Process
) - the socket server which uses
websockets
andasyncio
- A
client
which communicates to the socket server.
The test runs setup()
and all of the tests without any problem, but during teardown()
the socket server just hangs.
This is what I see:
setup complete
notebook server ready
file service ready
socket server ready
# irrelevant output from the test suite redacted. All tests pass.
teardown complete
The command prompt just hangs here. I can see that all sub-processes have stopped except one.
When I then hit CTRL+C I get this:
Process socket server:
Traceback (most recent call last):
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "C:\Data\github\websocket_trials\kernel\servers.py", line 225, in run
asyncio.get_event_loop().run_forever()
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\asyncio\windows_events.py", line 316, in run_forever
super().run_forever()
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\asyncio\base_events.py", line 596, in run_forever
self._run_once()
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\asyncio\base_events.py", line 1854, in _run_once
event_list = self._selector.select(timeout)
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\asyncio\windows_events.py", line 434, in select
self._poll(timeout)
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\asyncio\windows_events.py", line 783, in _poll
status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)
KeyboardInterrupt
Process launcher:
Traceback (most recent call last):
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "C:\Data\github\websocket_trials\kernel\launch.py", line 48, in run
if p.is_alive():
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\process.py", line 165, in is_alive
returncode = self._popen.poll()
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\popen_spawn_win32.py", line 118, in poll
return self.wait(timeout=0)
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\popen_spawn_win32.py", line 108, in wait
res = _winapi.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\util.py", line 357, in _exit_function
p.join()
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\process.py", line 149, in join
res = self._popen.wait(timeout)
File "C:\Users\madsenbj\Anaconda3\envs\py396\lib\multiprocessing\popen_spawn_win32.py", line 108, in wait
res = _winapi.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt
I struggle to interpret the traceback. As I see both lib\asyncio
and lib\multiprocessing
so I’m not sure which is the culprit.
When I use cmd.exe the suite runs, but the process hangs like this:
When I use nosetests in pycharm the test suite runs to the end and closes all processes.
Do you have any suggestions to what could be wrong?
Update
My suspicion is that this is an indeterminacy race bug between multiprocessing and asyncio.
How can I verify that it isn’t asyncio\windows_events.py that has a lock at the same time as multiprocessing\popen_spawn_win32.py holds a lock and both are waiting for one another?