By Asapanna Rakesh via Discussions on Python.org at 21Apr2022 00:20:
I have large number of i/o bound tasks. I am using multithreading to
run them on 2 core machine. How will my performance be affected if I
spawn a blocking process (subprocess in python) in my task’s code?
If the process is I/O bound, not at all. If your Thread is blocked
waiting for the subprocess, well that Thread will be stopped.
Assume that I am running os.cpu_count() number of threads – which I
assume is maximum possible threads.
No.
-
CPython uses a global interpreter lock (known as the “GIL”) around
all the pure Python stuff. So for CPU bound pure Python code, only one
CPU will be used. The GIL lets the interpreter assume that it is the
sole user of the interpreter’s internal data structures, meaning no
other locking is needed, which avoids deadlocks and complication. -
However, almost all blocking operations such as file I/O and waiting
for a subprocess release the GIL while waiting, allowing another
Thread to use the CPU for pure Python code. So you can run more or less
as many I/O bound Threads as you like, far in excess of the CPU count. -
Plenty of modules with C extensions, including parts of CPython which
use CPU bound C libraries such as hashlib, also release the GIL while
the C component runs. So the C component gets to use a CPU, flat out if
necessary, and the pure Python code in another Thread can continue using
a core.
So:
- if you/re I/O bound, run lots of Threads
- if you’re CPU bound pure Python, you can run lots of Threads too but
only one CPU will get used because the interpreter only runs one
thread of pure Python at a time - if your CPU bound stuff uses a C extension which releases the GIL, you
can run multiple threads and they will make use of more CPUs; on even
vaguely modern CPUs this is complicated (inside the CPU) and the
os.cpu_count() is probably not a good guide to what limits you might
impose; you can always run “quite a lot” of Threads and let the OS
sort out who gets the available CPU - if you need to use multiple CPUs with pure Python CPU bound code, you
might then reach for the multiprocessing module, which presents a
Thread-like API but runs the code in separate processors (and thus
separate Python interpreters) - or of course, you can just run your programme multiple times at once
from a script, working on separate data sets
Personally, I use Threads a fair bit and I have yet to want the
multiprocessing module. If nothing else, multiprocessing makes sharing
data somewhat more cumbersome - your are talking to a separate process
and so data has to be sent back and forth. There are others who almost
always reach for multiprocessing instead of threading because then they
don’t have to worry about deadlocks to the same degree. I remain
unconvinced, personally.
Cheers,
Cameron Simpson cs@cskk.id.au