Running a process inside multithreading while multithreading is uses 100% of CPU resources

I have large number of i/o bound tasks. I am using multithreading to run them on 2 core machine. How will my performance be affected if I spawn a blocking process (subprocess in python) in my task’s code?

Assume that I am running os.cpu_count() number of threads – which I assume is maximum possible threads.

By Asapanna Rakesh via Discussions on Python.org at 21Apr2022 00:20:

I have large number of i/o bound tasks. I am using multithreading to
run them on 2 core machine. How will my performance be affected if I
spawn a blocking process (subprocess in python) in my task’s code?

If the process is I/O bound, not at all. If your Thread is blocked
waiting for the subprocess, well that Thread will be stopped.

Assume that I am running os.cpu_count() number of threads – which I
assume is maximum possible threads.

No.

  1. CPython uses a global interpreter lock (known as the “GIL”) around
    all the pure Python stuff. So for CPU bound pure Python code, only one
    CPU will be used. The GIL lets the interpreter assume that it is the
    sole user of the interpreter’s internal data structures, meaning no
    other locking is needed, which avoids deadlocks and complication.

  2. However, almost all blocking operations such as file I/O and waiting
    for a subprocess release the GIL while waiting, allowing another
    Thread to use the CPU for pure Python code. So you can run more or less
    as many I/O bound Threads as you like, far in excess of the CPU count.

  3. Plenty of modules with C extensions, including parts of CPython which
    use CPU bound C libraries such as hashlib, also release the GIL while
    the C component runs. So the C component gets to use a CPU, flat out if
    necessary, and the pure Python code in another Thread can continue using
    a core.

So:

  • if you/re I/O bound, run lots of Threads
  • if you’re CPU bound pure Python, you can run lots of Threads too but
    only one CPU will get used because the interpreter only runs one
    thread of pure Python at a time
  • if your CPU bound stuff uses a C extension which releases the GIL, you
    can run multiple threads and they will make use of more CPUs; on even
    vaguely modern CPUs this is complicated (inside the CPU) and the
    os.cpu_count() is probably not a good guide to what limits you might
    impose; you can always run “quite a lot” of Threads and let the OS
    sort out who gets the available CPU
  • if you need to use multiple CPUs with pure Python CPU bound code, you
    might then reach for the multiprocessing module, which presents a
    Thread-like API but runs the code in separate processors (and thus
    separate Python interpreters)
  • or of course, you can just run your programme multiple times at once
    from a script, working on separate data sets

Personally, I use Threads a fair bit and I have yet to want the
multiprocessing module. If nothing else, multiprocessing makes sharing
data somewhat more cumbersome - your are talking to a separate process
and so data has to be sent back and forth. There are others who almost
always reach for multiprocessing instead of threading because then they
don’t have to worry about deadlocks to the same degree. I remain
unconvinced, personally.

Cheers,
Cameron Simpson cs@cskk.id.au