Python’s I/O models all face the same question: what happens when something blocks?
Asyncio can be stalled by one slow function. Threads require manual offloading, while processes are expensive and complicate resource sharing. Stackful runtimes such as gevent still run cooperatively on one thread, retaining asyncio’s core limitation.
The solution: work-stealing across OS threads
Go runs lightweight, stackful functions across real OS threads. If one thread stalls, work queued behind it would also stall, so other threads steal that runnable work. The blocked thread may remain stuck, but its remaining workload is rebalanced.
This combination – OS threads, stackful functions, and work-stealing – is a major reason Go performs so well in networking.
Adding work-stealing to Python free-threaded
These discussions cover virtual threads and asyncio in Python free-threaded:
The ideal runtime combines:
real OS threads (no gil, free threaded)
stackful fibers or goroutines
work-stealing
I built this as a C extension and achieved networking performance comparable to Go:
The remaining problem is migrating Python execution between threads.
The allocation problem
When a Python execution context moves to another OS thread, it may still be tied to the original thread’s mimalloc heap. The next allocation can then use allocator state owned by the wrong thread, causing corruption or a crash.
My CPython patch redirects allocations to the heap of the thread currently executing the context, enabling safe migration and work-stealing. I’d like to propose this patch as a possible PEP for Python seeking core dev sponsorship.
I haven’t looked closely at the internals of runloom, but does the stackful fiber work properly even for blocking implemented in the extension module?
My impression is that the transition to free threading itself is a big challenge for Python, and it would be difficult to accept additional complexity and maintenance costs beyond that.
asyncio can slow down the whole system in some cases. The GIL not only prevents multiple Python interpreters from running in parallel, but also interferes with the operation of the OS’s fair scheduler. The combination of multi interpreters and threading makes it difficult to share resources such as connection pools across multiple threads, and the improvement from multi-process is small.
Free threading solves those problems, which can significantly improve the 99%-tile response time when applying various types of load to a normal web application.
I think that focusing on the success of free threading and improving observability is important to maintain Python’s competitiveness in web backends.
does the stackful fiber work properly even for blocking implemented in the extension module?
Very good question. The short answer at the moment is it doesn’t. I’ll tell you about some design trade-offs I was thinking about and whether it might be worth revisiting this.
So as you’ve probably guessed: if you make a call to a C extension you enter deep inside code which blocks the fiber until it finishes. Go’s solution to this is simply to avoid such extensions entirely. That way: it has full insight into all the blocking parts and where it should yield. That design doesn’t work well with Python whose main strength is it’s extensibility. Runloom, after-all is a C extension!
So at the moment if these C calls block a fiber, they block the thread they’re assigned to. Fibers on that thread could still be work-stealed (providing the patch I wrote about in my OP was merged into Python. Though at the moment – only never-run fibers are safe to migrate.)
The other design I considered is having a thread dedicated to having these C extension calls off-loaded to it. But IMO, this is kind of complex and you essentially already have such a model with the regular threads (just use the thread it’s already running in and work-steal items if they’re idle.) That design seems much simpler to me and doesn’t require additional code.
Maybe there are other improvements to make here. I think this is an open research question worth exploring. I do have ideas to make this even smoother with extensions.
My impression is that the transition to free threading itself is a big challenge for Python, and it would be difficult to accept additional complexity and maintenance costs beyond that.
I think this is a key benefit to extensions. All my code other than this small patch is external to Python. CPython doesn’t need to change vastly to accommodate such a runtime. It can be another option in Python’s existing ecosystem.
Free threading solves those problems, which can significantly improve the 99%-tile response time when applying various types of load to a normal web application.
Agreed. I can already see people exploring free-threading to make asyncio scale higher. It’s going to be a direct boost to every Python networking library. People will have to be careful how they use it though. As thread spawning still has a cost.