PEP 805: Safe Parallel Python

I have a few thoughts about sections of the proposal, quotes below without explicit attribution or quoting something else on discourse are quotes of the pep text.

CPython is currently split into two: the default build and the free-threading build. Proponents of free-threading expect that free-threading will become the only version of CPython in a few years. The authors feel that this will be very challenging to achieve, and may be impossible. Removing the default build would involve breaking vast numbers of applications and libraries that are not safe to use with a free-threading build. Even though many libraries are marked as supporting free-threading, it is unlikely that they are all completely safe to use in a free-threading environment given the difficulty of eliminating race conditions.

As a user, I expect free-threading to become the only build when it’s ready to. That was the plan set into motion in PEP 703, and was considered to be phase 3. PEP 779 only reaffirmed this, while setting further conditions on phase 2 and re-stating that the conditions for phase 3 were not ready to be determined.

If you don’t believe this is a possible goal because removing the default build will break users, I think that’s asking to re-litigate free-threading’s acceptance, because it was accepted with exactly that source of breaking being the plan.

Almost all objects in Python have a __dict__ attribute. Freezing an object will convert its __dict__ into a frozendict. Synchronizing a module (or any object that both supports synchronization and has a __dict__) will convert the __dict__ into a SynchronizedDict.

This ran into problems when proposed as something users could do before: Use frozendict as type and module namespace (dictionary), can you explain what other changes will be made in the interpreter to avoid the issues the more narrow proposal ran into?

A generalized object freezing proposal also ran into issues multiple times before. The most common one was that without a deepfreeze, there were unacceptable edge cases; I don’t see those issues addressed here.

(PEP 351, as well as more recently Freezing data type for parallel access, and several others besides)

There’s also PEP 795 that does propose deep immutability, and that seems necessary to do any better than what free-threading does already for sharing arbitrary objects and enforcing immutability of them.

A new class, threading.ThreadGroup, will be added to help port applications that are currently relying on the GIL (accidentally or by design), using multi-processing, or using the _interpreters module, to parallel execution using threads.

All threads sharing a ThreadGroup object will be serialized, in the same way as all threads are currently serialized by the GIL. Using multiple ThreadGroups offers much the same capabilities as multi-processing, or multiple interpreters, but with lower overhead and with the ability to share objects without copying.

If an application is accidentally relying on the GIL, how will the authors know to use a ThreadGroup? If it isn’t relying on the GIL, they shouldn’t use it, serialization isn’t free. If they are relying on the GIL and know about it, they can replace it with a fine grained lock instead, which should be less expensive if done correctly.

This API seems like a trap to me.

3 Likes