But it does (with some limitations) make third party code thread-safe, and has done for decades now. So, it’s hardly surprising that people rely on it.
1 Like
There’s “rely” in the sense that race conditions are sufficiently rare that they are statistically never encountered, and there’s “rely” in the sense that a piece of code has been designed and stress-tested for potential race conditions.
I think most cases of user code made thread-safe by the GIL fall in the former sense. The latter sense usually involves careful usage of synchronization primitives (locks or queues, for example - perhaps implicitly through higher-level constructs such as a thread pool).
(but of course, the vast majority of code that serializes data to JSON also doesn’t mutate that data from another thread - GIL or not 
)
6 Likes
I’d say that a native extension that was being careful about not releasing the GIL and using borrowed references correctly for improved performance is one case where FT introduces bugs where there were none before.
This is not to discredit FT, which I believe is a great engineering achievement and the direction the Python community should take. Rather, the question we pose with this PEP is, do we want more explicit semantics when sharing data between threads?
The example of serialization probably shows that bare sequential consistency often falls short of being sufficient.
As another maybe more compelling example, I believe that the standard iterators of PEP 805 are significantly better than those of FT, where locking becomes almost always necessary to avoid segfaulting the interpreter, for use cases that, at least in my opinion, don’t exist. That is, I believe we all can come up with an example where sharing an iterator is a good feature, but in general sharing say range(10) makes little sense and always leads to non deterministic behavior. Yet, range must be thread safe. (Though range is special and it has specializations that mitigate the performance impact, other iterators like itertools.chain were not similarly blessed.)
- gh-129068: Make range iterators thread-safe by colesbury · Pull Request #142886 · python/cpython · GitHub
- gh-123471: Make itertools.chain thread-safe by eendebakpt · Pull Request #135689 · python/cpython · GitHub
Making the interpreter not crash after removing the GIL is of course very important, and the people who worked on FT have done an amazing job at doing that and keeping good performance. Nevertheless, we believe that it’s possible to do better.
The built in range doesn’t support concurrent access without locking? Alright, put that lock in Python code though, is all we’re suggesting. Then, there will be no need to implicitly slow down every other case.
7 Likes
I remember reading a statement somewhere that went something like this: “An interpreter crash caused by pure Python code is always an interpreter bug.” Two questions:
- Was that statement correct?
- Is it still true with FT?
1 Like
Was that statement correct?
Generally, yes.
In a few cases weird cases, like stack overflow, jail-breaking the dict out of a MappingProxy, using gc.get_referrents() to get part-constructed objects, the historical answer has been “won’t fix”, but I’m pleased to say that we seem to always fix these now (although the MappingProxy one still needs fixing)
Is it still true with FT?
Yes, I think?
The problem with FT is not race conditions that crash the interpreter, but race conditions that corrupt your program.
I didn’t know a destructor could release the GIL. So yes it can and no we don’t.
From trying locally, it looks like the GIL is blocking the destructor from happening in another thread whilst ujson.dumps() is still going and thereby keeping itself from being released? Or at least that’s my guess. I can only make it crash if the destructor is called on the same thread. e.g. Something esoteric like ujson.dumps(mutating_input, default=ClassWithDesctructor() and "actual default").
Code in case anyone's interested
import threading
import ujson
a = []
b = []
c = []
done = False
class Foo:
z = 0
def __del__(self):
self.z += 1
def mutate():
while not done:
a.append(b)
a.append(b)
b.append(c)
b.append(c)
if len(c) > 3:
c.clear()
b.append(c)
c.append(0)
c.append(Foo()) # destructor on background thread
c.append(0)
if len(a) > 3:
a.clear()
if len(b) > 3:
b.clear()
threads = [threading.Thread(target=mutate) for _ in range(3)]
[i.start() for i in threads]
try:
for i in range(100000):
#ujson.dumps(a)
ujson.dumps(a, default=lambda x: Foo() and {}) # destructor on same thread
finally:
done = True
A pure Python default function would allow releasing the GIL.
This crashes reliably for me:
import threading
import time
import ujson
l = [object()] * 1_000_000
def default(obj):
time.sleep(1e-3)
return None
def mutate():
time.sleep(1e-2)
l[:] = []
threading.Thread(target=mutate).start()
ujson.dumps(l, default=default)
(note the sleep calls only increase the likelihood of a race condition happening; the code is inherently racy even without the sleep calls)
And, yes, this doesn’t involve a destructor, but any pure Python destructor running in the middle of C code would present the same issue. I just don’t want to find how to trigger a pure Python destructor call inside ujson.dumps.
2 Likes
(To be clear: posting as myself, not as a SC member)
This proposal seems to have some pretty significant gaps, as well as a couple of fundamental misunderstandings. We do need to experiment with new threading APIs. I like S^M APIs and on the surface this seems pretty reasonable (albeit more complex than I would like), but I do not see how any of this can be implemented and enforced as described. I also don’t see how this can be introduced without breaking backward compatibility in extremely wide-spread, significant ways.
The PEP starts off pretending PEP 703 introduces a threading model. It does not. Python’s threading model has been the same since the introduction of threads, and PEP 703 doesn’t change anything about it. The PEP also claims this thread model (which it calls the C++/Java thread model) is not suitable for Python. I won’t argue it’s a good thread model, but it’s what we’ve always had. Threads have been actively used for decades despite the GIL, in many different ways, including when they should have no benefit because of the GIL That’s the reality of Python at the moment.
As has been brought up in this discussion a few times, free-threading doesn’t introduce many new threading bugs (barely any in Python code, and fairly easy to identify ones in extension modules) although it does make them more obvious. You’re worried about the community never being ready for free-threading, but in practice the amount of work to support free-threading is significantly less than what you seem to think. Despite working on and reviewing quite a few complex changes to support free-threading, I’ve seen none that are fundamentally problematic – because relying purely on the GIL for thread-safety is already provably unsafe, so it’s basically only done by packages not actually sharing data between threads, and so free-threading doesn’t affect them. (The PEP also makes the mistake of equating making free-threading the default with removing the GILful build, but those are distinct steps that are likely many years apart.)
Given that we have a fundamental disagreement on how free-threading is challenging, and how much, let alone whether it’s a viable project, are you sure you want to make it one of the PEP’s motivations? If free-threading proves not to be as challenging as you think, does that make this PEP unnecessary?
The fundamental proposal is very similar to past proposals for an object-capability model in Python. “If only we can control all the possible ways to get a reference, we don’t have to do any extra checks if you have a reference.” It’s very difficult to even accurately enumerate all the ways to get a reference, let alone control them all. Some of them may not have enough context to apply the rules you’re proposing here, so they would need to be fundamentally changed. There isn’t a single code path they all flow through, not even Py_INCREF – which isn’t suitable for this anyway because it can’t currently fail.
Even if we manage to add an appropriate check to all possible ways to get a reference, which includes a lot of third-party extension modules, a bug in any one of them invalidates all assumptions in the entire program. There is no defense in depth here, unless we add yet more checks in more places. Every check has performance impact, and introduces new failure modes to code not previously exposed to it.
As Petr pointed out earlier in the discussion (and as far as I can tell nobody addressed or even acknowledged) the downsides of introducing this mode of execution are severely downplayed in the backward compatibility section. In fact, that section confuses “free-threading” with any use of threads even with the GIL. It’s hard to tell if you expect a smooth migration path to this new API, and how you’ll allow mixing of the new API with code not using it. The mention of resolving breakages “on a case-by-case basis” suggests a migration path of some kind, but it’s hard to tell. Given that users of threads, like all users, generally call other code (like libraries), some of which may not be in their control, how much is all of this really going to break? Will all existing code just continue working? Will unsafe sharing start warning? Are you imagining this happening on a Python version upgrade? A runtime toggle? A per-module change? A new build mode?
As for performance, the entirety of that seems to hinge on the specializing interpreter, despite the fact that very many of these checks will need to be done throughout the runtime and in extension modules. Even disregarding how much the specializing interpreter (or the JIT) will be able to mitigate the performance impact, which I’m much less optimistic about than you, the overhead of these other checks will be significant. Again, how many checks we’d need will depend on how much defense in depth you’re imagining, but fundamentally they need to happen for any store or load of references, and the checks can be quite complicated. Do you want every call site of, say, PyList_GET_ITEM() to have checked or reasoned about the thread-locality of the list? Does that seem like a scalable, robust API? Have you done any experimentation to figure out how much overhead we’re talking about here? For example, by identifying a few fundamental places for these checks (say, dict and list access, including the fast path macro versions) and doing, say, a few atomic loads to simulate the impact? How much overhead do you consider acceptable, for this new API to be succesful?
10 Likes
This may be off-topic for this thread (which I have not followed in detail) but could explain what you mean by this? Or link to an explanation? Apparently you think this is a well-known fact, but I fail to follow what you mean; probably the crux is a specific meaning of “thread-safety” that I don’t know (since this is a vague term often bandied about without a precise definition).
3 Likes
I didn’t bother linking to it because one example is literally the discussion right before my post
If you look at examples of code that are supposedly safe because of the GIL (e.g. the += example given by Mark) you’ll find out that it comes with a bunch of caveats (like “when operating on exactly ints/floats and not subtypes” and “as long as destructors don’t execute arbitrary code”) that are impossible to rely on in real code. In extension modules it’s possible for the GIL to provide safety if you know exactly which calls could potentially release it, and you know very well that this is much harder than it seems.
2 Likes
Okay, thanks, now I think I get it.
-
For C code (the ujson example) it’s just really hard to remember what operations may release the GIL. (Personally I’d be suspicious of any C API call that’s not explicitly documented as guaranteed not to release the GIL.) Arguably you can write thread-safe code in C relying just on the GIL but you have to be very careful. I don’t believe this means relying on the GIL is “provably unsafe” though. At least the ujson example seems to violate the rules.
-
For Python code, the GIL was never meant to protect from thread-safety, and I agree that the fact that
a += 1happens to be thread-safe in recent CPython versions in a very constrained environment is not enough to rely upon.
2 Likes
Hi Thomas, I’ve trimmed your text as I’ve replied. Hopefully I haven’t changed the meaning by removing too much context.
This proposal seems to have some pretty significant gaps, as well as a couple of fundamental misunderstandings.
Could you be specific about where you think the gaps are?
but I do not see how any of this can be implemented and enforced as described.
That’s unfortunate. I’ve tried to be quite precise about the semantics, and outline a possible implementation. Which parts do you think cannot be implemented? It would help to know where we can strengthen the implementation appendix.
I also don’t see how this can be introduced without breaking backward compatibility in extremely wide-spread, significant ways.
Again, I had hoped that this is clear from the PEP. What do you see as breaking compatibility in “extremely wide-spread, significant ways”?
With a single ThreadGroup, this PEP degenerates to the behavior of the current with-GIL build, with a few minor exceptions.
Nothing should break.
The PEP starts off pretending PEP 703 introduces a threading model.
There is no pretence. By removing the GIL, PEP 703 changes the semantics of Python programs.
Parallelism becomes possible, and more race conditions can occur.
PEP 703 doesn’t specify exactly what the changes to semantics are, which is understandable as the with-GIL model wasn’t written down either.
But there is a new model, even if it is not written down.
We should specify what the model is for free-threading, but that’s out of scope for this PEP.
As has been brought up in this discussion a few times, free-threading doesn’t introduce many new threading bugs
One persons “not many” will be another persons “too many”. Why introduce new bugs at all?
the amount of work to support free-threading is significantly less than what you seem to think
Maybe. Maybe you think it is less than it is.
It doesn’t matter. But it is some unknown amount of work, and it is very hard to tell when it is done.
relying purely on the GIL for thread-safety is already provably unsafe
That is not entirely true. While hoping that the GIL will protect you from all race condition is not going to work, there are simple test-then-act constructs that the GIL does protect. For example: Race between UserDict.__getitem__ and __delitem__ can result in __missing_ not being called when item is missing. · Issue #156544 · python/cpython · GitHub
Another way the GIL does protect against race conditions is by discouraging the use of threads.
No threads: no race conditions.
The PEP also makes the mistake of equating making free-threading the default with removing the GILful build, but those are distinct steps that are likely many years apart
Fair enough, I’ll rephrase that part. The removal of the with-GIL build will not be the big break, it will be the switch to making free-threading the default.
Most people will use the default version, whatever that is, and that is when their code might break.
Given that we have a fundamental disagreement on how free-threading is challenging, and how much, let alone whether it’s a viable project, are you sure you want to make it one of the PEP’s motivations? If free-threading proves not to be as challenging as you think, does that make this PEP unnecessary?
I think that Java and C/C++ have already proven that weak memory models are very challenging to use. Not impossible, just very difficult. For C/C++ that difficulty is entirely justified: performance is everything. But not so for Python, correctness and ease of use are more important.
The fundamental proposal is very similar to past proposals for an object-capability model in Python.
An interesting comparison.
The boundaries are different though, and capabilities are aimed at sandboxing. PEP 805 does not aim to provide additional security over the with-GIL build.
The important boundary for PEP 805 is between thread and heap, and because of the way that C and Python (and most other languages) work, we have well defined places to put the checks.
OOI, do you have a link to the proposal for capabilities?
Even if we manage to add an appropriate check to all possible ways to get a reference … a bug in any one of them invalidates all assumptions in the entire program.
That is also true of reference counting, and critical sections for free-threading, to name just two.
But unlike those things, we can easily assert that a reference is legal without needing additional context, catching bugs close to their source in debug builds.
There is no defense in depth here…
We can do what we do for all the other invariants that CPython needs to maintain: assume that the checks have been inserted correctly in release builds to maintain performance, and add a generous helping of asserts in debug builds.
This argument could be applied to almost any new feature.
As Petr pointed out earlier in the discussion (and as far as I can tell nobody addressed or even acknowledged) the downsides of introducing this mode of execution are severely downplayed in the backward compatibility section.
I haven’t got around to updating that, yes, but calling it “severely downplayed” seems hyperbolic.
Here’s the list of breakages:
- Function’s
__kwdefaults__will become a frozen dict. (I really hope no one writes code that mutates the keyword defaults of functions) - Some, usually small, objects’ lifetimes may be extended. This is from using using deferred reference counting. Free-threading needs to do the same, and I suspect we are talking less than 1MB of extra memory in most cases.
- One time ABI breakage. Not API breakage. Just recompile your extension and your done.
Note: free-threading does the second and third of those as well.
In fact, that section confuses “free-threading” with any use of threads even with the GIL.
I do mean free-threading, but it could be clearer.
Try reading that section, as “the impact of PEP 805 on code that already works with free-threading”, and it might make more sense.
It’s hard to tell if you expect a smooth migration path to this new API
Do you mean the new API, or new execution model?
The new API is just that, an API: use as much or as little of it as you want.
The new model degenerates to the current with-GIL model if you don’t use the new API. Existing code will just work (apart from the __kwdefaults__ thing). Everything is a local object, and there is only one ThreadGroup.
Given that users of threads, like all users, generally call other code (like libraries), some of which may not be in their control, how much is all of this really going to break?
As libraries move to supporting parallelism, it will impact users in some cases.
Code that monkey patches libraries is going to break if the libraries start freezing objects.
If libraries do not make any changes, instances of their classes will be local and will not be shareable between ThreadGroups. This is less restrictive than free-threading as unmodified modules do not prevent parallelism, just sharing.
The mention of resolving breakages “on a case-by-case basis” suggests a migration path of some kind, but it’s hard to tell.
That refers specifically to making local objects shareable. It is up to the authors how they do that. I’d suggest using immutablility where possible, but adding locks is a valid solution.
Will unsafe sharing start warning?
Attempting unsafe sharing will raise an exception. That’s why the PEP has “Safe” in the title.
As for performance, the entirety of that seems to hinge on the specializing interpreter
Not the entirety. As you point out, checks will also need to be done on API boundaries.
But the interpreter is a big, important part of CPython performance. So, it makes sense to lean on the specializing interpreter where we can.
Even disregarding how much the specializing interpreter (or the JIT) will be able to mitigate the performance impact, which I’m much less optimistic about than you, the overhead of these other checks will be significant
Don’t be too pessimistic.
Because there is a well defined model that we can reason about, there are many optimizations that can be applied. Cumulatively, they can keep the overhead in the interpreter and JIT low.
But even in the C API, where neither SAI or JIT can help, there are still optimizations that can be applied:
We need an access check whenever a reference is copied from the heap to the stack, but whenever we make such a copy, we also need to do an incref.
By doing the access control and incref together, we can keep the additional cost of the access check low. Both need branches, but those branches are more predictable for access control, and both are dependent loads, so the access control adds no additional pointer chasing.
In many cases, we can use the access check to avoid additional checks for reference counting.
If an object is local we can use unsynchronized reference counting without the additional biased reference counting check.
I’ll add those details to the implementation appendix.
Again, how many checks we’d need will depend on how much defense in depth you’re imagining
Again, plenty of asserts in the debug build
but fundamentally they need to happen for any store or load of references
It is only loads from the heap to the stack that need checking*. Once in a local variable, loads do not need to be checked, and no stores need to be checked.
*plus a handful of extra checks for protected objects, see the Bytecode Compiler section in Appendix: Implementation | peps.python.org)
Do you want every call site of, say, PyList_GET_ITEM() to have checked or reasoned about the thread-locality of the list?
Lists are always local or protected, as they mutable, so we already know the thread-locality (technically ThreadGroup-locality) and no checks are needed for the list. This doesn’t just apply to lists. If a thread already has a reference to a object, then it must be allowed access to it, and no additional checks are needed.
We will sprinkle asserts liberally throughout the C code, to help catch bugs.
Does that seem like a scalable, robust API?
Yes, it does. The checks are only needed on the heap/stack interface, and only in the heap → stack direction.
Have you done any experimentation to figure out how much overhead we’re talking about here?
No, but I have done some simple statistical analysis of the number of heap->stack reference transfers based on the pyperformance stats. It seems like the overhead should only be a few percent, but I can’t say what it will be with any certainty.
doing, say, a few atomic loads to simulate the impact?
Because the state of an object can only change monotonically, in a way that has no race conditions, all access checks can use unsynchronized loads. So that wouldn’t be a useful test.
6 Likes