Do assignments ever require locking?

I recently came across some code where the author applies locks around “simple” assignments, in a multithreaded program:

class obj:
    def __init__(self):
        self.lock = threading.Lock()
        self.some_callback = None
...
with obj.lock:
    obj.some_callback = some_method
...
with obj.lock:
    cb = obj.some_callback
if cb is not None:
    cb(foo, bar)

I.e. the lock merely protects reading and writing the member variable.

Is there any situation, maybe in pypy or some other non-CPython implementation of Python, where the above code might be required to avoid a race condition?

The author of the code in question doesn’t want to remove the lock because they think it might be necessary on Jython or pypy or a future no-GIL CPython or [unknown].

My take is that assignments are atomic by definition, no matter the underlying Python implementation, and the lock is unnecessary overhead and impedes the code’s readability.

Who is correct?

I’d agree with you for reading a primitive.

In the code example, without the lock multiple threads could write to that method variable. And presumably it’s not about reading that method variable (like a primitive). It’s called call_back so the intention is to call it. This could do anything to other state on obj, and require other expensive or shared resources.

So the author could well have had good reason. If nothing’s broken, I would not mess around with it just to improve code quality.

My understanding is that assignments are not by definition atomic - that they are only atomic for simple assignments (using immutable objects like small integers or using built-in data types). Assignments for mutable types/complex objects correspond to multiple byte code instructions. My understanding that a thread switch can occur between any two byte code instructions.

I tried to find a good reference in the official Python docs. Best I could find was this:
What kind of global mutation are thread-safe? which is rather fuzzy.

I’m not sure, but I’m tempted to think the sample code does require locking (if thread-safety is a goal). The lock on the read could in some cases also be required, I think. Perhaps not in this example (I dont know), what if you have a longer block of code where the function would be called multiple times and you want to ensure consistency? Also, ‘some_callback’ itself would make me nervous, since it can be anything – for instance a C-level call that itself releases the GIL; it could by itself be thread-safe, but once it releases the GIL anyone is free to modify the parent object.

1 Like

Unfortunately, without being completely sure what the object is, we can’t actually say. If some_callback is actually a property, it could do anything (as it’s effectively a method call in disguise). Personally I would consider this to be bad style (if it’s a property and it absolutely needs atomicity, it should do the locking inside the property function instead), but it’s legal.

But my gut feeling is that this is cargo cult programming. Having run into threading problems and solved them with locks, the author threw them into more places. It’s really hard to be completely sure, though.

2 Likes

Let’s assume that obj.some_callback is not a property; if it was it could lock itself internally.
Likewise if there was an issue with the callback getting run from two threads then it could use an internal lock if it needed to.

My question thus boils down to this: Can “thread A updates an attribute while thread B reads it” result in any other outcome than “thread B reads either the old or the new value of the attribute” plus “there are no related consistency problems” (like broken reference counters)? Or, more to the point, does the specification of Python-the-language, as far as such specification exists, guarantee that anywhere?

No, there is no such formal python specification. However, by default CPython’s behavior is then what should be expected from all implementations, which in this case clearly says “no, pure python code should never be able to result in broken (e.g. wrong ref count) objects”

Starting with the last question: We’re currently (as of 2024) actually seeing quite a bit of specification needing to be written. Aspects that had previously not required any sort of documentation are now going to become important. Free threading is revealing that there are definitely some aspects of Python-the-language that have never actually been promised, and CPython implementation details are what we all go on. So, take all of what I say, and all of what everyone else in this thread says, with the proviso that more things may be specified in the future. I would generally assume that any new specs will be compatible with existing behaviour, though, so anything that IS specified should remain valid.

So. Let’s figure out exactly what happens when you mutate an object’s aftribute. As mentioned, properties and such can get in the way, but I’m going to assume the most simple and straight-forward behaviour there is: The object’s dictionary gets updated.

class Spam: pass

x = Spam()
x.ham = 1
print(x.__dict__["ham"])

So your question comes down to two things:

  1. Can thread A and thread B see two distinct dictionaries?
  2. If one thread is changing a dictionary, will another thread see problems?

I’ll start by excluding a number of behaviours which, if they were to occur, would definitely be called interpreter bugs:

  1. A “half-and-half” pointer error. On some CPU architectures, a pointer (such as a Python object reference) can’t be written in one go, so you have to write one half, then the other half. So you could read the first half of the new value paired with the second half of the old value, which would give you a nonsense pointer. This would be bad, very very bad; but it would also be one of the most obvious problems to look for. I think we can rule this one out on the basis that the exact same problem will show up in practically everything, and so it will NEED to be solved. (Which is quite easy on those CPU architectures that can write an entire pointer atomically.)
  2. A broken reference count. This one’s specific to reference counting, obviously, and one way to prevent it is to go for a different style of garbage collection (Python doesn’t mandate refcounting). But, again, if refcounting is a thing, broken refcounts are a critical failure, so this one will be dealt with.
  3. A failure within the dictionary itself. For example, thread A might be trying to add a new attribute, which causes the dictionary to enlarge itself. This would be a more insidious bug as it’s not going to trigger nearly as frequently as the other types, but since dictionaries are so important in Python, I would trust that people smarter than I will have already looked into this and made sure it won’t be a problem. (Which, again, might be quite simple if there’s a way to leave the old dictionary contents visible to other threads while the new hash table is being constructed.)

In general, anything that would outright segfault the interpreter should be considered a bug. [1] Bugs definitely do happen, but the more common the situation, the more likely that it’ll be discovered early. Core data types like dictionaries, lists, strings, etc should be safe from these kinds of critical failures.

That’s not to say there can’t be other sorts of errors, and a simple x += 1 could well produce strange results, but at very very least, it won’t result in a memory leak or a half-and-half pointer or anything like that.


  1. Unless you’re using ctypes - in which case, have fun, there’s endless crashes waiting for you! ↩︎

2 Likes

The GIL ensures that python will be able to execute its byte code ops atomically.
The lock is pointless, unless its a property. As Chris suggest may be cargo-culted.

If there was a read-modify-write pattern then a lock is mandatory for correctness.

If 2 threads read the some_callback then there is nothing to stop the callback being invoken by both threads.
Which may, or may not, be intended. More of the design would need be explained.

(This is a decent approximation, but there are exceptions. The GIL isn’t held for “one bytecode operation”; but you can certainly assume that the GIL could be released between any two operations as listed in dis.dis().)

Umm, well, there are Python versions without a GIL today, not to mention the current work to remove it entirely, long-term.

There will need to be an equivalent guarantee spelled out that we can reason about for multi-threaded code for the free-threading python.

Which, I assume, will be documented at some point.