Add RW locks to python?

Ideas
1

Hi all,

I was thinking that with the advent of free threaded python it might also be a good idea to add read/write locks.

My reasons are:

  1. you can’t subclass locks
  2. most OS’s have atomic rw locks. If you have to create a new class to implement it, they won’t necessarily be atomic

Of course, rw locks come with their own issues (easier starvation and deadlocks). And they can also be implemented via semaphores, so I can also see the case for not implementing them.

Thoughts?

2

Just to clarify, are you talking about adding an RWLock class to threading, or about adding a C API to a low-level native RW Lock? Or both?

IMO it’s probably a little too early to start adding stuff to the standard library for this, when we can experiment in projects outside of CPython and upload wheels to PyPI.

See GitHub - dpdani/cereggii: Thread synchronization utilities for Python, which could host such a Python type.

1 Like
3

Atomic requires direct access to memory. As you have in C etc
But python keeps programs well a way from memory.

Does it make sense to expose atomics in python?

We already have locks in the threading module API.

4

Actually, I’m mistaken. rwlocks aren’t atomic.

But I was thinking that similar to how threading locks/mutexes/etc in CPython just call the OS system call via a C extension, it would be beneficial doing the same for rw locks.

There are some (mostly unmaintained) rw lock implementations in pypi, but they’re in Python.

5

Another reason to put in rw locks at the language level as opposed to the application level is because once you start getting into multi-threading, you’re going to need things like thread safe sets/lists/dicts etc. rw locks are well suited for these things.

6

It would be really easy to wrap the C++ std library shared_mutex and expose it in an extension module.

If you want to use one now then there’s one in Qt, so both PySide and PyQt almost certainly have it (although it’s a big dependency just for a lock type).

This doesn’t really make sense to me. If you want it to be accessible from Python code then there’s very little difference between an extension module from PyPI and an extension module from the standard library. And if you’re implementing a thread safe container library you’re probably using c or c++ or rust anyway so will have access to their lock types.

7

Just a nit: CPython is C, not C++.

8

But it’s extensions can be C, C++, rust etc.

9

Hi @rlippmann :wave:

I agree that a readers-writers lock can be very useful in general and that it would make sense for it to be in the standard library.
At this time, cereggii doesn’t host that, but I opened an issue to add it.
You could also implement it in pure Python code, using the Lock and Conditions classes from the threading library.

On the issue of atomics in Python, they are not exposed directly. You may find some libraries on PyPI that do expose them raw, but my suggestion is to avoid that.
Java, for instance, prohibits using atomics directly but has AtomicReference in its standard library to solve the same problem at the object model level, instead of at the raw memory level.
If you wish to implement your own custom lock class, you could do it using cereggii’s AtomicRef, which is conceptually similar to Java’s AtomicReference.
For instance this is a simple test-and-set lock, it’s not efficient but it works:

from cereggii import AtomicRef


class Lock:
    def __init__(self):
        self._value = AtomicRef(False)

    def acquire(self):
        while True:
            if self._value.compare_and_set(False, True):
                break

    def release(self):
        self._value.set(False)

You’ll find that the Python built-ins are thread safe in free-threading builds; you can read more in PEP 703.

If you mean that you need the scalability that a readers-writers lock can provide, i.e. that only a single thread at a time may modify a set/dict/list, then I think that the optimistic lock avoidance described in the PEP should already provide you with the performance you need.

You can also checkout cereggii.AtomicDict :wink:

1 Like
10

@dpdani Ah, I wasn’t aware that built ins are thread safe in free threaded python. I am positive the python core developers are way smarter than I am and have figured out the best way to implement thread safety there.

The only other use case I can think of off the top of my head for rw locks is object attributes. But I’m not sure if that’s even necessary if the built ins are thread safe.

As for atomicity, yeah, not really necessary and you probably shouldn’t play with it unless you really know what you’re doing :slight_smile:

1 Like