Lockfile implementation in the standard library (IPC lockfile not packaging)

Ideas
1

I’ve been looking around to try and find why their is no lockfile implementation in the standard library. I do not mean packaging dependency lockfiles, I mean a form of mutex lockfile such are present in fasteners · PyPI

I do see some discussion >12 years ago Mailman 3 Re: [Python-ideas] Cross-platform lockfile (was: Implementation of shutil.move) - Python-ideas - python.org about including the now deprecated lockfile package, but it never seemed to catch on.

I assume it must be because it is OS specific and subject to many implementation details especially on more exotic systems?

2

Yes, basically this. And there are implementations on PyPI (as you note) for people that want them, so it’s not immediately obvious what benefit there would be in having something in the stdlib as well.

3

I might not understand the full picture, but I’m quite successfully using the multiprocessing.managers.SyncManager with queues to symc processes. Since that class also supports the other threading synchronization tools it might be worth a look.

4

I assume it must be because it is OS specific and subject to many implementation details especially on more exotic systems?

Since you mentioned IPC specifically, Named semaphores (also platform differences and not in standard library at this time) are made for this, and don’t have a problem persisting improperly if the application or host system crashes without cleaning up. Not sure on prior 3rd party libraries there, but it would only require platform detection + thin wrapping of a small number of syscalls.

If there’s a need for this kind of synchronization (beyond what’s available in multiprocessing, threading, or by designing how your application handles work differently to not need explicit synchronization), I’d expect Named Semaphores to be easier to advocate adding to the standard library as the system APIs should be available on a significant set of (all of except wasi/wasm? Definitely available on linux, windows, and mac) supported platforms, it’s made for this, and you won’t run into the kinds of edge cases possible with temporary files with some file systems or even mount options.

5

I see, that definitely looks good for a more rigid solution. I was looking for a quick-n-dirty solution to do some basic IPC with a bash script and python script. I saw the Ubuntu Manpage: lockfile - conditional semaphore-file creator package, so thats where I was looking for the python counterpart. I will probably use fasteners + lockfile for my proof-of-concept.

6

If I need something like this I generally use a file lock. There is filelock on pip for this.

I’ve also seen a case where a port number is passed around and everything tries to bind to the port and that works as a mutex.

Heck I’ve even seen code shell out to flock on Linux/Mac.

I’m curious: is there a situation where an IPC based file lock would work where a file lock doesn’t?

7

Already answered above…