Is there any way to share a common semaphore between different python programs. Currently multiprocessing module allows semaphore to be used between processes forked/spawned from current process. In C on using POSIX semaphores, we can have a producer and consumer as two different programs and share a common semaphore. similarly in python can two different .py files share a common semaphore using the built-in python modules?
Yes, but it has to be something outside of both. For example, it’s very common to use a filesystem lock to govern something, or have the two processes connect to a single manager that makes the decisions. It depends on your design as to what’s best.
Can’t it be something like the producer program creates the semaphore and then consumer and producer operate on the same (like in C using sem_open)? When a single manager makes decisions, the semaphore is in control of manager and neither programs will be able to use it to make decisions right?
What would then happen if two different processes run the same.py file (i.e. twice)?
[Edit] If such a thing existed, it would just bump the counter on the Semaphore an extra time.
A single manager that issues tasks to others? That would work fine; tasks can be issued through a multiprocess queue of some form, and the other processes will simply draw from that.
There are some multiprocess semaphores. The most common ones are found on the file system (at least on Unix - maybe not on Windows) or sockets/pipes.
1 Like
You can use flag files. Not sure if that’s applicable to your situation, and it’s not perfectly robust. But it is a way to have variables that are “global” on the level of “being accessible to all programs that are running”. And they can be changed atomically.
1 Like
Yes, but even that would be same as adding a new producer or consumer to the mix and the results should not be much different right? Applying the same analogy to a producer-consumer problem in C, we can run the either producer or consumer more than once, the user needs to take care of the logic.
I found another thread which discussed the same issue,
can python multiprocessing do semaphores between “unrelated” processes? - Stack Overflow
So, seems like for semaphores we cannot do this in python(unlike in C). Other ways like flag files pointed out by @peterc or ones pointed by @Rosuav can work but it seems to be absent for semaphores as of now.
Third party extension authors have tried to wrap POSIX Semaphores for use in Python. I don’t know which the best choice is (nor how suitable they actually are) but, e.g.:
1 Like
You can use named semaphrores on Unix and Windows, APIs are not the same, but it’s doable.
If you need example code I can post a link to one of my scripts that does this. Let me know.
Yes, please do provide the link to the script.
See the different _sendCommand functions that use es h OS method to talk the Barry’s emacs server process.
Code is here BarrysEmacs/Editor/PyQt6/be_client.py at master · barry-scott/BarrysEmacs · GitHub
The server code is C++, I can link to that if you need it for reference.
Thank you Barry, yes please do share the server code.
See BarrysEmacs/Editor/PyQt6/be_command_line_handler_posix.py at master · barry-scott/BarrysEmacs · GitHub and BarrysEmacs/Editor/PyQt6/be_command_line_handler_windows.py at master · barry-scott/BarrysEmacs · GitHub
You can then web search for the details of the APIs I’m using.
2 Likes
Thank you @barry-scott.