When would you use Python's queue or multiprocess.queue over the OS's APIs for communicating between processes?

Is there an advantage to using Python’s queues or multiprocess.queues over PyWin32’s CreateFile, Windows Named Pipes, or python-systemd or dbus-python’s ability to do inter-process communication? Do you primarily use the Operating System’s methods if you have 2 processes in 2 different languages that need to pass information back and forth? There isn’t really any more context here, other than I’m just trying to learn more about when people use which tools, or what makes each tool more practical.

For myself, I try to stick to Python’s multiprocessing queues/pipes so that I don’t have to worry about using different mechanisms on different operating systems. Someone else has already gone through the difficult work of making the multiprocessing module work on both Windows and Linux, so I don’t have to. :slightly_smiling_face:
Python”s multiprocessing tools also tend to be higher level than what an OS provides directly, so it’s easier to get something working quickly. It might not always be optimal, but it will work!

As you pointed out, if you’re communicating between programs implemented in different languages then you have to use something they both understand. If you don’t need to support multiple platforms and/or want to integrate more tightly with the OS to use some specific feature, then using using Windows APIs or dbus or whatever makes sense. If you still need cross platform support there are also IPC libraries that support multiple languages and help you implement protocols, like NNG or ZMQ.

Overall it’s all about trading complexity for capability. By adding complexity to your system you can gain specific capabilities (like performance, or scale, or flexibility). But complexity makes a system harder to implement correctly and harder to maintain, so you want to aim for the simplest tool that can handle your requirements.

1 Like

Thank you!