Proper way to create a daemon process

Python Help
1

I asked here but I figured the forum would be a better place.

Using Python 3.12+ I would like to create a daemon process that continues running after the current process exits, does not block from I/O and is immune to signals from the current process group. I have the following:

import subprocess
import sys

command = ["foo", "bar"]
if sys.platform == "win32":
    subprocess.Popen(
        command,
        creationflags=subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.DETACHED_PROCESS,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        close_fds=True,
    )
else:
    subprocess.Popen(
        command,
        start_new_session=True,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        close_fds=True,
    )

Is that sufficient or does one need to manually do that double fork technique on Linux? Perhaps there are more flags on Windows?

2

On linux can you have the daemon be a system or user systemd service? You could then start the service when needed.

3

Yes, I’m aware that is an option and is the “standard” way but I have a requirement for the short-lived process to be launched with a Python script frequently so I’m trying to figure out how to do this properly on a technical level.

4

It’s not so simple, and I don’t know if it’s the proper way, but this is how dmypy starts mypyd.

5

The instructions for creating a classic UNIX-style daemon can be found on almost any system by invoking the magic phrase…

man 7 daemon

Basically everything described there is doable directly in Python, I’ve done it myself in the past more than once. It’s maybe a dozen lines of code if you want to stick exclusively to the stdlib.

If you don’t mind reaching past the stdlib shelf, python-daemon · PyPI is quite popular and handles these tasks for you nicely too.

6

That doesn’t appear to support Windows. I suppose I could use that only on non-Windows systems but I would prefer a cross-platform approach using only the standard library (even if verbose).

Ah interesting, it looks like the startupinfo option is required on Windows as well.

7

The instructions for creating a classic UNIX-style daemon can be found on almost any system by invoking the magic phrase…

man 7 daemon

More direct to the initial question, quoting from the above:

5. Call fork(), to create a background process.

6. In the child, call setsid() to detach from any terminal and create
   an independent session.

7. In the child, call fork() again, to ensure that the daemon can never
   re-acquire a terminal again. (This is relevant if the program — and
   all its dependencies — does not carefully specify `O_NOCTTY` on each
   and every single `open()` call that might potentially open a TTY
   device node.)

8. Call exit() in the first child, so that only the second child (the
   actual daemon process) stays around. This ensures that the daemon
   process is re-parented to init/PID 1, as all daemons should be.

That part describes the reasons for the double-fork, and terminating the child in order to be sure the grandchild is reparented.

8

Yes that is the technique I mentioned in my opening post. Does start_new_session do all that?

9

That doesn’t appear to support Windows. I suppose I could use that only on non-Windows systems but I would prefer a cross-platform approach using only the standard library (even if verbose).

Fair, I have programmed BSD-type daemons all of my adult life, but have never written software intended for use on Windows so don’t know how it might differ. From what I’ve heard over the years, I get the impression the steps for creating a traditional background system service on Windows are quite entirely different so likely require completely separate code paths anyway.

10

Upon further reading it looks like there are two ways to achieve what that code does. Are these equivalent?

subprocess.Popen(
    command,
    creationflags=(
        subprocess.CREATE_NEW_PROCESS_GROUP
        | subprocess.DETACHED_PROCESS
        | subprocess.CREATE_NO_WINDOW
    ),
    ...
)

startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
subprocess.Popen(
    command,
    creationflags=(
        subprocess.CREATE_NEW_PROCESS_GROUP
        | subprocess.DETACHED_PROCESS
    ),
    startupinfo=startupinfo,
    ...
)
11

You are going to need to explain in more detail what you need to achieve.
Is this a system process or a user process?
Do you expect to process to survive the user logging out?
What OS do you need to this to work on? Windows and what else?

12

We want to collect telemetry from developer tooling/CLIs but we want collection to have no impact on responsiveness. So my idea is to launch a daemon process that reads from a temporary directory created in the main process. The main process will write data like which command was executed, the exit code and how long it took to files inside that directory. The daemon process will wait for the required files to be written and then send the HTTP payload with support for retries and such.

The target platforms would be wherever our developers are so macOS, Windows, and a few folks on Linux.

13

Then you should be able to launch a process and not wait for it to finish.
The process will be killed when the user logs out.
No need for magic fork-dance (that’s legacy for system deamons).

I take it doing the logging in a background thread will not work for you?

14

Right, that won’t work because we are submitting the data to Datadog (ourselves) over HTTP so we don’t want to add any overhead, not even importing the submission library, and we don’t want an error to impact the main process. The main concern is having to retry the network call, I just don’t want to impact CLI timings at all because Python already isn’t the best there.

15

Yes that is the technique I mentioned in my opening post. Does start_new_session do all that?

I don’t know, I’ve only ever trusted os.fork() in that capacity, since subprocess.Popen() seems designed more for spawning children you intend to interact with rather than orphan.

16

There’s also:

which never got into the stdlib.

17

systemd has made the unix deamonise dance no longer required.

18

systemd has made the unix deamonise dance no longer required.

At least on systemd-based Linux distributions (which, to be fair, is the majority of popular Linux distributions now). Not necessarily the case on BSD derivatives, which probably includes MacOSX/Darwin (I do use BSDs a lot, but have only a passing familiarity with Darwin). I guess it’s a question of how cross-platform the cross-platform need is in this case.

19

On macOS you would use a launchd service that is logically the same as systemd, indeed launchd was one of the inspirations for systemd.

Agreed that if you care about BSD then you need to do the old school daemon dance.