Asyncio seems to lose event loop on daemonization

I’ve got a pretty standard network protocol server that is using asyncio, but somewhere along the line of Python versions it started losing the event loop when the server daemonizes. I’m not sure exactly what version of Python that happened in, since I just noticed it after I upgraded my Debian OS. I know that it works on Python 3.7.3, but not on 3.9.2

I’m not sure exactly what I’m doing wrong. Since Python (at least to my knowledge) lacks its own implementation of Unix’ daemon(3), I had to write my own, and I’m not sure if there’s something in there that I’m doing or not doing to make asyncio lose the loop, but I do have a minimal example demontrating the problem:

import os, sys, asyncio

def daemon(nochdir=False, noclose=False):
    pid = os.fork()
    if pid != 0:
        os._exit(0)
    os.setsid()
    if not nochdir:
        os.chdir("/")
    if not noclose:
        sys.stdout.flush()
        sys.stderr.flush()
        os.close(0)
        os.close(1)
        os.close(2)
        fd = os.open("/dev/null", os.O_RDWR)
        if fd != 0: os.dup2(fd, 0)
        if fd != 1: os.dup2(fd, 1)
        if fd != 2: os.dup2(fd, 2)
        if fd >= 3: os.close(fd)

async def init():
    print(1)
    await asyncio.sleep(1)
    print(2)

async def finish():
    print(3)
    await asyncio.sleep(1)
    print(4)

async def main():
    await init()
    # loop = asyncio.get_running_loop();
    daemon(False, True)
    # asyncio._set_running_loop(loop)
    await finish()

sys.exit(asyncio.run(main()))

This crashes on the sleep call in finish, where asyncio can no longer find the event loop.

Uncommenting the commented lines in main()` seems to function as a work-around, but it is obvious to me that that’s just a hack and not the intended way to fix it. Does anyone know what the actual correct way is?

If you are on an linux OS that uses systemd you do need to do the deamon dance at all.

Or are you on a *BSD?

I’m on a Linux that doesn’t use systemd, but not all daemons are started by the init system to begin with, either, and I’d also like to be compatible with other systems. Also, in case this is related to forking in general, then I’d like to think that daemonizing isn’t the only reason to fork.

I recall that there are problems with using fork in python.

If you do deamon dance outside of python you can avoid needing to solve those issues.

On none systemd os you should find programs that will do this for you.

On systemd os i have not seen a deamon started outside of systemd in years. Desktop systems use systemd user service manager for the iser specific deamons.

My debian systems are all systemd, odd yours are not systemd as well.

On systemd os i have not seen a deamon started outside of systemd in years.

In this particular case it is a daemon that I most commonly run manually, quite simply. I start it via init scripts as well, but my most common use is to start it from the shell, as part of a custom development environment.

I recall that there are problems with using fork in python.

How so? I would love to learn more. I’ve been using fork and daemonization for years in Python, and this is the first problem I’ve had with it. The reason I suspected it to have with forking to do is because I was browsing through the asyncio source code and noted that it seemed to be doing some sort of PID-based caching, but I’m also not sure how relevant that is since the code in question is overridden by native replacements that I haven’t looked into yet.

I am curious if this also makes asyncio break when using Python’s multiprocessing. I haven’t used multiprocessing myself so I’m not very familiar with it, but if it doesn’t break I’d be curious to know how it avoids it.

My debian systems are all systemd, odd yours are not systemd as well.

Off-topic, but the reason is simple, I’ve uninstalled systemd from them since I’m not very fond of it.

For dev I would use & to put in background or use nohup if I’m going to lose the terminal.

I’m not sure of the details, its got to do with threads and locks being undefined after fork. Maybe other issues as well.

Are there thread running before you fork?
Try pausing the process before you fork and looking at it with debug tools to see if its got threads running.

Off-topic: I find that SysV init is hard to keep working and missing features.
systemd is, I find, easy to keep working and write robust reliable services with.
I do that as my day job for a cloud environment.

I’m not sure of the details, its got to do with threads and locks being undefined after fork. Maybe other issues as well.

Ah, yes, but that’s a general problem with forking in multithreaded programs, and not just limited to Python. The program in question isn’t multithreaded to begin with, and as you can see, the minimal example in the OP that exhibits the problem isn’t multithreaded either.

I wonder if async uses any worked threads that are not obvious.
That’s worth checking for.

Also I would change your code to do this and see if that helps.

daemon(False, True)
sys.exit(asyncio.run(main()))

Does that work?

Yes, that absolutely works, since no event-loop exists before main. The problem appears to be losing a current event-loop, not initially creating one in a forked process. The problem is, however, that I want to daemonize after initialization is complete, and I set up asyncio tasks as part of initialization, so moving daemonization to before initialization is not a solution for me.

Also, I checked (with strace, and it does not appear that asyncio creates and hidden worker threads.

Would it work if you consider your initialization and main running to be completely independent? After all, they’re going to be running in separate processes anyway. Something like:

asyncio.run(init())
daemon(False, True)
sys.exit(asyncio.run(main()))

Would it work if you consider your initialization and main running to be completely independent?

Depends on what you mean by “independent”, perhaps. An important part of the initialization procedure is to set up async tasks that should continue running after daemonization. Most importantly, an asyncio.start_server.

As far as I’m able to tell, following your example with two calls to asyncio.run() seems to just throw away any tasks started in the first call to run once it returns, but perhaps this is where my relative inexperience with asyncio comes in. I’m still not particularly sure just how event loops and tasks are managed and tracked, so I can’t say I’m particularly sure if there’s an obvious way to avoid that problem.

Yeah, then no, it’s not possible to make them imdependent. Oh well, was worth a try.

Is there an FD that is used by async that does not survive the fork?
Does strace show the child closing any FDs?

Is there an FD that is used by async that does not survive the fork?
Does strace show the child closing any FDs?

Not as far as I can tell. asyncio indeed does create and use an epoll FD, but it stays open as it should, and strace shows it successfully using it after the fork.

Actually, it might be possible. The following situation does appear to work as intended:

    with asyncio.Runner() as r:
        r.run(init())
        daemon(False, True)
        r.run(finish())

Using a Runner in this way seems to at least reuse the same event-loop before and after the fork, meaning tasks from init continue to run after the fork.
I’d technically prefer not having to split the program up in this way, but this does seem to be a functional workaround. I’ll verify later whether it works with the full program.

I’ll verify later whether it works with the full program.

Can confirm that it indeed did work.