Add a New Log Handler: `TimedMovingFileHandler`

Ideas
1

Background

In Python’s logging module, the TimedRotatingFileHandler rotate log files based on time intervals. It renames the current log file to a backup file when the specified time point is reached, and then creates a new log file to continue logging for the next period.

However, if users’ program has multiple processes, this rotation method can lead to multiple processes attempting to rename the log file simultaneously. Since the log file may be occupied by other processes for each individual process, this can lead to failures in file renaming and loss of logs for the new period.

Proposal

Can we have a new log handler which could be named TimedMovingFileHandler? The core idea is to avoid renaming files during rotation and instead directly create a new log file for logging in the next period.

This is my preliminary implementation of TimedMovingFileHandler based on the TimedRotatingFileHandler. The code changes are in these two commits: commit1, commit2. The first commit involves only copying the code of TimedRotatingFileHandler class and the second shows all the modifications to implement TimedMovingFileHandler.

Is this new log handler necessary to be added? Is moving rather than rotating good enough to aovid the logging problem in multiprocessing cases?

1 Like
2

I like this idea, though I think the ext option in the linked implementation is not necessary

1 Like
3

Having multiple processes write to the same log file is risky. You could end up with half writes or lines that end up written in the middle of other lines.

Even if you do have some sort of lock mechanism as soon as a rotation occurs all your processes will be writing to different files. There’s no guarantee that they rotate at /exactly/ the same time and thus generate /exactly/ the same timestamp.

Maybe I’m missing something, but you might want to re-think your logging. Perhaps have a single processes solely responsible for logging and send it messages?

1 Like