Background
In Python’s logging module, the TimedRotatingFileHandler rotate log files based on time intervals. It renames the current log file to a backup file when the specified time point is reached, and then creates a new log file to continue logging for the next period.
However, if users’ program has multiple processes, this rotation method can lead to multiple processes attempting to rename the log file simultaneously. Since the log file may be occupied by other processes for each individual process, this can lead to failures in file renaming and loss of logs for the new period.
Proposal
Can we have a new log handler which could be named TimedMovingFileHandler? The core idea is to avoid renaming files during rotation and instead directly create a new log file for logging in the next period.
This is my preliminary implementation of TimedMovingFileHandler based on the TimedRotatingFileHandler. The code changes are in these two commits: commit1, commit2. The first commit involves only copying the code of TimedRotatingFileHandler class and the second shows all the modifications to implement TimedMovingFileHandler.
Is this new log handler necessary to be added? Is moving rather than rotating good enough to aovid the logging problem in multiprocessing cases?