Python logging RotatingFileHandler writes to random file after the first log rotation

Python Help
1

I am using python logging.handlers.RotatingFileHandler for my log rotation. It is working fine and writting logs to let’s say app.log file but after the first rotation it doesn’t write to the app.log file all the time. sometimes it start writting to app.log.1 or any other file like that and also it rotates the file too early even though the max byte has not reached. The correct behaviour should be to write to the app.log file alawys. And i want that behaviour only . I have multiple module and every module write to a separate file. Every module use the below method to get its logger by passing log_name and log_file_name.

def get_logger(log_name, log_filename):
    log = logging.getLogger(log_name)
    logs_file = os.path.join(log_path, log_filename)
    log.setLevel(logging.DEBUG)
    file_formatter = logging.Formatter('%(asctime)s - %(pathname)s - %(levelname)s - %(message)s')
    file_handler = logging.handlers.RotatingFileHandler(logs_file, maxBytes=int(1.5e7), backupCount=15)
    file_handler.setFormatter(file_formatter)
    file_handler.setLevel(logging.DEBUG)
    log.addHandler(file_handler)
    return log

If anyone have face this kind of issue, what solution did you apply.

2

In this very similar question the problem was multithreading. Are you using multi-threading/-processing?

2 Likes
3

Do you know the solution. How to fix it?
Thanks in advance.

4

As mentioned in the other thread, make sure to only ever create one RotatinFileHandler per file name & logger. Currently if get_logger is called multiple times, potentially from different threads, multiple instances of RotatingFileHandler are created. You should have some kind of (thread-safe) cache here.

1 Like
5

So i have my api that get’s called by the user. And then api call some method of the module and module has this line at the top logger = get_logger(log_name=__file__, log_filename='app.log'). Do you think for every api call the module could create different rotating handler?

6

No, you should only ever create one RotatingFileHandler with the same logs_file argument. You shouldn’t attach handlers to each individual logger, but preferably only to the root logger, see this video for a more in-depth tutorial on how to handle logging.

In the original post you claim every module write to a separate file, but you are now suggesting that they are all writing to app.log. Both are valid, but you should make sure that you know what is supposed to happen.

2 Likes
7

Sorry if i was not able convery properly. But i took app.log for just refrence. Indeed every module has different log_filename. For eg: module1 would have module1.log filename and module2 would have module2.log like this for every module.