How to implement logging properly?

Hi,

I have several modules, each with a class. I want to be able to turn off the messages for each module/class independently, so that my screen does not get flooded with messages or that I can see the messages that matter.

My idea is:

  1. Create a logger instance for each class and make it a class attribute (not instance attribute)

  2. Create a logger manager class to store all those attributes. This class would be like a logger factory and would also enable me to set logging levels as I wish from my main function.

Of course the class would have to be something like a singleton, and shared across all the code. Is this a good design? Is there a better way to do this?

Thanks.

That’s logically what I have done for my non-trivia apps.

I’m guessing that you are thinking to process a command line arg like --debug=topic1,topic2 with your scheme to only have the debug messages you want and indeed a --version=topic3,topic4 etc.

I have found that some of my classes benefit from multiple log streams that I can control. I have a mapping of names to loggers that the UI can use with the --debug and --verbose options.

1 Like

The logging module already acts like the singleton factory you want. logging.getLogger takes a string argument, and always returns the same instance of Logger whenever the same argument is used. This name can be used by logging.fileConfig and logging.dictConfig for logger configuration, and the names live in a hierarchical namespace, e.g., configuration applied to getLogger("a") applies to getLogger("a.b"), getLogger("a.c"), etc.

1 Like

Dear Clint,

Thanks for your reply. I am currently using logzero and I imagine that it is also a factory of loggers like logging. However, wouldn’t it be better to have a kind of interface to the logger, such that for instance, we can switch loggers more easily in the future or have a central place where our loggers are configured for the whole project? I was thinking of something like:

import logzero

from logzero import logger as log
#------------------------------------------------------------
class log_store:
    d_logger = {}
    d_levels = {}
    @staticmethod
    def add_logger(name=None):
        if name is None:
            log.error(f'Logger name missing')
            raise
        elif name in log_store.d_logger:
            log.error(f'Logger name {name} already found')
            raise

        level          = logzero.INFO if name not in log_store.d_levels else log_store.d_levels[name] 
        logger         = logzero.setup_logger(name=name, level=level)
        log_store.d_logger[name] = logger

        return logger

    @staticmethod
    def set_level(name, value):
        log_store.d_levels[name] = value

    @staticmethod
    def show_loggers():
        log.info(f'{"Name":<20}{"Level":<20}')
        for name, logger in log_store.d_logger.items():
            log.info(f'{name:<20}{logger.level:<20}')
#------------------------------------------------------------

logzero already implements such a store behind the scenes, just like logging. (In fact, logzero appears to be an interface to the logging module, and probably lets logging manage the store.)

When you call logzero.setup_logger("foo"), it returns a logger bound to the name "foo", creating it first if necessary.

2 Likes

Are you sure that is a good idea? Does it not entangle the logging implementation too much with those classes and lead to a lot of reduplicated boilerplate code? Perhaps not – perhaps in your project it really makes sense to do this, but in the projects I worked on I have always preferred having just one logstream with a uniform format.

Logging itself is a prime example of a cross-cutting concern. So, imo, it’s good to actually also treat it that way in the implementation – which implies not tieing it (too closely) with particular business logic. Just reading a bit about aspect-oriented programming might give you more ideas about how to proceed.

2 Likes

Dear Hans,

Thank you for your reply and the links. My idea was that maybe it’s a good idea to have some sort of interface to the logger, such that if we want to move to a different library (e.g. because the old one is not maintained anymore) we could just change things in the interface instead of everywhere in the code. However I am not an expert and I will go through your links and consider this more carefully.

That smacks of premature abstraction. Worry about that when you change logging library/frameworks; the amount of work needed may not even be that much.

1 Like

Dear Clint,

I think logzero is not been maintained anymore, from what I see here:

if I were to use logzero directly, I would have to remove it in each file. While if I use it through an interface, I can remove it only in that interface. I think that’s another reason why having some sort of wrapper/interface is better.

Cheers.

Doesn’t the standard library’s logging module already provide the wrapper/interface you’re looking for? The docs for logzero say that it provides “a fully configured standard Python logger object”, so you should be able to switch to the built-in logging module without changing any of the actual log statements.

If you’re using the import statement suggested in the docs for logzero,

from logzero import logger

you would have to change it to the standard pattern,

import logging
logger = logging.getLogger(__name__)

but that should be it, it seems to me.

If you then want to plug in another logging library that doesn’t provide compatible logger objects, you could probably write a logging.Handler to send the messages from your standard logger objects to the new library.

The Advanced Logging Tutorial looks like a pretty good explanation of how all this is intended to fit together.

Dear Henrik,

Thanks for your comment.

I am not sure if in practice it would be that simple. If you stick to a logging library for years and leave logger lines all over the place and multiple people are touching the code and importing different things in different ways (e.g. instead someone could have used from logzero import logger as log or import logzero as lzr then use lzr.logger, etc) in hundreds of files. Furthermore, I havent really gone through logzero’s code, so the only guarantee that we have tha the interface of logzero and logging are the same is the:

line that you quote from their docs. So I would take that with a grain of salt.

Initially the idea was that I needed something better than logging that I would not have to configure myself, because I was pretty busy doing other things and I needed something that was already built (also I wasn’t an expert on it), designed in a more friendly way and ready to use. At that time logzero seemed like a good alternative.

I think loggers are special. In the sense that you do not use them in one file, but all over the place. Whatever is used all over the place, IMHO, should be wrapped in a class that acts as an interface. Even if the interface does not do anything. That way if something unforseen happens in the future, you would only change that interface that is fully under your control. In any case, that’s how I think right now, I might change my mind in a few years :smile:

Cheers.