A new feature is being added in `logging.config.dictConfig()` to configure `QueueHandler` and `QueueListener`

Python Help
1

In 3.12, I’m adding a new feature to allow configuration of logging.handlers.QueueHandler and logging.handlers.QueueListener via logging.config.dictConfig(). Here’s the relevant documentation - comments are welcome here or on the PR.

Configuring QueueHandler and QueueListener

If you want to configure a QueueHandler, noting that this is normally used in conjunction with a QueueListener, you can configure both together. After the configuration, the QueueListener instance will be available as the listener attribute of the created handler, and that in turn will be available to you using logging.getHandlerByName() and passing the name you have used for the QueueHandler in your configuration. The dictionary schema for configuring the pair is shown in the example YAML snippet below.

handlers:
  qhand:
    class: logging.handlers.QueueHandler
    queue: my.module.queue_factory
    listener: my.package.CustomListener
    handlers:
      - hand_name_1
      - hand_name_2

The queue and listener keys are optional.

If the queue key is present, the corresponding value can be one of the following:

  • An actual instance of queue.Queue or a subclass thereof. This is of course only possible if you are constructing or modifying the configuration dictionary in code.
  • A string that resolves to a callable which, when called with no arguments, returns the queue.Queue instance to use. That callable could be a queue.Queue subclass or a function which returns a suitable queue instance, such as my.module.queue_factory() .
  • A dict with a '()' key which is constructed in the usual way as discussed here in the documentation. The result of this construction should be a queue.Queue instance.

If the queue key is absent, a standard unbounded queue.Queue instance is created and used.

If the listener key is present, the corresponding value can be one of the following:

  • A subclass of logging.handlers.QueueListener. This is of course only possible if you are constructing or modifying the configuration dictionary in code.
  • A string which resolves to a class which is a subclass of QueueListener , such as 'my.package.CustomListener' .
  • A dict with a '()' key which is constructed in the usual way as discussed here in the documentation. The result of this construction should be a callable with the same signature as the QueueListener initializer.

If the listener key is absent, logging.handlers.QueueListener is used.

The values under the handlers key are the names of other handlers in the configuration (not shown in the above snippet) which will be passed to the queue listener.

Any custom queue handler and listener classes will need to be defined with the same initialization signatures as QueueHandler and QueueListener.

2 Likes
2

Thanks Vinay.

3

Is there a recommended way to setup a dict config for QueueHandlers and Listeners for python < 3.12?

I could setup a factory method which is pointed to for one of the handlers in the config which builds a QueueHandler, QueueListener, and queue that connects them and returns the QueueHandler so it can be connected to loggers, but how to connect handlers to the listener (without needing to resolve the config dicts into python objects myself in the factory method) or access the listener after configuration for start/stop?

Or is this just something that won’t be possible for python < 3.12 and I should give up and keep QueueHandler/Listener config in python code rather than a config file?

4

You could use the same basic scheme as the one I’ve added - just make it a custom handler factory (using the () key, as per the documentation) that does the configuration in code.

5

I’ve tried this but I’m not able to get it to work. I’ve created a QueueHandlerListener class in my.package:

# my.package
from logging.handlers import QueueHandler

class QueueHandlerListener(QueueHandler):
    def __init__(self, queue, handlers):
        super().__init__(queue)
        self.listener = QueueListener(queue, *handlers)

and I have logging_config.yaml:

version: 1
formatters:
    stream_formatter:
        format: '%(asctime)s | %(levelname)s | %(name)s | %(lineno)s | %(message)s'
handlers:
    stream_handler:
        class: logging.StreamHandler
        formatter: stream_formatter
    stream_queue_handler_listener:
        (): my.package.QueueHandlerListener
        queue:
            (): queue.Queue
            max_size: -1
        handlers:
            -  cfg://handlers.stream_handler
loggers:
    test_logger:
        handlers:
            - stream_queue_handler_listener

But when I configure logging by loading the yaml into a dict and using dictConfig the code runs but if I execute:

test_logger = logging.getLogger('test_logger')
handler = test_logger.handlers[0]
queue = handler.queue
listener = handler.listener
listener_handlers = listener.handlers
print('queue:', queue)
print('listener:', listener)
print('listener_handlers:', listener_handlers)

The output is

queue: {'()': 'queue.Queue', 'max_size': -1}
listener: <logging.handlers.QueueListener object at 0x0000017F23EF4F70>
listener_handlers: ('cfg://handlers.stream_handler',)

So the queue object and handler objects passed into the listener remain as their uninitialized config dictionary entries basically. Is this the expected behavior? Am I required to parse these dictionary config objects myself in QueueHandlerListener?

6

Ok, I got something to work with

# my.package

from logging.handlers import QueueHandler
from queue import Queue


class QueueHandlerListener(QueueHandler):
    def __init__(self, *args, max_size=-1, **kwargs):
        queue = Queue(max_size)
        super().__init__(queue)
        endpoint_handlers = args + tuple(kwargs.values())
        self.listener = QueueListener(queue, *endpoint_handlers)

    def start(self):
        self.listener.start()

    def stop(self):
        self.listener.stop()

and config yaml

version: 1
formatters:
    stream_formatter:
        format: '%(asctime)s | %(levelname)s | %(name)s | %(lineno)s | %(message)s'
handlers:
    stream_handler:
        class: logging.StreamHandler
        formatter: stream_formatter
    stream_queue_handler_listener:
        (): my.package.QueueHandlerListener
        handler_0: cfg://handlers.stream_handler
        max_size: -1
loggers:
    test_logger:
        handlers:
            - stream_queue_handler_listener

An arbitrary number of endpoint handlers can be configured for the queue hander listener in the config dict. The only slight funniness is allowing endpoint handlers to be passed into the QueueHandlerListener as either args or kwargs but this isn’t even that bad.

I guess the point is that the config syntax doesn’t recurse down into input arguments. That is, the (), cfg:// and ext:// etc. syntaxes can be used to resolve inputs to formatters, handlers, and loggers, but it seems they can’t generically be nested to realize construction of input arguments.

7

How to setup only QueueHandler in a dictConfig(), and skip initiating QueueListener at all?

My usecase is that I have multiple worker processes (that logs to a QueueHandler) with dedicated background logging process (that consumes the queue).
Workers does not need QueueListeners to be configured, while logger process does not need QueueHandlers.

Now I am just create QueueHandler object, and not using this nice feature you have introduced… :⁠-⁠(

8

Don’t specify a listener in your configuration.