Feature proposal: add a new `env://` value converter for logging configuration

Ideas
1

Feature or enhancement

Proposal:

It would be nice if Python logging config processed environment variables in a similar way cfg:// and ext:// are at the moment. For example:

import logging
import logging.config

config = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "simple": {
            "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.FileHandler",
            "level": "DEBUG",
            "formatter": "simple",
            "filename": "env://LOG_FILE",  # <---- this!
        }
    },
    "root": {
        "level": "DEBUG",
        "handlers": ["console"],
    },
}

logging.config.dictConfig(config)
logger = logging.getLogger(__name__)
logger.info("Hello, world!")

Something like this can already be implemented by users if they subclass logging.config.DictConfigurator:

import os
from logging.config import DictConfigurator


class DictConfiguratorWithEnv(DictConfigurator):
    value_converters = {
        "env": "env_convert",
        **DictConfigurator.value_converters,
    }

    def env_convert(self, value):
        return os.environ[value]

However, it’d be nice if Python had built-in support and it might be useful for configuring things like host and port for socket handlers.

I think this is a minor change if accepted, but folks might have thoughts about why this could be a bad idea.

Has this already been discussed elsewhere?

Not AFAICT.

3 Likes
2

This seems like a reasonable request to me. If nobody raises any objections within a week or so, please ping me (@zware) on the GitHub issue and I’ll reopen it.

1 Like
3

A similar thing with env vars and logging config has been discussed here: An environment variable to set a logging.config.dictConfig at startup, there’s a couple reasons not to that may overlap.

4

Thanks @mikeshardmind! I think they’re sufficiently different requests to be considered on their own.

My request here is for applications that already support a logging configuration file, being able to override certain handler parameters based on the environment.

For example, consider using the logging handler from GitHub - fluent/fluent-logger-python: A structured logger for Fluentd (Python), which is closer to the real use-case I’m imagining for this feature.

version: 1
formatters:
  fluent_fmt:
    '()': fluent.handler.FluentRecordFormatter
    format:
      level: '%(levelname)s'
      hostname: '%(hostname)s'
      where: '%(module)s.%(funcName)s'

handlers:
  fluent:
    class: fluent.handler.FluentHandler
    host: env://FLUENT_HOST
    port: env://FLUENT_PORT
    tag: test.logging
    buffer_overflow_handler: overflow_handler
    formatter: fluent_fmt
    level: DEBUG

Depending on the environment, I may not know the values of the Fluent host and port in advance, and I’d rather not write a logging configuration on the fly and just rely on values set in the environment.

Does that make more sense? Let me know what you think.

2 Likes
5

Yep, seems to make sense to me. I didn’t personally see a direct overlap, but it was extremely similar, so wanted to make sure nobody else saw a reason it overlapped with the prior concerns which wasn’t obvious to me; Would be a shame to have missed a prior discussed reason something wouldn’t be appropriate.

1 Like
6

The big difference between the two is that this is an option for how to specify a logging config, while the previous was a perhaps-too-broad mechanism for overriding any config.

Possibly this feature could be used to implement the other idea in a nice way: set up your tools to read from the environment and it’s easy to toggle on and off. I’m not sure that accomplishes the wide-ranging effect that @csm10495 wanted, but it’s closer.

1 Like
7

What happens if the env var isn’t set, does it just get an empty string?

I wonder how that would work for non string configurations. Should there be a way to say env var or X if the env var isn’t set?

8

I think erroring is the most sensible choice. But if you’re just making the point that this needs to be defined, I 100% agree.

I’d also leave the possibility of defaults out, at least for an initial version. Syntax could always be added for defaults later, if there’s evidence that it would be worth adding.

1 Like
9

Thanks for weighing in folks :pray:

I agree with @sirosen that erroring in the initial version probably makes the most sense.

Another aspect to consider is type coercion, which seems to be needed even for the first version. In my example, port has to be an integer, otherwise the user would see a TypeError.

One way I imagine supporting that is with the env://<VAR_NAME>:<TYPE> syntax, where TYPE could be str, int, bool and float in the first version.

Some examples:

  • env://HOST would return the value of the HOST environment variable as a string.
  • env://PORT:int would cast the value of the PORT environment variable to an integer with int(PORT).
  • env://DEBUG:bool would cast the value of the DEBUG environment variable to a boolean with DEBUG.lower() in ['true', '1', 'yes', 'on'] or something similar.
  • env://PI:float would cast the value of the PI environment variable to a float with float(PI).

Another option that could leave the door open to defaults in a future version is something like env://<VAR_NAME>?type=<TYPE>.

Some examples:

  • env://HOST would return the value of the HOST environment variable as a string.
  • env://PORT?type=int would cast the value of the PORT environment variable to an integer with int(PORT).
  • env://DEBUG?type=bool would cast the value of the DEBUG environment variable to a boolean with DEBUG.lower() in ['true', '1', 'yes', 'on'] or something similar.
  • env://PI?type=float would cast the value of the PI environment variable to a float with float(PI).

What do you think?

10

It all makes sense though erroring is sort of weird to me. I was thinking about environment variables like HTTP_PROXY and they often just don’t get used if not set. It’s sort of rare that a default configuration doesn’t allow environment vars to not be set.

That being said I’m not sure we really want to turn this string into a DSL of sorts. If we need that level of flexibility I start to think it shouldn’t be a string but rather an object of some sort (like a dict nested as json in the example).

Though it seems like multiple ends of the same string (metaphorically, lol) if that makes sense.