# Preserve unpacked \`extra\` in LogRecord

**URL:** <https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630>\
**Category:** Ideas\
**Created:** [May 9, 2022, 6:29pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630 "2022-05-09T18:29:40Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![adriangb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/adriangb/32/6992_2.png) [@adriangb](https://discuss.python.org/u/adriangb)\
**Post date:** [May 9, 2022, 6:29pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/1 "2022-05-09T18:29:40Z")

</div>

The logging module has some support for structured logs via the `extra` parameter (`logger.info(..., extra={"foo": "bar"})`), but it is handicapped by the unpacking of `extra` into `LogRecord. __dict__ `.

Two issues this leads to:

1. Every library needs to write code to separate user defined attributes from built-in ones (eg: [python-json-logger](https://github.com/madzak/python-json-logger/blob/838939b72a521f07eebab6d67dbeeb9041f29130/src/pythonjsonlogger/jsonlogger.py#L20) or [opentelmetry-sdk](https://github.com/open-telemetry/opentelemetry-python/blob/606d535551c14fbc5a68921d485be35ddb5a6ce2/opentelemetry-sdk/src/opentelemetry/sdk/_logs/ __init__.py#L277))
2. Keys that are not valid Python identifiers still get upacked, which is in general not great.

Since a large fraction of logging is now structured and meant to ingestion into various logging services (vs. being read from the console by a human), good support for structured logs is more important than ever.

I think the easiest step forward would be to check if there is an `" __extra__"` (or something like this) key and, if not, safe a reference to whatever the user passed in.  
Then filters / formatters / etc. could access `LogRecord. __extra__ ` without logic to separate out attributes and without having to access `LogRecord. __dict__ ` directly to support non-valid identifiers.

---

<div class="post-metadata">

**Author:** ![vsajip](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vsajip/32/44_2.png) [@vsajip](https://discuss.python.org/u/vsajip)\
**Post date:** [June 20, 2022, 9:43am UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/2 "2022-06-20T09:43:25Z")

</div>

What about if you pass `extra={'structured_data': {'foo': 'bar'}}`, for instance? Then a `Formatter` can certainly format stuff however it wants, and doesn’t _need_ to work at separating out built-in attributes from user-specified ones.

---

<div class="post-metadata">

**Author:** ![adriangb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/adriangb/32/6992_2.png) [@adriangb](https://discuss.python.org/u/adriangb)\
**Post date:** [June 20, 2022, 2:23pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/3 "2022-06-20T14:23:18Z")

</div>

I’ve used the approach before. There’s two issues you run into:

1. We all have to agree on `structured_data` as _the_ key. If you switch logging formaters, you have to replace every logging call.
2. It adds extra boilerplate to every logging call (and I’d argue the `extra=` is already quite boilerplatey) and instantiation of a dict (small but non-zero performance penalty).

---

<div class="post-metadata">

**Author:** ![vsajip](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vsajip/32/44_2.png) [@vsajip](https://discuss.python.org/u/vsajip)\
**Post date:** [June 20, 2022, 3:33pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/4 "2022-06-20T15:33:36Z")

</div>

I don’t know to what extent these are showstoppers. When it comes to structured logging, people use different libraries anyway (e.g. `structlog`), so the ship has perhaps sailed to standardize anything, anyway, outside of a specific group of collaborators who can agree a common approach. And the logging cookbook describes [yet another approach to structured logging](https://docs.python.org/3/howto/logging-cookbook.html#implementing-structured-logging). As to performance, returning `LogRecord`s from filters is probably a bigger performance hit than instantiating a dictionary 😉 You know what they say about premature optimization, right?

---

<div class="post-metadata">

**Author:** ![adriangb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/adriangb/32/6992_2.png) [@adriangb](https://discuss.python.org/u/adriangb)\
**Post date:** [June 20, 2022, 4:43pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/5 "2022-06-20T16:43:18Z")

</div>

> [@vsajip](#):
>
> I don’t know to what extent these are showstoppers.

I think the first point (coming up with an agreed upon key that is not codified anywhere) is kind of a show stopper because it makes it very difficult to build any sort of ecosystem / interoperability. Sure it will work within a group of collaborators / one org but not for the wider ecosystem.

> [@vsajip](#):
>
> As to performance, returning `LogRecord`s from filters is probably a bigger performance hit than instantiating a dictionary 😉 You know what they say about premature optimization, right?

I don’t think it’s fair to point at another relatively unrelated change made to justify a solution that requires an extra instantiation. That said, I do fundamentally agree with you: the creation of an extra dict is not argument enough for a new feature, at least from a performance standpoint.

> [@vsajip](#):
>
> people use different libraries anyway (e.g. `structlog`), so the ship has perhaps sailed to standardize anything

I have to disagree on this. I think we should be able to evolve the standard library to continue to provide value into the future. That’s why I focused a large part of the feature request on the increasing popularity of structured logging. There is a long precedent for the standard library learning from external modules, and I think this is a situation where we should do just that. For example, `structlog` allows the syntax `logger.info("msg", user="foo")`. I would greatly prefer that to the `extra=` syntax but since that is not possible I am trying to find a way to get as close as we can to what `structlog` provides without breaking existing use cases. In particular, I think that `LogRecord.extra` (or some other name) would make life a lot easier for `python-json-logger` or `opentelemetry-sdk` to process and interact with structured log records.

---

<div class="post-metadata">

**Author:** ![vsajip](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vsajip/32/44_2.png) [@vsajip](https://discuss.python.org/u/vsajip)\
**Post date:** [June 20, 2022, 5:04pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/6 "2022-06-20T17:04:22Z")

</div>

> [@adriangb](#):
>
> I don’t think it’s fair to point at another relatively unrelated change made to justify a solution that requires an extra instantiation.

That wasn’t my point - I was just kidding, hence the 😉 The serious point is that the performance cost of an instantiation is really not something to be worried about, given that `logging` is generally “fast enough”. But you get that, as you say as much in the same paragraph.

Notice that any change has to be backwards-compatible, and that is the challenge. Also, I am not really sure to what extent people are relying on the stdlib adding features in this area. I’m not saying that a lot of logging isn’t structured - I know it is - just that it isn’t clear there’s a burning need to make improvements in this area. Is there some sort of forum where structured logging people hang out where this can be discussed in a focused way? If so, have you or anyone else started a discussion about this which I can look at? This forum might be too general.

---

<div class="post-metadata">

**Author:** ![adriangb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/adriangb/32/6992_2.png) [@adriangb](https://discuss.python.org/u/adriangb)\
**Post date:** [June 20, 2022, 5:24pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/7 "2022-06-20T17:24:10Z")

</div>

I don’t think there is such a forum, at least not that I know of. Maybe we can include @hynek in this discussion?

> [@vsajip](#):
>
> Also, I am not really sure to what extent people are relying on the stdlib adding features in this area

I admit I haven’t done any sort of polling in this area, but I’d think that given popularity of structured logging and the existence of libraries like `python-json-logger` (which has ~ the same number of ⭐ as `structlog`) there is a desire from the community to be able to do structured logging within the standard library.

> [@vsajip](#):
>
> Notice that any change has to be backwards-compatible, and that is the challenge.

Agreed! This is going to be the hard part. Do you think that adding `LogRecord.extra` would be backwards compatible? It would look something like (somewhere in `LogRecord` or `Logger`):

```python
if "extra" not in extra:
    extra["extra"] = extra

```

---

<div class="post-metadata">

**Author:** ![aathan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aathan/32/7264_2.png) [@aathan](https://discuss.python.org/u/aathan)\
**Post date:** [June 20, 2022, 7:57pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/8 "2022-06-20T19:57:57Z")

</div>

Adrian, you may want to read this thread as well: [LogRecord construction: Why is extra not passed? - #12 by aathan](https://discuss.python.org/t/logrecord-construction-why-is-extra-not-passed/16673/12)

Like you, I would like to see LogRecord construction be more flexible than the current system – specifically, that LogRecord creation _at the logging site_ should accessible to users of the logging framework in ways that are on par with the logging framework itself.

In fact, the suggestion I made would allow you to do what you want.

Also I completely agree with your sentiments about improving stdlib and Python’s logging module. Structured logging is an important modern facility. Rust’s Tracing crate may be of interest there.

---

<div class="post-metadata">

**Author:** ![aathan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aathan/32/7264_2.png) [@aathan](https://discuss.python.org/u/aathan)\
**Post date:** [June 20, 2022, 8:06pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/9 "2022-06-20T20:06:15Z")

</div>

PS: In case you have not already done so, a way to achieve your goal is to monkeypatch Logger.makeRecord() or by subclassing Logger. If you want me to I’ll post a snippet; but I’m pretty sure you’ve thought of that.

In general it seems to me that instead of putting the “extra” handling in Logger.makeRecord() this should be left to the LogRecord factory or its dunder init perhaps with the inclusion of a utility method to do the current dunder dict key/value copying. Then, libraries could construct LogRecords as they see fit _and_ be able to do so in the context of the actual logging callsite.

---

<div class="post-metadata">

**Author:** ![adriangb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/adriangb/32/6992_2.png) [@adriangb](https://discuss.python.org/u/adriangb)\
**Post date:** [June 28, 2022, 8:18pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/10 "2022-06-28T20:18:49Z")

</div>

> [@adriangb](#):
>
> Do you think that adding `LogRecord.extra` would be backwards compatible?

@vsajip just checking back in on this, I think this is the key question here

---

<div class="post-metadata">

**Author:** ![vsajip](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vsajip/32/44_2.png) [@vsajip](https://discuss.python.org/u/vsajip)\
**Post date:** [June 28, 2022, 9:11pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/11 "2022-06-28T21:11:43Z")

</div>

I’ve been away and not been able to spend any time on this, but I hope to have some free time next week.

---

<div class="post-metadata">

**Author:** ![adriangb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/adriangb/32/6992_2.png) [@adriangb](https://discuss.python.org/u/adriangb)\
**Post date:** [June 28, 2022, 9:22pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/12 "2022-06-28T21:22:29Z")

</div>

No rush, sorry to ping you while you were away 😄

---

<div class="post-metadata">

**Author:** ![pwd](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pwd/32/661_2.png) [@pwd](https://discuss.python.org/u/pwd)\
**Post date:** [June 11, 2025, 2:04pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/13 "2025-06-11T14:04:33Z")

</div>

Not passing `extra` to record factory and instead handling it directly in `makeRecord()` is really unfortunate.  
@vsajip What do you think about modifying `makeRecord()` as below to give record factory access to `extra` argument and to allow it to suppress default handling of this argument?

```diff
     rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
- sinfo)
- if extra is not None:
+ sinfo, extra)
+ if extra is not None and not hasattr(rv, "skip_extra"):
         for key in extra:
             if (key in ["message", "asctime"]) or (key in rv. __dict__ ):
                 raise KeyError("Attempt to overwrite %r in LogRecord" % key)

```

---

<div class="post-metadata">

**Author:** ![vsajip](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vsajip/32/44_2.png) [@vsajip](https://discuss.python.org/u/vsajip)\
**Post date:** [June 25, 2025, 6:36pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/14 "2025-06-25T18:36:35Z")

</div>

Unfortunately, this approach seems to not be backwards compatible. If a user out there has a custom `LogRecord` factory, their factory callable will not be expecting that extra argument and will fail with a `TypeError`. Separately, using `hasattr` leads to the odd situation that if `rv.skip_extra` is `False`, the logic still skips `extra`! This is counter-intuitive. In any case, the backwards compatibility problem remains.

---

<div class="post-metadata">

**Author:** ![pwd](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pwd/32/661_2.png) [@pwd](https://discuss.python.org/u/pwd)\
**Post date:** [June 25, 2025, 7:00pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/15 "2025-06-25T19:00:52Z")

</div>

@vsajip How about this?

```diff
     rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
- sinfo)
- if extra is not None:
+ sinfo, extra=None)
+ if extra is not None and not getattr(rv, "skip_extra", False):
         for key in extra:
             if (key in ["message", "asctime"]) or (key in rv. __dict__ ):
                 raise KeyError("Attempt to overwrite %r in LogRecord" % key)

```

---

<div class="post-metadata">

**Author:** ![vsajip](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vsajip/32/44_2.png) [@vsajip](https://discuss.python.org/u/vsajip)\
**Post date:** [June 25, 2025, 9:01pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/16 "2025-06-25T21:01:15Z")

</div>

> [@pwd](#):
>
> How about this?

Then you haven’t actually passed the `extra` value, have you?

Look, I’m just uncomfortable about messing with stuff that could break backwards compatibility. I’ve just spent much of the day reverting a change (which had no API changes - just internal stuff) because someone convinced me it wouldn’t cause a problem. Guess what - some users have reported regressions with their code, so the changes have had to be reverted.

On the other hand, you could subclass `Logger` and override `makeRecord` - there are documented mechanisms for doing this. Why is this not an option for you? Notice that the logging cookbook documents a way to achieve structured logging, why is that not a possible approach you can use? I’m not opposed to accepting changes into `logging`, as a check of the history of the source for that package will confirm. Maybe I’ve missed some discussion as to why this functionality _has_ to be in the stdlib, and can’t use the approaches I’ve mentioned. So please outline them for me in some bullet points.

---

<div class="post-metadata">

**Author:** ![karolinepauls](https://avatars.discourse-cdn.com/v4/letter/k/58f4c7/32.png) [@karolinepauls](https://discuss.python.org/u/karolinepauls)\
**Post date:** [July 6, 2025, 5:56pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/17 "2025-07-06T17:56:51Z")

</div>

I’ve had to implement this feature in my `logstruct` library, with additional constraints of full `logging` compatibility, which also meant I couldn’t pass structured data as `extra={"structured_data": {...}` because existing tools that integrate with `logging` wouldn’t understand how to flatten my `extra` data.

It got worse since the `extra` dict cannot contain keys that conflict with `LogRecord`, like… `"name"`.

Logging ended up looking like: [src/logstruct/\_logger.py · master · Karoline Pauls / logstruct · GitLab](https://gitlab.com/karolinepauls/logstruct/-/blob/master/src/logstruct/_logger.py?ref_type=heads#L245-262)

While reading data back ended up as: [src/logstruct/\_formatters.py · master · Karoline Pauls / logstruct · GitLab](https://gitlab.com/karolinepauls/logstruct/-/blob/master/src/logstruct/_formatters.py?ref_type=heads#L33-42)

---

<div class="post-metadata">

**Author:** ![provinzkraut](https://avatars.discourse-cdn.com/v4/letter/p/c37758/32.png) [@provinzkraut](https://discuss.python.org/u/provinzkraut)\
**Post date:** [February 8, 2026, 2:50pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/18 "2026-02-08T14:50:57Z")

</div>

> [@vsajip](#):
>
> On the other hand, you could subclass `Logger` and override `makeRecord` - there are documented mechanisms for doing this. Why is this not an option for you? Notice that the logging cookbook documents a way to achieve structured logging, why is that not a possible approach you can use?

Approaches with subclassing / patching work (well) when you’re dealing with applications. If you’re writing a library, those aren’t a great option, because a user of that library would have to then update _their_ application to use your subclassed things everywhere.

I came across this issue because I am working on a library and trying to make it more agnostic to the logging environment it’s running in, specifically, supporting both stdlib logging and structlog. To that end, it would be very convenient if a library could just do `Logger.info("message", extra={"some_extra_key": "some_extra_value"})`, as you can easily make this compatible with both structlog and stdlib logging; Simply provide a formatter for your logger that understands the extras, and a processor for structlog that transforms it into structlog’s expected format.

You _can_ already sorta do this, by using the “magic key” approach shown above, but then all your logging calls end up looking like this:  
`Logger.info("message", extra={"very_special_key":{"attribute": "value"}})` instead of `Logger.info("message", extra={"attribute": "value"})` or ideally `Logger.info("message", attribute="value")`.

The goal would be to have an interface provided by `logging` that’s agnostic about how its records are processed. If all information passed in is preserved, any application/library can pass in stuff it deems relevant, and any logging library, be it structured or unstructured, would be able to hook into the configured loggers and format them as it pleases.

So does this mean this problem can only be solved in the standard library? No, I guess not. You _can_ work around it, it’s just so cumbersome that it’s not a viable approach and therefore isn’t really done.

---

<div class="post-metadata">

**Author:** ![provinzkraut](https://avatars.discourse-cdn.com/v4/letter/p/c37758/32.png) [@provinzkraut](https://discuss.python.org/u/provinzkraut)\
**Post date:** [February 8, 2026, 3:03pm UTC](https://discuss.python.org/t/preserve-unpacked-extra-in-logrecord/15630/19 "2026-02-08T15:03:53Z")

</div>

As for the backwards compatibility, how about instead of using a similar interface as exception notes (PEP 0678)?

Extras passed in could be preserved under an `__extra` key (using name mangling to ensure no conflicts with existing extras currently passed in via `extra`), and set and retrieved via a `set_extra` and `get_extra` method. Current functionality could remain unchanged, i.e. records would still have their `__dict__` populated, but now anything touching a record has a safe interface to retrieve extras.

```diff
 
        if extra is not None:
+ rv.set_extra(extra)
            for key in extra:
                if (key in ["message", "asctime"]) or (key in rv. __dict__ ):
                    raise KeyError("Attempt to overwrite %r in LogRecord" % key)
                rv. __dict__ [key] = extra[key]
        return rv

```