PEP 837: Extensible JSON serialization

Core Development
1

I have submitted PEP 837 (PR python/peps#5031 until it renders), which proposes an extension mechanism for the json encoder. It is a revival and rework of an idea with a long history — the implementation descends from my 2017 patch on gh-71549, and the idea itself has been proposed independently since 2010.

In fact, I published a proof-of-concept patch on the tracker in 2017 and have had a full implementation in my local branches since later that year. For this proposal I reworked some design details and wrote the documentation. I tried to write a PEP earlier, but it was too difficult for me — this time I finally managed to finish it.

Summary

Serialization of custom types can be customized at three levels, each covering a different need:

  • Library level — a class defines the JSON representation of its instances with a __json__() method. No imports, inherited by subclasses:

    class Money:
    def __json__(self):
        return {"amount": str(self.amount), "currency": self.currency}

  • Application level — a serialization function for a type you do not control is registered with copyreg.json(), following the copyreg.pickle() precedent. The copyreg.RawJSON wrapper emits an already encoded fragment verbatim, so an application can finally choose how Decimal serializes:

    copyreg.json(decimal.Decimal, str)                              # "1.10"
copyreg.json(decimal.Decimal, lambda d: copyreg.RawJSON(str(d)))  # 1.10

  • Call level — a particular encoder overrides the global registry with a dispatch_table attribute, mirroring pickle.Pickler.dispatch_table. default= is unchanged and runs last.

The more specific level takes precedence. deque, mappingproxy, ChainMap, UserDict, UserList and UserString — containers with an unambiguous JSON form — serialize out of the box. Exact basic types pay no overhead: the hooks are only consulted for objects that would otherwise head toward the isinstance checks, default= or TypeError.

Reference implementation: python/cpython#153607 (both the C and Python encoders, tests, documentation).

Deliberate non-goals

To keep the discussion focused, a few things are intentionally not in the PEP:

  • No default __json__ for Decimal, namedtuple, array.array or datetime — each has at least two legitimate JSON representations (number vs string; array vs object; array vs encoded string; many date formats). The registry exists precisely so the application declares this policy; the standard library should not guess.
  • Dict keys are not covered — the hooks apply to values only. A compatible extension for keys is sketched in Open Issues.
  • copyreg.copy() / copyreg.deepcopy() for __copy__-like hooks fit the same pattern but will be a separate proposal.

Prior discussions

This idea has come up many times; I have tried to address the points raised in all of them (see the PEP’s Motivation and Rejected Ideas):

The ecosystem already uses this convention informally: TurboGears serializes via a no-argument __json__() (with custom_encoders taking precedence over it — the same ordering as this PEP), Pyramid’s renderer calls __json__(request), simplejson added for_json() (choosing that name because dunders are reserved for the language), and projects like conda, Checkmk and TatSu define __json__ today.

Feedback particularly welcome on

  1. The registry’s home: copyreg.json() versus a function in the json module (the PEP argues for copyreg on import cost, pickle symmetry and neutrality for third-party encoders).
  2. The duck-typed __raw_json__ as the raw-output marker (see the alternatives in Rejected Ideas and the __str__-payload variant in Open Issues).
  3. Whether dict-key customization should be deferred (as proposed) or included.
26 Likes
Add copyreg.copy() and copyreg.deepcopy()
2

This will be unpleasant for ujson which already defines __json__ as returning JSON (akin to the PEP’s proposed __raw_json__) rather than just generic types that stdlib json supports…

If we’re going to have a dunder to reduce an object to widely serializable types then I’d still prefer to call it as such (__to_serializable__? __to_simple_types__?) then toml/YAML/msgpack/plist can all join in.

4 Likes
3

I think it’s a good choice to place registry’s home in copyreg, that is symmetry, and symmetry is grace

4

Wery bad for ujson which defines __json__ in incompatible way. First, dunders are reserved for Python core and stdlib. Second, __json__ with semantic like in the proposed PEP was discussed and implemented in other software years ago. ujson already had compatibility issues.

5 Likes
5

Quick thoughts - I like it, and I like that it mirrors pickle in nice ways. copyreg feels like the right choice for me, but does it need to be copyreg.RawJSON or can we just have copyreg.Raw (still with __json__ as described, and also other formats)? Perhaps __json_raw__() instead of __raw_json__() just to keep them together in lists?

Might be worth explicitly marking deserialization as out of scope/deferred, though I might be the only one to bring it up in the discussion. Still, if we’re going to have a JSON-related registry, and I imagine deserialization would require passing object (dicts) through a registry until one claimed it, it’s probably worth keeping it half in mind so we can reuse the same registry later on.

1 Like
6

RawJSON is the name of the class with the same function in simplejson (it does not have __raw_json__, it is tested by identity, which we want to avoid). It would be better to preserve it. And with such class name __raw_json__ is the most natural name for dunder, alternative order would confuse.

Deserialization is completely out of scope. In belongs to other level. There is no and cannot be any general approach. BTW, I am planning to try to implement a SAX parser for JSON, this might (or not) help to build external deserializer.

1 Like
7

Several standard library container types whose JSON representation is unambiguous — collections.deque, types.MappingProxyType, collections.ChainMap, collections.UserDict, collections.UserList, and collections.UserString — gain a __json__ method and therefore serialize out of the box.

I think dataclasses should also receive a default __json__ implementation that simply calls dataclasses.asdict(dataclass). As prior art, both Pydantic and msgspec serialize dataclasses as JSON objects[1][2]. Rust’s serde_json also serializes structs as JSON objects.[3]

Reusing __repr__/__str__ or a generic __serialize__
Proposed in cpython-issue:114285. These methods cannot serve as the marker for raw output: nearly every type defines them (and their results are usually not valid JSON), so the encoder could not distinguish raw-capable objects from others; a format-agnostic __serialize__ likewise cannot indicate which format it produces. __str__ can, however, serve as the payload once a separate marker exists — see Open Issues.

I still believe a generic __serialize__(self, serializer: SerializerProtocol) -> None that serializes the object into primitive types is the better approach (a la serde’s Serializer trait). However, in the absence of such a method, I’m +1 on __json__ in the meantime, mostly because it doesn’t block the introduction of a future __serialize__.

  1. JSON | Pydantic Docs ↩︎

  2. Supported Types ↩︎

  3. Structs and enums in JSON · Serde ↩︎

4 Likes
8

I haven’t looked into this PEP yet, but my gut reaction is similar. The reasoning is that we can’t anticipate all the serialization protocols out there now or in the future, and we’re not going to keep adding special dunder methods for each one.

I do urge folks to think about how to generalize this so that it’s adaptable to other 3rd party serializations and extensible for future, unknown protocols too.

7 Likes
9

I’d actually strongly recommend not adding __json__ to dataclasses this way. dataclasses.asdict does a lot of extra work that isn’t needed for __json__ and is slow. It recurses into other objects and makes deep copies of every unknown thing it encounters which is completely unnecessary if the goal is to subsequently convert everything to a JSON string.

For __json__ it would make more sense to create a (lazily) generated method that doesn’t recurse itself. It could then also write out the dict explicitly rather than needing to loop through fields at runtime which would make serialization significantly faster.

2 Likes
10

I do find it a bit confusing that __json__() methods output python objects instead of strings. But most libraries (outside of ujson) appear to have settled on that approach, and I guess it does help with uniformly applying the indent, as well as the other kwargs of json.dumps().

(Part of the issue for me here is that Decimal does not in fact have 2 legitimate __json__() representations as suggested in the OP[1] – it has 2 legitimate __raw_json__() representations, one of which is impossible to non-buggily convert to a __json__() equivalent.)

My intuition is that the right way to write a __json__() function for a dataclass is along the lines of

def __json__(self):
    return {
        attr: json.dump(value)
        for attr, value in self.__dict__.items()
    }

ie it would be normal to “recurse” into the __json__() method of other objects.

I’m not 100% sure whether the PEP text I read would encourage or discourage such an approach.

  1. specifically the original post text. The author of the original post does seem to be aware of that, since this problem was mentioned in the PEP as part of the motivation. ↩︎

11

This would convert things to strings too early and relies on the class having a __dict__, which slotted dataclasses wouldn’t.

When I talk about it being generated I mean it would be generated in the same way as other dataclasses methods and look like this:

def __json__(self):
    return {
        "field_1": self.field_1,
        "field_2": self.field_2,
        ...
    }

The encoder would then handle any nested structures without the __json__ method needing to handle them directly.

12

Yes I get the function I wrote isn’t good enough for the dataclasses module. It’s a 4-liner.

What I think is interesting though (and what I was trying to get at) is the question of whether json.dump(value) is the correct way of dealing with the attribute values.
Is there a better approach?
What happens when an attribute has a __raw_json__ but not a __json__?

13

You don’t need to handle json.dump for nested values manually. The encoder handles them for you.

Example tested on the branch 
import json
from dataclasses import dataclass, fields

def __json__(self):
    return {f.name: getattr(self, f.name) for f in fields(self)}

@dataclass
class Point:
    x: float
    y: float
    __json__ = __json__

@dataclass
class Line:
    points: list[Point]
    __json__ = __json__

l1 = Line([Point(0.0, 0.0), Point(1.0, 2.0)])

print(json.dumps(l1))

'{"points": [{"x": 0.0, "y": 0.0}, {"x": 1.0, "y": 2.0}]}'
2 Likes
14

It’s because it simplifies their implementation. If you had __json__ output a string and this was applied to a container type, then you would have to make sure to also serialize everything you contain. As well, if you return a string you have to handle formatting (e.g. pretty print versus compact). It’s a lot of redundant work that only really needs a single implementation if you say you only expect to get objects out and the caller handles creating the final string representation.

1 Like
15

I agree that new users might assume that __json__() must return real JSON (a string) and i think that further motivates for naming it __serialise__() or similar, where the required output format isn’t implied and hopefully people will look at the docs.

1 Like
16

Looking at the docs would be necessary anyway, but I think it’d work - any types that aren’t directly serialisable by whatever formatter is being used would have to be in the registry for that formatter.

So if you are doing JSON and something tries to serialise an unsupported type, you’ll get a TypeError just like today. And if you’re not in a position to modify that type’s protocol method, then you register a handler for that type and do it yourself that way.

My biggest problem with __serialize__/__serialise__ is the different English spellings of it :smiley:

We’d also have to have some kind of convention for telling the serialise method what the intended format is going to be, in case that affects the amount or kind of data, and that is likely to be the biggest challenge to get agreement on, and I wouldn’t blame anyone for choosing not to try and standardise it.[1]

But the question ought to be raised: when do we get __xml__ and __toml__? Why not now? If the difference is only going to be an extra argument to a similar looking function, why not now?

  1. FTR, I’d use an arbitrary string and claim "json" for ourselves, encouraging 3rd parties to use their own module name as a prefix for whatever they want. ↩︎

2 Likes
17

Meh, tokenize, usercustomize, weakref.finalize(), "string".capitalize(). plenty of .normalize()s… We Brits lost that battle a long time ago.

Although I did only just now realise that toml doesn’t support None so I’m feeling slightly less enthusiastic about the single method for all flavours of dict/list/str/float/int/bool formats.

3 Likes
18

I’m completely excited for the proposal :smiley:

Just my 0.0.1.alpha cents: I prefer copyreg.json. There’s a lot of serializing standard out there, and it seems to me that the current trend is

  1. use a sort of binary JSON
  2. extend JSON
  3. use a completely different binary protocol

IMHO it’s better to not add yet another dunder to classes.

19

The z in __serialize__ looks like a snake. The s is just a worm.

20

Oh, I didn’t mean that the answer wasn’t obvious! Just that I didn’t like it :wink:

There are always these kinds of quirks. If the data being serialised [sic] allows it, and you aren’t able/willing to register something to override it, then that format is simply unavailable to you and a TypeError is exactly the right result. (Though probably it requires registered handlers to have a way to say “just exclude this entire value”, probably through an exception type or returning a singleton.)