PEP 837: Extensible JSON serialization

[storchaka (Serhiy Storchaka)]

@Monarch

I think dataclasses should also receive a default __json__ implementation that simply calls dataclasses.asdict(dataclass).

If Python was more like JavaScript, it would be obvious idea. We could even simply use vars() to serialize arbitrary objects. But Python draws strict line between object and it’s namespace dict. Decision about serializing particular dataclass as a dict should be made at higher level. For example, if you are planning to restore it as the original object, you might need to add the class name.

@Monarch

I still believe a generic __serialize__(self, serializer: SerializerProtocol) -> None that serializes the object into primitive types is the better approach (a la serde’s Serializer trait).

Do you meant PEP 246? I would be glad to go that way, but it was rejected more than 20 years ago. It is currently only used in the sqlite3 module.

The drawback of such general protocol is that each class implementing it should know about the corresponding concrete protocol object, i.e. import it. PEP 837 allow you to support JSON without any import. Most of custom collection classes (as well as numeric classes) would only need to add

def __json__(self):
    return self

or something equally simple.

PEP 246 would allow to represent __str__, __repr__, __float__, __iter__, etc via single complex dunder. But I think we will spend next 20 years on bikeshedding. It is better to get something more specialized for JSON right now.

[Monarch]

@storchaka

If Python was more like JavaScript, it would be obvious idea. We could even simply use vars() to serialize arbitrary objects. But Python draws strict line between object and it’s namespace dict. Decision about serializing particular dataclass as a dict should be made at higher level. For example, if you are planning to restore it as the original object, you might need to add the class name.

I would assume an overwhelming majority want their dataclasses to be serialized as dict. It’s practically a defacto standard already:

>>> import dataclasses, cattrs, msgspec, orjson
... from cattrs.preconf.json import make_converter as cattrs_converter
... from pydantic import TypeAdapter as pydantic_converter
...
>>> @dataclasses.dataclass
... class Club:
...     id: int
...     name: str
...     members: list[str]
...
...
... obj = Club(
...     id=1,
...     name="example",
...     members=[
...         "Alice",
...         "Sam",
...     ],
... )
...
>>> cattrs_converter().dumps(obj)
'{"id": 1, "name": "example", "members": ["Alice", "Sam"]}'
>>> msgspec.json.encode(obj).decode()
'{"id":1,"name":"example","members":["Alice","Sam"]}'
>>> pydantic_converter(Club).dump_json(obj).decode()
'{"id":1,"name":"example","members":["Alice","Sam"]}'
>>> orjson.dumps(obj).decode()
'{"id":1,"name":"example","members":["Alice","Sam"]}'
>>>

If people want something more exotic, they can always override it but if we don’t provide a default implementation then I expect to see a ton of:

@dataclass
class Foo:
    def __json__(self):
        return dataclasses.asdict(self)

@storchaka

It is better to get something more specialized for JSON right now.

I agree. I’m +1 on __json__.

[Monarch]

I’m wondering whether __to_json__ would be a better name. If we introduce __from_json__ later, having __json__ on the serialization side would make the API feel asymmetric.

[bob (Bob Ippolito)]

I’ve always thought that the way out of the encoding/decoding protocol mess is to use schemas provided while encoding or decoding instead of global registries, but obviously not badly enough to standardize an implementation of it myself further than the options like default and object_hook.

The reason why I never implemented a global registry like copyreg is because there’s always going to be cases where the same application is going to want to encode an object different ways depending on the context.

[peterc (peter)]

I do write classes with a (static) .from_json() method, but I can’t imagine a scenario where it would make sense to systematically use .__from_json__()

[storchaka (Serhiy Storchaka)]

For __from_json__ you need to design protocol on top of JSON to communicate what class’ __from_json__ to be used. This is out of scope of the json module.

[storchaka (Serhiy Storchaka)]

@bob

there’s always going to be cases where the same application is going to want to encode an object different ways depending on the context.

Yes, and the PEP supports customization at different levels: at the library level, when you simply add __json__ to your classes, at application level, when you can use global registry for foreighn types, and you can also specify per encoder dispatch_table.

BTW, I am currently working on supporting pickling weakref types. For proxy types, the global registry is the only way. You cannot add __reduce__ in the proxy class – it will be redirected to the referrent.

[warden]

Why the focus on JSON over other serialization formats?

Several standard library container types whose JSON form is unambiguous [such as] collections.deque […] gain a __json__ method and therefore serialize out of the box.

Well, what is the unambiguous serialization of this deque:

collections.deque([1,2,3], maxlen=5)

Is it [1,2,3] or is it {“data”: [1,2,3], “maxlen”: 5} or is it something else?

The json.JSONEncoder has an unfortunate interaction with tuple sub-classes such as NamedTuple where the value is serialized as array before the default hook is ever reached. iterencode hacks no longer work in recent versions of Python. Given this behavior, what would happen here:

import json, typing

class NT(typing.NamedTuple):
    a: str
    n: int

    def __json__(self):
        return self._asdict()

x = NT("x", 127)
json.dumps(x)  # am I a list/array or am I a dict/object?

I have a few more questions, but don’t want to spam.

[bob (Bob Ippolito)]

I don’t think that a per-encoder dispatch_table is sufficient for a schema based encoding, at each field of an object you should be able to overlay another dispatch_table (or equivalent). Of course anything is possible if you do a full transform of the input to match the schema or return fully encoded RawJSON wherever the dispatch_table needs to differ, but that’s not very ergonomic.

[DavidCEllis (David Ellis)]

@warden

json.dumps(x) # am I a list/array or am I a dict/object?

With the current implementation, this is a dict/object.

>>> import json, typing
... 
... class NT(typing.NamedTuple):
...     a: str
...     n: int
... 
...     def __json__(self):
...         return self._asdict()
... 
... x = NT("x", 127)
... json.dumps(x)  # am I a list/array or am I a dict/object?
... 
'{"a": "x", "n": 127}'

[warden]

That is useful, but it contradicts this sentence in the proposal:

[…] the hooks are only consulted for objects that would otherwise head toward default= or TypeError.

The hooks apparently are consulted, despite NT not being caught by default= in 3.14. See:

import json, typing

class NT(typing.NamedTuple):
    a: str
    n: int

x = NT("x", 127)

def default(o):
    if isinstance(o, NT):
        return o._asdict()
    raise TypeError(f"Object of type {type(o).__name__} is not JSON serializable")

json.dumps(x, default=default)  # ["x", 127] instead of an object, despite default!

Maybe the PEP should clarify the meaning of “exact basic type”?
I would appreciate if there were a way to intercept tuple, set, frozenset etc. serialization anyway, because it could be useful to implement type-tagged JSON with the build-in encoder.

[storchaka (Serhiy Storchaka)]

Schema based encoding is completely different thing from type based encoding. For now, the only solution was to transform input according to scheme in a tree of basic types (tuples/lists, dicts) and pass it to json.dump(). But I think that the PEP allows to support this without materializing all tree in memory. You can wrap input in a proxy with __json__. If you need to serialize a JSON Array, __json__() should return an iterable which emits directly serializable objects or other proxies. The proxy can bear scheme-specific context. You can have several implementations of schema based encodings, this all is controlled outside of the json module. This will allow to create multiple third-party implementations, and one of them could even be included in the stdlib, but this is out of scope of this PEP.

[storchaka (Serhiy Storchaka)]

@warden

Maybe the PEP should clarify the meaning of “exact basic type”?

Basic types are dict, frozendict, list, tuple, str, int, float, bool, None. Exact basic type are exactly these types, not including subclasses.

[TomRitchford]

@storchaka

First, dunders are reserved for Python core and stdlib.

That bird flew a long time ago. numpy has numpy.__array__, torch has several dunderscore methods for derived classes, ujson apparently has one, and other important tools do too.

It’s very unfortunate. With hindsight, perhaps it would have been better to have been stricter about that rule.

[Lucas_Malor]

@storchaka

• Application level — a serialization function for a type you do not control is registered with copyreg.json()

Playing Devil’s advocate, adding copyreg.json will not introduce a global state? I don’t remember exactly where, but I remember that some core devs opposed to such an idea.

What if an extension X will register with copyreg a JSON serializer for a class, and another extension Y will register another serializer for the same class? Is there same bad precedent with copyreg.pickle usage in real world?

[storchaka (Serhiy Storchaka)]

@Lucas_Malor

Playing Devil’s advocate, adding copyreg.json will not introduce a global state?

Yes, of course. Application has a global state. sys.stdout and sys.modules is a global state.

@Lucas_Malor

What if an extension X will register with copyreg a JSON serializer for a class, and another extension Y will register another serializer for the same class?

They both will misuse the API. It is like two extensions will import different modules under same name, write to the same file, register different encodings under the same name, reconfigure sys.stdout with different options.

It is legal to change the global state. This is why we have a lot of API designed to change the global state. But there should be a single source of authority for this to avoid conflicts. Libraries/extensions should not do this unless applicating say them to do so.

[bwoodsend (Brénainn Woodsend)]

@Lucas_Malor

@storchaka

• Application level — a serialization function for a type you do not control is registered with copyreg.json()

Playing Devil’s advocate, adding copyreg.json will not introduce a global state? I don’t remember exactly where, but I remember that some core devs opposed to such an idea.

FWIW, as ujson maintainer, if this PEP does go through then there’s zero chance I’m implementing the copyreg portion of it. It’s global. It’s mutable. And it’s not so unreasonable for said mutation to happen mid-serialization that I can justify not clogging everything up with pycriticalsections.

[Lucas_Malor]

Ok. Probably I’m asking something obvious… I suppose that, if a package implements its own class or I have a work or private project, it’s good to use copyreg.json.

[Monarch]

I do share some reservations about a global mutable state like that. I feel like this can be deferred. Subclassing types you don’t own to implement __json__ will likely take you quite far and will be easier to reason about.

[bwoodsend (Brénainn Woodsend)]

Or just define a single default() function and pass it around your application. Or write a wrapper function around json.dumps() and use that instead. Making it global is such a trivial convenience.