A serde-style streaming serialization protocol. I’ve spent a couple days on this now because I got motivated to come up with something after seeing the __json__ proposal but it didn’t actually go my way. If you’ve used Rust’s serde you probably already know what I’m talking about. Anyway, here’s the proposal:
The Protocol
class Serializer(Protocol):
def serialize_bool(self, v: bool) -> None: ...
def serialize_int(self, v: int) -> None: ...
def serialize_float(self, v: float) -> None: ...
def serialize_str(self, v: str) -> None: ...
def serialize_bytes(self, v: bytes) -> None: ...
def serialize_none(self) -> None: ...
# Implementors should provide an accurate length when known, but it defaults to None.
# The use is left to the format. A serializer may use this as an optimization
# (maybe pre-alloc) or discard it. Some formats may require a length and error if it's not provided.
def serialize_seq(self, length: int | None = None) -> SerializeSeq: ...
def serialize_map(self, length: int | None = None) -> SerializeMap: ...
# Honestly, thought about calling this serialize_class but
# for now I'm sticking with struct. msgspec gets by with Struct so it's not entirely unfamiliar to Python devs.
# name should typically be type(obj).__name__ but it could be whatever makes sense.
# Again, the use is left to the format. length is the number of fields.
def serialize_struct(self, name: str, length: int) -> SerializeStruct: ...
def is_human_readable(self) -> bool: ...
class SerializeSeq(Protocol):
def serialize_element(self, value: Serialize) -> None: ...
def __enter__(self) -> SerializeSeq: ...
def __exit__(self, exc_type, exc, tb) -> None: ...
class SerializeMap(Protocol):
def serialize_key(self, key: Serialize) -> None: ...
def serialize_value(self, value: Serialize) -> None: ...
def __enter__(self) -> SerializeMap: ...
def __exit__(self, exc_type, exc, tb) -> None: ...
class SerializeStruct(Protocol):
def serialize_field(self, key: str, value: Serialize) -> None: ...
def __enter__(self) -> SerializeStruct: ...
def __exit__(self, exc_type, exc, tb) -> None: ...
# The star of the show
# This is what users would need to implement
class Serialize(Protocol):
def __serialize__(self, serializer: Serializer) -> None: ...
class SerializeError(Exception): ...
Recipes
A few assumptions here:
to_jsonandto_tomlare__serialize__-aware functions for the rest of this.- The built-in primitives (
bool,int,float,str,bytes, etc) each implement__serialize__and forward to the matching primitive method (int=>serializer.serialize_int(self), etc.)
Same type, two formats
class Serializable:
def __serialize__(self, serializer: Serializer) -> None:
fs = dataclasses.fields(self)
with serializer.serialize_struct(type(self).__name__, len(fs)) as s:
for f in fs:
s.serialize_field(f.name, getattr(self, f.name))
@dataclass
class Server(Serializable):
host: str
port: int
@dataclass
class Config(Serializable):
name: str
server: Server
config = Config("prod", Server("0.0.0.0", 443))
to_json(config) # {"name": "prod", "server": {"host": "0.0.0.0", "port": 443}}
to_toml(config)
# name = "prod"
#
# [server]
# host = "0.0.0.0"
# port = 443
Type with unowned field types
from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal
@dataclass
class Transaction:
amount: Decimal
when: datetime
def __serialize__(self, serializer: Serializer) -> None:
with serializer.serialize_struct("Transaction", 2) as s:
s.serialize_field("amount", str(self.amount))
if serializer.is_human_readable():
s.serialize_field("when", self.when.isoformat())
else:
s.serialize_field("when", int(self.when.timestamp()))
>>> tx = Transaction(Decimal("99.95"), datetime(2024, 1, 1, tzinfo=timezone.utc))
>>> to_json(tx)
'{"amount": "99.95", "when": "2024-01-01T00:00:00+00:00"}'
>>> to_toml(tx)
'amount = "99.95"\nwhen = "2024-01-01T00:00:00+00:00"\n'
Subclassing an unowned type
from datetime import datetime, timezone
class UnixTimestamp(datetime):
def __serialize__(self, serializer: Serializer) -> None:
serializer.serialize_int(int(self.timestamp()))
>>> ts = UnixTimestamp(2024, 1, 1, tzinfo=timezone.utc)
>>> to_json(ts)
'1704067200'
>>> to_toml({"t": ts})
't = 1704067200\n'
Enum
from enum import Enum
class Color(Enum):
RED = "red"
GREEN = "green"
BLUE = "blue"
def __serialize__(self, serializer: Serializer) -> None:
serializer.serialize_str(self.value)
>>> to_json(Color.RED)
'"red"'
`None` across formats
from dataclasses import dataclass
@dataclass
class User:
name: str
nickname: str | None
u = User("Ada", None)
>>> to_json(u)
'{"name": "Ada", "nickname": null}'
>>> to_toml(u)
# SerializeError: TOML has no representation for None
rtoml provides a knob to decide what to do with None.
A future __serialize__-aware rtoml can continue doing that by implementing something like:
class RTOMLSerializer(Serializer):
def __init__(self, none_value: Literal["null", "raise", None] = "null"):
self.none_value = none_value
self.buf = ... # internal buffer
def serialize_none(self):
match self.none_value:
case "null":
self.buf.write("null")
case None:
# omit, do nothing
case "raise":
raise SerializeError
# rest of the methods go here
Lazy serialization of a custom collection type
class Range:
def __init__(self, start, stop, step=1):
self.start, self.stop, self.step = start, stop, step
def __serialize__(self, serializer):
with serializer.serialize_seq() as s:
v = self.start
while v < self.stop:
s.serialize_element(v)
v += self.step
>>> to_json(Range(0, 5))
'[0, 1, 2, 3, 4]'
>>> to_toml({"nums": Range(0, 3)})
'nums = [0, 1, 2]\n'
Field rename within a dataclass
@dataclass
class User(Serializable):
user_id: int
display_name: str
def __serialize__(self, serializer: Serializer) -> None:
with serializer.serialize_struct("User", 2) as s:
s.serialize_field("id", self.user_id) # user_id -> id
s.serialize_field("display_name", self.display_name)
>>> to_json(User(42, "ada"))
'{"id": 42, "display_name": "ada"}'
>>> to_toml(User(42, "ada"))
'id = 42\ndisplay_name = "ada"\n'
Transparent wrapper
A newtype-style wrapper that serializes as its inner value with no struct overhead. Transparent(Transparent(42)) should produce 42, not {"inner": 42}:
from dataclasses import dataclass
@dataclass
class Transparent[T: Serialize]:
inner: T
def __serialize__(self, serializer: Serializer) -> None:
self.inner.__serialize__(serializer)
>>> to_json(Transparent(Transparent(42)))
'42'
>>> to_toml({"n": Transparent(42)})
'n = 42\n'
Open issues
-
The Transaction example (“Type with unowned field types”) above shoves
datetimeinto a string even for TOML, which has a real datetime type. Rust’s serde suffers from the same issue. In fact, writing this proposal out, I realizedserdedoesn’t really provide a format agnostic representation. It only really supports JSON types with a couple extra (bytes) thrown in. Maybe that’s good enough to cover most use cases considering how prolific Serde has become in the Rust ecosystem despite the same limitations and most format serializers are built on top of it. -
We cannot extend this in future. The first iteration will be the last because extending this will be a massively breaking change. The data model is fixed at
bool/int/float/str/bytes/none/seq/map/struct. A type outside it (datetime, Decimal, uuid, whatever the stdlib grows next) has to encode itself as one of those primitives and lose whatever the format would’ve done natively. Once again, Rust’sserdesuffers from the same issue.
After all this, __json__ might actually be better
As you can see, if this proposal was truly format agnostic, it wouldn’t suffer from the “datetime turns to string in TOML” issue.
So this might just be a glorified __json__ after all. There’s also the fact that A TOML/YAML/etc serializer can always just call __json__ and then serialize that. It also gives us an actual spec for what primitives are supported instead of coming up with our own set.
I may have talked myself out of __serialize__ while writing this, but I might as well post this now.