tomllib is not just about TOML. It adapts TOML to Python datatypes. Both of those matter.
For example, pi = 3.141592653589793238 is not a portable TOML document – and indeed most parsers will lose some of the digits. For a more extreme example: you could technically write a spec-compliant TOML reader that rejects n = 123 with “integer too big”.
IOW, we already guarantee things the TOML standard doesn’t. And the details of that are closely tied to Python datatypes.
In Python, of course, dicts are ordered.
It is a concern. If you read a TOML file and write it as JSON – or more importantly, if you do any processing on it and write out the result – you can get a reproducible output.
That’s a very practical property for a parser to have.
IMO, we should be clear that relying on dict order hurts interoperability, but I’d be fine with the Python implementation guaranteeing it.
cc @pradyunsg (as TOML spec maintainer). Do you have thoughts on this?
The TOML spec explicitlystates that no order is guaranteed.
We should follow the spec. We don’t want diverging implementations, which is the risk. Furthermore, if you rely on an order then you should explicitly apply that ordering - especially since you know it is not part of the spec.
Guaranteeing no order doesn’t conflict with an implementation guaranteeing one… The specs say that no order is guaranteed. That means implementations are free to use hash maps that don’t preserve ordering. But they are also free to use some that use hash maps that preserve ordering. As a matter of fact, we use a dict. And since CPython’s dicts maintain insertion order you inherit this guarantee as well. It doesn’t make sense to randomize the dict’s keys before returning it.
Once again, we are not talking about changing the implementation. It’s about exposing the implementation detail as a CPython detail. So if you use tomllib you can rely on that implementation detail. But if you decide to do so, then you likely won’t be able to switch to a parser that implements TOML parsing differently (e.g., with an alternative Python implementation for which dicts don’t preserve insertion order). But that’s not the problem tomllib needs to bother about.
Hi Bénédikt, libraries such as this implement a spec. If all implementations try and follow that spec then it decreases compatibility problems. In this case I would suggest that any mention of an ordering should state up front that:
An ordering is explicitly not guaranteed by the spec. This implementation however … … code that relies on this library specific ordering may need to insert explicit ordering for compatibility with other interpretations of the spec.
And this is exactly what I’ve been saying in my past posts. I’ve already said that we will mention it as an implementation detail and this will be rendered in the docs within a box.
We’re not here already. What I suggested was to document the implementation details, but the question is: do we even want to document it? if not, I also said that we can just restate what the TOML specs say about ordering to make sure that users don’t rely on the dict’s insertion ordering from CPython.
Expecting those digits to be preserved would be incorrect, and while we’re here, the parse_float parameter on the parser being both available and documented with the intent of allowing use of python’s decimal module is a mistake.
Spec says:
Floats should be implemented as IEEE 754 binary64 values.
If you actually need a number stored with arbitrary precision, toml only lets you store it as a string, and then handle it in the application.
Spec says:
Arbitrary 64-bit signed integers (from −2^63 to 2^63−1) should be accepted and handled losslessly. If an integer cannot be represented losslessly, an error must be thrown
TOML’s specification doesn’t define it’s use of “should” and “must” to mean something specific rather than their plain language meanings, so I don’t think that’s true, 123 is in a range the spec states should be accepted.
I did some looking around, and I think right now, msgspec is a great example of handling this in the right location, without extending the file format into some ad-hoc standard in the process.
When writing toml, msgspec allows ordering, but not when reading. This allows for things like better diffs to happen automatically for any mechanical modifications to toml files, without relying on the order for anything where it isn’t guaranteed when actually reading the data.
By default, msgspec doesn’t write using the order provided by python dict iteration either, it uses a “deterministic” ordering that is only guaranteed to be deterministic and based on internal details, along with offering a lexical sort by field name that is slower, but may producer easier for human consumption files.
Nothing msgspec is doing to enable this documents something that creates a problem for using toml for data, the order when writing has no impact on the toml meaning of data, so providing it on write for other benefits comes with no issues. Relying on it having meaning on the data when reading it is where any issues would occur.
Msgspec also naturally encourages parsing directly into its own struct types rather than a dict, where any implicit ordering also vanishes.
I’m personally not against documenting it, so long as it’s documented only as informational and directly states “don’t rely on this when dealing with toml”, but I struggle to see a benefit here to documenting the internal behavior with such a note over just stating that users should not rely on the order without pointing out that tomllib currently does preserve order.
Ah, but you’re quoting an older version of the TOML spec. If talking about the future of Python stdlib, the relevant spec is TOML 1.1.0, as parsed by tomllib 3.15 & 3.16:
Implementations are free to support any integer size. It’s recommended that at least 64-bit signed integers (from −2^63 to 2^63−1) are accepted and handled losslessly. If an integer cannot be represented losslessly, an error must be thrown.
(And similar for floats, but that’s less relevant since CPython-on-common-platforms matches TOML’s recommendation.)
Hopefully my post will make more sense with TOML 1.1.0 :)
