I came across a behavior in orjson that I’m curious about.
>>> n = 2**64 + 1
>>> s = json.dumps(n)
>>> json.loads(s)
18446744073709551617
>>> orjson.loads(s)
1.8446744073709552e+19
So orjson.loads() turns a JSON integer that doesn’t fit in 64 bits into a Python float.
What’s interesting is the asymmetry with encoding:
>>> orjson.dumps(2**64)
TypeError: Integer exceeds 64-bit range
I understand that orjson uses yyjson, which has a 64-bit integer representation and falls back to double on integer overflow.
But should a Python-facing JSON decoder silently change the type from int to float, given that Python int is arbitrary precision? Would rejecting the value or preserving it as an int be more expected?
I’m mainly interested in the general Python/API-design question rather than whether orjson specifically should change.
If it’s possible to provide the precision, it should be done. JSON (AFAIK) still doesn’t define the number type/limits, so as long as it’s not a DoS concern it should be integers if no fractional part is present.
This is the kind of behaviour that would cause bugs for me, because I’d never expect this to be happening. (And I don’t read the docs )
With ‘hindsight’ I can understand where it comes from. A lot of people view Jsons as representing javascript objects, rather than just being strings. So they view the ‘proper’ way to read json to first convert to a javascript-like types, and then to Python.[1]
I view jsons as strings, therefore capable of representing integers of arbitrary size, and natively representing Decimal numbers (and floats only as a by-product). ↩︎
Correct, it’s one of the most remarkably underspecified popular standards there is. Somehow it doesn’t cause massive interoperability problems
Personally I think it’s better to faithfully reflect the data, even if JavaScript wouldn’t have been able to; though with a pragmatic limit (I don’t, for example, think that we should emit decimal.Decimal for floats by default, but I’m glad there’s an easy way to do it if you want it). JSON does not strictly represent JavaScript data, and it should be possible to carry large integers through it. If I encode a large integer in Python, I can decode it in Pike, even though JS wouldn’t be able to handle it.
Whether a third-party orjson package follows this principle is up to its maintainers, though. Maybe its purpose is performance rather than maximum generality. That’s a valid tradeoff and a reason for having different JSON packages. But my default expectation is that JSON can carry large integers without loss of precision.
Conversion to float seems like a bad idea. It makes the encode/decode step lossy and irreversible. It may also be a schema violation if someone is using a typed dict.
6. Numbers
The representation of numbers is similar to that used in most
programming languages. A number is represented in base 10 using
decimal digits. It contains an integer component that may be
prefixed with an optional minus sign, which may be followed by a
fraction part and/or an exponent part. Leading zeros are not
allowed.
A fraction part is a decimal point followed by one or more digits.
An exponent part begins with the letter E in upper or lower case,
which may be followed by a plus or minus sign. The E and optional
sign are followed by one or more digits.
Numeric values that cannot be represented in the grammar below (such
as Infinity and NaN) are not permitted.
number = [ minus ] int [ frac ] [ exp ]
decimal-point = %x2E ; .
digit1-9 = %x31-39 ; 1-9
e = %x65 / %x45 ; e E
exp = e [ minus / plus ] 1*DIGIT
frac = decimal-point 1*DIGIT
int = zero / ( digit1-9 *DIGIT )
minus = %x2D ; -
plus = %x2B ; +
zero = %x30 ; 0
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is
generally available and widely used, good interoperability can be
achieved by implementations that expect no more precision or range
than these provide, in the sense that implementations will
approximate JSON numbers within the expected precision. A JSON
number such as 1E400 or 3.141592653589793238462643383279 may indicate
potential interoperability problems, since it suggests that the
software that created it expects receiving software to have greater
capabilities for numeric magnitude and precision than is widely
available.
Note that when such software is used, numbers that are integers and
are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
Note that it allows implementations to set limits, and that for interoperability, you should not expect more than double precision. But the specification does not mandate anything beyond that.
(You may also notice that Infinity and NaN are explicitly prohibited. Quite a few implementations ignore this, including by default Python’s. The interoperability argument goes both ways.)
)