Help: json.loads() cannot parse valid json

python code:

import json
x = json.loads('{"message":"variable `z` is assigned to, but never used","code":{"code":"unused_variables","explanation":null},"level":"warning","spans":[{"file_name":"main.rs","byte_start":191,"byte_end":192,"line_start":8,"line_end":8,"column_start":9,"column_end":10,"is_primary":true,"text":[{"text":"    let z = \"this is a relatively long string, to see the diff between strings and code.\";","highlight_start":9,"highlight_end":10}],"label":null,"suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[{"message":"`#[warn(unused_variables)]` on by default","code":null,"level":"note","spans":[],"children":[],"rendered":null},{"message":"consider using `_z` instead","code":null,"level":"note","spans":[],"children":[],"rendered":null}],"rendered":"warning: variable `z` is assigned to, but never used\n --> main.rs:8:9\n  |\n8 |     let z = \"this is a relatively long string, to see the diff between strings and code.\";\n  |         ^\n  |\n  = note: `#[warn(unused_variables)]` on by default\n  = note: consider using `_z` instead\n\n"}')
print(x)

gives the error message:

  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 303 (char 302)

however, the json that i used is valid


so i have no idea what could have happened. thanks!

It looks like the problem is with this particular part of the JSON string:

{"text":"    let z = \"this is a relatively long string, to see the diff between strings and code.\";"

and isolating it does indeed produce the error you’re getting:

import json
x = json.loads('{"text":"    let z = \"this is a relatively long string, to see the diff between strings and code.\";"}')
print(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jmorris/.pyenv/versions/3.10.0/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Users/jmorris/.pyenv/versions/3.10.0/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/jmorris/.pyenv/versions/3.10.0/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 23 (char 22)

My guess is that the problem lies with the two escaped quotes \" in the string as they do not stay escaped when interpreted by Python. Using a raw string (r'') mitigates this:

import json
x = json.loads(r'{"text":"    let z = \"this is a relatively long string, to see the diff between strings and code.\";"}')
print(x)

The string is not valid JSON because of the escape sequences.

You have at least four cases of the escape sequence \" (backslash
quote) in your string, but in the Python interpreter, they are escape
sequences and are interpreted as a plain old quote:

>>> print("abcd\"efgh")
abcd"efgh

You need to either escape the backslash so that they remain in the
string and are visible to the JSON encoder, or use a raw string.

Tested and works:

You can fix the escape sequence problem by using a raw string. At the
beginning of your JSON loads command, change this:

x = json.loads('{"message": ... # blah blah blah blah

to this:

x = json.loads(r'{"message": ... # blah blah blah blah

The r prefix tells the interpreter to treat the backslashes as ordinary
characters, not escape sequences. So they are inserted into the string
and passed to the JSON encoder.

A second possible solution, which I have not tested, is to escape the
backslashes by using two backslashes in a row: \\, but if you do that
you might find that the bare quote now matches another quote, so you
need three(!) backslashes: \\\" instead. So try either doubled-
backslashes, or tripled-backslashes, and see which works.

Or just use the raw string r'...' syntax instead.

ooh, thanks, this worked!