JSONDecodeError exception's message - verbose

Ideas
1

Issue

One of the porposes of exception’s message is to help the developer to identify the root cause of the bug and fix it.
Currently, JSONDecodeError doesn’t do it well enough.

Consider the following example:

import json
import requests

try:
    server_raw_response = requests.get(url)
    server_json_response = json.loads(server_raw_response)
except JSONDecodeError as error:
   print(error)

# JSONDecodeError: Expecting , delimiter: line 1 column 23 (char 22)

Can you understand what is the root issue?
You can see that there is a problem with loading the data into json that needs ‘,’ in some character location.
But what actually caused the exception? What was the data that did it exactly?

In order to answer this question you would need to see data itself, right?
So you would go and print/log the data itself which caused the bug and then understand what you need to do.

As you can see, the error you’re getting isn’t indecative enough to understand what really the bug is.

Proposition

I would like to propose a simple solution to this:

import json
import requests

try:
    server_raw_response = requests.get(url)
    server_json_response = json.loads(server_raw_response)
except JSONDecodeError as error:
   print(error.verbose())

# JSONDecodeError: Expecting , delimiter: line 1 column 23 (char 22), got: "'status': 200 'data': 3}"

Using the new verbose function from the JSONDecodeError class you get a verbose exeception message.
This message shows the data itself that caused the bug. You can even control the amount of context to get, by specifing the range of characters to show around the error.

The implementations of the solution don’t necessarily need to be as I suggested.
The important part is that you will have an easy and clear error message.

Benefits

• Easy to debug - verbose and clear error message.

• Easy to use - built-in function to get the verbose error message.

• Data efficiency - The amount of context of the error is controlled by the user.

• Backwards compatibility: this change is fully backwards compatible since the proposed feature will only add function to JSONDecodeError (not removing or changing)

1 Like
2

Linking to a github comment of a developer that have implemented this feature on his own:

This shows and support the necessity of the feature.

3

This is not a bug; it is just malformed JSON data.

Expecting ',' delimiter

line 1 column 23 (char 22)

If you want to make it verbose, you should print the actual malformed JSON data.

This is not helpful. We need the JSON data before this chunk.

While I am neutral on the proposition, implementing it to print meaningful information from the malformed JSON data that fails to decode requires considerable effort compared to manually checking the JSON data.

4

The malformed JSON caused the exception which is not the wanted result.
We would like it to sucessfully decode the response from the server into json.

As I have written in the post, you can also get all of the data by passing a parameter to the verbose function which will define the range of context to show.

There are simple ways to implement this by using the properties of the JSONDecodeError class.
Fir example, One possible way to solve it has been suggested here:

5

Then… fix the JSON data. It’s not valid. Again, without seeing the data, we can’t advise further.

6

I would suggest using the well-known JSON Schema to validate JSON data: JSON Schema Validators for Python.

The current exception object provides computable-friendly information, and a simple code to print the character is:

import json

json_string = '{"name": "John Doe", "age": 42, \n"bio"; "Loves programming!"}'

try:
    data = json.loads(json_string)
    print(data)

except json.JSONDecodeError as e:
    print(e)
    char = e.doc.split('\n')[e.lineno - 1][e.colno - 1]
    print(char)

…but it doesn’t add value to print ; or the whole key-value pair "bio"; "Loves programming!". (If it adds value, how?) It might help in manual debugging, but in your use case, it will be a production exception, and you only need to know the position of the offending character.

7

What might help is this:

except json.JSONDecodeError as e:
    print(e)
    print(e.doc.split('\n')[e.lineno - 1])
    print("-" * (e.colno - 1) + "^")

which will put a ruler underneath the errant line and show the point where the error was detected. But you still often need to scan back from that point to find the actual error.

8

I’d be +1 on making the error message say / underline the issue it’s upset about with some amount of context.

Maybe similar logic can also expose bad encode/decode behavior too?

1 Like
9

Maybe I didn’t explain it well enough.
In the example, we get a response from remote server which we are not responsible of.
In that case we can’t fix the data itself, but we should be able to understand what caused it to be malformed and deal with it.
(Report the the server owner/deal with it on another way)

And “seeing the data” is exactly what is missing from this exception’s message! You can’t figure out what is the problem with the current message.

10

Ohh… Are you asking for a way to get the text response from Requests rather than the JSON?

11

They aren’t asking anything, this is not a help thread. They are suggesting adding a way to see the context around the json decode error, without having to manually keep track of anything else.

1 Like
12

You keep repeating this issue, and the message is quite descriptive, for example, Expecting ':' delimiter: line 2 column 6 (char 38). If you open the JSON string in an editor, it would be straightforward to locate the character at position 38 (line number 2, column number 6).

For example, printing "bio"; "Loves programming!" would still not make it clear where this string is in the JSON string without using the lineno and colno attributes.

As I mentioned in my previous posts, it is not easy to print a meaningful debugging message because JSON data is quite unpredictable. The message may not fit in the console, it may get wrapped, and using a caret to point to the offending character can be difficult. Additionally, printing a partial JSON data type is not going to be helpful. You cannot even pretty print it because it is not valid JSON data.

1 Like
13

Thanks, now I think I have understood your problem with the suggested feature.

I agree that it could be problomatic to print all of the data because of the size of the window.

But printing even part of the data should also help a lot.
For example, if you get the following exception message:
<html><head>...

You understand that the data returned by the server (the data you tried to decode) is an html page and not a valid json.
This could indicate multiple things, but for example, it could indicate that the server is unavaliable so the server returned an html page with a message of “Service unavaliable 503”.
Only by understanding that we got a html page we can reduce the possible problems that caused it dramatically.
And by expanding the context range of the message you can get even more input to understand the problem.

There are many cases in which getting partial data is very useful.

Additionally, I think that even printing all of the data without caret to indicate the error is good enough because you know the location of the bad character(from the exception’s message).

14

The focus remains on printing the JSON string, whether partially or fully. What issues arise with simply using print(e.doc)?

15

The size of the doc could be huge so you wouldn’t want to print it all in an exception’s message.

16

You can easily implement this function in one or two lines using e.doc, as you described:

Do you have a sample implementation of the function?

Here is an example:

import json

json_string = '{"name": "John Doe", "age": 42, \n"bio"; "Loves programming!"}'


def verbose(e, ctx=0):
    print(e)
    
    pos = 0
    lineno = e.lineno - 1
    colno = e.colno - 1
    char = b''
    for i, line in enumerate(e.doc.split('\n')):
        if lineno == i:
            pos += colno
            char = line[colno]
            
            break
        
        pos += len(line) + 1
        
    left = e.doc[pos - ctx:pos] if pos - ctx >= 0 else e.doc[:pos]
    right = e.doc[pos + 1:pos + ctx] if pos + ctx <= len(e.doc) else e.doc[pos + 1:]
    
    message = 'Verbose message:\n%s --> %s <-- %s' % (
        left,
        char,
        right
    )
    print(message)

try:
    data = json.loads(json_string)
    print(data)

except json.JSONDecodeError as e:
    verbose(e, 16)

Result:

Expecting ':' delimiter: line 2 column 6 (char 38)
Verbose message:
age": 42, 
"bio" --> ; <--  "Loves program

Would it be necessary to include it in the standard library?

17

This is not so easy as you imagine, here is a link possible implementation:
JSONDecodeError verbose implementation

Yes, I think it would be helpful to include it in the standard library.
First and most important, it will add the missing piece of information to the exception’s message in order to better understand the exception.
In addition, it isn’t easy to implement this function, so built-in function would help to reduce complexity and save time.

A side note:
I think a better implementation would be:

except json.JSONDecodeError as e:
    e.verbose(16)
18

This can be provided by a third party library, then it can be used right now. I’ve implemented this in jsonyx · PyPI, so you can use that. It’s not a drop-in replacement for json, so this might also need to be implemented in simplejson · PyPI.

BUT, currently the traceback doesn’t display all information:

Traceback (most recent call last):
  File "<stdin>", line 1
    ...t": {"GlossEntry": {"ID": "S..., "XML"]}, "GlossSee": "markup"
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jsonyx.JSONSyntaxError: Unterminated object

Because the context is truncated, you can’t determine the start and end column. Am I supposed to monkey patch traceback.format_exception_only() or can this be implemented by Python? Either by truncating the context when displayed or passing the start and end column?

Edit: changed the number of columns to 69 to not wrap on my iMac, I’m starting to wonder if I better posted an image…

1 Like