About Dictionary Based Syntax Errors~Python 3.10

sandraC · May 14, 2022, 8:16pm

I was working on a new project which was based on dictionaries and recognized some weird(?) things about syntax errors which pop out when you do not separate the key:value pairs with commas.

I have created an example to ask my question better:

# Dummy Keys
key = "Jules"
key2 = "Potato"
key3 = "Doctor"
key4 = "Computer"
key5 = "Chair"
key6 = "Answers"

no_comma_dict1 = {
    key: {
        key2: "Please put a comma here->" # syntax err: perhaps you forgot a comma
        key3: "Ok." 
        },
    key4: {
        key5: "Is this what you were expecting ->",
        key6: "Yes, exactly."
        }
    }

In the code block above, I have a dict of two sub-dicts. I haven’t put a comma between the key-value pairs in the first sub-dict and got a syntax error about commas as expected.

no_comma_dict2 = {
    key: {
        key2: "Here is a comma ->",
        key3: "That's right." 
        },
    key4: { #syntax err: expression expected after dictionary key and ':' --- why not the same ^ error?
        key5: "Huh"
        key6: "No commas!"
        }
    }

Here, I have done the same thing inside the second sub-dict and the error I got wasn’t about commas. Can someone please help me with understanding this - am I missing something? Or, is it a “bug”? If yes, is it possible to make an improvement?
Thanks in advance.

rob42 · May 14, 2022, 8:28pm

What coding IDE are you using?

With my IDE both blocks raise SyntaxError

dict1

key3: "Ok."

dict2

key6: "No commas!"

sandraC · May 14, 2022, 8:35pm

Thank you for your fast reply.
I also get errors in both situations. My main question was more about what syntax error I get.
In dict1, I get:

SyntaxError: perhaps you forgot a comma?

In dict2:

SyntaxError: expression expected after dictionary key and ':'

So, my question is:
Why do I not get the same error in the second dict ?

BTW, I am using IDLE

rob42 · May 14, 2022, 8:41pm

No worries.

I see what you’re driving at. Sorry I’m no help here as my IDE (Wing) simply complains about ‘bad Syntax’ and then leaves me to sort out where I messed up.

rob42 · May 14, 2022, 8:56pm

I know this won’t help you but as an FYI:

With IDLE3.6.9 it simply flags the offending lines and pops up with “Syntax Error” with no kind of help message at all.

saaketp · May 14, 2022, 8:59pm

These new error messages were introduced in 3.10 I believe.
Older versions had very unhelpful SyntaxErrors like you see.

rob42 · May 14, 2022, 9:06pm

Thank you.

Yes, I did read somewhere about that very feature.

I’m kinda stuck on 3.6 until my OS provider gets to updating things (maybe it has and I need to be pro-active) or I distro-hop.

I did try to up-grade to 3.8, but that simply crashed my Window manager, so I had to ‘Time Shift’ back again.

sandraC · May 14, 2022, 9:09pm

Thank you both @saaketp and @rob42 for heading up. I have tried running it with 3.9 and yes, I didn’t get any specific errors.

rob42 · May 14, 2022, 9:12pm

I kinda think that what you’re pumping into is a case of ‘you can’t please all of the people, all of the time’. The system is trying to be as helpful as it can be, given what it ‘knows’.

sandraC · May 14, 2022, 9:20pm

@rob42,
You are definitely right, but mustn’t you at least try to “please all of the people”?

rob42 · May 14, 2022, 9:23pm

yes, you can try, but you gunna fail! It’s an impossible task!!

saaketp · May 14, 2022, 9:33pm

I’m doing some guess work, but I think this can explain the reason for the different error messages.

These error messages are defined in the python grammar file

github.com

python/cpython/blob/aa0f056a00c4bcaef83d729e042359ddae903382/Grammar/python.gram#L1107

      
        
            invalid_legacy_expression:
                | a=NAME !'(' b=star_expressions {
                    _PyPegen_check_legacy_stmt(p, a) ? RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b,
                        "Missing parentheses in call to '%U'. Did you mean %U(...)?", a->v.Name.id, a->v.Name.id) : NULL}
            
            
invalid_expression:
                # !(NAME STRING) is not matched so we don't show this error with some invalid string prefixes like: kf"dsfsdf"
                # Soft keywords need to also be ignored because they can be parsed as NAME NAME
               | !(NAME STRING | SOFT_KEYWORD) a=disjunction b=expression_without_invalid {
                    _PyPegen_check_legacy_stmt(p, a) ? NULL : p->tokens[p->mark-1]->level == 0 ? NULL :
                    RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Perhaps you forgot a comma?") }
               | a=disjunction 'if' b=disjunction !('else'|':') { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "expected 'else' after 'if' expression") }
            
            
invalid_named_expression(memo):
                | a=expression ':=' expression {
                    RAISE_SYNTAX_ERROR_KNOWN_LOCATION(
                        a, "cannot use assignment expressions with %s", _PyPegen_get_expr_name(a)) }
                | a=NAME '=' b=bitwise_or !('='|':=') {
                    RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '==' or ':=' instead of '='?") }
                | !(list|tuple|genexp|'True'|'None'|'False') a=bitwise_or b='=' bitwise_or !('='|':=') {
                    RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot assign to %s here. Maybe you meant '==' instead of '='?",

github.com

python/cpython/blob/aa0f056a00c4bcaef83d729e042359ddae903382/Grammar/python.gram#L1315

      
        
                    RAISE_INDENTATION_ERROR("expected an indented block after class definition on line %d", a->lineno) }
            
            
invalid_double_starred_kvpairs:
                | ','.double_starred_kvpair+ ',' invalid_kvpair
                | expression ':' a='*' bitwise_or { RAISE_SYNTAX_ERROR_STARTING_FROM(a, "cannot use a starred expression in a dictionary value") }
                | expression a=':' &('}'|',') { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "expression expected after dictionary key and ':'") }
            invalid_kvpair:
                | a=expression !(':') {
                    RAISE_ERROR_KNOWN_LOCATION(p, PyExc_SyntaxError, a->lineno, a->end_col_offset - 1, a->end_lineno, -1, "':' expected after dictionary key") }
                | expression ':' a='*' bitwise_or { RAISE_SYNTAX_ERROR_STARTING_FROM(a, "cannot use a starred expression in a dictionary value") }
                | expression a=':' {RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "expression expected after dictionary key and ':'") }

The one about comma is actually the less general one, that just happened to be more helpful for in this case (I guess). It is raised for any invalid expression, whether it makes sense to have comma there or not.

The other one about dict key and value is raised in the second case, because it knows that you’re defining a dict there, maybe because it parsed the other key: value already. In the first case, the error happens while it was parsing the first value, so it doesn’t know that you are defining a key: value pair for a dict.

aroberge · May 14, 2022, 9:33pm

If you want improved error messages with Python 3.6+, you might want to use friendly/friendly-traceback. In both cases shown by OP, friendly highlights correctly the location of the error and makes the correct suggestion of adding a comma where it is missing.

rob42 · May 14, 2022, 9:42pm

Thank you; I’ll have a look at it.

To be fair, it’s seldom that I get into a situation where I can’t figure out what said syntax error I’ve created. If I do, then I think to myself “Rob, you can make that code way simpler!”

sandraC · May 14, 2022, 10:16pm

I didn’t mean such a “deep” reason. Just a reason about my example code(to check if i have missed some points). But, really thank you for sharing your thoughts and the related piece of the src code with us.
( also thank you, @rob42; for wasting your time with this :).
It seems, there is no more need to discuss on this topic. It is just a field which needs improvement. I am not sure if it is important enough to create an (“enchancement”) issue on GH, but I decided not to.

ferdnyc · May 17, 2022, 12:54pm

Sandra C.:

Thank you for your fast reply.
I also get errors in both situations. My main question was more about what syntax error I get.
In dict1, I get:
SyntaxError: perhaps you forgot a comma?
In dict2:
SyntaxError: expression expected after dictionary key and ':'
So, my question is:
Why do I not get the same error in the second dict ?

The interesting thing here, actually, isn’t so much the form of the error, but where it occurs. If you enter the second example interactively in the Python 3.10 REPL, here’s the error message that’s spit out:

>>> key = "Jules"
>>> key2 = "Potato"
>>> key3 = "Doctor"
>>> key4 = "Computer"
>>> key5 = "Chair"
>>> key6 = "Answers"
>>> no_comma_dict2 = {
...     key: {
...         key2: "Here is a comma ->",
...         key3: "That's right." 
...         },
...     key4: {
...         key5: "Huh"
...         key6: "No commas"
  File "<stdin>", line 6
    key4: {
        ^
SyntaxError: expression expected after dictionary key and ':'

It’s decided the syntax error is with the key4 expression, in this case; nothing to do with key5 or key6 at all, it’s backtracked even farther than before, in flagging the error location.

Same thing if you enter it all on one line (can’t decide whether that makes it better or worse):

>>> no_comma_dict2 = { key: { key2: "Here is a comma ->", key3: "That's right." }, key4: { key5: "Huh" key6: "No commas" } }
  File "<stdin>", line 1
    no_comma_dict2 = { key: { key2: "Here is a comma ->", key3: "That's right." }, key4: { key5: "Huh" key6: "No commas" } }
                                                                                       ^
SyntaxError: expression expected after dictionary key and ':'

Ultimately I think it just comes down to “the parser is complex and inscrutable”. Pretty much all error messages are at least improved over earlier Python releases… even if different positions on the parse tree are still treated differently at times.

pablogsal · November 17, 2022, 10:00pm

@sandraC @ferdnyc

Hi, I’m one of the core devs in charge of the Python "complex and inscrutable” parser . This problem has been already fixed in Python 3.10.6 in this commit:

This is what you get in 3.10.6+:

  File "/Users/pgalindo3/github/python/main/lel.py", line 1
    x = { key: { key2: "dsfsdf", key3: "dsfsdf" }, key3: { key5: "dsfsdf" key6: "dsfsdf" }}
                                                                 ^^^^^^^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?

this is one of the reasons you should always use the latest bug fix version

Rosuav · November 17, 2022, 10:37pm

Interesting. My guess is that, in that case, the parser isn’t sure whether you’re attempting to write a set or a dict, or possibly a set/dict comprehension, and it’s really hard to provide useful messages when there’s that many possibilities. Once you’ve established that it’s definitely a dict, there are fewer possibilities (you might have {key: value for...} or {key:value, key: value} but that’s all), so it’s easier to try to guess at a probable solution.

ferdnyc · November 22, 2022, 8:41am

Yay! That’s good news, awesome.

In fairness, 3.10.6 didn’t even exist yet, when @sandraC originally posted this question back in May. It was still 3 months from being born.

But now that the fix has been committed, we can enjoy a more scrutable¹ pyparser!

Notes

(I’m not sure that’s really a word, but just roll with it.)

Topic		Replies	Views
Dict() Syntax error Python Help	14	949	September 23, 2023
Issues with the correct syntax for my nested dictionary Python Help	2	801	September 7, 2020
Can't assign to literal error Python Help	9	1245	April 5, 2023
Dictionary in Format String Syntax versus f-string Python Help	3	526	February 18, 2024
Please help: Key Error: ' ' Python Help help	5	5051	June 9, 2021

About Dictionary Based Syntax Errors~Python 3.10

Notes

Related Topics