How are fstrings lexed & parsed in cPython?

Python Help
1

This might be a question for core but figured I would start with help first. I have a semi-working tokenizer and thanks to LibCST a nearly rock solid parser implemented.

In Python/tokenizer.c, fstrings are wrapped up as dumb ‘f"Hello {place}"’ string that are parsed somewhere else.

Somewhere after that middle man parser, fstrings go through the PEG parser with this rule cpython/python.gram at main · python/cpython · GitHub but that rule isn’t prepared to handle ‘f"Hello {place}"’.

Rust in Python steps character by character over a fstring to create a mini-ast which makes me think cPython does the same but I haven’t been able to find where.

For the curious, my code base is at github under user devdave, project name rython4. The antispam filter won’t let me post more than two hrefs to a post so that’s why there isn’t a direct link.

NOTE the Python software foundation license txt file hasn’t been copied into the repository yet. Regardless, because I am almost directly translating from cPython’s code base to rust, it’s a PSF project.

2

https://docs.python.org/3.6/reference/lexical_analysis.html#formatted-string-literals
maybe this is what you are looking for

3

I have the language reference open/pinned on my browser and unfortunately the lexical rules for fstrings is only a little helpful, the valuable discovery would be finding the C code that processes f"string" to something the PEG parser can consume.

4

Possibly in string_parser.c

5

Only partially joking when I wonder if perhaps I will pretend f-strings don’t exist.

6

In May, Pablo Galindo proposed moving f-strings into the grammar, I don’t know what’s the progress on this.