Looking for a pointer/overview of how cPython handles fStrings

I asked this in help and apologize if this is not the correct place to ask this.

I have a semi-working tokenizer and thanks to LibCST a nearly rock solid parser implemented.

In Python/tokenizer.c, fstrings are wrapped up as dumb ‘f"Hello {place}"’ string that are parsed somewhere else.

Somewhere after that middle man parser, fstrings go through the PEG parser with this rule cpython/python.gram at main · python/cpython · GitHub but that rule isn’t prepared to handle ‘f"Hello {place}"’.

The Python in Rust project steps character by character over a fstring to create a mini-ast which makes me think cPython does the same but I haven’t been able to find where.

For the curious, my code base is at github under user devdave, project name rython4. The antispam filter won’t let me post more than two hrefs to a post so that’s why there isn’t a direct link.

2 Likes

Currently, this happens here:

But this will change soon as we are working to integrate this with the PEG parser

Is the plan to try and parse something like f"Hello {place=}" with the PEG parser or more like the Python in Rust project where I think they break it down to a AST?

The idea is to parse it with the peg parser itself yeah

@pablogsal Any chance there is an experimental fork/branch with this new ruleset for parsing fstrings?

There is:

But don’t draw any conclusions from it yet, we are still heavily working on it.

1 Like

This is what Instagram’s LibCST developers tried for parsing f-strings - LibCST/grammar.rs at main · Instagram/LibCST · GitHub