Both of these are valid with our current implementation:
>>> x = 1
>>> f"___{
... x
... }___"
'___1___'
>>> f"___{(
... x
... )}___"
'___1___'
It also handles debug expressions:
>>> f"___{
... 1
... +
... 1
... =}___"
'___\n1\n+\n1\n=2___'
>>> print(_)
___
1
+
1
=2___
That could be driven by the parser though, right? The parser can have lookaheads for }, : and :=, and then signal to the lexer to switch back.
Humm seems that I didn’t clarify this correctly and there is some confusion. […]
Got it. I glanced over that (I seem to be missing a lot of detail today – must be the weather :-).
I checked and we handle that correctly at the time:
Ah, but who says I don’t want to allow }} in a format specifier? Previously maybe we couldn’t but now we could, right?
(Also, could we allow \{ and \} to escape curlies? If we could, would we want to? The doubling feels jarring because that’s not how Python escapes anything else.)
Both of these are valid with our current implementation:
That’s a relief. I guess it’s another part I skimmed incorrectly. The PEP has “Comments, using the # character, are possible only in multi-line f-string literals” which made me think that these also weren’t possible. But if I added a comment to my examples those should work, right?
Definite +1 on the PEP as a whole.
Like others in the thread, I voted in favour of grammar consistency in the poll (allowing quote reuse), but I would also like to see the PEP more strongly discourage actually relying on that in manually written code, in the form of a proposed addition to PEP 8.
The section on string quotes would be the appropriate place for an addition like:
When an f-string contains nested f-strings in subexpressions, use different quote styles at each level of nesting to make it easier to distinguish nested strings from the end of the containing f-string. (By implication, f-string nesting should never exceed 4 levels outside auto-generated code intended solely for machine consumption)
(The part in parentheses probably shouldn’t be added to PEP 8, but could be included in the PEP 701 section on adding the new PEP 8 text)
Maybe tweak the phrasing to be “f-strings that span multiple lines”? (As you say, it’s possible to span multiple lines without using triple quotes to make an explicitly multi-line string)
In my view, the various mentions of nested f-strings and unreadable code as a possibility are not particularly compelling arguments. It is always possible to write unreadable code.
O0OO0 = 1
OO0O0 = 0
OOO00 = O0OO0 + OO0O0 - ...
And I haven’t even started defining functions yet!
The backslash restriction shows up pretty naturally when working with \n. The other day, I ran into the following real-world example:
f"{'\n'.join(mylist)}\n"
It is not intuitive that this raises an error at runtime, which makes it a (possibly unnecessary) point of friction for developers.
“It is already possible to write unreadable code so we should not discourage more of it” is definitely not a compelling argument. You might as well jump straight to “it’s possible to write unsafe Python code so there’s no compelling argument against C++”.
I take your meaning, but that’s not what I’m trying to say.
There are many examples in this thread which are very confusing but also intentionally constructed, like
f"{some + f"{nested + f"string {concat}"}"} example"
This doesn’t seem to me to be an attempt to show how the feature, being used merely “cleverly” or by a beginner, makes code difficult to understand. Rather, it intends to show that it is possible to construct something unreadable.
There is a good amount of discussion about nested f-strings here, but I don’t know that anyone in favor of the PEP cares very much about nested f-strings? More than one comment seems to suggest a view that no one yet knows of a legitimate and useful case for nested f-strings, so we’d probably keep them out during code reviews.
That line of argument might not be convincing to those against. But nobody seems to be strongly in favor of nested f-strings as a particular consequence of this proposal.
This will depend on what one considers legitimate, but I found myself writing
print(f"{some_dict["foo"]}")
not once, but twice in the same day when experimenting in the REPL recently. Even though I hade recently read this thread I was still confuced for a while before I realized what the error was. So quick experiments in the repl is one use-case that I would like this feature for.
That usage features use of the outer quote character inside of the f-string.
By nested f-strings, I mean any use of an f-string inside of the {} format specifiers:
f"f-string {'containing ' + f'another {"f-string"}'}"
We actually started with that but it turned out to be a bad idea for the following:
tokenize.py and others) the parser is actually not available.In general, we found that just modifying the tokenizer to know when to switch is not only much easier, but also much cleaner and debuggable, and requires many many fewer changes. At the end of the day, one of our cornerstones is making this part of CPython more maintainable so our implementation went this route.
Thanks, this actually makes sense.
Trying to get the discussion back on track (away from .join()) a bit, have you thought about my parenthetical remark about using \{ and \} to escape curlies instead of, or in addition to, {{ and }}? I believe that when PEP 498 was discussed, there was a good reason to use the latter, but somehow I can’t find that reason in the PEP, and I can’t recall what it was. Maybe it was for compatibility with str.format()? The doubling has always irked me, and I think I’ve forgotten that it existed several times over the years (because I so rarely have a real need to put a curly brace in an f-string).
Not yet, I want to discuss this in detail with @lys.nikolaou and @isidentical and currently we are focusing on making the tokenize.py changes and acting on the feedback so far. We will discuss it and check if is too hard to support. If it proves to be trivial and we all agree we can add it to the proposal. If it proves to be tricky we can do it as a follow-up (maybe in an issue and not in a PEP).
In any case, I think the question is if we can support both, right (for backwards compat). The double { makes it somehow harder to implement but I don’t think there is a way around supporting it
I believe that when PEP 498 was discussed, there was a good reason to use the latter, but somehow I can’t find that reason in the PEP,
Meanwhile, maybe @ericvsmith knows the answer to this?
Yes, that was it.
It’s unclear from the PEP whether following (new line in f-expression) are valid or not:
f'{1
+
1}'
f'{1\
+\
1}'
# triple-quoted f-string
f'''{1
+
1}'''
f'''{1\
+\
1}'''
similar code is valid in string interpolation, e.g. C# (see [Proposal]: Remove restriction that interpolations within a non-verbatim interpolated string cannot contain new-lines. · Issue #4935 · dotnet/csharplang · GitHub), Kotlin (see Kotlin language specification).
Thanks for your comment! I answered this partially here:
But I can confirm that this will be incorporated soon into the PEP document 
The SC (sans Pablo, who didn’t get a vote since he’s one of the PEP authors) is happy to accept PEP 701. Thanks for the hard work in getting f-strings in an even better place 
A big thanks to the authors of this PEP and its implementation. This is a nice improvement!
I’m implementing support for PEP 701 in pyright, which has its own tokenizer and parser. I tried to implement it based on the documentation in the PEP, but I found that there’s an important detail missing in the “How to produce these tokens” section. Step 3 doesn’t say anything special about what happens after a : character is encountered. It says that the tokenizer should go back to step 2 after emitting a : token and popping the current tokenizer mode from the stack. However, when in “format specifier” mode (i.e. after the : is encountered), I think the rules for step 2 need to change so double braces ({{ or }}) are no longer assumed to be part of an FSTRING_MIDDLE token.
To test this, I tried f"{'':{{1}.pop()}}" with the latest CPython implementation. This f-string parses without error. The {{ after the : is clearly being treated as two separate { tokens rather than part of an FSTRING_MIDDLE token.
If I use the same subexpression prior to the :, it works consistently with the PEP’s “step 2” description, and the {{ is treated as part of an FSTRING_MIDDLE token. As proof, the f-string f"{{1}.pop()}" results in a syntax error unless a space is added between the two opening braces.
Could the authors of the PEP confirm that my assumption is correct? If so, would it make sense to update the PEP so it accurately reflects the intended behavior?
Edit: Another related question… in “step 3”, the PEP implies that when a = or ! token is seen, the tokenizer should switch back to step 2. I presume that’s incorrect and it should stay in step 3 and continue to produce normal tokens (as opposed to FSTRING_MIDDLE tokens) until it sees a : or '}`.
Another small topic that isn’t mentioned in the PEP has to do with nested format specifiers. It appears that the implementation supports two levels of nesting but not three.
f"{'':*^{1:{1}}}" works
f"{'':*^{1:{1:{1}}}}" produces a syntax error (f-string: expressions nested too deeply)
The PEP indicates that there is a “lower bound of 5 levels of nesting”. I presume this statement applies to nesting of f-strings, not of format specifiers within an f-string. Would it make sense for the PEP to mention a separate lower bound for format specifier nesting?
Thanks @erictraut for the message! I am very excited to know that you are adding PEP 701 support in pyright! Thanks a lot for the great work
Let me answer the points that you raise.
“How to produce these tokens” section. Step 3 doesn’t say anything special about what happens after a
:character is encountered. It says that the tokenizer should go back to step 2 after emitting a:token and popping the current tokenizer mode from the stack. However, when in “format specifier” mode (i.e. after the:is encountered), I think the rules for step 2 need to change so double braces ({{or}}) are no longer assumed to be part of anFSTRING_MIDDLEtoken.
I think we may want to be a bit more explicit here, but the section was intended to be indicative and not prescriptive as different tokenizers would require different ways to approach the problem. You have a good point though because there is ambiguity over how the format specifier should be tokenized in general so we should describe that better. I will try to make a PR after beta freeze as we are currently fully immersed in the Python tokenizer changes.
You are right here but note this is not new as this is something we just preserved from the previous implementation:
ython 3.11.1 (main, Dec 15 2022, 18:32:52) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f"{'':{{1}.pop()}}"
' '
[quote="Eric Traut, post:120, topic:22046, username:erictraut"]
Another small topic that isn’t mentioned in the PEP has to do with nested format specifiers. It appears that the implementation supports two levels of nesting but not three.
`f"{'':*^{1:{1}}}"` works
`f"{'':*^{1:{1:{1}}}}"` produces a syntax error (f-string: expressions nested too deeply)
[/quote]
This is also something we preserved from the previous implementation:
Python 3.11.1 (main, Dec 15 2022, 18:32:52) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f"{'':*^{1:{1:{1}}}}"
File "<stdin>", line 1
f"{'':*^{1:{1:{1}}}}"
^
SyntaxError: f-string: expressions nested too deeply
For both of these things we just preserved existing behavior, while for the nesting of f-strings we had to dictate it because now arbitrary nesting is possible while before it was not.
Would it make sense for the PEP to mention a separate lower bound for format specifier nesting?
I think this is something that makes sense, just so is more clear what the default prescription/recommendation is.