Do you think that sentence is sufficient to explain the need to describe the idea of keeping only the newline without content, while already describing the idea of allowing content after the opening quote?
Allowing content after opening quote is more consistent.
And the reason that idea is rejected would just repeat most of the reasons for rejecting the idea of allowing content immediately after the opening quote.
After reading the open PR, I think it is the same topic. I’d like it to be clearer that this is a question of being consistent with other multiline strings or not.
I’m not totally satisfied with the emphasis in the new text, but the topic is covered, which is the main thing I care about.
I do see use cases for writing on the first line of the d-string. It’s more compact, more similar to the way we currently write docstrings, easier to switch…
But I’m convinced linters will fix those problems before they ever get to me. “Add a new line to the start of a d-string if it doesn’t start with a new line” doesn’t seem like a hard rule to add.
OTOH, I think keeping the door open for annotated d-strings, something in the direction of
d"""#SQL
select statement from table;
"""
is valuable.
(Maybe there’s other better things to use that first line for. I think we’ll have a better view on that in the future regardless.)
I’ve been trying to exhaustively envision the use cases of d-strings… and at the end I think there are two ways they can be used cleanly : a ‘vertical flavor’ and a ‘horizontal flavor’…
The vertical flavor wants to copy paste snippets of text into indented code and add the quotes over and under, at the end it wants to concatenate vertically (thus the last newline is actually convenient)
... :
snippet = d"""
- Lorem ipsum
Bla bla bla
"""
text += snippet
The horizontal flavor wants to define blocks and incorporate them afterwards in a template, it probably wants to keep the number of lines minimal, and does not require the indentation level to be defined by the closing quotes (because it wants to reindent in the template) :
... :
statement = d"""#some comment
do_something()"""
... :
instructions = df"""
for in in range(n):
{statement}
""""
(Note ; Here the remaining problem is the multiline reindentation of {statement} in the template.)
I don’t think there are really transverse ways (besides the vertical and horizontal flavors) for using d-strings properly.
→ If I’m right, keeping the last newline should actually be more convenient… and ‘transverse flavors’ that would require to remove it would be anecdotical and clumsy, thus discouraged intrinsically by the syntax, and the python neatness will be preserved.
I feel torn on this PEP for a few reasons:
\n removal, but I’m unsure of the leading-but-not-trailing \n removalAll that said, I really like the idea of improving the dedenting mechanisms in Python, whether through a string method, a new syntax, or just an enhancement to textwrap.dedent
I find myself copy-pasting these two functions around between various projects:
This dedent version that removes the leading newline only:
def undent(text):
return dedent(text).removeprefix("\n")
And this version which strips a trailing newline as well (like inspect.cleandoc):
def undent(text):
return dedent(text).removeprefix("\n").removesuffix("\n")
It seems that I use the first version most of the time but I use the second version (to remove both prefix and suffix \n) about one-third of the time.
It would be nice to avoid using .rstrip() with d-strings, but that would make my primary use case more awkward.
Given this string:
code = r"""
def strip_each(lines):
new_lines = []
for line in lines:
new_lines.append(line.rstrip("\n"))
return new_lines
""".strip("\n")
I would love it if this:
example = d"""
Example function:
{code}
That function was indented properly!
"""
Resulted in this:
>>> print(example)
Example function:
def strip_each(lines):
new_lines = []
for line in lines:
new_lines.append(line.rstrip("\n"))
return new_lines
That function was indented properly!
But I understand that this could complicate the mental model even further, especially raising questions of “what about replacement fields that are mid-line”.
The “How to teach this” section doesn’t currently address teaching this to new Python programmers.
I suspect that I would teach this most often to folks who have never heard of textwrap.dedent and who may never hear of it (if d-strings become successful enough to supersede its use).
My main concern with teaching this to beginners is explaining how it works. The prefix/suffix newline removal feels a bit magical and I’m suggesting even more (likely unfeasible) magic above, but dedenting is also a bit magical, especially if a {...} replacement field includes a string that contains a newline.
I hope some of the above concerns may be useful to consider. I know I’ve repeated/overlapped at least a bit with previously expressed concerns.
Thanks for pushing this idea forward in various forms over the years @methane! I’ve found what I’ve read from the previous threads inspiring.
Hello @methane !
The Python Steering Council has reviewed PEP 822. While we’re generally positive about the PEP, we think it would be best to defer this feature until Python 3.16, to give it more time to solidify.
The main feedback we have is that this feature is baking into the language specific formatting style and expectations, which would be very difficult to change if we get some of the corner cases wrong. We note that the discussions about this PEP still express different opinions and we feel like consensus and convergence hasn’t been reached. Rather than prematurely approve this PEP, we want to give it more time and think that a Python 3.16 target release provides that confidence that we’re making the right choices.
Some details:
Can we also find some projects where d-strings would help a lot and get some testimonials from their maintainers? We think that experience would help a lot in establishing confidence that we’re adding a language feature that truly helps, or just another way to do something similar to the alternatives, but a bit different.
The PSC thanks you for your work on this and urges you to continue to let this idea bake. We look forward to its resubmission for Python 3.16.
IMO it’s easy, since, if I have to dedent a multistring, also the first line must be indented.
My 0.0.1 cents.
PS: I’m happy the PEP is deferred and not rejected. 
Thank you for evaluating this PEP. And I’m sorry for the delay. Python 3.15 has been released as beta, and I have been a bit away from Python.
I tried to migrate dedent() in stdlib and find other places d-string can be used.
This repository contains some examples in README, dstringify (migrate dedent() to d-string) script and patch, and many code snippets uses string concat to build multiline strings.
The README explains the effects of removing the leading blank line and the technical debt in the existing code caused by having a blank line at the beginning.
The leading newline has little effect in most cases. However, I think the problem caused by removing it is smaller than the problem caused by keeping it.
It is difficult to quantitatively determine which is more intuitive and easier to remember. However, there is a clear reason for the asymmetry.
dedent("""\) or kept even though it is clearly unnecessary. (e.g. empty first line in Python source code.)\ to remove the final newline. It is enough to write the closing quote at the end of the line. (e.g. ...lastline""")Anyway, I will write a draft of what I would add to the tutorial about d-string.
It is difficult to find projects that use multiline strings more than CPython’s test code.
Also, I checked the source code of psycopg2, and all the t-string tests were written as single line strings.
The great benefit of d-string is that it can be used together with t-string, but it is difficult to find multi-line t-string queries in famous open source libraries.
I don’t know if a good project can be found, but I will try to find it.
While working on documentation for d-strings, I have been reconsidering how blank lines (lines consisting only of whitespace) are handled.
The current PEP specifies the following behavior:
<TAB><newline>—an IndentationError is raised.Although this specification is theoretically consistent, it makes the conditions for IndentationError hard to explain.
If a TAB character is unintentionally mixed into a text block indented with spaces, and the line is not a blank line, no error occurs; the common indent to be removed just becomes shorter than intended. This is because it is hard to tell whether the TAB character is intentional or not.
On the other hand, when an unintended TAB is in a blank line, we can detect the mismatch with the common indent and raise an error. However, explaining to beginners why the behavior differs between blank lines and other lines is difficult.
To make this easier to explain, I am considering a spec change that adds normalization of blank lines to empty lines to the d-string behavior, just like dedent() does.
With this change, a d-string never raises IndentationError, so there is no need to teach the conditions for it, and since the difference from dedent() becomes smaller, teaching that difference also becomes easier.
For example, in the following code, the blank line contains only a TAB character (---> represents a TAB). Under the current spec this raises IndentationError, but under the proposed spec the blank line is normalized to an empty line, so no error occurs.
s = d"""
foo
--->
bar
"""
# Current spec: IndentationError
# Proposed spec: "foo\n\nbar\n"
Meanwhile, I would like to keep the ability to preserve some indentation using the closing quote, because I found several places where it is useful, both in CPython’s own code and elsewhere. That is, the line containing the closing quote is treated as a non-blank line when computing the longest common indent.
Accordingly, I would like to exclude the line containing the closing quote from the blank-line normalization as well.
This is a difference from dedent(), and it may look strange at first glance that a line consisting only of spaces remains in the result.
However, if we define the line containing the closing quote as a non-blank line, the rules stay simple.
s1 = d"""
foo
bar"""
assert s1 == "foo\nbar"
s2 = d"""
foo
bar
"""
assert s2 == "foo\nbar\n "
The rules for indent removal and blank-line normalization are as follows.
These steps are applied to the physical lines of the source code, before escape sequences are processed. Therefore, escape sequences such as \x20 and line-continuation backslashes are not treated as whitespace.
Do you have any feedback on this spec change?
P.S. I used Opus 4.8 and Fable 5 on Cursor to develop this spec change and write it up in English.
+1. Just a minor observation: isn’t it easier to say that no blank line is considered while calculating the prefix, except for the one with the closing quote? This way furthermore the last line is stripped.
In most case, if there are only whitespace characters before the closing quote of a multiline string literal on the last line, I think it is okay to make it empty.
However, there are cases where str + str is used instead of f-string to insert another value in the middle of a multiline string. (e.g. d"""...""" + expr + d"""..."""). In this case, which result would be preferable in the following sample?
baz = "baz"
s1 = d"""
foo
bar """ + baz
assert s1 == "foo\nbar baz"
s2 = d"""
foo
bar
""" + baz
print(repr(s2))
# "foo\nbar\n baz" ?
# "foo\nbar\nbaz" ?
I think the best way is the one that does not concede on practicality while keeping the mental model the most concisely explainable.
I think for now, two rules are enough to grasp the most of it :
For the remaining white space handling, maybe I don’t get the whole picture right, but I think you would like to have whether no special handling at all, whether that third -simple- rule :
This is what Java Text Blocks does.
Sometime, text block needs trailing spaces. For example, Markdown uses two trailing spaces for <br>.
Java introduced the new escape sequence \s to support trailing spaces.
// copied from the JEP 378.
String colors = """
red \s
green\s
blue \s
""";
So “no special handling at all” is the most simple approach.
It’s an interesting aspect I didn’t considered. Anyway, to be honest, I will be really confused to such a concatenation. Probably I will end up doing:
s2 = fd"""
foo
bar
{baz}"""
Much more readable IMHO.
I just updated PEP 822.
To make it easier for users of the tokenize module to calculate the dedent amount, I exclude lines inside replacement fields from dedent amount calculation too.
This ensures that replacing all replacement fields with {""} does not affect the dedent amount.
s1 = df"""
Hello, {
1 + 2
} World
"""
s2 = df"""
Hello, {"3"} World
"""
assert s1 == s2
Currently, psycopg is the most widely used library that supports t-string.
Psycopg supports writing queries with t-string, but it does not dedent the received query. Since PostgreSQL supports multiline strings, dedenting Python strings could unintentionally affect strings within the query, making it difficult for psycopg to handle dedent.
Therefore, users of psycopg should dedent the query on their side. At that time, Template.dedent() or d-string will be needed. Without these, users are forced to use multiline string literals without indentation or to use string concatenation.
@dvarrazzo If you don’t mind, could you share your opinion as a psycopg developer?
Hello @methane
There are a few things I don’t understand:
De-denting is a Python lexical feature I understand: the execute() function will receive a dedented normal string. I don’t think Psycopg has anything to handle, unlike t-strings which require a Template Processor.
I don’t get the “should”, the “needed”, the “forced” here. Psycopg users can dedent or not a string, manually or with a d-string if introduced. There’s nothing that they “should” do. They “can” use a feature. They are definitely not “forced to use string concatenation”, in fact they are thoroughly discouraged to do so.
Psycopg doesn’t manipulate whitespaces because Postgres is capable to handle them itself and we tend to do only the least amount of operations on a query: pretty much only parameters replacing (either with inline escape values or with Postgres-style $1, $2, … placeholders, according to whether the query is merged to the arguments server- or client-side).
So the users has two choices at the moment: the query indented with the code:
def f():
with psycopg.connect() as conn:
conn.execute("""
SELECT ...
""")
or the query on newline, which they do either to avoid pushing too much on the right or because they don’t want the whitespace in the logs. In the latter case they might write:
def f():
with psycopg.connect() as conn:
conn.execute("""\
SELECT ...
""")
I think that d-strings would only be useful for the case of people not wanting excess of whitespace in the logs but wanting to continue the indent flow of the code in the query. This is a purely stylistic source formatting choice which stops at the AST, AFAICS: psycopg will never receive a “dedented string” object, just a string.
Even a td-string would be received as a t-string by psycopg. If dedenting breaks the query template, that’s a bug in Python or a bug introduced by the user (with the d-string culpable to introduce a new footgun): Psycopg would be just the victim.
I apologise if I missed something in the previous conversation: this is my level of understanding after having skimmed the PEP 822. Because Postgres manages extra whitespace itself, using d-strings would be either unneeded or potentially cause bugs because Psycopg would not receive the query that the user passed. I don’t have an immediate example in mind but I think one can be crafted using Postgres multi-line strings inside a Python multi-line string.
I was using dedent to refer to all methods of removing common indentation from multiline strings. I wanted to confirm that it is unrealistic that psycopg to use textwrap.dedent() after processing a t-string received from the user. So dedent should be done before passing query to psycopg.
The reason I wanted to confirm this point is that if dedent is easy to do as post-processing, str.dedent() and Template.dedent() are likely to be more convenient than d-string. On the other hand, if post-processing is difficult, dedent needs to be done for each individual query, so d-string is more likely to be useful.
Sorry, my story lacked some context. The premise is that user need to remove Python-level indent from their query. For example, to increase readability of the query log or query string in tracing.
Is that case common enough to justify adding syntax to Python, or is it a rare case?
Of course, I am not proposing to add d-string just for the users of Psycopg 3. However, since there are not many well-known libraries that support t-strings yet, I want to check whether it would be useful for Psycopg 3 users first.
Another idea, let’s take
...
lines = d"""
line 1
line 2
"""
snippet = df"""
header
{lines}
"""
.
Can we consensually assume that the desired snippet is
header
line 1
line 2
and not
header
line 1
line 2
?
Thus it might be a good idea to make lines in df strings only containing a pair of curly braces inserting multiline strings automatically reindented. I think there is no “counter use case” for this.
Opinions ?