PEP 822: Dedented Multiline String (d-string)

PEPs
145

FWIW, you have written similar idea, but you did not say let’s include it in this PEP.

Comparing with your previous idea, there are several drawbacks.

If the embedded multiline string is already indented, it will end up being indented twice. The behavior of f-strings and df-strings would also differ much more than under the current PEP, making mechanical rewriting more difficult.

If we change the behavior of df-strings, dt-strings should behave the same way. However, that would require changing the dynamic behavior of Template, which could affect libraries that already support Template.

How much indentation should the embedded string have in a situation like the following? (For now, please set aside the issue of SQL injection.)

snippet = df"""
    header
        first {lines} second
    """

query = df"""
    UPDATE table
        SET message = "{lines}"
    WHERE
        id = {id}
    """

Also, indentation in embedded strings should be useful outside d-strings as well.

snippet = f"""
header
    {lines}
"""

After considering these points, I concluded that df-strings and dt-strings should not affect embedded strings. If this functionality is to be added, I think it would be more appropriate to extend f-strings and t-strings instead, as your previous idea.

2 Likes
146

IMO we have to be careful here not to take “do what I mean” too far.

What about

    lines = "line 1\nline 2"

    snippet = df"""
    header
        {lines}
    """

or

    lines = "line 1\nline 2"
    snippet = f"header\n    {lines}"

?

Because what matters most for me is that d-strings behave just like the equivalent “normal” strings, it’s essential to me that all 3 of these snippets act the same. So with that in mind, no, I don’t agree that the desired snippet is the one you suggest.

2 Likes
147

@pf_moore
Yes I get your arguments are valid, thus my idea was bad.

@methane
After I did suggest the reindentation symbol I had an answer that this can be a different PEP, I still think the simplified deindentation is not that meaningful without a simplified reindentation but this is my opinion and yes it might be an orthogonal proposal later.

148

One suggestion that might solve the current disagreements about whether newlines should be stripped at the start/end of d-strings:

What if this proposal also included two other new string-types, say s-stringsand e-strings(for “start” and “end”, respectively *). So these new string modifiers would strip newlines (or possibly all whitespace) from, respectively, the start and end of any string type.

With that, the following would all be valid ways of expressing the same string content:

"Hello\nworld!"

s"""
Hello
world!"""

e"""Hello
world!
"""

....ds"""
....Hello
....world!"""

....dse"""
....Hello
....world!
...."""

While the extra two letters for the main form proposed in PEP 822 might look a bit less elegant and would take some time to get used to, the consistency, simplicity and logic would in my opinion (at least right now) far outweigh the clumsiness. If these new string types would also include stripping of whitespace in general, we might even start to see forms like:

fe"{mystring}"

instead of:

mystring.rstrip()

* I would have preferred l and r, for “left” and “right”, but r is unfortunately already taken. However, a positive argument for s, e, and d is that they are all close together on a keyboard.

149

I would become -1 on this PEP in that case. That is way too much complexity for something that doesn’t show up in every program.

7 Likes
150

I would become -1 on this PEP in that case. That is way too much complexity for something that doesn’t show up in every program.

Well, I sort of agree, so let’s simplify. The end stripping is really not needed (I believe the main arguments in this thread in this direction was to strip newlines at both the beginning and the end for symmetry’s sake). Use cases are few. It is the stripping at the start which creates the inconsistencies with multi-line strings, vertically. So if we reduce this idea to a combination of two string types, d-strings and l-strings(for “left”-strip), then the usability improves and clunkiness reduces significantly. This allows for new use cases, such as:

MY_VERY_LONG_MODULE_LEVEL_CONSTANT = l"""
Hello
world!
"""

MY_VERY_LONG_MODULE_LEVEL_TEMPLATE_CONSTANT = tl"""
{some_text}
"""

Importantly, it allows for docstrings with indentation that preserves the current docstring style with a summary in the first line, e.g.:

def my_hello_world_func():
d"""Simple function printing "Hello, world!"

Here is a longer description of this amazing
function.
"""

Which is a single-character edit of existing docstring content that is completely consistent with normal multiline style. Alternatively:

def my_hello_world_func():
dl"""
Simple function printing "Hello, world!"

Here is a longer description of this amazing
function.
"""

Main arguments are consistency with existing string types, docstring style, textwrap.dedent() and lstrip(), to put a higher value on “Explicit is better than implicit”, allow for a few more use cases and variations, at the cost of an extra character.

151

I personally still find it to be too much for what it the use case is. I suspect Inada-san will choose an approach, and if the SC accepts it then we will all just internalize it on our first misuse the chosen semantics as a lesson learned.

3 Likes
152

Python strips indent from docstring already.
It is possible to use d-strings in docstrings, but there is no benefit to making d-strings more complex for the sake of replacing existing docstrings with d-strings.

I found many places where I could replace large amounts of `textwrap.dedent()` with d-strings, and other multi-line literals that could be replaced with d-strings, but I don’t think the fact that d-strings start on the line after the first quote makes them difficult to use at all.

From the votes in the previous thread, I gave up on using a feature like from __future__ import dedent_strings. There is no need to replace all triple-quoted strings with d-strings; it is sufficient to allow users to replace only the strings they want to use as d-strings.

Therefore, if d-string is to be added, I am convinced that it is best for them to start on the line after the opening quotes. If you want to argue against that, please show specific use cases where the current specification would be inconvenient.

The main reason I think it would be better to add d-strings rather than str.dedent() is the poor compatibility between t-strings and str.dedent(), but t-strings are a relatively new feature, so it is difficult to get feedback from users who are using multiline t-strings on a large scale. The SC has said that they want to find an external sponsor to prove that d-strings are useful outside of the Python core, so this PEP is suspended until that is found.

9 Likes
154

No, this was not about usability. In fact, personally I would like to be able to directly replace the hundreds of dedent("""\ blocks I have scattered throughout my code with d""" as defined in the PEP!

I mistakenly understood from my reading of this thread that inconsistencies with other string types was a potential blocker for the PEP (which I support), and since the inconsistencies also annoyed me a bit I wanted to put my idea out there. With a better understanding of the situation I fully agree that it was too complex for the purpose, so I withdraw it.

155

I was not aware that docstrings now are actually dedented (from 3.13 onwards). I experienced a related issue recently that I (from memory) chalked up to the lack of dedenting of docstrings in CPython, which partly lead to my misguided and now withdrawn idea above. I realize now that the real cause of my issue was rather the lack of support for handling nested dedenting in docstrings, e.g. :

def my_func(number: int):
"""my_func takes a number and does something with it.

Args:
number (int): a number provided as input. this
description is long, so it needs multiples lines.
"""

Here, the parameter list is frequently visualised with whitespace blocks in developer tooling (IDEs, type checkers, I cannot remember the exact combinations used, but it does not really matter here). PEP 257 – Docstring Conventions | peps.python.org makes docstring formatting an issue for tools to handle, which makes sense in the context (early Python), but my experience is that the lack of standardized docstring formatting and tooling in stdlib is causing major inconveniences for developers today.

Leaving my misguided idea above behind and relating to the current PEP822 draft only, I do have another idea I would like to share that I think could be a major use case for d-strings, which I will present in a new post below.

1 Like
156

New (and I believe much better!) idea for a docstring macro engine, as a use case for the current PEP822 draft (going slightly off tangent here, but bear with me):

Most developers have the DRY (Don’t Repeat Yourself) principle in their backbone, however annoyingly, Python out-of-the-box does not support a way to avoid duplicating content in docstrings. This is, I believe, one of the arguments for the now withdrawn PEP 727 – Documentation in Annotated Metadata | peps.python.org which would have allowed attaching e.g. parameter docs to the actual parameters through the type system. One of my issues with PEP 727 was the cumbersome notation, but I found the idea sound and useful in some, but not all cases. In any case, I believe there is room for an official, but complementary approach to allow DRY for regular docstrings. The lack of such a solution has lately caused me to spend considerable effort to implement my own in-house docstring macro system (where supporting correct indentation was one of the main headaches), which I would be happy to scrap if I had an alternative.

Consider something like this:

  • t-strings (template strings) and d-strings are supported for docstrings
  • variables to be substituted into docstring templates are defined in a confined way, not fully runtime, e.g. in a module.dpy file (or some other clever way)
  • The Python parser or other preprocessor does the template substitution pre-runtime (somehow, somewhere, I don’t know the architecture of CPython)

Then, one could do something like:

mymodule.py:

def my_func(number: int):
    dt"""
    my_func takes a number and does something with it.

    Args:
    {number_param}
    """

mymodule.dpy:

number_param = d"""
        number (int): a number provided as input. this description is long, so it
            needs multiples lines.
    """

For this to really be powerful, template insertion with dedent handling could be supported. This would allow moving the dedenting from the d-string to the insertion, so that one could e.g. write the more user-friendly variant (here following the {-> notation proposed above PEP 822: Dedented Multiline String (d-string) - #36 by hprodh ):

mymodule.py:

def my_func(number: int):
    dt"""
    my_func takes a number and does something with it.

    Args:
        {->number_param}
    """

mymodule.dpy:

number_param = d"""
    number (int): a number provided as input. this description is long, so it
        needs multiples lines.
    """

So my argument in favor of the current PEP as it stands is that a powerful and relatively simple docstring macro engine could be built on top of it with:

  • PEP 822 providing a convenient way to:
    • define multistring macro content (handling the correct amount of dedenting out-of-the box)
    • alternative docstring formatting, with the description line below the “”" line (which looks better to me)
  • A new docstring macro PEP would add most of the convenience straight away (complementing PEP 727-like approaches)
  • A template substitution with dedenting PEP would provide additional power/usability

Thoughts?

1 Like