PEP 750: Tag Strings For Writing Domain-Specific Languages

Thank you to the PEP authors for their hard work in putting together the proposal. However, from the perspective of someone who teaches python to college students, I am not seeing how the practical utility of tag strings outweighs the significant downsides identified at various points in this thread.

Could the authors add to the PEP some compelling ‘before and after’ examples from the python standard library and popular python packages to show how the proposed approach would have helped in actual code bases?

If the idea is accepted, it would be very nice to disambiguate the tag from surrounding text. Could something similar to what is done for html tags or regex named groups (angle brackets) be used: "?P<tag> my string" or simply <tag>"my string"?

Thanks.

8 Likes

Not what was asked for, but a short common example, strictly textual templating with escaping:

# Context
class Safe(str): pass
def escape(s:Safe|str) -> Safe:
    return s if isinstance(s, Safe) else Safe(s.replace('bad','good'))
# A way of approaching this prior
Safe(f'<div>{"".join(escape(items))}</div>')

# New
# The supporting code for this is pretty long comparatively so, not including
html'<div>{items}</div>'

This still supports multiple items, this still does escaping, it still marks the result as safe to include in other template strings. In addition, due to it being able to return whatever it wants, it can set itself up to iteratively go through the tree of templates used, to be able to produce the page in a streaming fashion, allowing the browser to get the information the moment the web app can produce it, if involving slow queries etc, instead of needing to wait for the whole page to be built.

1 Like

Would eagerly binding but deferred evaluation be an option?

tag"abc{i+x}def"
# Not this
tag("abc", lambda: i+x, "def")
# Instead this
tag("abc", (lambda i,x: lambda: i+x)(i, x), "def")

Would be a anti-edge-case edge case to learn about though

I think it would still be surprising, not so much for +, but for calls and attribute lookups:

x = [1, 2, 3]
t = tag"I have {len(x)} items"
x.append(4)
print(t)

In the above case, you might have enough context to know what the output will be, but that isn’t a typical feature of the language. Semantics like this should be consistent enough that you don’t have to care what tag actually does in order to figure out whether len(x) happened before or after the .append.

3 Likes

As a data point, I read that without thinking, and my immediate expectation was that the output would say 4 items. I have no idea why that felt like the natural answer to me - I’m happy to concede that if it had been an f-string, I would have expected 3.

It’s possible that my instincts have been confused by this whole discussion, but I don’t think so - I really wasn’t thinking about early/late binding when I read this (even though it’s the point of the message), I just skimmed the code and then did a massive double-take when I realised that (a) the answer wasn’t obvious and (b) the message was claiming that people wouldn’t expect 4…

That’s not intended as an argument for either interpretation. And whatever we decide are the actual rules, people will be able to learn them. But I mention it as a cautionary note for people who claim that instincts from f-strings will “obviously” transfer across to tagged strings in users’ minds…

(Feel free to believe that I’m just being dumb because I’m tired and had a beer earlier. That may indeed be the problem here :slightly_smiling_face:)

2 Likes

To me, this is a pretty compelling example but in the opposite direction to what’s intended. It seems to say that I can either have:

  • A trivial comprehension loop which concatenates some escaped objects and embeds them in a <div> (2 lines, easy to follow)
  • A slightly punchier line of code which tells me nothing about what it’s really doing, backed by some hidden stuff that’s too long to include which I expect means that it’s not nearly as simple as the trivial comprehension loop
2 Likes

But if I miss a Safe, the entire site breaks. If I miss an escape, the entire site breaks and I have a security issue. Currently a manual unavoidable process which makes such work default to being insecure. The solution currently is to have a full blown DSL in the form of a templating engine like Jinja, which means you are no longer using python in python, or start shopping elsewhere.

To further make a small point of it, I only just realized now that I messed up the more “simple” f-string version, by forgetting the comprehension.

And the long backing code for the tagged string is, IMO still very trivial
from collections.abc import Decoded, Interpolation, Iterable

def render(item):
    if isinstance(item, Safe):
        return item
    if isinstance(item, str):
        return item.replace('bad', 'good')
    if isinstance(item, Iterable)
        return ''.join(render(i) for i in item)
    return render(str(item))

def html(*args):
    o = ''
    for arg in args:
        match arg:
            case Decoded as s:
                o += s.raw
            case Interpolation as i:
                o += render(i.getvalue())
    return Safe(o)

I’m pretty sure I can keep adding unrelated lines between lines 2 and 3 until you don’t immediately think it :wink: But yeah, contrived examples are contrived.

What’s worse is that today it might always produce 3, and then tomorrow the tag author decides to “optimise” their implementation by switching to lazy evaluation, at which point it changes.

2 Likes

Maybe. My feeling is that I went “print t, so what’s t, it references len(x) which is 4, so OK”. What bothers me is that logic is exactly transferrable to f-strings, but that’s not how I think in that case. Maybe tagged strings “feel” different? I’ve no idea why, though.

Like you say, contrived examples are contrived. And unrepresentative.

Yeah, that’s bad. But arguably no more so than any other bug or regression in tag

I remain ambivalent. I’ve not seen any really compelling examples that demonstrate the usefulness of lazy evaluation. But I’m absolutely convinced that if we don’t default to lazy, lazy evaluation will simply be unavailable[1]. I’d rather not close the door on potential use cases, but I’d be much happier if the use cases were concrete rather than potential :slightly_smiling_face:


  1. As far as I’m concerned, workarounds like lambda: expr are dead in the water - no-one will use them, and they simply don’t fit with the “DSL” use case ↩︎

My personal preference would be to default to eager evaluation, but accept {-> expr} as syntactic sugar for {(lambda: expr)} in the substitution field syntax.

That could easily be added later based on concrete examples using the lambda syntax rather than needing to be part of the baseline proposal.

I’m not a fan of needing to “just know” whether a given tag string defers evaluation or not (although I do like that better than all evaluation being lazy by default).

7 Likes

I will try to summarize some of the concerns here:

  • Deferred evaluation has a number of issues, including with respect to static typing analysis and ergonomics.
  • Surprise, in that it overly generalizes the capabilities of f-strings.
  • Potential inadvertent use, such as len'foo' (which would be the same as len('foo')).

Therefore let me propose the following additional changes to the PEP to see if this would help reach consensus:

  • Tag strings should provide standard function calling semantics. So instead of lambda-wrapped expressions using annotation scope, all interpolations are strictly evaluated from left-to -right. Evaluations are therefore straightforward to read and follow, including by static type analysis.
  • Interpolation.value provides access to this evaluated value, instead of Interpolation.getvalue provding call-by-name or fexpr like semantics.
  • Uses a new dunder method, tentatively called __tag_call__, which will be called with the earlier Template I brought up. If this method is not available, a suitable TypeError is raised, much like trying to use len(42).

Tag strings would still support the key ability to interpolate with respect to the context of the target DSL, such as interpolating properly for HTML or SQL (as seen in popular JavaScript tagged template libraries like lit or squid or the somewhat similar React JSX). Supporting structured logging would be possible.

It would also still be very much possible to support at a library level deferred/lazy evaluation, much as we see with Django’s QuerySet or Panda’s DataFrame. In particular, my earliler example of markup'{top} ... {middle} ... {bottom}' would still work in that middle and bottom could do a minimal amount of work to setup (much like a generator, including making use of a generator) before returning, allowing top to complete its rendering before doing the additional work in middle and bottom.

On the other hand, it would not possible to write a lazy f-string using this revised approach.

3 Likes

I’m not sure I fully understand what you are proposing but from my understanding it seems like it should still be possible. There are different aspects to what counts as “lazy” here so perhaps I am thinking about a lazy f-string as meaning something different from what you are referring to.

Let me give a concrete example for where I would want a lazy f-string. In SymPy it is quite common to have a function that does something like:

def some_func(expression):
    if some_condition(expression):
        raise ValueError(f"does not work for {expression}")
   ...

When the exception message is shown in a traceback it is usually nice to see the expression like x**2 + 1 that caused the exception. The problem then is that expression here means a symbolic expression and those can be large. Converting the expression to a string can therefore be expensive (in some contexts the string representation grows exponentially).

It can also then be possible that some other code catches this exception in a loop like:

for expression in expressions:
    try:
        results.append(some_func(expression))
    except ValueError:
        results.append(other_func(expression))

Now our expensive string conversion is happening over and over in a loop and the string is always discarded. In most cases the expression is not super large but it can still be wasted CPU cycles to make the string representation and I have found cases where profiling revealed this to be the dominant cost of a higher-level operation.

The problem though is not that f"{expression}" eagerly retrieves the variable expression from the local namespace. Rather the problem is that it eagerly converts the expression into a string like str(expression). We don’t want to convert the expression into a string eagerly if the string is never going to be displayed.

There are of course ways round this like old format strings:

class MyValueError(ValueError):
    ...
    def __str__(self):
        return format(self.format_string, *self.args, **self.kwargs)

raise MyValueError("does not work for {0}", expression)

We had those before f-strings came along though and people still put loads of work into making f-strings because they are a bit nicer.

My understanding here is that I should be able to do something like:

raise MyValueError(lazy"does not work for {expression}")

Here the tag string eagerly retrieves the object expression from the namespace but the lazy tag implementation can lazily convert that object to a string later if the string representation is needed.

Are you saying that this kind of lazy f-string would not be possible?

Am I misunderstanding what it is that you mean by a “lazy f-string” and what it is wanted for?

3 Likes

I’m not the PEP author, but I believe that’s still possible. The “value” of the Interpolated object passed to the lazy function (tag) would be the expression object. You can keep that as a complex object in the return value of the lazy function, which would itself be an object rather than a string. That object would only calculate the string form of the expression in its __str__ method.

The sense in which “lazy” is being used here is that the name expression isn’t looked up at the time the lazy"..." syntax is encountered. Instead the lookup is deferred until later, when the tag object calls getvalue() on the Interpolated value.

So, for example, if your code mutates expression in the time between when you call lazy"...{expression}..." and when you use the value of that tag string, then the code will see the mutated value of expression.

Hmm, trying to describe that makes me realise that “lazy” semantics in the sense I just attempted to explain are really difficult to state precisely. That pushes me to -0.5 at least on lazy semantics as long as it’s true that there’s no identified use case that needs those semantics. Which seems to be the case, although I’m still not completely clear how the markup'{top} ... {middle} ... {bottom}' example needs lazy semantics (to be honest, though, I can no longer find the details of that example, so maybe I’m not remembering the details).

1 Like

There are two parts here:

  • Eager evaluation of the interpolations to the tag string. This is the change I’m suggesting, we go from deferred (lambda-wrapped expressions, available in Interpolation.getvalue) to eager (Interpolation.value).
  • Construction of some object, possibly a string, but it could be a DOM for HTML, a query object for SQL, JSON structured logging, etc. The tag function can use Interpolation and Decoded args from Template.args, possibly memoized with Template.source, to construct this object. This part is not changing with my suggestion to go from deferred to eager evaluation of interpolations.

So it depends on what we mean by a lazy f-string, but the one you describe can be constructed by this process.

1 Like

This is correct. In addition, the tag function could memoize any aspect of this calculation, as it sees fit. Perhaps for example it has already stringified the expression before.

One example that comes to mind is someone doing

log.debug(f'Logging something with {expensive_calculation()}')

so if we had dotted names, this could be

log.debug(lazy.f'Logging something with {expensive_calculation()}')

(or something equivalent like lazy_f or fl).

The difficult to state semantics you mention are maybe straightforward here: log.debug would complete using the lazy f-string before itself returns. In general, I would be very surprised that some other thread is mutating the namespace that expensive_calculation is working with, and of course this would be true if it were a lazy f-string or the usual f-string.

The problem with deferred evaluation is of course that one could do something like

keep_around = lazy_f'Logging something with {expensive_calculation()}'
# do other things, including modifying what expensive calculation is based on
log.info(keep_around)

and now there’s some inconsistency.

With my

markup"{top} ... {middle} ... {bottom}"

example, I was trying to look at some general idea in a markup language like HTML where I want to render things from top to bottom. But of course I only need to look at the top stuff first when it’s rendering to a browser. I don’t need what’s part of bottom to do that. So if have a tree of deferred expressions I can readily support that.

But of course I can always write my own version of that deferred expression tree, such as the SymPy expression mentioned here as I understand it. It doesn’t have to be part of this PEP, even though it would simplify this one case of a lazy_f tag function.

Thanks Jim. This addresses all my concerns with the proposed semantics (though I would suggest keeping getvalue and having it reraise any exceptions raised during evaluation - that way a tag that wants to ignore the captured values can do so[1]).

My other concern is still namespace pollution (that is, the presence of an xml tag meaning you can’t use xml as a variable), but I’m not sure that it will be a real problem, or how to prove now that it will/won’t be. I suspect that xml only having a single meaning throughout is better than having xml"{xml}" refer to two different xmls, but I don’t know how to convince myself of that. Maybe take this concern as a voice in the crowd, and only give it weight if it turns out quite a few people are nervous about this aspect.


  1. And on an implementation note, I suspect we’d need to wipe the exception after __tag_call__ returns to avoid leaking entire frames, or perhaps only capture the exception type in the first place. ↩︎

1 Like

The problem with this example is that log.debug("Logging something with %s", expensive_calculation()) doesn’t complete formatting the string when it returns - it bundles it up into a log record that keeps the value of expensive_calculation() directly and formats it later (potentially immediately, but potentially later on a separate thread).

So we already have a counter-example to the trivial lazy.f"" example, and the caller doesn’t have to make things any more complicated to be bitten by even-more-lazy semantics.

1 Like

Indeed and actually I don’t think you described it quite right. At least I would have avoided using the word “mutate”. The eager part is the name binding as you say but the object could still be mutable:

lazy_things = eager_things = []

string = lazy"here are {lazy_things}"

# Mutate the object:
eager_things.append("eager")

# Rebind the name:
lazy_things = ["lazy"]

print(string) # eager or lazy?

If I have understood the difference between the eager and lazy evaluation proposals here then it is whether this is going to print eager or lazy. Note that under the eager proposal it is still possible for the tag string to process the object eagerly or lazily so whether or not “eager” appears in the printed output does depend on how the lazy tag string implementation handles its inputs (it might call str(obj) right away or it may delay that until the print call).

2 Likes

After skimming through the posts in this thread, and mulling over this PEP for a little while I think it’s time to offer my 2 cents.

I have one big issue here, which is both a gripe and a suggestion.

The syntax should be foo(t'bar'), not foo'bar'. It follows that rather than a *args pack, the function foo should accept one argument, of some new built-in type that’s instantiated whenever t'bar' is encountered (and contains the information that would have been in the *args pack), but I don’t think that should be controversial.

(Replace t with some other fixed prefix if you wish, I’m not here to bikeshed that.)

Python’s existing string prefixes - b'', r'', f'', and some I don’t know about - give a far greater difference in functionality than replacing what is ultimately one variable name with another. b'' creates bytes rather than str. r'' changes the handling of backslashes. f'' interpolates variables. Tag strings are one specific piece of functionality, they should have one specific prefix.

I see several clear and significant benefits to the foo(t'bar') syntax:

  • Python keeps open its opportunities for the future to add new string prefixes.
  • It is a simpler and more obvious syntax: you want to call foo passing some special string expression, so write a function call to foo (foo(...)) passing a special string expression (t'...').
  • The function responsible for interpreting the tag string can take additional arguments, which may well come in useful if the function author wants to allow callers to use those additional arguments to affect how the interpreting is done. (Indeed, the t'...' expression alone could be used anywhere, not just as a function argument, but that’s besides the point.)

On the other hand, I cannot see any reason why the proposed foo'bar' syntax carries any benefit, other than saving 3 characters.

Hopefully this can be taken into consideration. To me, it’s a no-brainer to have t'...' just create an object of a particular type that can be used like any other Python object; am I alone in this thinking?

37 Likes

I’m not heavily invested in this design space, but I’ll say you’re not alone - to me this approach [1] is much less “magical”, in a good way. It seems easier to use and explain.


  1. using a t string prefix to create an object ↩︎

14 Likes