Deferred Evaluation for Template Strings

Interpolations of template strings are evaluated immediately upon their creation. However, in many situations, deferring the evaluation of an expression until it is actually needed can be useful. Currently, Python lacks an effective mechanism to support this capability.

Current Limitations

Within the current constraints, one approach to implementing deferred evaluation involves using lambda expressions, as demonstrated below.

template = t"Result: {(lambda: expensive_func(val1, val2, flag=True))}"

However, the problem with this approach is that it is verbose and ambiguous. Expressions such as {(lambda: ...)} are particularly cumbersome, and few developers would realistically choose to program this way, since such expressions reduce readability, clarity, and simplicity.

Furthermore, there is no clear way—either from a developer’s perspective or for the code that will later evaluate the template string—to distinguish whether a value is simply callable, or is intentionally wrapped for deferred evaluation. This ambiguity is especially problematic when the value itself has meaning as a callable object, such as a class, among other examples.

Because of these limitations, implementing deferred evaluation in current template strings is practically impossible, though not technically impossible.

!d Conversion

To address these issues, this document proposes introducing a new !d conversion (where 'd' stands for 'defer') to template strings.

This conversion wraps the internal expression in a lambda function and sets the conversion value to "d".

deferred_template = t"{1 + 2!d}"
interpolation = deferred_template.interpolations[0]

assert interpolation.conversion == "d" # Conversion is set to "d"
assert interpolation.expression == "1 + 2"
assert callable(interpolation.value) # Value is callable
assert interpolation.value() == 3 # Compute actual value when it is called

Unlike the previous approach, the conversion attribute makes it clear—both to developers and to code implementations that handle the template string—that the interpolation is deferred.

Examples

The following examples further illustrate the !d conversion.

Implementing f-string Behavior

The following code shows an example implementation of f-string-like behavior for template strings in fstring.py from the pep750-examples repository.

def convert(value: object, conversion: Literal["a", "r", "s"] | None) -> object:
    """Convert the value to a string using the specified conversion."""
    # Python has no convert() built-in function, so we have to implement it.
    if conversion == "a":
        return ascii(value)
    if conversion == "r":
        return repr(value)
    if conversion == "s":
        return str(value)
    return value

def f(template: Template) -> str:
    """Implement f-string behavior using the PEP 750 t-string behavior."""
    parts = []
    for item in template:
        match item:
            case str() as s:
                parts.append(s)
            case Interpolation(value, _, conversion, format_spec):
                value = convert(value, conversion)
                value = format(value, format_spec)
                parts.append(value)
    return "".join(parts)

Adding only two lines to this code enables support for !d conversion.

def convert(value: object, conversion: Literal["a", "r", "s", "d"] | None) -> object:
    """Convert the value to a string using the specified conversion."""
    # Python has no convert() built-in function, so we have to implement it.
    if conversion == "a":
        return ascii(value)
    if conversion == "r":
        return repr(value)
    if conversion == "s":
        return str(value)
    if conversion == "d": # This line is added for deferred evaluation support
        return value()
    return value

def f(template: Template) -> str:
    """Implement f-string behavior using the PEP 750 t-string behavior."""
    parts = []
    for item in template:
        match item:
            case str() as s:
                parts.append(s)
            case Interpolation(value, _, conversion, format_spec):
                value = convert(value, conversion)
                value = format(value, format_spec)
                parts.append(value)
    return "".join(parts)

This simple change naturally integrates the !d conversion into the existing system.

When Values Do Not Always Need Evaluation

Depending on the logging level set, log messages may not require evaluation. In such cases, the !d conversion can prevent unnecessary expression evaluation.

# If the logger level is set above DEBUG, expensive_func is not evaluated at all.
logger.debug(t"User stats: {expensive_func(val1, val2, flag=True)!d}")

When Values Change in Real Time

Another application of the !d conversion occurs when dealing with values that change in real time.

The following example retrieves a value from a specific API every second and logs it. Using the !d conversion both avoids unnecessary calculations when debug messages are not required, and permits the template string to be created once and reused repeatedly.

log_value = t"[{datetime.now().isoformat()!d}] current value is {value!d} (Attempt {i!d})"

for i in count(1):
    value = httpx.get("https://...").json()
    logger.debug(log_value) # datetime.now() and value are evaluated at the time the log is printed
    do_something(value)
    time.sleep(1)

The following example demonstrates periodically updating news headlines every minute in a hypothetical GUI library. In this implementation, the content of the text element is periodically updated through a deferred template string.

from gui import GUI
from news_api import get_headline

def main():
    window = GUI.create_window()
    text_element = window.add_element("text")
    # Fetch a new headline every 60 seconds.
    text_element.set_text(
        t"Today's News | {get_headline(section='economic')!d: <100} | {datetime.now().strftime('%H:%M')!d}",
        refresh_interval=60
    )
    window.start()

Relationship to Tagged String

PEP 750 originally proposed the concept of tagged strings. This approach deferred all interpolation by default. However, the PEP later evolved into proposing template strings, evaluated immediately in the same way as f-strings.

Unlike tagged strings, the !d conversion requires users to explicitly opt-in to deferred evaluation. This explicit opt-in approach also allows for more flexible control, as it can be applied only to specific interpolations that require deferred evaluation.

2 Likes

This was discussed and rejected in PEP 750. While it’s certainly a possible future enhancement, I strongly suggest that we wait until t-strings have been released (in Python 3.14) and we have some real-world practical experience with them before proposing extensions - especially extensions that were considered and deferred in the PEP process.

7 Likes

I agree. That said, I do hope this comes back around in a couple of years – I think the idea has merit. Maybe we can finally stop reading endless threads about lazy evaluation and have a quieter Ideas forum.

I’m a little unclear on some of the examples given, since they don’t defer the expressions in a lambda. Which results in things getting evaluated and bound to the Interpolation, unless I’ve forgotten some key details of t-strings.

2 Likes

Could make use of Builtins.lazy for lazy arguments if it was adapted.

template = t"Result: {lazy(expensive_func, val1, val2, flag=True)}"

Please don’t hijack every thread about any kind of deferred evaluation to push your preferred deferred. This is exactly what makes these sorts of proposals a pain to discuss. Can you focus on someone else’s proposal and its merits?

(Note: I don’t know for sure that you personally have hijacked any other threads, but it does seem to be a thing that happens a lot with deferred-evaluation threads. I’ve certainly seen a lot of it.)

3 Likes

How about making the whole template lazy?

>>> lazy = lambda: f"Hello {name}" # example with f-strings
>>> name = "world"
>>> lazy()
'Hello world'
3 Likes

Well, Python 3.14 isn’t out yet :wink:

But the alpha version allows some interesting things, including delayed evaluation.

def hello(name):
    print(f"hello {name}")

who = 'bob'
flavor = 'spicy'
embedx = t'Call function {hello:!fn} {who} {flavor}'
who = 'jane'

r = render(embedx)
print(r)
# → Call function hello jane spicy
embedn = t'Call function {hello} {who} {flavor}'
r = render(embedn)
print(r)

outputs
Call function hello jane spicy
Call function <function hello at 0x7914f551fe20> jane spicy

(because the evalutation only happens with the “!fn” specifier).

Not perfect. e.g. “who” must be defined as something, even if not the value used at rendering. But that could be worked around with quoting e.g.
embedx = t"Call function {hello:!fn} {'who'} {flavor}"

render implementation hacked together in half an hour

import sys
import io
import inspect
from string.templatelib import Template

def render(template: Template, ctx: dict[str, object] | None = None) -> str:
    """
    Render a PEP 750 t-string, only treating interpolations whose
    format_spec == "!fn" as calls.  Any callable marked !fn will
    consume as many following interpolations as its positional args,
    be invoked, and its stdout captured inline.  All other
    interpolations (including those consumed as args) and static text
    are rendered in order.
    """
    ctx = ctx or globals()
    strings = template.strings
    interps = template.interpolations

    out: list[str] = []
    i = 0
    while i < len(interps):
        # 1) static text before this interpolation
        out.append(strings[i])

        interp = interps[i]
        expr = interp.expression
        spec = interp.format_spec

        if spec == "!fn":
            # deferred-eval the callable
            fn = eval(expr, ctx)
            if not callable(fn):
                raise ValueError(f"{expr!r} is not callable")

            # inspect how many fixed args, and if *args is present
            sig = inspect.signature(fn)
            params = sig.parameters.values()
            pos_count = sum(
                1 for p in params
                if p.kind in (inspect.Parameter.POSITIONAL_ONLY,
                              inspect.Parameter.POSITIONAL_OR_KEYWORD)
            )
            has_varargs = any(
                p.kind is inspect.Parameter.VAR_POSITIONAL
                for p in params
            )

            # decide how many following interps to consume
            available = len(interps) - (i + 1)
            take = available if has_varargs else min(pos_count, available)

            # eval each of the next `take` expressions
            args = [
                eval(interps[i + j].expression, ctx)
                for j in range(1, take + 1)
            ]

            # capture stdout of the call
            buf = io.StringIO()
            old_stdout = sys.stdout
            sys.stdout = buf
            try:
                fn(*args)
            finally:
                sys.stdout = old_stdout

            out.append(buf.getvalue().rstrip("\n"))
            i += 1 + take
        else:
            # normal interpolation: deferred eval and str()
            val = eval(expr, ctx)
            out.append(str(val))
            i += 1

    # trailing static text
    out.append(strings[-1])

    result = "".join(out)
    return result

I understand that you might be frustrated thinking that I intend to hijack a thread. However, I assure you this is not my intent.

Objectively, the situation is as follows:

  1. This is a thread on deferred evaluation for Template Strings
  2. My proposal for lazy type is a valid potential solution (among many others)

I have understood and appreciated the proposal and offered potential alternative solution. I think it is quite standard to pitch in alternative solutions and it has never been a problem.

I don’t think it matters whether potential solution is already existing feature, new one that I just came up with or something that I am working on in parallel.

And your accusation that I am “Hijacking a thread” given my brief comment consisting of 1 sentence is inappropriate at the very least - it is distorting objective situation at hand at my expense.

The fact that there are others who agree with your accusation towards me and no one bothers to say anything about such comment saddens me.

1 Like

That’s precisely what I mean. You look at a proposal, and immediately say “HEY! My proposal would do this better!”. Do you see the problem here? It’s hard enough to have a good discussion about deferred evaluation, given the many difficulties with getting a viable implementation, without having threads get redirected towards a completely different proposal.

Perhaps you should take it into account rather than be saddened by it.

I pushed the experimental code to pypi / github in case anyone is interested.

https://pypi.org/project/tstring-util/

1 Like

I see that this is one possible interpretation.

But what I said was “Could make use of Builtins.lazy if it was adapted.”

Notice “could”. It is a soft suggestive. could is used to talk about something that can happen. And theoretically, it can. I did not say that it should.


There were other similar suggestions (the only difference is that they were not referring to the ideas that they are working on). But you did not choose to interpret them in the same manner.

I don’t understand why you chose this interpretation of my words among many other while neither my language nor the context was explicitly suggestive of it.

Finally, my suggestion does not offer a complete solution to this proposal, but is rather a remark that there is possibility for using such for signalling.

Thus, I can not see in a slightest how interpretation of “hijacking a thread” is appropriate. And as it is a speculative accusation based on one of the worst possible interpretations, in turn is disrespectful.

1 Like

Have you done any benchmarks for overhead?

I.e. how expensive function would need to be to be worth it?

Of course, if the need is to bind arguments at evaluation, then it is worth in either case. However, if deferral is used for sole purpose of not doing unnecessary evaluations, it would be interesting to know (at least to me).