Make print() use a single write() call

Ideas
1

Currently, print("a", "b") makes four separate calls to sys.stdout.write():

sys.stdout.write("a")
sys.stdout.write(" ")
sys.stdout.write("b")
sys.stdout.write("\n")

I propose changing print() to combine its output into a single write() call whenever possible:

sys.stdout.write("a b\n")

Motivation

The main benefit is preventing output from concurrent print() calls from being interleaved.

For example, structlog provides two logger implementations:

  • PrintLogger uses print().
  • WriteLogger uses a single write() call per log message.

When two threads log "msg1" and "msg2" concurrently using PrintLogger, the output can be msg1msg2\n\n instead of msg1\nmsg2\n.
WriteLogger avoids this particular problem by writing each complete log message in one call.

This means users need to understand the internal implementation of Python’s print() function to choose the safer logger for multithreaded applications.

Benefits

  • Better behavior with multiple threads: Output from a single print() call is less likely to be interleaved with output from other threads.
  • Better performance for small outputs: Combining several writes into one can reduce Python-level calls and I/O overhead.

Drawbacks

  • Performance and memory overhead for large outputs: Constructing a combined string requires additional memory and copying, potentially making large outputs slower.
  • Changed error behavior: Currently, if converting an argument to a string or writing to the stream raises an exception, earlier arguments may already have been written. With a single write, conversion errors would occur before any output is written.

Questions

  1. Do you think this change is worth making, given the compatibility and performance trade-offs?
  2. Would this change require a PEP, or could it be handled through the normal CPython issue and PR process?
4 Likes
2

Usually, when you use print() with multiple arguments, the semantic is “I want to print that, but I don’t want to format it a string directly and just want to use sep= and end= features”. So I think it’s an improvement here. Usually errors come from computing the data being printed, not for rendering it as a string IMO.

I think this can be handled without a PEP but I’m not entirely sure because print is something REALLY fundamental in Python and used a lot in many CLIs for instance (and this could affect them because they wouldn’t necessarily like to be slower here).

Instead of constructing a combined string, we can write it using “adaptive chunks” in the sense that you try to fit as many arguments you can in a single call, and if one argument is larger, then you just print it on its own. We don’t need to reduce the number of syscalls to 1, but reducing it where it makes sense would be better.

To make it maybe more reasonable when one uses file=..., we could also add a atomic parameter (name to be determined) which is set to True to indicate that you fit it in a single syscall, None for the current behavior or False to indicate that you can one or more syscalls.

That being said, I don’t know if this could negatively affect CLIs in general where they would just print multiple things in a loop. Is the syscall slower or is the pre-formatting slower? that’s a something to investigate.

I’m in favor of this proposal for the thread safetiness but, OTOH, one could circumvent this by (1) documenting the fact that each argument, separator and end value are printed with different syscalls (2) by advising users to format the strings first.

2 Likes
3

TextIOWrapper.writelines(lines: list[str]) currently inherits the implementation from IOBase.writelines(), which simply calls self.write(line) repeatedly. This means it acquires and releases the lock for each line, adding overhead and allowing other write() or writelines() calls to interleave between lines.

One possible approach would be to override TextIOWrapper.writelines() so that it calls self.write() for each string in the iterable while holding a lock across the entire operation. Then, print(a, b) could call:

sys.stdout.writelines([str(a), " ", str(b), "\n"])

This might prevent interleaving without the overhead of constructing a potentially large concatenated string.

The advantage of this approach is that it could both prevent interleaving in print() and improve the performance of writelines().

However, we would need to consider how to handle subclasses of TextIOWrapper that override write(), or whether it would be acceptable not to support that case.

P.S. This approach cannot completely prevent interleaving because TextIOWrapper releases its critical section while calling buffer.write(). In particular, when Python is started in unbuffered mode (-u), stdout uses write-through rather than line buffering, so buffer.write() is called for each line, causing the critical section to be released.

4

Would it be worth doing this sort of coalescing with some sort of threshold? All the benefits you’ve described are going to be very real for short writes (four separate calls for four individual characters), but won’t make as much difference if it’s larger. Or would the extra complication undo the benefit?

5

Of course, we could implement a mechanism that flushes the buffer once the concatenated output is about to exceed, say, 4,000 characters.

The primary motivation for this proposal is to reduce interleaving in multithreaded programs. Since small print() calls are more common in real-world applications, improved performance would also be a benefit, but performance optimization is not the main goal.

Therefore, rather than choosing the buffering threshold based on the point where buffering starts to hurt performance, I think we should set it relatively high, as long as the performance overhead and memory usage remain acceptable.

2 Likes
6

Thank you for posting this. This idea would greatly benefit IDLE and likely other applications that replace sys.stdout/stderr with something for which .write is substantially slower than in a console. I discovered the multiple-write behavior when a user filed an issue about print output being so much slower in IDLE than in REPL. My ‘solution’ was to add the following to the “Running user code” section of the IDLE doc, which discusses differences from REPL behavior.

Sending print output from one process to a text widget in another is slower than printing to a system terminal in the same process. This has the most effect when printing multiple arguments, as the string for each argument, each separator, the newline are sent separately. For development, this is usually not a problem, but if one wants to print faster in IDLE, format and join together everything one wants displayed together and then print a single string. Both format strings and str.join() can help combine fields and lines.

I considered having the replacement .write join strings, but end='' suppresses what would normally be the ‘send it’ signal. (A time-based send might be an alternative, but getting such right is more tricky.)

About back compatibility: I was surprised to discover that one print did not equal one write. I do not remember reading anything making piecemeal writing part of the builtin definition, and I have considered ‘print’ to be a higher-level ‘write’ with most formatting options. The doc for the flush option Output buffering is usually determined by *file*. However, if *flush* is true, the stream is forcibly flushed. says to me that even multiple prints may be combined before delivery.

7

I am :+1: overall, I think having choices in how many write calls happen will be useful. In a number of cases where Python already has a big in-memory buffer being passed to print (think bytes, str, …). In those cases collecting all the buffers then doing one list of buffers writev type API call helps a lot while actually reducing memory usage vs today.

It would be useful if we maintain a “minimal buffering” option for PYTHONUNBUFFERED. In particular I have written __str__ methods which result in a SIGSEGV or similar and having as much output as possible printed at that point can help save debugging rounds. To me though that is a small edge case.

I think it’s important if a __str__ or other method raises we do write to the underlying device as much as possible before returning / flush in the finally effectively.

8

I’m very +1 on this proposal. Avoiding interleaving in multithreaded situations is very useful.

However,

On its own, I would be against this - I would expect the call to either write everything, or raise (e.g. if a __str__ raises) and write nothing (“strong exception guarantee”). So that semantic would fit with the idea of “join everything first, then write”.

Especially, if I wrote

print(a, b)

and b.__str__ raised, it would be surprising that it printed a.__str__() and then didn’t put a newline after it, since I didn’t specify end=''.

But if there are compelling reasons to write arguments separately such that print might raise part-way through (such as if you’re printing lots of huge objects, though I doubt that would be a frequent occurrence), then it wouldn’t really cause any trouble for me (after all, it behaves that way at the moment).

I think the following (obviously pseudocode) implementation could be a sensible compromise:

def print(*args, sep=' ', end='\n'):
  if not args:
    write(end)
    flush()
    return
  first, *rest = args
  r = first.__str__()
  try:
    for arg in rest:
      if should_send_now:  # optional
        write(r)
        r = ''
      r += sep + arg.__str__()
  finally:
    write(r + end)
    flush()

In other words, in all cases, end would always be written, unless an exception occurred when trying to write the first argument, in which case nothing would be written at all.

should_send_now, if used, would be whatever heuristic condition was chosen to allow very large prints to be sent in multiple pieces.

9

This is the behavior of Python currently when using print to sys.stdout. Most the time that is “internally buffered” until there is a newline (linebuffered). Each object has its __str__ called and then is written to the file-like stream separately.

I’d be okay with “print always does the newline + flush if specified” at the end but that feels like building more special behavior which is both not the present behavior or something that is easy to explain/understand. Code which wants to “format everything ahead then print all or none” does still work in that model, you can build a buffer of data then just call print() with one arg.

1 Like
10

Indeed, I’ve just checked, on the REPL:

$ python3
Python 3.11.2 (main, Oct  7 2026, 12:35:07) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Bad:
...   def __str__(self):
...     raise Exception('foo')
... 
>>> 
>>> print(1, Bad())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __str__
Exception: foo
1 >>>

The fact that 1 appears immediately left of >>> shows that it’s not printing the newline.

Though I can’t say I’ve ever experienced this before (since I don’t tend to have __str__ methods that raise!), I would say that this existing behavior is surprising.

It’s entirely possible that this is so unlikely to ever be hit in practice that it’s not worth fixing, and that’s fine. But if we’re talking about potential behavior changes anyway, we probably should aim for the “nicest” behavior, and I think the following output (which my pseudocode implementation would have produced) would be “nicer”:

>>> print(1, Bad())
1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __str__
Exception: foo
>>>
11

int.__str__ raises. It is the one that I’ve had to protect from the most often.

L = [_**_**_ for _ in range(7)]
print(*L)
1 Like
12

I like the idea of joining print() arguments + end argument into a single string to only call file.write() once.

The usual case is that a line is around 100 characters. The change can have a negative impact on performance if the line is very long (ex: 1 MB): print(long_line) would compute long_line + '\n' which requires to duplicate the memory (double memory usage) just to add a newline character at the end. Maybe we can implement an advanced mode using writev() to pass an array of bytes strings, but that can be addressed later.

If print() arguments contain newlines, I still consider that it’s a good thing to only call file.write() once. I don’t think that we should split the strings at newline to call file.write() once per line.

I don’t think that it’s a good idea to write end if formatting an object as str raised an exception.

Instead, I would propose:

import sys

def atomic_print(*objects, sep=' ', end='\n', flush=True, file=None):
    if sep is None:
        sep = ' '
    elif not isinstance(sep, str):
        raise TypeError(f"sep must be None or a string, not {type(sep).__name}")

    if end is None:
        end = '\n'
    elif not isinstance(end, str):
        raise TypeError(f"end must be None or a string, not {type(end).__name}")

    if file is None:
        file = sys.stdout
        if file is None:
            return

    parts = []
    try:
        for obj in objects:
            if parts:
                parts.append(sep)
            arg = str(obj)
            parts.append(arg)
        parts.append(end)
    finally:
        # call write even if str(obj) raised an exception
        try:
            line = ''.join(parts)
        except:
            # most likely error: MemoryError
            for arg in parts:
                file.write(arg)
            if flush:
                file.flush()
            raise

        file.write(line)
        if flush:
            file.flush()

Note: the code can be micro-optimized if sep, end and/or line are empty strings. But I prefer to keep my code short here :slight_smile:

Example:

class BadStr:
    def __str__(self):
        raise Exception("not good")

end = '\\n\n'
atomic_print('Hello', 'World!', end=end)
try:
    atomic_print('before error', BadStr(), end=end)
except Exception:
    pass
atomic_print('exit', end=end)

Output:

Hello World!\n
before error exit\n

3 lines are written, but only 2 lines (and 2 end) are displayed: the second print() doesn’t write the end since BadStr() raised Exception.

13

Sample implementation of that idea.

Performance is still not very good, but we may be able to improve it by adding an internal API to TextIOWrapper that accepts a list[str] and using that API.

2 Likes