# Change \`open().write()\` to guarantee all bytes will be written or an exception will be thrown

**URL:** <https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082>\
**Category:** Ideas\
**Created:** [November 13, 2024, 7:26pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082 "2024-11-13T19:26:48Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![cmaloney](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cmaloney/32/15851_2.png) [@cmaloney](https://discuss.python.org/u/cmaloney)\
**Post date:** [November 13, 2024, 7:26pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/1 "2024-11-13T19:26:48Z")

</div>

I’ve been looking a bit at what open() does under different circumstances, and found what to me is surprisingly different contracts for behavior:

1. 

```python
with open("testfile.txt", mode='wb') as f:
    print(f"{type(f) = }")
    count = f.write(b"test")

with open("testfile.txt", mode='rb') as f:
    print(f"{type(f) = }")
    buffer = f.read()

assert len(buffer) == count, "Should have read and written everything"

```

1. 

```python
with open("testfile.txt", mode='wb', buffering=0) as f:
    print(f"{type(f) = }")
    count = f.write(b"test")

with open("testfile.txt", mode='rb', buffering=0) as f:
    print(f"{type(f) = }")
    buffer = f.read()

assert len(buffer) == count, \
        "Usually reads and writes everything, but not guaranteed"

```

This differs

1. `.write()` both cases in the CPython codebase and ones I’ve looked at in open source code expect a `write()` to either write everything or throw an exception. CPython code and quite a few projects use `open(buffering=0)` or `_io.FileIO` directly with a single call and seem to expect that behavior[1]. For `read()` this is relatively safe because read-all or fixed size does fullfill that contract, and most the time `write()` does as well. The partial write only occurs under some cases, and CPython already retries a number. It comes up when there’s system limits (ex. [`getrlimit`](https://www.man7.org/linux/man-pages/man2/getrlimit.2.html) on Linux[2]) that the user encounters.
2. `.read()` in the [docs](https://docs.python.org/3/library/io.html#io.RawIOBase.read) doesn’t talk about [PEP-475](https://peps.python.org/pep-0475/) retries. In [code comments](https://github.com/python/cpython/blob/main/Modules/_io/fileio.c#L842-L851) the readall case isn’t noted currently.

This difference in behavior is what was originally specified by [PEP-3116](https://peps.python.org/pep-3116/), “Raw I/O” isn’t supposed to make more than one system call generally while BufferedIO will loop. With [PEP-475](https://peps.python.org/pep-0475/) a lot of cases were updated to try for better standard behavior.

I’m curious if it would be better to leave the behavior as is or if there are ways we could improve the standard I/O patterns.

I’ve been pondering at a couple possible tweaks:

1. Update `read()` code comments and docs that it retries per [PEP-475](https://peps.python.org/pep-0475/) and include the readall behavior (`read()` without a size retries until it finds `EOF` as indicated by a `read()` returning size 0).
2. Add a documentation warning around `FileIO`, and its `write` method that users must check the return size to guarantee everything has been written.
3. Make BufferedIO able to be used without a zero-sized buffer (pass through) efficiently. Update so `open(buffering=0)` always uses BufferedIO so code gets the “read and write are batteries included” behavior they expect when using `open()`.
4. Potentially updating `write()` in FileIO to retry partial writes. That will likely make code that relies on non-blocking I/O slower (goes from partial write returning size to an exception being thrown)

Curious other thoughts on how to possibly make this safer for what seems to be commonly used pattern which contains a subtle bug in the Python ecosystem.

* * *

1. a. [Code Search buffering=0 language:Python](https://github.com/search?q=buffering%3D0+language%3APython&type=code)  
b. [gh-126606](https://github.com/python/cpython/issues/126606)
2. [write(2) - Linux manual page](https://www.man7.org/linux/man-pages/man2/write.2.html)

> Note that a successful write() may transfer fewer than count  
> bytes. Such partial writes can occur for various reasons; for  
> example, because there was insufficient space on the disk device  
> to write all of the requested bytes, or because a blocked write()  
> to a socket, pipe, or similar was interrupted by a signal handler  
> after it had transferred some, but before it had transferred all  
> of the requested bytes. In the event of a partial write, the  
> caller can make another write() call to transfer the remaining  
> bytes. The subsequent call will either transfer further bytes or  
> may result in an error (e.g., if the disk is now full).

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 13, 2024, 8:46pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/2 "2024-11-13T20:46:24Z")

</div>

> [@cmaloney](#):
>
> Add a documentation warning around `FileIO`, and its `write` method that users must check the return size to guarantee everything has been written.

That is already [documented](https://docs.python.org/3/library/io.html#io.RawIOBase.write).

---

<div class="post-metadata">

**Author:** ![cmaloney](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cmaloney/32/15851_2.png) [@cmaloney](https://discuss.python.org/u/cmaloney)\
**Post date:** [November 13, 2024, 9:12pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/3 "2024-11-13T21:12:52Z")

</div>

My thought is adding a warning / more attention box that size must be checked (and possibly a linter / code style rule). CPython `importlib` had it off, and quite a few packages like `numpy` and `tensorflow` contain cases of it. It’s a usage pattern that seems to be common but has bugs (be nicer to me to change the behavior, but that has a lot of things it could break). `gevent` made a “WriteISWriteallMixin” to change the behavior. Numpy allows io.FileIO but doesn’t check the write size ([numpy/numpy/lib/format.py at main · numpy/numpy · GitHub](https://github.com/numpy/numpy/blob/main/numpy/lib/format.py#L743-L750)). So it feels like the information isn’t getting to end users or was lost somewhere along the way and needs its visibiltity raised.

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 13, 2024, 9:22pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/4 "2024-11-13T21:22:43Z")

</div>

Adding a hint in the [open()](https://docs.python.org/3/library/functions.html#open) documentation would make it more visible, as the [io](https://docs.python.org/3/library/io.html#text-i-o) module directs users straight to `open()`.

> The easiest way to create a text stream is with open(), optionally specifying an encoding:
> 
> f = open(“myfile.txt”, “r”, encoding=“utf-8”)

Not that users are likely to read the `open()` documentation either, but that hint would still be helpful for seasoned users.

---

<div class="post-metadata">

**Author:** ![xitop](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xitop/32/13242_2.png) [@xitop](https://discuss.python.org/u/xitop)\
**Post date:** [November 14, 2024, 1:26pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/5 "2024-11-14T13:26:19Z")

</div>

I’d like to add a brief note, that there is a difference between writing bytes (where partial writes are possible) and writing text strings.

---

<div class="post-metadata">

**Author:** ![ericvsmith](https://avatars.discourse-cdn.com/v4/letter/e/34f0e0/32.png) [@ericvsmith](https://discuss.python.org/u/ericvsmith)\
**Post date:** [November 14, 2024, 3:25pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/6 "2024-11-14T15:25:14Z")

</div>

Are you saying partial writes aren’t possible if a file is opened in text mode? That would be great. Is it documented somewhere? I’ll admit that all of the io abstractions make it hard to find things like this in the docs.

---

<div class="post-metadata">

**Author:** ![xitop](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xitop/32/13242_2.png) [@xitop](https://discuss.python.org/u/xitop)\
**Post date:** [November 14, 2024, 4:46pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/7 "2024-11-14T16:46:47Z")

</div>

> [@ericvsmith](#):
>
> Are you saying partial writes aren’t possible if a file is opened in text mode?

This is how I understand the `io.TextIOBase.write`:

> `write(s, /)`
> 
> Write the string _s_ to the stream and return the number of characters written.

and it makes sense, because - contrary to writing bytes in chunks - a partial write, e.g. one byte of a 3-byte UTF-8 char would make it very hard to write the next chunk.

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [November 14, 2024, 5:16pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/8 "2024-11-14T17:16:11Z")

</div>

I was referring to `open()` because its documentation covers the `buffering` parameter. A link to `RawIOBase.write()` would be useful there to explain unbuffered I/O.

Unbuffered Text I/O is not supported:

```python
open('/dev/random', buffering=0)
# ValueError: can't have unbuffered text I/O

```

---

<div class="post-metadata">

**Author:** ![cmaloney](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cmaloney/32/15851_2.png) [@cmaloney](https://discuss.python.org/u/cmaloney)\
**Post date:** [November 14, 2024, 5:36pm UTC](https://discuss.python.org/t/change-open-write-to-guarantee-all-bytes-will-be-written-or-an-exception-will-be-thrown/71082/9 "2024-11-14T17:36:12Z")

</div>

Yes, TextIO is largely safe / all text ends up written or an exception is raised (and I think where the user expectation of that comes from).

* * *

The `BufferedIOWriter` code does retry partial writes (so will get to all data written or an exception). And TextIO (generally) requires BufferedIO generally / gets it by default. `TextIOWrapper` is used directly around a `FileIO` ([which some code does](https://github.com/python/cpython/blob/1e3497e745d6559d30dbb65c914d19c759d60dc5/Lib/getpass.py#L47-L51)) contains a [write loop](https://github.com/python/cpython/blob/1e3497e745d6559d30dbb65c914d19c759d60dc5/Modules/_io/textio.c#L1625-L1628) for its various write paths as well so should largely be safe (although that could likely use a more comprehensive audit, some cases don’t seem to check the buffered write size…).

Splitting individual Unicode characters / partial utf-8 encoded character writes can and does happen, has had a couple bugs largely reported in WindowsConsoleIO to date ([gh-110913](https://github.com/python/cpython/issues/110913), [gh-82052](https://github.com/python/cpython/issues/82052)). There are some additional code paths I think have latent bugs around that (ex. `_Py_write` in fileutils caps at \_PY\_WRITE\_MAX and doesn’t watch for multi-byte character boundaries), but usually don’t happen to things which read immediately and occur when have particularly large chunks of data to a TTY.