Writing out of bounds with the help of `seek`: Always NULL characters?

Python Help
1

In the following code:

text = "0"

with open("random.txt", "w") as f:
    num = f.write(text)
    f.seek(15)
    f.write("Hello")

When I open the `random.txt` file, I’ll see the first character is `0`, then 14 NULL characters, and then the “Hello” sequence.

The documentation on the io module says that it’s undefined behaviour when I use an integer not given by the `tell` method, or when it’s not zero. io — Core tools for working with streams — Python 3.14.6 documentation

Is it really undefined behaviour, similar to what one would get in C?
But then, why should we not get something like a segmentation fault, or some type of warning from the standard library?

Are there any security or stability implications in this behaviour?

Note: I’ve just noticed that Python’s official documentation is outdated.
The offset integer is now what’s called a cookie integer!

2

Compared to C, it’s more like “unspecified behaviour”. Allow me to demonstrate…

with open("example", "w") as f:
    f.write("\N{PENCIL}")
    f.seek(1)
    f.write("*")

with open("example", "r") as f:
    f.read()

You’ll probably get a UnicodeDecodeError. Why? Because f.tell() returns a byte position. In a text file, that could potentially land in the middle of a character (as it does here - U+270F PENCIL encodes into UTF-8 as E2 9C 8F, and we then overwrite the second byte with an asterisk, making the invalid sequence E2 2A 8F). You’ll never get that back from tell() as it will always ensure that an entire character has been written (it goes straight from 0 to 3), so depending on the encoding, you can cause all manner of weird results.

I don’t think you’ll ever have security risks from this particular problem, but you are largely on your own here. Note that the concerns you’re looking at there are specific to text files. Here’s the corresponding method on binary files:

So I would generally expect that the failures you’ll see are mostly related to text encoding. However, there are no guarantees at all when you do something other than the four supported operations (one of which is useless). In theory, you could get other sorts of errors or failures.

3

Thanks, for the information. :wink:

4

The Python interpreter is written in C and when it calls a C function with undefined behavior, its behavior is of course also undefined.

However, the C standard defines only minimum requirements which any C implementation on any OS must fulfill. Implementations are free to provide stronger guarantees. In particular, POSIX precisely defines the meaning of the offset. So if you are running on a Unix-like OS like Linux or MacOS, you are not only safe, you can determine exactly what this code must do. It might still do weird things or simply crash on other operating systems.

Because “undefined behavior” means just that: The standard doesn’t define the behavior. The code might do what you expect, it might do something unexpected but still reasonable in retrospect, it might do something completely random, it might crash, it might set your CPU on fire, …

5

You’re probably thinking of C APIs like lseek(2) - Linux manual page but there’s nothing there about undefined behaviour. As noted, there’s nothing undefined about seeking in binary files, only text.

6

No, I’m thinking about fseek(). In the C standard calling fseek() on a text stream is undefined except in the cases mentioned by the OP. However, in POSIX its behavior is defined. lseek() is not a standard C function but a POSIX function and it doesn’t care about text or binary streams since that distinction doesn’t exist at the UNIX system call level.

Granted I don’t know whether CPython does call fseek() (I have the vague memory that it used to use stdio but that was changed at least for unixoid platforms at some point), but if it does, the behavior of fseek() directly impacts Python programs. And the restrictions in the Python documentation seem to be at least inspired by the restrictions in the C standard.

7

Ah. It’s also worth noting that the C definition of “undefined behaviour” won’t apply to a function, as the compiler can’t optimize things away based on assumptions that turn out to be false. Can you find anywhere in the C standard that describes fseek as having “undefined behaviour”, or only wording like “the remainder of the file [is] undefined”? Those are very different claims.

8

Hmm, I don’t see what encodings and text mode have to do with the “out of bounds” seeking they’re talking about (seeking to 15 when the length is only 1). That applies to binary mode as well…

9

@ialvata The documentation is where behavior is defined. So if it doesn’t define some behavior or even explicitly says that some behavior is undefined, then it is undefined.

10

And it’s not undefined in binary mode - see the linked Python docs. Seeking past the end may be unusual but is not a problem. The reference to “undefined” came from the text documentation as the OP linked.

11

I don’t see it. Where exactly?

12

I can’t find it too. Anyway:

>>> with open("example", "wb") as f:
...     f.write("\N{PENCIL}".encode("utf-8"))
...     f.seek(20)
...     f.write(b"*")
...     
3
20
1
>>> with open("example", "r") as f:
...     f.read()
...     
'✏\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00*'

at least on my machine.

EDIT: adjusted the code for having an actual pencil ^^’

13

Compare:

Only one of these mentions limitations.

14

To be clearer: You said “it’s not undefined”. So where is it defined?

15

It does what you would expect. Living in fear of C’s “undefined behaviour” bugbear is not justified unless it actually says “undefined behaviour”.

16

That’s not true, I would not expect that.

17

That’s not what “undefined behavior” means. It means that the standard doesn’t define what will happen and that the implementation can do whatever it wants. This is true for many standard functions if you call them with different values than specified.

7.23.9.2 The fseek function
…
For a text stream, either offset shall be zero, or offset shall be a value returned by an earlier successful call to the ftell function on a stream associated with the same file and whence shall be SEEK_SET.

Violating a “shall” requirement causes undefined behavior, as chapter 4 Conformance points out:

If a “shall” or “shall not” requirement that appears outside of a constraint or runtime-constraint is violated, the behavior is undefined.

IIRC correctly, the reason for this restriction was OSs with record-oriented text files. For these implementations, the offset parameter couldn’t be a simple byte offset but had to encode the record number and offset within the record somehow which could mean that not all values encoded valid positions. Rather than trying to specify an algorithm which would work for all OSs or at least mandate specific error handling, the standards committee decided to leave the behavior undefined. They did that a lot.

18

Here: Change the stream position to the given byte offset, interpreted relative to the position indicated by whence, and return the new absolute position.

Note that this is quite subtle. Here is says “byte offset” and doesn’t limit that further. So you can seek to any byte offset. For text streams it says just “offset” without specifying what kind of offset this is and says that only 0 and values returned by ftell are allowed.

19

No we’re talking about what happens when you seek/write out-of-bounds, whether that always results in NULL characters. That’s not defined there, is it? Does Python define that anywhere?

20

The second paragraph does not explicitly limit seeking to within bounds (emphasis mine):

Independent of its category, each concrete stream object will also have various capabilities: it can be read-only, write-only, or read-write. It can also allow arbitrary random access (seeking forwards or backwards to any location), or only sequential access (for example in the case of a socket or pipe).

As I understand io, the precise behavior is platform-dependent, for example in the ability to create holes and seek to them. Python’s io abstraction does not assume or require these capabilities; it only exposes them when the underlying implementation supports them.