Is there a way to change the linesep in open()?

Python Help
1

The infile was encoded in UTF_16_LE and each line is end with “0x0d 0x00 0x0a 0x00”. When running the following code, Python read each line with end of “0x0d 0x00 0x0a” and causing the following error.

with open(infile, "rb") as f:
    for line in f:
        line_text = line.decode('utf_16_le')

Traceback (most recent call last):
File “d:\Works\Python\2utf8.py”, line 25, in
line_text = line.decode(‘utf_16_le’)
^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: ‘utf-16-le’ codec can’t decode byte 0x0a in position 384: truncated data

2

You can pass the encoding while opening the file

>>> f = open("file.dat", "rb")
>>> next(f)
b'f\x00o\x00o\x00\n'
>>>
>>>
>>> f = open("file.dat", encoding="UTF_16_LE")
>>> 
>>> 
>>> next(f)
'foo\n'
>>>
3

If there’s a reason that you have to read the file in binary and then decode line by line (as opposed to, as recommended, decoding the file as a whole before splitting into lines), I would recommend doing everything yourself including the line splitting. Read the file as pure bytes - the simplest way is to just read the whole thing into a single variable - and then split it as needed.

4

struct is also good for this. You can define the struct and then iterunpack the steam.

If it’s a single file and it’s all the same structure though, it’s probably best to just read it all at once and use the encoding argument of open.

5

Yes, it seems that this is the best solution at this moment.

6

I didn’t think of using struct as a solution, even in my dream:) I will take a look later.

Unfortunately, there is no encoding argument when open at the ‘rb’ mode:(

7

Why are you reading in bytes mode if you want an encoding argument? Wouldn’t it be better to simply read it using the correct encoding?

8

Expanding on this: If you open in text mode ("rt") rather than binary ("rb") you can specify newline to open() docs. For Binary I/O (rb) the line terminator is always b'\n'. There are more details about this in the file.readline() docs.

9

I was checking the docs for the open() function, and I could not see any clear statements that a newline argument is not allowed in binary mode. Maybe, the docs could be improved a bit there.

Actually, I tend to think that reading a binary file line-wise is a miss-feature. However, if that feature exists, I am not aware of a compelling reason why one cannot specify which sequence of bytes constitutes a newline, however specifying an encoding does not make sense.

10

:+1: would be nice to have. For a very commonly used function the open docs are already really, really long… need to find some way to split them into a more digestible form. There have been some long open PRs to improve this case and others but they tend to get stalled in details.

It is implemented as a direct byte comparison in a complex loop that would be hard to generalize. Multi-byte comparison runs into more edge cases of “what if one byte is in the last buffer and the next byte requires a read”. Currently Binary I/O doesn’t have multiple buffers at the time. Have some thoughts on supporting that, but with the current code that would be very complex.