# How to fix utf-8 error when reading text file?
**URL:** https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117
**Category:** Python Help
**Created:** [April 2, 2024, 1:37pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117 "2024-04-02T13:37:53Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@c-rob](https://discuss.python.org/u/c-rob)
#### Post date: [April 2, 2024, 1:37pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/1 "2024-04-02T13:37:53Z")
I have Python 3.12 on Windows 10.
I have a program to find a string in a 12MB file .dat file which was exported from Excel to be a tab-delimited file. However when the file is read I get this error: `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in byte position 7997: invalid continuation byte`
When I open the file in my text editor (Notepad++) and go to position 7997 I don’t see any special characters when I turn on “Show special characters”. The cursor is between 2 normal letters: H and S.
Here’s my function which reads the text file and searches through it.
```python
def search_file_lookahead(filename, search_strings):
r"""Searches a text file for occurrences of
multiple strings in any order using lookahead assertions.
"""
with open(filename, 'r', encoding='utf-8') as file:
lines = file.readlines() # ERROR is here then "lines" has no value
pattern = f"(?=.*{')(?=.*'.join(search_strings)})" # Construct a single pattern with lookaheads
matches = []
for line_number, line in enumerate(lines):
if re.search(pattern, line):
matches.append((line_number + 1, line.strip()))
if matches:
print(f"Found matches in {filename}:")
for line_number, line in matches:
print(f"- Line {line_number}: {line}")
else:
print(f"No matches found in {filename} for the given strings.")
```
I trap the error and this error causes the program to quite even though I put a “pass” in the exception clause. This also causes the variable `lines` to be undefined.
How do I ignore UTF-8 errors like this and read the text file to process it and use regex on it to search for strings?
Thank you. I’m stumped after searching for some web pages about this.
---
### Author:  [@MegaIng](https://discuss.python.org/u/MegaIng)
#### Post date: [April 2, 2024, 1:45pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/2 "2024-04-02T13:45:18Z")
Notepad++ tells you what encoding your file is in. Use that instead of trying to ignore utf-8 errors.
---
### Author:  [@c-rob](https://discuss.python.org/u/c-rob)
#### Post date: [April 2, 2024, 1:49pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/3 "2024-04-02T13:49:52Z")
I did. Notepad++ reports the file is UTF-8. I still get the error above.
I updated the function to show the error if someone wants to run it.
FIXED: I added “errors=‘ignore’” to open statement.
```python
with open(filename, 'r', encoding='utf-8', errors='ignore') as file:
```
---
### Author:  [@barry-scott](https://discuss.python.org/u/barry-scott)
#### Post date: [April 2, 2024, 3:20pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/4 "2024-04-02T15:20:11Z")
> [@c-rob](#):
>
> The file is UTF-8. I still get the error above.
Clearly it is not utf-8, the error proves that.
---
### Author:  [@onePythonUser](https://discuss.python.org/u/onePythonUser)
#### Post date: [April 2, 2024, 3:45pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/5 "2024-04-02T15:45:03Z")
Hi,
per your original post, you stated that you are getting a `UnicodeDecodeError: utf-8`. From the following link, could it be that you have it encoded in another unicode version. From the following post:
[https://sites.pitt.edu/~naraehan/python3/reading\_writing\_methods.html](https://sites.pitt.edu/~naraehan/python3/reading_writing_methods.html)
`"UnicodeDecodeError" means you have a file encoding issue. Each computer has its own system-wide default encoding, and the file you are trying to open is encoded in something different, most likely some version of Unicode. If this happens, you should specify the encoding using the encoding='xxx' switch while opening the file. If you are not sure which encoding to use, try 'utf-8', 'utf-16', and 'utf-32'`
---
### Author:  [@jeff5](https://discuss.python.org/u/jeff5)
#### Post date: [April 2, 2024, 5:06pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/6 "2024-04-02T17:06:19Z")
> [@c-rob](#):
>
> FIXED: I added “errors=‘ignore’” to open statement.
Errors are there to help you. @barry-scott is correct: the file is not encoded with UTF-8. Windows is not very good at that, by default at least: too much history of byte-encoded workarounds.
So if you care about the data, I recommend you find out what the encoding actually is. `\xe9` is “é” in code page 1252, which is quite a common choice for file encoding on Windows and crops up a bit in otherwise English text.
---
### Author:  [@MRAB](https://discuss.python.org/u/MRAB)
#### Post date: [April 2, 2024, 5:52pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/7 "2024-04-02T17:52:34Z")
Read the file as, say, Latin-1, and look for that character and the surroundng text.
For example:
```python
import re
with open(filename, 'r', encoding='latin-1') as file:
text = file.read()
print(re.findall(r'\w*[\x80-\xFF]\w*', text))
```
Does it look OK? Is it in a word that contains `é`?
---
### Author:  [@barry-scott](https://discuss.python.org/u/barry-scott)
#### Post date: [April 2, 2024, 9:21pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/8 "2024-04-02T21:21:45Z")
It is typical of a Windows program to save as the local 8-bit code page, cp1252 being common.
Using `encoding="cp1252"` is often needed when reading Windows files.
The other char that breaks utf-8 assumption is 0xa0 its the right-hand smart quote.
Its so common that a web search for “0xa0” (and only that) takes you to sites telling you how to fix your code!
---
### Author:  [@kknechtel](https://discuss.python.org/u/kknechtel)
#### Post date: [April 2, 2024, 11:28pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/9 "2024-04-02T23:28:16Z")
> [@c-rob](#):
>
> a 12MB file .dat file which was exported from Excel to be a tab-delimited file.
**Exactly** what does this mean? Walk me through the process of creating this file. The extension `.dat` is highly suspicious here; this is _almost never used for a file that should be interpreted as containing text_. Is all the data created by typing things in Excel, or is it ultimately sourced from somewhere else? When the data is “exported”, what options are chosen for export?
> [@c-rob](#):
>
> When I open the file in my text editor (Notepad++) and go to position 7997 I don’t see any special characters when I turn on “Show special characters”. The cursor is between 2 normal letters: H and S.
The position given in the error message is a byte position, while Notepad++ will presumably show you a character position (and might also translate newlines).
Let’s see if we can diagnose the file’s encoding (and verify that it is, in fact, supposed to be interpreted as text) by looking at some raw byte values around the (first) problematic position. Please show the result you get from:
```python
import binascii
with open(filename, 'rb') as file:
file.seek(7900)
for i in range(16):
data = file.read(16)
print(*map('{:02x}'.format, data), sep=' ')
```
---
### Author:  [@petersuter](https://discuss.python.org/u/petersuter)
#### Post date: [April 3, 2024, 6:21am UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/10 "2024-04-03T06:21:29Z")
To investigate file encoding problems it can also be useful to view the raw file in a hex editor. For example just open [https://hexed.it/](https://hexed.it/) in a browser and drag the file into it to open it locally. Enter 7997 on the right in the “Go to” field.
---
### Author:  [@c-rob](https://discuss.python.org/u/c-rob)
#### Post date: [April 3, 2024, 9:20am UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/11 "2024-04-03T09:20:15Z")
> [@kknechtel](#):
>
> The position given in the error message is a byte position, while Notepad++ will presumably show you a character position (and might also translate newlines).
A tab-delimited file uses tabs to separate columns in each line, instead of commas in a CSV file.
I use a macro to save the current spreadsheet tab as a tab-delimited file. But a tab-delimited file from Excel is made like this:
1. Open the Excel file to export.
2. Go to File, click More, then Export.
3. Click Change File Type. Choose tab-delimited .txt file.
4. Click Save Icon.
> The position given in the error message is a byte position, while Notepad++ will presumably show you a character position (and might also translate newlines).
No, it’s a programmer’s text editor so it can take me to a line position, or byte position. I thought it had a hex view mode but I couldn’t find it. Maybe it has a plugin to show stuff in hex mode.
---
### Author:  [@c-rob](https://discuss.python.org/u/c-rob)
#### Post date: [April 3, 2024, 9:24am UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/12 "2024-04-03T09:24:52Z")
> [@petersuter](#):
>
> For example just open [https://hexed.it/](https://hexed.it/) in a browser and drag the file into it to open it locally.
That’s a great tool I will save, but I cannot use it for this file as the file has customer data. Can you recommend a Windows Desktop app for a hex editor? I haven’t used one in 15 years. I will need to find and probably edit certain hex values in the file then save the 12MB file.
---
### Author:  [@kknechtel](https://discuss.python.org/u/kknechtel)
#### Post date: [April 3, 2024, 9:58am UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/13 "2024-04-03T09:58:04Z")
> [@c-rob](#):
>
> But a tab-delimited file from Excel is made like this:
And there’s no option in this process anywhere to select a text encoding?
How did data get input to begin with? Did the customer just give you a `.xlsx` file to start with or something?
> [@c-rob](#):
>
> Can you recommend a Windows Desktop app for a hex editor?
Reading the file in binary mode and translating the byte values into hex digits works. It’s certainly not user-friendly, but it shows you the important part for debugging. That was the point of the code I suggested to you.
---
### Author:  [@petersuter](https://discuss.python.org/u/petersuter)
#### Post date: [April 3, 2024, 11:08am UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/14 "2024-04-03T11:08:13Z")
> [@c-rob](#):
>
> - Click Change File Type. Choose tab-delimited .txt file.
This is not UTF-8 encoded here, but uses some locale, probably `cp1252`.
You can instead click “Save as Another File Type” there and then “Unicode Text”, that’s then tab-delimited UTF-16 I think.
> [@c-rob](#):
>
> That’s a great tool I will save, but I cannot use it
I _think_ it doesn’t upload the data anywhere, it’s a local app running in your browser only, but I understand.
> [@c-rob](#):
>
> Can you recommend a Windows Desktop app for a hex editor?
“010 Editor” is excellent, but this question is probably offtopic on this forum.
@kknechtel’s advice above to use Python is also excellent of course.
---
### Author:  [@MRAB](https://discuss.python.org/u/MRAB)
#### Post date: [April 3, 2024, 3:42pm UTC](https://discuss.python.org/t/how-to-fix-utf-8-error-when-reading-text-file/50117/15 "2024-04-03T15:42:39Z")
I use EditPad Pro (paid), but there’s also EditPad Lite (free).