Major changes from the previous version:
- Target Python version changed from 3.12 to 3.13.
locale.getencoding()instead of changing
locale.getpreferredencoding(False)to ignore UTF-8 mode.
EncodingWarningbecause UTF-8 mode affects it (opt-in).
- Add section about
- Add “Use
PYTHONSTDIOENCODINGfor PIPEs” section to rejected ideas.
This PEP proposes enabling UTF-8 mode by default.
With this change, Python consistently uses UTF-8 for default encoding of files, stdio, and pipes.
UTF-8 becomes de facto standard text encoding.
- The default encoding of Python source files is UTF-8.
- JSON, TOML, YAML use UTF-8.
- Most text editors, including Visual Studio Code and Windows Notepad use UTF-8 by default.
- Most websites and text data on the internet use UTF-8.
- And many other popular programming languages, including Node.js, Go, Rust, and Java uses UTF-8 by default.
Changing the default encoding to UTF-8 makes it easier for Python to interoperate with them.
Additionally, many Python developers using Unix forget that the default encoding is platform dependent. They omit to specify
encoding="utf-8" when they read text files encoded in UTF-8 (e.g. JSON, TOML, Markdown, and Python source files). Inconsistent default encoding causes many bugs.
Python will enable UTF-8 mode by default from Python 3.13.
Users can still disable UTF-8 mode by setting
-X utf8=0 .
Since UTF-8 mode affects
locale.getpreferredencoding(False) , we need an API to get locale encoding regardless of UTF-8 mode.
locale.getencoding() will be added for this purpose. It returns locale encoding too, but ignores UTF-8 mode.
warn_default_encoding option is specified,
locale.getpreferredencoding() will emit
open() (see also PEP 597).
This API will be added in Python 3.11.
PEP 597 added the
encoding="locale" option to the
TextIOWrapper . This option is used to specify the locale encoding explicitly.
TextIOWrapper should use locale encoding when the option is specified, regardless of default text encoding.
"UTF-8" in UTF-8 mode even if
encoding="locale" is specified for now. This behavior is inconsistent with the PEP 597 motivation. It is because we didn’t expect making UTF-8 mode default when Python changes its default text encoding.
This inconsistency should be fixed before making UTF-8 mode default.
TextIOWrapper should use locale encoding when
encoding="locale" is passed even in UTF-8 mode.
This issue will be fixed in Python 3.11.
Most Unix systems use UTF-8 locale and Python enables UTF-8 mode when its locale is C or POSIX. So this change mostly affects Windows users.
When a Python program depends on the default encoding, this change may cause
UnicodeError , mojibake, or even silent data corruption. So this change should be announced loudly.
This is the guideline to fix this backward compatibility issue:
- Disable UTF-8 mode.
EncodingWarning(PEP 597) to find every places UTF-8 mode affects.
encodingoption is omitted, consider using
locale.getpreferredencoding()is used, consider using
- Test the application with UTF-8 mode.
- Ruby changed the default
external_encodingto UTF-8 on Windows in Ruby 3.0 (2020).
- Java changed the default text encoding to UTF-8 in JDK 18. (2022).
Both Ruby and Java have an option for backward compatibility. They don’t provide any warning like PEP 597’s
EncodingWarning in Python for use of the default encoding.
Deprecating the use of the default encoding is considered.
But there are many cases that the default encoding is used for reading/writing only ASCII text. Additionally, such warnings are not useful for non-cross platform applications run on Unix.
So forcing users to specify the
encoding everywhere is too painful. Emitting a lot of
DeprecationWarning will lead users ignore warnings.
Java also rejected this idea in JEP 400.
To ease backward compatibility issue, using
PYTHONIOENCODING as the default encoding of PIPEs in the
subprocess module is considered.
With this idea, users can use legacy encoding for
subprocess.Popen(text=True) even in UTF-8 mode.
But this idea makes “default encoding” complicated. And this idea is also backward incompatible.
So this idea is rejected. Users can disable UTF-8 mode until they replace
For new users, this change reduces things that need to teach. Users don’t need to learn about text encoding in their first year. They should learn it when they need to use non-UTF-8 text files.
For existing users, see the Backward compatibility section.