PEP 686: Make UTF-8 mode default (Round 2)

I updated the PEP 686 based on discussions in the previous topic.

Major changes from the previous version:

  • Target Python version changed from 3.12 to 3.13.
  • Add locale.getencoding() instead of changing locale.getpreferredencoding(False) to ignore UTF-8 mode.
    • locale.getpreferredencoding() will emit EncodingWarning because UTF-8 mode affects it (opt-in).
  • Add section about encoding="locale" behavior.
  • Add “Use PYTHONSTDIOENCODING for PIPEs” section to rejected ideas.

Abstract

This PEP proposes enabling UTF-8 mode by default.

With this change, Python consistently uses UTF-8 for default encoding of files, stdio, and pipes.

Motivation

UTF-8 becomes de facto standard text encoding.

  • The default encoding of Python source files is UTF-8.
  • JSON, TOML, YAML use UTF-8.
  • Most text editors, including Visual Studio Code and Windows Notepad use UTF-8 by default.
  • Most websites and text data on the internet use UTF-8.
  • And many other popular programming languages, including Node.js, Go, Rust, and Java uses UTF-8 by default.

Changing the default encoding to UTF-8 makes it easier for Python to interoperate with them.

Additionally, many Python developers using Unix forget that the default encoding is platform dependent. They omit to specify encoding="utf-8" when they read text files encoded in UTF-8 (e.g. JSON, TOML, Markdown, and Python source files). Inconsistent default encoding causes many bugs.

Specification

Enable UTF-8 mode by default

Python will enable UTF-8 mode by default from Python 3.13.

Users can still disable UTF-8 mode by setting PYTHONUTF8=0 or -X utf8=0 .

locale.getencoding()

Since UTF-8 mode affects locale.getpreferredencoding(False) , we need an API to get locale encoding regardless of UTF-8 mode.

locale.getencoding() will be added for this purpose. It returns locale encoding too, but ignores UTF-8 mode.

When warn_default_encoding option is specified, locale.getpreferredencoding() will emit EncodingWarning like open() (see also PEP 597).

This API will be added in Python 3.11.

Fixing encoding="locale" option

PEP 597 added the encoding="locale" option to the TextIOWrapper . This option is used to specify the locale encoding explicitly. TextIOWrapper should use locale encoding when the option is specified, regardless of default text encoding.

But TextIOWrapper uses "UTF-8" in UTF-8 mode even if encoding="locale" is specified for now. This behavior is inconsistent with the PEP 597 motivation. It is because we didn’t expect making UTF-8 mode default when Python changes its default text encoding.

This inconsistency should be fixed before making UTF-8 mode default. TextIOWrapper should use locale encoding when encoding="locale" is passed even in UTF-8 mode.

This issue will be fixed in Python 3.11.

Backward Compatibility

Most Unix systems use UTF-8 locale and Python enables UTF-8 mode when its locale is C or POSIX. So this change mostly affects Windows users.

When a Python program depends on the default encoding, this change may cause UnicodeError , mojibake, or even silent data corruption. So this change should be announced loudly.

This is the guideline to fix this backward compatibility issue:

  1. Disable UTF-8 mode.
  2. Use EncodingWarning (PEP 597) to find every places UTF-8 mode affects.
    • If encoding option is omitted, consider using encoding="utf-8" or encoding="locale" .
    • If locale.getpreferredencoding() is used, consider using "utf-8" or locale.getencoding() .
  3. Test the application with UTF-8 mode.

Preceding examples

  • Ruby changed the default external_encoding to UTF-8 on Windows in Ruby 3.0 (2020).
  • Java changed the default text encoding to UTF-8 in JDK 18. (2022).

Both Ruby and Java have an option for backward compatibility. They don’t provide any warning like PEP 597’s EncodingWarning in Python for use of the default encoding.

Rejected Alternative

Deprecate implicit encoding

Deprecating the use of the default encoding is considered.

But there are many cases that the default encoding is used for reading/writing only ASCII text. Additionally, such warnings are not useful for non-cross platform applications run on Unix.

So forcing users to specify the encoding everywhere is too painful. Emitting a lot of DeprecationWarning will lead users ignore warnings.

PEP 387 requires adding a warning for backward incompatible changes. But it doesn’t require using DeprecationWarning . So using optional EncodingWarning doesn’t violate the PEP 387.

Java also rejected this idea in JEP 400.

Use PYTHONIOENCODING for PIPEs

To ease backward compatibility issue, using PYTHONIOENCODING as the default encoding of PIPEs in the subprocess module is considered.

With this idea, users can use legacy encoding for subprocess.Popen(text=True) even in UTF-8 mode.

But this idea makes “default encoding” complicated. And this idea is also backward incompatible.

So this idea is rejected. Users can disable UTF-8 mode until they replace text=True with encoding="utf-8" or encoding="locale" .

How to teach this

For new users, this change reduces things that need to teach. Users don’t need to learn about text encoding in their first year. They should learn it when they need to use non-UTF-8 text files.

For existing users, see the Backward compatibility section.

7 Likes

We should add the new function immediately (3.11) and mention that here. That will give library authors more time to adapt.

I think we can also fix encoding="locale" immediately to use getencoding() and avoid UTF-8 mode.

Other than that, I think I’m happy enough with this to start promoting it around “regular” users and see what new concerns are raised.

1 Like

OK, I will add schedule to each spec.

Thanks!

FWIW, I found “Inside Java” article written by the JEP400 author.

If the Python UTF-8 mode has issues, I agree with @steve.dower, it should be fixed right now. I’m not sure that these bugfixes need a PEP.

I’m fine with adding a new locale.getencoding() function which ignores the UTF-8 Mode.

But changing locale.getpreferredencoding(False) to ignore the UTF-8 Mode can cause mojibake on code expecting the Python 3.10 behavior or who didn’t know the UTF-8 Mode when writing their application. PEP 540 changes locale.getpreferredencoding(False) so existing applications (calling this function) switch automatically to UTF-8 if the UTF-8 Mode is enabled.

It’s unfortunate that the UTF-8 Mode… which ignores the locale on purpose… affects a function of the locale module :frowning:

This problem reminds me the old time.clock() function which had a different behavior (monotonic or not) depending on the platform. After long discussions, the function was deprecated and then removed, to be replaced with time.perf_counter() and time.monotonic() which behave the same on all platforms.

I propose a similar migration plan:

  • Deprecate locale.getpreferredencoding() and locale.getpreferredencoding(False)
  • Add locale.getencoding() which ignores UTF-8 Mode
  • Add sys.getencoding() which is similar to locale.getpreferredencoding(False)

locale.getencoding() and sys.getencoding() should read the locale encoding at Python startup, and then always return the same encoding. Otherwise, we create another source of mojibake. “Changing the encoding” at runtime is always a bad idea. IMO using the same encoding everywhere (command line, filenames, environment variables, subprocess pipes, stdio) reduces greatly the risk of mojibake.

Deprecating locale.getpreferredencoding() can happen later, or maybe remain part of PEP 686.


The very few special functions which require the current LC_CTYPE locale encoding can call the existing locale.nl_langinfo(locale.CODESET) function. Example:

$ ./python -q
>>> import locale
>>> locale.nl_langinfo(locale.CODESET)
'UTF-8'
>>> locale.setlocale(locale.LC_CTYPE, "fr_FR")
'fr_FR'
>>> locale.nl_langinfo(locale.CODESET)
'ISO-8859-1'

Sadly, this function is not portable on Windows where locale.getencoding() should be used instead to the ANSI code page. But I expect that functions which care about the current LC_CTYPE encoding are really specific to Unix (ex: ncurses).


If these APIs are fixed in Python 3.11, PEP 686 can be greatly simplified to just enable the UTF-8 Mode by default in your favorite Python version :wink:

3 Likes

Strictly speaking, this is not an issue until this PEP is accepted.
If SC reject this PEP and we decide to change the default TextIOWrapper encoding without UTF-8 mode, current behavior has no problem.
That’s why I include this issue in the PEP.

On the other hand, I think we can change the encoding="locale" behavior in Python 3.11 without waiting this PEP accepted.
It is very small behavior change so almost zero users will be affected.

But we need locale.getencoding() to fix this issue in _pyio anyway.

For clarify, previous version of this PEP had proposed to change the locale.getpreferredencoding(False) to ignore UTF-8 mode.
But current version (written in this topic) doesn’t propose it:

  • Add locale.getencoding() which is same to locale.getpreferredencoding(False) except it ignores UTF-8 mode.
  • locale.getpreferredencoding() emits EncodingWarning if warn_default_encoding is specified, so that user can know where UTF-8 mode affects.

I want to make encoding="locale" behaves same to current encoding=None as possible to ease migration.
Current encoding=None and locale.getpreferredencoding(False) returns current locale encoding. So I proposed locale.getencoding() returns current locale too.

Many users might need to keep current behavior until major version up of their applications.
I really want to provide API that is same to locale.getpreferredencoding(False) but ignores UTF-8 mode.

On the other hand, I don’t think we need sys.getencoding().
We already have sys.getdefaultencoding() and sys.getfilesystemencoding(). sys.getencoding() will be too confusiong.
If we need “locale encoding at Python startup”, it can be sys.getlocaleencoding() and it will ignore UTF-8 mode.

I don’t see why we’d need it, and certainly not anywhere so obvious as the sys module.

How do you get the encoding at startup in other languages? Presumably by reading the encoding before you change it, and the same can be done in Python. If someone is injecting an encoding change even earlier than that, it’s probably meant to override whatever you’re supposed to think the original encoding was… this game goes on for a long time :slight_smile:

Being able to read the current encoding from locale is good enough until proven otherwise, and I’d be willing to bet that whoever thinks they can prove otherwise is still probably wrong.

1 Like

For the record, R 4.2.0 was released at April 2022 also adopt UTF-8 on Windows.

It seems they uses UTF-8 code page by manifest.

As far as I know, there are neither backward compat option like PYTHONUTF8=0 nor warning like EncodingWarning.
They just bit the bullet.

3 Likes

Hi,
As Thomas said in the other thread:

we would like the change to the default to be for Python 3.15 instead of 3.13. Unless you have strong objections to the longer timeline, please update the PEP accordingly, and consider it accepted.

The PEP was updated, so please consider it accepted. You can mark it as such at your convenience.

– Petr, on behalf of the SC

6 Likes