Sys.flags.utf8_mode is -1

Python Help
1

Any idea why sys.flags.utf8_mode is -1 when using PyInstaller?
(Setting the PYTHONUTF8 environment variable does not change this, and -X flags are not supported by PyInstaller.)

Thanks.

2

It is a bug. Is it fixed by Python 3.7.1? If not, please report the bug.

2 Likes
3

Upgrading to Python 3.7.1 indeed solved the problem. Thank you! :confetti_ball:

2 Likes
4

Out of curiosity, how do you enable the UTF-8 mode in an executable generated by PyInstaller ?

5

I don’t know PyInstaller. At the C level, you can use “Py_UTF8Mode = 1;”. In Python 3.7.1, setting PYTHONUTF8=1 environment variable should work as well.

2 Likes
6

Unfortunately the environment variable still seems to be ignored with PyInstaller:

C:\>mkdir test-pyinstaller && cd  test-pyinstaller 

C:\test-pyinstaller>py -3.7 --version
Python 3.7.1

C:\test-pyinstaller>py -3.7 -m pip install pyinstaller
Collecting pyinstaller
...
Successfully installed pyinstaller-3.4 ...

C:\test-pyinstaller>echo import sys; print("UTF8 mode is ", sys.flags.utf8_mode) >main.py

C:\test-pyinstaller>py -3.7 main.py
UTF8 mode is  0

C:\test-pyinstaller>py -3.7 -m PyInstaller main.py
45 INFO: PyInstaller: 3.4
46 INFO: Python: 3.7.1
... completed successfully.

C:\test-pyinstaller>dist\main\main.exe
UTF8 mode is  0

C:\test-pyinstaller>set PYTHONUTF8=1

C:\test-pyinstaller>py -3.7 main.py
UTF8 mode is  1

C:\test-pyinstaller>dist\main\main.exe
UTF8 mode is  0
7

Again, I don’t know PyInstaller, I don’t know how it uses the C API. But I can now say that I wrote unit tests on the C API to initialize Python to check that the configuration is read properly! Previously, there was basically zero test for that. And there is a test to ensure that environment variable are read when you use Py_Initialize(). And I see a test for PYTHONUTF8=1.

So. I checked PyInstaller: it explicitly disable PYTHON environment variables. src/pyi_pythonlib.c:

 /* This flag ensures PYTHONPATH and PYTHONHOME are ignored by Python. */
 *PI_Py_IgnoreEnvironmentFlag = 1;

I suggest you to open an issue in PyInstaller, there is nothing Python can do for you :slight_smile:

2 Likes
8

Thanks for checking. I don’t know much about PyInstaller internals myself. I opened https://github.com/pyinstaller/pyinstaller/issues/3866

At the moment I don’t need to enable UTF8 mode, but I expect it will be very useful later. :relieved:

1 Like
9

Windows uses UTF-8 by default since Python 3.6 for many functions. The UTF-8 Mode mostly impacts open() if you don’t specify the encoding.

2 Likes
10

(I encountered the impact with p = subprocess.Popen(["example.exe"], stdout=subprocess.PIPE, ...) without specifying the encoding. It seems the example.exe process (not under my control) uses the default system locale to encode stdout, and Python (without UTF-8 mode) decodes it successfully. With UTF-8 mode Python attempts to use UTF-8 instead and fails to decode the stdout of the process.

If I understood correctly it is affected because p.stdout.encoding defaults to locale.getpreferredencoding(False) via io.TextIOWrapper.)