Difference in behavior of Python versions when no virtual environment is found

ivanov17 · March 14, 2023, 4:10pm

Hello!

I would like to know if there any behavior changes since version 3.6 that are relevant to my issue.

I am using uWSGI containers to run Django applications. I recently decided to upgrade my containers to use Python 3.9, but ran into an issue that doesn’t seem to be uWSGI version dependent.

With Python 3.6 if uWSGI master process doesn’t find virtual environment directory, this causes a segmentation fault, the process exits with code 1 and can then restart with the fallback configuration.

With Python 3.9 uWSGI master process exits without a segmentation fault, so it doesn’t use the fallback configuration.

It seems strange to me that with Python 3.9 I get <no Python frame> message instead of a backtrace. I tried searching the web for more details, but didn’t find anything informative.

Please, explain to me what’s going on and how can I get back the desired behavior.

Thank you.

uWSGI 2.0.18 / Python version: 3.6.8

[uWSGI] getting INI configuration from /etc/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Sat Mar 11 21:33:34 2023] ***
compiled with version: 8.3.1 20190507 (Red Hat 8.3.1-4) on 27 January 2020 10:45:00
os: Linux-6.1.14-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Feb 26 00:13:26 UTC 2023
nodename: 22c3284023fa
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /var/www
writing pidfile to /run/uwsgi/uwsgi.pid
detected binary path: /usr/sbin/uwsgi
your processes number limit is 30960
limiting address space of processes...
your process address space limit is 268435456 bytes (256 MB)
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 524288
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
[busyness] backlog alert is set to 16 request(s), step is 2
[busyness] backlog non-zero alert is set to 60 second(s)
uwsgi socket 0 bound to TCP address 127.0.0.1:9000 fd 3
Python version: 3.6.8 (default, Jan 25 2023, 15:03:30)  [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
!!! Python Home is not a directory: env !!!
Set PythonHome to env
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f1a5cefb600 (most recent call first):
!!! uWSGI process 1 got Segmentation Fault !!!
*** backtrace of 1 ***
/usr/sbin/uwsgi(uwsgi_backtrace+0x2e) [0x46a37e]
/usr/sbin/uwsgi(uwsgi_segfault+0x27) [0x46a767]
/lib64/libc.so.6(+0x4eb50) [0x7f1a5a7e6b50]
/lib64/libc.so.6(abort+0x203) [0x7f1a5a7b9f81]
/usr/lib64/libpython3.6m.so.1.0(Py_FatalError+0x17c) [0x7f1a59c76bb7]
/usr/lib64/libpython3.6m.so.1.0(+0x9d5bc) [0x7f1a59cab5bc]
/usr/lib64/libpython3.6m.so.1.0(_Py_InitializeEx_Private+0x21b) [0x7f1a59e1b0fb]
/usr/lib64/uwsgi/python3_plugin.so(uwsgi_python_init+0x142) [0x7f1a5a3650b2]
/usr/sbin/uwsgi(uwsgi_start+0x5a7) [0x46b9d7]
/usr/sbin/uwsgi(uwsgi_setup+0x1317) [0x46dd27]
/usr/sbin/uwsgi(main+0xf) [0x418f7f]
/lib64/libc.so.6(__libc_start_main+0xe5) [0x7f1a5a7d2d85]
/usr/sbin/uwsgi(_start+0x2e) [0x418fbe]
*** end of backtrace ***
VACUUM: pidfile removed.
Sat Mar 11 21:33:34 2023 - !!! /usr/sbin/uwsgi (pid: 1) exited with status 1 !!!
Sat Mar 11 21:33:34 2023 - !!! Fallback config to /etc/uwsgi.d/safe.ini !!!
[uWSGI] getting INI configuration from /etc/uwsgi.d/safe.ini
*** Starting uWSGI 2.0.18 (64bit) on [Sat Mar 11 21:33:34 2023] ***
compiled with version: 8.3.1 20190507 (Red Hat 8.3.1-4) on 27 January 2020 10:45:00
os: Linux-6.1.14-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Feb 26 00:13:26 UTC 2023
nodename: 22c3284023fa
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /var/www
detected binary path: /usr/sbin/uwsgi
chdir() to /var/www
your processes number limit is 30960
limiting address space of processes...
your process address space limit is 268435456 bytes (256 MB)
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 524288
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address 127.0.0.1:9000 fd 3
Python version: 3.6.8 (default, Jan 25 2023, 15:03:30)  [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
Python main interpreter initialized at 0x1fba2b0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 218760 bytes (213 KB) for 2 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1fba2b0 pid: 1 (default app)
gracefully (RE)spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 2, cores: 1)
spawned uWSGI worker 2 (pid: 3, cores: 1)
*** Stats server enabled on 127.0.0.1:9999 fd: 11 ***

uWSGI 2.0.18 / Python version: 3.9.16

[uWSGI] getting INI configuration from /etc/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Mon Mar 13 09:22:18 2023] ***
compiled with version: 11.2.1 20210728 (Red Hat 11.2.1-2) on 27 January 2021 00:00:00
os: Linux-6.1.14-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Feb 26 00:13:26 UTC 2023
nodename: 8c5999467f48
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /var/www
writing pidfile to /run/uwsgi/uwsgi.pid
detected binary path: /usr/sbin/uwsgi
your processes number limit is 30960
limiting address space of processes...
your process address space limit is 268435456 bytes (256 MB)
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 524288
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
[busyness] backlog alert is set to 16 request(s), step is 2
[busyness] backlog non-zero alert is set to 60 second(s)
uwsgi socket 0 bound to TCP address 127.0.0.1:9000 fd 3
Python version: 3.9.16 (main, Dec  8 2022, 00:00:00)  [GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]
!!! Python Home is not a directory: env !!!
Set PythonHome to env
Python path configuration:
  PYTHONHOME = 'env'
  PYTHONPATH = (not set)
  program name = 'env/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/var/www/env/bin/python'
  sys.base_prefix = 'env'
  sys.base_exec_prefix = 'env'
  sys.platlibdir = 'lib64'
  sys.executable = '/var/www/env/bin/python'
  sys.prefix = 'env'
  sys.exec_prefix = 'env'
  sys.path = [
    'env/lib64/python39.zip',
    'env/lib64/python3.9',
    'env/lib64/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f8817848780 (most recent call first):
<no Python frame>
VACUUM: pidfile removed.

uWSGI 2.0.21 / Python version: 3.9.16

[uWSGI] getting INI configuration from /etc/uwsgi.ini
*** Starting uWSGI 2.0.21 (64bit) on [Sat Mar 11 21:11:36 2023] ***
compiled with version: 11.2.1 20220127 (Red Hat 11.2.1-9) on 28 October 2022 00:00:00
os: Linux-6.1.14-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Feb 26 00:13:26 UTC 2023
nodename: f6e77e6845c9
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /var/www
writing pidfile to /run/uwsgi/uwsgi.pid
detected binary path: /usr/sbin/uwsgi
your processes number limit is 30960
limiting address space of processes...
your process address space limit is 268435456 bytes (256 MB)
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 524288
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
[busyness] backlog alert is set to 16 request(s), step is 2
[busyness] backlog non-zero alert is set to 60 second(s)
uwsgi socket 0 bound to TCP address 127.0.0.1:9000 fd 3
Python version: 3.9.16 (main, Dec  8 2022, 00:00:00)  [GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]
!!! Python Home is not a directory: env !!!
Set PythonHome to env
Python path configuration:
  PYTHONHOME = 'env'
  PYTHONPATH = (not set)
  program name = 'env/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/var/www/env/bin/python'
  sys.base_prefix = 'env'
  sys.base_exec_prefix = 'env'
  sys.platlibdir = 'lib64'
  sys.executable = '/var/www/env/bin/python'
  sys.prefix = 'env'
  sys.exec_prefix = 'env'
  sys.path = [
    'env/lib64/python39.zip',
    'env/lib64/python3.9',
    'env/lib64/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f0703dcc780 (most recent call first):
<no Python frame>
VACUUM: pidfile removed.

barry-scott · March 14, 2023, 9:53pm

It seems that the module encoding has been stripped out of your python installations.
I see error messages about it in all the examples.

My guess is that encodings being missing was not an issue but now it is.

I think you must have the encoding support the encoding for the file system which is UTF-8 by default.

There where changes about file system encoding after 3.6 I recall to make UTF-8 a default.

I guess you have encodings stripped to make a small image size for your container.
In which case you will need to strip all the encoding except utf_8.py and utf_8_sig.py

ivanov17 · March 14, 2023, 11:22pm

No, I didn’t strip the encodings. I use packages from the distribution repositories with no modifications.

ls -1 /usr/lib64/python3.9/encodings | wc -l
123

In all the examples the env directory doesn’t exist, so uWSGI cannot find any Python files in that path at all, after which the master process exits.

But with version 3.6 I see segmentation fault and backtrace message, then the master process successfully restarts with the fallback configuration. With version 3.9 I don’t see segmentation fault, backtrace message is absent, instead I only see <no Python frame> and the master process exits without restarting.

My guess is that something has changed in how virtual environments work. I wrote to uWSGI developers, but they replied that it was probably a behavior change in Python.

barry-scott · March 15, 2023, 4:49am

The next clue is the error about PYTHONHOME, suggest you investigate why that error is reported.

ivanov17 · March 15, 2023, 8:18am

I set home option for uWSGI to the relative env path. My container’s working directory is /var/www, so the path /var/www/env doesn’t exist in all the examples. This causes the error, but with different Python versions I have a different behavior after the error.

ivanov17 · March 16, 2023, 9:42am

I see that in Python 3.8 the initialization process has changed: Python Initialization Configuration — Python 3.11.2 documentation.

I guess that uWSGI does not work correctly with configuration initialization in modern versions of Python.

So I think it’s a uWSGI bug.

If anyone is interested, I created an issue on GitHub:

github.com/unbit/uwsgi

uWSGI doesn't throw an error if no virtual environment is found

opened 04:19PM - 12 Mar 23 UTC

ivanov17

I'm running uWSGI containers with podman and using fallback configuration to cre…ate stateless containers. I used uWSGI 2.0.18 with CentOS 8 Stream containers and my configuration worked well. If the master process does not find virtual environment directory, this causes a segmentation fault, the process exits with code 1 and then restarts with the fallback configuration. But when I try to use uWSGI 2.0.21 with CentOS 9 Stream, daemon process exits without a segmentation fault and then doesn't use the fallback configuration. I've looked at the changelogs for behavior changes but haven't found anything that could be related to my case. Please, tell me if the behavior actually changes. If it doesn't, then it looks like a bug that breaks my configuration. Or maybe I can restore uWSGI behavior in the new version by just adding some options to my config file? ### Command line options `uwsgi --ini /etc/uwsgi.ini --fallback-config /etc/uwsgi.d/safe.ini` (as regular user) ### Configuration files **/etc/uwsgi.ini** ``` [uwsgi] strict = true master = true pidfile = /run/uwsgi/uwsgi.pid plugins-dir = /usr/lib64/uwsgi plugin = python3,cheaper_busyness log-4xx = true log-5xx = true vacuum = true die-on-term = true need-app = true home = env wsgi-file = wsgi.py callable = application cheaper-algo = busyness processes = 10 cheaper = 2 cheaper-initial = 4 cheaper-overload = 1 cheaper-step = 2 cheaper-busyness-multiplier = 30 cheaper-busyness-min = 20 cheaper-busyness-max = 70 cheaper-busyness-backlog-alert = 16 cheaper-busyness-backlog-step = 2 disable-logging = true enable-threads = true single-interpreter = true harakiri = 60 limit-as =256 reload-on-rss = 128 max-requests = 10000 max-worker-lifetime = 3600 worker-reload-mercy = 60 socket = 127.0.0.1:9000 stats = 127.0.0.1:9999 ``` **/etc/uwsgi.d/safe.ini** ``` [uwsgi] strict = true master = true pidfile = /run/uwsgi/uwsgi.pid plugins-dir = /usr/lib64/uwsgi plugin = python3 vacuum = true die-on-term = true chdir = /var/www wsgi-file = wsgi.py callable = application processes = 2 disable-logging = true enable-threads = true single-interpreter = true harakiri = 60 limit-as =256 reload-on-rss = 128 max-requests = 10000 max-worker-lifetime = 3600 worker-reload-mercy = 60 socket = 127.0.0.1:9000 stats = 127.0.0.1:9999 ``` ### Output **podman logs uwsgi-python3.6** ``` [uWSGI] getting INI configuration from /etc/uwsgi.ini *** Starting uWSGI 2.0.18 (64bit) on [Sat Mar 11 21:33:34 2023] *** compiled with version: 8.3.1 20190507 (Red Hat 8.3.1-4) on 27 January 2020 10:45:00 os: Linux-6.1.14-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Feb 26 00:13:26 UTC 2023 nodename: 22c3284023fa machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 2 current working directory: /var/www writing pidfile to /run/uwsgi/uwsgi.pid detected binary path: /usr/sbin/uwsgi your processes number limit is 30960 limiting address space of processes... your process address space limit is 268435456 bytes (256 MB) your memory page size is 4096 bytes *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** detected max file descriptor number: 524288 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) [busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2 [busyness] backlog alert is set to 16 request(s), step is 2 [busyness] backlog non-zero alert is set to 60 second(s) uwsgi socket 0 bound to TCP address 127.0.0.1:9000 fd 3 Python version: 3.6.8 (default, Jan 25 2023, 15:03:30) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] !!! Python Home is not a directory: env !!! Set PythonHome to env Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings' Current thread 0x00007f1a5cefb600 (most recent call first): !!! uWSGI process 1 got Segmentation Fault !!! *** backtrace of 1 *** /usr/sbin/uwsgi(uwsgi_backtrace+0x2e) [0x46a37e] /usr/sbin/uwsgi(uwsgi_segfault+0x27) [0x46a767] /lib64/libc.so.6(+0x4eb50) [0x7f1a5a7e6b50] /lib64/libc.so.6(abort+0x203) [0x7f1a5a7b9f81] /usr/lib64/libpython3.6m.so.1.0(Py_FatalError+0x17c) [0x7f1a59c76bb7] /usr/lib64/libpython3.6m.so.1.0(+0x9d5bc) [0x7f1a59cab5bc] /usr/lib64/libpython3.6m.so.1.0(_Py_InitializeEx_Private+0x21b) [0x7f1a59e1b0fb] /usr/lib64/uwsgi/python3_plugin.so(uwsgi_python_init+0x142) [0x7f1a5a3650b2] /usr/sbin/uwsgi(uwsgi_start+0x5a7) [0x46b9d7] /usr/sbin/uwsgi(uwsgi_setup+0x1317) [0x46dd27] /usr/sbin/uwsgi(main+0xf) [0x418f7f] /lib64/libc.so.6(__libc_start_main+0xe5) [0x7f1a5a7d2d85] /usr/sbin/uwsgi(_start+0x2e) [0x418fbe] *** end of backtrace *** VACUUM: pidfile removed. Sat Mar 11 21:33:34 2023 - !!! /usr/sbin/uwsgi (pid: 1) exited with status 1 !!! Sat Mar 11 21:33:34 2023 - !!! Fallback config to /etc/uwsgi.d/safe.ini !!! [uWSGI] getting INI configuration from /etc/uwsgi.d/safe.ini *** Starting uWSGI 2.0.18 (64bit) on [Sat Mar 11 21:33:34 2023] *** compiled with version: 8.3.1 20190507 (Red Hat 8.3.1-4) on 27 January 2020 10:45:00 os: Linux-6.1.14-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Feb 26 00:13:26 UTC 2023 nodename: 22c3284023fa machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 2 current working directory: /var/www detected binary path: /usr/sbin/uwsgi chdir() to /var/www your processes number limit is 30960 limiting address space of processes... your process address space limit is 268435456 bytes (256 MB) your memory page size is 4096 bytes *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** detected max file descriptor number: 524288 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 inherited INET address 127.0.0.1:9000 fd 3 Python version: 3.6.8 (default, Jan 25 2023, 15:03:30) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] Python main interpreter initialized at 0x1fba2b0 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 218760 bytes (213 KB) for 2 cores *** Operational MODE: preforking *** WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1fba2b0 pid: 1 (default app) gracefully (RE)spawned uWSGI master process (pid: 1) spawned uWSGI worker 1 (pid: 2, cores: 1) spawned uWSGI worker 2 (pid: 3, cores: 1) *** Stats server enabled on 127.0.0.1:9999 fd: 11 *** ``` **podman logs uwsgi-python3.9** ``` [uWSGI] getting INI configuration from /etc/uwsgi.ini *** Starting uWSGI 2.0.21 (64bit) on [Sat Mar 11 21:11:36 2023] *** compiled with version: 11.2.1 20220127 (Red Hat 11.2.1-9) on 28 October 2022 00:00:00 os: Linux-6.1.14-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Feb 26 00:13:26 UTC 2023 nodename: f6e77e6845c9 machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 2 current working directory: /var/www writing pidfile to /run/uwsgi/uwsgi.pid detected binary path: /usr/sbin/uwsgi your processes number limit is 30960 limiting address space of processes... your process address space limit is 268435456 bytes (256 MB) your memory page size is 4096 bytes *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** detected max file descriptor number: 524288 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) [busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2 [busyness] backlog alert is set to 16 request(s), step is 2 [busyness] backlog non-zero alert is set to 60 second(s) uwsgi socket 0 bound to TCP address 127.0.0.1:9000 fd 3 Python version: 3.9.16 (main, Dec 8 2022, 00:00:00) [GCC 11.3.1 20221121 (Red Hat 11.3.1-4)] !!! Python Home is not a directory: env !!! Set PythonHome to env Python path configuration: PYTHONHOME = 'env' PYTHONPATH = (not set) program name = 'env/bin/python' isolated = 0 environment = 1 user site = 1 import site = 1 sys._base_executable = '/var/www/env/bin/python' sys.base_prefix = 'env' sys.base_exec_prefix = 'env' sys.platlibdir = 'lib64' sys.executable = '/var/www/env/bin/python' sys.prefix = 'env' sys.exec_prefix = 'env' sys.path = [ 'env/lib64/python39.zip', 'env/lib64/python3.9', 'env/lib64/python3.9/lib-dynload', ] Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Python runtime state: core initialized ModuleNotFoundError: No module named 'encodings' Current thread 0x00007f0703dcc780 (most recent call first): <no Python frame> VACUUM: pidfile removed. ```

@barry-scott Thanks for the help.

Topic		Replies	Views
Virtual environments vs. *nix Python upgrades Python Help	11	1588	February 22, 2022
Trouble getting pyo3 to use the virtualenv (Windows 10) Python Help help	0	896	September 16, 2021
How to install uWSGI package from PyPI Python Help	4	6394	March 23, 2023
Encountering "Fatal Python error: Segmentation fault" error in python script Python Help help	3	3630	May 18, 2023
Multiple Python versions on Linux in user space Python Help	8	8588	March 29, 2022

Difference in behavior of Python versions when no virtual environment is found

Related Topics