Is this a bug about python's stat module?

Python Help
1

Hi all,
I find a problem when using python on Mac (not happen on CentOS), here is how to reproduce it on Mac:

brew install python@3.8
python3.8 -m venv debug_python_stat
source debug_python_stat/bin/activate
touch stat.py
python3.8

Then the error occurs:

Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.8/3.8.17_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site.py", line 580, in <module>
    main()
  File "/opt/homebrew/Cellar/python@3.8/3.8.17_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site.py", line 566, in main
    known_paths = addusersitepackages(known_paths)
  File "/opt/homebrew/Cellar/python@3.8/3.8.17_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site.py", line 315, in addusersitepackages
    if ENABLE_USER_SITE and os.path.isdir(user_site):
  File "/opt/homebrew/Cellar/python@3.8/3.8.17_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/genericpath.py", line 45, in isdir
    return stat.S_ISDIR(st.st_mode)
AttributeError: module 'stat' has no attribute 'S_ISDIR'

I guess it’ related to python stdlib’s stat module, but I didn’t figure out why.

Any help is appreciated.

2

It is a common issue that will not be considered a bug. It is the unfortunate consequence of a few different design decisions.

This is one specific instance of the general problem:

In your case, when the interpreter starts up it will try to import the stat standard library module, indirectly, as a part of its setup work. However, because

  • standard library modules use absolute imports to find each other
    • which they must because they are not in a common package
      • partly for convenience and I’m not sure the concept of packages was present from the beginning
  • absolute import will search sys.path in order in most cases
  • your current working folder is automatically on sys.path
    • which is for convenience so that you can split your code across files in the same folder without understanding packages
    • and possibly also something to do with providing backwards compatibility when the import system rules were reformed between 2.4 and 2.5
  • that folder contains a stat.py,

that attempt to import stat will import your, blank, module instead. When the startup code later attempts to use functionality from stat, it isn’t present.

Aside from using a different name for your module, you can:

  • make sure that the current working directory isn’t on sys.path
    • in 3.11 onward, by using the -P command-line option to Python, or setting the PYTHONSAFEPATH environment variable
    • in 3.4 onward, by running in “isolated mode” with -I (but this will cause PYTHONPATH and PYTHONHOME environment variables to be ignored, and may prevent access to some third-party libraries)
    • not by manipulating sys.path directly, because you won’t get a chance before the problematic code runs
    • No matter how you do this, it will most likely interfere with any intentional absolute imports you might have from one of your source files to another. Of course, you can temporarily fix sys.path to work around that.
  • prevent startup code (the site standard library module) from running with -S (this will definitely prevent access to third-party libraries, and may also lose a lot of other nice functionality such as tab-completion in the REPL)

In 3.10 and up, to know what names could cause a problem, check sys.stdlib_module_names; otherwise, check the standard library documentation.

1 Like
3

You have a file shadowing the stat module.

Inside the code, it does import stat and now finds your module instead of the one from stdlib.

You can do similar by making other files with the same name as stdlib modules.

1 Like
4

What’s interesting to me is that this doesn’t always cause a problem, depending (I think) on how python was installed/configured. As the OP says, it didn’t happen for them on CentOS, and it isn’t a problem for me on macOS with a conda env. Seems odd.

5

It depends on the contents of the site module, which may vary by Python version and by the requirements of the platform; and it could depend on environment variables that affect sys.path, i.e. PYTHONPATH. I’m not at all surprised that this specific reproduction of the issue wouldn’t be consistent.

For example, on my machine, adding a blank token.py breaks the interactive help for the 3.8 install that came with my OS, but not for the 3.11 that I built myself. stat.py doesn’t cause a problem for me with either.

After digging around a bit in the source code: It appears that Linux-based systems provide a built-in stat that comes from a Linux system call and is exposed as os.stat. The generic implementation of os.path.isdir tries to use os.stat when it’s made available that way; otherwise it falls back on figuring it out using the stat module, as in OP’s exception traceback.

1 Like