Stdlib module hides local module (not the other way around)

It took me embarrassingly long to get a rough idea of what was happening here. So I thought I’d post it to see if I missed something obvious and if there could improvements be made.

Usually, at some point when starting to learn Python, something like this happens:

$ python3 -VV
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110]
$  ls -a  # empty folder
.  ..
$ cat <<EOF >gettext.py
> print("I am a local gettext.py")
> EOF
$ python3 -c 'import argparse'
I am a local gettext.py
$

(Depending on your Python version, you might also get an ImportError immediately. I am on Debian 11 “bullseye” with Python 3.9.)

Explanation: The user has an (unrelated) gettext.py in their folder, without knowing that there’s an stdlib module of that name. The user wants to use argparse which imports gettext which leads to errors, because the local gettext.py hides stdlib’s gettext.

I’m aware of this and wanted to deliberately take advantage of the behavior to edit a local copy of a stdlib module to quickly try out some changes. However, I noticed the exact opposite behavior:

$ rm gettext.py  # clean-up
$ python3 -m venv venv  # create venv for later
$ cat <<EOF >reprlib.py
> print("I am a local reprlib.py")
> EOF
$ python3 -c 'import reprlib'
$ # notice no output from line above
$ python3 -c 'import reprlib; print(reprlib.__file__)'
/usr/lib/python3.9/reprlib.py
$

To my surprise, the local reprlib.py has been ignored.

Also, interestingly, the virtual environment behaves differently, despite being the same version:

$ ./venv/bin/python3 -VV
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110]
$ ./venv/bin/python3 -c 'import reprlib'
I am a local reprlib.py
$

With a current build, the issue disappears for reprlib when executed directly:

$ # build from source, steps skipped
$ ./python -VV
Python 3.12.0a0 (heads/main:ffcc7cd57f, May 11 2022, 16:39:40) [GCC 10.2.1 20210110]
$ ./python -c 'import reprlib'
I am a local reprlib.py
$

But it is still there for the interactive shell:

$ ./python
Python 3.12.0a0 (heads/main:ffcc7cd57f, May 11 2022, 16:39:40) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import reprlib
>>> 

Furthermore, for other stdlib modules like io, the issue persists for both command line and interactive shell:

$ cat <<EOF >io.py
> print("I am a local io.py")
> EOF
$ ./python -c 'import io'
$ ./python
Python 3.12.0a0 (heads/main:ffcc7cd57f, May 11 2022, 16:39:40) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import io
>>> 

I assume all of this has something to do with some modules being imported - every time, by default - when starting the interpreter. Which specific modules this concerns or at which time the import happens, seems to depend on Python version, virtual environment and interactive shell.

The inconsistent behavior can probably be explained by looking at the internals, but this made it really confusing for me to find out what’s even going on. While I’m aware that it’s generally good practice to just avoid re-using names of stdlib modules, I thought that my use-case warrants to do so. In any case, it’s unfortunate that the whole “wrong import” situation results in a, more or less, silent error.

Is all of this missing from the docs or at least not easy to find or - most likely - am I just really bad at searching for it? Both for my issue as well as the more frequent “hiding an stdlib module with a local module”? Is there a conclusive list of names to avoid? If my assumption from above is correct: which modules are automatically being imported at start-up? I guess python -c 'import sys; print(sorted(sys.modules))' might be misleading because I manually import sys?

I suggest that you check your “site-packages” (AKA “dist-packages” in Debian-based Linux distros) directory (or directories) for “.pth” files. For example, in my current Ubuntu setup I have “/usr/lib/python3/dist-packages/repoze.lru-0.7-nspkg.pth”. The “repoze.lru” package is a dependency of the “routes” package. Executing “repoze.lru-0.7-nspkg.pth” imports the standard reprlib module early on before it can be shadowed by a local “reprlib.py” in the current directory (-c command, -m module) or script directory.

By default virtual environments do not include site packages directories, and thus their “.pth” files won’t be executed.

$ find /usr/lib/python* -iname "*.pth" | wc -l
0

Apparently there’s no .pth file. If I understood you correctly, this is not what you expected to see?

Also, here is a reproducible example with the official Docker image:

$ sudo docker run -it --rm -w /tmp python:3.10 /bin/bash
root@7cb29a128739:/tmp# cat <<EOF >io.py
> print("I am a local io.py")
> EOF
root@7cb29a128739:/tmp# python -c 'import io'
root@7cb29a128739:/tmp# python
Python 3.10.0 (default, Nov 17 2021, 15:26:39) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import io
>>> 

Compare the output of the command python3.9 -c "import sys; print(*sys.path, sep='\n')" to that of the command python -Sc "import sys; print(*sys.path, sep='\n')" (no site module) to discover which site packages directories get added by your distro, including the per-user site packages.

The io module is a core module in the standard library that gets used early on no matter what’s installed. AFAIK, you can’t rely on being able to replace it with an “io.py” file in the script directory or the initial working directory (if there’s no script). In 3.11+, the io module is also pre-compiled and frozen in the interpreter, so even editing the “io.py” source file has no effect when 3.11 is installed, unless the interpreter is executed with the command-line option -X frozen_modules=off. (The latter is the default for a local build. You have to actually install 3.11+ for frozen modules to be enabled by default.)