When does a local file shadow a standard library module?

Python Help
1

Today, I was debugging an issue that seemed to boil down to the infamous “a local file in my directory shadows a standard library module”. However, it only happened sometimes, so I was curious and ended up with this. This is the python:3.12 container.

$ mkdir empty
$ cd empty
$ for mod in abc mailbox threading; do python3 -c "import $mod; print($mod.__file__)"; done
/usr/local/lib/python3.12/abc.py
/usr/local/lib/python3.12/mailbox.py
/usr/local/lib/python3.12/threading.py

That was quite expected. Let’s shadow the modules:

$ for mod in abc mailbox threading; do touch $mod.py; done
$ ls
abc.py	mailbox.py  threading.py

And try again:

$ for mod in abc mailbox threading; do python3 -c "import $mod; print($mod.__file__)"; done
/usr/local/lib/python3.12/abc.py
/empty/mailbox.py
/empty/threading.py
Exception ignored in: <module 'threading' from '/empty/threading.py'>
AttributeError: module 'threading' has no attribute '_shutdown'

Why is abc not imported from the current directory?

Even more weird, on my regular system, polluted with .pth files etc.:

$ for mod in abc mailbox threading; do touch $mod.py; done
$ ls
abc.py  mailbox.py  threading.py
$ for mod in abc mailbox threading; do python3 -c "import $mod; print($mod.__file__)"; done
/usr/lib64/python3.12/abc.py
/home/.../empty/mailbox.py
/usr/lib64/python3.12/threading.py

Now even threading is imported from the system path, but not mailbox.

But again, using a clean venv:

$ for mod in abc mailbox threading; do venv/bin/python -c "import $mod; print($mod.__file__)"; done
/usr/lib64/python3.12/abc.py
/home/.../empty/mailbox.py
/home/.../empty/threading.py
Exception ignored in: <module 'threading' from '/home/churchyard/tmp/empty/threading.py'>
AttributeError: module 'threading' has no attribute '_shutdown'

What am I missing? Is this expected behavior?

it seems that Python modules imported on startup are imported from the system dirs, but the rest is shadowed.

2

I suspect, but haven’t verified, that the current directory is only added sometime later in site.py. Stuff that is imported before that doesn’t see it, and once the module is in sys.modules it doesn’t get checked again.

3

That is pretty much what I suspect as well, but it is rather confusing.

4

This may provide a hint:

$ for mod in abc mailbox threading; do python3 -c "import sys; print('$mod', '$mod' in sys.modules)"; done
abc True
mailbox False
threading False

You may find it useful to explore the output of python3 -v to see when abc gets imported, and then look beyond that for changes to the path.

2 Likes