# How exactly does \_\_init\_\_.py influence module search order?

**URL:** <https://discuss.python.org/t/how-exactly-does-init-py-influence-module-search-order/24759>\
**Category:** Python Help\
**Tags:** documentation, help\
**Created:** [March 12, 2023, 7:05am UTC](https://discuss.python.org/t/how-exactly-does-init-py-influence-module-search-order/24759 "2023-03-12T07:05:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dragondive](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dragondive/32/11519_2.png) [@dragondive](https://discuss.python.org/u/dragondive)\
**Post date:** [March 12, 2023, 7:05am UTC](https://discuss.python.org/t/how-exactly-does-init-py-influence-module-search-order/24759/1 "2023-03-12T07:05:26Z")

</div>

The documentation on Packages says this:

> The \_\_init\_\_.py files are required to make Python treat directories containing the file as packages. This prevents directories with a common name, such as string, unintentionally hiding valid modules that occur later on the module search path.

The documentation on Module Search Path says this:

> When a module named `spam` is imported, the interpreter first searches for a built-in module with that name.

* * *

To properly understand this, I tried a few experiments:

1. Create a local directory `string` in the current directory `c:\WORK\dragondive\heavens-arena\python\modules` (from where python is called). Then do:

2. In the above local directory, create an empty ` __init__.py`. Run the same code again:

* * *

My `sys.path` looks like this:

> PS C:\WORK\dragondive\heavens-arena\python\modules\> python main.py  
> [‘C:\WORK\dragondive\heavens-arena\python\modules’, ‘C:\WORK\python’, ‘C:\Users\aravi\AppData\Local\Programs\Python\Python311\python311.zip’, ‘C:\Users\aravi\AppData\Local\Programs\Python\Python311\Lib’, ‘C:\Users\aravi\AppData\Local\Programs\Python\Python311\DLLs’, ‘C:\Users\aravi\AppData\Local\Programs\Python\Python311’, ‘C:\Users\aravi\AppData\Local\Programs\Python\Python311\Lib\site-packages’]

* * *

**Questions**

How exactly does the presence of an ` __init__.py` cause the Python interpreter to “prefer” the local `string` over the built-in string?

The module search path documentation says built-in modules are searched first, but the packages documentation says ` __init__.py` prevents hiding valid modules that occur _later_ on the module search path.

Does this imply that regular packages are loaded before modules?

This [Stack Overflow answer](https://stackoverflow.com/a/6393913) (from 2011) suggests so, but there’s also some disagreement in the comments.

Is the expected behaviour documented somewhere explicitly?

---

<div class="post-metadata">

**Author:** ![komoto48g](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/komoto48g/32/6978_2.png) [@komoto48g](https://discuss.python.org/u/komoto48g)\
**Post date:** [March 12, 2023, 4:10pm UTC](https://discuss.python.org/t/how-exactly-does-init-py-influence-module-search-order/24759/2 "2023-03-12T16:10:14Z")

</div>

I am not sure I can answer your question correctly.

But `pkgutil`, for example, does not recognize a directory without ` __init__.py` as a package.  
My guess is that the directory containing ` __init__.py` takes precedence.

By the way, I am interested in your situation and I experimented in the following environment:

```auto
~/workspace/
    string/
        __init__.py
        myscript.py

```

I got:

```auto
C:\temp\workspace>py
Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "C:\usr\home\.py", line 12, in <module>
    from PIL import Image
  File "C:\Python310\lib\site-packages\PIL\Image.py", line 30, in <module>
    import logging
  File "C:\Python310\lib\logging\ __init__.py", line 28, in <module>
    from string import Template
ImportError: cannot import name 'Template' from 'string' (C:\temp\workspace\string\ __init__.py)
>>>

```

( **EDIT:** The line `from PIL import Image` is written in the PYTHONSTARTUP file.)  
So, `logging` and other libraries can conflict with the local `string` package.  
If you plan to share your library with others, I strongly recommend that you do not use package names that conflict with the standard library.

---

<div class="post-metadata">

**Author:** ![dragondive](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dragondive/32/11519_2.png) [@dragondive](https://discuss.python.org/u/dragondive)\
**Post date:** [March 12, 2023, 5:53pm UTC](https://discuss.python.org/t/how-exactly-does-init-py-influence-module-search-order/24759/3 "2023-03-12T17:53:07Z")

</div>

@komoto48g Thank you for your response.

Indeed, I will not use this code in a production environment. Nonetheless, it is important for me to understand why. This is also frequently helpful to debug problems when reusing code from multiple “internal” projects, which can get messy quickly. Especially with a language like Python which makes changes across versions, that are not always fully backward compatible.

The documentation (and the information available online, for example, on Stack Exchange sites and Reddit forums) isn’t satisfactory in my view. “Just create an \_\_init\_\_.py, it makes the directory a package and prevents conflict later” (as the documentation suggests) isn’t sufficiently reliable explanation.

I hope someone who has worked on the “behind the scenes” implementation and/or documentation can shed more light on this.

---

<div class="post-metadata">

**Author:** ![dragondive](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dragondive/32/11519_2.png) [@dragondive](https://discuss.python.org/u/dragondive)\
**Post date:** [March 13, 2023, 6:08am UTC](https://discuss.python.org/t/how-exactly-does-init-py-influence-module-search-order/24759/4 "2023-03-13T06:08:40Z")

</div>

I found the most clear explanation on this topic here from Victor Skvortsov: [Python behind the scenes #11: how the Python import system works (tenthousandmeters.com)](https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works/). In particular:

> How does it work? When Python traverses path entries in the path (`sys.path` or parent’s ` __path__ `) during the module search, it remembers the directories without ` __init__.py` that match the module’s name. If after traversing all the entries, it couldn’t find a regular package, a Python file or a C extension, it creates a module object whose ` __path__ ` contains the memorized directories.
> 
> The initial idea of requiring ` __init__.py` was to prevent directories named like `string` or `site` from shadowing standard modules. Namespace package do not shadow other modules because they have lower precedence during the module search.

This explanation resolves my queries. Although it’s not clear if this behaviour is officially documented. I wrote to Victor thanking him for the explanation, and he pointed me to [PEP 420 – Implicit Namespace Packages | peps.python.org](https://peps.python.org/pep-0420/) for further reading. I will take my time to understand the PEP and update here what I learn.

---

<div class="post-metadata">

**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\
**Post date:** [March 13, 2023, 7:55pm UTC](https://discuss.python.org/t/how-exactly-does-init-py-influence-module-search-order/24759/5 "2023-03-13T19:55:18Z")

</div>

First observation: Python treats packages _as a kind of module_. They’re cached in `sys.modules` and represented with the `module` type (not even a subtype). Whereas the attributes of an “ordinary” module (created by `import`ing a `.py` file, for example) are the global names in the top-level code, the attributes of a package are its contained modules - as well as anything defined in ` __init__.py`, as a special case.

Second observation: Python doesn’t actually create packages (rather, `module` objects representing packages) “from” the folder. It creates them according to the rules of the `import` system. With the filesystem-based importer, there are two ways this happens: explicitly by executing code from a `.py` file (or cached `.pyc` bytecode, etc.), or _implicitly, from nothing at all_ (a namespace package, the need for which is deduced from the `import` statement syntax).

> [@dragondive](#):
>
> The module search path documentation says built-in modules are searched first, but the packages documentation says ` __init__.py` prevents hiding valid modules that occur _later_ on the module search path.

There is no contradiction.

**Built-in** modules (which _does not mean_ “standard library modules”, but _only_ the ones listed in `sys.builtin_module_names` - ones which the reference implementation implements in C) are checked before any attempt to look for source code in the places mentioned in `sys.path`.

**The requirement to use** ` __init__.py` to make Python treat the directories as packages, is so that e.g. the standard library `string` module (which is _not_ built-in) won’t be _unavoidably_ hidden unintentionally by a local folder called `string`.

Because of the requirement, the `string` folder doesn’t prevent the standard library `string` from being imported, all by itself. But if an ` __init__.py` is added, the folder will be recognized as a package; and since _its containing_ folder appears (by default) earlier on `sys.path` than the standard library folders (deliberately placed at the end, so that this option exists), `import string` will import that package.

“Explicit is better than implicit”, as they say. (Something the _documentation writer_ could take to heart; it’s not very clear what “this” refers to in the first quotation.)

> [@dragondive](#):
>
> How exactly does the presence of an ` __init__.py` cause the Python interpreter to “prefer” the local `string` over the built-in string?

I tried many ways to write this explanation, because there are a **lot** of ways to customize the Python import system. Ultimately, though, I think it’s better here to focus on the important parts of the one specific situation you tested.

The `string` folder can’t create a module at all. If there is a ` __init__.py` file, Python will find that, and create a module from it, before it looks at the standard library. Python _already_ “preferred” this path.

Without ` __init__.py`, after making every possible attempt to identify `string` as an ordinary module, Python concludes that `string` must be the name of a package instead. During the search for source code, Python will have found and remembered the `string` folder, and _because it found_ such a folder, it will create a `string` “namespace package”.

The details of this process are described in PEP 420 as you already linked. I think this constitutes explicit documentation. You may also be interested in the [non-tutorial](https://docs.python.org/3/reference/import.html) official documentation for the import system.