ModuleNotFoundError: No module named '...' (pyproject.toml, venv, pip install -e, vanilla Python)

Python Help
1

Hi. Being still new to Python ecosystem, I’d appreciate to get a helping hand with minimal pip install -e + pyproject.toml configurations for local package development.

Please consider following example:

mylib

(src-layout here)

/tmp/demo-lib/
├── pyproject.toml
└── src
    ├── mylib
    │   ├── helper.py
    │   └── main.py

# pyproject.toml
[project]
name = "mylib"
version = "0.1"

# src/mylib/main.py
def say_hello():
    print("Hello world")

# src/mylib/helper.py
def say_hello_helper():
    print("Hello helper")

myclient

/tmp/demo-client/
├── main.py
└── venv

# main.py
import mylib.main as T
T.say_hello()

Testcases

. /tmp/demo-client/venv/bin/activate
pip install -e /tmp/demo-lib/
# ...
# Successfully built mylib
# Installing collected packages: mylib
# Successfully installed mylib-0.1

1.) Single module works

(venv) [myuser@sys demo-client]$ python main.py 
Hello world

2.) helper.py causes ModuleNotFoundError

To reproduce, change /tmp/demo-lib/src/mylib/main.py:

import helper

def say_hello():
    print("Hello world")
    helper.say_hello_helper()

(venv) [myuser@sys demo-client]$ python main.py 
  File "/tmp/demo-lib/src/mylib/main.py", line 1, in <module>
    import helper
ModuleNotFoundError: No module named 'helper'

(venv) [myuser@sys ~]$ pip --version
pip 23.3.1 from /tmp/demo-client/venv/lib64/python3.11/site-packages/pip (python 3.11)

I am able to navigate to demolib and run main.py (but this won’t solve problem client issue):

[myuser@sys ~]$ cd /tmp/demo-lib/src/mylib/
[myuser@sys mylib]$ python -c 'import main; main.say_hello()'
Hello world
Hello helper

Question

I am interested in the correct or recommended way to get this going natively - just vanilla Python/pip, not using any third-party dependencies as poetry.

From what I’ve read about PEP 660, this scenario with interactive pip install and pyproject.toml should be supported out of the box right? And shouldn’t the module import resolution for helper handled transparently within mylib from perspective of client, when using pip install -e?

I totally might have missed something obvious, so please correct me in this case.

1 Like
2

I recently converted my CLI tools projects to use tooling as you want to with a pyproject.toml for each tool.

The code is here: GitHub - barry-scott/CLI-tools: A collection of command line (CLI) tools

Look at build.sh to see the workflow that I use.

3

I guess you need a [build-system] table in your pyproject. What happens when you do pip install -e /tmp/demo-lib?

4

Thanks @barry-scott for your answer.

Honestly I am not quite sure what to carry over from that project into above minimal example. What I noticed: All pyproject.toml files have dependencies key and [build-system] table entries amongst others defined.

IIRC dependencies is for pip package installation (not module resolution). As mylib is found in Python module path from the client, everything should be fine here. I added pyproject.toml to myclient for testing purposes:

[project]
name = "myclient"
version = "0.1"
dependencies = [
    "mylib",
]

and renamed main.py → myclient.py, without change in results.

[build-system] seems to be optional, as defaulting to setuptools.

To name another resource, I found blog article Python packages with pyproject.toml and nothing else (not affiliated) simple to follow and got that very example to work. Unfortunately it only uses a distribution package with a single module, not multiple modules, which is my issue.

5

Rerun pip install -e /tmp/demo-lib, here is the output:

(venv) [myuser@sys demo-client]$ pip install -e /tmp/demo-lib/
Obtaining file:///tmp/demo-lib
Installing build dependencies … done
Checking if build backend supports build_editable … done
Getting requirements to build editable … done
Preparing editable metadata (pyproject.toml) … done
Building wheels for collected packages: mylib
Building editable for mylib (pyproject.toml) … done
Created wheel for mylib: filename=mylib-0.1-0.editable-py3-none-any.whl size=1088 sha256=0ef8719055b7b53f5953e9ace5eda260193a49d7c7d4df5aa022be72b5c6608e
Stored in directory: /tmp/pip-ephem-wheel-cache-wenmus_5/wheels/5a/4c/85/37da9e7b4650244f4e8a82144bc7a1adc716e62c5686bc9be1
Successfully built mylib
Installing collected packages: mylib
Attempting uninstall: mylib
Found existing installation: mylib 0.1
Uninstalling mylib-0.1:
Successfully uninstalled mylib-0.1
Successfully installed mylib-0.1

pip list confirms, it’s installed in venv:

(venv) [myuser@sys demo-client]$ pip list
Package    Version Editable project location
---------- ------- -------------------------
mylib      0.1     /tmp/demo-lib
pip        23.3.1
setuptools 62.6.0

Also tried to add

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

to both projects, as suggested by above blog entry (build-system defaults to setuptools according to it), no change.

6

I could follow your issue. But I noticed that a small change to the import made it work. Please try:

import mylib.main
mylib.main.say_hello()

To my surprise this worked for me. I checked the pip bugtracker but could not readily find an issue for this. I think this is an issue with editable installs, maybe you want to create an issue there.

7

Presumably you need to import mylib.helper as helper here? Alternatively you can use import .helper for a relative import.

1 Like
8

Yes, this actually worked - great! My interpretation so far is (please correct me otherwise):

  • In src-layouts, intra-package imports per default need to prefix module imports by the package name under src - e.g. import mylib.helper.
  • “Per default”, as we created a minimal pyproject.toml, where most standard settings are inferred.
  • Regarding import module resolution the default is Automatic discovery.
  • Automatic discovery recognizes src-layout structure for our project, and sets package_dir 1 to {"" = "src"}, i.e. src becomes package root.

@Mholscher If I change the import with package prefix, the alias import works as well :+1:

1 http:/setuptools.pypa.io/en/latest/references/keywords.html#keyword-package-dir
(can only post 2 links as new user)

9

Hi Luis and welcome.

First off, thanks for the excellent example; it shows everything necessary to understand the problem. I might want to refer others with a similar problem to this thread in the future.

It seems you’ve resolved the issue, but I want to correct your interpretation like you asked for, and add a bunch of background information so that you can properly understand the system (and also to practice writing out the key ideas so I can make a proper article about it in the future). It’s kinda long but hopefully entertaining enough to keep you reading :wink:

Okay; you’ve correctly identified some of the things that happen with a src layout that are different from a non-src layout - but these things are not relevant to the problem you experienced.

The reason import helper doesn’t work in the original setup is because that is an absolute import, and the mylib folder isn’t in the list of paths that Python will search in order to perform absolute imports. But that has nothing to do with src-layout. After all, this is happening after you installed the demo-lib to your venv. It also doesn’t matter whether you do this in the standard way or in editable mode with -e.

What happens is that some metadata files are put into the venv’s site-packages folder, so that when Python starts up, this metadata is read and used to add the necessary path to sys.path. But this is really an implementation detail. With a full install, it would work by setting up a mylib folder within site-packages directly.

Either way, the mylib folder (and therefore the mylib package) is available as a top-level absolute import, but helper.py (and therefore the helper module) is not. Again, this is not because you have a src layout, but because you have installed the mylib package.

helper is supposed to be part of the mylib package after all. An absolute import should look like import mylib.helper or from mylib.helper import say_hello_helper, etc. That is supposed to be the same regardless of whether you installed the package or are just developing it “directly”. Anything that enables you to import helper is a hack that is deliberately ignoring the package structure that you explicitly set up, presumably with a reason.

This is the same reason that /tmp/demo-client/main.py says import mylib.main as T, and not import main as T, and would be like that even if there weren’t a name conflict.

The opinionated part starts here.

The right way to solve the problem is with a relative import: in the library main.py, use from . import helper (or you can alias it with as, import the function only like from .helper import say_hello_helper, etc. etc.) By using relative imports within the library, we never have to worry about whether the code is installed or not (or where), or think about sys.path. It’s the responsibility of the driver code to do one absolute import to an appropriate entry point in the library; doing so ensures that the top-level package is in sys.modules and that the appropriate __package__ attributes are set up correctly. Thereafter, relative imports from one part of the library to any other part of the library will Just Work.

Doing things this way gives you the best of all worlds. Within library code, a relative import documents the intent to work inside the library, while an absolute import documents the intent to work with a third-party dependency or with the standard library. Minimizing absolute imports minimizes dependency on correct sys.path configuration, avoids name conflicts (you can have e.g. tokenize.py as part of your library without issue, as long as it isn’t supposed to be an entry point for the application code) and is just generally making proper use of an important namespacing feature, giving your library code a real identity as library code. Meanwhile, the application code’s initial absolute import into the library means that it assumes the responsibility for that initial setup. Your library doesn’t even have to know its own name to function.

Friends don’t let friends hack (i.e., by writing code) sys.path. This message brought to you by… well, me, I guess. But do keep in mind that many of the largest and most influential Python projects have zero, or perhaps one, mention of sys.path across hundreds of thousands of lines of code each - often in some ancillary code needed to run Sphinx or something like that.

1 Like
10

More than welcome - thanks very much for the explanations @kknechtel !

Understood. So the actual error cause is a missing entry '/tmp/demo-lib/src/mylib' in Python’s sys.path for this case, when running the client.

I also see your point with the src layout. Just to make sure to have grasped it fully: Isn’t awareness of src layout here still relevant? You can only use import mylib.helper, because Python/pip recognizes <project-root>/src as known/pre-defined project layout via auto discovery and automatically adds '/tmp/demo-lib/src' to sys.path right? To test this, I added

import sys
print(sys.path)

to /tmp/demo-client/main.py and got

[‘/tmp/demo-client’, ‘/usr/lib64/python311.zip’, ‘/usr/lib64/python3.11’, ‘/usr/lib64/python3.11/lib-dynload’, ‘/tmp/demo-client/venv/lib64/python3.11/site-packages’, ‘/tmp/demo-lib/src’, ‘/tmp/demo-client/venv/lib/python3.11/site-packages’]

Great reminder, will definitely try these out. I lost sight of relative imports after having tried them couple months ago, having experienced issues. But I guess it was due to the reason I tried to use relative imports in combination with a script instead of modules.

11

Well, it’s useful to know that src layouts and non-src layouts exist, and that people have reasons for choosing them.

You mostly have the right idea about the discovery process, but it’s important to understand that sys.path gets set up like that when you start Python, not when you install the project. When Python starts up, it will run a special standard library module called site.py, which looks for the site-packages folder and makes the needed changes to sys.path. In your case, sys.path has '/tmp/demo-client/venv/lib64/python3.11/site-packages' and '/tmp/demo-client/venv/lib/python3.11/site-packages' because site.py added them automatically, and 'tmp/demo-lib/src' because of the metadata it found in one of those folders. (It has the '/usr/lib64' subfolders because that’s where the standard library components are; and it has '/tmp/demo-client' because of how you started up Python.)

But let’s focus on the install process for a moment. In order for everything to work smoothly, after all, the installation has to cooperate with the system implemented by site.py and the site-packages folder.

In your case, the layout was detected automatically. It’s actually the build backend which does this - in your case, this is probably Setuptools (since if it were anything else, you would have had to understand all this stuff in order to choose). When your package is installed, Pip first has to “build a wheel” from it; by default, it uses Setuptools to do that, and Setuptools has some code to look at the folder names and guess what packages should be stored in the wheel and how they’re organized in your project folder.

If this isn’t working, or if you’d rather be explicit (generally considered a virtue), you can edit the pyproject.toml to describe how the project is laid out. Currently, this needs to be done in a backend-specific way. For Setuptools, use the tool.setuptools.packages entry. In your case (if I’m thinking clearly!) it might look like:

[tool.setuptools.package-dir]
mylib = "src/mylib"

Unfortunately, there are a few ways to do it, and none of it is very well documented at the moment from what I can tell.

There are also other backends you can choose besides Setuptools; that looks like

[build-system]
requires = ["flit"]
build-backend = "flit.api:main"

for Flit, or

[build-system]
requires = [ "poetry>=0.12",]
build-backend = "poetry.masonry.api"

for Poetry. Then from there you need to check the backend’s documentation for any other configuration options. However, generally these tools try to help you write the pyproject.toml file, rather than expecting you to edit it manually.

1 Like