Hi Luis and welcome.
First off, thanks for the excellent example; it shows everything necessary to understand the problem. I might want to refer others with a similar problem to this thread in the future.
It seems you’ve resolved the issue, but I want to correct your interpretation like you asked for, and add a bunch of background information so that you can properly understand the system (and also to practice writing out the key ideas so I can make a proper article about it in the future). It’s kinda long but hopefully entertaining enough to keep you reading
Okay; you’ve correctly identified some of the things that happen with a src layout that are different from a non-src layout - but these things are not relevant to the problem you experienced.
import helper doesn’t work in the original setup is because that is an absolute import, and the
mylib folder isn’t in the list of paths that Python will search in order to perform absolute imports. But that has nothing to do with src-layout. After all, this is happening after you installed the
demo-lib to your venv. It also doesn’t matter whether you do this in the standard way or in editable mode with
What happens is that some metadata files are put into the venv’s
site-packages folder, so that when Python starts up, this metadata is read and used to add the necessary path to
sys.path. But this is really an implementation detail. With a full install, it would work by setting up a
mylib folder within
Either way, the
mylib folder (and therefore the
mylib package) is available as a top-level absolute import, but
helper.py (and therefore the
helper module) is not. Again, this is not because you have a
src layout, but because you have installed the
helper is supposed to be part of the
mylib package after all. An absolute import should look like
import mylib.helper or
from mylib.helper import say_hello_helper, etc. That is supposed to be the same regardless of whether you installed the package or are just developing it “directly”. Anything that enables you to
import helper is a hack that is deliberately ignoring the package structure that you explicitly set up, presumably with a reason.
This is the same reason that
import mylib.main as T, and not
import main as T, and would be like that even if there weren’t a name conflict.
The opinionated part starts here.
The right way to solve the problem is with a relative import: in the library
from . import helper (or you can alias it with
as, import the function only like
from .helper import say_hello_helper, etc. etc.) By using relative imports within the library, we never have to worry about whether the code is installed or not (or where), or think about
sys.path. It’s the responsibility of the driver code to do one absolute import to an appropriate entry point in the library; doing so ensures that the top-level package is in
sys.modules and that the appropriate
__package__ attributes are set up correctly. Thereafter, relative imports from one part of the library to any other part of the library will Just Work.
Doing things this way gives you the best of all worlds. Within library code, a relative import documents the intent to work inside the library, while an absolute import documents the intent to work with a third-party dependency or with the standard library. Minimizing absolute imports minimizes dependency on correct
sys.path configuration, avoids name conflicts (you can have e.g.
tokenize.py as part of your library without issue, as long as it isn’t supposed to be an entry point for the application code) and is just generally making proper use of an important namespacing feature, giving your library code a real identity as library code. Meanwhile, the application code’s initial absolute import into the library means that it assumes the responsibility for that initial setup. Your library doesn’t even have to know its own name to function.
Friends don’t let friends hack (i.e., by writing code)
sys.path. This message brought to you by… well, me, I guess. But do keep in mind that many of the largest and most influential Python projects have zero, or perhaps one, mention of
sys.path across hundreds of thousands of lines of code each - often in some ancillary code needed to run Sphinx or something like that.