Which is the right way to use modules within a package?

Packaging
1

Trying to understand the following which is always giving me trouble - Let’s say, I have a minimalistic python package that looks as follows:

maciamug@maciamug-PNMD6V mygalaxy % tree .
.
├── README.md
├── poetry.lock
├── pyproject.toml
├── src
│   └── mygalaxy
│       ├── __init__.py
│       ├── calc.py
│       └── planet.py

planet.py uses calc.py module, I am importing it as follows:

from mygalaxy.calc import planet_mass, planet_vol

This works just fine (trial and error). What I wonder is, why was the following ways not working:

from calc import planet_mass, planet_vol

or else

from .calc import planet_mass, planet_vol
1 Like
2

Hi Gloria, the import system in Python evolved quite a bit since its first implementation, so right now we have 2 (main) ways of writing an import:

  1. absolute imports: this corresponds to your first example (from mygalaxy.calc import ...)
    When using this syntax you need to write the whole path to your package/module.
  2. explicit relative imports: this corresponds to your last example (from .calc import ...)
    When using this syntax you need to pay attention to the location of the file where you are writing the import statement.
    from .calc import ... means that the calc module is a sibling of the module you are writting, i.e. it is located inside the same folder.
    This should also work depending on how you are calling your Python scripts.
    If it is not working probably there is some detail in your development environment that is not quite right… For example, you might need to run pip install ./pip install -e . (or set PYTHONPATH accordingly) to make the package under development available to your tests.

There is a nice article about this in Absolute vs Relative Imports in Python – Real Python.

The statement from calc import ... will not work because it tries to use an implicit relative import which is no longer supported by Python (it was removed for being ambiguous and the cause of common problems).

2 Likes
3

By Gloria Macia via Discussions on Python.org at 26Apr2022 14:15:

Trying to understand the following which is always giving me trouble -
Let’s say, I have a minimalistic python package that looks as follows:

maciamug@maciamug-PNMD6V mygalaxy % tree .
.
├── README.md
├── poetry.lock
├── pyproject.toml
├── src
│   └── mygalaxy
│       ├── __init__.py
│       ├── calc.py
│       └── planet.py

planet.py uses calc.py module, I am importing it as follows:

from mygalaxy.calc import planet_mass, planet_vol

This works just fine (trial and error). What I wonder is, why was the following ways not working:

That’s a normal “absolute” import, where the module name has a path from
the top of the tree. It presumes that “src” is in your syspath list.

from calc import planet_mass, planet_vol

This is an absolute import, and the path is wrong (“calc” is not at the
top of the tree).

or else

from .calc import planet_mass, planet_vol

I’d expect this to work. Can you shows the full error traceback? Also
your sys.path (inside Python: import sys; print(sys.path) and/or
$PYTHONPATH environment variable.

It may matter how you invoked things. Are you:

  • invoking something else which imports from mygalaxy etc, and finds
    it with sys.path
  • invoking planet.py directly? (in which case I would not expect from mygalaxy.calc to work)

Can you show the invocation command (how you start Python to test
these)?

My personal habit would be to either (a) install the mygalaxy package
using pip or (b) add /path/to/src to my $PYTHONPATH environment
variable.

I’d hope to use this form:

from .calc import planet_mass, planet_vol

because planet.py knows that calc.py is part of its own package, and
this avoids needing to know the absolute name of the package.

If planet.py was found by an import, then it knows it is in a package.
If you invoke planet.py directly, I believe it does not consider it
part of a package, and therefore the relative import path .calc would
not work.

So a script foo.py which goes:

from mygalaxy.planet import something

would let planet.py get to .cacl.

Cheers,
Cameron Simpson cs@cskk.id.au

4

@abravalheri you were right. Absolute import work, explicit relative import works - implicit relative import fails.

I am running on poetry and it was needed to do “poetry install” to update the version of my package in the poetry virtual environment.

1 Like
5

Very interesting @cameron

This is my print(sys.path) output

['', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/maciamug/Library/Caches/pypoetry/virtualenvs/mygalaxy-dawQFL-m-py3.9/lib/python3.9/site-packages', '/Users/maciamug/Desktop/repos/mygalaxy/src']

So if I understood you here correctly (please bear in mind with me, a package beginner):

from calc import planet_mass, planet_vol

That’s a normal “absolute” import, where the module name has a path from
the top of the tree. It presumes that “src” is in your syspath list.
type or paste code here

Does not work because it would expext “calc.py” to be at the same level as “src”. I moved calc.py at the source level like this:

maciamug@maciamug-PNMD6V mygalaxy % tree .
.
├── README.md
├── calc.py
├── poetry.lock
├── pyproject.toml
├── src
│   └── mygalaxy
│       ├── __init__.py
│       └── planet.py

And after poetry install to update the package it worked!

My only question remaining is, when I run poetry build I see only files inside src are taking. In other words, only files inside src folder get uploaded to PyPi which I guess it means that someone trying to download from there this package would get something broken (calc.py missing as shown below). Is this correct?


maciamug@maciamug-PNMD6V dist % tree mygalaxy-0.1.3
mygalaxy-0.1.3
├── PKG-INFO
├── README.md
├── pyproject.toml
├── setup.py
└── src
    └── mygalaxy
        ├── __init__.py
        └── planet.py

So in practice then this is not a great way to import and I should rather use absolute or explicit relative imports.

from calc import planet_mass, planet_vol

Thanks a lot for your guidance, much appreciated as I learned a lot from you.

6

Not quite; what @cameron means is that it presumes calc.py is a top-level module inside your src directory, from which it is presumed that your modules/import packages are installed or is visible to Python (via one of many possible ways). I.e.

├── README.md
├── poetry.lock
├── pyproject.toml
├── src
│   |── calc.py py
│   └── mygalaxy
│       ├── __init__.py
│       └── planet.py

But this is presumably not what you want to do, as long as you want calc to be part of your mygalaxy import package as opposed to a totally separate module.

This is most certainly not what you want to do. Since it is outside your import package and outside your installed package directory, if it still works to import it, it only does so purely by accident (e.g. mygalaxy-0.1.3 happens to be your current working directory, and your particular method of starting Python puts the CWD on the module search path). It will not work for anyone else who installs and tries to use your package (as you indeed discovered), or if that isn’t your CWD (or otherwise on your path), or invoking it in a way that doesn’t have your CWD on your path, or… It also means that the local copy of code can be picked up by the installed copy of your package, which can lead to all sorts of unexpected behavior. In short, please do not ever do this.

Instead, if you want to use a relative import, simply be explicit and prefix code with .—as indeed, it seems you figured out on your own.

Best of luck!