I wish to know the RAW data about the relative imports "issue"

Disclaimer: Please read the whole post before reply. The post is not as it appears at the first glance.

My post is about the (in)famous exception ImportError: attempted relative import with no known parent package. Wherever I look in the Net about the exception it seems to be a highly controversial topic. It is very unfortunate for me, because I really want to know the raw data about what people were about to do when this exception “stand in their way”. I’m not interested in what way is the “correct” or “right” one.

Before you ask: Yes, I think there may be a real issue here, but before I give my verdict on this I want to solve it like a scientist: no opinions, but first look at the data.

I tried to ask it elsewhere, but my post was always flagged as “opinion-based” :frowning: It seems no one did a research about the topic.

So, If you remember or have yourself meet with the exception, please share it with me. Or, if you know some other source where I can “dig” for the data, then share it too. I’ll be very grateful :smiley:

I’m not sure what you mean by raw data here. Do you want to see all the code that raised this exception? Or wWhat they wanted to do [1]?

What’s the hypothesis? Speaking as a scientist: we don’t work in secrecy–we share their ideas with others, especially when we’re asking for people to volunteer their time and effort.


  1. how could anyone record this? ↩︎

What they wanted to do. Yes this may be hard to find, but usually people when they post about this error they also say (now or later) what they wanted to do.

If there is an real issue or not.

There’s no controversy, just a lot of people who don’t understand what the error means.

Relative imports, like relative paths, need to resolve relative to something. That something is the current working directory for paths, but the current package for imports.

The current package of a module can be found in the __package__ attribute, which means relative imports like

from . import foo
from .subpackage import bar

are “equivalent” to imports like

# These aren't valid import statements, of course.
from __package__ import foo
from __package__.subpackage import bar

So the question is, how does __package__ get set to the correct value?

In a top-level module (found directly in one of the directories on your search path), __package__ is set to ''.

>>> import sys
>>> sys.__package__
''

In a module contained in package, __package__ is set to the name of the package.

>>> import unittest.mock
>>> unittest.mock.__package__
'unittest'

In a script, __package__ is set to None:

$ python3 -c 'print(__package__)'
None

That’s it. When you get the “no known parent” issue, it means you are either attempting a relative import from a top-level package or a script.

You can assign the name of any package you want to __package__ to force relative imports to resolve (see PEP 366), but usually the error means you haven’t properly installed a module before importing it or you are trying to execute a module as a script.

3 Likes

This seems to be what people are doing. But what I want to know is: why they do it? Is it because they don’t know well about the import rules? Or maybe there is something that they want to do that requires running a module as a script?

Maybe I should give an example: During my experience as Python programmer I usually encounter this exception when I try to run my testing code that tests my fresh new module. What I want to do is test my application and separate the test code from the app code by putting them in two different sibling catalogs. This is what I find to be a kind of standard on GitHub for repositories referenced on the PIP site.

In my view of this the exception forces me to find other ways than simply run a .py file. I understand the error, but I’m asking myself if others also have this or similar problems like me. I have got a proposal of how to “fix” it, but first I want to know two things: Are there any other use cases? Is my yet untold proposal really goes to the heart of the problem?

Suppose you have a directory tree like

.
├── script.py  # import src.mod1
└── src
    ├── mod1.py
    ├── mod2.py
    └── test_one.py  # import mod1

The unaware user (e.g., me, several years ago) might use import src.mod1 in the script because they figure, src is in the current directory and mod1 is in src, not realizing that the current directory has nothing to do with how the package is found. It’s the directory that script.py lives in that matters.

The same user might notice that import mod1 has to be used by test_one.py for the same reason, possibly not noticing that it works with either cd src; python test_one.py or python src/test_one.py. And everything is fine… as long as mod1.py doesn’t use a relative import.

But now the user adds from . import mod2 to mod1.py, because they think the . refers the same directory containing mod1.py, not the package containing mod1. And it works when they run script.py, because src is a package from script.py’s perspective. But from test_one.py’s perspective, mod1 is not in any package; it’s a top-level module, so now the relative import fails.

The solution, of course, is that

  1. script.py should not be treating src as a package in the first place, and using PYTHONPATH=src python script.py with script.py using import mod1, or
  2. test_one.py should be treating src as a package, using something like PYTHONPATH=. python src/test.py and import src.test1.

In the former, mod1 should not and cannot use a relative import to refer to mod2, because the two do not live in the same package.

In the latter, mod1 can use a relative import, because now mod1 and mod2 both live in `src.

Either way, script.py and src/test_one.py agree on whether src is a package or not.

2 Likes

I can give an contr-example. When I referred to the “GitHub standard” I was thinking about this dir structure:

.
├── my_package
|   ├── __init__.py
|   ├── mod1.py
|   └── mod2.py
└── tests
    ├── test_mod1.py
    └── test_mod2.py

I think we both can agree that tests directory is not a package, but my_package is.
The question arises: how test_modX.py can test modX.py?

In my view there is a kind of confusion in the Python language. Namely: in Python a directory structure == an application structure. I can agree it is a good thing, because it simplifies how an application is build. In other programming languages it is harder to create an application structure.

But there is one thing missing: in the above directory structure we have got 2 applications: one the main app called my_package and a second app that tries to test the first one.
Of course if we could do a thing like this in test_mod1.py:

from ..my_package import mod1

then a test code could test the mod1.py and do its job.

Of course we can’t do that and I see 2 problems why we can’t:

  • we are, as You stated, confuse directories with packages/modules. In the from ... import ... statement it looks like if my_package and tests belonged to the same app, which is wrong.
  • maybe the my_package/__init__.py does something important? In fact it could initialize the package so mod1 and mod2 could do what they should do.

My solution is to allow a construction like that:

from . in "../my_package" import mod1

This way it is clearly stated that mod1 is not a part of the test app. It also tells the Python interpreter where to look for an other application, so it can set the __package__ variable correctly. But what is the most important here is now the interpreter can execute the right code from __init__.py file/s on its way to the imported module.

I’ve read the one post from Guido where he said this is an antipattern. And I agree with this statement, but under two conditions:

  • We should forbid to “escape” from an application in imports when they are trying to import something that is not a part of their “realm”
  • But we should let people in some way allow them to import something from other application if that’s the way they do its job.

I would like to end this with the sentence from the Python’s Zen: “Simple is better than complex”. And doing any other things than to run a .py file directly is complex.

The correct way to do this is to install the package in a test environment, which is more controlled, more reproducible, and doesn’t require syntax changes.

3 Likes

I think it’s quite easy to find. You look at the existing questions that were asked. Most people asked them because of the situation they were getting into. They usually explain what they wanted to do, because they’re trying to figure out how to do that thing.

It sounds like you’re talking about Stack Overflow. Did you try using their site search, or adding site:stackoverflow.com to a search with an ordinary search engine? Anyway, tons of “research” has been done. If your question was closed on Stack Overflow, that has nothing to do with other people not knowing how to answer the question, and it has nothing to do with either your or their skill level, or anything else like that.

It is purely a matter of whether your question meets the site’s standards to be answered separately. It must be:

  • in scope for the site (questions about what you “should” do are not accepted);

  • properly scoped (ask one thing; if you want to know how to do something, make sure it’s precisely specified, and isn’t obviously something that could be broken down into smaller steps; if you are trying to fix your code, find the problem yourself first, make sure you have short code that directly reproduces the problem, and ask a question instead of expecting “help”);

  • clear (if they don’t understand you, this is generally your responsibility to fix, not theirs);

  • not a duplicate (it’s not inherently wrong to ask duplicates if you couldn’t find what you were looking for, but the answers are supposed to all go in the same place. Sometimes a duplicate can be helpful, because it improves searching for the next person).

(Part of the reason there are so many unhelpful Q&As on Stack Overflow is because most people asking questions don’t understand, or seem to care, how the site is intended to work or what its actual purpose is.)

But generally speaking: what they’re trying to do is to expect the folder structure on disk to dictate how relative imports work, without having to do any setup. It doesn’t and can’t work that way, because modules can be imported in many other ways besides “read a .py file and interpret the Python source code in it”, and because a given package can be split across multiple folders.

Incidentally, just because you will commonly see misinformation: __init__.py files do not fix this problem, whether or not they are empty. They can have many minor effects on the import system, which are often desirable, but “make Python treat this folder as somewhere that it can find a package using relative import” is not one of them. The primary effects are to prevent splitting the package across other folders and bail out early from the path-searching process (because it knows there are no “portions” to look for).

I can’t understand what you mean. If you are trying to say that you think there is a bug in Python, then no, the problem is that it was very specifically designed to work a certain way, and some people expected it to work differently. In any event, there isn’t any “data” to “dig” for. There is not any mystery about why the error occurs. The way that the system works is one of the most thoroughly described things you can find in the Python documentation. We’re just lacking in the other aspects of good documentation for the import system.

Better yet, follow established development practice by setting up a virtual environment; then one can pip install -e .. Then the path to src is reliably on sys.path. This becomes clearer if we use a “src layout” where src contains a mypackage folder that actually names the package. Then script.py, despite being outside of src, can directly use the sensible import mypackage.mod1 rather than the artificial and semantically wrong import src.mypackage.mod1; and mod1 can still from . import mod2. (Plus it will still work after packaging the project into a wheel, putting it on PyPI, then having someone else install it with Pip. :wink: )

By installing my_package first, and then letting the tests use absolute imports normally (import my_package.mod1 etc.). For the developer, this could look like pip install -e . in the top-level folder (with this layout; with a src layout, it might look like pip install -e src). Similarly if you cloned the repository from GitHub (you are now a “developer”, possibly on a fork of the project).

For the end user, normally the tests aren’t provided anyway. If they are, the developer will normally arrange the build process so that tests becomes a sub-package (but still using absolute imports). It would be impolite to make the wheel install a separate my_package package and a tests package - imagine if everyone did that; Pip would try to put all the folders for the tests packages into the same site-packages folder, and they would overwrite each other.

This doesn’t work the way you want it to, in part because the developer requires a different folder structure from the end user. In particular, the end user isn’t building wheels for the project, or using Markdown source files within the project to create online documentation on readthedocs, etc. etc. Ideally (for most users), the end user doesn’t even need setup.py or pyproject.toml, because their job has already been done (and the user gets a wheel instead of an sdist).

It’s certainly possible to import code from an arbitrary specified path, and create a module from it. (Admittedly, it’s nowhere near as simple as that kind of syntax). But for most situations it creates more problems than it could solve, so making it easier doesn’t seem to be a priority.

I can’t imagine any way that syntax like what you describe, could enforce this.

1 Like

test_modX.py simply uses

import my_package.mod1

and the test runner is responsible for making sure that my_package can be found, typically by adding . to the search path. (Installing my_package, as others have recommended, does this by putting my_package in a directory already on your search path.)

The problem is when one confuses the directory structure of a source repository for a properly installed Python script and/or package.

No, there are not two applications. There is a package my_package and a directory tests full of scripts that all want to import part or all of my_package.

PEP 366, mentioned earlier, provides a way to allow relative imports inside a module that can be run using the -m switch. It’s not generally meant to enable relative imports inside arbitrary top-level modules or scripts.

2 Likes