Improve ModuleNotFoundError to cater for condition when directory exists

there are conditions when the directory exists eg. library
but python complains -
ModuleNotFoundError: No module named library

the library directory has the necessary file - but because the code was not invoked with

python -m module - then it fails to find them. from secuity perspective - the error may make sense - but end user - its frustrating and google doesn’t result in clear resolutions.

a) when i execute the code
python should be smart enough to inspect the library folder in this condition - with hardened security (whatever) to just test if the directory exists and has the in it - and then complain -

ModuleNotAccessibleError: Module named ‘library’ inassessible due to security decisions from Python. (Run with -m or pip install -e editable to run run with modules )
something different from the vanilla not found error - because you’ve found it.

b) if you can’t do that introspection with compromising security of python
change the error message to indicate - it maybe inaccessible.
ModuleNotFoundErrorOrModuleNotAccessibleError: No module named library or Module named library maybe inaccessible (we don’t know because we cant run introspection without compromising security.


I suggest the error should give more context - and provide feedback that the module is inaccessible - which then opens different paths for recovery.


How was the code invoked? Python can do both python -m library (where library is a package in a directory that’s on sys.path, including the current working directory) or python library (where library is a path to a directory that is a package)

Hi, John, welcome to the forum. Unfortunately, I can’t see anything here that could possibly make an actionable suggestion - since it’s based on serious, fundamental misunderstandings of how the existing system actually works. I’m moving this to the Python Help section, and giving a more detailed explanation below.

This is wrong on multiple levels.

First off, since Python 3.3, it has not been necessary to include an file in the directory simply to be able to import it as a package. Adding just turns what would otherwise be a namespace package into a regular package (and also provides a place for package-level initialization).

Second, ModuleNotFoundError comes from failed absolute imports only - not relative imports. From looking at the linked Conda issue, it seems that you are starting from the sd-scripts subdirectory of some cloned kohya_ss repository; and from there you want to run finetune/ But the project is simply not designed to work that way; you need to follow the instructions in the project’s README, and/or take it up with that project’s issue tracker - not with Conda’s, and not with the Python dev team. (The fact that you set up a venv for this project, following the directions, is also not relevant to why the problem occurs - because the library folder isn’t expected to be in site-packages anyway, so it’s not relevant which Python’s site-packages are used. You’re using code that was designed to be cloned and run in place, not “installed”; you’re asked to use a venv to have an isolated environment for third-party dependencies like Torch.)

At any rate, when you use python finetune/, yes, the finetune folder is added to sys.path - not the current folder. This is clearly documented: “The first entry in the module search path is the directory that contains the input script, if there is one.”. It’s also explained in the documentation specifically for sys.path: “python command line: prepend the script’s directory.”. (I.e.: not necessarily the shell’s PWD.)

Third, this behaviour is not a security measure. The reason the module is not found is because it isn’t in any of the places that Python was told to look. It isn’t failing to look in the library folder for any security reason. It’s failing to look in the library folder because it has no reason to look there. From the interpreter’s perspective, there’s nothing special about that folder path or about its name; no reason to suspect that this folder contains a; and no reason to believe that said is the right one for the import in question. And, again, an file is not needed to make any given folder get interpreted as a package; thus, checking for its presence is not any kind of security measure (nor a “prevent mistakes” measure).

That has more to do with a huge number of people believing strange and very wrong things about Python’s import system, even though the normal operation of import is quite simple.

For me, putting something like where does python look for modules into a search engine gives plenty of good (or at least correct) results. You need a better search query or a better search engine. Or you need a willingness to go right to the documentation to understand the system in detail, or to look up the section that specifically explains sys.path configuration (previously linked), or to follow the built-in tutorial.

I can only make sense of this in the context of the example you posted on the Conda issue tracker.

There is, under these conditions, nothing special about this folder, as far as Python is concerned, and no reason why it would look there. The proposed rule seems to be something like: “if Python can’t otherwise find a module, check if it would work if the current working directory were on sys.path”. But in that case, it would make far more sense to just put it there in the first place.

Aside from being really long, this would be lying to the user. The problem doesn’t have anything to do with permission to look in the library folder. It’s simply a matter of not having guidance to look there - just like it wouldn’t look in any other unrelated folder on your filesystem, even if that file is perfectly accessible and everyone is allowed to read it.

“Not found” and “not accessible” mean the same thing here. The problem, again, isn’t because of some inability to look into folders; it’s because the folders that were looked into don’t contain the module. “Not found” perfectly describes that already. The only other “context” that could possibly be offered is the current contents of sys.path.

1 Like

As an aside: I wouldn’t really consider python -m finetune.make_captions a proper fix. While it does result in putting the current directory onto sys.path, the purpose for that is to make it possible to find finetune - which has to be imported first, so that its __package__ can be set before finetune.make_captions is imported. Importantly, this allows for make_captions to use relative imports within the finetune package. The -m option is there specifically to support relative imports (and avoid ImportError: attempted relative import with no known parent package), not absolute imports. If you were willing to run finetune/ directly, that implies not having any interest in ensuring that finetune is recognized as a package.