Import files directly from file location

Importing new modules in Python is a pain, and far from straightforward. The trivial way to import a new module would be to give the full path of the file containing the required code - which could be a python __init__ module itself, setting up a fully functional module.

The import behaviour could change such a way that every beginner would feel involved, and we wouldn’t have long discussions about importing modules.
The new behaviour - added to existing - would look like:

import ‘path/to/module/module_name.py’

OR

import ‘path/to/module/module_name.py’ as module_name

The first one would use the filename as the module_name by default.

I’m aware that this behaviour is already solved if the path of the file is in the python path, however it only works if python starts AFTER the python path is defined AND the file is there already in the path. Which makes it very confusing using the import functionality from an ipynb notebook.

Solution proposal:

import sys
import types
import importlib.machinery

define the name and path of your module

module_name = ‘new_module’
module_path = ‘/path/to/new_module.py’

create a loader for the module

loader = importlib.machinery.SourceFileLoader(module_name, module_path)

load the source code from the .py file

code = loader.get_code(module_name)

create a new module object

new_module = types.ModuleType(loader.name)

add the source code to the module object

exec(code, new_module.dict)

add the module to the cache

sys.modules[loader.name] = new_module

Can you elaborate on that? There’s an import path, a simple import module statement, and a shortcut for beginning where the current directory is on the import path. After learning and doing simple scripts, more features of the import and install systems can be used (site packages, user site-packages, virtualenvs, different compiled code for platforms…) and imports still work without changing the code.

On the contrary, that would be a less easy way. More details of the install would be in the code. In other languages like javascript I have found it much less pleasant to need to give file paths.

For reference, my answer to the question python - How can I import a module dynamically given the full path? - Stack Overflow is nearly solely responsible for my 20.000 karma on that site. And it’s continuously rising. There is also a closed ticket about a convenience function: Consider leaving importlib.abc.Loader.load_module() · Issue #65635 · python/cpython · GitHub

I don’t think this warrants changing the import statement, but a convenience function in importlib (like it existed in the Python 2 imp module) would be rather nice.

See also gh-104212: Add importlib.util.load_source_path() function by vstinner · Pull Request #105755 · python/cpython · GitHub

1 Like

What’s specifically wrong with that? Why does having the file already exist on a path cause issues? If you’re specifying file paths you’re already hard-coding code layout, so I’m not seeing why this is a concern?

It also circumvents package structures and any other aspect of the import system where resolution by name is used (e.g. __path__). And this ignores how relative paths should be resolved which instantly become broken when you move a file in your file system. Also note that Python code does not need to come from the file system, which would break down if you forced all modules to be specified by path (e.g., what would you expect someone to do if they used a URL or SQLite database to store their module code?).

1 Like

The problem with the current behaviour is that otthon is ám interpreted language, and I would bet that a vast majority(!) uses it on a daily basis from ipython notebook, through a browser. While everything else seems trivial, the importing functionality is a nightmare, especially from an already running notebook.
Maybe the file path content could be dynamically updated at every import statement then? That would leave the current syntax intact, and would give a more intuitive behaviour.
My - and I’m sure many - people’s biggest issue was always: “why doesn’t it work, I placed it right next to my ipynb file!”. And it’s not documented well either, that the file has to be there BEFORE you start the kernel.
I know, sounds lame to argue from an ipynb notebook standpoint, but it made python popular, better accept it.

I don’t think that’s true; Python is used in a huge number of contexts, including embedded within other applications, scripts that run from the command line (and their users don’t necessarily even know they’re implemented in Python), web applications and other server-side things, and all manner of other use-cases. In those situations, it’s usually fine to use the existing import machinery.

Now, that sounds like something that can be pursued. Rather than seeing this as a Python problem, perhaps look at it as an IPython problem. IPython is already capable of doing many things that Python itself doesn’t, mostly using percent commands. It might well be that the best way to do this would be a magic command that would be something like “%load /some/file/name.py” and would, in effect, do this:

exec(open(“/some/file/name.py”, “rb”).read())

or the same thing but in a new module object, or similar.

I can guarantee you that isn’t true.

I’m not quite sure what you mean here, but if you mean update sys.path then you can already do that. Otherwise you could always construct your own function or finder to do imports the way you want. The import system is plenty flexible.

Not well documented where? The import system is thoroughly documented at 5. The import system — Python 3.12.0 documentation so you will have to be specific.

And the .ipynb file format has nothing to do with Python itself and is entirely a construct of the Jupyter project. If you have issue with how Jupyter sets up your Python environment then my suggestion is opening a feature request with them.

Once again, I can guarantee you that isn’t true, and I know both Brian and Fernando and they would never make that claim about .ipynb files either.

1 Like

I love Python, but would’ve never tried it without the need for data manipulation - that’s me, I know.
And it’s a great language and everyone knows it, just look at JavaScript memes vs Python memes.
But all I tried is to propose some other solution for importing a local file sitting anywhere on the machine and was trying to give a reason for it. Ignore my reasons if you don’t like them (those are just my personal opinions anyway), but please be open for a more inclusive solution towards newbies. Because the import logic is still too painful too many times.

When you make a proposal, you need to be prepared for feedback and for the answer to be “no”. Ideas should be backed up with concreted data, not “I think most people do this” or “I just know this would be helpful” or “but other language does it” since those statements on their own aren’t justification for implementing something.

It’s not a judgement of you or what you find useful, it’s a judgement of what is reasonable to implement and maintain. In this case, multiple people have explained that this would enable a pattern that we don’t want to encourage, as there is already a robust import system with complex details. They’ve also indicated it may be more appropriate in Jupyter instead of the core language.

3 Likes

Closing because the OP fell back to quoting more memes (deleted) instead of engaging with the topic and feedback, and the decision seemed to be “no” anyway.