Determine Python module location without `exec`? - `importlib.find_spec` `exec`s

Packaging
1

I am working with Python libraries and frameworks like TensorFlow and PyTorch, and doing various manipulations and inferences on their interfaces. Executing slows things down and causes compatibility issues. How do I find the module filepath?

Old version (that works but execs):

from os import path
from importlib.util import find_spec

def find_module_filepath(module_name, submodule_name):
    """
    Find module's file location without first importing it

    :param module_name: Module name, e.g., "cdd.tests"
    :type: ```str```

    :param submodule_name: Submodule name, e.g., "test_pure_utils"
    :type: ```str```

    :return: Module location
    :rpath: ```str```
    """
    module_spec = find_spec(module_name)
    assert module_spec is not None, "spec not found for {}".format(module_name)
    module_origin = module_spec.origin
    module_parent = path.dirname(module_origin)

    return next(
        filter(
            path.exists,
            (
                path.join(
                    module_parent, submodule_name, "__init__{}py".format(path.extsep)
                ),
                path.join(module_parent, "{}{}py".format(submodule_name, path.extsep)),
                path.join(
                    module_parent, submodule_name, "__init__{}py".format(path.extsep)
                ),
            ),
        ),
        module_origin,
    )

Specifically I am asking for how module resolution works; assuming that each module is already referencing an actual filesystem location.

Is it something like: first sys.path then PYTHONPATH then distutils.sysconfig.get_python_lib? - And do I just build a tree in memory using os.walk?

2

Is the answer always it is the standard site-packages folder?
Just wondering if you need to duplicate the import algorithm at all.

3

In the simple case, look in the Python install’s site-packages direcotry for a directory with the module name that you want.

For the general case, you need to use find_spec as you are. The whole import process is described at 5. The import system — Python 3.11.2 documentation but as it relies on whichever loaders are running at the time, it’s harder to emulate than to just run Python and do the actual import.

1 Like
4

(Which you can get via site.getsitepackages(), BTW)

5

Unless you’re trying to avoid running Python, which was one of the constraints :wink:

6

Maybe the OP can clarify; I wasn’t sure at first either but I interpreted “without exec” to mean without executing code/the import system/the module for each module, since they didn’t actually ask “without Python” as opposed to “without exec”, their example code is in Python and they were asking about specific Python functions they should use:

FWIW, import-ing site and a single call to site.getsitepackages() took on the order of 10 microseconds (with -n 1 -r 1, to avoid import or any other caching) for me on a Conda-based install with a long search path.