Optional imports for optional dependencies

Ideas
1

Issue

It’s common practice for Python packages to define extras for optional features, e.g. pandas[excel]. When developing such packages, you cannot import the extra packages at top level — otherwise, a ModuleNotFoundError will be raised at import time on installations missing that extra.

Typical workarounds are:

  • Importing the dependency inside every function that needs it, or

  • Doing a top-level try/except and setting the module to None, later checking if mod is None inside functions to disable the optional feature.

If the same dependency is used in multiple places, this logic must be repeated everywhere.

There’s currently no clean language-level mechanism for this, even though optional extras are a well-established concept in Python packaging.

Proposition

Inspired by PEP 810 – Explicit lazy imports, I’d like to propose a similar construct:

optional import numpy as np

This would attempt to import the module, and if it fails with ModuleNotFoundError, assign a special sentinel object (e.g. an instance of MissingOptionalDependency). This object would raise a MissingOptionalDependencyError on any access — much like how lazy imports reify on PEP 810.

Example:

optional import numpy as np

def non_optional_func():
    return "foo"

def optional_func1():
    return np.random.random()

def optional_func2():
    return np.random.random() * 3

On an installation without numpy, you will be able to import and use non_optional_func. Only when calling optional_func1() or optional_func2() would the user get a clear MissingOptionalDependencyError explaining that numpy is not installed.

Benefits

  • Keeps optional imports at the top level, where imports naturally belong.

  • Reduces boilerplate and repetition of try/except or None checks.

  • Produces clearer error messages (MissingOptionalDependencyError vs. ModuleNotFoundErroror ‘NoneType’ object has no attribute ‘random’).

  • Provides a standardized, language-level way to express a very common pattern in Python packages.

Compatibility

This introduces new syntax that currently raises a SyntaxError, so it would not conflict with existing code.

5 Likes
2

So, to be clear,

import sys

optional import numpy

rest_of_module_here()

Is the same as:

import sys

lazy import numpy

try:
    rest_of_module_here()
except ModuleNotFoundError as e
    raise MissingOptionalDependencyError from e

If so, it does seem like a nice syntactic sugar! But not sure if it’s worth another soft keyword or not.

1 Like
3

The main feature missing from lazy import that would be useful in this situation is really just having a way to know if the import would succeed without having to complete the import (as discussed in the PEP 810 thread).

If I understood it the proposal here is just that compared to lazy import a different exception could be raised on reification. That doesn’t seem useful to me.

3 Likes
4

PEP 810 gives us the __lazy_import__ builtin, so we can build additional tooling around lazy modules.

class MissingPackageModule(types.ModuleType):
  def __init__(self, message: str):
    self._message = message

  def __getattr__(self, name: str):
    raise MissingOptionalDependencyError(self._message)

def optpkg(module_name: str) -> types.ModuleType:
  if importlib.util.find_spec(module_name):
    return __lazy_import__(module_name)
  else:
    return MissingPackageModule(f'Package {module_name} cannot be loaded')

np = optpkg('numpy')

We will probably do something similar in nibabel/nibabel/optpkg.py at master · nipy/nibabel · GitHub once we hit a minimum of Python 3.15.

3 Likes
5

I know it’s not a particularly satisfying answer but if there really are enough usages of the optional dependency that a try/except per function is too tedious then you could split the functions that use the optional dependency into a submodule that eagerly imports it with a try/except then lazy from ._numpy_bits import functions_that_need_numpy them back into the right namespace.

6

You can add a comment to the import line, and it will show up in the traceback:

lazy import numpy as np # optional
np

Traceback (most recent call last):
  File "//main.py", line 1, in <module>
    lazy import numpy as np # optional
ImportError: deferred import of 'numpy' raised an exception during resolution

Maybe we could somehow detect that a module is meant to be optional (like if it’s imported under a optional_<name> alias) and indicate that in the error message.

7

I think one issue here is that not every package may want to do the same thing on missing an optional dependency.

For example the optional dependency may be a compiled accelerator module which can replace a slower pure python component. I don’t think having it raise an error like this would be appropriate in this case.

Alternatively a package may want to make UI changes based on the presence of an optional package, and may want to do without triggering the import.

I think in both cases, checking with importlib.util.find_spec is probably the more appropriate approach.

from importlib.util import find_spec

# Package has an option to use a compiled accelerator
if find_spec("faster_json_package"):
    lazy import faster_json_package as json
else:
    lazy import json

# Replacement for the current try/except None check
if find_spec("numpy"):
    has_numpy = True
    lazy import numpy as np
else:
    has_numpy = False
    np = None

def build_ui():
    ...
    if has_numpy:  # using this doesn't trigger the import
        ... # enable numpy related UI options

Now this is slightly different from the current try/except because it only tells you if the module can be found not if the import would actually succeed, but the only way to completely check that is to import the module at which point this is no longer lazy.

2 Likes
8

No, not quite the same. The main difference is that when numpy is installed, it will be imported at import time — not lazily on first use. Also, there are some other considerations that make lazy unsuitable for optional dependencies as discussed in the PEP 810 thread.

If you think about it, optional imports and lazy imports are orthogonal (maybe combinable) use cases. You might not want your optional dependency to be loaded lazily. It feels like we’re trying to fit the “optional dependencies” problem into PEP 810’s goals, when it’s really addressing a different concern (I also tried).

Every suggestion on how to import optional dependencies (which I really appreciate) feels like a workaround to an issue. The variety of proposed patterns suggests that we don’t yet have a clear, standardized way to handle such a well-established concept as package extras in Python.

I fully understand the cost of introducing new syntax, and I agree we shouldn’t add syntax just to cover niche cases. I just don’t think optional dependencies are that niche — they’re already an integral part of Python’s packaging ecosystem, supported by pyproject.toml and installation tools. It feels natural that the language itself could provide a consistent mechanism to match that.

4 Likes
9

Okay, thanks for the clarification! That does actually seem quite useful, though I agree with @DavidCEllis that this does feel very implementation-specific, and that some combination of their and @effigies’ code would suffice as a helper function.

Some potential/current use cases of this specific paradigm (e.g. not updating UI, etc., but just raising an error on unsupported code paths) might be helpful to support the proposal.

10

To me, the variety of approaches shows that different projects want to handle the missing import differently. I don’t think of these as workarounds. To me they seem like “I have the tools to deal with this the way I want.”

Not only is new syntax a heavy cost for this change, but the exception MissingOptionalDependencyError doesn’t seem that different than ModuleNotFoundError. If you want a slightly different exception for this case, you have the tools to create one and raise it.

12 Likes
11

An even easier solution is to make a wrapper module, which is then lazy imported. For example:

optional_numpy.py

try:
    from numpy import *
except ImportError:
    raise OtherError("please install my_pkg[numpy] to use this operation")

other.py

lazy from .optional_numpy import array
2 Likes
12 
lazy import numpy

should raise an ImportError as the module is not installed. My understanding is that a lazy import still checks for the presence of the module and creates an object that links to the file without actually executing it until it’s used.

13

Lazy imports do not check for the presence of a module

You can try it in the reference implementation demo

>>> lazy import numpy
>>> 
>>> numpy
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ImportError: deferred import of 'numpy' raised an exception during resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
4 Likes
14

No, it’s not like that. Maybe I misguided you all by mentioning PEP 810 at all (my apologies🙏). The optional import I’m proposing here is not lazy — it does not defer the import to the first usage. It only defers the ImportError, allowing the import to remain at the top of the module (where imports belong) and enabling the usage of all functions that don’t require the optional dependency if it’s not installed.

I believe there are other concerns that make lazy unsuitable for optional dependencies, like the fact that laziness is not always guaranteed, or the fact that they may trigger reification in autocompleters. In the end, optional and lazy imports solve different problems as I try to explain here.

Many replies here are trying to address the issue by using lazy import:

All these approaches will make your package unimportable when the global lazy imports flag in the user’s environment is set to "none" (see). I think that lazy import is not the solution for optional dependencies. In fact, I believe that they are something complementary and I could even imagine combining them like optional lazy import.

3 Likes
15

Setting aside the discussion about lazy imports, which I addressed in the post above, I can still see that you are making some good points here.

Agree. I think this is a different kind of optional dependency: not about an extra feature, but about a different implementation. In those cases, using a try/except block to define the module one way or another is fine, since the code will work either way. If you really want to do this lazily, then yes — find_spec is probably the way to go.

This one I do consider an extra feature. Enabling or disabling these extras based on the current installation makes sense. Providing a standard syntax to handle “maybe-present” packages allows the coder to use them as needed without messy logic. For example, if MissingOptionalDependency had a special boolean-false behavior, you could write:

optional import numpy as np

def build_ui():
    ...
    if np:
        ...  # Your np-based UI tweaks here
    ...

In this case the result would be exactly the same as doing a try/except and assigning np to None on failure, but still there’s 4 lines versus 1. However, it’s not the (only) point of the proposal to save these 3 lines. With the None approach, you would need to check if the symbol is None and raise a proper error in every function that uses it, or otherwise risk getting a ‘NoneType’ object has no attribute ‘whatever’ error when calling a function that requires the optional dependency.

There’s also a different syntax involving or that was proposed in this old thread. Such syntax could also apply to the first use case of your post:

import faster_json_package or json as json

I haven’t explored all the implications of that approach (because I thought that optional was cleaner), but it could be another way to handle optional dependencies in general.

1 Like
16

It’s possible that I’m seeing optional dependencies from a biased perspective. For me, they have always been about providing extra functionality in a package, in which case I do see a potential benefit to the proposed syntax. But it’s possible I’m overlooking other common use cases, like the one mentioned by @DavidCEllis regarding alternative, higher-performance implementations. I’m definitely open to discussing other patterns and use cases for optional dependencies as well.

I can see that point. It’s true that MissingOptionalDependencyError is very similar to ModuleNotFoundError. It’s a subtle distinction, and if nothing else is built on top of it, it may not justify a new built-in exception. I wouldn’t mind if MissingOptionalDependency simply raised the standard ModuleNotFoundError instead.

17

If you wish to have a MissingOptionalDependency sentinel object that raises this error, you can define and use it instead of using None.

Something along these lines:

class MissingOptionalDependencyError(ModuleNotFoundError):
    pass

class MissingOptionalDependency:
    __slots__ = ["_module_name"]
    def __init__(self, module_name):
        self._module_name = module_name
    def __getattr__(self, name):
        raise MissingOptionalDependencyError(
            f"Optional dependency {self._module_name!r} not installed"
        )
    def __bool__(self):
        return False

try:
    import numpy as np
except ModuleNotFoundError:
    np = MissingOptionalDependency("numpy")
1 Like
18

Yep, that would do the job. Still, I think it would be nicer to have this as a language feature rather than having to implement it by yourself in every project, and in every module using optional dependencies, first import that MissingOptionalDependency from wherever you defined it and later do that try/except for each optional dependency.

# builtin imports
...

# external imports
...

# internal imports
...
from .optional_utils import MissingOptionalDependency

# then you can import optional dependencies with the try/catch
try:
    import numpy as np
except ModuleNotFound:
    np = MissingOptionalDependency("numpy")

# your module
...

While this works, that doesn’t look super pleasant to me. Just to be clear, I’m not looking for some way to import optional dependencies. Of course, there are plenty of ways to address the situation —people have been using them for a while now and managing their way one way or another. In my case, I’ve been using a small context-manager + decorator pattern that works like this:

from .utils import OptionalImports

with OptionalImports() as optional:
    import numpy as np

@optional
def my_optional_func():
    ...

And yes, that also does the trick. Just as importing the optional dependencies inside the functions, do the try / catch and assing None and many other paths you can go.

My point here is that none of them feels natural (to me). And I think it makes sense, since optional dependencies are a thing in Python, for the language to provide a clean, standard mechanism to maybe-import them within the Python conventions —at top level, and not inside any logic— and be treated as regular, well-defined MissingOptionalDependency objects when not available — not something that might be None or directly unbound.

For the sake of comparison, there were also tools to manage lazy imports, and you could write your own implementations, but the community decided in PEP 810 that it made sense to have it as a standard language feature: lazy import. The question here is whether something like optional would also make sense, or if people think it’s not worth it.

1 Like
19

I would be interested in seeing your implementation of OptionalImports. This conversation prompted me to try to write a context manager that injects a MetaPathFinder into sys.meta_path and I think I got a decent part of the way yesterday. I like that this approach allows type checkers to see normal import statements, which np = optional_package('numpy') doesn’t.

My implementation 
import sys
import importlib.util

def call(func):
    return func()

class MissingPackageError(AttributeError):
    pass

class MissingPackageModule(type(sys)):
    __slots__ = ("_module_name",)
    def __init__(self, module_name):
        super().__init__(module_name)
        self._module_name = module_name
    def __bool__(self):
        return False
    def __repr__(self):
        return f"<MissingPackageModule: {self._module_name}>"
    def _raise(self, attr):
        raise MissingPackageError(f"Optional package '{self._module_name}' is not installed.")
    def __getattr__(self, attr):
        self.__getattr__ = self._raise

@call
class MissingPackageLoader:
    __slots__ = ()
    def create_module(self, spec):
        return MissingPackageModule(spec.name)
    def exec_module(self, module):
        pass

@call
class optional_imports:
    __slots__ = ()
    def find_spec(self, fullname, path, target=None):
        return importlib.util.spec_from_loader(fullname, MissingPackageLoader)
    def __enter__(self):
        sys.meta_path.append(self)
    def __exit__(self, exc_type, exc_value, traceback):
        sys.meta_path.remove(self)

from optpkg import optional_imports

with optional_imports:
    import numpy as np
    import scipy.stats

I don’t think it’s worth it. I think it’s a use case worth having good tools to handle, but I don’t think we need additional syntax.

5 Likes
20

This is not the same as lazy imports that are expected to succeed but are just deferred. The reason is that there is not a single way to handle optional dependencies. Your preferred approach is that it is imported eagerly but if that fails then it works like a lazy import but fails later on when used. To me this does not match what I want to do with optional dependencies at all.

The fact that the dependency is optional strongly hints that it should be imported lazily. If the library/application can provide some functionality without that dependency then it would be better if it could do that without incurring the import overhead unconditionally.

Your suggested approach means that if the dependency is not available then it becomes a dummy object that will later raise an exception when used. This is also not what I would usually want to do. Rather if the optional dependency is not found then it will mean that we import some other implementation instead or we will want to know that it is not found and do something else different. The primary role of the line of code that attempts to import the optional dependency is that it answers the boolean question “do we have the dependency”. The proposed optional import statement here is designed for someone who does not want to answer that question and just wants the code to fail later if an attempt is made to used the dependency.

If this optional import statement was provided alongside lazy import then I would never use the optional import statement because it seems strictly inferior to using lazy import for the same purpose. The differences are that when using optional import:

  • If the dependency is available then we import it eagerly (a bad thing)
  • If not then when using it later a MissingOptionalDependency exception is raised instead of ImportError (not particularly useful)

The thing that is awkward to do right now for optional dependencies is just checking if the dependency is available without eagerly importing it but the suggested optional import statement here fails to provide that. Instead you can do:

import importlib.util

lazy import numpy as np
using_numpy = importlib.util.find_spec("numpy") is not None:

To me this is superior to the suggested optional import statement. One thing that could be improved is the non-obvious spelling of find_spec but if there were a function with a clearer name it would be fine:

from importlib import is_importable

lazy import numpy as np
using_numpy = is_importable("numpy")

There could be a way to combine those two things so that you have a function that returns np and the boolean, but then I think it is fine to just return None in that case.

def import_optional(name)
    try:
        return __import__(name)
    except ImportError:
        return None

np = import_optional('numpy')

An argument for having a central function for this in stdlib could be so that type checkers can special case it. While I can easily write that function I think I would have to use

if TYPE_CHECKING:
    import numpy as np
else:
    np = import_optional('numpy')

to get it so that a type checker knows what np.cos means. With a lazy import type checkers will understand it automatically (at least if numpy is installed at checking time).

6 Likes