I think outside of a very small handful of libraries intentionally designed to have compatible APIs, you end up needing to do more to swap out providers for some functionality than just changing the import anyhow.
A recent example I personally wrote code for: if you are decompressing data that doesn’t encode size information, the behavior of zstandard (available on pypi) and compression.zstd (stdlib) is different enough that you’ll need to handle the difference[1].
As for packages that have parts of their api just not function without the “optional” dependencies, I find these packages to be more of an annoyance than if they had just put the optional functionality in a seperate library that depends on the core library + the 3rd party package. We don’t have anything in python like rust’s features where declaring the features reduces the api surface, and even if you do it dynamically at runtime yourself, that interacts extremely poorly with static analysis.
Wow, that’s way more interesting than my silly implementation. Mine simply catches the import error and returns a decorator that is meant to be used to decorate any optional feature. Since you asked, this is how it looks like:
My (silly) implementation
class MissingDependencyError(ImportError):
pass
class OptionalImports:
def __init__(self, group_name):
self.group_name = group_name
self.failing_dep = None
def __enter__(self):
return self.decorator
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_type is ModuleNotFoundError:
self.failing_dep = exc_val.name
return True
return False
def decorator(self, func):
if self.failing_dep:
def wrapper(*args, **kwargs):
raise MissingDependencyError(
f"Missing optional dependency '{self.failing_dep}'. To enable this feature, install the package with extra '{self.group_name}'."
)
return wrapper
return func
And this is how it’s supposed to be used:
import pandas as pd
from .utils.optional_deps import OptionalImports
with OptionalImports("plotting") as optional_plotting:
import matplotlib.pyplot as plt
import seaborn as sns
with OptionalImports("analytics") as optional_analytics:
import statsmodels.api as sm
df = pd.DataFrame({"x": [1, 2, 3], "y": [1, 4, 9]})
@optional_plotting
def func1():
sns.lineplot(df, x="x", y="y")
plt.show()
@optional_analytics
def func2():
X = sm.add_constant(df["x"])
model = sm.OLS(df["y"], X).fit()
return model.summary()
In my implementation, if the coder uses the optional dependency anywhere without using the decorator, then a NameError would raise on execution, which is not ideal. I like yours better, to be honest.
I share your perspective here. That’s one important point on why to look for a way to import optional deps using Python’s standard import mechanisms rather than implementing custom functions.
That’s a valid opinion, and I appreciate it. I think it would be coherent with the existence of optional dependencies to have a standard optional import mechanism, but that’s just my opinion and I can understand if community doesn’t feel that way. I would also understand if the community feels that optional dependencies are diverse enough that each package should handle them in its own way, and that a standard mechanism doesn’t apply – even though I don’t quite agreey[1].
I think that all optional dependencies have something in common, and it’s the fact that at some point in your code you’ll need to import them, and that import may fail. So I think any optional dep would benefit from an explicit kind of import statement that doesn’t raise the error, but handles it gracefully instead. ↩︎
You seem to favor using lazy imports over regular (eager) imports for optional dependencies, as if it should be the only reasonable way to go.
I don’t agree with that, honestly. For instance, when developing a web server, I’d prefer all imports to happen at launch time, rather than being imported on the first execution of some endpoint. I do see the value in lazy import —I really do, in some cases like CLIs it’s crystal clear—, but I just don’t think that everything should be lazy. Lazy and optional serve different purposes, and there should be room for optional dependencies to be loaded eagerly or lazily. More on that later .
In the first part, you mention that returning an object that raises an exception when called isn’t what you would usually want, while in the second you suggest assigning None to the module symbol. I’m trying to reconcile these two points.
You could create any logic you want with the value assigned to the module after optional import foo. If you want to do something when foo was not found, then if not foo: ... can do it. That would be pretty similar to your import_optional function, but with some differences:
You would be using standard import syntax which is nicer both for the coders and for type checkers.
In case you don’t check for that None and the dependency is called, the error would be something more understandable than ‘NoneType’ object has no attribute ‘whatever’.
AFAIK[1]lazy import just can’t be used as it is to import optional dependencies in the global scope. It’s not because of the fact that it’s not telling you before-hand whether the module can be imported or not, it’s because lazy import produces a potentially lazy import —that is, not guaranteed that to be always lazy. So if your package contains something like this:
And a user with global lazy imports flag set to "none" and without the optional numpyattempts to use it, your package is unimportable.
I think this is a strong enough argument to not use lazy import for optional dependencies.
I think you are exactly right with this point:
The statement lazy import is expected to succeed, but just deferred. So, AFAIK, you shouldn’t use it in places where it’s not expected to succeed.
If you want your optional dependency to be loaded lazily, which is something that makes sense in many cases, then I could image combining the two things and have optional lazy import foo[2] that will load foo lazily on first usage, but that late import will not fail and might return a MissingOptionalDependency('foo') instead. That would make your dependency to be loaded when you want, while robust against any global lazy imports flag (and other concerns mentioned in the PEP810 thread).
This won’t solve the main concern you are presenting here, though, which is knowing before-hand if that import will fail or not without actually importing it.
To me, that’s not the main issue with optional dependencies. My opinion is that the only fully reliable way to know if something is importable is to actually try importing it. You can use find_spec or other approaches if you want to avoid the import at some conditions, but ultimately you can’t guarantee that another ImportError won’t occur. So no, this proposal doesn’t address that specific concern, and I would suggest continuing to use find_spec if you need to check whether the import is attemptable. I understand if that’s not ideal for your case.
Then, whenever you use the optional dependency, check first for if it’s None and if it is, respond appropriately (an error message to a log somewhere, a failed web request, whatever is the right thing for your application).
You seem to be assuming that raising an (uncaught?) exception is the only possible appropriate behaviour if the optional dependency gets used when not available. In my experience, that’s the least user friendly and maintainable option.
To be honest, I think you have a very unusual view of what “optional dependencies” are.
True. And if you need that level of certainty, you have it - and ImportError is the language standard way of reporting that the name isn’t importable. Once you know that, you can do whatever you want to deal with that fact. And this is where you part company with everyone else, because you seem to be unable to accept that people might want to do different things, and you’re focusing solely on the case where you want to have the name exist, but be bound to an object that raises a new exception on any use. That’s a perfectly valid desire (although as I say, it’s not one I’ve ever seen myself in applications I’ve worked with) but it’s only one of many, and as such doesn’t deserve to be given special status in the form of a language keyword.
Ok, I can accept that. I appreciate all the comments and clarifications. I’ll take some time to think about these points and won’t be adding further to this thread.
I agree that a home-made optional import mechanism can be easily implemented, but I also agree that it feels weird that we have optional extra dependencies but no official way to do optional imports. In my opinion, having a clear syntax for doing this provides a similar level of value as `lazy import` provides.
I don’t agree with this approach. Lots of things in Python are try-expect wrappers and lots of things can be implemented around try-except - bypassing existing high level mechanisms. I don’t think “this can be handled via try - except “ is a valid reasoning to disagree with a proposal - and if we follow that logic, big portion of what we have should be considered useless.
Yes, but again, that’s not the point. try-except being powerful is one thing, having a clear mechanism - syntax for a common need is another thing. Also, the original post already mentions that why the existing solution can and should be improved.
Same issue here.
From what I understood, all these arguments can be used to oppose to lazy imports too. You could even say
You can wrap next() with try-except and for is not needed. It is just a different version of the same thing.
Nobody’s suggesting that the proposal should be rejected just because it can be done already with try…except. But conversely, a proposal that offers nothing more than a different way to write some cases of try…except doesn’t provide anything like enough value to be worth pursuing.
The OP claimed try…except was a workaround for the lack of an “optional import” statement. The point being made here is that it’s not a workaround, it’s a perfectly valid way of using existing language features to get the behaviour you want. And because you’re composing existing general features, the try…except approach is far more flexible than the proposed new syntax.
The proposal here seems to be based on some sort of presumption that “optional imports” are a unified concept, that can be satisfied by a single feature which behaves in one particular way. That simply isn’t true - people have explained various different behaviours that they might want from an “optional import”, and no-one has explained why the one particular form that the OP is looking at is more deserving of dedicated syntax than any other.
@dg-pb, if you want to lazily import an optional dependency, you’ll need to decide what that means and how it is presented to your users.
I have no intention of lazily importing my optional dependencies without defining wrapper modules or otherwise thinking about how to control the behavior. Combining these ideas carelessly would be irresponsible. I am confident that other maintainers will also be thoughtful in how they adopt new syntax.
In that case, you did not understand my meaning.
You should compare any new proposal against what already exists, and ask “is it better, or is it just different.” My contention is that the proposal is not better than try-except.
It provides for none of the use-cases I cited, and is most closely equivalent to[1]…
Adding my two cents about optional imports as a user of optional and lazy imports, this is my first time posting in this forum.
I think that the current ways of handling optional imports are sometimes weird and unintuitive. I agree, that the try-from syntax is well suited for some use-cases, since it gives a lot of flexibility in handling the optionality of an import. But at the same time, using that syntax can become repetitive while also separates the optional imports from the other imports.
For me, the existence of the unified possibility to create optional dependencies in the pyproject.toml implies that optional imports should be also handled in a unified way. Currently, there are multiple use-cases for optional dependencies, and for most of these use-cases there are again multiple ways to implement them.
In general, I think having imports not at the top of a module is an antipattern. Lazy imports improve this situation, but there are use-cases with optional dependencies where it is necessary to have imports inside functions.
As a library developer, I wish there would be a unified way of handling optional imports.
I think that having an optional keyword could streamline these use-cases and remove code duplication across different modules, while making it easier to read and understand code which handles optional imports. This would make the addition of such a keyword better than existing solutions. So I created a (probably uncomplete) list of use-cases with example implementations and potential replacements using an optional.
I really liked @DavidCEllis idea of having a wrapper structure which fulfills different requirements, in my following examples I had his concept behind the syntax sugar of an optional keyword in mind.
Compatible APIs
try:
import orjson as json
except ImportError:
import json
could be replaced by something like
optional import orjson as json or import json
Logical dispatch for incompatible APIs
try:
from ._pyyaml import YAMLLoader
except ImportError:
from ._ruamel import YAMLLoader
could be replaced by something like
optional from ._pyyaml import YAMLLoader or from ._ruamel import YAMLLoader
Customized errors
If a library wants to tell the user of specific instructions on how to install an optional dependency, e.g. via extras.
Customized errors from ImportError at module-level
try:
import coverage
except ImportError as e:
raise MissingOptionalDependencyError("Loading the coverage plugin requires that you install the 'test' extra: pip install \"mylib[test]\"") from e
could be replaced by something like
optional import coverage
if not coverage:
raise MissingOptionalDependencyError("Loading the coverage plugin requires that you install the 'test' extra: pip install \"mylib[test]\"") from coverage
If coverage can’t be imported, deriving an Error from it could point to the underlying ImportError.
Customized errors from ImportError in function (lazy import)
def foo():
try:
import coverage
except ImportError as e:
raise MissingOptionalDependencyError("Loading the coverage plugin requires that you install the 'test' extra: pip install \"mylib[test]\"") from e
could be replaced by something like
optional lazy import coverage
def foo():
if not coverage:
raise MissingOptionalDependencyError("Loading the coverage plugin requires that you install the 'test' extra: pip install \"mylib[test]\"") from coverage
Similar as above to derive from the ImportError.
Eager import with runtime checks
try:
import cupy as cp
except:
cp = None
def foo():
if cp is not None:
...
else:
...
could be replaced by
optional import cupy as cp
def foo():
if cp:
...
else:
...
Lazy import with runtime checks
if find_spec("cupy"):
lazy import cupy as cp
else:
cp = None
def foo():
if cp is not None:
...
else:
...
could be replaced by
optional lazy import cupy as cp
def foo():
if cp:
...
else:
...
Which would result in the same code as in the eager case, just with the lazy keyword attached.
This conversation prompted me to try to write a context manager that injects a MetaPathFinder into sys.meta_path and I think I got a decent part of the way yesterday.
I really like this model, both for the clean syntax that results (complete with an obligatory indent to call out something different about the imports) and for how it plays well with type checkers.
My vote would be for something like this to be included in importlib rather than adding a new keyword.
A key conceptul difference between lazy import and optional import is that with lazy import, you still generally expect that module to be available, just that you don’t need it right now. And for the sake of import times you’re willing to pay for conceptual price of handling or propagating ModuleNotFoundError in every part of your code that reifies the import.
Whereas with optional import you write your code fully knowing that the module might not exist, and you will certainly do something differently depending on the availability of that module. This is the important part. For example, third-party modules that provide API-compatible[1] drop-in replacements for stdlib modules (such as regex, arrow, json, xml, etc.):
try:
import pcre2 as re
pattern = R"I like PCRE regex syntax so it goes here"
except ImportError:
try:
import regex as re
pattern = R"PyPI regex goes here"
except ImportError:
import re
pattern = R"Plain old boring builtin regular expression"
def foo():
....
# I don't have to worry about which pattern suits which regex engine
m = re.match(pattern, string)
In this case, only eager regular imports can suffice, since I’m using a different regex pattern to take advantage of pcre2 and regex over re.
Meanwhile, for use cases such as providing “optional” methods, moving the import statement into the method makes the most sense to me, avoiding any of the optional / global-eager-import pitfalls:
class SimpleMatrix:
"""The ndarray-related methods raise ImportError if numpy is not installed."""
def do_something(self, other):
# Method that doesn't need numpy.
# In fact, most users don't need numpy for our use cases.
...
@classmethod
def from_ndarray(cls, andy):
# This import statement is essentially free
# since the caller presumably has already imported numpy
# in order for it to get a hold of an ndarray.
import numpy
...
def to_ndarray(self):
# This import statement may take a while
import numpy
return numpy.ndarray(..., dtype=...)
In short, given the multitude of ways to customize the behavior of missing imports afforded by existing soslutions, lazy and optional imports for the sake of optional imports do not sound attractive to me.
They are a thing in Python applications, but I don’t see them as a ‘thing’ in the Python language or Python code. (Whereas I can see lazy imports as such.) So I don’t see language syntax as the proper solution. I can see something, possible plural, added to importlib as appropriate.
.