Conditional imports in stub files

[srittau (Sebastian Rittau)]

This came up during the transition from types-Pillow to inline annotations in Pillow. See Depend on Pillow instead of types-Pillow by srittau · Pull Request #11720 · python/typeshed · GitHub.

The Package python-xlib has a single function that takes an Pillow Image object. Now to annotate this correctly, we’d have to pull in the whole Pillow package when someone installs types-python-xlib, whether they are using Pillow or not. For now, we’ve worked around this by using Any, although I will look into using a protocol later.

Still, I wonder whether it would make sense to support some form of conditional imports in stub files. Something like this:

try:
    from PIL.Image import Image
except ImportError:
    type Image = Never

[Daverball (David Salvisberg)]

I’ve come across this myself when writing stubs for gevent where I used the exact same solution. There’s many packages with optional dependencies or even soft-dependencies where some of the functions have been designed to work with types from other libraries without ever importing that type from that library at runtime. In the latter case a Protocol is usually a good enough solution, but anywhere a type gets propagated, such as a return type or readable attribute, this solution is usually insufficient and you’re stuck with Any.

---

One potential concern I have with a feature like this is testability. It would force you to setup an increasing number of environments depending on how many optional dependencies you have, if you wanted to ensure that you aren’t introducing some kind of inconsistency when one or more of those dependencies are missing. It would probably need to come with more configuration options to mask out a given set of packages, so the type checker pretends they aren’t installed, similar to the Python version switches to test multiple Python versions.

[erictraut]

I don’t think that conditional imports are a good idea in stubs. I especially dislike the idea of using a try/except statement, which is dynamic in nature and cannot be evaluated statically.

I’ll also note that the code block above violates the current typing spec because type aliases cannot be redeclared or redefined on any code path.

In my opinion, stub and library authors should decide how they want a symbol to be defined for the purposes of static type analysis. If a symbol may be imported from different sources (e.g. different versions of an import library) or fail to be imported at runtime, the stub author should pick their preferred import source and tell the static analyzer in an unambiguous manner to use that source.

If an import cannot be resolved at static analysis time, a type checker will default to Any for that symbol, which is a reasonable fallback. In the above example, the from PIL.Image import Image statement is all that’s needed. Adding a second definition for the Image symbol only creates ambiguity.

[Daverball (David Salvisberg)]

@erictraut

I especially dislike the idea of using a try/except statement, which is dynamic in nature and cannot be evaluated statically.

I’ll have to agree with you on that. As soon as you enclose more than one import spread across multiple statements it becomes difficult or even impossible to ensure you are modelling the correct behavior. I would prefer something similar in structure to sys.version_info / sys.platform checks.

There’s a proposal from five years ago that may have easier to justify semantics, although it never really went anywhere due to the lack of motivating use-cases.

Conditional imports in one line using `else`

Conditional imports are somewhat common in code that supports multiple platforms, or code that supports additional functionality when some extra bonus modules are available. Today this is often written as: # pylint: disable=import-not-at-top try: from monty.python.au import bruce else: bruce = None Four lines of code, plus a potential linter directive asking tooling not to whine about logic before imports or non top level imports. What if we could simply write any variant of the followin…

But we could provide a simple function in typing_extensions with the same semantics that will later get added to sys or typing or wherever feels most appropriate, to implement a static import with a fallback other than Any. This could just be a simple wrapper for __import__ at runtime.

Alternatively it could be a static check, that doesn’t perform the import, it just checks whether it would raise an ImportError. This would be the most flexible and most closely mirror the version/platform switches, since this allows you to execute arbitrarily complex code in the two cases. Although there would probably be some special cases where this would break unless the function performed the actual import and just skipped the step where it put it into the local namespace.

Chameleon for example provides two translate functions in their i18n module, but one of them only exists if you happen to have zope.i18n installed. Any does not seem like a good fallback in this case, in fact any fallback would be wrong, the symbol should just be missing. Although to be fair this function has since been deprecated, due to conflicting expectations of zope’s API. This is just an example I have seen more recently. I’ve definitely come across more examples that do similar things, but the most common case is probably that one of the modules will fail to import completely at runtime, because it is only meant to be used once an optional dependency has been installed, which is a little less bad than the other cases, since it is more likely to be caught during development, this also can’t be fully modelled with any of the proposed solutions.

---

But to get back to the crux of the matter: This is about providing the most accurate types in the presence of optional dependencies without polluting people’s environments with those same dependencies or create a bunch of noise if they don’t have follow_imports=silent enabled. You will have to do weird stuff like type: ignore[import-not-found,unused-ignore] to shut up the type checker in both cases^[1].

If we can find another way to improve the situation here, I would be happy enough. Maybe something as simple as a type: optional-import comment to clue the type checker into shutting up if the module is missing completely^[2]. We could even consider changing the fallback to Never in this specific case, since we have explicitly marked the import as optional, so the potential for a false positive is much lower.

---

1. and I’m not even sure that works, I’ve tried to come up with a combination of ignores in the past that works with typeshed, but I haven’t found one that actually works for both mypy and pyright ↩︎

2. it should still complain if the module is present but missing type information ↩︎

[mikeshardmind (Michael H)]

@erictraut

If an import cannot be resolved at static analysis time, a type checker will default to Any for that symbol, which is a reasonable fallback.

I don’t think it’s a reasonable fallback in any case where I have conditional imports.

Here’s a quick example, though not in a stub

try:
    import orjson
except ModuleNotFoundError:
    import json

    def _to_json(obj: Any) -> str:
        return json.dumps(obj, ensure_ascii=True)

    _from_json = json.loads
else:
     def _to_json(obj: Any) -> str:
        return orjson.dumps(obj).decode('utf-8')

    _from_json = orjson.loads

There are a lot of libraries in the python community that exist solely to do something faster, written in another language, that offer either an identical or near identical API. If I could only attribute one reason to python being as popular as it is, it would be the libraries.

Some of these libraries aren’t fully supported on every platform, conditional imports like this allow a cross-platform library to use other libraries when they are available.

This is extremely commonplace throughout the ecosystem, a common extra you’ll see for various libraries is “speed” or “speedups” or similar that even contain those optional dependencies.

a handful of others:

cchardet vs chardet
aiohttp conditionally using aiodns and Brotli when available
pandas vs polars
conditionally using hyperscan (Which has zero windows support)
conditionally using numba (which doesn’t work everywhere)