Hey y’all,
type checkers complain about this:
from contextlib import suppress
def foo(x: int, y: int) -> float:
with suppress(ZeroDivisionError):
return x / y
and that’s great, that’s exactly what is expected 
Any calls to foo() with y=0 return None and the type checker will be happy as soon as we correct the return type of foo tofloat | None.
Type checkers know that because suppress.__exit__'s return type is bool.
Context managers that annotate their __exit__ as returning bool | None are assumed to be always non-suppressing–that’s a heuristic that works for most context managers and it’s fine.
Problem
Currently, all generator context managers (so context managers created from generators using the @contextmanager decorator) and generator context manager decorators (decorators from @contextmanager-based context managers) are assumed to be non-suppressing (__exit__() -> bool | None).
Therefore, all the cases below pass type checking, because of incorrect unreachability inference:
from collections.abc import Generator
from contextlib import contextmanager, suppress
@contextmanager
def suppressing1() -> Generator[None]:
with suppress(ZeroDivisionError):
yield
def foo() -> float:
with suppressing1():
return 1 / 0
@suppressing1()
def bar() -> float:
return 1 / 0
@contextmanager
def suppressing2() -> Generator[None]:
try:
yield
except ZeroDivisionError:
pass
def foo2() -> float:
with suppressing2():
return 1 / 0
@suppressing2()
def bar2() -> float:
return 1 / 0
@contextmanager
def suppressing3() -> Generator[None]:
with suppressing2():
yield
def foo3() -> float:
with suppressing3():
return 1 / 0
@suppressing3()
def bar3() -> float:
return 1 / 0
assert (
foo() is None
and bar() is None
and foo2() is None
and bar2() is None
and foo3() is None
and bar3() is None
)
The passing assertion at the bottom makes it visible that all float-annotated functions in fact return None.
A fix is needed for generator context managers and generator context manager decorators.
Proposal
- Change typeshed to allow specifying
__exit__return types using type variables- Done in Make `__exit__` return types of generator context managers specifiable by bswck · Pull Request #14439 · python/typeshed · GitHub along with my thought process
- Implement a PoC to analyze context managers around
yields in type checkers to infer the types bound to those type variables- Work in progress, any tips from mypy / pyright / ty / other type checkers folks are very welcome!
- Describe that in the typing specification
Technical overview (very brief and general)
The work in Make `__exit__` return types of generator context managers specifiable by bswck · Pull Request #14439 · python/typeshed · GitHub reveals errors in 2 packages, which I described in the PR comments. Overall, this patch seems to play out well with all packages checked by primer and is therefore review-ready.
What was changed in typeshed
Instead of relying on a type variable for carrying the function in context manager decorator, we’re using a ParamSpec and joining the return type with _ExcReturnT that should always be Never or None. We can specify the __exit__ return type of a generator context manager as well.
Inference rules
Type checkers need to treat bodies of generator context managers specially. I believe some sort of check has to be added that such a generator context manager has one yield in every path (which is a separate issue, as type checkers allow more than 2 yields in one branch as well! See also Special semantics for generator context managers).
Then, if a context manager has a “suppressing context” around the yield (either a try-except or another context manager with bool exit) it needs to infer that the _ExcReturnT is None and _ExitT_co is bool or, if the context around yield is not suppressing, _ExcReturnT is Never and _ExitT_co is bool | None (as default).
Questions
- Should this be a PEP?
- Do you think it’s a useful change?
- Do you have any concerns regarding that change?