Some improvements to function overloads

Typing
1

I’ve floated around some ideas about this previously in python/typing and a GitHub issue that was caused by another shortcoming of the current way we specify function overloads.

There are currently two major annoyances I’ve come across multiple times when writing overloads:

  1. Optional arguments that are both keyword and positional can get annoying to deal with very quickly because you often will find yourself writing the same overload twice (once for the positional version and once for the keyword version) because we’re not allowed to have optional arguments follow required ones (unless they are keyword-only).

  2. While it’s obvious that overloads should be matched in the order they are defined in, it doesn’t always make sense to use the first overload as the unsolved case (i.e. the overload that is picked when the type solver can’t pick a specific one). This is somewhat similar to an unsolved TypeVar, although type checkers seem to have decided that it’s better to pick one of the overloads than to try to carry around an unsolved overload. This will often result in false positives. Sometimes the simple solution is to change the order of the overloads, but for overlapping overloads this may not always be possible, so we’re forced to decrease the accuracy of our overloads and contend with false negatives instead.

I would be interested in exploring if we can come up with some small extensions to the current typing constructs that would solve the above issues (and any others we can come up with) with overloads and collect them in a new PEP.

For 1) I have suggested either re-using typing.Required to mark a parameter as faux-required in an overload, i.e. a type checking only marker that has no effect on the runtime behavior of the function, or to add something like a RequiredParameter sentinel value that can be set as the default and means the same thing.

@overload
def foo(x: int = ..., y: Required[int] = ..., z: int = ...) -> float: ...
@overload
def foo(x: int = ..., y: None = ..., z: int = ...) -> float | None: ...

For 2) I have suggested something like a @default_overload that behaves like a regular @overload but type checkers will default to this overload, rather than the first one, if they can’t solve the overload.

@overload
def foo(x: LiteralString) -> LiteralString: ...
@default_overload
def foo(x: str) -> str: ...
1 Like
2

@Jelle already mentioned it in the linked typing discussion, but I’d like to repeat it here for visibility: I previously suggested to allow non-default arguments after default arguments on python-dev, but there wasn’t much enthusiam: Mailman 3 [Python-Dev] Proposal: Allow non-default after default arguments - Python-Dev - python.org

I still think that this would be an easy usability improvements without any major downsides.

2 Likes
3

Overload resolution behavior is under-specified by the typing spec. Every type checker does it slightly differently. Despite my best efforts to replicate mypy’s overload behavior in pyright, there are edge cases where the behaviors deviate. Mypy’s behavior doesn’t even fully match its own documentation, and I’ve found edge cases where it seems to be internally inconsistent.

For reference, here is my attempt to document pyright’s overload behavior. As you can see, it’s way more complicated than “overloads should be matched in the order they are defined in”.

Before we consider modifying or extending overload behavior, I think we need to formally specify the overload resolution behavior. Inconsistent behavior across type checkers is painful for library authors and typeshed maintainers. IMO, consistency and deterministic behavior is significantly more important than succinctness.

If the proposed typing governance process is approved, this would be a good area for the new typing council to tackle. Once there’s a formal specification in place for the existing overload mechanism, we can then explore extensions and improvements to it.

16 Likes