Yet another `@overload` verbosity reduction idea

[ncoghlan (Alyssa Coghlan)]

(Some similarities to Make `@overload` less verbose here, but a simpler variation, since each overload is still required to list all the parameter names, it just doesn’t have to repeat the annotations if they have no bearing on how the overload resolves. This simplification means the proposal is already valid Python syntax, and hence would only require updates to static type checkers, allowing it to be used on all currently supported Python versions, rather than being restricted to 3.14+ or 3.15+)

For a current project, I needed to overload a bunch of APIs with several optional parameters, but only one of them influenced the returned type (if it’s None, you get Result[str], if it’s set you get Result[Mapping[str, Any]]).

The overload syntax allows default values to be omitted from the overload definitions by specifying them as ....

It would be nice to be able to eliminate duplicated parameter annotations in the same way: param_name: ... would mean “required parameter with the same annotation as the implementation function”, and param_name: ... = ... would mean “optional parameter with the same annotation and default value as the implementation function”.

That way overloads would only need to spell out the parameters that actually influence the result type:

@overload
def f(relevant_param: None, required: ..., optional: ... = ...) -> Result[str]: ...
@overload
def f(relevant_param: FieldSpec, required: ..., optional: ... = ...) -> Result[Mapping[str, Any]]: ...
def f(relevant_param: FieldSpec|None, required: SomeType, optional: SomeOtherType|None = None) -> Result[str] | Result[Mapping[str, Any]:
    ...

Rather than having to repeat all the annotations, even when they don’t affect the overload resolution:

@overload
def f(relevant_param: None, required: SomeType, optional: SomeOtherType|None = ...) -> Result[str]: ...
@overload
def f(relevant_param: FieldSpec, required: SomeType, optional: SomeOtherType|None = ...) -> Result[Mapping[str, Any]]: ...
def f(relevant_param: FieldSpec|None, required: SomeType, optional: SomeOtherType|None = None) -> Result[str] | Result[Mapping[str, Any]:
    ...

At runtime, the overloads are overwritten by the implementation definition anyway, so there wouldn’t be any change to runtime annotations here.

Type checkers are already able to detect overload inconsistencies with each other and with the implementation functions, so presumably they’re analysing them collectively and would be able to fill in the omitted annotations from the implementation function before continuing with the rest of their work.

Edit: as per @TeamSpen210’s comment below, to support type stubs and protocols, rather than the ... resolution coming specifically from the “implementation function”, it would instead need to come from something like the “implementation function, or the last listed overload when there is no implementation function provided”. This is a slightly narrower definition than Spencer suggested, but it aims to ensure the runtime annotations on protocol definitions aren’t allowed to contain an unresolved ellipsis (and similarly for implementation function annotations).

[TeamSpen210 (Spencer Brown)]

A slight wrinkle is definitions in type stubs or protocols, which lack an implementation giving nowhere to copy from. Perhaps instead the rule is it can come from anywhere, but there has to be only one definition. Probably also prohibit mixing different */** arg types since that’s also confusing.

[blhsing (Ben Hsing)]

@ncoghlan

(Some similarities to Make @overload less verbose here, but a simpler variation, since each overload is still required to list all the parameter names , it just doesn’t have to repeat the annotations if they have no bearing on how the overload resolves)

I’m not sure why the premise here is that each overload is required to list all the parameter names, when in the thread you linked to the proposal is specifically about using ... to omit parameter names that are already specified in the function implementation, which makes sense to me.

Can you clarify why you think omitting parameters that do not affect the resolution of an overload doesn’t work? Granted that proposal requires a syntax change, but given how much duplicate code that can help reduce especially from function overloads with long lists of parameters, and how much more readable it can make them, I think a syntax change may be justified.

[jamestwebber (James Webber)]

I assume the whole point is to avoid the discussion from that thread?

[blhsing (Ben Hsing)]

Yes, but it would be nice if it is clarified why it is presumed that “each overload is still required to list all the parameter names” in the OP.

[jamestwebber (James Webber)]

Again, I’m just basing this on context, but I’d say “Because this version reduces verbosity but doesn’t require a syntax change”.

It’s not “presumed”. The thing you’re quoting is just a brief summary of the full post.

[blhsing (Ben Hsing)]

It’s apparently a presumption because the OP uses the conjunction “since” before the premise.

Anyway, I get that this thread is about an idea that’s easier to implement. I just don’t think conceptually an overload is necessarily required to list all the parameter names.

EDIT: Ah, I’m sorry. I think I misread the OP, where the conjunction “since” is really about explaining why this is a simpler version but not about stating a fact.

[blhsing (Ben Hsing)]

@ncoghlan

@overload
def f(relevant_param: None, required: ..., optional: ... = ...) -> Result[str]: ...
@overload
def f(relevant_param: FieldSpec, required: ..., optional: ... = ...) -> Result[Mapping[str, Any]]: ...
def f(relevant_param: FieldSpec|None, required: SomeType, optional: SomeOtherType|None = None) -> Result[str] | Result[Mapping[str, Any]:
    ...

How about a middle ground, allowing an overload to skip inconsequential parameters by simply not listing them, and letting them resolved from the implementation definition? That is:

@overload
def f(relevant_param: None) -> Result[str]: ...
@overload
def f(relevant_param: FieldSpec) -> Result[Mapping[str, Any]]: ...
def f(relevant_param: FieldSpec|None, required: SomeType, optional: SomeOtherType|None = None) -> Result[str] | Result[Mapping[str, Any]:
    ...

The only catch is that if an inconsequential parameter happens to be positioned before a relevant one and is not keyword-only, then and only then it needs to be redundantly listed in the parameter list of an overload.

[ncoghlan (Alyssa Coghlan)]

@jamestwebber is correct, I’m aiming to avoid needing a syntax change. I’ll edit the first post to make that clearer.

The project with the repetitive overloads that would benefit from this shortcut supports back to Python 3.11, so something that only worked on 3.14+ or 3.15+ would take years to be beneficial. By contrast, my suggestion here would be valid syntax in all currently supported versions of Python.

[Daverball (David Salvisberg)]

That’s already how it works right now. You can use any signature you want in an overload, as long as all overload signatures are compatible with the signature of the implementation. If you’re suggesting that the omitted parameters should be back-filled using the implementation, so those overloads can still be matched, even when people explicitly use a parameter that wasn’t specified, that would be a breaking change, which changes the meaning of all existing overload definitions. So that’s a complete non-starter, that’s worse than a syntax change.

Also this naïve case, where the relevant parameter for overload resolution is the first parameter is really not the problematic case. Almost all the verbosity comes from the other cases. Especially since you will often find yourself defining two redundant overloads for specifying the positional call and the keyword call. Multiply that by the number of arguments that are relevant for the return type and you quickly end up with a big mess of overloads.

I would like to see something that lets you violate the rule about where optional parameters can occur, so you don’t end up with two overloads per positional or keyword parameter. I.e. it lets you write something like

@overload
def foo(x: int = ..., y: int) -> int: ...

Type checkers will understand that there’s only two valid ways to call that: foo(y=y) and foo(x, y), since you can’t skip positional parameters. Now unfortunately this is a syntax error, so we would need something like the following to express this in a backwards compatible way:

@overload
def foo(x: int = ..., y: Required[int] = ...) -> int: ...

Which is something I have proposed in the past and seems like an elegant re-use of an existing special form.

[blhsing (Ben Hsing)]

@Daverball

If you’re suggesting that the omitted parameters should be back-filled using the implementation, so those overloads can still be matched, even when people explicitly use a parameter that wasn’t specified, that would be a breaking change, which changes the meaning of all existing overload definitions. So that’s a complete non-starter, that’s worse than a syntax change.

We can maintain backwards compatibility by making the partial matching overloads a keyword-based option like:

@overload(partial=True)
def f(relevant_param: None) -> Result[str]: ...
@overload(partial=True)
def f(relevant_param: FieldSpec) -> Result[Mapping[str, Any]]: ...
def f(relevant_param: FieldSpec|None, required: SomeType, optional: SomeOtherType|None = None) -> Result[str] | Result[Mapping[str, Any]:
    ...

The goal here is to avoid redundantly listing parameters irrelevant to the overload being defined.

@Daverball

I would like to see something that lets you violate the rule about where optional parameters can occur, so you don’t end up with two overloads per positional or keyword parameter. I.e. it lets you write something like

@overload
def foo(x: int = ..., y: int) -> int: ...

Type checkers will understand that there’s only two valid ways to call that: foo(y=y) and foo(x, y), since you can’t skip positional parameters. Now unfortunately this is a syntax error, so we would need something like the following to express this in a backwards compatible way:

@overload
def foo(x: int = ..., y: Required[int] = ...) -> int: ...

That’s a good proposal for a rather different problem, and I don’t see the two conflicting with each other.

[Daverball (David Salvisberg)]

@blhsing

We can maintain backwards compatibility by making the partial matching overloads a keyword-based option like:

You would still need to use a backported overload from typing_extensions, since the current implementation can’t be parametrized, but that would work, yes.

@blhsing

That’s a good proposal for a rather different problem, and I don’t see the two conflicting with each other.

True, although as far as verbosity goes, I think having each overload contain the complete list of parameters that can be supplied, is more readable. Tooling can help with that, but everything you need a tool for, so it’s still readable, just ends up obfuscating the code base^[1], so the fewer things like that we have, the better, in my book.

So reducing the number of required overloads seems like a better place to invest our time. That being said @ncoghlan’s proposal is relatively well-balanced between readability and not repeating yourself as much.

Although I’d personally still repeat all the annotations and default values, since that way you can look at each overload individually and don’t need to look at multiple overloads at once to figure out what type or default value any given parameter has. Typeshed also uses default values redundantly in all overloads as a policy.

---

1. especially when viewing the code online on GitHub ↩︎

[hauntsaninja (Shantanu Jain)]

Folks interested in this should also take a look at Default argument with incompatible type discussing the current use of ... for default values on overloads.

[ncoghlan (Alyssa Coghlan)]

The work project where I ran into this problem has been published now, so I can replace my simplified example from above with a code snippet from the actual SDK:

    @overload
    def respond(
        self,
        history: Chat | ChatHistoryDataDict | str,
        *,
        response_format: Literal[None] = ...,
        config: LlmPredictionConfig | LlmPredictionConfigDict | None = ...,
        on_message: PredictionMessageCallback | None = ...,
        on_first_token: PredictionFirstTokenCallback | None = ...,
        on_prediction_fragment: PredictionFragmentCallback | None = ...,
        on_prompt_processing_progress: PromptProcessingCallback | None = ...,
    ) -> PredictionResult[str]: ...
    @overload
    def respond(
        self,
        history: Chat | ChatHistoryDataDict | str,
        *,
        response_format: Type[ModelSchema] | DictSchema = ...,
        config: LlmPredictionConfig | LlmPredictionConfigDict | None = ...,
        on_message: PredictionMessageCallback | None = ...,
        on_first_token: PredictionFirstTokenCallback | None = ...,
        on_prediction_fragment: PredictionFragmentCallback | None = ...,
        on_prompt_processing_progress: PromptProcessingCallback | None = ...,
    ) -> PredictionResult[DictObject]: ...
    @sdk_public_api()
    def respond(
        self,
        history: Chat | ChatHistoryDataDict | str,
        *,
        response_format: Type[ModelSchema] | DictSchema | None = None,
        config: LlmPredictionConfig | LlmPredictionConfigDict | None = None,
        on_message: PredictionMessageCallback | None = None,
        on_first_token: PredictionFirstTokenCallback | None = None,
        on_prediction_fragment: PredictionFragmentCallback | None = None,
        on_prompt_processing_progress: PromptProcessingCallback | None = None,
    ) -> PredictionResult[str] | PredictionResult[DictObject]:

And the same API with the invariant parts of the overload declarations replaced with ...:

    @overload
    def respond(
        self,
        history: ...,
        *,
        response_format: Literal[None] = ...,
        config: ... = ...,
        on_message: ... = ...,
        on_first_token: ... = ...,
        on_prediction_fragment: ... = ...,
        on_prompt_processing_progress: ... = ...,
    ) -> PredictionResult[str]: ...
    @overload
    def respond(
        self,
        history: ...,
        *,
        response_format: Type[ModelSchema] | DictSchema = ...,
        config: ... = ...,
        on_message: ... = ...,
        on_first_token: ... = ...,
        on_prediction_fragment: ... = ...,
        on_prompt_processing_progress: ... = ...,
    ) -> PredictionResult[DictObject]: ...
    @sdk_public_api()
    def respond(
        self,
        history: Chat | ChatHistoryDataDict | str,
        *,
        response_format: Type[ModelSchema] | DictSchema | None = None,
        config: LlmPredictionConfig | LlmPredictionConfigDict | None = None,
        on_message: PredictionMessageCallback | None = None,
        on_first_token: PredictionFirstTokenCallback | None = None,
        on_prediction_fragment: PredictionFragmentCallback | None = None,
        on_prompt_processing_progress: PromptProcessingCallback | None = None,
    ) -> PredictionResult[str] | PredictionResult[DictObject]: