An Issue with a Union of Callables

Hi,

I have a few functions that generate different types of files. Each of them has one of two signatures (they differ in the DocType parameter). I introduced a variable called handler (an attribute of a more complex object, actually) and I used the following annotation:

handler: Union[
    Callable[[Path, Entry, DocType, Settings], str],
    Callable[[Path, Entry, Settings], str]
]

However, when I call the function with four arguments, Pyright says it expected 3 positional arguments. If I call it with only three arguments, Pyright complains that it expected 1 more positional argument. In other words, it seems to choose the wrong branch every time.

The issue occurs regardless of whether I use an explicit Union, or the | operator.

Is the annotation just too complex for Pyright to handle, or have I missed something?

Thank you for your insights!

The type as shown is unusable from a typing perspective because it might be a function that must be called with 3 arguments or it might be a function that must be called with 4 arguments. The conclusion from a typechecker is then that there is no allowable number of arguments that can be passed to this object safely.

In your actual code you presumably have some way of knowing which kind of handler function you have somehow using some other variable or because the type is more specific in a subclass or something. A typechecker is not going to be able to understand this without somehow also having that other knowledge.

How does your code know how to call handler correctly?

It is determined at runtime. The functions have unique names and are not overloaded. However, they are assigned to the handler attribute, which makes their specific names irrelevant.

To be more specific, I have a class structured like this:

@dataclass
class Entry:
    pattern: str    # a glob (fnmatch) pattern
    handler: Callable[...]
    
entries: list[Entry]

Essentially, I am scanning the file system, matching file paths against the patterns, and applying the corresponding handlers.

The question is what should go inside the Callable[] argument list. Of course, the code works fine without type annotations, but I took it as a challenge (or exercise) to see if I can write statically correct code.

For the record, I also tried a solution based on the Protocol class:

class EntryGenerator(Protocol):
    @overload
    def __call__(self, path: Path, entry: Entry, doctype: DocType, settings: Settings) -> str: ...

    @overload
    def __call__(self, path: Path, entry: Entry, settings: Settings) -> str: ...

@dataclass
class Entry:
    pattern: str    # a glob (fnmatch) pattern
    handler: EntryGenerator
    
entries: list[Entry]

What is curious is that an error is reported when the individual functions are assigned to the handlers, but not at the place where they are called. At the call site, the static checker correctly allows either three or four arguments, refusing incorrect argument types.

You haven’t shown the call-site i.e. the code that the type checker complains about. No one will be able to understand what you mean if you don’t show this. What do you do with entries?

for entry in entries:
    # Somehow know how to call the function?

Make a self-contained runnable (including all imports etc) code demonstration that shows what you mean.

Correct. Consider this code:

f: handler = g

The assignment is fine whether g takes 3 or 4 arguments, but f doesn’t “remember” how many it actually takes. There’s no way to call f, because you don’t know if it needs 3 or 4 arguments, only that it doesn’t (for example) take 2 or 5 arguments.

If you turn the runtime mechanism that decides how to call the function into a type guard, static type checkers will also be able to follow the logic when type checking (they will be aware the handler takes 3 args on one branch after the guard check and 4 args on the other, and the guard definition tells them which branch is which)