Protocol classes should not match `Callable[..., Proto]`

Typing
1

The following code type checks under mypy and pyright, but raises an error at runtime.

from typing import Callable, Protocol

class Proto(Protocol):
    def foo(self): ...

def foo(t: Callable[..., Proto]):
    t()

foo(Proto)  # TypeError: Protocols cannot be instantiated

Should we specify that type checkers should error here?

Similar thing for abstract classes (although the spec doesn’t yet really talk about abstract classes):

import abc
from typing import Callable, Protocol

class Abstract(abc.ABC):
    @abc.abstractmethod
    def foo(self): ...

def foo(t: Callable[..., Abstract]):
    t()

foo(Abstract)  # TypeError: Can't instantiate abstract class Abstract without an implementation for abstract method 'foo'

My secret goal is to make headway on T.__init__ unsoundness, by encouraging use of Callable[..., T] instead. A subgoal is preserving soundness on attempts to instantiate protocols or abstract classes. Some relevant discussion here

2 Likes
2

I don’t feel that strongly either way on this topic. I don’t think this is a source of bugs today, so I think that adding this rule would have little or no impact.

If we were to add this rule, I presume we’d need to carve out some special cases for protocol classes that define a __new__ and/or __init__ method?

3

The signature for foo looks reasonable to me. I don’t agree with the error. As has been discussed in many places, the type system can’t always know if it has that type or a subtype. There are valid subtypes of Abstract that in some cases the type system would only have a bound of type[Abstract]

I don’t think this is the right approach. We’ve seen more and more impacts of LSP exceptions (most recently, accurate typing and static error detection of copy.replace), I think we should be looking at ways to narrow these exceptions to the bare minimum that are caused by object and type and get user code that creates unnecessary issues fixed over time.

4

Quick and dirty mypy draft PR here: Disallow assignment of protocol or abstract classes to Callable by hauntsaninja · Pull Request #18347 · python/mypy · GitHub primer looks fine (if you ignore the prefect bug). Both hits are true positives in my opinion (though diagnostics need to be improved).

Hm, good question. IIRC Protocol.__new__ / __init__ failed at runtime prior to 3.11. Given that it seems it was added to make mixins work, I’d say we don’t need a carve out. This is similar to how pyright and mypy do not have a carve out for direct instantiation

To be clear, the request is for a call site error, not an error on definition of foo. The type checker can actually sort of know at the call site… The requested behaviour would be similar to the behaviour here, whose specification dates back to PEP 544. Note this specified behaviour means we actually do mostly know a x is not Proto where x: type[Proto], so we don’t have transitivity issues.

(The PEP 544 behaviour can get annoying when applied to abstract classes like mypy does, because there are several possible uses of type beyond instantiation. But as I mentioned in the thread I linked in OP, I think more use of Callable can help here)

Ah interesting. I thought of my goal as complementary.

__init__ LSP violations are endemic. If we can get to a point where --strict-type-instantiation is viable (that disallows instantiation of unknown non-final type[C] and assignment of unknown non-final type[C] to Callable[P, C]), maybe users can get soundness even if using the many libraries that have __init__ LSP violations. But this may not be viable (or may need intersections to be viable)!

(to avoid confusion for readers, my secret long term goal is technically a separate topic from OP)

1 Like
5

I’m a bit concerned that this is going to catch other things that should be valid, but I guess we can deal with that if it comes up that type checkers are emitting false positives as a result of this.

Yeah, and that’s a problem in it of itself. It creates more problems than just telling people to use **_kwargs: object if they want to allow subclasses to add kwargs (and other such things). If we were able to limit all of the exceptions to those that are part of the data model or from object/type, and model the exceptions so that they only apply to overriding a default implementation, we could get actual soundness of type I’ve been working on language towards this route with `__hash__`, `__eq__`, and LSP

I don’t see this as a goal worth pursuing, I think this is a configuration option that does more harm than good, much like those that flag Any/Unknown, they restrict valid code rather than gradually improve what is actually expressible and work towards making what exists possible to use soundly.

6

IMO, this would be an awesome step in the right direction.

For additional support, this problem is extensively discussed here in which many people don’t like how MyPy complains about abstract types not being passable (since MyPy assumes that any type can be instantiated). Separating callability from the concept of a type would eliminate this entire problem.

7

A Protocol can’t be instantiated, but the things that are assigned to it at runtime can:

import operator
from collections.abc import Iterable
from typing import Protocol

class CanPos[T, RT](Protocol):
    def __init__(self, x: T, /) -> None: ...
    def __pos__(self, /) -> RT: ...

def pos_map[T, RT](
    mapper: type[CanPos[T, RT]],
    things: Iterable[T], 
    /,
) -> Iterable[RT]:
    things_that_can_pos = map(mapper, things)
    return map(operator.pos, things_that_can_pos)

Here it’s valid to pass a type[int] to mapper:

>>> list(pos_map(int, ["1", "2"]))
[1, 2]