Enforcing __init__ signature when implementing it as an abstractmethod

Typing
1

Hello. I noticed that Pyright doesn’t check the signatures of abstractmethod implementations when the abstractmethod is __init__(). Here’s an example:

from abc import ABC, abstractmethod

class AbstractA(ABC):
    @abstractmethod
    def __init__(self, x: int, y: int):
        pass

    @abstractmethod
    def do_something(self, z: int, u: int):
        pass
        
class RealA(AbstractA):
    def __init__(self, x: int):  ## No static type checker error
        self.x = x

    def do_something(self, z: int):  ## Static type checker error
        print(f"I hope I don't need anything other than {z}")
        

inst = RealA(3)  ## No type checker errors for missing/bad method implementations

Here, Pyright gives me an error at the definition of RealA.do_something() as expected (“overrides AbstractA in an incompatible manner”), but I get no such error for RealA.__init__() even though it is missing y in its signature.

I don’t think this is a Pyright bug, at least not an accidental one. The developer addresses this issue here (for Protocol rather than ABC, but I think it should be similar enough). I’ll admit that I haven’t taken the deep dive into the spec to decide if the Pyright dev is correct that there’s too much uncertainty to address this issue at present, but I’ll yield to their expertise and accept that premise for my question.

Is it possible either now or in the future to write AbstractA in such a way that it enforces a minimum signature for construction? I can’t think of a way around this that doesn’t involve trusting the author of RealA to remember to call super().__init__() in order to make sure that the correct type checker errors are thrown, and if the author does forget to add super().__init__(), the runtime errors (“RealA has no attribute y”) aren’t immediately intuitive, especially if AbstractA is part of an installed package that the author of RealA didn’t write themselves.

This issue could also be avoided by implementing __init__() in AbstractA and having all implementations use it implicitly, but in my particular case, I came across this problem because I wanted to have implementation-specific keyword arguments for constructing RealA on top of a minimally required signature for constructing any AbstractA implementation.

Apologies if this is a duplicate topic, searches for various combinations of “__init__”, “signature”, “ABC”, etc. didn’t turn up any relevant posts.

2

A long time ago there was a decision not to type check __init__ methods since subclasses often override them by adding some needed init values for new attributes or similar. This is unsafe from a type theory pov, but a big enough ease of use advantage that it was deemed worth it. I think it’s pretty reasonable to say that this line of reasoning doesn’t really apply to protocols or abstract base classes, especially if the method is decorated with @abstractmethod. But since this is an underspecified part of the typing spec an effort to enforce this behaviour would probably have to be part of a larger proposal defining more precise behaviour for various __init__ and __new__ related topics.

AFAIK the best workaround right now is to create a class method that effectively becomes the init method and then always use that method. For example, something like this:

class AbstractBase(ABC):
    x: int
    y: int

    def __init__(self, **kwargs: object) -> None:
        for key, val in kwargs.values():
            setattr(self, key, val)

    @abstractmethod
    @classmethod
    def create(cls, x: int, y: int) -> Self: ...

If you (and users of your library) always use the create method instead of the usual class call, you get the same behaviour but type checkers won’t do the special treatment of __init__ with it.

5 Likes
3

Unfortunately __init__ isn’t sound under the current type system. Neither is __new__, use of type(self), generic bounds in a constructor, and much much more. The list of things actually grows with python versions.

I don’t think the classmethod solution above is a good idea, it’s yet another ugly wart caused by a bad decision years ago, and it’s writing non-idiomatic code to get a tool to be useful rather than fix the tool to just be useful. I think we need to fix this fundamentally in the type system.

for now, this will alert you for any incompatibilities without needing to subclass or end up needing to work around abc.ABCMeta as a metaclass.

Code sample in pyright playground

from typing import Protocol


class ProtocolA(Protocol):
    def __init__(self, x: int, y: int):  ...
    def do_something(self, z: int, u: int):  ...

class _RealA:
    def __init__(self, x: int):  pass
    def do_something(self, z: int, u: int):  ...

RealA: type[ProtocolA] = _RealA  # errors here
del _RealA  # optional

Use ProtocolA in type hints where expecting any instance of any implementation of it.

3 Likes
4

What exactly is unsound about __init__ and __new__?

Is the issue that type checkers don’t model the distinction between uninitialised and initialised types or that the type system cannot express the information that the type checkers need to make this well defined?

5

One of the problems as far as I can tell is variance. The spec mentions that function arguments behave contravariantly.

Any covariant attribute of a class would thus result in being invariant:

class Foo1: ...
class Foo2(Foo1): ...

class HasFoo1:
    foo: ReadOnly[Foo1]  # ReadOnly -> covariant
    def __init__(self, foo: Foo1):  # parameter -> contravariant
        self.foo = foo  # covariant & contravariant -> invariant

class HasFoo2(HasFoo1):
    foo: ReadOnly[Foo2]
    def __init__(self, foo: Foo2):
        self.foo = foo

cls: type[HasFoo1] = HasFoo2
cls(Foo1())
6

I’m confused by this example because there are no type parameters. What type is having variance with respect to what? Do there not need to be type parameters before we can speak of variance?

7

I think this can be more easily explained by just reasoning about how we expect a method to behave.

Given some object a where isinstance(a, A), what is the signature of a.__init__? You might think it’s compatible with A.__init__. It’s not. You could have some derived class B < A that has whatever signature it wants. if a = B(), then a.__init__ will have B.__init__'s signature.

This is unlike every other method that obeys LSP: those methods have to be compatible with parent class methods.

Essentially __init__ is acting more like a free (factory) function than like an initializer method. This causes havoc with expectations about how super().__init__ should work.

I agree that this was a design choice for convenience.

2 Likes
8

Neil covered most of it already above, but the fact that type checkers exempt certain methods from LSP means that any situation where you take a user provided type is very fragile, even if you specify what the type must comply with, because the type system ignores a lot of it, by (IMO faulty) design.

This isn’t just __init__ either, there’s a lot of places this comes up, and new ones like __replace__

9

It makes use of type objects unsound. See the following:

class A:
    def __init__(self, x: int): ...

class B(A):
    # in practice, very common to add params in a subclass
    def __init__(self, x: int, y: str): ...

def problem(a_t: type[A]) -> None:
    # problem(B) will error
    a_t(1)

def not_problem(a: A) -> None:
    # not_problem(B(0, "boom")) will error
    # pyright will happily let you do this
    # mypy is sound here and gives you:
    # Accessing "__init__" on an instance is unsound, since instance.__init__ could be from an incompatible subclass
    a.__init__(1)

The variance thing mentioned also poses a challenge. It’s similar in prevalence to the mutable attribute hole that mypy leaves unsound under default configuration: Error codes for optional checks - mypy 1.14.0 documentation

2 Likes
10

I disagree that mypy is more sound there. Well-written subclassing only has problems when you get to object.__init__, and we could narrow the exception to only apply to object, or even fix this at the language level by making object.__init__ take *args: object, **kwargs: object and discard them. The error mypy has on accessing a.__init__ is incorrect, because the use of that is actually sound if the rest of the program is sound. It’s a bandaid over the fact that mypy isn’t erroring for B.__init__'s definition.

11

But it’s not just the methods that have an implementation from object, it it was, the problem would be easy to fix. You’ve got an example of how to do it with your thread on hashability, and that one might be palatable to people. With constructors though, the problem is that people have been writing the unsound thing for years, even if you haven’t. They write constructors that change the arguments taken regularly, without having kwargs or something on a common base to absorb this.

There’s years of people doing their best to type their code and being told their typing is fine now relying on this rather than having been informed that they’ve designed something unsound, so now you’ve got libraries that were written with typing in mind that may pass strict type checking, but aren’t sound.

I know your opinion on this; I can hear the conversations we’ve had, “it’s already unsound whether type checkers catch it right now or not, informing the user of the unsoundness isn’t breaking them.” and “If they don’t care about the theoretical unsoundness, they can ignore the new error”, but I think we’re gonna need a typing 2.0 or for you* to continue working on a type safe python-like language that emits python, similar to TS. Even though I want to share your opinion on this, I know there are workplaces where static analysis isn’t used as just another tool, it’s mandated that it passes, and there’s now years of people in that situation sitting on a pile of problems that their specific use doesn’t crash currently at runtime.

*Or anyone else

2 Likes
12

Okay, I see. It is not just common to add parameters to subclass constructors but I think actually quite necessary in many situations. Other languages I have used with subclassing have all had this but the difference between Python and say Java/C++ is that Python has type objects. In those other languages the constructor is a function whose name is the exact class being constructed so there is no question of substitutability for constructors because you must always know exactly which class is being constructed when you call the constructor.

Since Python has type objects the question then is what is the type of the type objects themselves and then what exactly is the relationship between type[A] and type[B] if B is a subclass of A. I would say that type[T] is invariant in T because constructors and other class methods can and often must have different signatures in different classes. The unsoundness here comes from treating type[B] as a subtype of type[A]. There are of course many situations where it does make sense to think of type[B] as a subtype of type[A] but it does not hold in general.

1 Like
13

This isn’t the unsoundness. if type[B] isn’t a subtype of type[A] when B is a subtype of A, this breaks the entire concept of subtyping. The root cause is not enforcing LSP compatibility in some places.

I think this is only going to get worse the longer we don’t address it, and is probably the single largest complicator to even creating a roadmap to things people want like HKTs and generic type var bounds.

14

I agree with this, and with @Liz above. I think it would be a bad idea to aim to enforce LSP on __init__ or __new__, both practically and theoretically. From a theoretical standpoint, constructors are not part of the instance type at all, so instance typing does not need LSP on constructors to be sound; the only issue is with the type[] type. This was already discussed at length in another thread; as I said there, we should not let the “tail” of type[] wag the dog of instance typing here.

I don’t think that we should sacrifice type[] covariance here, though; I think the most promising path to soundness is to sacrifice type[] callability and use explicit callable types for that instead (and in future, intersections between callable types and type[] types.) So I think the direction @hauntsaninja is exploring here is a useful one.

(The above applies only to constructors, not to other methods that currently have LSP issues. In those other cases I think the right path forward is to try to stick to LSP.)

5 Likes
15

It doesn’t work to sacrifice type callability. Unlike the other languages compared to, you can pass around types and expect them to be constructable and compatible. There’s a lot of real world code that has the shape:

def foo[T: SomeBase](typ: type[T]) -> T:
    ...

specifically with DI patterns.

Beyond that, LSP exceptions are also already breaking more than just callability.

one of the new exceptions people pushed for excluded the ability to properly type __replace__ and copy.replace in 3.13+ (mypy and pyright have already both decided to exclude this from compatibility checks) which means

def update_thing[T: Thing](x: T, payload: ChangeSet):
    n = copy.replace(x, **payload)   # crash here possible if LSP violation
    some_state.emit_event(x, n)
16

It would be painful, but far less painful than the other options mentioned here. At least there would still be a way to support the patterns you mention, via callable types.

Enforcing LSP on constructors is IMO a non-starter (its impact on the ecosystem would dwarf the impact of sacrificing type callability, by many orders of magnitude), so if sacrificing type callability is impractical, I think in practice that means sticking with the unsound status quo.

1 Like
17

So basically, the entire ecosystem is too unsound to actually have a sound type system and it’s not okay to inform people what they’ve written is unsound?

Why even bother with having a type system if that’s the approach?

Callable types here aren’t just painful, they aren’t correctly expressible currently, and we’re at least 2 typing features away from typing them.

We’d need intersections to have type[T] & Callable[..., T] and this doesn’t enforce the parameters from T.__call__, so we’d need type[T] & Callable.from_callable[T]

At which point we could just say type[T] means that, and type[B] isn’t assignable to type[A] if calling B isn’t compatible with calling A

18

It isn’t just a question of the callability of the type object. Many class methods besides __new__ are also essentially constructors that may need to have different parameters in subclasses for the same reasons. In general I don’t think that LSP necessarily applies to class methods in the sense that you can expect to be able to exchange a type object for a superclass with a type object for a subclass and call the class methods in the same way.

In practice I think that there is a distinction between class methods that are supposed to have a compatible interface across subclasses and class methods that are not. Maybe there should be a decorator or something to indicate this distinction.

19

I can see how this is useful for discussion purposes, so that we can e.g. describe how the current type[T] would be spelled in the new regime. But is the concern that there is enough existing code that needs both subclassing/instance-of and callability at the same time, and so we’d need these new features as a blocker to removing callability from type[T]?

I ask because I can easily think of code that would need either one or the other (and so in the new regime, they’d use type[T] or Callable[..., T], respectively). But I’m having trouble thinking of a situation where you’d need to use that intersection type. Your DI example, for instance, would be

def foo[T: SomeBase](typ: Callable[[], T]) -> T:
    return typ()

At a call site like foo(A), we’d have to determine whether that particular class A satisfies that signature, by looking at its metaclass’s __call__ and its __new__ and __init__. But foo doesn’t care about that, it just needs something callable that returns the correct type.

(I actually think that check would be easier if we introduce a new special form Only[T], which signifies an instance of T but not any subclasses. So like @final, but checked separately at each use, and not enforced globally for all uses of the class. The inferred type of A in the call above would be type[Only[A]], and type[Only[T]] would be callable—and therefore assignable to Callable[..., T].)

20

Only[T] doesn’t work with DI,

To be clear, I’m arguing that we shouldn’t remove callability from type, and that type[T] should imply type[T] is callable with the parameters expected by T, and using what we need to express that to point out the absurdity of needing that versus just making type[T] mean that.

Only[T] doesn’t work for DI at all, the user is expected to come with their own type.

That’s only if T isn’t expected to take any parameters, and without a construct for this other than callable, or the ability to extract the parameters from a Type, people have to duplicate the signature from T and not have it automatically “just work”.

If we imagine “CallableAs” to mean a Callable compatible with a given callable, is it possible to replace use of type? well… no, because that would mean we’re intentionally separating it such that given x = X(), type(x) doesn’t mean CallableAs[X] that breaks things like the below:

class Point[T: Coordinates]:
    def __init__(self, coords: T):
        self.coordinates: T = coords
    def __add__(self, other: Self) -> Self:
        return type(self)(self.coords + other.coords)

And if you look at it enough, you’ll find it basically dooms HKTs and generic type var bounds from having a reasonable interpretation in the future for similar reasons

I also have real world use cases that aren’t coverable in python’s type system. One of them involves recursive serialization with custom serialization types where the user needs to register a set of unique types and ids for those types. generating the id for the user is inappropriate because the ids should not be re-used for different types (versioned data) an the obvious idiomatic way here is a classvar for the id.

Arguably, my test suite covers this on the library side, and runtime validation is necessary anyhow, but it would be nice for the type system to work here and inform users (other developers) of issues before they hit runtime errors.