Using `Self` with classmethod and a `Self` class attribute

Typing
,
1

Hello,

I’m trying to type the following code, where a class has a class attribute containing a reference to itself (something like a Singleton for instance).

It would look like this:

from typing import Self

class MyClass:
    _singleton: Self | None = None

    @classmethod
    def new(cls) -> Self:
        if cls._singleton is None:
            cls._singleton = cls()

        return cls._singleton

The type checker complains with:

Incompatible return value type (got "MyClass", expected "Self")

… and I’m trying to figure out why exactly :slight_smile:

Reading the PEP for Self, is this code part of the “invalid” use of Self explained in the class Foo: example:

class Foo:
    def return_concrete_type(self) -> Self:
        return Foo()  # Rejected (see FooChild below for rationale)

Or is it a different problem?

In my example, if I change the return type of new to MyClass instead of Self, the type checker is much happier.

Thanks for your help!

Jonathan

2

Note that with just the given code it doesn’t really work and you will in fact only get a single MyClass instance even from all subclasses because the lookup of _singleton will recurse up the mro.

However, that’s not something the type checkers can figure out. MyPy and pyright disagree here, and I am inclined to believe that this is a bug in mypy and pyright is correct in letting this work directly.

Also note that you forgot ClassVar which is something I would have expected both type checkers to complain about.

1 Like
3

In case of subclassing, the behavior is interesting (at least to me :slight_smile: ):

  • If I create an instance from the parent class first, I will only ever get that instance even from subclasses.
  • If I create an instance from the child class, then the parent class, I get only:
    • The child instance when calling new() from the child class
    • The parent instance when calling new() from the parent class

I think in the context of where this actual code is taking from, it doesn’t matter too much though.

I also only tested using MyPy and didn’t check with pyright: indeed, pyright doesn’t complain about this construction… I’ll check if I can find something in MyPy source code that would raise this kind of error, otherwise I’ll open an issue there.

4

I also checked this one before posting, I forgot to add it back in my initial example, but like you said, it doesn’t make any difference in this case, although it is more correct.

5

mypy is correct to disallow this

Yeah, the problem with this can be shown pretty quickly:

>>> class A:
...     _x: Self | None = None
...
>>> class B(A):
...     ...
...
>>> A._x = A()
>>> B._x
<__main__.A object at 0x000001A9063879E0>

You can’t have something that is assignable as a classvar annotated with a type of Self safely.

If you’re interested in this solely for singletons, the type-safe handling for this would look something like:

class SingletonMeta(type):

    _instances: dict[type, Any] = {}

    def __call__(cls, *args: Any, **kwargs: Any):

        if cls not in cls._instances:
            instance = super().__call__(*args, **kwargs)
            cls._instances[cls] = instance
        return cls._instances[cls]

class MyClass(metaclass=SingletonMeta):
    pass

This also avoids users bypassing it by not using the MyClass.new() classmethod, but this doesn’t help for other possible uses of classvar assignment.

6

I don’t think so.

An alternative solution that doesn’t involve introducing the mess that is metaclasses is this:

from typing import Self, ClassVar

class SingletonBase:
    _singleton: ClassVar[Self | None] = None

    def __init_subclass__(cls, **kwargs):
        cls._singleton = None

    @classmethod
    def new(cls) -> Self:
        if cls._singleton is None:
            cls._singleton = cls()

        return cls._singleton

This works perfectly at runtime. (and instead of new we could use __new__ to prevent people from bypassing it, at the cost of __init__ being called each time)

7

mypy correctly prevents the problem shown immediately after I said that, this actually has come up before though, and it’s less cut and dry because mypy is doing the right thing for type safety, pyright is doing the right thing for specification…

unfortunately, the specification both allows this and specifies something unsafe here

discussion didn’t really go anywhere at the time, with objections such as:

The specification unfortunately very frequently picks “easier for users to write, but with an element of type unsafety” over “type safe”, and it seems like every time I point this out, people are fine with a type system that is intentionally not providing type safety.

8

There’s no mess to be found with metaclasses. people shying away from their use when they have exactly the kinds of situations they exist to solve isn’t helpful.

9

And mypy is therefore telling me that perfectly functional and safe code has a type error that it just isn’t smart enough to figure out doesn’t exists and gives confusing error messages.

yes, there is: Reduce metaclass conflicts with less eager proclamation, automatic metaclass merges

Not sure if you saw these discussions, but the problem is that metaclasses don’t compose, so it’s best to avoid them as much as possible if you want your code to be modular and reusable. (it’s not possible for your singletons to also subclass any ABC for example, at least not without extra effort.)

2 Likes
10

I don’t see that as an issue with metaclasses but with people having more complex inheritance structure than they need, and creating problems for themselves without wanting to have also to resolve the metaclasses. (you can create a metaclass with the merged behaviors of multiple metaclasses if you actually need this).

I also see subclassing ABCs as a mistake in typed code, protocols and a separate base implementation serve the same purpose with less complexity and while avoiding an unnecessary dependency on a specific implementation (playing nicer with duck typing as well), incidentally, the better option also leaves the metaclass free for things where it actually is doing something involved with initializing instances and types, meaning that it’s less likely to require manual resolution.

11

The specification was very much written to support gradual typing and adding types to previously untyped code, which was not necessarily written to be typing friendly. Python is a dynamic language, and total type safety is an unrealistic goal and detrimental to the Python language overall.

4 Likes
12

The specification appears to not match runtime behavior here, so I think it’s incorrect to chalk this up to gradual typing. I agree with you that python is very dynamic and not everything is going to be type safe. But I also agree with Michael that type hints when accurate should be type-safe, and that’s something that I’ve seen people push back on too, even in cases where it isn’t due to gradual typing. In other words, the unsafe parts should be limited to where we haven’t fully expressed type information because we can’t.

13

Let me just point out that I was not commenting on the topic in this thread. I was replying to the out of context quote of my (imagined) “objection” from another thread.

I mostly see people in the typing community trying to find a practical way to reconcile the various goals and wishes – of which type safety is an important one, but not the only one. This means pushing back on new ideas from time to time. On the other hand, I lately see an uptick of quite vocal comments about how typing is “wrong” and how things should be done, instead. And while those comments often have merit, they often fail to fully understand that there are multiple – often conflicting – goals that need to be juggled. And their demanding tone is often one reason why I tune out those discussions.

7 Likes
14

Gradual typing is not the same as outright lying about the type of something and not handling detectable type errors. The latter creates unresolvable situations to complicate future matters, the former is “Okay, we don’t have a good way to type this, lets leave it as Any until we do”.

I haven’t seen anyone argue for total type safety, that’s clearly not possible. The argument I’ve personally made is that for things that are typed, people should have type safety. Little lies throughout the type system and typeshed undermine this from even being possible.

2 Likes