Why do some (all?) type checkers discard `classmethod`/`staticmethod`/`property` instead of using the descriptor protocol?

Typing
1

In mypy when builtin decorators that implement the descriptor protocol are encountered, the function signature is modified and then the decorators are removed. Is there a specific reason why?

I’m guessing the answer to this question is “because mypy was created long before ParamSpec and others followed suit”, but is there any other reason? I assume the same is true for all other type checkers because the descriptor behavior of builtins isn’t accurate in typeshed.

This causes problems with nested descriptors, eg functools.cache, one other attempt to solve the functools.cache call signature proposed to allow self binding for generic ParamSpec, however unless there’s a way to distinguish between a binding callable and a nonbinding callable (the descriptor protocol?) it might cause as many problems as it solves.

I’ve got a very WIP fork of mypy where I’ve updated the builtin descriptor annotations to be (I think) accurate, then keep the appropriate decorators. I’ve got this working locally for a very basic example (inheritance breaks stuff, I need to promote inferred explicit types to Self, other things). This would be useful because for example it can be combined with changing functools.cache to use ParamSpec, and then correct binding behavior is out of the box. But have I missed something? It seems doable now that we have ParamSpec.

2

As you imply, this is likely simply because mypy’s handling of these decorators predates ParamSpec. I don’t think this behavior is mandated by the spec. If you can modify mypy in a way that handles these descriptors in a more generalized way, I’d personally welcome such a change.

1 Like
3

I’m not sure what you mean by “decorators are removed”. I presume that’s an internal mypy implementation detail? Could you provide a code sample that demonstrates the user-visible behavior that you’re seeing — and explain how that differs from what you’d like to see instead?

Keep in mind that the decorators classmethod and staticmethod need to be handled in a special manner because they affect the signature of the method they are decorating. In particular, they dictate whether the first parameter of the method is treated as self, cls, or neither. For all other decorators, the signature of the pre-decorated method must be established prior to applying the decorator. The resulting callable (and its full signature) is an argument to the decorator call.

The property decorator requires significant special-casing in a type checker because the property class was not defined as generic, but it acts like it’s generic. Attempts have been made to fix this, but this resulted in a variety of problems (worse error messages, backward compatibility issues, etc.), so these attempts were abandoned.

1 Like
4

In mypy the decorators are literally removed. EG in mypy:

elif refers_to_fullname(d, "builtins.classmethod"):
    removed.append(i)

and later

for i in reversed(removed):
    del dec.decorators[i]

Then special handling is applied elsewhere for accessing those classmethods/etc. This is fine in most cases. Until it’s combined with another decorator that doesn’t directly return the function type, but instead returns a class with a __call__ method. In that case the special handling in mypy breaks and a callable with the full original call signature is returned, even though at runtime the first cls parameter has been removed. To demonstrate expected behavior -

If the classmethod definition in typeshed is changed to:

_Class_co = TypeVar("_Class_co", bound=Callable[Concatenate[type[Any], ...], Any], covariant=True)
class classmethod(Generic[_Class_co]):
    @property
    def __func__(self) -> _Class_co: ...
    @property
    def __isabstractmethod__(self) -> bool: ...
    def __init__(self, f: _Class_co, /) -> None: ...
    @overload
    def __get__(self: classmethod[Callable[Concatenate[type[_T], _P], _R_co]], instance: None, owner: type[_T], /) -> types.MethodType[_P, _R_co, classmethod[_Class_co]]: ...
    @overload
    def __get__(self: classmethod[Callable[Concatenate[type[_T], _P], _R_co]], instance: _T, owner: type[_T] | None = None, /) -> types.MethodType[_P, _R_co, classmethod[_Class_co]]: ...
    if sys.version_info >= (3, 10):
        __name__: str
        __qualname__: str
        @property
        def __wrapped__(self) -> _Class_co: ...

Then typing.MethodType:

_W = TypeVar("_W")

@final
class MethodType(Generic[_P, _R_co, _W]):
    @property
    def __closure__(self) -> tuple[CellType, ...] | None: ...  # inherited from the added function
    @property
    def __code__(self) -> CodeType: ...  # inherited from the added function
    @property
    def __defaults__(self) -> tuple[Any, ...] | None: ...  # inherited from the added function
    @property
    def __func__(self) -> _W: ...
    @property
    def __self__(self) -> object: ...
    @property
    def __name__(self) -> str: ...  # inherited from the added function
    @property
    def __qualname__(self) -> str: ...  # inherited from the added function
    def __new__(cls, func: Callable[..., Any], obj: object, /) -> Self: ...
    def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _R_co: ...
    def __eq__(self, value: object, /) -> bool: ...
    def __hash__(self) -> int: ...
    def __get__(self, instance: object, owner: type[Any] | None = None,/) -> Self:...

Then what that says is that when a classmethod is in the type dictionary wrapping a callable (any callable - not just a function defined with def), when accessed via either the class or an instance, it returns a MethodType which has a call signature with one less parameter than the wrapped callable.

If functools._lru_cache_wrapper is changed to:

_C = TypeVar("_C", bound=Callable[..., Any])
_P = ParamSpec("_P")
_R_co = TypeVar("_R_co", covariant=True)
_Cache: TypeAlias = _lru_cache_wrapper[Callable[_P, _R_co]]
_MCache: TypeAlias = _lru_cache_wrapper[Callable[Concatenate[_T, _P], _R_co]]

@final
class _lru_cache_wrapper(Generic[_C]):
    __wrapped__: _C
    def __call__(self: _Cache[_P, _R_co], *args: _P.args, **kwargs: _P.kwargs) -> _R_co: ...
    def cache_info(self) -> _CacheInfo: ...
    def cache_clear(self) -> None: ...
    if sys.version_info >= (3, 9):
        def cache_parameters(self) -> _CacheParameters: ...

    def __copy__(self) -> Self: ...
    def __deepcopy__(self, memo: Any, /) -> Self: ...

    @overload
    def __get__(self, instance: None, owner: type[Any], /) -> Self: ...
    @overload
    def __get__(self: _MCache[_T, _P, _R_co], instance: _T, owner: type[_T] | None = None, /) -> types.MethodType[_P, _R_co, _lru_cache_wrapper[_C]]: ...

Then if it wraps a standard function it implements the same descriptor protocol as a function. It’s also callable, however and so if combined with classmethod:

class MyClass:
    @classmethod
    @cache
    def fn(cls, arg: int, arg2: str) -> str: ...

When the above fn is accessed through either an instance of the class or the type of the class it should return a Callable that takes an int and a str. Currently, mypy doesn’t implement this and I don’t think other type checkers do either (I’ve tried using pyright with the above annotations and didn’t get expected results).

If the above behavior was implemented this would mean that functools.cache wrapped functions would have correct call signatures (something that people keep asking for) as well as having no false positives for access to _lru_cache_wrapper.cache_clear, a strong requirement of the typeshed repo devs.

I may be wrong in some way (or many ways), please correct me if so, I have a very good understanding of the c++ type system (including variadic templates), but most of my understanding of the python type system is based on guesses that it works similarly to the c++ type system combined with reading documentation.

(edits - using the right class defs, apologies)

1 Like
5

I’ve corrected the quoted type definitions in the previous post as edits, sorry about that.

6

Mypy has special handling for attributes that implement the descriptor protocol, as such it’s possible to write a shim property-like class:

from dataclasses import dataclass
from typing import Callable, assert_type
@dataclass
class MyProperty[T, R]:
    fn: Callable[[T], R]
    def __call__(self, val: T) -> R:
        return self.fn(val)
    
    def __get__(self, instance: T, owner: type[T] | None) -> R:
        return self(instance)

class MyClass:
    @MyProperty
    def fn(self) -> int:
        return 1

my_class = MyClass()
assert_type(my_class.fn, int)

and the above type checks correctly in pyright and mypy (I can’t work out how to share a link to mypy online). No special handling needed for that property. It gets a bit more complicated when using property.setter etc. Which do need some special handling.

7

Being able to write a custom generic property isn’t the issue. The built-in property not being generic is.

I wrote a more complete example of a generic property (playground); pyright not reporting on __delete__ returning Never is somewhat surprising

8

A function is allowed to return Never, it implies that the function doesn’t return and doesn’t warrant an error from type checkers.

I personally prefer to allow the callabe fget/fset/fdel attributes to be fully generic types then constrain on usage of __get__, etc. This way if they are not trivial functions then the type interface doesn’t discard the additional attributes (eg if the getter is a ‘functools.cache’ wrapped function). And this can be type checked in pyright. I’ve used _ to get around pyright erroring on functions being shadowed, this would need to be special cased by type checkers, but it might be less special casing than currently while more consistent with runtime behavior.

9

What I meant is that the code down the line isn’t flagged as unreachable.

10

If you really want to abuse the type system you can also use this to give pseudo-structural type unions.

from typing import Never, reveal_type

class Foo:
    foo_attr: int
    def __getattr__(self, name: str) -> Never:
        ...
class Bar:
    bar_attr: int
    def __getattr__(self, name: str) -> Never:
        ...

def foo_bar(arg: Foo | Bar) -> None:
    reveal_type(arg.foo_attr) # revealed type is int, no error
    reveal_type(arg.bar_attr) # revealed type is int, no error
11

So pyright is implementing the standard and the standard says that Never is the bottom type, there isn’t really a way to specify a function that throws an exception, so from a standards perspective it doesn’t strictly mean that the function causes any following code to be unreachable.

12

I thought of that approach, but correctly constraining the callables would require HKT.

That’s clever, haven’t thought of that. Too bad del _id afterwards does not hide the attribute in the autocomplete suggestions

It’s less so about the function throwing an exception and more so about indicating this call should not be made.
It looks to me like an inconsistency in pyright that a direct call to Foo.id.__delete__(foo) does result in the following code being marked as unreachable, while del foo.id does not.

1 Like