Support different type for class variable and instance variable

Typing
1

After creating this feature request in pyright repo, i was told that this is not supported by python type spec and it’s best to discuss it here.

The problem

In django-modeltranslation, before we added types, we used class variable to populate instance variable with same name, but different type. And now this decision is biting our asses.

Here’s cleaned up code:

from typing import Any, ClassVar, Iterable

class TranslationField:
    ...

class FieldsAggregationMetaClass(type):
    fields: ClassVar[Iterable[str]]

    def __new__(cls, name: str, bases: tuple[type, ...], attrs: dict[str, Any]) -> type: ...

class TranslationOptions(metaclass=FieldsAggregationMetaClass):
    def __init__(self) -> None:
        self.fields: dict[str, set[TranslationField]] = {f: set() for f in self.fields}

class BookTranslationOptions(TranslationOptions):
    fields = ["name"]

This leads to error when assigning fields, because it’s being treated as instance variable, not class var.

Pyright:

error: Expression of type "list[str]" is incompatible with declared type "dict[str, set[TranslationField]]"
  "list[str]" is incompatible with "dict[str, set[TranslationField]]" (reportAssignmentType)

Mypy:

typing-test.py: note: In class "BookTranslationOptions":
typing-test.py:20: error: Incompatible types in assignment (expression has type "List[str]", base class
"TranslationOptions" defined the type as "Dict[str, Set[TranslationField]]")  [assignment]
        fields = ["name"]
                 ^~~~~~~~

Suggestion

It would be really nice, if on the class level fields was treated as class variable, allowing assignling a list. But on the instance level - it was using type from __init__ (or type from annotation without ClassVar.

If it’s possible at runtime, maybe it should be supported by typing spec? What do you think?

Any suggestions how to fix this by refactoring is also welcome. Probably, we should rewrite TranslationOptions and use self._fields instead, or something like this. But i really don’t want to rewrite all of this, and maybe just use Any as a type. It’s a library after all, and not a public interface.

2

How about cls.field_names?

3

Changing this is probably not realistic, since there’s too much ambiguity between when you are accessing an instance attribute and a class attribute, since class attributes can be accessed through the instance until they’re shadowed. So it seems more sensible to forbid dual-use in static type checking.

You could try some trickery using a descriptor:

from typing import Generic, TypeVar, overload

ClassT = TypeVar("ClassT")
InstanceT = TypeVar("InstanceT")

class ClassOrInstanceAttribute(Generic[ClassT, InstanceT]):
    def __init__(self, class_value: ClassT) -> None: ...
    @overload
    def __get__(self, obj: None, owner: type[object]) -> ClassT: ...
    @overload
    def __get__(self, obj: object, owner: type[object]) -> InstanceT: ...
    def __set__(self, obj: object, value: InstanceT) -> None: ...

The implementation of __init__, __get__ and __set__ are left as an exercise to the reader[1].

  1. if you don’t mind ignoring a type error on the base class you can also leave it as a pure Protocol with no implementation and just annotate the attribute with the corresponding type ↩︎

4

That would be changing public API of a library. We’d better be off by changing private code (e.q self.fields → self.fields_map, or something like that).

1 Like
5

Oh, that’s exactly what we need.

Here’s my current version (going with protocol). And, it works perfectly for mypy (actually no, it does not throw error when assigning, but revealed types is wrong), with pyright throwing an error on a line i care about. But that’s probably problem on the side of pyright, thanks a lot!

from __future__ import annotations

from typing import Any, Generic, Iterable, Protocol, TypeVar, overload
from typing_extensions import reveal_type

ClassT = TypeVar("ClassT")
InstanceT = TypeVar("InstanceT")


class ClassOrInstanceAttribute(Protocol, Generic[ClassT, InstanceT]):
    @overload
    def __get__(self, obj: None, owner: type) -> ClassT: ...
    @overload
    def __get__(self, obj: object, owner: type) -> InstanceT: ...
    def __set__(self, obj: object, value: InstanceT) -> None: ...


class TranslationField: ...


class FieldsAggregationMetaClass(type):
    fields: ClassOrInstanceAttribute[Iterable[str], dict[str, set[TranslationField]]]

    def __new__(cls, name: str, bases: tuple[type, ...], attrs: dict[str, Any]) -> type: ...


class TranslationOptions(metaclass=FieldsAggregationMetaClass):
    def __init__(self) -> None:
        ...
        # self.fields: dict[str, set[TranslationField]] = {f: set() for f in self.fields}


class BookTranslationOptions(TranslationOptions):
    fields = ["name"]  # This is a problematic line


reveal_type(BookTranslationOptions.fields)
reveal_type(BookTranslationOptions().fields)

Mypy:

types-test.py:10: error: Invariant type variable "ClassT" used in protocol where covariant one is expected  [misc]
    class ClassOrInstanceAttribute(Protocol, Generic[ClassT, InstanceT]):
    ^
types-test.py: note: In member "__new__" of class "FieldsAggregationMetaClass":
types-test.py:24: error: Missing return statement  [empty-body]
        def __new__(cls, name: str, bases: tuple[type, ...], attrs: dict[str, Any]) -> type: ...
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
types-test.py: note: At top level:
types-test.py:37: note: Revealed type is "typing.Iterable[builtins.str]"
types-test.py:38: note: Revealed type is "builtins.list[builtins.str]"

Pyright:

types-test.py:10:7 - warning: Type variable "ClassT" used in generic protocol "ClassOrInstanceAttribute" should be covariant (reportInvalidTypeVarUse)
types-test.py:34:14 - error: Expression of type "list[str]" is incompatible with declared type "ClassOrInstanceAttribute[Iterable[str], dict[str, set[TranslationField]]]"
    "list[str]" is incompatible with protocol "ClassOrInstanceAttribute[Iterable[str], dict[str, set[TranslationField]]]"
      "__get__" is not present
      "__set__" is not present (reportAssignmentType)
types-test.py:37:13 - information: Type of "BookTranslationOptions.fields" is "Iterable[str]"
types-test.py:38:13 - information: Type of "BookTranslationOptions().fields" is "dict[str, set[TranslationField]]"
1 error, 1 warning, 2 informations
6

For a generic protocol just inherit from Protocol[ClassT, InstanceT], inheriting from Generic is redundant, both base classes need the parameters if you inherit from both.

Also the covariant error pyright is throwing is technically correct, that’s my bad, just add covariant=True to the ClassT type var.

1 Like
7

That took care of some errors, but main problem still stands.

class BookTranslationOptions(TranslationOptions):
    fields = ["name"]

This throws an error in both mypy and pyright.

I’ve added __set__ overloads here, similar to __get__. (And removed covariant, as suggested by pyright after adding __set__.

from __future__ import annotations

from typing import Any, Iterable, Protocol, TypeVar, overload
from typing_extensions import reveal_type

ClassT = TypeVar("ClassT")
InstanceT = TypeVar("InstanceT")


class ClassOrInstanceAttribute(Protocol[ClassT, InstanceT]):
    @overload
    def __get__(self, obj: None, owner: type) -> ClassT: ...
    @overload
    def __get__(self, obj: object, owner: type) -> InstanceT: ...
    @overload
    def __set__(self, obj: None, value: ClassT) -> None: ...
    @overload
    def __set__(self, obj: object, value: InstanceT) -> None: ...


class TranslationField: ...


class FieldsAggregationMetaClass(type):

    def __new__(cls, name: str, bases: tuple[type, ...], attrs: dict[str, Any]) -> type:
        return super().__new__(cls, name, bases, attrs)


class TranslationOptions(metaclass=FieldsAggregationMetaClass):
    fields: ClassOrInstanceAttribute[Iterable[str], dict[str, set[TranslationField]]]

    def __init__(self) -> None:
        self.fields = {f: set() for f in self.fields}


class BookTranslationOptions(TranslationOptions):
    fields = ["name"]   # Problematic line


reveal_type(BookTranslationOptions.fields)
reveal_type(BookTranslationOptions().fields)

> mypy types-test.py
types-test.py: note: In class "BookTranslationOptions":
types-test.py:38: error: Incompatible types in assignment (expression has type "List[str]", base class
"TranslationOptions" defined the type as "ClassOrInstanceAttribute[Iterable[str], Dict[str, Set[TranslationField]]]")
[assignment]
        fields = ["name"]  # Problematic line
                 ^~~~~~~~
types-test.py: note: At top level:
types-test.py:41: error: Cannot determine type of "fields"  [has-type]
    reveal_type(BookTranslationOptions.fields)
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
types-test.py:41: note: Revealed type is "Any"
types-test.py:42: error: Cannot determine type of "fields"  [has-type]
    reveal_type(BookTranslationOptions().fields)
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
types-test.py:42: note: Revealed type is "Any"
Found 3 errors in 1 file (checked 1 source file)

> pyright types-test.py
/home/serg/src/django-modeltranslation/types-test.py
  /home/serg/src/django-modeltranslation/types-test.py:38:14 - error: Expression of type "list[str]" is incompatible with declared type "ClassOrInstanceAttribute[Iterable[str], dict[str, set[TranslationField]]]"
    "list[str]" is incompatible with protocol "ClassOrInstanceAttribute[Iterable[str], dict[str, set[TranslationField]]]"
      "__get__" is not present
      "__set__" is not present (reportAssignmentType)
  /home/serg/src/django-modeltranslation/types-test.py:41:13 - information: Type of "BookTranslationOptions.fields" is "Iterable[str]"
  /home/serg/src/django-modeltranslation/types-test.py:42:13 - information: Type of "BookTranslationOptions().fields" is "dict[str, set[TranslationField]]"
1 error, 0 warnings, 2 informations
8

Yeah that won’t work, descriptors only have a class level getter, the setter and deleter both only work at an instance level. So you’ll have to reapply the annotation at the class level assignment and ignore the assignment error to retain the descriptor.

If you want to be able to do it without errors then you’ll need to make it a real descriptor and create an instance with the class level values, i.e. something like this:

fields = ClassOrInstanceAttribute(["name"])
9

If reusing the same name is an implementation detail and not part of the public API, you can also only use the descriptor in your subclasses, although I’d recommend to just ignore the errors there for now with the intention to later clean it up and splitting it into two separate attributes.

1 Like
10

FWIW the code will probably be easier to understand if you rename the instance variable, so that’s a long-term maintenance win.

1 Like