Discussion: Optional class and protocol fields and methods

[yangdanny97 (Danny Y)]

In Optional class and protocol fields and methods · Issue #601 · python/typing · GitHub, I proposed some potential semantics for modeling non-required attributes. The goal is giving users a way to annotate attributes that may not be initialized or methods that may not be present on a class.

I’ve had some trouble trying to extend the semantics to support methods, so I’m cross-posting it here to get more feedback and see if anyone has some good ideas. I’d also like to hear opinions on whether the proposal in its current form is worth pursuing further.

To save you from scrolling through the whole thread, here is the proposal in one continuous text:

NotRequired

The NotRequired qualifier will be allowed for attribute annotations in classes and protocols.

Structural Typechecking

If a protocol has a NotRequired attribute, structural typechecking will depend on finality:

• If the class is final, it can omit the attribute, declare it normally, or declare it with NotRequired
• If the class is not final, it must declare the attribute normally or with NotRequired; the attribute may NOT be omitted

class Proto(Protocol):
    x: NotRequired[int]
class Class1:
    pass
@final
class Class2:
    pass
class Class3:
    x: NotRequired[int]
class Class4:
    x: int
a: Proto = Class1()  # not OK
b: Proto = Class2()  # OK
c: Proto = Class3()  # OK
d: Proto = Class4()  # OK

The requirement that implementing classes must declare the attribute as NotRequired (instead of allowing it to be omitted) is motivated by this example from @JelleZijlstra

class P(Protocol):
    a: int
    b: NotRequired[str]

def user(p: P) -> None:
    if hasattr(p, 'b'):
        print(len(p.b))

class A:
    a: int
    def __init__(self) -> None:
        self.a = 3

class B(A):
    b: int
    def __init__(self) -> None:
        super().__init__()
        self.b = 1

def accept_a(a: A) -> None:
    user(a) # OK, A has an attribute a of the right type and doesn't have an attribute b

accept_a(B()) # boom

@runtime_checkable

Attributes annotated with NotRequired in a runtime_checkable protocol will be skipped when checking an object at runtime. The current runtime checkable behavior only checks for the presence of an attribute with the right name without checking the type.

Overrides

Subclasses may only remove the NotRequired qualifier from an overridden attribute.

Final

NotRequired will not be compatible with the Final qualifier in the same type annotation, since attributes with the latter are required to be initialized so NotRequired wouldn’t do anything.

ReadOnly

PEP767 may introduce read-only attributes. Subclasses will be allowed to override ReadOnly attributes to remove NotRequired (making a non-required attribute required).

class Class1:
    x: ReadOnly[NotRequired[int]]
class Class2(Class1):
    x: ReadOnly[int]  # OK
class Class3(Class1):
    x: ReadOnly[NotRequired[int]]  # OK

Pattern Matching

Not-required attributes are not allowed to be used with __match_args__, and may not be matched in general.

Assignment/Deletion

There will be no changes to assignment or deletion behavior at runtime. For the purposes of typechecking, assignment to NotRequired attributes will work the same as attributes annotated without the qualifier.

Currently, despite all attributes being “required”, none of the major typecheckers prevent attributes from being deleted. This behavior will stay the same, and both regular and NotRequired attributes will be deletable.

Access

There will be no changes to access behavior at runtime. Typecheckers may error if the attribute is accessed without being narrowed (using a hasattrcall or assigning to it) or if getattris used without a default.

This would be similar to emitting errors for accessing non-required TypedDict keys, and narrowing would work the same way (Mypy and Pyre don’t support this kind of error/narrowing, but Pyright does).

Given the lack of standardization of the equivalent typechecking behavior for TypedDicts, I think we probably want to make this behavior optional for now.

class Class1:
    x: NotRequired[int]
c = Class1()
if hasattr(c, "x"):
    c.x  # OK
c.x  # not OK (optional)

Uninitialized Attribute Checks

Typecheckers including Mypy, Pyright, and Pyre, can check for uninitialized attributes, though this is generally a best-effort check and opt-in/experimental in some cases. Typecheckers should not raise an error if the uninitialized attribute is annotated with NotRequired.

Dataclasses

There are 2 options here:

1. Ban this type qualifier in dataclasses
2. Have some special handling based on inspecting the type, just like we do for ClassVar. We could create a new type UninitializedType that’s inhabited by a single value (similar to NoneType or EllipsisType). It wouldn’t have a use outside of dataclasses, but if it’s passed as an argument to the constructor or used as a default for a dataclass then the attribute could be left uninitialized, like in the below example:

@dataclass
class Foo:
    x: NotRequired[int]

# generated __init__ method would look like this

def __init__(self, x: int | dataclass.UninitializedType = dataclass.Uninitialized): ...

@not_required decorator for methods

NotRequired works decently for attributes, but modeling methods with an equivalent decorator is tricky. This is because methods declared in the class body are always bound in the class, even abstract ones, so there’s no way of declaring a type for a method without adding the method to the class.

We could just not allow this to be used with methods, but that would limit the expressiveness of the feature.

There is a way to have this decorator, but only have it work in a limited context, as follows:

1. The decorator will only work in Protocols (except in stubs where it can be used anywhere). When a concrete class inherits from a protocol, the decorator does nothing and the method will be present as usual.
2. In a normal file, the only way for a concrete class to match the protocol without declaring the method is for the class to be final. In a stub file, you can pair the decorator with @type_checking_only in any concrete class.

I have mixed feelings on this section since it’s pretty confusing to even write this out.

Effect for Users

When implementing all the required proposed features as described above (but none of the optional ones), this is what changes.

Compared to leaving the attribute unannotated:

• We know what type the attribute is when accessing/assigning (instead of giving an unknown attribute error/defaulting to Any)
• We can restrict the type of the attribute in a subclass

Compared to annotating the attribute with an unqualified type:

• No typechecker complaints on uninitialized attributes
• The presence of the qualifier documents that the attribute may be absent

Things the proposed semantics do NOT guarantee:

• Absence of NotRequired means that the attribute is present
• Attribute accesses to NotRequired attributes must be gated
• Banning deletion of required attributes

[mikeshardmind (Michael H)]

@yangdanny97

If the class is not final, it must declare the attribute normally or with NotRequired; the attribute may NOT be omitted

This line here seems problematic for structural subtyping and describing existing code.

There’s no reason to require this, absence isn’t an incompatibility with NotRequired. It should only prohibit the presence of an incompatible type

I’ll have to give the rest more thought. I generally think this is a good addition to the type system, but the specifics of it are going to be tricky to get to what’s needed from it to usefully cover even the data model and the accurate typing of object

[Liz (Elizabeth King)]

@mikeshardmind

This line here seems problematic for structural subtyping and describing existing code.

There’s no reason to require this, absence isn’t an incompatibility with NotRequired. It should only prohibit the presence of an incompatible type

class A:
    ...
class B(A):
    x: int = 1

class Proto(Protocol):
    x: NotRequired[str]

def mixing_nominal_and_structural_subtyping_sucks(boom: Proto):
    value = "ka" + getattr(boom, "x", "")

You’re still right that this massively reduces the utility of this, but it is necessary.

@yangdanny97

There will be no changes to access behavior at runtime. Typecheckers may error if the attribute is accessed without being narrowed (using a hasattrcall or assigning to it) or if getattris used without a default.

I would like to see this explicitly include try/except AttributeError as a valid means of accessing NotRequired attributes; If it is expected to have the attribute in most cases, this is the version that should be preferred for runtime behavior with exception handling being free in the “normal” path

[ostr00000]

I wonder how we could improve decorator for methods. We must meet the following constraints:

• to add specific properties to the whole function, currently in typing only decorators are used, so it should be decorator,
• we should not introduce runtime change,
• we cannot bind function to the real name (no way of declaring a type for a method without adding the method to the class)
• we cannot express methods as Callable - unspecified and inconsistent behavior between type-checkers

So I was thinking : Is there any precedent in a native Python where we bind a function to another name? Actually, yes - property.setter. In a runtime, there is of course a descriptor mechanism used, but we can only take from this example how to write such decorator.
Next, we must bind to some irrelevant name - for this purpose, it may be used any name, but I propose to use a _ character - it is frequently used to mark an irrelevant name.
Finally, we must still define a method name for type-checker - we may use here a property setter example - let’s define the method name as an attribute access on the decorator:

class ClassWithOptionalMethods:
    @not_required.function_name
    def _(self, a: int, /, b: str, *, c: str) -> int: ...

    @not_required.another_function
    def _(self) -> None: ...

(alternatively, maybe use a factory decorator with a string argument @not_required('function_name')?)

This solution allows us to define function:

• with parameter kinds (positional/keyword) and type annotations,
• function_name is not bound (although there may be leftover as function _),
• no runtime change required.

The only incompatibility I see here is when somebody already has a method named _ and the method is meaningful. But as a workaround, the already present method may be defined as the last one (some type-checkers (pyright) may report error if a method is defined twice, but it turns out that _ in current implementation is an exception to this rule!)

---

I am also afraid that

@Liz

this massively reduces the utility

When you add a new NotRequired attribute to Proto class, you also need to search all classes that are used in place of Proto argument to add this declaration. It is not scalable and probably will be introducing many unnecessary errors (runtime will not care about new, not required attributes, but type checker will raise an error).

I tried to come up with something smart here, but every time I ended up with the same solution as Danny proposed

[yangdanny97 (Danny Y)]

@Liz

I would like to see this explicitly include try/except AttributeError as a valid means of accessing NotRequired attributes; If it is expected to have the attribute in most cases, this is the version that should be preferred for runtime behavior with exception handling being free in the “normal” path

This might be pretty tricky to implement in a typechecker, but if it’s optional behavior and the default just leaves accesses unsafe that could be OK?

@ostr00000

It is not scalable and probably will be introducing many unnecessary errors

Yeah, I guess the problem is most classes aren’t final. If we were like Kotlin and classes were final by default then this change wouldn’t have as much friction.