PEP 767: Annotating Read-Only Attributes

[mikeshardmind (Michael H)]

Issues with using negation for things of this nature have been discussed before, but even if negation ends up supported, the problem with it for API surfaces is that the type system is lossy on certain information, and negation would only hold for local narrowing: you’d have to reaffirm the lack of it to use it elsewhere.

[jamesdow21 (James Dow)]

I figured there’d be some reason it wouldn’t be that simple

It’s been a while since I’ve read through the discussions about them, but I remember there being a ton of nuance

[randolf-scholz]

I admit that Never, how it’s currently handled by type-checkers, is not a perfect fit, but that’s the old discussion whether we are allowed to witness an uninhabited type. My opinion on this is that we actually need to disentangle NoReturn and Never, and have type checkers error when an instance of the bottom type is produced by a function.^[1] I’ll edit my post to address this nuance.

We actually need a way to formalize not just the absence of a capability, but that adding that capability isn’t allowed for that type, not just that a certain call always raises.

This is exactly what the __setattr__ overload formulation achieves, because any subtype would need to have a compatible __setattr__. But since the return type is Never for that attribute, it must also be Never on the subtype. So the call must fail on the subtype as well.

---

1. That is, we need a true uninhabited bottom type such that type checkers error when a function produced an instance of that type. The original purpose of NoReturn is not really even a type. It was introduced to annotate functions that literally never return, and hence never actually produce an instance of the bottom type. ↩︎

[mikeshardmind (Michael H)]

Changing Never to behave that way isn’t feasible, discussion on that is happening elsewhere so I won’t repeat it here.

The Wrong special form also won’t fix it, for much of the same reasons.

[beauxq (Doug Hoskisson)]

@Eneg

@beauxq

I acknowledge the value of something that would say an attribute is read-only externally, while still being mutable internally.

But there is also a need to have covariant overrides of attributes.

These 2 cannot be the same thing safely, and should not be the same thing.

Is there a type theory basis that they cannot?

I optimistically assumed that they can, perhaps under some premise - for example, any methods that write to the attribute are required to be overridden (if the type is narrowed), though that alone is not enough.

If we could come up with some rules to make it safe, I think it would be complex and difficult to learn, and difficult for type-checkers to implement checks for.

Are we also going to ban super()?

    def indirect(self) -> None:
        super().rename("abc")

---

I also think internal mutability for the name ReadOnly is counter-intuitive.
Even if we made it safe and documented it well, there would be lots of bug reports to type-checkers saying “Why is it letting me change this ReadOnly variable?”
This is why people are trying to find a clear name for it here: Proposal: `PrivateSet[T]` — a type hint for externally read-only, internally writable fields - #29 by JoBe

[Eneg]

I’ve updated the PR (diff)
The gist of changes:

• revert 11f4522 - “Remove internal mutation restriction”
• brief “Terminology” section
  • explanation of “attribute mutation” and “read-only”
• allow subclasses to initialize parent’s read-only attributes
• allow combining Final and ReadOnly, although redundant
  • Final semantics take priority over read-only

---

Under Initialization > Instance Attributes, it is currently specified that

Assignment to an instance attribute must be allowed in the following contexts:

• […]
• In __new__, on instances of the declaring class created via a call to a super-class’ __new__ method.

Additionally, a type checker may choose to allow the assignment:

• In __new__, on instances of the declaring class, without regard to the origin of the instance. (This choice trades soundness, as the instance may already be initialized, for the simplicity of implementation.)
• In @classmethods, on instances of the declaring class created via a call to the class’ or super-class’ __new__ method.

I’m considering whether assignment within classmethods could be allowed by default (that is, to move the classmethod bullet point to the “must be allowed” section)

It feels to me that it shouldn’t be any less safe than what we allow for __new__ (sourcing instance from super())

Thoughts?

[Eneg]

Origin for those rules for context: (cc @carljm)

Expanding `ReadOnly` to normal classes & protocols

I think the argument that __new__ is the correct place to initialize attributes that are actually read-only at runtime is strong, and it will be a problem if we don’t allow initialization of ReadOnly attributes in __new__. I think this is not a problem; a type checker is free to in general disallow explicit calls to __init__ on an existing instance, and mypy already does. (I would be fine with specifying that type checkers should do so, but it’s not an issue where standardization has a lot of …

[carljm (Carl Meyer)]

Yeah, I think really we can treat __new__ and classmethods the same in the PEP. In both cases, it is sound if we require that the instance originated from a call to __new__ on the same class within the method, but that requirement could also be loosened for implementation simplicity.

[Eneg]

@carljm

In both cases, it is sound if we require that the instance originated from a call to __new__ on the same class within the method

To make it clear, is the following correct?

[must be allowed in]

• in __new__, on instance created by a call to super().__new__()
• in classmethods, on instance created by a call to cls.__new__() or super().__new__()

[may be allowed in]
in __new__ and classmethods, without regard for the origin of the instance

Perhaps the two bullet points could be merged into one, though calling cls.__new__() from __new__ results in a recursive call

[carljm (Carl Meyer)]

Yes, I think that’s right, on all counts.

[oscarbenjamin]

@Eneg

To make it clear, is the following correct?

[must be allowed in]

• in __new__, on instance created by a call to super().__new__()
• in classmethods, on instance created by a call to cls.__new__() or super().__new__()

There can be good reasons to bypass calling the __new__ method of the immediate superclass i.e. not using super() for this.

It’s late here so I can’t distil an easily understandable real example but I mean something like this:

from __future__ import annotations

class Rectangle:
    width: int
    height: int

    def __new__(cls, width: int, height: int):
        obj = object.__new__(cls)
        obj.width = width
        obj.height = height
        return obj

class Square(Rectangle):
    def __new__(cls, side: int):
        obj = object.__new__(cls)
        obj.width = side
        obj.height = side
        return obj

I’m not sure if this is or is not consistent with what you are suggesting but when you get into using __new__ you can also end up wanting to use object.__new__(cls) rather than super().__new__(cls) precisely because the former gives you full control over creation and initialisation.

[carljm (Carl Meyer)]

Yeah we should allow setting it on an object returned from object.__new__.

[Eneg]

1. Should it be possible for classmethods to assign to read-only attributes on an instance sourced from <ClassName>.__new__()?
   While possible, I don’t think this has any benefits over doing cls.__new__().
2. What should type checkers do in the following scenario?

class HasName(Protocol):
    name: ReadOnly[str]

class User:
    name: ReadOnly[str]

    def __init__(self: HasName, name: str) -> None:
        self.name = name  # allowed?

I think this should not be allowed, as it would allow writing to arbitrary objects by doing User.__init__(has_name).

__init__ has currently specified the following:

[must be allowed in]

• In __init__, on the instance received as the first parameter (usually self)

It’s probably useful to clarify that the type of self must be of the declaring class type?

[oscarbenjamin]

@Eneg

Should it be possible for classmethods to assign to read-only attributes on an instance sourced from <ClassName>.__new__()?
While possible, I don’t think this has any benefits over doing cls.__new__().

The benefit is always to do with public vs private constructors. If cls.__new__ is the public constructor that accepts, parses and canonicalises many possible input types/values then there is always going to be a reason to bypass it. This happens for any code where the types are known and canonicalisation is not needed e.g. to avoid a GCD calculation in the Fraction constructor.

[Eneg]

@oscarbenjamin

If cls.__new__ is the public constructor that accepts, parses and canonicalises many possible input types/values then there is always going to be a reason to bypass it.

I get that, but it doesn’t answer my question.
Are there classmethod patterns that need to use <ClassName>, instead of object, super() or cls?

[oscarbenjamin]

@Eneg

Are there classmethod patterns that need to use <ClassName>, instead of object, super() or cls?

I certainly see a lot of code that does do that in sympy although it possibly just predates super. I would have to check the class hierarchies to see if they could all be changed to using super…

… Actually I asked AI (gpt-5.4) to do that and it found 65 classes where <ClassName>.__new__ is used and changing to super would not be equivalent.

  - numbers.py:1886: Integer calls Expr.__new__, but super().__new__ would hit Rational.__new__ /
    Number.__new__ first.
  - matadd.py:52: MatAdd calls Basic.__new__, but super().__new__ would hit MatrixExpr.__new__ first.
  - rv.py:369: RandomMatrixSymbol calls Basic.__new__, but super().__new__ would hit
    RandomSymbol.__new__ and MatrixSymbol.__new__.
  - piecewise.py:155: Piecewise calls Basic.__new__, but super().__new__ would hit DefinedFunction /
    Function / Application.
  - ellipse.py:1591: Circle calls GeometryEntity.__new__, but super().__new__ would hit
    Ellipse.__new__.

A simplified version of how this happens for Integer.__new__ is

from __future__ import annotations

class Basic:
    _args: tuple[Basic, ...]
    _mhash: int | None

    def __new__(cls, *args: Basic) -> Basic:
        obj = object.__new__(cls)
        obj._mhash = None
        obj._args = args
        return obj

class Expr(Basic):
    pass

class Number(Expr):
    def __new__(cls, obj) -> Number:
        if isinstance(obj, Number):
            return obj
        if isinstance(obj, int):
            return Integer(obj)
        if isinstance(obj, tuple) and len(obj) == 2:
            return Rational(*obj)
        raise TypeError

class Rational(Number):
    p: int
    q: int

    def __new__(cls, p: int, q: int) -> Rational:
        obj = Expr.__new__(cls)
        obj.p = p
        obj.q = q
        return obj

class Integer(Rational):
    q: int = 1

    def __new__(cls, i: int) -> Integer:
        obj = Expr.__new__(cls)
        obj.p = i
        return obj

So Integer.__new__ needs to skip Rational.__new__ and Number.__new__ in the MRO. There are more class methods involved than just __new__ in the actual code but Expr.__new__ is used because super().__new__ would do the wrong thing. I would probably argue that Number.__new__ doesn’t need to be there but I have certainly seen people using it and it can’t be removed now.

[Eneg]

I wonder how to specify this.
My initial thought was to enumerate three cases:

• the instance can be sourced from super().__new__().
• the instance can be sourced from cls.__new__(), where cls is the first parameter.
• the instance can be sourced from <ClassName>.__new__(), where <ClassName> is a class appearing in the mro.

But this shows a better, more generic way is needed.

“a call to __new__ on any object of type type[T], where T is …”?
Would be nice to replace T with Self, but if I understand variance correctly, Self is covariant, and T should actually be contravariant.

@carljm what do you think?

[Eneg]

I think the phrasing would be “…where T is a nominal supertype of the declaring class”

I think it’s important to specify nominal, as it doesn’t make sense for T to be a structural type
OTOH, you can implement __new__ on a protocol, and can inherit from that protocol?

[carljm (Carl Meyer)]

A protocol that you explicitly inherit from is still a nominal supertype.

I’m not clear why “a more generic way is needed” – I think your three bullet points still capture all of the cases mentioned here?

I think super().__new__() deserves special mention either way, since it is not clearly “a call to __new__ on any object of type type[T], where T is …”.

[Eneg]

@carljm

I’m not clear why “a more generic way is needed”

It seems to me the three cases have a lot of overlap, hence the conclusion.

@carljm

I think super().__new__() deserves special mention either way

Yeah, makes sense.