Hi, I’m presenting PEP 767.
It enables the use of ReadOnly within attribute annotations, as a way to remove [1] the ability to reassign or delete an attribute of an object.
Any feedback is welcome!
at a type checking level ↩︎
14 Likes
Read-only attributes are covariant.
Does that mean that the type parameter T@ReadOnly is covariant in the same way as e.g. T@Sequence, or does it mean that the T that’s used in a ReadOnly[T] of a generic class or protocol must always be covariant?
To illustrate, consider this example:
class Box[T]:
thing: ReadOnly[T]
def match(self, other_thing: T, /) -> bool:
return other_thing == self.thing
In the first case, the T@Box would inferred as invariant (so the same behaviour as with a @property instead of a ReadOnly). But in the second case, it would be covariant.
The reason I’m asking, is because if T is explicitly marked as read-only within its defining generic type, then it’s safe to assume it won’t mutate. So even if T is used to annotate a method parameter (or any anything else that in other situations might have led to a mutation of something that has type T), the attribute of type T won’t be changed. So there’s no need to make it invariant, and can therefore “stay” covariant.
I believe that this is a special case of what’s commonly referred to as use-site variance, and avoids situations where T@Box is inferred as invariant.
This. It cannot force a type variable to be covariant. Rather, it limits the variance of T to covariance. Further places where T occurs may limit its variance further.
In your example, it appears in the type of a method parameter. Parameters of callables limit the variance of type variables to contravariance; application of both limits results in T being invariant.
Typing docs for reference
2 Likes
Ok that’s too bad. Because I think PEP could also be used to improve the current variance inference, which can be unnecessarily restrictive, as my example shows.
That’s a pretty good explanation of the way that variance is currently inferred of Python.
But the point I’m trying to make is actually a bit more subtle than that, and was trying to show that Box[T] with T covariant is type-safe:
a: Box[int]
b: Box[object] = a
b.match(object()) # no problem
And even if we add another method to Box with the same signature as match, then this would still be type-safe. That’s because there is nothing of type T in Box[T] that can be mutated, thanks to ReadOnly[T].
Note that this is only the case for Box itself, because there might be subtypes that override thing in a mutable way.
There are some ways that variance could be fixed that have been brought up before, but that should probably be handled for variance as a whole rather than try to use this change to make it happen. For what you want to be sound, it has to be the case that subclasses that restrict variance more than the parent are considered an LSP violation and rejected (as they are no longer a suitable replacement for the parent behaviorally). This probably won’t ever happen because there are some extremely deeply rooted cases, like Sequence/MutableSequence, so the narrowing by the assumption of covariance can’t be safely done, and people are still proposing new exceptions to LSP despite the problems LSP violations cause, such as excluding __replace__ in 3.13+'s dataclasses.
1 Like
The match method in your example doesn’t really depend on the type of T, so it could be likely replaced with object (as is typically done for __eq__ or __contains__)
A subclass doesn’t need to enable mutability for thing to make covariant T unsafe:
class Box[T]:
thing: ReadOnly[T]
def match(self, other_thing: T, /) -> bool: ...
class Rect(Box[int]):
def match(self, other: int, /) -> bool:
return self.thing.denominator == other.denominator
a: Box[object] = Rect()
a.match(object()) # boom
You could remove the ability to subclass Box by decorating it with @typing.final, in which case I can’t think of other problems.
However, that won’t work for protocols.
I think that makes sense, especially considering the difficulty of this topic.
This will probably have to wait until we variance can be manually declared (again).
Yes your example nicely illustrates when I said
This is something that with the current automatically inferred variances is difficult (but not impossible) to deal with.
I like the PEP, especially the open question at the bottom regarding __init_subclass__.
I’ve done patterns where I utilize __init_subclass__ to force a child class defines class variables through class parameterizations, these class variables are read only from my point of view.
In the example below the desired behavior is for subclasses of OurABC to be forced to have a class variable abstract_classvar and for that classvar to be read only i.e. I want Concrete.abstract_classvar to be readonly.
from abc import ABC
from inspect import isabstract
from typing import ClassVar, Optional
class OurABC(ABC):
abstract_classvar: ClassVar[int]
abstract: ClassVar[bool]
def __init_subclass__(cls, *, abstract_classvar: Optional[int] = None, abstract: bool = False):
cls.abstract = True
if not abstract and not isabstract(cls):
cls.abstract = False
if abstract_classvar is None:
# handling of requirement of classvar to be passed as class parameter
raise Exception
cls.abstract_classvar = abstract_classvar
return super().__init_subclass__()
class Concrete(OurABC, abstract_classvar=5):
...
It seems to me like it makes sense to allow initialization of ReadOnly[ClassVar[...]] in __init_subclass__
pyre also has support for a read-only attribute (via pyre_extensions.ReadOnly) but is more ambitious in the sense that it disallows any mutation. So this:
from dataclasses import dataclass
from typing import Protocol
from pyre_extensions import ReadOnly
@dataclass
class Game:
name: str
class HasGames(Protocol):
games: ReadOnly[list[Game]]
def add_games(shelf: HasGames) -> None:
shelf.games.append(Game("Half-Life")) # line 14
shelf.games[-1].name = "Black Mesa" # line 15
shelf.games = [] # line 16
del shelf.games # line 17
Leads to the following pyre output:
main.py:14:4 ReadOnly violation - Calling mutating method on readonly type [3005]: Method `list.append` may modify its object. Cannot call it on readonly expression `shelf.games` of type `pyre_extensions.PyreReadOnly[typing.List[Game]]`.
main.py:15:4 ReadOnly violation - Assigning to readonly attribute [3003]: Cannot assign to attribute `name` since it is readonly.
So, amusingly, pyre errors exactly on those lines that the PEP says shouldn’t be errors and is fine with the lines that the PEP says should be errors.
I’m not saying that this is the correct way to interpret “ReadOnly” but it should maybe be discussed in the PEP.
EDIT: here is how to mark methods as not mutating in pyre:
@dataclass
class A:
x: int
def possibly_mutating(self) -> None:
self.x = 1
def not_mutating(self: ReadOnly[Self]) -> None:
print(self.x)
def f(a: ReadOnly[A]) -> None:
a.possibly_mutating() # ReadOnly violation - Calling mutating method on
# readonly type [3005]: Method
# `main.A.possibly_mutating` may modify its object.
# Cannot call it on readonly expression `a` of type
# `pyre_extensions.PyreReadOnly[A]`.
a.not_mutating() # OK
2 Likes
What pyre calls ReadOnly should really be called Immutable[1].
typing.ReadOnly is not a new symbol, and it mirrors the semantics PEP 705 describes for TypedDict, it would be weird to use the name for something completely else now, so the ship has already sailed for any naming discussions.
I’m guessing pyre_extensions.ReadOnly predates typing.ReadOnly. It’s unfortunate that there’s a naming conflict, bur pyre should probably consider deprecating that name in favor of something less ambiguous like Immutable, now that typing.ReadOnly exists and has a very different meaning.
This loosely mirrors Rust with its
mutkeyword, although there everything is a immutable reference by default and you need to mark mutable references instead of the other way around ↩︎
4 Likes
Something like this is desperately needed in the type system, but I would suggest a few changes:
A read-only attribute
name: Ton aProtocolin principle defines two requirements:
hasattr(obj, "name")isinstance(obj.name, T)
I wouldn’t call such a thing ReadOnly, but Readable. A read-only member of a protocol needs to satisfy an additional axiom: (3) setattr(obj, name, T) raises an Exception. I believe this is a simple oversight in this section, but at the end of the day, both concepts are necessary in order to facilitate correctly annotating duck-typed code: Need a way to type-hint attributes that is compatible with duck-typing - #19 by randolf-scholz
So if it is possible, it would be very nice to get Type Qualifiers Readable, ReadOnly, Writeable and WriteOnly and Mutable all in one go.
2 Likes
I raised the same objection originally with PEP 705, but we ended up deciding that ReadOnly meaning Readable is fine, since there’s no compelling need for a true ReadOnly qualifier and people seemed to generally prefer ReadOnly over Readable, considering it to be less confusing.
So I fear that ship has sailed as well. If there is ever a need for a strict qualifier it could be called StrictlyReadOnly.
2 Likes
I don’t think this is an oversight, and I don’t think we should add that third requirement in the PEP.
The semantics of typing.ReadOnly are what you call “Readable”, and this naming decision is a ship that I think already sailed with PEP 705.
If you have an object of a type with an attribute typed as ReadOnly, attempting to set that attribute will be a type error, because the mutator method for that attribute is not part of the type. But a runtime object which does permit that mutation can still inhabit that type.
1 Like
The problem with this of course, is interaction with runtime checkable protocols.
class A:
def __init__(self):
self.x: ReadOnly[int] = 1 # This breaks inexplicable if it isn't 1
@runtime_checkable
class B(Protocol):
x: int
def hmm(objs: set[object]):
for val in objs:
if isinstance(val, B):
val.x = 0
hmm({A(),})
It’s trivial to end up in a situation where ReadOnly will be violated if it’s not a requirement that it can’t be set. While this is a toy example, real world code isn’t self-contained where every contributor has knowledge of every place it interacts, that’s actually what type systems are supposed to help developers with
That’s unfortunate. Having Mutable[T] = Readable[T] & Writeable[T] really scratches that itch for beauty in code. Mutable[T] = ReadOnly[T] & WriteOnly[T] looks absolutely horrid. The natural expectation from the plain English terminology is that the intersection type of ReadOnly[T] and WriteOnly[T] should be Never.
2 Likes
True. It can be added to the list of the many other more serious ways in which runtime-checkable protocols are not, and never will be, sound. (I think it was a mistake to add them.)
2 Likes
While I agree it was a mistake to add them, I’m somewhat hesitant about whether the current proposed wider definition of ReadOnly is a good definition within the system we have since people will come to rely on this as if it is enforced, even when it cannot be, or if the system we have is a reason why some definitions should be stricter.
I think the primary motivation for runtime checkable protocols was that some pre-existing ABCs already had this (problematic) behavior. One option would be to soft deprecate them, and recommend that they are not used for any new use cases.
1 Like