Make __replace__ stop interfering with variance inference

Typing
1

Consider:

from dataclasses import dataclass
from typing import Generic, TypeVar, override


if False:  # Switch to True is a type error.
    @dataclass(frozen=True)
    class C[T]:
        x: T  # T is inferred to be invariant!
else:
    U = TypeVar('U', covariant=True)

    @dataclass(frozen=True)
    class C(Generic[U]):
        x: U


class X:
    def f(self) -> C[object]:
        return C(1.0)


class Y(X):
    @override
    def f(self) -> C[float]:
        return C(1.0)

Eric explains the problem clearly here, but the whole thread is worth reading.

Would it be possible to move towards a resolution on this? I personally like option 2 because “broken” variance inference is quite difficult to work around, and hampers adoption of PEP 695.

2 Likes
2

The variance inference is correct here, and all special casing would do is break the ability to safely use copy.replace or for that to ever be typed appropriately. Subclasses that override unsafely cause problems.

I’d actually prefer if typecheckers rejected an old-style type variable with the incorrect variance.

If we need non-inheritance based methods of composing dataclasses, I could see that being worth working on to improve various structural types, but as-is, this just is a problem with overuse of inheritance.

1 Like
3

The __replace__ docs state that

This method should create a new object of the same type, replacing fields with values from changes.

And although there’s room for interpretation, I think this suggests that __replace__ is not meant to mutate the data. If we naively assume that this indeed never happens, then type-checkers don’t need to view the __replace__ parameter as an input position. The self is not mutated, so it would be type-safe (still being naive here) to annotate the generated __replace__ as e.g.

new:

def __replace__[$typesig](self: $ident[$typesig], $callsig) -> $ident[$typesig]: ...

(the $ thingies indicate template-like placeholders, similar to a macro in rust)

For comparison, this is how I picture the current generated __replace__ method:

current:

def __replace__(self, $callsig) -> $ident[$typesig]: ...

In case of one value of some (data)class class Spam[T], these would result in

new:

def __replace__[T2](self: Spam[T2], *, spam: T2) -> Spam[T2]: ...

current:

def __replace__(self, *, spam: T) -> Spam[T]: ...

I’m sure that there are many edge-cases where this isn’t type-safe. However, the current situation is a breaking change in Python 3.13. And I think the latter is a much bigger problem.

2 Likes
4

This is a rather unfortunate situation and I’d love to find a solution that allows frozen generic dataclasses to still be covariant. The current situation is not good for users.

However, it’s worth noting that it really is necessary for soundness that even frozen dataclasses become invariant:

from dataclasses import dataclass

@dataclass(frozen=True)
class Base[T]:
    x: T

    def method(self) -> T:
        return self.x

class Child(Base[int]):
    def method(self) -> int:
        return self.x + 1

def func(x: Base[object]) -> object:
    return x.__replace__(x="hello").method()

func(Child(x=1))  # mypy and pyright error here because Child is not assignable to Base[object]

This program fails at runtime (we end up with a Child whose x attribute is a str, so Child.method fails), and mypy and pyright catch it only because of the invariance issue.

@andersk on GitHub gave another example of unsoundness with __replace__:

from dataclasses import dataclass

@dataclass(frozen=True)
class A:
    x: int | str
    def get_str(self) -> str:
        return ""

class B(A):
    x: str
    def get_str(self) -> str:
        return self.x

a: A = B(x="")
five: str = a.__replace__(x=5).get_str()  # 5
five.upper()  # AttributeError at runtime

Type checkers currently don’t catch this, but one way to detect this unsoundness would be by noticing that B overrides __replace__ incorrectly. The synthesized methods look like this:

class A:
     def __replace__(self, x: int | str) -> Self: ...
class B(A):
     def __replace__(self, x: str) -> Self: ...

So to achieve soundness, type checkers should also reject covariant inheritance of immutable dataclass fields.

3 Likes
5

It makes sense that variables in a NamedTuple and a frozen dataclass should be considered read-only and should therefore not cause a TypeVar to be inferred as invariant.

Interestingly, Python 3.13’s addition of a __replace__ method effectively breaks this.

Seems really related to what I was worried about in Proposal: Optional Explicit covariance/contravariance for PEP 695.

Maybe explicit is better than implicit after all.

6

People had a useful tool (covariant frozen dataclasses) and then someone else comes along forcing something on those people that they didn’t ask for (the __replace__ method). And then the people that had the useful tool (covariant frozen dataclasses) don’t have that useful tool anymore?

I think Eric’s option 2 is best.

  1. We add a special-case exemption for replace in the TypeVar variance algorithm. There are already exemptions for __init__ and __new__, so maybe this is acceptable.

__replace__ is another constructor dunder method like __init__ and __new__ - So I think it really fits.

If someone wants to avoid false negatives, it could be through a type-checker option to report an error on usage of copy.replace or __replace__ when a precisely known type can’t be seen as safe - the same way it could be reported for __init__ and __new__.

def foo(c_type: type[C]):
	c_instance = c_type()  # this is not safe - __replace__ is the same story
1 Like
7

I don’t think special casing the variance algorithm for __replace__ is the right solution.

Dunder methods such as __replace__ are in some sense python’s way of realizing multiple dispatch. They are usually not supposed to be called from the object itself, but through another method or operator. The typical schema is:

class Example:
     def __dunder__(self, *args, **kwargs): ...

def dunder(obj, *args, **kwargs):
    if hasattr(obj, "__dunder__"):
        return obj.__dunder__(*args, **kwargs)

So, the imo right question to ask is how we would annotate an @overload for copy.replace for a given type SomeDataclass. Such an overload obviously couldn’t contain any bound type-vars of that class, as they would be out of scope.

The signature of SomeDataclass.__replace__ should be identical to that @overload. As it doesn’t contain any type-vars bound to SomeDataclass, it doesn’t influence variance; problem solved.

Therefore, I believe @jorenham’s approach is on the right track.

2 Likes
8

Consider a non-dataclass variant of this example from the thread in mypy. I use property to emulate covariance. With the suggestion by @jorenham, we would get: Code sample in pyright playground

class A[T: int | str]:
    @property
    def x(self) -> T: ...

    def __replace__[S: int | str](self: "A[S]", *, x: S) -> "A[S]": ...

class B(A[str]):
    @property
    def x(self) -> str: ...

    def __replace__(self: "B", *, x: str) -> "B": ... # ❌ override

This highlights that the actual problem with __replace__ is not variance, but that this method by design is not safe for subclassing, as @andersk beautifully demonstrated.

So, type-checkers can decide to special case __replace__ in the sense of ignoring this override error, but variance doesn’t need to be touched.

9

Thanks for the clear exposition—you’re right that one way to restore soundness would be to block covariant inheritance of dataclass fields.

However, I think that’s too high a price to pay just for the convenience of replace. I’d like to propose an alternative.

First, let’s observe that the problem arises with class factories written like this:

class C:
  @classmethod
  def create(cls, ...) -> Self:
    return cls(...)  # You can't actually know the parameters of cls!

You could have create return C, which would be more accurate, but that raises the question: why are we inheriting this class factory in subclasses at all?

The core issue is that class factories are inherited, even though they probably shouldn’t be. The intended use of a class factory like create is via an explicit call like C.create. You don’t typically need something like:

def f(t: type[C]) -> C:
  return t.create(...)
f(C)

If you do, a better approach might be:

def f(factory: Callable[..., C]) -> C:
  return factory(...)
f(C.create)

In short, create should only be callable on a specific class, not on a variable whose type merely is a subclass of that class.

A similar situation arises with __replace__. Consider:

def f(x: D) -> D:
  return replace(x, some_member=1)

As others have noted, we can’t guarantee that some_member hasn’t been narrowed in a way that makes assigning it an int invalid.

A straightforward solution would be to treat replace more like a class factory. If class factories shouldn’t be inherited, then replace shouldn’t be either. Instead, you’d write:

def f(x: D) -> D:
  return D.__replace__(x, some_member=1)

By explicitly calling the “replace factory” of D, we sidestep the covariance issue—we’re constructing an instance of D and using x only as an input.

This also justifies an optional type warning for the following:

replace(x, some_member=1)  # What type are you trying to construct? We can't be sure some_member is valid here.

—unless x has a final type.

However, if you write:

D.__replace__(x, some_member=1)

then there’s no ambiguity, and no warning would be necessary.

I agree that specifying the class explicitly isn’t ideal—but the current situation, where inheritance introduces unsoundness, is much worse.

This is consistent with Randolph’s hypothesis:

It follows Doug’s intuition that __replace__ is a constructor:

I think it’s a variation of the Joren’s idea that:

in that __replace__'s parameter is treated specially. It’s neither mutated, nor used to polymorphically choose between class factory return types.

2 Likes
10

I think this is reasonable in most cases. But there are some situations where covariant inheritance is perfectly sound, for example:

@dataclass(frozen=True)
class Base[T: complex]:
    x: T

    def method(self) -> T:
        return self.x

    # def __replace__[T2: complex](self, x: T2) -> Base[T2]: ...

class Child1[T: complex](Base[T]):
    def method(self) -> T:
        return self.x + 1

    # @override
    # def __replace__[T2: complex](self, x: T2) -> Child1[T2]: ...

class Child2(Base[complex]):
    def method(self) -> T:
        return self.x + 1

    # @override
    # def __replace__(self, x: complex) -> Child2: ...

If we assume that no other subclasses exist (i.e. they’re @sealed), then T@Base and T@Child1 can be covariant without violating LSP.

11

My use case of covariant inheritance is something as simple as:

@dataclass(frozen=True)
class InferenceResult[C_co: Configuration = Configuration]:
    configuration: C_co
    # ...

def f(r: InferenceResult) -> Configuration:
  return r.configuration  # Should work.

def g(...) -> InferenceResult[SomeConfiguration]: ...
result = g(...)
result.configuration  # Should have type SomeConfiguration.

I think this should work.

12

If people are really uncomfortable with allowing subclasses to introduce unsound behavior[1], then instead there should be the ability to turn of the generation of __replace__ to remove the restriction on variance caused by it.

  1. IMO practicality beats purity here, but that’s not the point of this comment ↩︎

2 Likes
13

Something like a @dataclass(replace=False) could indeed help for those that realize that the invisible __replace__ method is the reason behind this backwards-incompatible variance issue. But for most users that are facing this issue, this probably isn’t the case.

1 Like
14

But that could be improved via better type-checker error messages.

1 Like
15

Could you give an example of that?

16

If type checker notice that covariance is an issue, they could point to the methods that cause this variance - especially if it’s a single synthesized method. I don’t know enough about type checker implementations to create a POC, but I would be surprised if this is information that is hard to get for them.

E.g. the pyright error message for the example in OP:

Method "f" overrides class "X" in an incompatible manner
  Return type mismatch: base method returns type "C[object]", override returns type "C[float]"
    "C[float]" is not assignable to "C[object]"
      Type parameter "T@C" is invariant, but "float" is not the same as "object"  (reportIncompatibleMethodOverride)

Could be extended to include

"T@C" is invariant because of the synthesized "__replace__" method.

The inclusion or exclusion of this message could be controlled by heuristics and/or type checker options.

1 Like