Questions regarding datetime subclass

Typing
1

I’m trying to type the replace() method for a subclass of Python 3.13 datetime class, and I’m somewhat puzzled.

First, the tzinfo arg in the implementation here has the default value True and in the documentation here self.tzinfo and the type annotation in Typeshed here says _TzInfo | None. For now I’m going with what the type annotation says, but True seems really far off.

Second, the existing type annotations here for replace() state SupportsIndex = ... which, confusingly, doesn’t look like an optional arg. If in my subclass I use that exact same annotation

class DateTime(datetime):
    def replace(
        self,
        year: typing.SupportsIndex = ...,
        # The rest.
    ) -> Self:
        # Do stuff.

then I get a mypy error

error: Incompatible default for argument "year" (default has type "EllipsisType", argument has type "SupportsIndex")  [assignment]

If I go with year: int | None = None — where int indeed implements the SupportsIndex protocol, and making it optional — I get an error

error: Signature of "replace" incompatible with supertype "date"  [override]
note:      Superclass:
note:          def replace(self, year: SupportsIndex = ..., month: SupportsIndex = ..., day: SupportsIndex = ...) -> datetime
note:      Subclass:
note:          def replace(self, year: int | None = ..., month: int | None = ..., day: int | None = ..., hour: int | None = ..., minute: int | None = ..., second: int | None = ..., microsecond: int | None = ..., tzinfo: tzinfo | None = ..., *, fold: int | None = ...) -> datetime
error: Argument 1 of
"replace" is incompatible with supertype "datetime"; supertype defines the
argument type as "SupportsIndex"  [override]
note: This violates the Liskov substitution principle
note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

I poked around for more details and found this and this article, but nothing that helps this situation.

Can somebody please shed some light on all of this? Thank you!

2

Try SupportsIndex | None = None.

If the method on the parent class accepts some non-int SupportsIndex, the method on the child class must as well (as per Liskov substitution).

(Said differently, parameters are contravariant)

3

The = ... syntax is only valid in .pyi files and means that it’s some complex/unknown/sentinel default value that is not part of the interface.

4

@hauntsaninja I tried that but it didn’t work either:

error: Argument 1 to "replace" of "DateTime" has incompatible type "SupportsIndex | None"; expected "SupportsIndex"  [arg-type]

I’m actually a little puzzled by how the type annotations in Typeshed are supposed to express “optional” :thinking:

@MegaIng thank you, that’s good to know! Does that include “optionality” of an argument?

5

It’s valid in abstract definitions but not in an implementation. Can (and should) be used outside pyi files in protocols and overloads.

1 Like
6

The datetime module has two implementations, a C implementation and a fallback Python implementation. Unfortunately, both implementations accept different types. The C implemenation – which is used by default – is stricter, and the type annotations reflect that:

>>> from datetime import date, datetime
>>> date.today().replace(year=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object cannot be interpreted as an integer
>>> datetime.now().replace(tzinfo=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: tzinfo argument must be None or of a tzinfo subclass, not type 'bool'

The C implementation is not that easy to override. I would suggest to use @hauntsaninja’s suggestion and then assemble the arguments provided to datetime.replace() manually:

from datetime import datetime
from typing import SupportsIndex

class MyDateTime(datetime):
    def replace(year: SupportsIndex | None = None, ...):
        kwargs = {}
        if year is not None:
            kwargs["year"] = year
        # ...
        return super().replace(**kwargs)

Alternatively, you could force using the Python implementation, but that has no type annotations:

from _pydatetime import datetime
1 Like
7

@srittau I had tried that suggestion already, see my response above.

In my next attempt to solve the problem I created a mydatetime.pyi stub next to the mydatetime.py implementation and moved all type annotations there, leaving the implementation completely without. But that seems to have the effect that now the types work but the implementation file itself won’t be checked anymore — because the stub takes precedence!? And if I add both then mypy complains

error: Duplicate module named "test_package.mydatetime" (also
at "src/test_package/mydatetime.py")
note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules for more info
note: Common resolutions include: a) using `--exclude` to avoid checking one of them, b) adding `__init__.py` somewhere, c) using `--explicit-package-bases` or adjusting MYPYPATH

Is it not possible to have both an implementation and its stub file and type-check both?

8

Using SupportsIndex | None as the argument annotation is only part of the solution, he gave you the rest of the solution and the proper context for why it’s necessary. You got two different errors. One was for violating LSP by using int instead of SupportsIndex. The other is for passing None to super().replace().

datetime.replace is implemented in C, not in Python. In C you can have optional parameters without a default value, which can’t be expressed in pure Python. So in the Python version you need to use None or a different sentinel as your default value to filter out those parameters before passing them on to the superclass, since the superclass can’t deal with None.

And to head off another potential hurdle: For tzinfo you can’t use None as the sentinel, you will need to define a different one, since replace(tzinfo=None) means “remove the timezone”.

1 Like
10

@srittau @Daverball actually it almost works, but not quite. Following Sebastian’s suggestion

kwargs = {}

doesn’t quite work and neither does

kwargs: dict[str, SupportsIndex | datetime.tzinfo | int] = {}

because both cause errors when passed to replace(). However, declaring the following typed dictionary seems to work (and no fiddling around with None either):

class _ReplaceArgs(TypedDict):
    year: NotRequired[SupportsIndex]
    month: NotRequired[SupportsIndex]
    day: NotRequired[SupportsIndex]
    hour: NotRequired[SupportsIndex]
    minute: NotRequired[SupportsIndex]
    second: NotRequired[SupportsIndex]
    microsecond: NotRequired[SupportsIndex]
    tzinfo: NotRequired[datetime_.tzinfo]
    fold: NotRequired[int]

kwargs: _ReplaceArgs = {}
11

Correct, correctly overriding methods is one of the things that can quickly become quite painful in Python due to the many different kinds of parameters and C bindings allowing signatures that can’t be expressed in Python.

You found one of the methods that is the most annoying kind to override because it features heterogenous parameter types, optional parameters without a default value and all parameters are positional or keyword. But it can get even worse with overloaded methods.

In this case you aren’t quite finished yet. I can give you a small regression test that will show that you’re currently not providing the complete functionality of datetime:

x = DateTime().now(UTC)
y = x.replace(tzinfo=None)
assert y.tzinfo is None

So you will need a sentinel other than None to distinguish between “retain the original timezone” and “remove the timezone”.

Edit: Unless your subclass happens to always be tz-aware, in which case I guess you got lucky, not having to deal with this. Although it seems slightly error-prone to accept None and have it mean something else in your subclass. So I personally would probably still use a different sentinel.

12

@Daverball So you will need a sentinel other than None […]

Yes, and like the Python implementation here I used True as that sentinel.

Unless your subclass happens to always be tz-aware, in which case I guess you got lucky, not having to deal with this.

Ah, you saw right through this :nerd_face: Yes, the subclass is indeed intended to ensure aware datetime objects always and therefore replace(tzinfo=None) now raises a ValueError. And yes, I have a test for that too!

1 Like