_datetime should be a static module

Core Development
1

In CPython, _datetime is the last remaining extension that contains static types. This is because we expose some of _datetime through the public C API: DateTime Objects — Python 3.13.5 documentation.

Unfortunately, _datetime is built as a shared library, which has two primary downsides:

  1. It can be disabled, meaning that the PyDateTime_IMPORT macro will silently set an exception. Based on a code search, nobody checks PyDateTime_IMPORT for failure, so any usage of the _datetime C API will probably just crash when it’s disabled.
  2. Unlike all other static types, _datetime’s static types have to be initialized during import rather that interpreter initialization. This isn’t thread-safe and causes crashes: gh-136421, gh-136423.

We can solve both of these issues by making _datetime static, as then it can’t really be disabled and we can move the static type initialization to the interpreter. But, this is technically a breaking change; does anyone rely on _datetime being non-required when building?

5 Likes
2

I think making the _datetime a static module is a good option. However, it would be a design issue for the runtime to depend on an extension at startup/shutdown. As long as you offer _pydatetime, _datetime should be optional.

3

At this point we should assume not and IMNSHO see how that goes. It has been C for over 20 years and there is little reason to assume it will ever be any other way in CPython.

cc @pganssle specifically for datetime opinions.

This is CPython, C based things are always usable. _datetime doesn’t even have any third party dependencies. Making it static makes sense. Leave the existence of the _pydatetime thing that nobody uses out of the decision.

Our pure Python implementations of things that everyone uses the C version (datetime, decimal, io, etc) of are for historical and prototyping purposes. Projects like PyPy might use them. But outside of our own test suite to attempt to guarantee compatibility, we should not.

I doubt we should even be shipping _pydatetime anymore in our releases. But excluding it from release builds/installs is not a task anyone likely wants to bother spending time on.

5 Likes
4

Do note that making the extension statically linked does not mean it’s always available. You can still build with it disabled (or even shared). It sounds like a really good idea to statically link it by default.

1 Like
5

Is that your consensus as a whole? To be in sync with the C implementations, pure Python versions appear to be still maintained with some extra cost, which are imported as fall back modules when the C module is missing or unavailable. If we use only C versions, just raising an ImportError would suffice.

The C implementations have been introduced for acceleration, but they would not always be advantageous when the faster-cpython project is mature.

Again, I agree to making the _datetime a static module as a default option.

6

AFAIR, the C _datetime module was there before the _pydatetime module. The latter is intended as a reference implementation and may also be used by non-CPython implementations. I don’t think the idea ever was that CPython should be built with _datetime disabled, especially as the latter exposes a C API for third-party extensions.

1 Like
7

Writing up a brief conversation I had with Peter B earlier today – is there a good reason to keep this C API exposed? I believe that moving _datatime away from static types would be a good thing, and might mean that in the future we can remove the special-cased building of _datetime that this suggestion proposes. Are there any strong objections against looking at deprecating it?

A

8

NumPy at least uses the datetime C API to implement numpy datetimes: numpy/numpy/_core/src/multiarray/datetime.c at b5d6eb4300f782f67dde893dc72f2dca94945b6c · numpy/numpy · GitHub

2 Likes
9

more open source uses via github code search: Code search results · GitHub

1 Like
10

This flows from a now somewhat dated feeling PEP 399 – Pure Python/C Accelerator Module Compatibility Requirements | peps.python.org where we declared for the sake of the multi-vm ecosystem that we must keep them in sync and prefer to have Python first implementations. I believe the number of viable runtimes attempting to use our stdlib code has shrunk since.

The ImportError fallbacks to importing the pure python version were more of a maintenance burden reduction thing for other Python implementations using our stdlib code so that they did not need to make changes. In reality those existing does not mean we ever expected normal builds of CPython that did not go out of their way to exclude something to take that ImportError code path.

1 Like
11

Thank you all for the explanations. Now that the _datatime module seems mature enough, the shared DLL may not be so useful for development purposes.

I think I’m fine with the built-in _datatime module that can be accessed from each interpreter directly.

12

Would NumPy be opposed to switching to usual APIs (e.g., import datetime and call things via PyObject_Call)?

Or, could we deprecate how we expose _datetime’s C API? Instead of the horribleness that we’re doing with PyDateTimeAPI, how about we expose some sort of context that works per-interpreter? For example:

static PyObject *
my_method(PyObject *self, PyObject *something)
{
    PyDateTime_CAPI datetime_api;
    if (PyDateTime_Init(&datetime_api) < 0) {
        return NULL;
    }

    if (datetime_api->PyDate_Check(something)) {
        /* ... */
    }

    PyDateTime_Close(datetime_api);
    Py_RETURN_NONE;
}
1 Like
13

Here are the currently exposed macros:

#define PyDateTime_CAPSULE_NAME "datetime.datetime_CAPI"
/* Define global variable for the C API and a macro for setting it. */
static PyDateTime_CAPI *PyDateTimeAPI = NULL;

#define PyDateTime_IMPORT \
    PyDateTimeAPI = (PyDateTime_CAPI *)PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)

The static pseudo-global variable is indeed horrible, but otherwise the PyCapsule_Import pattern is sound and should hopefully be compatible with multiple interpreters (is it? @eric.snow ).

So perhaps we should direct users to call PyCapsule_Import directly. The capsule name can certainly be considered stable.

1 Like
14

Oh hey - I opened the issue about this last year!

I ran into this working on PyO3. Since PyO3 can’t call macros, we have to re-implement them. It looks like we ended up handling the errors as well using a single-initialization API:

I don’t have a strong opinion about the proposed API changes - although @pitrou’s suggestion above seems to be the least disruptive and does fix the issue of a macro that can’t fail calling a function that might fail. I’m not dure if users calling PyCapsule_Import themselves might lead to races, but that’s also not a problem if it happens during module init.

I raised NumPy using this API as well as the links to open source uses to try to emphasize that the current API is used all over the place, so the instructions for dealing with the deprecation should be very clear and hopefully not require a substantial refactor to adopt.