Hello,
I have a relatively large C/C++ program that embeds Python using C-API to call user external python modules. This program runs in cycles and after each cycle, it needs to call Py_FinalizeEx() then Py_Initialize(). Without going in a lot of details why I had to do that way, it was working great until one of the users created an external python script that made this fail. I get this Python error: Assertion failed: PyUnicode_CheckExact(ep->me_key), file D:\a\1\s\Objects\dictobject.c, line 936
After a lot of tracing, I found that the “import datetime” statement is what will produces this error in starting from the second cycle. First cycle works fine.
Below is a very simple C program that produces the issue:
int main (int argc, char *argv[])
{
Py_Initialize();
PyRun_SimpleString("import datetime\nprint('Hello, World!')");
Py_FinalizeEx();
Py_Initialize();
PyRun_SimpleString("import datetime\nprint('Hello, World!')");
Py_FinalizeEx();
return 0;
}
The output is: Hello, World! Assertion failed: PyUnicode_CheckExact(ep->me_key), file D:\a\1\s\Objects\dictobject.c, line 936
As you can see, the first cycle works then I get the error in the second cycle. So far I found that “import datetime” is what will cause this. Other modules imports works fine
Why is that? What am I missing to make this work? or any workaround?
Bugs and caveats: The destruction of modules and objects in modules is done in random order; this may cause destructors (__del__() methods) to fail when they depend on other objects (even functions) or modules. Dynamically loaded extension modules loaded by Python are not unloaded. Small amounts of memory allocated by the Python interpreter may not be freed (if you find a leak, please report it). Memory tied up in circular references between objects is not freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.
You should try really hard to avoid having to do this[1], and if you do, there is probably nothing you can do to make datetime work. It might be possible to change the _datetime module to work if reinitialized in this way, but there are probably always going to be some extension modules that just don’t work.
Maybe subinterpreters are able to fulfill your need?
I.e. call Py_Initalize again after Py_FinalizeEx↩︎
Thank you for your reply. Yes I am guessing there will be more modules that causes the same issue. what I am really trying to achieve is have a completely fresh start every cycle. So is there another way to have the python engine clears all loaded modules in memory and reset the state?
Also what subinterpreters you are referring to? Do you mean something else besides CPython?
I believe that the answer for extension modules is no, at least not reliably. They are allowed to rely on static global state (if they aren’t using the new interfaces at least), and don’t need to provide cleanup for that. There are lot’s of discussions around these topics that you can find.
I can reproduce this issue on Python 3.12. But the code just works fine on Python 3.13 and 3.14.
The workaround is to use Python 3.13.
The datetime module was modified in Python 3.13 to implement the multiphase initialization API (PEP 489). Before that, apparently, it’s not possible to unload/reload the extension and it fails badly as you can see.