C++ Code crashes at call to import python module

Python Help
1

Hello Python Team,

We are embedding python in C++ for establishing connection to Rest API’s and getting the response. We are compiling the C++ code and generating DLL for the code.

When we trigger or call the C++ method for the first time the code is calling the python file and connecting to Rest API’s and returning back the response. Everything is working fine for the first time. But when we again call the same method in the same session code crashes at a point i.e. “PyObject* pythonFile = PyImport_ImportModule(“PythonCode”)”. Second time Python initialization is happening but at the above line code crashes. When we refresh the page, a new session is assigned and again when we run it works fine.

I request you to please look into this issue and help us out resolving this issue as soon as possible. If you have any questions, we can connect through teams, etc. Sharing you the code snippet below.

C++ Code:

std::string main(std::string PartID )
{
    std::string ret="";
    Py_Initialize();
    if (Py_IsInitialized())
    {
        PyObject* pythonFile = PyImport_ImportModule("PythonCode");
        if (pythonFile != Py_None && pythonFile != NULL)
        {
            PyObject* funcName = Py_None;
            funcName = PyObject_GetAttrString(pythonFile, "callCode");
            if (funcName != Py_None && PyCallable_Check(funcName))
            {
                PyObject* output = Py_None;
                PyObject* args = PyTuple_New(1);
                PyTuple_SetItem(args, 0, Py_BuildValue("s", PartID.c_str()));
                output = PyObject_CallObject(funcName, args);
                if (output != Py_None)
                {
                    PyObject* temp_bytes = Py_None;
                    temp_bytes = PyUnicode_AsEncodedString(output, "UTF-8", "strict");
                    char* result = NULL;
                    result = PyBytes_AS_STRING(temp_bytes);
                    ret = std::string(result);
                    result = NULL;
                    Py_DECREF(temp_bytes);
                }
                Py_DECREF(args);
                Py_DECREF(output);
            }
            Py_XDECREF(funcName);
        }
        Py_XDECREF(pythonFile);
    }
    Py_Finalize();
    return ret;
}

Python Script file name “PythonCode.py” :

import re
import string
import requests

def callCode(partId):
    retMsg = ''
    s = requests.Session()
    username = 'username'
    password = 'password'
    headers = {'Content-Type': 'application/x-www-form-urlencoded'}
    authRequestData = 'login=' + username + '&apiKey=' + password
    url = "https://url"
    authRequest = s.post(url, data=authRequestData, headers=headers, verify = False)
    authResponse = authRequest.json()
    authRespStatus = authResponse['Status']
    authRespMessage = authRespStatus['Message']
    print(authRespMessage)
    if(authRespMessage=="Authentication Succeeded"):
        print("Authentication Succeeded")
        partRequestData = 'partNumber=' + partId
        url = "https://partsearchurl"
        partSearchRequest = s.post(url, data=partRequestData, headers=headers, verify = False)
        partSearchResp = partSearchRequest.json()
        retMsg = partSearchResp
        print(partSearchResp)
    else:
        retMsg = "Error : Authentication was not successful"
    s.close()
    return retMsg

Thanks & Regards,
Dheeraj

2

Hi @Dheeraj - Can you please edit your question and put the code inside a code block – surrounded by triple back-quote marks so it looks like

import this

Currently it’s very hard to read.

3

The Python code might be raising an exception.

If that’s the case, then PyObject_CallObject will return NULL. Your C++ code checks for Py_None (None in Python), but not NULL.

1 Like
4

Hello Hans,

Thanks for the reply. I have added the code inside code block can you please check what is the issue and suggest some workaround.

Thanks & Regards,
Dheeraj

5

You have a pattern of init variable the immediate assign to it.

Just write

PyObject* funcName = PyObject_GetAttrString(pythonFile, "callCode");

Here declare output on first use and remove the init of it.

The if check is wrong check for NULL not None.
See Object Protocol — Python 3.8.17 documentation

1 Like
6

I’m assuming the main function is just a normal function called multiple times in your server process, right? (I would call it something else to avoid confusion when reading the code.)
And is it called in a multithreaded C-program from different threads, possibly at more or less the same time?
The code you used seems to have been mostly copied from the embedded tutorial - which is clearly only intended for a command line, single-process app.
I don’t know enough about the current CPython code (and it’s global state, global references), but I would not re-initialize a complete Python interpreter in every call (and would be even more wary if this was in a multithreaded process). I would try to initialize one main interpreter, when the server starts, and then look into using sub interpreters, if you need to use an interpreter in each thread.
This maybe helpful: Initialization, Finalization, and Threads — Python 3.12.0 documentation
Note:

The “main” interpreter is the first one created when the runtime initializes. It is usually the only Python interpreter in a process. Unlike sub-interpreters, the main interpreter has unique process-global responsibilities like signal handling. It is also responsible for execution during runtime initialization and is usually the active interpreter during runtime finalization.

This is probably also relevant to you (it’s a pretty exciting change in the upcoming Python 3.12):

7

I’m curious - Assuming your service is a web app - There are some mature and well-designed Python web frameworks available, like tornado · PyPI or Flask · PyPI, did you consider those rather than going to the long and arduous road of trying to embed a Python interpreter? If you used those, you could directly use the python script in your code. Or turning this around - If you need to work with an existing server, written in C++, wouldn’t it be easier to just port the Python script into C++? Unless you have lots of different scripts that you’d want to execute, I don’t quite understand the point of using an embedded interpreter.

8

Hello Hans,

There is a PLM tool “Teamcenter” in which on action(on click in UI) we can only call a C++ code. So that’s the reason I have tried to embed Python in C++.

I have observed if the python script does not have requests python module calls then its working fine but for connecting to Rest API’s we need requests module. Can you please help on this?

Thanks & Regards,
Dheeraj

9

Like I said - I’m don’t know enough about the current CPython code, especially not about the import mechanism, so I would also need to dive in to really understand why your code crashes. I can just tell you my guess: Something goes wrong with the global state of the Python interpreter when you re-initialize it and then re-import your script - some pointer may be freed twice, or a null pointer may be used somewhere. So, I think you cannot do the re-initialize + re-import as you’re doing now. But I don’t know how you can fix this.
(It would be good actually if the Python docs had some sample code for this kind of multithreaded scenario - perhaps there is sample code already?)

Anyway - perhaps someone with more knowledge about PyImport_ImportModule in a multi-threaded process can comment on this or give hints?

10

HI @Dheeraj - I don’t know why no one has picked up your question - and won’t speculate. Your question seems interesting enough. Anyway - if I was in your shoes, and wanted results, I would forget about Python. It seems you simply want to do some http post request - Isn’t it much simpler to do this directly in C++? (Not to mention, much and much more efficient?)

11

@Dheeraj

  • It would be much easier to help you if you gave a self-contained example with the instructions to run it. http://sscce.org/
  • Have you tried getting a backtrace?
1 Like
12

Can anyone please help on this. It would be a great favor.

13

Allow me to repeat:

It is not useful to ask for help again when you have not provided sufficient information for other people to help.

1 Like
14

It seems to me that it’s overkill to try to write an embedded Python interpreter just for doing some HTTP requests and printing the response. You can do the same thing in less then 10 lines of C code with a system call to ‘curl’.