C++ program embedding Python crashes when calling PyUnicode_AsUTF8

Hi all,

I’ve posted this in the Python form at python-forum[dot]io but I really think this is a more suitable place for this post, so I’m posting it here too (will report back to python forum if resolved).

I’m embedding Python into an openFrameworks (a C++ toolkit for creative coding) program through the C/C++ embeddings of the Pyo module (a Python module for DSP - Digital Signal Processing - written in C).

It works fine when calling PyRun_SimpleString() to run a Python chunk of code, but when trying to get more information in case something goes wrong, the program calls PyUnicode_AsUTF8() and it crashes. I run the program with GDB and it pointed that this function calls PyUnicode_AsUTF8AndSize(), which is where the crash happens. This happens in unicodeobject.c from Python’s sources (which by the way I couldn’t locate in my system, only unicodeobject.h).

From Python’s GitHub’s cpython directory, I found unicodeobject.c and found the PyUnicode_AsUTF8() in line 4139 (https://github.com/python/cpython/blob/3…deobject.c). Indeed, this funciton calls PyUnicode_AsUTF8AndSize(PyObject *unicode, NULL), which is where GDB points to. Here’s the bit of code from Pyo’s C/C++ embeddings where this happens:

INLINE int pyo_exec_statement(PyThreadState *interp, char *msg, int debug) {
    int err = 0;
    if (debug) {
        PyObject *module, *obj;
        char pp[26] = "_error_=None\ntry:\n    ";
        memmove(msg + strlen(pp), msg, strlen(msg)+1);
        memmove(msg, pp, strlen(pp));
        strcat(msg, "\nexcept Exception, _e_:\n    _error_=str(_e_)");
        PyEval_AcquireThread(interp);
        PyRun_SimpleString(msg);
        module = PyImport_AddModule("__main__");
        obj = PyObject_GetAttrString(module, "_error_");
        if (obj != Py_None) {
            strcpy(msg, PyUnicode_AsUTF8(obj));
            err = 1;
        }
        PyEval_ReleaseThread(interp);
    }
    else {
        PyEval_AcquireThread(interp);
        err = PyRun_SimpleString(msg);
        PyEval_ReleaseThread(interp);
    }
    return err;
}

The call to PyUnicode_AsUTF8() happens in line 14 in the chunk above, with a valid PyObject* argument, since this is what is supposed to be returned by PyObject_GetAttrString(), in line 12 of the chunk above. The full code this chuck belongs to is here https://github.com/belangeo/pyo/blob/mas…ed/m_pyo.h. The code chunk above is in line 470.

I think this is all the info I’ve got. I would appreaciate any help.
Thanks.

Are you absolutely sure that msg is backed by a writable buffer large enough to contain the extended program? No code visible verifies this which is a sign of horrible C-coding practices. What happens if you run your example with valgrind?

except Exception, _e_ is a syntax error in Python 3, so PyRun_SimpleString will be returning -1 to indicate that an exception occurred, which you’re not checking for.

I suspect that PyImport_AddModule and PyObject_GetAttrString are returning NULL for the same reason, which you’re not checking for either.

If you’re passing NULL to PyUnicode_AsUTF8, I’m not surprised it’s crashing. Don’t do that!

And in particular, always check the return value of a C function call that has error cases. You have to handle them, otherwise you get issues like this.

That was it! I now changed the code to the following and it doesn’t crash:

INLINE int pyo_exec_statement(PyThreadState *interp, char *msg, int debug) {
    int err = 0;
    if (debug) {
        PyObject *module, *obj;
        char pp[26] = "_error_=None\ntry:\n    ";
        memmove(msg + strlen(pp), msg, strlen(msg)+1);
        memmove(msg, pp, strlen(pp));
        strcat(msg, "\nexcept Exception as _e_:\n    _error_=str(_e_)");
        PyEval_AcquireThread(interp);
        err = PyRun_SimpleString(msg);
        module = PyImport_AddModule("__main__");
		if (module != NULL) {
			obj = PyObject_GetAttrString(module, "_error_");
        	if (obj != Py_None && obj != NULL) {
				if (PyUnicode_AsUTF8(obj) != NULL) {
        	    	strcpy(msg, PyUnicode_AsUTF8(obj));
				}
        	    err = 1;
        	}
		}
        PyEval_ReleaseThread(interp);
    }
    else {
        PyEval_AcquireThread(interp);
        err = PyRun_SimpleString(msg);
        PyEval_ReleaseThread(interp);
    }
    return err;
}

The problem is that the way this code is written it can only execute single lines, as it inserts a try statement with four spaces but doesn’t insert an additional four spaces for every line in the string to execute. So, if I try to execute the following:

def foo():
    print("bar")

it will result in executing:

_error_=None
try:
   def foo():
   print("bar")
except Exception as _e_:
   _error_=str(_e_)

so it will not indent print(“bar”), which results in an indentation error.

I have to do some C string juggling there, I guess.