How to get traceback in embedded Python?

[Alexandros (Alexandros Drymonitis)]

I’m using the following C function to run Python inside a C++ program.

INLINE int pyo_exec_statement(PyThreadState *interp, char *msg, int debug) {
    int err = 0;
    if (debug) {
        PyObject *module, *obj;
        char pp[26] = "_error_=None\ntry:\n    ";
		char tab[5] = "    ";
		char *starting_pos = msg;
		size_t num_newlines, i, j = 0;
		size_t *newlines_ndxs;
        memmove(msg + strlen(pp), msg, strlen(msg)+1);
        memmove(msg, pp, strlen(pp));
        strcat(msg, "\nexcept Exception as _e_:\n    _error_=str(_e_)");
		/* find number of newline characters in the string */
		for (i = 0; msg[i]; msg[i] == '\n' ? i++ : *msg++);
		num_newlines = i;
		/* reset the msg pointer to its starting position */
		msg = starting_pos;
		/* allocate memory for the array that will hold the indexes of the newline characters */
		newlines_ndxs = (size_t*)malloc(sizeof(size_t) * num_newlines);
		for (i = 0; msg[i] != '\0'; i++) {
			if (msg[i] == '\n') {
				newlines_ndxs[j++] = i+1;
			}
		}
		/* we ignore the last two newlines and the first two
		   we loop in reverse so the newlines indexes don't chage as we insert tabs
		*/
		for (i = num_newlines-3; i > 1; i--) {
			insertStringInPlace(msg, tab, newlines_ndxs[i]);
		}
        PyEval_AcquireThread(interp);
        err = PyRun_SimpleString(msg);
        module = PyImport_AddModule("__main__");
		if (module != NULL) {
			obj = PyObject_GetAttrString(module, "_error_");
        	if (obj != Py_None && obj != NULL) {
				if (PyUnicode_AsUTF8(obj) != NULL) {
        	    	strcpy(msg, PyUnicode_AsUTF8(obj));
				}
        	    err = 1;
        	}
		}
        PyEval_ReleaseThread(interp);
    }
    else {
        PyEval_AcquireThread(interp);
        err = PyRun_SimpleString(msg);
        PyEval_ReleaseThread(interp);
    }
    return err;
}

It runs fine and I get the traceback in my C++ program (copied to the msg char array) when executing single lines. When I execute more than one line though, it does execute fine, but I don’t get the proper traceback.

For example, if I try to execute the following:

def foo():
    print("bar")
        for i in range(10):
            print(i)

The code obviously doens’t run because of the indentation error, but inside the C++ program, the chunk of code above, wrapped in a try/except structure is copied to the msg char array, while in the terminal I get the traceback, which, in the case above is:

  File "<string>", line 5
    for i in range(10):
IndentationError: unexpected indent

How can I get the traceback above in my C/C++ code?

[ZeroIntensity (Peter Bierma)]

The traceback should always be included with the exception object, under the __traceback__ attribute. I see the traceback when running something similar locally:

#include <Python.h>

int
main(void)
{
    Py_Initialize();
    const char *script = "def test():\n"
                         "    0/0\n"
                         "test()";
    int error = PyRun_SimpleString(script);
    if (error < 0) {
        PyErr_Print();
    }
    Py_Finalize();
    return error;
}

$ gcc a.c -lpython3.13 -I/usr/include/python3.13 -o out
$ ./out
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "<string>", line 2, in test
ZeroDivisionError: division by zero

So, to get only the exception without the traceback, someone needs to be printing str(exception) rather than calling PyErr_Print.

At a glance, this could be problematic:

err = PyRun_SimpleString(msg);
module = PyImport_AddModule("__main__");

In the case that PyRun_SimpleString sets an exception, PyImport_AddModule will be called with an exception set, which generally isn’t allowed for C API functions. You need to handle the error before you call PyImport_AddModule and clear the thread’s exception indicator.

[Alexandros (Alexandros Drymonitis)]

I don’t want to just print the error to the console, I want to get it in the char array inside the C program and handle it from there. In your example, you’re calling PyErr_Print() which I guess prints the error to the console. Does PyRun_SimpleString() copy the error to the char array passed as its argument, in your case script?

[ZeroIntensity (Peter Bierma)]

Ah, I missed that from your original question. You can do that using traceback.format_tb. Something like this:

PyObject *exception = PyErr_GetRaisedException();
PyObject *the_traceback = PyException_GetTraceback(exception);
PyObject *traceback = PyImport_ImportModule("traceback");
if (traceback == NULL) {
    /* ... */
}
PyObject *format_tb = PyObject_GetAttr(traceback, "format_tb");
if (format_tb == NULL) {
    /* ... */
}
PyObject *formatted = PyObject_CallOneArg(format_tb, the_traceback);
if (formatted == NULL) {
    /* ... */
}
// formatted is a list of strings containing each line in the traceback, do
// what you want with it.

Alternatively, you could set up a custom file object and call PyTraceBack_Print.

[Alexandros (Alexandros Drymonitis)]

Where is this code supposed to go? After PyRun_SimpleString()?

[ZeroIntensity (Peter Bierma)]

Yeah, essentially you want to emulate the following Python code:

try:
    exec(script)
except BaseException as error:
    tb = error.__traceback__
    import traceback
    tb_str = ''.join(traceback.format_tb(tb))

[Alexandros (Alexandros Drymonitis)]

@ZeroIntensity

Ah, I missed that from your original question. You can do that using traceback.format_tb. Something like this:

PyObject *exception = PyErr_GetRaisedException();
PyObject *the_traceback = PyException_GetTraceback(exception);
PyObject *traceback = PyImport_ImportModule("traceback");
if (traceback == NULL) {
    /* ... */
}
PyObject *format_tb = PyObject_GetAttr(traceback, "format_tb");
if (format_tb == NULL) {
    /* ... */
}
PyObject *formatted = PyObject_CallOneArg(format_tb, the_traceback);
if (formatted == NULL) {
    /* ... */
}
// formatted is a list of strings containing each line in the traceback, do
// what you want with it.

But how do I treat the PyObject structure to get the list of strings from it? What do you mean by list, an array? Can you give an example that prints the strings contained in formatted?

[CEXT-Dan (Dan)]

Maybe you can get inspiration from this

if (PyErr_Occurred() != NULL)
    auto trace = PyRxApp::the_error();

//handled 
PyErr_Clear();

[Alexandros (Alexandros Drymonitis)]

Thanks for this, but this is C++ code. Even though my project is C++, the file where I’m executing the Python bits is a C header file that is common for both C and C++ projects, so I’d like to maintain C coding style.

[ZeroIntensity (Peter Bierma)]

@Alexandros

But how do I treat the PyObject structure to get the list of strings from it? What do you mean by list, an array? Can you give an example that prints the strings contained in formatted?

You can use PyList functions with PyUnicode functions. I think this should work for copying to a C++ vector:

std::vector<std::string> traceback_vector;
for (Py_ssize_t i = 0; i < PyList_GET_SIZE(formatted); ++i) {
    PyObject *tb_line = PyList_GET_ITEM(formatted, i);
    const char *c_str = PyUnicode_AsUTF8(tb_line);
    if (c_str == NULL) {
        /* ... */
    }
    std::string cpp_str(c_str);
    traceback_vector.push_back(cpp_str);
}

[Alexandros (Alexandros Drymonitis)]

I’m trying this out, but the following line produces an error:

PyObject *format_tb = PyObject_GetAttr(traceback, "format_tb");

The error is this:

/home/alex/Applications/openFrameworks/of_v0.12.0_linux64gcc6_release/apps/myApps/livelily/src/m_pyo.h:546:67: error: cannot convert ‘const char*’ to ‘PyObject*’ {aka ‘_object*’}
  546 |                 PyObject *format_tb = PyObject_GetAttr(traceback, "format_tb");
      |                                                                   ^~~~~~~~~~~
      |                                                                   |
      |                                                                   const char*

So PyObject_GetAttr() expects two PyObjects as arguments, so I tried PyObject_GetAttrString() instead which compiles, but my program chrashes when PyException_GetTraceback is called.

GDB produces this output:

PyException_GetTraceback (self=0x0) at Objects/exceptions.c:381
381	    return Py_XNewRef(base_self->traceback);

I changed the code to exit when one of these objects is NULL to the following (I just try to print to the console for now, to see if I can get the full traceback inside my program):

        PyEval_AcquireThread(interp);
        err = PyRun_SimpleString(msg);
		PyObject *exception = PyErr_GetRaisedException();
		if (exception == NULL) {
			return err;
		}
		PyObject *the_traceback = PyException_GetTraceback(exception);
		PyObject *traceback = PyImport_ImportModule("traceback");
		if (traceback == NULL) {
			return err;
		}
		PyObject *format_tb = PyObject_GetAttrString(traceback, "format_tb");
		if (format_tb == NULL) {
			return err;
		}
		PyObject *formatted = PyObject_CallOneArg(format_tb, the_traceback);
		if (formatted == NULL) {
			return err;
		}
		for (Py_ssize_t i = 0; i < PyList_GET_SIZE(formatted); ++i) {
		    PyObject *tb_line = PyList_GET_ITEM(formatted, i);
		    const char *c_str = PyUnicode_AsUTF8(tb_line);
		    if (c_str == NULL) {
		        /* ... */
		    }
			std::cout << "traceback: " << c_str << std::endl;
		}
        PyEval_ReleaseThread(interp);

It doesn’t crash like this, but when I try to execute the following:

def foo():
    print("bar")
        for i in range(10):
            print(i)

I get the output printed by Python and not from my program (I have put this std::cout << “traceback: “ to see if it is printed from inside there).

What am I doing wrong here?

[ZeroIntensity (Peter Bierma)]

1. That’s my mistake, sorry. You want PyObject_GetAttrString, not PyObject_GetAttr.
2. PyRun_SimpleString does not let you handle exceptions, per the documentation:

If there was an error, there is no way to get the exception information.

You want PyRun_String("...", Py_file_input, your_globals, your_locals) instead.

[Alexandros (Alexandros Drymonitis)]

@ZeroIntensity

1. PyRun_SimpleString does not let you handle exceptions, per the documentation:

If there was an error, there is no way to get the exception information.

You want PyRun_String("...", Py_file_input, your_globals, your_locals) instead.

OK, thanks. But what are the three arguments Py_file_input, your_globals and your_locals? Say that I want to execute a function definition. How do I go about this?

[ZeroIntensity (Peter Bierma)]

Python needs to know where to get the globals and locals dictionary. In embedding, that usually looks like this:

PyObject *globals = PyDict_New();
if (globals == NULL) {
    /* handle failure */
}

int result = PyRun_Whatever(/* ... */, globals, globals);

Py_file_input is just telling Python that you want to compile some long source code and not just a single line or expression.

[Alexandros (Alexandros Drymonitis)]

I read the docs (The Very High Level Layer — Python 3.13.7 documentation) and I see that this function returns a PyObject. I can’t tell from the documentation how to use this. Should I replace the line err = PyRun_SimpleString(ms) with a call to this funciton and then go on with the rest of the code I posted above?

Sorry for the endless questions and thanks for all the help so far, I’m trying to get my head around this.

[ZeroIntensity (Peter Bierma)]

@Alexandros

Should I replace the line err = PyRun_SimpleString(ms) with a call to this funciton and then go on with the rest of the code I posted above?

Yeah, and make sure to Py_DECREF the result at the end. Should look like this:

PyObject *result = PyRun_String(/* ... */);
if (result == NULL) {
    // ...
}
Py_DECREF(result);

[Alexandros (Alexandros Drymonitis)]

This is my code so far:

INLINE int pyo_exec_statement(PyThreadState *interp, char *msg, int debug) {
    int err = 0;
    if (debug) {
		int err = 0;
		PyObject *result, *exception, *the_traceback, *traceback, *format_tb, *formatted, *globals;
        PyEval_AcquireThread(interp);
				globals = PyDict_New();
		if (globals == NULL) {
			std::cout << "null globals\n";
			return err;
		}
		result = PyRun_String(msg, Py_file_input, globals, globals);
		if (result == NULL) {
			std::cout << "null results\n";
			err = 1;
			exception = PyErr_GetRaisedException();
			if (exception == NULL) {
				std::cout << "null exception\n";
				return err;
			}
			the_traceback = PyException_GetTraceback(exception);
			traceback = PyImport_ImportModule("traceback");
			if (traceback == NULL) {
				std::cout << "null traceback\n";
				return err;
			}
			format_tb = PyObject_GetAttrString(traceback, "format_tb");
			if (format_tb == NULL) {
				std::cout << "null format_tb\n";
				return err;
			}
			formatted = PyObject_CallOneArg(format_tb, the_traceback);
			if (formatted == NULL) {
				std::cout << "null formated\n";
				return err;
			}
			for (Py_ssize_t i = 0; i < PyList_GET_SIZE(formatted); ++i) {
			    PyObject *tb_line = PyList_GET_ITEM(formatted, i);
			    const char *c_str = PyUnicode_AsUTF8(tb_line);
			    if (c_str == NULL) {
					std::cout << "null c_str\n";
					return err;
			    }
				std::cout << "traceback str: " << c_str << std::endl;
			}
			return err;
		}
		Py_DECREF(result);
        PyEval_ReleaseThread(interp);

    }
    else {
        PyEval_AcquireThread(interp);
        err = PyRun_SimpleString(msg);
        PyEval_ReleaseThread(interp);
    }
    return err;
}

I put all the previous code inside the if (result == NULL) test because when I was trying to execute foo I was getting null results printed in the console, so I figured that result is NULL when there’s an error.

If I run something like foo I get null results and if I then run print(“foo”) my program crashes with this output

Fatal Python error: _PyThreadState_Attach: non-NULL old thread state
Python runtime state: initialized
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'foo' is not defined

Extension modules: pyo._pyo (total: 1)

(Note, I’m running this from Pyo’s C embeddings, see m_pyo.h here pyo/embedded at master · belangeo/pyo · GitHub, that’s why there’s this line at the end).

Running this with GDB, I got this output:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007ffff5e9e9ff in __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
#2  0x00007ffff5e49cc2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff5e324ac in __GI_abort () at ./stdlib/abort.c:73
#4  0x00007ffff788b4f3 in fatal_error_exit (status=-1) at Python/pylifecycle.c:3148
#5  fatal_error (fd=<optimized out>, header=header@entry=1, prefix=prefix@entry=0x7ffff7bba5b0 <__func__.5> "_PyThreadState_Attach", msg=msg@entry=0x7ffff7bba490 "non-NULL old thread state", status=status@entry=-1) at Python/pylifecycle.c:3294
#6  0x00007ffff7aa75d3 in _Py_FatalErrorFunc (func=func@entry=0x7ffff7bba5b0 <__func__.5> "_PyThreadState_Attach", msg=msg@entry=0x7ffff7bba490 "non-NULL old thread state") at Python/pylifecycle.c:3380
#7  0x00007ffff7aaab21 in _PyThreadState_Attach (tstate=0x5555561bee20) at Python/pystate.c:2084
#8  _PyThreadState_Attach (tstate=0x5555561bee20) at Python/pystate.c:2074
#9  0x000055555565fd33 in pyo_exec_statement (interp=0x5555561bee20, msg=0x555555f12c64 "print(\"foo\")", debug=1) at /home/alex/Applications/openFrameworks/of_v0.12.0_linux64gcc6_release/apps/myApps/livelily/src/m_pyo.h:540

The line of the m_pyo.h file is different as I’m adding some stuff there (mainly printing to the console) while trying to get this right. The error refers to PyEval_AcquireThread(interp);.

I don’t know if I’m even close to getting this right. Any tip is welcome.

[ZeroIntensity (Peter Bierma)]

How is this being called? You generally don’t need to call PyEval_AcquireThread yourself at all.

[James_David]

The reason you don’t see the traceback is because IndentationError and SyntaxError happen at compile time, before your try/except ever runs. That’s why single-line code works but multi-line with bad indentation doesn’t.

Fix:

Instead of wrapping user code directly in try:, you need to compile first, then exec inside the try.

Quick Python wrapper idea:

import traceback
_error_ = None
try:
    _src_ = r"""<user_code_here>"""
    code = compile(_src_, "<string>", "exec")
    exec(code, globals(), globals())
except Exception:
    _error_ = traceback.format_exc()

Or in C API:

1. Use Py_CompileString(msg, "<string>", Py_file_input).

   • If it fails → fetch error with PyErr_Fetch + format with traceback.format_exception.

2. If it compiles → run with PyEval_EvalCode, and again catch runtime errors the same way.

Bottom line: move compilation into the try. That way you’ll also catch IndentationError/SyntaxError and can copy the full traceback into msg.

[Alexandros (Alexandros Drymonitis)]

@James_David

Or in C API:

1. Use Py_CompileString(msg, "<string>", Py_file_input).

• If it fails → fetch error with PyErr_Fetch + format with traceback.format_exception.

2. If it compiles → run with PyEval_EvalCode, and again catch runtime errors the same way.

Bottom line: move compilation into the try. That way you’ll also catch IndentationError/SyntaxError and can copy the full traceback into msg.

Thanks for the tip. I know I’m asking you to write code for me, but could you provide (at least a minimal) example of how this is done in C?