Minimal debugger for C++ application Python script

Hi,

we’ve a C++ Qt application that’s scriptable by Python. We would
like to have a minimal debugger when such a Python script is executed
with the application. Being able to set breakpoints, single stepping and
stepping into function calls.

The C++ application and the Python interpreter have to run in the same
thread, because the Python script modifies the state of the C++
application. When a Python breakpoint is reached the C++ application shouldn’t
be blocked, it should still be possible to interact with it.

I looked into the Python trace API, but when running in the same thread it
would block the C++ application. I wonder if async/await could be
somehow used to create a Coroutine at the breakpoint and later continue
it?

Any ideas or hints are highly appreciated!

Greetings,
Daniel

Hi again,

at the moment I’m playing with the idea to execute the Python script inside a sub Python interpreter. So having a separate Python interpreter for the running C++ application and the Python script execution. That could allow me to use the Python trace API and suspend the execution on a breakpoint with a stackfull coroutine, like the ones from the boost C++ library.

Has someone tried something like this? Any hints?

Greetings,

Daniel

Hi Daniel,

I wouldn’t recommend either of those two approaches.

async/await won’t solve this because suspension occurs only at explicit coroutine suspension points (await); it doesn’t provide a mechanism for externally suspending arbitrary synchronous Python execution at an arbitrary bytecode boundary.

Swapping native stacks out from under an active interpreter with Boost fibers is also dangerous. CPython’s evaluation machinery maintains C-level execution state associated with the active native call stack-such as the active cframe, recursion state, and C-level exception-unwinding context. Swapping to another native stack and later resuming the interpreter can therefore leave the evaluator in an inconsistent state, particularly across different CPython versions.

If Python has to run on the main thread because scripts directly interact with non-thread-safe Qt/C++ state, a practical approach would be to block inside your trace/monitoring callback with a nested QEventLoop.

The conceptual flow is:

Python execution
   ↓
CPython frame evaluator
   ↓
Trace / monitoring callback
   ↓
Breakpoint detected
   ↓
QEventLoop::exec()
   ↓
User clicks Continue
   ↓
QEventLoop exits
   ↓
Trace callback returns
   ↓
CPython continues execution

The important point is that this doesn’t suspend the interpreter by switching its native stack. The Python thread remains inside the trace callback on the same native stack while Qt processes events.

In C++, roughly:

void onBreakpointHit() {
    emit debuggerPaused();

    QEventLoop pauseLoop;

    auto c1 = connect(
        ui->continueBtn,
        &QPushButton::clicked,
        &pauseLoop,
        &QEventLoop::quit
    );

    auto c2 = connect(ui->stepBtn, &QPushButton::clicked, [&]() {
        enableSingleStep();
        pauseLoop.quit();
    });

    pauseLoop.exec(QEventLoop::AllEvents);

    disconnect(c1);
    disconnect(c2);
}

The subtle danger here is re-entrancy and object lifetime. A nested Qt event loop changes the normal control-flow assumptions of the application. You need to ensure that the nested event loop cannot synchronously re-enter the same Python interpreter, start another script, or invalidate/delete C++ objects that the paused Python call stack is still referencing.

If that re-entrancy/lifetime surface is too brittle for your architecture, another option is to move the Python interpreter to a dedicated worker thread, use a QWaitCondition for breakpoint waits, and marshal script-to-widget calls back to the GUI thread with QMetaObject::invokeMethod(..., Qt::BlockingQueuedConnection). That cleanly separates the debugger wait from the UI loop, although it introduces the usual deadlock considerations around blocking queued calls.

Hope that helps!

Hi David,

thank you for your reply!

You need to ensure that the nested event loop cannot synchronously
re-enter the same Python interpreter, start another script, or
invalidate/delete C++ objects that the paused Python call stack is
still referencing.

I can’t ensure this, because the C++ application uses Python bindings to
its C++ objects to implement the logic of its GUI. Interacting with
the C++ application in any way also means Python gets executed in the
interpreter.

Swapping native stacks out from under an active interpreter with Boost
fibers is also dangerous. CPython’s evaluation machinery maintains
C-level execution state associated with the active native call
stack-such as the active cframe, recursion state, and C-level
exception-unwinding context. Swapping to another native stack and later
resuming the interpreter can therefore leave the evaluator in an
inconsistent state, particularly across different CPython versions.

But is this still an issue if I would use a separate Python interpreter
instance for the debugger? How could it then come to an inconsistent
state? The C++ application could still use its own Python interpreter
and wouldn’t mess with the one of the debugger.

If that re-entrancy/lifetime surface is too brittle for your
architecture, another option is to move the Python interpreter to a
dedicated worker thread, use a QWaitCondition for breakpoint waits,
and marshal script-to-widget calls back to the GUI thread with
QMetaObject::invokeMethod(..., Qt::BlockingQueuedConnection). That
cleanly separates the debugger wait from the UI loop, although it
introduces the usual deadlock considerations around blocking queued
calls.

Moving the Python interpreter into a separate thread also isn’t an
option, because of the Python bindings to the C++ objects.

Greetings,
Daniel

I think these are incompatible requirements. You need python to be synchronous with the C++ application thread, but you also need it to be asynchronous to allow C++ to continue executing when python is blocked in the debugger. I would be very skeptical of the validity of debugging asynchronously because the state will not be protected as it normally would be.

Hi Daniel,

Lonnie makes a fair point about the design conflict here. But on the sub-interpreter idea specifically: it doesn’t bypass the native-stack issue.

Sub-interpreters isolate Python-level interpreter state, such as sys.modules, builtins, and module state, but they don’t provide a separate native C stack for each interpreter. CPython can have multiple interpreters in the same OS thread and switch between them by changing the active PyThreadState. The CPython documentation on multiple interpreters describes this interpreter/thread-state model.

So if you swap native C stacks underneath an active evaluation loop using Boost fibers and then try to execute Python code for GUI callbacks on that same OS thread, you’re still dealing with the same fundamental problem: the interpreter’s C-level execution state is intertwined with the currently executing native call stack. A sub-interpreter doesn’t make arbitrary native-stack switching safe.

There’s also another complication with the Qt bindings. C/C++ extension modules aren’t automatically isolated between sub-interpreters. The Isolating Extension Modules HOWTO explains the problems that can arise when extension modules rely on C-level global state rather than properly isolating state per interpreter.

The bigger hurdle, though, is the workflow itself. If the GUI logic is driven by Python and you pause a script at a breakpoint, you can’t simultaneously treat that Python state as safely mutable while arbitrary GUI callbacks continue running. For example, if the paused script is halfway through modifying a widget hierarchy and the user clicks a button that invokes another Python callback which expects that same state, you’ve introduced re-entrancy into the paused execution.

In practice, I’d make the breakpoint pause effectively modal. A nested QEventLoop can keep Qt responsive enough to repaint the debugger UI and process debugger commands, while the normal application interaction is disabled. The debugger can expose only things such as Continue, Step, and variable inspection.

If the requirement is instead that the user must be able to freely interact with the GUI while the script is paused, I’d decouple the two execution contexts entirely: run the script on a worker thread, communicate through signals/message queues rather than holding raw widget pointers, and have the worker wait on a condition variable when it reaches a breakpoint.

That gives you a much cleaner separation between the GUI event loop and the debugger’s paused execution state.

Hope this helps clarify the trade-offs.

These won’t work together, so one of them has to go. The easier option would be to do your debugging differently, and not set breakpoints; but if you definitely want breakpoints, you’ll need to rework your C++/Python boundary. TBH this is probably worth doing anyway; Python code can then take as long as it likes, without making the Qt application window nonresponsive.

Kinda, but it’s overkill. Fundametnally, what you need is for your C++ app to signal [1] Python to say “please do your stuff” (eg “this event happened” or whatever your context is), then hold whatever continued action is waiting on that. Flag it as pending. Set up a callback for “when the Python code is done, carry on from here”. An async function is basically that, but designed for lots and lots of interruption points; from the sound of things, that wouldn’t be necessary for what you’re doing.

So you’ll need to rearchitect the C++ app a bit, separating out the “before Python” and “after Python” parts, with Python itself running on a separate thread (or even process). The easiest way to do that would be to pass the Python function a callback that it has to call for the thing to be done:

def thing_happened(state, done):
    # examine and optionally manipulate state
    done()

This can be error-prone as forgetting that call or raising an exception would result in the original action not happening, but it’s a good start.


  1. not in the Unix sense ↩︎