When using python as an embedded interpreter it would be nice if there was an easy way for the host to force the script to end. This could be used for run away scripts or to implement a timeout mechanism.
Web searches turned up suggestions to call PyErr_SetInterrupt() but that can be masked with while 1 try type code. Solutions I found for timeouts got more involved with wrapping user code in python threading stubs
Poking through ceval.c _PyEval_EvalFrameDefault it was pretty easy to add a callback to the host app and then goto exit_unwind on command.
The callback also allows the host UI time to process messages and remain responsive. A script timeout is then easy to implement.
I searched the web quite a bit looking for a solid built in solution, but still pretty new to the code base if something better already exists.
While interrupts can be caught, it’s uncommon for good code to try catch
BaseException. I haven’t run an embedded interpreter so I don’t know the most common use-cases, but running unverified, untrusted scripts seems like a bad idea.
my use case is allowing end users to automate the application similar to office vba.
Also allows me to use complex GUI apps as basically code libraries while keeping the data display features and automate code testing in bulk.
my apps have a very small user base of highly technical users so they can manage their own risk on untrusted scripts.
Generally speaking, what you want here is a very hard problem if signals (such as SIGINT) are insufficient. It gets to the point of impossibility (well, impossible to do safely) depending on if the embedded Python interpreter is completely unrestricted in nature, and you’ll find some parallels with discussions on why we can’t just kill threads and the problems with daemon threads in non-embedded use cases.
There are ways to prevent signal handling that don’t even involve registering a signal handler that ignores signals or catching BaseException. One example of this, is signals aren’t handled by the interpreter while native code is executing, being handled when next possible for the interpreter to do so. If that next possible moment never happens due to being indefinitely in native code with the GIL held, the signal is never handled.
Simply killing the embedded process from the perspective of an observer works, but abruptly killing a process with SIGKILL that isn’t handling signals anymore for whatever reason can lead to open resources not being cleaned up properly. As you want this to interact with an application to automate it, this may not be desirable.
One possible method of balancing the issues here would be to parse user-provided python yourself and ensure only a limited subset of things you consider “safe” are acceptable here, and then just execute it.
Be aware that there are many things that may not be safe in this regard, and that it is difficult to predict everything that might be unsafe.
max(itertools.count()) might look obviously problematic to a human reviewer specifically looking for someone trying to do this, it’s harder to detect programmatically all the ways someone could cause a problem here.
A slightly stronger solution here would be to write your own scripting mini-language, parse that, and execute it. There are various parsing libraries for python that would enable this while keeping the ease of a debug interface you find python provides for you.
I don’t know what a good technical solution would be, maybe a customized interpreter that provides some sort of heartbeat? Otherwise, I would just document that catching
BaseException is a bad idea, and provide the users some way to kill the entire process remotely.
thanks for the detailed response. I havent tested it with threads yet. Since I am already using a modified python311.dll I am actually tempted to remove support for threading since its beyond what I would need.
For resource cleanup, I was thinking of exploring the garbage collector and seeing how it kept track of objects after PyFInalize (in case my ref counts get off). Could also come in handy for use as a live object viewer for the debugger part.
What I am trying to start with is pretty basic and hopefully not to dangerous. Seems to be working in my tests so far: Coders Corner - python abort / script timeout
It seems like the generic try except eats the interrupts by default. Automation code is usually quick one offs so I would not expect anyone to limit the exceptions they handle. I already had to hook exit and quit so they would not shut down the host application.
The heart beat aspect of the callback is actually really important since it allows the UI to stay responsive and the timeout timer to tick. (I am limited to a single threaded UI)
I figured there had to be a reason something like this wasnt implemented yet. Wasnt sure if a solution was already baked in or maybe that embedding was just not a very popular use.