Hello all, Victor asked me to share my past experience with the stable Python API wrt. integration of Python as a scripting language/console into a C++ GUI application. So here goes my reply that I already sent to him by email previously:
A bit of background first: Our application is a large complex C++ code base
with lots of dependencies targeting all three major desktop platforms. We use
Python both externally (for headless data processing, e.g. UMAP) as well as
internally for an integrated python console that allows biologists to run
python scripts with access to some API that allows automation of tasks in the
C++ GUI.
Originally, we hoped to be able to use the stable python ABI to allow
biologists to “bring your own python”. The idea was that they probably have a
custom set of python libraries and code that they would like to continue
using. Our integrated API - so we thought - was a tiny addition that should
work with any Python out there, so we used the stable ABI.
This turned out to be a dead end, and I believe we can (should?) now use the
non-stable ABI of python. Allowing end users to BYO Python caused far too much
setup problems and support issues for us that it was not worth it in the end.
Instead, we now rather want to ship a custom Python with a custom prefix that
they can pip install custom libraries into as needed.
The problems we faced are not directly related to the stable ABI - quite the
contrary. Rather, it was due to thirdparty python libraries that we shipped
which themselves are not compatible across python version increments. E.g. for
the integrated console we use qtconsole/jupyter, which worked in an archaic
version with python 3.9 but requires newer versions for python 3.11+.
The ton of dependencies pulled in by UMAP was even worse, with numba and
pydnndescent and llvmlite often taking months to support newer python
versions.
Finally, we faced tons of issues when we tried to combine different python
prefixes - e.g. the user has Python 3.X and we ship Python 3.Y, both having a
prefix of their own with some PIP packages installed. This often led to
incompatibilities, when e.g. some package dropped (or added) a third party
dependency which was then picked up in another prefix.
Overall, I can safely say that integrating Python as a scripting language into
an application in a meaningful way (i.e. not just the interpreter, but also
some thirdparty/PIP packages) is probably one of most time draining
experiences I have had in my professional career.