I have some rather slow Python code that I use frequently. I’d like to have the option of running its main loop in C++20, and I’d strongly prefer to use HPy for that instead of “the” C extension interface.
I have a Cython version of that loop, and it’s faster, but not by a lot. And I’ve been mostly running it on Pypy lately, but even that’s rather slow.
Are there any tools, mature or nascent, that facilitate creating HPy modules from C++? It looks like for both Cython and pybind11, the C extension module interface is the only way.
I don’t know about pybind11 but Cython does not support HPy yet. If/when it does though the Cython code you would write would still be the same but would probably just have some compile time directives for building against the HPy interface. Either way the CPython C API or the HPy API would be mostly behind the scenes.
If the issue is that your Cython code is slower than you could get with C++ then you can just call the C++ code from Cython. If the issue is that the Cython code is slow because it is calling lots of Python code then using C++ is unlikely to help.
I prefer HPy because it’s better abstracted, is much faster on Pypy, and requires less recompiling. I’m willing to invest a little time in it, because it strikes me as the way of the future.
Yes, there’s code with its logic in Cython, and then there’s code with its logic in C/C++ that is called by Cython. A little uncomfortably, they’re both sometimes called “Cython”.
I want the module to run well on Pypy though, and the C extension API doesn’t really do that.
My loop doesn’t call back into Python. It’s just chopping up a stream of bytes into content-defined blocks.
I use C++20 with boost::python, in an embedded context, in Python for AutoCAD
I mention embedded context, because I’m wondering what you mean by “main loop”. In my case, AutoCAD is the main loop, embedded Python makes calls in to C++ via boost wrappers.
boost::python is pretty fast, 2x faster than pybind11 in my case, the caveat is that it has lousy docstrings. Also, there’s little tricks you can do to eke out performance depending on your data, and how it’s being used in Python
Anyway, you can ignore me if it’s not applicable to you project