Packaging a C extension with a dependency on another C extension

Packaging
1

Hi,

Assume the following situation:

There is a C/C++ library called LibA, and a Python binding for LibA, called PyLibA (in my case, LibA = Qt5 and PyLibA = PyQt5).

There is a C/C++ library called LibB, which depends on LibA. I want to create a Python binding for that library, called PyLibB (in my case, LibB = poppler-qt5 and PyLibB = python-poppler-qt5). The Python binding is built on top of PyLibA. E.g., if there is a LibB function returning an instance of a LibA type, my binding returns an instance of the corresponding PyLibA type.

Since building C/C++ extensions is complex (have a look at Issues · frescobaldi/python-poppler-qt5 · GitHub, almost all issues are about installation…), I would like to provide wheels for PyLibB on PyPI.

In PEP 491, I read

The wheel filename is {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.

(here)

My problem is that my wheels depend not only on the Python tag (i.e., Python implementation) and the (Python) ABI tag as well as the platform tag, but also on the version of LibA that is used, since my PyLibB extension links into PyLibA and therefore into LibA.

My question is this: is there a way, for one PyLibB version, to distribute wheels for several PyLibA versions? Or should I just pin the PyLibA version and distribute wheels for that version only, constraining all users of my PyLibB library to use a specific version of LibA?

Also, is conda maybe more appropriate for that sort of thing than the PyPA ecosystem? (I’ve never used it.)

2

You might want to see how Qscintlla does this.
See QScintilla · PyPI

Also there is a mailing list to support PyQt that has experts on PyQt.

https://www.riverbankcomputing.com/mailman/listinfo/pyqt

1 Like
3

NB, you might want to move this to the Packaging category instead—you’re likely to get better expert help there.

I’m not a C extensions expert, but a somewhat similar problem crops up with things like different CPU/GPU/CUDA flags/implementations for PyTorch, TensorFlow, etc. You can’t do it with different wheels for the same distribution name, but there are some ways to achieve this—PyTorch does it with an extra index URL with different indices for different CUDA versions that serve the correct wheels, but you could also do it by just having different dependencies in a metapackage or “core” package that doesn’t contain any binding-specific code, delegating to binding-specific dependencies via extras.

Yup, this is one of the things Conda was designed to handle better than the PyPA tools. You can have a single package recipe with multiple outputs, one built against each version of the binary dependency with different dep specs, and give them different build tags accordingly. You can also have constraints that don’t specify a hard requirement, but rather just require a specific version of a package if it is installed. And you can directly specify the supported Qt versions to be solved against and installed, if needed. And you can also more easily create meta-packages and feature flag packages to control the dep version.

4

Good idea, I’ve posed on that mailing list.

5

OK, I’ll do that next time. I was a bit afraid of posting about user problems, interpreting that category as intended for development of packaging tools and not for questions. But I guess complex packaging problems call for discussions on improving the tools as much as on how to work around their limitations.

I don’t think I have the permissions to move this topic now that it’s been created, though?

Thanks, good to know I’m not missing something obvious. The workarounds do sound complicated.

Thanks again. I’ll take a look and see if this ends up easier. python-poppler-qt5 was mainly developed for the benefit of one app (Frescobaldi) and I am mostly interested in this app, so I will most likely end up requiring PyQt5 == x.y in my wheels in the near future (edit) short term, but it would be nice to make the library more generally useful outside of Frescobaldi and that may entail distributing it on conda.

6

You should be able to at least as TL2, just click the edit button (:pencil2:) next to the thread title and you should get the option in a dropdown over the category name. I know I can move others threads as a TL3 (but I haven’t been as active for a while so I got downgraded, otherwise I’d do it for you).

7

Ah yes, thanks, I thought that pencil icon was for editing the title only.

1 Like
8

Yes, and in fact there are already separate conda packages for qt and pyqt (as opposed to PyPI, where a binary PyQt5 package bundles the underlying Qt with it).

1 Like
9

I have a question related to this. Suppose I accept the fact that I’ll be shipping my extension with a pin of the PyQt5 version. Is there a way to do this only for the wheels, but leave a lax requirement in the sdist for people building from source?

10

N.B. python-poppler-qt5 uses sip as a build backend.

11

I’m not sure it is useful to let the sdist not be pinned.
You will have only tested with that pinned version of Qt.

If you are packaging for debian or fedora then that what you do on PyPI will not matter.

Also until you have ported to Qt6 (Qt5 is close to EOL I think)
that will be a different source.

12

Good point. Ideally, I would test with more versions… but I don’t have the time.

Not sure what you mean by “a different source”?

And, yes, I know we should port to Qt6. Basically, we’re a few volunteers trying to take over maintenance of a project (the Frescobaldi music sheet editor) whose initial author doesn’t have time for it anymore. First we want to fix lots of packaging / distribution / installation issues, then the next thing to look at will probably be Qt6.

13

I am not familiar with the C++ porting works.
At least with PyQt6 i ended up with lots of changes, especially around enums.

It did not look easy to have qt5 and qt6 in the same source files.

14

By the Qt Compnay, perhaps, due to their new strategy to phase out FOSS LTS releases and push their commercial releases instead, but few folks get Qt directly from them and distributors (Conda, Linux distros, Homebrew, KDE) are likely to support it for many years to come, like with Qt4 and GTK2. My impression of existing PyQt/PySide users is that a substantial majority of production installs (at least on the scientific and related side) are still using Qt5. In fact, some even only dropped Qt4 support in the last few years.

Yeah, it’s not easy. I help maintain QtPy the most widely used compatibility layer between PyQt and PySide, and Qt5 and Qt6 (previously Qt4 and Qt5 until the past couple years). Of course, that’s aimed at the Python level, not C API users, though perhaps it will be at least a helpful guide.