PEP 725: Specifying external dependencies in pyproject.toml

[rgommers (Ralf Gommers)]

Thanks for all your thoughts @newkozlukov.

@newkozlukov

My understanding is that the motivation for this PEP, at least partly, is to facilitate easier cross-compilation for python packages. Do you think it would make sense to include a deprecation plan for ctypes.util.find_library in the PEP and start discouraging python package developers from using it?

Cross compilation is important, but not a primary motivation for this particular PEP. It is more a case of “we want good cross compilation support too at some point, hopefully soon, so let’s get the cross-compilation aspects of external dependencies first-time-right”.

I’d much prefer to not mix that into this PEP, since it’s a fairly orthogonal topic. However, I’d certainly discourage Python package developers from using ctypes (and certainly find_library) - there’s a host of reasons for why that is not reliable. Okay for prototyping or quick hacks, or very specific usage scenarios within a single package - but when things go wrong it’s much better to build a Python extension module and get an error at build time, rather than weird errors, crashes or incorrect results at runtime.

Your examples of how Python packages use find_library shows that we indeed need a good alternative. I’d say that that is pkg-config indeed, and that we could at least mention that as the better method in the ctypes.util.find_library docs.

@newkozlukov

but I’d like to add that “virtually always” (yes, that’s a lie) native dependencies come with either CMake or pkg-config targets maintained upstream (e.g. meson is capable of generating .pc files, unless I’m mistaken)

Yes, it is a bit of a lie - but to make the situation better, I think there is little choice but to start using .pc / .cmake files more and treating their absence in a distro as a bug.

There is of course a complementary problem: we still need to solve the issue that you cannot sensibly distribute .pc/.cmake files inside a Python package (or more accurately, inside a wheel). It’s not hard to fix conceptually, but adding a new component/location to install schemes is in practice quite a bit of work. I think if that were to be tackling, it’d be best to tackle all this in one go and add it inside site-packages:

include/
lib/
lib/pkgconfig
lib/cmake

We have this problem for numpy right now - I’d really like to ship a numpy.pc file because there are so many NumPy C API users, but there is nowhere to install it to.

@newkozlukov

I don’t think it’s immediately obvious that we’d want the python packaging community to maintain standards and registries describing things like cxx_lambdas or cuda_std_17, as opposed to explicitly offloading this task to existing tools

Agreed. I’d draw the line at a standard version, but not individual language features - that’s too detailed, and best left to build systems.

@newkozlukov

I maybe have one question right away that I didn’t find answers to in the comments: are the declared “external dependencies” to be validated by python build “front-ends” (not sure, what’s the appropriate name to summarize pypa-build and pip) prior to the build and how?

Those are front-ends, yes. That validation is not specified on purpose. For one because prescribing behavior of individual tools tends to be out of scope for packaging PEPs. For another, because it may shift over time. I’d imagine that we add metadata to packages first, then at some point front-ends will start using that to add better diagnostic messages. And then I’d expect a more experimental front-end to start validating more strictly and/or offering users to opt into installing external dependencies automatically. pip is probably going to be the last to do that, since it has to be quite conservative given its role and large user base.

@newkozlukov

On the other hand, a wheel built in reasonable isolation is already an evidence in support of correctness

Yes, that is what I’d like to rely on. It only needs a single CI job on one platform for a package to validate this. And anyway, most dependencies won’t change quickly or at all.