Wanting a singular packaging tool/vision

Python has a way, way, WAY harder context here than rust[1], since it exists as a glue language to just about everything. So without a comprehensive and cross-platform answer how to deal with non-python artefacts (building, distribution) that live in python projects - anything from C/C++/Fortran/Rust/Java/NodeJS/CUDA… - a cargo-like experience is very far away. And doubly so for a solution that doesn’t just hard-copy binary artefacts (e.g. openblas) into wheels, redundantly across packages.

Poetry is well-loved for its UX, but suffers from the same issues as all wheel-based installers – i.e. no comprehensive & portable solution how to deal with non-python dependencies. Conda/mamba has the most complete (though certainly not perfect) answer to this set of problems, but doesn’t get much recognition (especially on DPO), for reasons that elude me - even though it’s effectively been offered for that purpose.

Perhaps this is because packaging is an unglamorous task with lots of responsibility, and consequently tends to attract few people, and those that do work on the topic regardless tend to be people who don’t require much recognition, but have lots of opinions:tm: about how things should be done (witness the proliferation of python build tools/frameworks, or almost any packaging PEP discussion).

Don’t get me wrong, I’d love to see python packaging make a big leap towards a cargo-like experience, but it’d IMO need a project that’s at least as organised/funded as faster-cpython, for a likely even larger body of work[2], in a fractious ecosystem, for – let’s be honest – a topic that’s way less sexy (=harder to sell) than performance.

Undoubtedly this is something the SC could push for, I just fear that the problem space is so big that inevitably people go “nah, not touching that”, which is pretty much what happened periodically ever since numpy/scipy first ran into these issues (ironically, that was the catalyst for conda being created), and settling on standardizing a variation of the most comprehensive existing solution seems to be completely off the table due to non-technical(=human) constraints.


  1. where dependencies are mostly mono-lingual and can be built with the same build system, which conveniently also comes with a uniform compiler everywhere ↩︎

  2. for reference, even the huge C++ ecosystem famously has no standardized solution for building/distribution, and this is but a subset of the problem a cargo-for-python would have to tackle at least partially. ↩︎

12 Likes