Hi all, I package things for nixpkgs and have been following this thread for a while. Instead of adding more opinions I wanted to try PEP 725 out on Nix and see what happens, so here’s what I found.
Some context on why I think Nix is a useful test case: in nixpkgs every package spells out its build tools (nativeBuildInputs) and the libraries it links against (buildInputs), and builds run in a sandbox where nothing is in /usr. If something isn’t declared, the build fails. So it shows pretty quickly whether [external] metadata is complete. I also have a selfish interest here. I’d love to generate Nix expressions for Python projects from metadata (say, from a uv lock file) instead of writing overrides by hand for every package with a C dependency.
The experiment
I took the [external] tables @rgommers wrote for the 37 compiled packages in external-deps-build, mapped them to Nix packages using nix.mapping.json from the PEP 804 registry, filled in the Python side from each sdist’s [build-system] and [project] tables, and generated a buildPythonPackage for each. Then I built a handful and compared all 37 to what nixpkgs has today. The code is here: GitHub - arunoruto/pep725-nix: Experiment: generating nixpkgs derivations from PEP 725 [external] metadata · GitHub
| Package | Result |
|---|---|
| numpy 2.4.4 | built as generated, links nixpkgs’ BLAS/LAPACK |
| pandas 2.3.3 | built as generated |
| cryptography 48.0.0 | built after two Nix-specific fixes (crate hash, maturin hook) |
| pillow 12.3.0 | needed pkg-config added, 3 optional libs still not picked up |
| scipy 1.17.1 | needed -Dblas=blas -Dlapack=lapack |
| pyyaml 6.0.3, cross to aarch64 | built as generated |
| all 37 compared with nixpkgs | 18 identical external deps, 19 differ |
Cross-compilation
This was the part I was most curious about, given the long build/host discussion above. nixpkgs already works this way: under cross-compilation, nativeBuildInputs come from the build platform and buildInputs from the host platform. When I evaluated the generated files for aarch64, every build-requires entry turned into an x86_64 tool targeting aarch64 (cross gfortran, cross rustc, pkg-config), and every host-requires entry into an aarch64 library. The pyyaml cross build then just worked, no changes needed.
nixpkgs uses the same build/host/target terms internally, so for what it’s worth, the current naming worked for us without any translation.
The virtual packages fit too. nixpkgs has blas and lapack packages that wrap a switchable implementation, and dep:virtual/interface/blas maps straight onto them.
Where it broke
Interestingly, most failures weren’t wrong dependency lists. The dependencies were there, but the build didn’t find them or didn’t pick the right one:
- pillow had every library it needed and still didn’t detect imagequant, openjpeg and xcb. The nixpkgs pillow expression has the same problem and patches
setup.pywith hard-coded paths. - “Any BLAS” is the right thing to declare. numpy finds nixpkgs’ BLAS on its own, but scipy looks for OpenBLAS by name until you tell it otherwise.
- nixpkgs builds scipy against system boost/qhull/xsimd and aiohttp against system llhttp, while the tables describe the default build with bundled copies. That’s not wrong, but there’s no way to say “can use X from the system instead of the bundled copy”.
- Rust packages need a hash of the vendored crates, and
maturinis a setup hook in nixpkgs rather than a Python module. That’s purely our problem.
I don’t think PEP 725 needs to solve any of this. Distros will always carry some per-package fixes. It might be worth saying in “How to teach this” that [external] tells you what a package needs, not how its build finds it, so nobody expects the metadata alone to make builds work.
The tables
A few things I noticed:
- pillow’s table is missing
dep:generic/pkg-config. Without it pillow’ssetup.pyonly looks in the usual FHS paths and can’t find zlib. - numpy’s table lists a Fortran compiler (so does nixpkgs, to be fair), but numpy 2.4.4 builds fine without one.
- pillow 12 added AVIF support, and the table is older than that.
None of this is a criticism of the tables, which were written for an experiment. But it shows how quickly tables kept outside a project go stale, which I think is a good argument for projects owning their own [external] table. It also means a sandboxed build with only the declared dependencies works as a cheap completeness check.
PEP 804
One thing that’s relevant to the comparison @rgommers suggested: the only commands the mapping describes for Nix are imperative installs (nix-env -iA, nix-shell -p). We never install anything that way when packaging. What we need is the package name to put into the derivation. I’d guess conda-forge recipes and distro spec files work the same way. So for Nix, PEP 725 plus the registry of canonical names and per-ecosystem mappings is what we’d actually use, and the standardized install/query commands matter a lot less.
Happy to rerun this as the tables or the registry change, or to try other packages if anyone’s curious about a specific one.