(NixOS developer, maintain some Python packages in nixpkgs and has knowledge of the issues we encounter classically with the Python packaging ecosystem wrt to native dependencies. I don’t represent the NixOS project or community.)
I read the whole thread and found the remarks of @steve.dower very interesting. Firstly, I have to say we know that Nix ecosystem is probably a small subset of Python users and probably the one for which Python packaging works more or less the best in our closed garden. We have native cross-compilation support to a certain extent, and I can even trivially run stuff on RISC-V without thinking too much about what needs to be done.
The concept of a key infrastructure and letting distributions & system integrators provide packages for users is compelling for us, as this is 99 % what you need to do on NixOS. (of course, we have escape hatches, but this is a very much desired state in my perspective).
Currently, we have around ~5K7 Python packages (not all of them are working) and we do not support all versions. It’s already a lot of efforts to manage this set of packages, notably because of those native dependencies and various intricacies caused by pleasant features which does not work that well in our model (setuptools-scm for example.)
Sometimes, we have posts such as https ://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741 which shows the difficulty to do this at scale on particularly challenging pieces of software, e.g. TensorFlow.
We have many attempts to bottle Python packages automatically into our Nix ecosystem, I won’t do a list of them, one of them which works really well is
poetry2nix (which has support for multiple build backends in fact: https ://github.com/nix-community/poetry2nix/blob/master/overrides/build-systems.json).
Unfortunately, this labor of love (IMHO) requires careful overrides to represent many of the lacking information in the packaging metadata: poetry2nix/default.nix at master · nix-community/poetry2nix · GitHub ; cryptography being one of the most difficult case : https ://github.com/nix-community/poetry2nix/blob/master/overrides/default.nix#L373-L421 where we need to manually pin the vendor hashes of the Cargo dependencies for each release.
So, even if,
pip, let’s say would use our Nix expressions to download our binaries or kick in some “from source” compilation using the right environment, it would still force some level of efforts on the maintainer side and may not provide what a user expects, i.e. all versions of a PyPI package.
Given the current discussion, I am not certain what will be done to address such things and whether current policies will impose everyone to find solutions downstream.
More specifically, multiple people talked about more standardization, for example, I wonder if Python considered integrating CUDF formats into his core and if there are reasons to not adopt this.
Building on SBOM standardization process, if there was a way to specify native dependencies using a more or less unique identifiers, I think, it would help drastically the ecosystem.
Also, I have to add that Nix community is probably very interested into all of this and is interested into helping how they can and collaborate on this, we probably don’t have anything for Windows (except through WSL2 I suppose or if someone finishes a Nix for Windows at some point) though.
BTW, I see that other language ecosystem were mentioned, I have to say that I’m not exactly sure that Node.js or Java is somewhat better at all, as distribution maintainers, we sometimes have worse problem than Python ecosystem which provided stuff like
site and all to enable our use cases.
Thank you for your time and the interesting exchanges here.
[Had to split the links because of Discourse hostility to new users…]