Mach-nix - Tool to create highly reproducible python environments

The Nix user DavHau created a tool called mach-nix, which allows one to package Python projects and environments with Nix, requiring minimal knowledge of Nix. The tool can be used like pip and virtualenv to create a shell with a certain tool or environment, or it can be used in conjunction with other Nix expressions.

Why would you want to use this tool? Reproducible builds with all build and run-time dependencies provided by the same package manager, regardless of whether they’re Python dependencies or not.

Not all Python packages work out of the box, in which case parts need to be patched or overridden, which is more involved. mach-nix piggybacks here on the Nixpkgs distribution, which provides specific configuration for more “complicated” packages such as the scientific Python stack and machine learning packages.

The upcoming version of Nix features a new feature called Flakes, which are essentially lock files for Nix expressions. That way, you can fetch Nix expressions and lock them, allowing for a more decentralized packaging. The cli is also improved and as soon as this PR is in one could obtain a shell with a set of Python packages by writing nix shell mach-nix#with.requests.tensorflow.aiohttp. This invocation is a pretty ugly hack (I guess somewhat inspired by Nixery), but it does show a strength of the tool.

2 Likes

One more reason to use mach-nix is that nix provides a global content-addressable store for (reusable and isolated) installed packages together with a way to compose these packages in an ad hoc way. This means, for example, that you can install a specific Pytorch version once and then link it in many project-specific (“virtual”) environments, saving on storage. This may be substantial for binary distributions and is a reason some people in DS use conda instead of venv+pip