Nix-like package dependencies using hirarchical venv-like environments for packages

Hi there!

This might seem like a crazy idea and it would probably require lots of changes in how python deals with packages…

I was wondering if it would be possible (or even feasible) to have nix-like package dependencies in python, where each package has it’s own venv-like environment to alleviate issues where one (or more) packages have conflicting dependencies.

Potential benefits:

  • alleviates certain issues with conflicting dependencies
  • easier development and testability of applications and packages

Possible drawbacks:

  • would possibly require some fundamental changes in python’s runtime and how it deals with modules/packages
  • dependencies of installed packages wouldn’t be directly accessible by the user
  • some packages/dependencies might need to be the same for all packages, in which case we would need to define them as fixed or singeltons
  • necessity of optional dependencies or a way to install additional dependencies into the virtual env. of a specific package
  • this wouldn’t alleviate all issues with dependencies; e.g. singelton packages and directly requested packages would still need to be resolved
  • could also have an impact on existing packages, not just python itself

As I said, the idea is kind of crazy and I can imagine that it might never even be considered because of how many things it would have an impact on, but the python eco-system is getting quite complex and I guess I’m not the only one that is experiencing dependency-related issues on an almost monthly if not weekly basis (independent of whether it is nix, conda, pip/poetry etc.)

1 Like

Hi Darko. It’s not a crazy goal in the slightest, but from what little I know I think it’s a huge job with far reaching consequences. The first obvious question is, which benefits does PyPi and PyPa re-engineering their infrastructure in a more Nix like fashion bring, over Nix introducing a pipeline to include whatever Python packages Nix users wish. For Linux and Unix users at least.

Nix already has a great deal of Python packages in its index.
https://search.nixos.org/packages?channel=22.11&from=50&size=50&sort=relevance&type=packages&query=Python#

How does Nix work internally by the way? Am I over-simplifying it to say that a Nix package is referenced by its hash, and either has a copy of all its own dependencies, or refers to other Nix packages it depends on by their hashes? So basically it’s eliminated dependency conflicts via good old-fashioned static linking, albeit with a nice recursion to other Nix packages,and hashes to eliminate duplication?
https://nixos.org/manual/nix/stable/

If so, I’m guessing reengineering a dynamic linking system to a static linking system is a non-starter. But there may still be other ways to achieve the same end.

Hi James.

Please take my experiences with nix with a grain of salt… I’m not a novice, but I’m certainly nowhere close to somebody that’s been contributing to this project for years.

Nix actually doesn’t use static linking. The best way (at least for me) to think of nix’s package store is as a self-referencing object DB, with packages having UUIDs (consisting of a hash, based on all the contents of a package, the name and the version of a package). Dependencies of a package are hence simply back-references into this object DB and nix environments can be thought of as subsets (w/o dangling references) of this object DB. Dependencies (from what I can tell) also seem to be staticly defined, however, one can easily define their own derivations, with new build-instructions, dependencies etc. Downside of this is that these custom derivations aren’t as tried and tested as the default ones in the nix store.

Regarding nix and python: it’s actually been a somewhat painful experience. Because of nix’s static nature (w.r.t. how it deals with dependencies), one has to resolve possible confilcts manually. This isn’t an issue if the packages support “concurrent” installations, but it can be tedious with environments that don’t support concurrent installations (as with python). Needless to say, contributors to the nix project are kind of swimming against the current here, but nix’s idea of how to deal with many dependency issues seems to be the only fasible (and tested) approach that I’ve seen.

Thanks for the explanation of Nix Darko. That was very helpful. As I already said, I don’t think it’s a crazy idea at all. I just suspect it’s not one that’s worth modifying the core language for everyone over, or even to modify pip over.

Nonetheless, if nobody’s come up with a system already, I reckon Python already provides the features for a Nix -like Python dependency manager to be made, that anyone that wants it can then use.

First and foremost a Python Nix module can use import hooks to customise how the import statment works, i.e. to depend on the dependencies of the module/package that’s calling __import__, and on any metadata in a Python Nix DB. All it needs to do, is only find the version of the package in its dependencies with the specified hash, and then import that as normal (instead of following the normal import search order). Else raise an error.

Secondly if needed, there are at least 4 builder backends already. It would be straightforward to create another that simply records a package’s meta data (including its hash and any environment variables in a venv) in the DB, or somewhere the Python Nix or pip Nix can find it.
https://packaging.python.org/en/latest/tutorials/packaging-projects/

Any party could then use that to build Python Nix packages, from a PyPi sdist.

Other questions are how should this interact with, or be distributed by regular pip (could it just use pip --no-deps and target a custom Python Nix package index?) and PyPi.

Taking over the whole import system or dependency manager, could break an awful lot of stuff! So if nothing else, a solution like the one I described above based on an optional import, makes that optional.