Structured, Exchangeable lock file format (requirements.txt 2.0?)

I’ve been dabbling with the namespace idea (not implementing it, but to leave room for future support in the lock file format), and notice that Conda seems to lack documentation on exactly how packages are specified, how environment files are consumed, etc. There is documentation on how to use them, and Conda is effectively the reference implementation, but without proper documents it makes things really difficult for people trying to interoperate with Conda :frowning:

was something like this what you were looking for?

https://conda.io/projects/conda-build/en/latest/concepts/package-anatomy.html

or were you looking more for the way that individual requirements are specified?

https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#build-version-spec

env files are definitely under-documented. We’re very interested in improving that, and maybe this topic will be a good standard to unify on. We’ll either be adopting whatever comes out of this discussion, or otherwise trying to unify our 3 (!) ways of specifying environments (conda’s lists of specs, conda-env’s YAML files, and anaconda-project’s YAML files).

I was looking for an overview of what exactly can go into an environment.yml file, something similar to this:

The best I can find currently is:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually

But that only covers general cases, and I can’t find e.g. what top-level sections are possible other than name, channels, and dependencies.

The version spec you linked was also quite helpful for understanding what can go into dependencies though, thanks. (Are there any other special dependecy entries beside pip that can have nested requirements? Or can channels define that themselves, say to require an NPM install?)

I’ve been treading water with this lock file format (with accompanying JSON schema):

There are still several holes (e.g. VCS support and editable install, but I don’t want to spec the latter at least until it is spec-ed), but I feel the general structure is quite serviceable.

Some important characteristics (IMO):

  • JSON format (more easily verified than requirements.txt, but still expressive enough, and parsable with built-in tools).
  • Dependency keys are decoupled to package names. This makes the structure much simpler, and solves a difficult problem in Pipfile (and Poetry also, if I’m not mistaken), where you cannot specify a package’s version based on environment markers (e.g. v1 on Windows, v2 otherwise).
  • Package information is not tied with a dependency entry. This (with the previous point) allows for room for potential support to other package managers (e.g. Conda)
  • Keeps dependency information in graph (so tools can install each package with --no-deps, but still know what depends on what without resolving).

I hope this could be helpful if the topic is brought up during the mini-summit :slight_smile:

2 Likes

Apologies for not keeping up with the threads on here, turns out that discourse is one too many things to stay on top of. I look forward to discussing this at PyCon with some of you / @uranusjr if there is anything in particular you want me to bring up shoot me an email or a msg and I can make sure to try and cover it

If you are using pipx, pointing to the pipx-installed tools would probably be a good solution here.