Lock files, again (but this time w/ sdists!)

I was tagged a couple days ago from the pipenv perspective (CC @oz123), I’ve tried -rereading it, and ever since this post is exploding – very hard for us slow readers to digest all that :sweat_smile: from a pipenv perspective I will note that cross platform locking continues to be a problem, some users have requested multiple lock files to support multi-platform environments, but I have typically pushed back on this. Without a standard to define how this would work best, it has felt like projects would have two separate development paths and it maybe works out or maybe doesn’t depending on level of parity between the disparate lock file variants, and I still think this is largely true.

What motivates the desire for multiple lock files is a deficiency for collecting hashes for system dependent sdists. I don’t fully understand this, why the sdist has to be built for a hash to be collected but I think it also stems from the fact pipenv was written ontop pip and with the pip resolver being written from the perspective of installing dependencies on the system being resolved on. If the hashes were just the sources compressed file, there is still the problem of sub-dependencies. If the sdist cannot be built on the system where lock is invoked, then it becomes harder somehow to follow the sub-dependencies.

This is an important problem to solve; but I don’t have a clear idea of how multiple lock overrides creates a consistent environment across platforms. Though makers already kind of have this problem and are often applied as a weak workaround to something that could be better defined. I keep coming back to it would be great if the resolver (pip) could resolve a complete chain of dependencies and return hashes without having to build the sdists to get there, but honestly not sure how feasible it is nor does it solve the problem of having a generic specification for lock files. I think though the goals I am describing, when abstracted are similar to this specification goals.

One thing I’ll note about pipenv is the meta hash is the hash of the Pipfile (specifiers) and not of the lock-file itself. It has always felt challenging to try and hash the lock file and place that in the _meta of the lockfile because updating that hash would essentially change the hash of the file. So therefore, devs sometimes will edit the lockfile directly, and it either works or it doesn’t, but the hash validation won’t complain since it was based on the dependency spec file.

Another thing this spec calls out lock.git, but what about a more generic lock.scm and a sub key for the type – there are multiple source vaults, pipenv supports mercurial still (I think) and while git is the most populate today, it might not forever be that way.

2 Likes