How should a lockfile PEP (665 successor) look like?

[pradyunsg (Pradyun Gedam)]

It’s an approach — Another would be to review all the existing formats and design something that works for their needs. Or to pick one with the “least worse” trade offs.

PEP 665’s research involved PDM, Poetry and also other ecosystems (npm, cargo); and I don’t remember more off the top of my head. Fulfilling the needs of most of such projects is a useful thing IMO.

[ofek (Ofek Lev)]

@pf_moore

would it be easier to simply standardise an existing format like this?

Yes, but I think we shouldn’t. We should instead do what Brett is trying to achieve now and satisfy use cases, and the format will be shaped by those. The format is an implementation detail and we should not put too much emphasis on it (which is what PEP 665 did).

[davidism (David Lord)]

This PEP is about the lock file format, so I don’t think that format is an implementation detail. I was just using PDM as an example though because I like its layout, and found it easy to add the extra stuff I think is needed.

[brettcannon]

@pradyunsg

PEP 665’s research involved PDM, Poetry and also other ecosystems (npm, cargo); and I don’t remember more off the top of my head.

1. Dart
2. Rust
3. npm
4. yarn
5. pdm
6. pipenv
7. poetry
8. pyflow
9. pip-tools

[ofek (Ofek Lev)]

@davidism

This PEP is about the lock file format

Well sure, but that’s not the first step and we’re not there yet. It’s like writing a database schema without business logic.

[VannTen (Max Gautier)]

Here is some use cases we have

Thoth use AI to curate python dependencies trees criteria. (TL;DR: a fancy dependency resolver)

Currently, we only support Pipfile / requirements.txt
Depending on the form a lock file would take, we could be both producer and consumer.
Relevant in our context:

1. we mostly target the container space => so the build/packaging/install environment is not necessarily the run environment.
2. we’re quite oriented towards the data science field. On that subject, this caught my eye

How should a lockfile PEP (665 successor) look like?

Here’s a legitimate workflow that depends on the dynamic nature of sdists to work around a tooling limitation: a package that has different dependencies (say… for mxnet) based on what GPU/CUDA/pick-your-custom-accelerator is available on the machine. That’s not malware or random.choice or machine name. That’s a legitimate use case, that installs different dependencies based on the build environment. You can’t represent this with environment markers. If the build environment doesn’t match the in…

Supporting that sort of stuff (customizing the dependency tree depending on device availability) is of interest to us. However given 1 this can’t happen at build time ; unless for example the lockfile record all possibilities and we have a builder creating several container images.

---

We’re also currently on something else, “Custom Notebook Image”, aka, creating a container image to serve a jupyter notebook from a source provided by the user. The source being:

• manually entered list of packages
• git repository
• notebook file (.ipynb)
• …

In that case, a standard lock file seems like it would be a good intermediary representation to which we would convert every source so that the build part of the pipeline is the same. For now, we’re going to use a Pipfile.lock or the output of pip-compile --generate-hashes, we’re still working on it.

---

That’s for the use cases.

I’m also going to be the new maintainer of micropipenv. In my opinion, it tries to solve the same problem that this discussion, which is the main reason I’m participating (I think it was already mentioned on the PEP 665 thread). I think it could become a first implementation, as either a converter from other format, an “universal installer” as PEP 665 was calling them, or both.

I’ve reached out to our users on the subject to see if there is other use cases which could guide the discussion.