PEP 665, take 2 -- A file format to list Python dependencies for reproducibility of an application

Are you saying you view url only for the direct case?

Right, but what’s wrong with that? As long as that lock file doesn’t leave the control of the environment then there shouldn’t be an issue. This implies the lock file is being shipped with closed-source code, so the potential of the lock file getting out with private URLs is going to be the least worrisome issue if the entire code base leaked.

In my work environment I don’t think having an internal URL for internal code is problematic.

I don’t see why installers can’t be instructed to prefer hitting indexes instead of URLs and making the URL the fallback.

But this actually sounds more like a locker policy/option to not insert URLs unless absolutely necessary. And so I would assume it would be controlled at that point and not at the installer level.

I assume you mean by having someone speak up? Otherwise I’m not quite sure how to explore this specifically beyond asking Twitter for feedback from a wider audience.

1 Like

Hello! I have implemented something that looks a lot like this format in a proposed pip resolve command: metadata-only resolve with new `pip resolve` command! by cosmicexplorer · Pull Request #10748 · pypa/pip · GitHub until @pf_moore noted that the JSON output I was hacking together felt a lot like this PEP. I’m getting more familiar with the specification atm, but I will almost definitely be modifying the output of the pip resolve command to produce this file format instead. It also may not be called pip resolve; instead pip install --dry-run --report was proposed as an alternative which I think I like more.

3 Likes

I wanted to announce that the PEP has been rejected (@pf_moore messaged me privately about it). The reason for the rejection is the lack of sdist support was seemingly enough to cause a lukewarm reception overall to the PEP. I am now going to go think about what an opinionated lock file would look like from me and consider implementing it as part of my mousebender project.

Thanks to everyone who provided feedback! Thanks to @frostming @sdispater for providing early feedback. Extra thanks to @frostming for writing a proof-of-concept locker. And a special thanks to my co-authors, @pradyunsg and @uranusjr .

4 Likes

My thanks as well to everyone for their participation in the discussion here.

As Brett said, we had a discussion offline as to how to proceed with the PEP, and the conclusion was that it should be withdrawn/rejected.

This discussion has persuaded me that we definitely need a lockfile format that’s better than the current “pinned requirements file” approach[1], but I think we need a better understanding as a community of what we actually want in terms of functionality. The question of how and if we handle sdist support is the biggest issue, but a clearer understanding of use cases in general (many of which are “hidden” behind closed-source corporate environments) is also needed.

Thanks to the PEP authors, @brettcannon @uranusjr and @pradyunsg for their work on this, and I’m sorry we didn’t get the result you hoped for.


  1. Recent examples of supply chain issues with npm and similar demonstrate that locking is becoming far more critical. ↩︎

Hi, thanks for the update. What I’m not sure about is where next. If someone were to pick this up what would they do to get the ball rolling? You alluded to the sdist problem, but what about the rest? (I’m not sure how much corporate environment overview we can get here, from the communities POV)

PS. I’m personally sad that the PEP failed due to not supporting sdist. I was hoping we were taking an iterative approach as was the case with PEP-517 and then PEP-660. Start with wheel support, and let a future PEP extend/handle sdists. If this path is not feasible perhaps the rejection motivation should clearly state why not, so a future iteration can address it.

2 Likes

Honestly, I think the key think is for any new PEP to start from a very clear and well-defined list of scenarios that it is designed to support/address. This was where I felt PEP 665 fell down, it focused on reproducibility, rather than on specific use cases. And as a result, it was extremely hard to determine whether it intended to handle particular scenarios.

For example, I never really got a clear answer on whether the PEP was intended to support deployment of an app to a service like Heroku. In a broad sense, “obviously” it would, but when you get into details, suppose I had a dependency that’s not on PyPI as a wheel, how do I ship the combination of “lockfile + wheel” to Heroku in a way that works regardless of what lockfile installer Heroku chooses to use? That was the root of the “should we allow relative paths in the URL field” question, which was still open at the point where we decided to reject the PEP.

I think the reality is that someone is going to have to pick (or maybe even invent) some use cases that they believe are important, and write a PEP based on those, and be prepared to defend the PEP on the basis of those use cases. That would at least mean that readers could quickly see whether the PEP helps them in their particular situation (and then, if necessary, propose additional use cases that the PEP should add).

It wasn’t solely sdists. As I say, it was mostly a lack of clear definition of what use cases the PEP was trying to support, and “not all of our dependencies are distributed as wheels” is part of that.

The PEP was still (in my mind) perfectly viable when the scope was reduced to wheel only. In fact, on a personal note, I prefer not requiring all installers to be able to build from source. What finally tipped the scales, though, was the question of how to distribute a lockfile with a set of wheels, which is important for the use case where the locker hits some sdist-only packages, and wants to build wheels for them and provide those wheels with the lockfile. That problem is solveable in a number of ways, but we got stuck on the question of whether that is a problem the PEP even wanted to solve.

I’m still perfectly happy with that approach, as long as the initial (wheel support only) phase is clearly viable in its own right, and not simply a placeholder that no-one will use until sdist support gets added.

1 Like

I’m not certain everyone saw @cosmicexplorer’s post above, but it seems clear to me that whatever comes next will surely be based on/utilize the functionality introduced in metadata-only resolve with `pip download --dry-run --report`! by cosmicexplorer · Pull Request #10748 · pypa/pip · GitHub