PEP 751: one last time

[brettcannon]

@frostming

in some cases users may need to switch to a mirror index which provides exactly the same set of package files with the same file hashes. index url plus file name is just equal to file url and either one should be applied, not both.

So choose one, but not both in a single lock file.

@frostming

Restrict the [tool] table to data that is disposable: this is also acceptable to me(and PDM).

So that puts uv and PDM (and @sethmlarson ) in favour of a disposable [tool] table and Poetry as neutral from an export format perspective. That seems like enough to go with the disposable approach!

@frostming

Recording dependencies and/or dependents: I support recording only dependencies

@charliermarsh didn’t express an opinion, @radoering said they would put in [tool], so it’s kind of hazy as to whether to put this in or not (and @sethmlarson liked having it).

@frostming

Recording extras and groups: I have an question, since extras and groups serve similar functions in a lock file, should we distinguish between extras and groups or merge them into a single field?

It first depends if we are even going to try and record them. But after that I think it’s a question of UX if you are handed a lock file and nothing else. Do people care about distinguishing between extras and dependency groups? Previous discussions suggested people did.

@EpicWink

As the PEP is right now, if PyPI were to switch to a different domain (from files.pythonhosted.org) or URL path structure for files, all existing lock files would become unusable and would have to be recreated.

Correct.

@EpicWink

Perhaps I misunderstand, and you mean that you can use the simple repo API to get the file download URL given the index URL and the wheel filename (and computing the package name from the filename).

Yes. With the index server’s URL you can get the project name from packages.name and tack that on. You can then make the request and then find the file you’re after.

@aragilar

would pip modify the cache to try to handle different files coming from different indices?

I personally don’t see why the cache would care. You got the file you wanted and I believe pip hasn’t cared about the hash once the file is on disk when the requirements file contains hashes. I don’t see why this would be different.

@aragilar

There are also tools like devpi (which is something I install on all my systems, because it means that you can work offline/with a flakey connection to PyPI), which as things currently stand would no longer work if the URL is always looked up.

Because you connect via IP address and it changes every time? So you could never rely on anything be written down in the PEP for where to find something because your IP constantly changes?

@pf_moore

I hadn’t even considered the wheel cache. It’s built based on the principle that any wheel with the same filename (name, version and tags) is functionally identical. That principle doesn’t really seem to apply for a lockfile (where an exact content match is required), so my instinct is that we’d have to bypass the wheel cache.

I think it depends on how thorough you want to be. If you wrote down the hash(es) that some wheel file has so you can validate before using it from a cache then you could do that to check if things match. But I’m also fine to make it explicit in the PEP that all the security mechanisms are about file acquisition and once you have it on disk then it’s up to your tool to decide if its cached version of the wheel file meets your needs or not (e.g. you can bypass the cache or use it based on name alone).

@pf_moore

But this does bring up a more fundamental question. Is a lockfile based on the idea that the installer will fetch the exact same files as the locker used, and install from those? Or is it based on the idea that the installer can use any file that has the same hash value (i.e., content, as long as we ignore the risk of hash collision attacks) as the original?

The latter for me (and if you include file size then hash collision attacks go way down). I think I brought this up way back when PEP 751 was first proposed, but I personally view the lock file as saying what bits you want to get installed into an environment. I think getting pedantic to the point that it has to come from a specific URL instead of relying on hashes and file size to make sure you’re getting what you want isn’t beneficial for the user. If I’m remembering correctly I considered the URL a hint of where to look, but I think you didn’t like the URL being viewed as that and wanted it to be canonical.

I’m old enough to remember download mirrors and BitTorrent (for anyone unfamiliar with mirrors, think of them like CDNs where you had to manually choose which location to download from). In both situations you weren’t downloading a specific file from a specific place on the internet, you were downloading specific bits from wherever you could get them the fastest and you could verify you got what you expected by checking the hash and file size (if you weren’t so lazy as to skip that step which I know I almost always did ). In that case you only cared about the content.

@pf_moore

maybe this should be another open question? Put me down as mildly in favour of the “fetch the exact same files” model, but I’m open to changing my mind.

Sure, but I think we need to be very clear about how far we would be taking this. If we say the lock file fundamentally cares about recording what file contents are expected when downloading a file, then the URL and index are more for auditing purposes as well as a quick way to know where to consider looking for a file. But we could also say installers could in fact have users provide alternative locations to look as well if they so desired (e.g. an internal mirror or CDN). Think of the weekend hobbyist wanting to just grab the files from PyPI and the corporate user who has an internal mirror of PyPI, both wanting to use the same lock file. Do we tell e.g. the corporate user to recreate the lock file because the URL would be different, or just say to check the hashes and file sizes to make sure you’re getting the same thing as the weekend hobbyist from the internal mirror?

If we take a “fetch the exact same file” and it’s all about the URL on the internet, then it has to be the exact URL or the index, but not both. And it also means users can’t use their own mirrors.

I’m personally for the “content” view over the “location” view. @charliermarsh @radoering @frostming do you have opinion on this one? Would you ever support letting users download files from places other than what’s in a lock file as long as the hash and file size match?