I’m assuming I’ll be PEP delegate for this. Could we please ensure that any discussions on the PEP (other than purely editorial issues like typos) happen here, and not on the PR? I will not be reviewing the PR comments, and even if I did, I find it difficult to follow the discussion given that changes are getting force-pushed, meaning a lot of things are marked as “outdated”.
Also, you still don’t appear to be following the normal process. I see that @dstufft is the PEP sponsor, I suggest having a word with him to ensure that you’re following the correct process. I would have expected a draft PEP to come out of an initial discussion on Discourse, and the PEP to link to that initial discussion for background. This PEP appears to have sprung out of nowhere, with no initial discussion or even mention on Discourse. Also, you need to get the PEP committed and assigned a number before even starting to talk about submitting it for approval, and as was explained in the previous thread
I’ll add some further comments on the proposal itself in a follow-up message.
As I said, that’s not normally how the process works for packaging PEPs. There should be a discussion, and once there’s a clear consensus that the idea is good, then the PEP should be submitted. If no-one comments on the PEP, and you submit it as you propose, then I’ll probably reject it on the basis that no-one seems interested in it.
The initial thread on this PEP had no comments on the actual proposal, just on the process. This thread has so far only had your comments, and the note from @gpshead. The fact that no-one else has commented either for or against the proposal concerns me. Packaging proposals are never this quiet
That’s what I’m doing. I left some grammatical comments and one follow-up comment on something that initially caught my eye, but otherwise I’m waiting until it has a PEP number since it already has a sponsor.
The motivation section feels weak to me - there’s a lot of examples of things that “might” be useful, but no actual examples where it will be used. If there’s no actual use cases, how do we know that the design is right? And how do we know that we’re not just adding complexity which might never get used? Do we have examples of people asking for this information, that can be linked to in the PEP?
It’s not at all clear to me how the “url” field will be filled in. What URL is appropriate for (for example) pip install ./app? It seems pointless recording the source directory, the user could rename or delete it. Or simply change the content - which makes the link useless for provenance purposes. And what should the installer do if (for example) it installs a cached copy of a wheel built from source? The cached wheel is identified purely by name/version, so there’s no guarantee that it came from the same location as would have been chosen if the cached file wasn’t there. Without good examples of real use cases, it’s impossible to guess what a “reasonable” answer is to these types of question.
You say there’s no backwards compatibility issue because it’s a new file. That’s not entirely accurate - it’s also important that the file is optional - otherwise, consumers would be impacted as they would have to be prepared to deal with “legacy” installations that have neither provenance_url.json nor direct_url.json. As it is, there’s no backward compatibility issue, but there is a usability issue - if the data available is by definition potentially incomplete, is that sufficient? Once again, without use cases, it’s impossible to say. (I imagine, for example, that filing SBOM paperwork that states that the information provided might be incomplete because there’s no requirement for the tools to have recorded everything, might be an issue…)
You’ve provided a reference implementation for pip (although I haven’t looked at it in any detail). What about other installers like installer? Or conda (my understanding is that conda tends to implement most of the “installed database of packages” standards, for compatibility if nothing else? As a standard, the expectation is that all installers will implement this proposal. Have other installer projects confirmed that they are OK with this? For that matter, has pip? In the same way as the PEP, the pip PR seems to have appeared, but had no feedback or comment from the pip maintainers, so I don’t think the mere existence of the PR implies that pip is willing to implement this PEP. This is another place where getting supportive feedback is important to getting the proposal approved.
That’s fair, but on that basis, the idea of setting a deadline for when the PEP will be submitted for approval seems even less reasonable… (It was the deadline that prompted me to comment, otherwise I would also have ignored this until it was merged).
It could definitely be useful for SBOMs, and along those lines it would allow for pip freeze (or similar) to generate a very accurate lock file-like output for the current environment. For instance, if we ever get a lock file format there won’t be a way to take an environment that already exists and work backwards to help generate that initial lock file.
I don’t think there’s anything wrong with having a draft PEP to kick off discussion. PEP 1 certainly doesn’t require you to wait to write a draft, while it does suggest doing it, that suggestion’s purpose is, as documented in PEP 1, to save the author the time of writing a PEP if the idea is going to be rejected anyways rather than being an inherent part of the process [1].
In this case, I think it would be silly to pretend there wasn’t a draft PEP already written simply to blindly follow some idealized process, the document was already written, it seems entirely reasonable to include it in the discussion about whether this idea should progress further or not.
Fridolin wasn’t saying he was going to submit it for approval, he said he was going to submit it as a PEP to get a number assigned.
I told Fridolin that I frequently express when I intend to do something when I plan to submit some PEP, because IME unless someone is extremely against something they’ll often times put it on their list to comment on later, and then get surprised when I submit something or do something before they thought I was going to.
Thus I recommended mentioning when he intended to submit for a PEP number, assuming of course that there wasn’t ongoing discussion at that point or that the discussion hadn’t changed those plans, so that nobody is surprised by it, and it tends to seem to push people to comment more often IME.
I haven’t had a chance to comment yet, but I think this proposal is a reasonable idea that improves the auditability of an existing installation to know where what came from. I have longer form thoughts on it that I don’t have time to get into at the moment, but I’m +1 on the overall idea.
The “deadline” (which wasn’t really a deadline, just Fridolin communicating up front when he intended to do something unless the current discussion altered that plan) was for submitting the PEP to get a PEP number assigned.
OK. That explains the confusion. I suspect there won’t be much discussion before then, in that case. People have already pointed out that they prefer to wait for a PEP number.
I generally think this is a useful idea. It enables the “pip freeze with hashes” use case that is sometimes requested on the pip tracker.
I have not had time to review in depth, but one thing I would like to suggest is to reference the direct URL data structure instead of re-specifying it in the PEP with possible subtle differences that would make its generation and consumption harder for implementers. To enable that, I had submitted a PR to packaging.python.org to have the data structure specification in a standalone document so it is easier to reference independently of the direct_url.json / PEP 610 context.
I have a related question (sorry for the ignorance, probably this is more related to PEP 610 itself).
Imagine a scenario where a package is build from a transient local directory, that will be removed from the machine after the installation is completed (let’s think for example in a backend bootstrapping scenario[1]).
According to the text in the PEP, this process would have only to create a direct_url.json file but no provenance_url.json, right? Is the direct_url.json required even if the local directory will be deleted later and will not be available for anyone reading direct_url.json?
Will it be OK for a package to not have neither direct_url.json nor provenance_url.json? (Will installers like pip infer anything in this circumstance and avoid updating the package?).
Just chiming in that I generally like the idea as well and am also waiting for this to be merged into the PEP repo as a draft before providing a detailed review.
This case should be covered by PEP-610. The proposed draft PEP is addressing what is missing in PEP-610, which is installing packages by their names, and optionally their version specifiers:
The provenance_url.json file SHOULD be created in the *.dist-info
directory by installers when installing a distribution identified by their
name, and optionally their version specifier.
This file MUST NOT be created when installing a distribution from a requirement
specifying a direct URL reference (including a VCS URL).
Only one of provenance_url.json and direct_url.json from :pep:610
files MAY be present in *.dist-info directory.
There is also section Example pip commands and their effect on provenance_url.json and direct_url.json. pip creates direct_url.json file for this case as of today - it follows what is stated in PEP-610:
$ pip install ./app
...
$ cat .venv/lib/python3.10/site-packages/foo-1.0.0.dist-info/direct_url.json | jq
{
"dir_info": {},
"url": "file:///Users/fridolin.pokorny/git/fridex/pip/app"
}
$ cat .venv/lib/python3.10/site-packages/foo-1.0.0.dist-info/provenance_url.json
cat: .venv/lib/python3.10/site-packages/foo-1.0.0.dist-info/provenance_url.json: No such file or directory
This should be addressed in this comment. I’ve adjusted the draft PEP to explicitly state this case.
Based on the draft PEP the proposed provenance_url.json:
The provenance_url.json file SHOULD be created in the *.dist-info
directory by installers when installing a distribution identified by their
name, and optionally their version specifier.
Is SHOULD sufficient here to avoid the mentioned compatibility issues and keeping adoption by other installers optional?
I will not be reviewing the PR comments, and even if I did, I find it difficult to follow the discussion given that changes are getting force-pushed, meaning a lot of things are marked as “outdated”.
I tried to keep git history clean, hence force pushing - I did changes between meetings and sometimes with typos or with merge conflicts. If someone would like to review only commits, they should state relevant diff and comments. BTW, GitHub marks comments outdated because they were added to a line that no longer exist because of subsequent changes (example).
