PEP 627: Updating PEP 376; making RECORD optional in installed .dist-info

[sinoroc]

@pf_moore

I’ll also blame Discourse for having no way to mark messages as unread or for later review

(You can set a bookmark with a reminder.)

[pf_moore (Paul Moore)]

OK. My notes on the PEP and on the proposed wording of the packaging spec.

1. You say you don’t want to get sucked into REQUESTED, and I agree. But the current wording feels like it implies more than you want to. I’d suggest adding a note at the top of the section on the REQUESTED file saying something like “Installers are not required to maintain REQUESTED, and so consumers must not assume that the lack of a REQUESTED file means anything in practice.” (Or you can take the same view as PEP 376, that tools will maintain this data - in which case you need to amend the comment that “Almost all information is optional” to reflect that REQUESTED data is not optional).
2. I don’t see much discussion of the removal of the requirement for hashes/sizes to be present. I get your argument that no tools currently use that information, but once we remove the requirement for it, it’s going to be more or less impossible to re-introduce it. Which means future tools won’t be able to say things like “refusing to uninstall FOO as local changes have been made”. Is there any concrete reason for relaxing this requirement? (I don’t buy “simplifies the spec” and I don’t see the issue with expecting shebang rewriters to update the hash - RECORD explicitly doesn’t need a hash to make that process straightforward).
3. I’d like to see an explicit statement that tools which rely on Python’s package database¹ MUST refuse to uninstall projects that have no RECORD. See point (5) below, as well. (Tools like system package managers that rely on other data can do what they want - there’s no particular need to make this point, though).
4. If INSTALLER is intended for use in messages, the content should be usable in that context - so I’d suggest that the spec should say something like “the file must contain a single line containing the name of the installer - for example ‘pip’ or ‘conda’ or ‘Mega-Corp Super Installer’”.
5. Question - if an installer omits RECORD, should it be required to write INSTALLER, so the user at least knows who to blame for the package that “normal” tools can’t uninstall? (That can be a future revision, I’m not going to block acceptance on it, but I think it’s a reasonable requirement).

Overall, though, this looks pretty good. No-one has raised any fundamental issues, so it looks like we have a reasonable consensus. If you can address the above points, I’m OK with accepting the PEP.

¹ A minor cosmetic problem with this rewrite is there’s no longer a good noun phrase for “the database of installed packages”

[pf_moore (Paul Moore)]

@sinoroc

(You can set a bookmark with a reminder.)

Yes, but then I have to choose when to be reminded. What I want is for the message to remain in my list of unread (and therefore “for my attention”) messages until I deal with it.

Another thing that sucks is that you can only have one reply “on the go” at any one time

[pf_moore (Paul Moore)]

@steve.dower

What if the file contained an uninstall shell command

Very much no. That would require installers to be able to write future-proof uninstall commands that work with all operating systems they support. I’m pretty sure we’d end up with all sorts of broken corner cases if we tried to do this.

[sinoroc]

@pf_moore

What I want is for the message to remain in my list of unread (and therefore “for my attention”) messages until I deal with it.

Another thing that sucks is that you can only have one reply “on the go” at any one time

(I think I somehow managed to configure my account so that all this is feasible from my email client. – Wanted to answer as private message, as to not pollute this topic, but it won’t let me. I will stop here.)

[dstufft (Donald Stufft)]

@uranusjr

pip does indeed only clean up packages installed by pip though, so I should have said this instead:

Are you sure about this? This definely didn’t used to be the case, and some quick searching of the code I don’t see any checks done by pip. Conceptually speaking pip should not be locked to only uninstall things installed by pip, it should be barred from uninstalling things whom the source of truth is not the Python metadata.

[encukou (Petr Viktorin)]

@pf_moore

• You say you don’t want to get sucked into REQUESTED , and I agree. But the current wording feels like it implies more than you want to. I’d suggest adding a note at the top of the section on the REQUESTED file saying something like “Installers are not required to maintain REQUESTED , and so consumers must not assume that the lack of a REQUESTED file means anything in practice.” (Or you can take the same view as PEP 376, that tools will maintain this data - in which case you need to amend the comment that “Almost all information is optional” to reflect that REQUESTED data is not optional).

I’ll update.
I’d like to take the same view as PEP 376 (even though I don’t agree with it). That means the REQUESTED file is optional, but handling it is mandatory – leaving it out must me a conscious choice.

@pf_moore

• I don’t see much discussion of the removal of the requirement for hashes/sizes to be present. I get your argument that no tools currently use that information, but once we remove the requirement for it, it’s going to be more or less impossible to re-introduce it. Which means future tools won’t be able to say things like “refusing to uninstall FOO as local changes have been made”. Is there any concrete reason for relaxing this requirement? (I don’t buy “simplifies the spec” and I don’t see the issue with expecting shebang rewriters to update the hash - RECORD explicitly doesn’t need a hash to make that process straightforward).

Here I’m actually trying to keep the status quo. PEP 376 says “The hash is either the empty string or [the data]”. It also gives some examples and notes about when the hash and file size are left out – specifically, .pyc, .pyo and RECORD are mentioned.
For the file size, it just says “the file’s size in bytes”, but later examples/notes contradict that by leaving it out, so I read that as a shorthand for “either the empty string, or file’s size in bytes, as a base 10 integer.“

I believe that’s a valid interpretation, given how non-rigorous PEP 376 is generally. But I worded my PEP so that it works even for those who PEP 376 it as “file size is mandatory“.

I don’t want to get bogged down in specifying which files can have the details left out.
I believe individual tools can make good choices, and it’s in their interest of serving users to allow messages like “refusing to uninstall FOO as local changes have been made” where possible.
But anyway, the proposed spec does say:

For other files, leaving the information out is not recommended, as it prevents verifying the integrity of the installed project.

@pf_moore

• I’d like to see an explicit statement that tools which rely on Python’s package database¹ MUST refuse to uninstall projects that have no RECORD. See point (5) below, as well. (Tools like system package managers that rely on other data can do what they want - there’s no particular need to make this point, though).

OK, I’ll add that.

@pf_moore

• If INSTALLER is intended for use in messages, the content should be usable in that context - so I’d suggest that the spec should say something like “the file must contain a single line containing the name of the installer - for example ‘pip’ or ‘conda’ or ‘Mega-Corp Super Installer’”.

The proposed spec already says If present, INSTALLER is a single-line text file naming the tool used to install the project..

I didn’t say it this way in the PEP, since the “single-line” and “contains name of installer” requirements aren’t changed from PEP 627. Do you think it’s necessary?

@pf_moore

• Question - if an installer omits RECORD, should it be required to write INSTALLER, so the user at least knows who to blame for the package that “normal” tools can’t uninstall? (That can be a future revision, I’m not going to block acceptance on it, but I think it’s a reasonable requirement).

Uninstallers will need to handle missing INSTALLER anyway – it’s better if the spec acknowledges that the file can be missing.
For installers, again I think it’s fine to rely on the tool to do what’s best for the users. Who knows what reasons to leave INSTALLER out might come up. IMO a spec should ensure interoperability, not good tools :‍)

[pf_moore (Paul Moore)]

Drat, I looked at this message and now I can’t “mark it as unread”. I’ve bookmarked it for tomorrow, but I’ve no idea if alerts go anywhere I’ll notice. If I haven’t responded to this by Monday, ping me as it means I’ve forgotten.

(This is one place Discourse really sucks IMO).

[encukou (Petr Viktorin)]

OK!
I took the opportunity to sneak in some edits into my earlier post. I also updated the PEP and spec proposal.

[pf_moore (Paul Moore)]

@encukou

I didn’t say it this way in the PEP, since the “single-line” and “contains name of installer” requirements aren’t changed from PEP 627. Do you think it’s necessary?

I found the comment “If the installer is executable from the command line, INSTALLER
should contain the command name” unclear, as I originally thought it allowed for tools to insert a full command line in there. But on reflection, that’s not true, and I think this is fine.

@encukou

I’d like to take the same view as PEP 376 (even though I don’t agree with it). That means the REQUESTED file is optional, but handling it is mandatory – leaving it out must me a conscious choice.

I think the rewording is worse, as it’s no longer backward compatible. As worded. installers must write a REQUESTED file if there’s any chance that the user requested the install. But that would imply that consumers are allowed to assume that the lack of a REQUESTED file guarantees that the install wasn’t user-requested - which isn’t true, because existing installs don’t reliably add REQUESTED.

I think that like it or not, we need to acknowledge existing reality, which means that the presence of REQUESTED has to mean “the installer is sure that the user requested this install” and consumers can rely on that, leaving "no REQUESTED file as meaning “not user requested, or we’re not sure, or the installer didn’t record that information”. Yes, that makes the data mostly useless, but we don’t have any good use cases for needing it anyway, so I think that’s acceptable.

Apart from the one issue around REQUESTED, and a typo that I noted against the PEP commit, I think this is OK now.

[encukou (Petr Viktorin)]

The intention was to give REQUESTED some useful semantics, with the caveat that tools don’t follow them, so the data is unreliable until the ecosystem catches up.

But, it’s becoming clear that I can’t really salvage REQUESTED. I too can’t see any use case where it can work reliably today. So, I’d rather remove it from the spec altogether, which would mean REQUESTED becomes a tool-specific file that pip adds for its own use.
A different solution, which could encode all three values of yes/no/don’t know, can be standardized later.

Would that work?

[pf_moore (Paul Moore)]

I think it’s clear at this point that there’s a question over REQUESTED. Let’s get some feedback from others. Once we have some sort of consensus, you can update the PEP.

See https://github.com/pypa/pip/issues/7811 for the issue where adding REQUESTED to pip was discussed. That issue also notes that flit implements REQUESTED as well. But I don’t know of any code that uses REQUESTED, just tools that write it… I think the only reason it got added to pip was because it was in the standard… Maybe @sbidoul has a view here?

On a personal note, I’ve never seen an implementation of the sort of semantics that REQUESTED is trying to capture that worked. So frankly, I’d rather see it removed altogether. It’s one of those things where, if it’s in the standards, tools must implement it (or risk devaluing the standard, as happened with PEP 376). But if there are no consumers of the data, implementing it is pointless.

I’m definitely -1 on having it remain in pip (and flit) if it’s removed from the specification.

[sbidoul (Stéphane Bidoul)]

My vision is to have REQUESTED behave similarly as what can be done with apt-mark auto/manual in debian. In my mind this is a prerequisite for automatic removal scenarios.

Mid term, I see REQUESTED evolving to hold a version specifier so if the user installed “foo<3.0”, the pip resolver could use that information to keep the dependencies in the requested state when installing additional packages. (I think that’s how I responded to the upgrade strategies survey)

When working with disposable virtualenvs or with virtualenvs managed via a lockfile/requirements.txt this is not very useful I agree. When managing the --user environment however, having that feature similar to what linux package managers have sounds useful to me.

I’m not aware of any tool that make use of it today, but the REQUESTED spec makes sense to me and I figured implementing it in pip was a useful enablement first step.

I understand the discussion here mostly revolves around a transition when some tools implement it and other not? In that respect I would be strict in the spec: “installers MUST create REQUESTED for packages that it knows are user supplied, as opposed to dependencies”. The transition period could be managed by asking confirmation to the user when auto-removing packages.

[encukou (Petr Viktorin)]

@sbidoul

Mid term, I see REQUESTED evolving to hold a version specifier so if the user installed “foo<3.0”, the pip resolver could use that information to keep the dependencies in the requested state when installing additional packages. (I think that’s how I responded to the upgrade strategies survey)

That means the current spec – neither PEP 376 nor my current proposal (permalink to current state) – is adequate for the mid term. So this will need an update to the spec, and we lose nothing by not leaving REQUESTED out of the spec right now.

@sbidoul

I’m not aware of any tool that make use of it today, but the REQUESTED spec makes sense to me and I figured implementing it in pip was a useful enablement first step.

Unfortunately, there is no distinction between “the file was installed automatically” and “the file was installed with an older version of pip”. So, the information is pretty useless to consumers that would like to uninstall unneeded projects.

@sbidoul

I understand the discussion here mostly revolves around a transition when some tool implement it and other not? In that respect I would be strict in the spec: “installers MUST create REQUESTED for packages that it knows are user supplied, as opposed to dependencies”. The transition period could be managed by asking confirmation to the user when auto-removing packages.

If you can design a spec that can express all three values, yes/no/don’t know, the transition will be much easier to manage.

[sbidoul (Stéphane Bidoul)]

So there are two possibilities?

1. remove REQUESTED from the pep and specification and develop a new standard
2. keep REQUESTED with a MUST language for installers, and mention that tools must ask user confirmation before taking destructive action based on the absence of REQUESTED, since it’s absence may mean installers do not implement it correctly

It may be better to leave it out of the spec until fully implemented. Perhaps such specs could have a short section pointing to elements of the PEPs that have not been translated to specs by lack of sufficient implementation feedback (as opposed to PEP elements that have been abandoned)?

[encukou (Petr Viktorin)]

Short-term, any tool in its right mind will want to be compatible with pip, which now writes REQUESTED. And the PEP explicitly allows tool-specific files. So, strictly speaking I don’t think need any extra text in the spec. But I did add a note.

I’ve also updated the PEP so we have concrete text to discuss.

Pip is now free to experiment. Hopefully we get an update that standardizes an existing, proven implementation. Which answers:

@pf_moore

I’m definitely -1 on having it remain in pip (and flit) if it’s removed from the specification.

Flit, sure. But I think pip should be free to drive this feature and then standardize it (probably as an optional extension which would take care of the backcompat troubles).

[pf_moore (Paul Moore)]

@encukou

Flit, sure. But I think pip should be free to drive this feature and then standardize it (probably as an optional extension which would take care of the backcompat troubles).

I’m speaking personally, not as PEP delegate here, but I’m not particularly interested in pip being the innovator over this. I’d personally argue that we remove it from pip until someone comes up with a usable spec for a replacement feature that can be standardised and addresses backward compatibility issues. An obvious approach would be to have a mandatory USER_REQUESTED file (we can’t use REQUESTED, as that’s been taken by PEP 376 ) that contained the values “yes” or “no”, with a missing file meaning “installed by a tool that doesn’t respect this standard”.

I would expect @sbidoul to argue with me, though, and honestly I don’t care enough to block him working on this.

[sbidoul (Stéphane Bidoul)]

Nope, I’m not gonna argue If people think the backward compatibility issues are going to be a show stopper for REQUESTED, feel free to dump it.

[encukou (Petr Viktorin)]

Is there anything more the PEP needs?

[pf_moore (Paul Moore)]

If you feel the REQUESTED issue has been addressed, let me know and I’ll take another look.