PEP 627: Updating PEP 376; making RECORD optional in installed .dist-info

Very much no. That would require installers to be able to write future-proof uninstall commands that work with all operating systems they support. I’m pretty sure we’d end up with all sorts of broken corner cases if we tried to do this.

2 Likes

(I think I somehow managed to configure my account so that all this is feasible from my email client. – Wanted to answer as private message, as to not pollute this topic, but it won’t let me. I will stop here.)

Are you sure about this? This definely didn’t used to be the case, and some quick searching of the code I don’t see any checks done by pip. Conceptually speaking pip should not be locked to only uninstall things installed by pip, it should be barred from uninstalling things whom the source of truth is not the Python metadata.

1 Like

I’ll update.
I’d like to take the same view as PEP 376 (even though I don’t agree with it). That means the REQUESTED file is optional, but handling it is mandatory – leaving it out must me a conscious choice.

Here I’m actually trying to keep the status quo. PEP 376 says “The hash is either the empty string or [the data]”. It also gives some examples and notes about when the hash and file size are left out – specifically, .pyc, .pyo and RECORD are mentioned.
For the file size, it just says “the file’s size in bytes”, but later examples/notes contradict that by leaving it out, so I read that as a shorthand for “either the empty string, or file’s size in bytes, as a base 10 integer.“

I believe that’s a valid interpretation, given how non-rigorous PEP 376 is generally. But I worded my PEP so that it works even for those who PEP 376 it as “file size is mandatory“.

I don’t want to get bogged down in specifying which files can have the details left out.
I believe individual tools can make good choices, and it’s in their interest of serving users to allow messages like “refusing to uninstall FOO as local changes have been made” where possible.
But anyway, the proposed spec does say:

For other files, leaving the information out is not recommended, as it prevents verifying the integrity of the installed project.

OK, I’ll add that.

The proposed spec already says If present, INSTALLER is a single-line text file naming the tool used to install the project..

I didn’t say it this way in the PEP, since the “single-line” and “contains name of installer” requirements aren’t changed from PEP 627. Do you think it’s necessary?

Uninstallers will need to handle missing INSTALLER anyway – it’s better if the spec acknowledges that the file can be missing.
For installers, again I think it’s fine to rely on the tool to do what’s best for the users. Who knows what reasons to leave INSTALLER out might come up. IMO a spec should ensure interoperability, not good tools :‍)

Drat, I looked at this message and now I can’t “mark it as unread”. I’ve bookmarked it for tomorrow, but I’ve no idea if alerts go anywhere I’ll notice. If I haven’t responded to this by Monday, ping me as it means I’ve forgotten.

(This is one place Discourse really sucks IMO).

1 Like

OK!
I took the opportunity to sneak in some edits into my earlier post. I also updated the PEP and spec proposal.

I found the comment “If the installer is executable from the command line, INSTALLER
should contain the command name” unclear, as I originally thought it allowed for tools to insert a full command line in there. But on reflection, that’s not true, and I think this is fine.

I think the rewording is worse, as it’s no longer backward compatible. As worded. installers must write a REQUESTED file if there’s any chance that the user requested the install. But that would imply that consumers are allowed to assume that the lack of a REQUESTED file guarantees that the install wasn’t user-requested - which isn’t true, because existing installs don’t reliably add REQUESTED.

I think that like it or not, we need to acknowledge existing reality, which means that the presence of REQUESTED has to mean “the installer is sure that the user requested this install” and consumers can rely on that, leaving "no REQUESTED file as meaning “not user requested, or we’re not sure, or the installer didn’t record that information”. Yes, that makes the data mostly useless, but we don’t have any good use cases for needing it anyway, so I think that’s acceptable.

Apart from the one issue around REQUESTED, and a typo that I noted against the PEP commit, I think this is OK now.

The intention was to give REQUESTED some useful semantics, with the caveat that tools don’t follow them, so the data is unreliable until the ecosystem catches up.

But, it’s becoming clear that I can’t really salvage REQUESTED. I too can’t see any use case where it can work reliably today. So, I’d rather remove it from the spec altogether, which would mean REQUESTED becomes a tool-specific file that pip adds for its own use.
A different solution, which could encode all three values of yes/no/don’t know, can be standardized later.

Would that work?

I think it’s clear at this point that there’s a question over REQUESTED. Let’s get some feedback from others. Once we have some sort of consensus, you can update the PEP.

See https://github.com/pypa/pip/issues/7811 for the issue where adding REQUESTED to pip was discussed. That issue also notes that flit implements REQUESTED as well. But I don’t know of any code that uses REQUESTED, just tools that write it… I think the only reason it got added to pip was because it was in the standard… Maybe @sbidoul has a view here?

On a personal note, I’ve never seen an implementation of the sort of semantics that REQUESTED is trying to capture that worked. So frankly, I’d rather see it removed altogether. It’s one of those things where, if it’s in the standards, tools must implement it (or risk devaluing the standard, as happened with PEP 376). But if there are no consumers of the data, implementing it is pointless.

I’m definitely -1 on having it remain in pip (and flit) if it’s removed from the specification.

My vision is to have REQUESTED behave similarly as what can be done with apt-mark auto/manual in debian. In my mind this is a prerequisite for automatic removal scenarios.

Mid term, I see REQUESTED evolving to hold a version specifier so if the user installed “foo<3.0”, the pip resolver could use that information to keep the dependencies in the requested state when installing additional packages. (I think that’s how I responded to the upgrade strategies survey)

When working with disposable virtualenvs or with virtualenvs managed via a lockfile/requirements.txt this is not very useful I agree. When managing the --user environment however, having that feature similar to what linux package managers have sounds useful to me.

I’m not aware of any tool that make use of it today, but the REQUESTED spec makes sense to me and I figured implementing it in pip was a useful enablement first step.

I understand the discussion here mostly revolves around a transition when some tools implement it and other not? In that respect I would be strict in the spec: “installers MUST create REQUESTED for packages that it knows are user supplied, as opposed to dependencies”. The transition period could be managed by asking confirmation to the user when auto-removing packages.

1 Like

That means the current spec – neither PEP 376 nor my current proposal (permalink to current state) – is adequate for the mid term. So this will need an update to the spec, and we lose nothing by not leaving REQUESTED out of the spec right now.

Unfortunately, there is no distinction between “the file was installed automatically” and “the file was installed with an older version of pip”. So, the information is pretty useless to consumers that would like to uninstall unneeded projects.

If you can design a spec that can express all three values, yes/no/don’t know, the transition will be much easier to manage.

So there are two possibilities?

  1. remove REQUESTED from the pep and specification and develop a new standard
  2. keep REQUESTED with a MUST language for installers, and mention that tools must ask user confirmation before taking destructive action based on the absence of REQUESTED, since it’s absence may mean installers do not implement it correctly

It may be better to leave it out of the spec until fully implemented. Perhaps such specs could have a short section pointing to elements of the PEPs that have not been translated to specs by lack of sufficient implementation feedback (as opposed to PEP elements that have been abandoned)?

Short-term, any tool in its right mind will want to be compatible with pip, which now writes REQUESTED. And the PEP explicitly allows tool-specific files. So, strictly speaking I don’t think need any extra text in the spec. But I did add a note.

I’ve also updated the PEP so we have concrete text to discuss.

Pip is now free to experiment. Hopefully we get an update that standardizes an existing, proven implementation. Which answers:

Flit, sure. But I think pip should be free to drive this feature and then standardize it (probably as an optional extension which would take care of the backcompat troubles).

:man_shrugging: I’m speaking personally, not as PEP delegate here, but I’m not particularly interested in pip being the innovator over this. I’d personally argue that we remove it from pip until someone comes up with a usable spec for a replacement feature that can be standardised and addresses backward compatibility issues. An obvious approach would be to have a mandatory USER_REQUESTED file (we can’t use REQUESTED, as that’s been taken by PEP 376 :slightly_frowning_face:) that contained the values “yes” or “no”, with a missing file meaning “installed by a tool that doesn’t respect this standard”.

I would expect @sbidoul to argue with me, though, and honestly I don’t care enough to block him working on this.

Nope, I’m not gonna argue :slight_smile: If people think the backward compatibility issues are going to be a show stopper for REQUESTED, feel free to dump it.

1 Like

Is there anything more the PEP needs?

If you feel the REQUESTED issue has been addressed, let me know and I’ll take another look.

It has. Please look again when it’s convenient.
Thank you!

I’ve reviewed the PEP and the PR, and it all looks good now, so PEP 627 is now formally accepted! Congratulations @encukou and thanks for all the work you put into getting this done :smile:

I’ve added a few review comments to the PR - nothing major, and nothing that affects the acceptance, but if you could take a look that would be good.

Also, if you could update the PEP to note its accepted status, that would be great.

1 Like

Thank you!