I don’t really see why upload-time warrants a mention in this PEP, honestly. If it’s significant enough to warrant inclusion in the HTML representation, then surely it could still be added after this PEP - the text says that it SHOULD NOT be modified, not that it MUST NOT. My interpretation is that that provision is there precisely to allow adding functionality in special cases like upload-time.
1 Like
You can point users to PEP 700 for the reason. It has nothing to do with PEP 833 - excluding the new fields from the HTML format was a deliberate decision in PEP 700, and reversing that decision would require a change to PEP 700. That change can request an exemption from the restrictions PEP 833 implements - and such an exemption would IMO certainly be granted, precisely because the cooldown use case is so compelling.
A change to PEP 700 would be easy to justify - the PEP originally describes the added fields as being mainly for compatibility with the legacy PyPI JSON API, and of little significance. The appearance of the cooldown use case changes the picture completely, and I consider extending PEP 700 to support the HTML index format to be a perfectly reasonable proposal.
Someone needs to formally make that proposal, of course. But it’s independent of PEP 833, which only says that PEPs SHOULD not change the HTML format, not that they MUST not.
2 Likes
Because I believe that this missing field, and associated use case, means the specification SHOULD be modified, and while that it is still the case I am -1 on a PEP that says it SHOULD NOT be modified.
I spoke to @dstufft, out of band and before this latest thread, on why PEP 700 excluded upload-time from HTML, because were I part of the discussion at the time I would have pushed for it, as I asked for the field to exist in the PEP 691 discussion as I saw it as a useful field for resolver and installer clients.
My understanding from him, and you or he can correct me if I’m misinterpreting this, is that PEP 700 was aiming to get users who were using the non-standard PyPI JSON API to use a standards based JSON API. The motivation was not specifically to add support for features that common installers, say pip, would want to use.
I am happy to propose a clarification, but like konsti I do not currently have the capacity to take on the burden of writing a shepherding a PEP through acceptance.
It directly relates because this use case means, IMO, the HTML format SHOULD change, in direct violation of the text of this PEP.
As the author of PEP 700, I can 100% assure you that the reason was little more than completeness. I had a personal need for the data, and having to use both the new JSON API and the legacy PyPI JSON API was painful. Adding the remaining data from the legacy API seemed a simple and uncontroversial change. And not adding it to the HTML format was basically because it was only intended as a replacement for a JSON API, so only having it in the JSON index API wouldn’t be a problem for consumers. There was no expectation that installers would ever want to use the new data.
If cooldowns had been a common mechanism at the time, things would have been very different.
Sorry, I’m not sure what I can do here. For me, modifying PEP 700 to change “For the HTML version of the API, there is no change from version 1.0” to include a bunch of new fields is more than I feel comfortable describing as a “clarification”.
Maybe someone else can step up and write a PEP. Or if there’s overwhelming community consensus that a change like this doesn’t need a PEP, I’ll reconsider. Otherwise, this will likely have to wait until the Packaging Council is up and running, and ready to consider changes to our standards process.
I don’t agree with your reading of PEP 833, but I’ll let @woodruffw decide what (if anything) he wants to do with your feedback here.
3 Likes
Two parts to this:
- Per this, uv is not planning on implementing the Last-Modified hack – there are too many soundness risks with it, and it’s fundamentally a layering violation. Unless pip is considering implementing it independently, this means that neither of the two largest installer clients is considering it.
- The reason people want the
Last-Modifiedhack is not because of the absence ofupload-timein the HTML per se – it’s because third-party index providers don’t provide anything beyond the bare minimum in PEP 503. That ends up coveringupload-timebecause it’s JSON-only, but in practice it means that, even if the HTML representation were to standardize it, there’s no reason to believe that third-party indices would actually adopt it. So we’d be in the same end position around the HTML representation being de facto frozen and end users wanting hacks to work around their service suppliers.
(As abductive evidence for this, I point to PEP 740: we went out of our way to add it to the HTML representation too, and it has obvious value-add benefits to third-party index providers. And yet I’m not aware of any third-party index updating their HTML representation to include PEP 740 attestations.)
Yeah, my intent with the language in PEP 833 is to allow people to make changes to the HTML representation if they really want to, which is why it’s a SHOULD NOT instead of a MUST NOT.
Per RFC 2119 a SHOULD is stronger than a SHOULD NOT, so if @notatallshaw thinks that the HTML representation SHOULD be extended to include upload-time in a future PEP, then I have no objection to that and PEP 833 should not be interpreted as an absolute prohibition 
(Or in other words: the current community norm is “we should update both representations usually,” and I want to change it to “we should only upload the JSON representation by default, but we can update the HTML representation if we have a good reason to.”)
3 Likes
My main point of contention is we are in a worse place because they can’t adopt it as opposed to merely won’t adopt it, and I read this PEP as enshrining that problem.
I’ll freely admit I am poorly educated on specification word lawyering, and my objection may be entirely faulty based on my own misunderstanding. I think everyone understands by point though and I don’t have anything else to add, so I’ll leave it up to PEP authors and approvers to decide if my point is worth considering.
1 Like
It’s worse than that, actually. Because not all index servers can be guaranteed to even have an “upload time” value available, the field is optional even in the JSON version. So adding it to the HTML version would still allow index providers to omit it.
There’s value in adding the HTML format to PEP 700, because it would clearly define the attribute names to be used for the fields defined by the PEP. But there’s no reason to believe it would make any difference to what index providers would actually offer.
I’m repeating myself at this point, but they can’t adopt it because PEP 700 doesn’t allow them to, not because of PEP 833. And that’s something that has been true for years, and isn’t changed by PEP 833.
I’m curious. Has anyone ever asked an index provider to support upload-time, and been told “no”? If so, what was the reason for the refusal? And in particular, has any index provider ever stated that they can add upload time to the HTML index, but they have decided not to? Because if they did, they were wrong in thinking that they could.
3 Likes
I should say, just for balance, that I’m just as frustrated as @notatallshaw appears to be with the poor state of adoption of index API standards by providers. Tools like pip and uv are stuck in a situation of having to either provide a poor user experience, or work around limitations that our standards have solved years ago, just because providers seem to think that index standards don’t apply to them, and are somehow only for PyPI 
But we can’t use technical changes to solve people issues. We’ll only ever make progress if we talk to the index vendors, and try to reach some sort of agreement.
4 Likes
If it helps, I’m happy to add some qualifying language to the PEP to explain the weight of the SHOULD NOT and why it doesn’t preclude all future changes to the HTML representation.
I think I can fairly say without exaggeration that I’ve tried to get in contact with various index providers for years, and have been given the cold shoulder repeatedly. It’s been a frustrating experience, since I’ve sunk nontrivial hours of my personal time into making their products better (and presumably more profitable by extension).
Another frustrating thing about this is that making these kinds of compatibility-risk changes tends to get the attention of the end users of these index providers, but not the index providers themselves. I’ve tried to get some of these users to use their support channels to get attention, but even that gets a lot of pushback (a lot of it even before they reach out, because they just assume that the index provider won’t change anything).
All around, very frustrating and to your point does not help the tooling situation.
1 Like
Perhaps a FAQ or adding to the Rationale to say that this PEP is not intended to be an absolute prohibition to further PEPs targeting the HTML representation, and still allows future PEPs to make changes to it, but rather it means that a PEP that wants to change the HTML representation needs to make the case for why the change they want to make to the HTML representation is important enough add it even though we’ve “closed” it.
I’m personally thinking of it kind of like Python 2.7 was treated, it technically was no longer accepting new features, but a number of features did end up being backported to 2.7 because a case was made that those features were important enough to override the standard “no new features in a patch release” rule.
SHOULD NOT X is allowing for the possibility of X, but it’s got an implication that X is expected to be rare or have caveats, issues, or other such things that may cause issues if you X.
Roughly speaking:
- MUST and MUST NOT - These are hard requirements of the specification, and if you don’t follow them then you’re “wrong”.
- SHOULD and SHOULD NOT - These are things that ideally would be MUST or MUST NOT, but the spec recognizes there may be cases where they are needed, so it allows it, but “only if you understand the consequences”. In the context of a PEP like 833, the consequences of adding something like
upload-timeis probably pretty minimal, so low risk (but for instance, it’s not unheard of in the land of RFCs for simplified tools to make assumptions that get invalidated if someone doesn’t follow a SHOULD or SHOULD NOT. - MAY and MAY NOT - These are things that the spec fully allows you to both do or not do, and there typically is an implication that either choice is perfectly valid and equally reasonable. Compliant tooling is expected to handle these being followed or not followed.
- RECOMMENDED and NOT RECOMMENDED - These aren’t actually part of the normal RFC 2119 language, but they get used sometimes. Most of the time they’re basically the same as MAY and MAY NOT, but with an implication that one of the choices is better for some reason.
I do agree it’s kind of an awkward look if we end up accepting 833 and then shortly after we accept a hypothetical PEP that adds upload-time to the HTML, but 833 does allow for that.
Unfortunately the nature of OSS is that we the stuff that gets worked on is the stuff that someone feels motivated to spend their time on, so sometimes that means those awkward ordering of things are just going to happen. [1]
I’m not sure how useful of a anecdotal data point it is, but the linked PR on uv where someone wanted to use Last-Modified to hack in support for something like upload-time, the author of that PR mentioned that they choose to switch to an entirely different vendor that allowed them to effectively implement --exclude-newer server side instead rather than try to interact with their original vendor to try and get them to add support for anything.
That was about implementing PEP 691/700, not about adding upload-time to HTML which isn’t quite what you asked, but the impression I’m getting from that thread is, at least that person’s experiences, is that their company was dreading trying to get their original vendor to do anything.
Which does then circle back to 833, and IMO, adds strength to it’s argument that the HTML format is effectively de facto frozen already, by nature of basically zero uptake in new features by anyone that isn’t PyPI.
I suspect there is a very real chance that, if we get a hypothetical PEP to add upload-time to HTML, that the effective out come is the same whether we accept it or not-- that PyPI would support it (but installers that understand it are fetching JSON from PyPI anyways) and nobody else (or practically nobody else).
The only reason I’m waffling at all and calling it a chance, and not a nearly certain outcome, is that specifically for upload-time itself, the effort to add that data (if you have it) is so trivial that maybe folks would have an easier time getting those vendors to do things.
Of course, it also sometimes means that good features are blocked by something nobody actually wants to work on too! ↩︎
1 Like
Thanks all, I’ve proposed some qualifying language to the PEP draft here: PEP 833: add some qualifying language by woodruffw · Pull Request #4944 · python/peps · GitHub
2 Likes
In the interest of transparency/correcting myself, I wanted to point out that I have been made aware of a place in which standardizing the HTML representation of upload-time would likely be beneficial: PyTorch’s own index includes upload-time on the HTML representation. This is technically non-standard, but it turns out uv does actually respect it.
So, to @notatallshaw’s point, I think there’s a very strong argument for standardizing upload-time in the HTML representation. I would heartily support a PEP that does so
(I think this doesn’t pose a problem for this PEP however, per the “should not” discussion.)
7 Likes
It’s been over a week since the last piece of feedback and discussion here, so I’m hereby requesting pronouncement from @dstufft on this PEP
2 Likes