PEP 700: Additional Fields for the Simple API for Package Indexes

OK, I’ve updated the PEP with the following changes:

  1. API Version is now 1.1
  2. Fields are all mandatory, but upload-time may be returned as None if the server doesn’t hold that data.
  3. Fractional seconds in upload time are optional.
  4. Versions are unordered, and it’s explicitly noted that non-standard versions are possible.
  5. A bunch of FAQs have been added clarifying the reasoning.

I’ve added a brief rationale section, but to be honest I’m not sure how much value it adds. If people feel it’s not worth having, let me know and I’ll drop it. (Or if it’s useful, but needs improvements, I’d be grateful for suggestions :slightly_smiling_face:)

Edit: The PR for the changes is at PEP 700: Update following duscussions on Discourse by pfmoore · Pull Request #2859 · python/peps · GitHub

In addition, as a technical point, this PEP reserves all field names in the JSON form beginning with an underscore as available for private use by indexes - no future PEP will assign a standardised meaning to such names.

Can we just patch this into PEP 691 instead? Seems out of place in this PEP.

This key, if present, MUST contain a list of version strings conforming to PEP 440, which list all of the project versions uploaded for this project.

+1 to the questions about ordering, and the sentiment that the PEP should be explicit. Also, I agree with the concern about versions that don’t comply with PEP 440. It’s probably unreasonable to ask the index to omit them.

I think there’s a risk of people interpreting the versions list as something ordered simply because it’s a list, even if the PEP says it’s not. My opinion is that we should find a way to sidestep ordering/lists entirely, by turning versions into a mapping, like @layday mentioned.

This makes me wonder whether instead of some kind of major/minor versioning, we should provide a field with the PEP numbers the API supports, something like:

...
'supports': [691, 700]
...

I don’t mind if someone wants to do that, but I think having it in this PEP is just as good for all practical purposes.

See the PR updating the PEP that I referenced earlier today. This is covered in that. I’ll merge that PEP when I next get a chance.

I’m not taking that on in this PEP. If someone else wants to create a different PEP that covers this, that’s fine, but it won’t be me. One thing I will say, though, is that such a field should definitely not reference PEP numbers. These standards should ultimately be moved to PyPA specifications - Python Packaging User Guide, at which point the PEPs are no longer the definitive references. (No-one appears to have got round to this yet for the simple API, but that’s just a matter of resources, not policy).

That would making typing this via TypedDict difficult in any way beyond listing every field as optional required of what protocol version was used.

1 Like

There are proposals to support typed dict intersections, for what it’s worth. I don’t think we should worry too much about not being able to (cleanly) express API feature sets as static types.

FWIW, if we do go this route, it’d be friendlier IMO to use some sort of legible names for the features instead (eg: upload-times, versions) rather than using numbers instead. Besides, 691 is implied. :slight_smile:

See also Consider renaming this project to not include the PEP number · Issue #10 · FFY00/python-pyproject-metadata · GitHub, Rename `pep517` to `pyproject-hooks`, and trim functionality · Issue #136 · pypa/pyproject-hooks · GitHub and Name for pyproject.toml builds and PEPs about not using PEP numbers for user-facing functional pieces (yes, this is a format detail but still – similar argues apply since the consumers are written by people :wink:).

I’d personally rather we keep the simple numbering scheme. You can have features superseding other features, mutually incompatible features, features which depend on earlier features… this can get messy really easily and WAGNI anyway, if we don’t anticipate that any of the new additions will pose a problem for implementers.

Can I suggest that if people want to discuss a "supports": [list of features] mechanism, it gets moved to a different thread? As I said, I won’t be adding this to PEP 700, so it’s not on topic for this thread.

Personally, I’m -1 on doing this. As a consumer, I doubt I’d ever use it even if it was present.

1 Like

I’m still not convinced that the fields need to be required. The only argument I remember is that the version bump doesn’t make sense of the new fields aren’t present, but that seems like the version bump is simply justifying it’s own existence.

As a JSON producer, the most naturally why to add functionality of to add fields. As a JSON consumer, the most natural way to see if functionality exists is to check for the existence of the fields.

The PEP says:

handling optional fields is an inconvenience for clients using the API

Could you expand on this? I don’t find handing optional fields an inconvenience any more than checking the version number.

One thing I will suggest is that if the index server provides a fields for one project (or file), it should provide or for all projects (or all files).


The versions key is for project responses. The PEP doesn’t explicitly state this.


Keys (at any level) named with a leading underscore

“Keys named” doesn’t make sense: the key is the name. This should be “Keys with a leading underscore”

Nothing complicated. Simply that it’s easier to not have a check for None (using .get()) over doing so. It’s a minor point, certainly.

Re-reading that FAQ entry, I am uncomfortable with it, but (I suspect unfortunately, from your perspective) mainly because I think it’s simply making excuses. The reality is simply that as PEP author I think those fields should be mandatory. And my reading of the discussion here is that consensus is in favour of a version bump, but (with the exception of you) people are fairly indifferent about whether fields in general should be mandatory. So let’s look on a case by case basis.

There’s no question in my mind about versions. Being able to get a list of project versions is essential for many types of enquiry, and releases with no files are possible (at least for PyPI) so only the server can provide that information reliably. Servers that have no better means of doing so, can extract the version list from the file list (which they are serving anyway).

For the file size information, every file has a size, so there’s no sensible reason to make the field optional. And once again, some clients have good reason to need it and the server has the data so it should publish it.

Regarding the upload time, certainly there might be servers that don’t record this (or files for which that data isn’t available) so there is an argument for allowing it to not be returned. The PEP currently says the field is required but can be None. I’m OK with changing that to say the field is optional, but if present must be a valid date (that actually follows the lead of fields like gpg-sig and yanked). If no-one says they prefer one approach over the other (or there’s an argument, which I doubt!), I’ll make an executive decision. I’m not going to over-constrain here by saying that if any file has an upload date, every file must, or anything like that. Servers should (and I hope, will) provide what they have, so demanding what they can’t provide is pointless.

The FAQ section, I’ll delete. The PEP shouldn’t be making excuses for itself.

1 Like

I guess that’s my main concern. For file size and upload time, I’ll send them regardless. For versions, I will have to parse the files. For wheel versions, that’s easy, as it’s part of the spec, but for sdists, the version is not necessarily normalised (in my case, they at least match the version in the metadata). I will have to perform normalisation on those sdist versions, which is fine.

This is because I don’t store metadata, rather simply use the file details in S3 (with hash in the custom metadata). I will be competing the versions set each time.

I’m not too worried, if you feel the benefits are justified. I’m personally happy with the PEP as-is.

1 Like

Although hashes is required to exist with at least an empty dictionary, so one could argue we have examples of both keys that have reasonable defaults but are required to always be there and optional keys. But one thing to notice about the optional keys is they have a 3-way state of unknown, true, or false. Now one could go with null/None for the unknown case, but PEP 691 went with “key not set” instead.

Thanks, that seems like a good way to look at it. I’ll alter the upload time so that omitting the key means “not known”.

PEP 700 has been updated with the latest round of changes. Unless anyone has any final concerns, I think this is now ready for pronouncement. @dstufft could you take a look please?

Bah. Apparently this is “too similar” to the post I made in error to a different topic. I’m adding this junk here to placate Discourse…

Reading PEP 700 – Additional Fields for the Simple API for Package Indexes | peps.python.org , it says:

For the JSON version of the API, the following changes are made:

  • The api-version must specify version 1.1 or later.
  • A new versions key is added at the top level.

Now PEP 691 defines two JSON payloads/responses: project list and project detail. PEP 700 doesn’t specify which of these the PEP’s new keys apply to. While size and upload-time are scoped to files (so you can infer where it applies), version doesn’t have such implied clarification and thus could be read as applying to the project list which I don’t think is intended.

My other question is whether version 1.1 would apply to the project list even though it doesn’t have any changes? You could argue for consistency it should even though no fields changed. Or you could argue that the version only changes for the project details. I personally vote for consistency as it will make it easier to make requests of servers without having to have different Accept HTTP headers based on whether you’re requesting a project list or detail.

Sigh. To be honest that never even occurred to me. I think it’s more something that should be handled as a clarification to PEP 691 (which is another way of saying I don’t really want to think about it :slightly_frowning_face:). As far as I’m concerned, a server implements version X.Y of the simple API, and states that version in its API responses. So the version field a given server replies with should always have the same value.

I agree that PEP 691 doesn’t spell this out explicitly though.

Regarding the new keys in PEP 700, yes they are all for the project details response. I’ve created PEP 700: Clarify that the new fields are in the project details by pfmoore · Pull Request #2885 · python/peps · GitHub to clarify this.

1 Like

I don’t think it occurred to any of us. :sweat_smile: But I am about to release a new version of mousebender that supports PEP 691, and while thinking about how I would add PEP 700 support I realized this case of an API version bump only tweaking one of the two possible responses wasn’t even considered.

Same here; the whole server implements one API version.

@dstufft do you want me to send you a PR with such a clarification?

PEP sounds good.

Reading it tho I think the following would help readers understand more:

  • A more descriptive title rather than “additional fields” - Maybe something like:
    • Add more release metadata fields to save calls to Warehouse JSON API
      (otherwise I feel we’re going to end up with lots of ‘Additional Fields’ PEPs as we move forward)
  • s/JSON form/JSON format/g maybe … That makes more sense to me
  • Maybe add some example JSON showing the new fields as we put examples in PEP691 and I feel they are helpful gaining understanding

At this point, the PEP has been submitted for pronouncement, so I’d rather not make last minute edits.

@dstufft - a gentle reminder in case the request for pronouncement got missed among the follow-up comments.

I’ll take a look later today or tomorrow. Thanks!

1 Like