Backwards Incompatible Change to PyPI JSON API

[pradyunsg (Pradyun Gedam)]

I’d argue that dependency information (or the METADATA being implemented on PyPI) is also a very valuable addition to have here.

[pf_moore (Paul Moore)]

Metadata is already standardised, and will be present as soon as Warehouse implements the appropriate PEPs. I’d love that to happen sooner, but that’s up to the Warehouse developers’ workload (and volunteer contributions) and there’s nothing more from a standardisation point needed here.

[pf_moore (Paul Moore)]

@dstufft

I’d say more accurately would be installer and mirroring and other related tools are the primary audience.

That’s fair. I was going off what the PEPs said, but real-world use cases are more important. (I’m just hesitant about pushing my use case too much, because “mirror all the metadata you can get your hands on” is by definition going to be in favour of adding everything regardless!)

@dstufft

There’s a project called pypi-timemachine that filters the list of files by a date, you could easily imagine moving that feature into the installer if the installer API provides the upload date.

We’ve had feature requests in pip for “don’t install things older than X”.

OK, consider me convinced (although I still think we should sanity-check, and I still think there’s a possibility of cases where “we should keep this available but the simple API isn’t the right place” is the right choice).

@dstufft

We attempt not to break compatibility where we have a reasonable option, but we can and will as needed, particularly to aid in supporting the health of the service.

Understood (and I agree in principle). I’m just concerned that if we tie replacements to standardisation, there’s an unhelpful pressure on “we need to standardise quickly before the current solution gets removed”. Just something to be aware of, I guess.

@dstufft

That’s requests that make it to the PyPI origin, the dark purple is the json endpoints

… I do hope that wasn’t me My scraping app is pretty naive in some ways - I know I need to be careful of resource usage, but I don’t honestly know if what I’m doing is right.

@dstufft

So the idea behind PEP 691 is that eventually HTML will (hopefully) fall out of use and we do not need to constrain the simple api to the lowest common denominator of what we can or want to add to each serialization.

Thanks, I thought there was something like that but couldn’t find it.

@dstufft

I agree that it’s not a natural successor to the JSON/XML-RPC APIs, but it exists and we’ve expended significant energy to scaling it and making it supportable long term, so for data that installers or mirroring and related tools either need or may be useful, then it is a good option to point people towards.

OK, that’s fair. As long as we don’t end up adding stuff to the simple API that means it becomes a problem like the existing APIs, then I’m fine with that. I’ll trust you to keep us honest on that score, though. I don’t have enough knowledge to do that.

So thanks for your response. I guess that puts us back at a point where I’ll create a PEP adding a versions key to the project page, and upload-time and size keys to the file elements. I’ll try to get that done in the next few weeks.

[dstufft (Donald Stufft)]

@pf_moore

That’s fair. I was going off what the PEPs said, but real-world use cases are more important. (I’m just hesitant about pushing my use case too much, because “mirror all the metadata you can get your hands on” is by definition going to be in favour of adding everything regardless!)

@pf_moore

OK, consider me convinced (although I still think we should sanity-check, and I still think there’s a possibility of cases where “we should keep this available but the simple API isn’t the right place” is the right choice).

It’s a fair question! And to be honest, if you had said something like “hey can we add the author, maintainer, description, etc” kind of thing, I would have pushed back on those particular pieces of metadata because they don’t fit great here.

@pf_moore

… I do hope that wasn’t me My scraping app is pretty naive in some ways - I know I need to be careful of resource usage, but I don’t honestly know if what I’m doing is right.

I doubt it, they were issuing something like 600 requests per second and we ended up blocking the IP doing that Fastly, so if you can still access PyPI you’re good.

I would probably summarize one of the bigger problems with the JSON api is a combination of two things:

• Because we have an URL per version number we have a very large number of unique cache keys with different content.
• We include information inside of the responses that can grow unbounded ^[1], the releases key being one of the main culprits, which we have projects with thousands of entries in the releases key and querying/computing that took a big chunk of time. For example, we use UUIDs as primary keys, and simply producing uuid.UUID() objects when querying data from the database was taking ~200ms due to the sheer number of them on some of those urls.

@pf_moore

Thanks, I thought there was something like that but couldn’t find it.

It’s of course a dangerous power to have! Since you could imagine us adding say… a YAML serialization format and having some features only supported in JSON and some only supported in YAML, which would be an awful state to be in, but it’s a useful feature to let us not break backwards comparability with HTML but still evolve the API in JSON (or any future serializations).

@pf_moore

So thanks for your response. I guess that puts us back at a point where I’ll create a PEP adding a versions key to the project page, and upload-time and size keys to the file elements. I’ll try to get that done in the next few weeks.

Sounds good!

---

1. The same is true of the simple API! Every file added to a project is added to the same response so it grows unbounded. Fortunately we’ve carefully limited the data we pull in about those files so our query is still relatively fast and there is only a single response per project that we can cache for a long time (with a plan in the future to pre-render and store it to make it even faster). ↩︎

[neersighted (Bjorn Neergaard)]

I was very happy to stumble upon this thread and realize that there was ongoing discussion of many of the concerns we have from the Poetry side of things.

We are nervous as to the possibility of data being removed from the JSON API (or the JSON API being fully removed before PEP 658 and PEP 691 are fully implemented) – @pf_moore seems to have expressed all the same concerns however, and the hypothetical PEP he proposes would allow Poetry to migrate fully to PEP 691 once it is implemented.

I don’t think I have anything additional to add to the discussion, except that I’ll keep an eye on this and will speak up for Poetry if needed (or likewise, you can tag me if you need to communicate with the Poetry project).

[pf_moore (Paul Moore)]

Sorry, this completely dropped off my radar. I’ve just created a PR - PEP 700: Additional Fields for the Simple API for Package Indexes by pfmoore · Pull Request #2840 · python/peps · GitHub