Riffing off the work done in PEP 691, I’ve gone ahead and used a similar pattern to write a new upload API for PyPI, this time documented as a standard PEP rather than implementation defined like the existing upload API.
The new API is a bit more complicated (it would be pretty hard to get less complicated than the old API), but with it we unlock a host of new features:
- We move to an asynchronous API, that allows servers to respond quickly to requests, and enable any sort of extended processing of a file to happen out of band, while providing clients the ability to poll to monitor the status.
- We provide an opt in mechanism (on the client side) to support resumable file uploads, so that if something happens and your network connection is interrupted, you can ask the server how much data they got and send only the remaining data.
- We provide a mechanism to send the file in (sequential) chunks, allowing clients to work in situations where some middlebox may be arbitrarily restricting how much data, or how long an individual request may take.
- We provide the ability to upload multiple files, in separate HTTP requests, but “commit” them together as one atomic action, making either all of them or none of them available in the repository.
- We provide the ability for users to upload files, hold off on publishing them, and test installing and using their artifacts being committing to them, using an ephemeral URL.
- Allow the client to “pre-validate” that the upload will be OK, by sending the metadata and additional information prior to actually uploading any bytes on the wire.
- Provide better mechanisms for the server to communicate warnings and errors back to the client throughout the process.
The most basic flow to replicate the existing semantics involves making a number of new HTTP requests:
POSTto create an upload session.
POSTto create a file in that session.
POSTto actually upload the file.
POSTto complete the session and publish the artifacts.
GETto query the status, assuming the server didn’t respond with a
Steps 2/3 can be completed multiple times for each file, and you may also delete and overwrite files within the session at will prior to completing it.
Unlike PEP 691, this does not attempt to match the semantics or live at the same URLs as the existing, legacy API. This represents an entirely new API, starting at version 2. However, it uses the same versioning mechanism as PEP 691 (but this is largely academic, as there is currently only the single version, and only JSON serialization).
In either case, discussions should go to this thread!