Since before PyConUS, I’ve been making substantive improvements to PEP 694, mostly in response to @dustin detailed and appreciated feedback, as he is the PEP Delegate. Please follow up here rather than in the Round 2 thread, or the original first round.
Most of those suggestions I’ve either adopted, used as motivation to elaborate, or thought about aspects more deeply or thoroughly. I’ve now published the latest update for your enjoyment. It’s a long read, but I hope it is still approachable.
For example, there’s a lot more detail about the state machine for creating and managing a session, and uploading files into a session/stage, but at least now I have pretty diagrams!
One thing I haven’t done yet is to incorporate any changes in response to @cjames23’s Pre-PEP regarding separation of stages from Upload 2.0. I am still thinking about this, and now that this big update is published, I have more brainwidth to think about it.
The protocol largely stays the same so at a high level you still have the process: create a session, upload files into that session, publish the session. I’ve mostly (hopefully!) closed many of the corner cases and underspecified bits of the protocol. There’s a new section on the security implications and a new section on the (non-normative) recommendations for tool UX. Details in the Change History section, listed here for convenience:
Session actions now use dedicated endpoint links instead of an action key in request bodies.
Publishing sessions add links.publish and links.extend
File upload sessions add links.complete and links.extend.
The links.session and links.file-upload-session endpoints are now used only for GET (status) and DELETE (cancel) operations.
Specify that attempting to replace an in-progress file upload returns a 409 Conflict.
Specify that uploading a file matching one already published for an existing release returns a 409 Conflict, since published artifacts are immutable.
Clarify the wording of the Multiple Sessions client recommendation example.
Relax session access from the exact creating credentials to any principal authorized to upload to the project, evaluated contemporaneously on each request.
Supports rotating Trusted Publishing tokens and multiple publishers contributing to one session
Notes the related security implications.
Remove the optional metadata key from the file upload session creation request. The uploaded file is the authoritative source of metadata, which the index extracts from the file itself.
Define an explicit publishing session state machine.
Rename the session-level pending status to open.
Add a transitional processing status for deferred (202 Accepted) publishing.
Document the error status as a still-editable state that records a failed deferred publish (with the reason reported in notices).
Specify that any completion failure – synchronous or deferred – moves the session to error, that an error file cannot be repaired in place (the client cancels or deletes it and starts a new file upload session),
The server MUST reject a DELETE with a 409 Conflict while a session is processing.
Add state transition diagrams to the Publishing Session States and File Upload Session States sections, alongside the existing transition tables.
Make the suggested twine and uv command-line interfaces consistent: group the staged-session operations under a session subcommand (session publish/session cancel/session status).
Give uv the same staged-session follow-ups and session-id output as twine, and align the GitHub Action’s stage input with the --stage flag.
What’s the expected error for mutation commands (including publication) if an incompatible state change takes place in the background? Cancellation calls out 409 for the processing state, but I didn’t see a general statement that 409 should be emitted while in processing/cancelled/published for any operation that is only valid in open/error
Are servers expected to keep session status URLs valid indefinitely, or can they revert to a 404 some period of time after the session is terminated? The cancellation section implies this outcome, but I didn’t find it clear, and I found it even less clear for successful publication. This also ties into the expected error handling (sufficiently stale links may report 404 instead of 409)
Same 2 questions as above, but for file upload session states (amending the question details as necessary)
Edit to add a suggestion:
in the “name required” discussion, it may be worth mentioning that even with that restriction, this PEP still improves multi-project releases, because they can all be fully staged before a final step runs through and publishes each one. It also lays a foundation that could support a “publish multiple projects” operation as a separate future endpoint.
Thanks! I appreciate the review and all the previous feedback.
This is a good observation. There are a number of underspecified gaps where a race condition could happen while a stage is being processed, and they’ll all have to be closed. E.g. while a stage is open or being prococessed, someone uploads a conflicting wheel via the legacy API.
Things to add:
Recheck all artifacts in the stage for conflicts during the processing step, and transition to the error (or stay in open, if it’s an immediate publication) if there’s a conflict.
During publication, all artifacts in the stage have to be reserved for purposes of conflict detection. Meaning, during processing, you would not be able to upload a conflicting artifact using the legacy API, but if the stage publication failed, that reservation would be lifted.
Another excellent callout. I’m adding a section that describes the URLs for terminal states SHOULD be retained for some index-specific period of time, but leaves that retention period up to the index.
Yes. We’ll echo the same language for file upload session status retention except that it’s tied to the parent publishing session’s lifetime.
Just want to call out, just in case it was not on your radar, that this should also be specified for the URL for Staged Previews as well. I would expect the open ended releases discussion to apply to staged preview urls as well in terms of the period of time being discussed there for PyPI.
Keep in mind that the server can expire sessions (and thus their stages) at any time, although the spec says they SHOULD remain active at least until the expires-at timestamp in the session status. Clients can request extensions, but it is up to the index to grant them. So if a server expires a session, its stage goes away and the session status is subject to the retention policy of the server.
I think this is a little different since stages are not generally public, although if the session token/stage URL is shared it can be accessed by others without authentication (by design).
So I think the lifetimes of stages and open releases are different, but certainly if a release is closed, then no publishing session can be created for that release, and if there are any open sessions at the time the release gets closed, I would expect the server to expire any open publishing sessions.
Note that because the session SHOULD not MUST stay open until the expires-at timestamp, I think it’s valid that if there are publishing sessions open when the release close timeout occurs, the index is free to do what it wants, which could include either closing all open publishing sessions when the release closes, or keeping the release open until all publishing sessions are completed or expire.
In the authentication section, can we specify how tokens as well as username and password are passed to registries? With the current level of details, it’s not yet possible to write a client that connects and authenticates with arbitrary registries.
For example, we could say:
Authentication
Authentication uses the Authorization header from RFC 7235. For authentication with a username and a password, the Authorization header is used with the Basic scheme. For authentication with a token, the Authorization header is used with the Bearer scheme.
Registries and clients are free to provide additional authentication mechanisms.
Personally, I’m fine with not specifying details on authorization as long as those two common auth flows that users need are supported. Note that platforms such as GitHub respond with 404 instead of 401/403 when you don’t have access, to prevent probing for projects, so we shouldn’t require 401/403 responses.
Upload authentication is also not standardized.
PyPI uses HTTP Basic Authentication with API tokens as the password and the username token. Trusted Publishers authenticate via OpenID Connect and receive short-lived API tokens that are used in the same way.
While I wouldn’t want us to copy what the legacy upload API does, the legacy upload API auth pattern is supported across registries, and registries that don’t support it initially usually add support for it. For the legacy upload API, both uv publish and twine upload can authenticate with arbitrary registries.
I might be misreading the spec, but I think the intended flow is:
Server responds with WWW-Authenticate: <scheme>
Client responds with credentials in the format implied by <scheme>
So, for example, a registry that wants to accept username/password pairs could respond with WWW-Authenticate: Basic, which would signal to the uploading client that it should use the normal Basic authentication format (of base64(username:password)) for a username/password pair.
I do wonder if we can simplify this, though – right now the spec leaves it open to the registry to determine the scheme, but it might be nice (since this is all greenfield) to stipulate a single acceptable scheme. I think it’d be cool if PEP 694 stipulated just Bearer authentication, for example, with the following requirements:
Normal credentials (API tokens) are passed verbatim as bearer credentials
Username/password pairs are passed as base64(username:password), i.e. the same format used for Basic but placed in the Bearer format instead.
((2) is explicitly allowed by the Bearer RFCs, since any opaque token can be a bearer credential.)
twine only supports HTTP Basic at the moment – we cram whatever credentials the user gives us into that shape. However, I would personally be supportive of twine doing Bearer authentication as the uniform credential shape (including for username/password pairs).
It feels like there’s an assumption that an index can only support one authentication scheme.
Small modification, WWW-Authenticate allows for multiple schemes to be communicated as acceptable, either via multiple headers or a comma-separated list. This is ultimately the most flexible for index operators and provides a discovery mechanism for clients which is why it was written into 694 to use RFC 7235.
I don’t much like masquerading basic auth under the Bearer path FWIW, especially since an implementer could explicitly advertise both, and even deprecate one later.
It’s a pretty small nitpick from me – I’m personally happy with servers advertising the authentication mechanism they support and uploading clients selecting the right one for the kind of credential they have!
While relying on RFC 7235 makes sense to me, I don’t otherwise have strong opinions on the authentication flow. My only preference is for there to be a single standardized flow across all the ways to interact with an index (and PyPI specifically), but currently and in the future. Meaning, when we have an API, can we adopt the same authentication standards[1]?
I think this concern only comes up when probing for project names that haven’t had their first published upload, but for which there is an active stage in progress. Once a project has a single published upload, the name is of course public anyway.
One caveat: the spec currently says that indexes MAY reserve a project name before the session is published, but if an index chooses not to, then there could be a name claim race condition. If an index doesn’t place a temporary reservation on a name, you’d still get a 404, so you wouldn’t know if a first-publish stage is active, but it might be possible to open a competing first-publish stage, and beat the original to the publishing step. So I think we should consider strengthening that temporary reservation to MUST.
I know this is off-topic so feel free to consider this a rhetorical question ↩︎
– I’m personally happy with servers advertising the authentication mechanism they support and uploading clients selecting the right one for the kind of credential they have!