Agreed, if “push a button” also means “call an API”.
1 Like
I started to reply with a “but won’t people just…” objection, and then I realised I’m assuming we’re trying to force people into behaviours that we think are good but they might not want to do. I don’t know where I got that assumption from, so I’ll ask explicitly. Are we trying to design something here that will allow people who want to add some form of 2-phase release to do so, or are we trying to design something that ensures that all projects go through a 2-phase process, because we think it’s a good practice that we should impose on users?
I can’t see why anyone would object to having an option to do a phased release, but I can see problems with requiring it.
1 Like
By “2-phase” do you mean staged releases as 694 describes them?
Leaving aside the scanning aspects, at least the way I’ve been thinking about it, 694 in both its new upload mechanisms and stages are for the foreseeable future nice-to-have improvements. I think there are a lot of projects for which both pre-release staging, stage testing, and atomic releases will be really useful.
Probably less so for pure-Python packages since you’d have one wheel and maybe one sdist, and atomicity or pre-release testing isn’t as important.
If some day we decide to retire the legacy upload API, I think it’s going to be a very long deprecation period. To be clear, that isn’t even being proposed.
I’m not really sure. Maybe it’s my database background, but I see 694 releases as being more a single extended “transaction”, with the whole upload and release process being one interaction with the index. But the way people are talking in this thread feels more like two separate “transactions”, with an upload action (which people are characterising as being done by CI, although clearly it’s possible to do it manually) and a publish action (which is done via direct interaction with the index, either via the index UI or maybe via an authenticated API).
The security position seems to be that by having 2 actions, which use 2 different credential sets, you limit the exposure if one set of credentials gets compromised. But for that to work, you have to explicitly make the “publish” action something that can’t be executed by CI, and therefore necessarily a manual action by the user (where “manual” might mean “run a tool that calls the publish API”).
2 Likes
I am saying that I want to have a project-level option configured on the PyPI side to require that releases are staged and that a staged release cannot be published without an additional authorisation step and that I would opt-in to that for at least some projects. That final step could be clicking a button in the PyPI website or it could be done with a project PyPI token. The configuration for this requirement must not be something that can be overridden by the upload client in any way.
I suspect that making that final step compulsory for all projects would be counter-productive because the most obvious workaround would be to take whatever token you need for the approval step and then stick that in the repo defeating the main benefit of trusted publishing that you don’t leave long-lived tokens lying around.
I’m not sure how this would be implemented but I would actually want the “upload action” (specifically the pypa/gh-action-pypi-publish GitHub Action step) to wait while the sign-off and publishing is done and to fail/abort the CI workflow if publishing is rejected or times out waiting for approval. This is because I would want the workflow to have other jobs that depend on this job (publishing a GitHub release, updating docs websites etc) and I want those other jobs to be cancelled if the PyPI release is cancelled. Note that this is not entirely about security: I might also just want to check things and maybe cancel and retry the release.
1 Like
I think it’s just naturally going to be two different credentials, but that doesn’t mean CI can’t do both.
With 694, I think you just use TP as the preferred token for each individual upload into the stage. That token[1] will almost certainly not be the one used to interact with the stage’s API. The publishing step will probably be manual, with both through the web UI and via an API call.
I can imagine a workflow where your CI systems are building the wheels and uploading them into the stage via TP. Once all wheels are uploaded, maybe you have a CI that does some installation and smoketesting on the staged wheels and if those pass, it hits the publish API. Or maybe there’s a manual QA step and publishing happens by a human via the web UI or a script.
or a long lived legacy upload token ↩︎
1 Like
Can you elaborate why you want that forcing configuration option? I mean, the alternative is just to have your project’s releases go into a stage. Either way, you’re in control so the forcing config seems unnecessary.
Are you worried that the upload client itself (or the CI action) could be compromised to bypass the staged upload choice you’d normally make?
In the 694 workflow, this would be a polling wait. We can’t keep the HTTP transaction open while whatever backend verification step is being performed. And for 694’s chunked upload mechanism (e.g. to get around the current 1000MiB hard file size limit), you’re going to have multiple transactions anyway.
You’ll get a 202 when the upload’s been accepted for processing and will have an endpoint you can poll for status changes. So you automated publish action will have to wait for that completion status.
1 Like
So I am approaching it from both definitions myself, one that is explicit for a 2-phase release that says maintainers should be able to upload and then have access to grab those artifacts and run further integration tests against them before saying they are safe to release. The other is the index policy based for PyPI to state that in the current landscape, where AI is not just a tool maintainers have it is also a tool that malicious actors have and are using to find exploits faster and faster, that we should have a policy that at least provides some level of checks that can help to reduce any attack blast radius.
I think the underlying mechanisms are the same for both cases. Its just the scope of the 2nd phase that is being debated here. I think the policy based portion is an input signal on being able to release where if the signal back from a scanner is flagging something as malicious you cannot release without asking for an approval from a PyPI admin. This just protects against false positives so those are able to still release. I would obviously want Mike’s input here on what he would like to see happen with regards to requiring approval from PyPI admins if something was flagged as malicious.
1 Like
I’m not entirely sure what question you are asking here. I want to set something in PyPI that restricts how releases can be made. I don’t want any upload tool like twine to be able to bypass those restrictions by saying something like “let’s just use the legacy upload API”.
Yes, absolutely. Suppose that everything in GitHub is compromised. Maybe an attacker has taken control of my GitHub account but they don’t have my PyPI password or 2FA. Or maybe an attacker has hacked GitHub itself. They might be able to initiate the trusted publishing process from CI to PyPI but I don’t want them to be able to publish the malicious release making it available to installers like pip.
2 Likes
Seems like under that scenario, there’s an incredibly large swath of damage they could do. But I do get the motivation to want to have a backstop in the index itself.
As an alternative suggestion (and I’m aware this is somewhat off-topic for this discussion) wouldn’t that sort of scenario be better (or at least as well) served by some sort of “suspend publication on PyPI” flag that you could set in PyPI’s UI, while you dealt with the issue?
Or is the problem simply that the way you expect to find out that your Github credentials have been compromised is via an unexpected release appearing on PyPI? Because I suspect there’s a lot of other damage someone could do with your GitHub credentials that didn’t require releasing on PyPI…
1 Like
Unfortunately, recent events have more or less shown that the most damage that can be done is to release a new version of an existing package that contains malicious code. This is an active issue, and GitHub credentials are specifically being targeted for this purpose.
And yes, every case that I’m aware of has found out because other people were compromised by the update and reported it to the owners, who were unaware that they’d been targeted. This is partly why I set up the CPython release process to include email notifications to the entire release team, so even if someone manages to get into the system, they can’t release without notifying the rest of us.
From a security POV, having new releases act as if they’ve been pre-yanked for the first ~12 hours is probably enough, provided the RSS/equivalent notifications still go out so the groups doing scanning for us still know about them. Whether that’s a client side thing or a server side thing isn’t a big deal, except that if it’s server side then we can more easily say we’re doing everything we can to protect our users. I’m not a huge fan of hiding new releases until they’re scanned, as other ecosystems do, mostly because it’s an opaque process that relies on a lot of false-positive handling by a team with special access. Having the files be public immediately but temporarily ineligible for unpinned install is a practical solution for the malicious user problems.
There are other problems/needs that may be better handled with a more interactive mechanism. But I don’t think “compromised GitHub credentials” needs them or would benefit any more from them than it would from a simple delay.
3 Likes
This would entirely hinge on reusing yanking as a mechanism and I think that muddies the waters enough that it would be harder to keep track of when compared to the idea of the artifacts are not released and available for download unless you are the maintainer or a scanner with permissions.
I am going to give @barry some time here to think about this proposal now that 694 feedback changes have been put out. I want to make sure that there is some alignment between 694 and this proposal if it needs to become a full PEP.
Thank you to everyone who has already provided valuable feedback. This is what makes the Python community so amazing! I remain humbled by all of the people who are willing to jump in and provide feedback for someone that only in the past year has come out of the shadows of just using Python to wanting to provide more of my time to giving back to the ecosystem and community.
3 Likes
I’m just catching up on this, and I just wanted to mention one thing with regards to:
In theory, Macaroons allows you to build this without any help from PyPI (though obviously it wouldn’t be a button on PyPI that does it), but the Macaroons implementation on PyPI doesn’t currently expose the knob you’d need to support it (though adding it wouldn’t be difficult).
Macaroons allow you to constrain a token such that, in order to successfully use it, you need to get an additional token from some other service (or services), which is designed in such a way so that PyPI itself doesn’t have any knowledge of that other service.
So you could have an “approver service”, which never sees your actual API token, and the API token that your CI uses, is itself not valid, unless you submit it with an approver service token, so compromising the CI pipeline itself isn’t enough (and your approver service can do whatever you want, including just be a button that you push to say “yea allow this”).
@barry just wanted to check in here and see if you have had the time to give more thought to my proposal.
@cjames23 I have thought about some of this and full disclosure, spent some quality token budget to evaluate your proposal in light of the current 694 text.
There are several elements to your proposal, and a lot of overlap with 694. Here at least are some thoughts.
- Where there are overlaps on the “control plane” API, such as publishing, discarding a stage, getting its status, I think we should just use 694’s protocol. Whether that’s exactly and specifically the same protocol or state machine is TBD, but at least the shape and transitions should be identical. That way, clients only need to know one dance.
staged=trueon legacy upload. That could be an interesting way to enter the 694 protocol through the legacy upload API. I haven’t fully thought through what the response to a legacy upload withstaged=truewould be, but I think there should be a path into the publishing session state so that everything can happen the same way from session initiation onward (dovetails with the previous bullet point).- Security scanning. Indexes are entirely free to defer a publishing request on an open stage (returning
202and doing further processing). This seems like the natural point for PyPI to inject a security scanning or even temporary hold gate. The fail-open behavior also fits nicely into this protocol, and can lead directly to thesuccessstate if no human or automated review happens within whatever index-specific timeout is required. If the security review fails, the session moves to theerrorstate, and there will be enough information in the status messages for the clients to resolve the problem[1]. - Publishing permissions. Right now, 694 says that if you have upload permissions, you can both upload artifacts and issue a publish request. We could potentially separate those two roles. It’s possible we could say that the “publish” role must be completed via the webUI, for the first release, or every release, although I’d argue that the latter at least would be pretty inconvenient for CI publishing. Either way, this fits in with 694 nicely.
- Auto-publish after timeout. IIUC, This is one place where your proposal diverges from 694, since 694’s publish event is always client initiated. Perhaps an index can auto-transition from the
openstate to theprocessingorpublishedstate when the legacy upload +staged=true+ fail-open timeout trips. That would only be allowed for publishing a single artifact using the legacy upload protocol, but it could use the 694 machinery under the hood (and perhaps still expose the same status end points). - Public visibility of stages. This is a big divergence from 694. If we want to expose the stage base URL in the 691 JSON metadata, to enable scanners (third party, automated, manual) I think that would require an amendment to 694. Perhaps a key saying something like “public=true” so that the unguessable stage URL (and session token) are easily guessed, say from the project+version tuple. PyPI could even enforce this setting for first-upload. Alternatively, PyPI could share these stage tokens only with trusted security partners. Either way, I think it’s important to to preserve 694’s embargo functionality and balance that against some level of security scanning by third parties.
Have I missed anything?
if possible – detected malware may never exit the
errorstate, and would end up timing out the session, and getting culled ↩︎
2 Likes
I do not think you missed anything.
I am fully onboard with just using 694s protocols here in terms of the control plane.
I do not think this needs to be completed through the webUI for every release. I think the primary separation I want to see here is that separating out those two roles to provide some security in layers such that if one or the other role has compromised credentials then it still makes it much harder to get malware through the process.
This really only differs because I was trying to find a way that says we wont block forever if a security scanner does not come back. So I would read the timeout here as only applying to automated scanners. I was treating no signal after a given window as a positive signal instead of a negative signal.
I was more thinking more that PyPi would share stage tokens with trusted security partners. I like the embargo functionality of 694. It fits in with what I am trying to achieve here which is providing a bit more proactive security in layers.
I am more than happy to have this end up as mostly amendments to 694 if we think that is the right path forward. Or still happy to write a separate PEP here for the legacy API portions.
2 Likes
I’ve been building a client against the current draft—a prototype in a fork of uv, prompted by astral-sh/uv#20789 ran into a few places where I wasn’t sure how to interpret the spec. Thought I’d flag them in case any are easy to clarify before implementations start diverging. I don’t think Round 4 covers these.
1. blake2b doesn’t pin a digest length.
The spec says any hashlib algorithm may be used as a key, with at least one from hashlib.algorithms_guaranteed, and the example shows:
"hashes": {"sha256": "...", "blake2b": "..."}
But hashlib.blake2b defaults to a 512-bit digest, while PyPI’s legacy upload API has always used Blake2b-256 (blake2_256_digest). Since the server has to choose a digest length to compute and compare, the key name alone isn’t enough—a client following the hashlib default and a server carrying over legacy behaviour would both believe they’re conformant and still disagree.
A possible fix would be to pin the digest length for variable-output algorithms, or name the key blake2b_256.
2. The mechanism object is snake_case while the rest of the protocol is kebab-case.
Everything else is api-version, expires-at, session-token, file-upload-session, but the mechanism object uses file_url and attestations_url. This feels like something that’s easy to normalize now, but harder once implementations exist.
3. The error example uses details; RFC 9457 specifies detail.
Since the PEP mandates RFC 9457 problem details, a client using an off-the-shelf RFC 9457 parser would silently miss that field.
4. Minor: the 409 on a duplicate name/version MUST carry a Location header, but the text doesn’t say whether it should be absolute. Resolving it relative to the endpoint handles both cases; explicitly stating it would remove the ambiguity.
On @EWDurbin question about listing open publishing sessions: from the client side, I don’t think it’s necessary for automation. The CI retry case only needs a deterministic way to re-address the session for a given (name, version), and the 409 + Location on duplicate creation already provides that. A list endpoint seems more useful for human cleanup and observability than for tooling.
Happy to be corrected on any of these—these are just things I noticed while reading the draft and implementing against it, rather than from testing against a real server, since there isn’t one yet.
1 Like
Thanks! PEP 694 – Upload 2.0 API for Python Package Indexes | peps.python.org Does indeed have this covered! I’ll reply in other topic.
1 Like