Pre-Pep: Staged Releases separated from PEP-694

Packaging Standards
1

Some thoughts after chewing on PEP 694 for a while. 694 lists “staging” as one of its benefits — upload something, preview it, publish it when you’re ready. I think the staging idea is strong enough to stand on its own, and I’d rather not see it gated on whether the whole Upload 2.0 redesign lands.

What I actually care about here is supply-chain safety. Right now an artifact is installable the instant the upload finishes. I’d like an index to be able to hold a freshly uploaded release in a non-public “staged” state for a bounded window, run the slower security analysis async, and only flip it to publicly available once it clears (or discard it if the verdict’s bad). We already have the two ends of this — fast synchronous reject at upload time, and reactive quarantine after something’s already public. Staging is the missing middle. It closes the window where a malicious artifact is sitting there installable until someone figures out it is malicious, which unfortunately usually means they were impacted by the malicious code.

The thing that makes this separable: it’s really a repository API concern, not an upload API one. A staged release just doesn’t show up in the canonical Simple/JSON index until it’s published. No installer changes, no mirror changes — you can’t install what you can’t see. And I believe it is buildable on the legacy upload API today.

I’d love thoughts on: does pulling this out of 694 make sense? is “hide it from the canonical index until an async gate clears” the right minimal core? how much of the verdict-reporting channel is worth standardizing now vs. leaving to 694? Should any version that is flagged by these scans trigger alerts to the project owner that it was rejected? What is the approach that would be the most comfortable for handling any potential false positives on scanning and rejection?Will also loop in the 694 authors since this pokes directly at their scope.
@barry @dstufft @EWDurbin

4 Likes
PEP 694 -- PyPI upload API 2.0 (Round 3)
2

I generally support this. 694 will take time to finalize, more time to implement, and then even more time before the legacy upload can be fully deprecated.

I am concerned about one thing though. Upload 2.0 has the benefit of async processing being built-in before a release/file can be published by the caller. While the current upload endpoint is immediate and results in an installable file immediately from each successful request.

Given that there is no delay now, forcing all legacy uploads to a staging state would be unexpected (and likely unwelcome), so that would need to be opt-in. Someone who’s goal is to publish malware would likely never do so, which kind of defeats the purpose of implementing staged releases on the current api.

2 Likes
3

I think in this case that the state is internal to the index with regards to this type of staged release. I haven’t fully worked out the best approach to notifying in the event something is flagged. But maintainers of a project as listed in the index could be notified and be given a link to flag as a false positive to trigger a human review. I would not expect some form of scanning to take more than 5-10 minutes which I think is an acceptable trade-off in this case.

4

Just to better clarify here, this proposal is composable with 694. This would have a status that would be usable by 694 one day but for now provides an opportunity for gating whether that is user(npm style) or malware scanning based gating. I think this is work that can be done in the legacy API and reused in Upload 2.0 APIs.

5

Is there any interoperability to specify? Could this not simply be a feature request in Warehouse? There shouldn’t be any changes to the installers to support the staging URLs, right?

An idea to make the packages public, but not available for install (by default), is to initially and temporarily mark them as yanked. I don’t like this hack though, as it overloads the meaning and could normalise installing yanked releases.

6

Part of the reason for this being a pre-pep discussion here is that staged releases should be designed in a way that it ensures interoperability with 694 when it gets accepted. And I would rather start with the discussion focused as a PEP and decide based on feedback that it could just be a feature request to warehouse.

That feels like it would have the potential to increase attack surfaces and blast radius rather than the goal of reducing the blast radius of malware supply chain attacks.

1 Like
7

It would only do this by normalising pinning the versions of things you install, which sounds like an overall improvement to me. But agreed that it’s a bit of a hack.

8

There are a couple of things in my mind to consider. The idea behind “staging” in 694 is so that releases can be tested by the uploader before they are published by them. This could be to ensure that the actual, uploaded files are installable, or that all the files for all platforms are available at the same time (i.e. atomic releases). So in this sense, the stage is a feature for the uploader. That said, I think the 694 protocol gives the index an opportunity to scan the artifacts and inform the uploader of malware, bad dependencies, or corrupted files before publishing occurs.

To piggyback index scanning on the legacy upload API means that the “stage” would have to be created implicitly, and some mechanism to delay visibility would have to be provided, but there is likely no way to inform the uploader of this through the existing legacy protocol. You’d still need an additional API to query the state of the implicit stage, perhaps through some future REST API or just (inconveniently) the web UI.

Let’s tease out how this middle ground of staging is different than, say an implicit auto-quarantine for new legacy uploads, which would get released (un-quarantined) when the scan completes cleanly. For something in this auto-quarantine, would you imagine that the publisher could still access it, and if so, what can they do with it?

3 Likes
9

I think you can still separate out the staging here. It might require a new API as you suggest until 694 lands but that API could also be used until the legacy API is completely deprecated. This is definitely more of an implicit staging than explicit staging from an uploader. I think scanning as a stage has to be implicit. If it is not implicit that to me would imply that there is some way for uploaders to explicitly skip scans.

The reason I am not proposing this as an auto-quarantine for new legacy uploads where the quarantine would be released when the scan completes, at least under the current quarantine model it is not at the correct granularity which is the whole project. I do not think stopping any installs from already accepted versions is the right experience, this is what quarantine does today. I think the experience should be that for a newly created version it is in a “quarantine” state until either the scan completes or a bounded time window has passed where we can say we are not getting any scan signals back that indicate this is malicious so we will proceed as though the signal back was the artifacts are not malicious.

1 Like
10

I’m not sure how feasible it would be in advance of 694, but it could be potentially be done as a complement to 694 (where avoiding the implicit quarantine required migrating to explicit staging).

Implicit quarantine before that honestly feels more feasible as a client side feature (where it’s just a non-zero default cooldown period)

12

I think there is a few things here that could be decoupled:

  1. staging vs. cooldown
  2. staging vs. quarantinee
  3. staging as gatekeeping
  4. now or later

I’m not familiar with the details PEP 694, so please excuse me if I redefine things already decided there. From my perspective as a user, maintainer, and scanner/reporter:

at. 1: I belive staging should be index-level feature, while cooldown installer-level. I’d welcome default cooldown periods, but I don’t belive we should try to fix index-level security via it. Cooldown should be in a strong control of the user and let them adjust it to the risk model.

at. 2. I find staging as a maintainer-controlled feature that has also security implications. As a scanner, I definietly support triggering scans during the stage phase. For that, I need a) information about the staged uploads, b) some metadata. The usage of the staging should, however, be in the hands of the maintainer.

at. 3. Ideologically, I welcome the gatekeeping idea. Practically, I don’t belive we can in general hotfix it into legacy API. While the staging can act as index-enforced gatekeeping, I’d dear to ask: do we have scanners that will be able to leverage this right now? What would be the outcome of forced gatekeeping at the moment? From my perspective: having 30 minutes window to scan before package is publish would stop some repeating abuses by new packages - but they already live 3-10 minutes. To stop most of the malware, in my opinion, we would have to introduce hours-long window, the best 24h. However, I think we could introduce a middle-ground: opt-in staged releases for existing projects and opt-out for new projects. Allowing enforcing manual approvals from maintainers would also be a good security measure for projects that cannot benefit from trusted publishing and CI-level manual approvals.

at. 4. I definietly belive we could and should introduce first staging version now, in a simplified version. I’d advertise it as experimental feature, suggest collecting experience, and build the capacity for better implementation with PEP 694.

Answering some of initial questions from my perspective:

is “hide it from the canonical index until an async gate clears” the right minimal core?

I belive users should be able to install the package during the staging/gatekeeping, assuming they explicitly allow it. It could be achieved two ways: 1) publish staging information in index API and let installers decide, 2) provide link to the installable release file.

I’m not a fan of introducing - right now - gatekeeping as private-only scan feature. I don’t know how exactly the structure of reporters looks like at the moment, but I’d be afraid keeping access behind closed doors could actually delay the detection. I’d like the first staging or gatekeeping to be, if any, a minimal feature that keep the release away from regular users, but still let it use when explicitly enabled - similar to how already mentioned yanked or archived versions work, but more strict (e.g. I would suggest installer to have something like a flag --allow-staging=).

My suggestion would be to add “staged” field or endpoint/parameter in the existing Index API, with all necessary informations (or at least release files data) and separated RSS feed. I’d also introduce it as a release-level status, something that quarantinee is lacking at the moment. I think it would be relative small change comparing to providing separate private access. Based on experience, we could then introduce or not the gatekeeping with the bigger change.

how much of the verdict-reporting channel is worth standardizing now vs. leaving to 694?

The reporting API I’m aware of lacks some features, but I belive the biggest win would be to introduce staged releases and treat reports the same way as for live releases vs. trying to standardize the reporting (and, I assume, onboard more scanners so it would make sense). I’d rather leave standardazing API for the next step, maybe even experiment more with what could be collected and what would be helpful.

Should any version that is flagged by these scans trigger alerts to the project owner that it was rejected? What is the approach that would be the most comfortable for handling any potential false positives on scanning and rejection?

The last is actually the most important question. As I already mentioned, I’m able to stop repeating abuses timely, but anything less common requires manual verification. I don’t collect good statistics, but generally from ~200 alerts daily I produce 5-10 actual reports. I still have a limited scanning scope, and I hope there are people with more efficient systems, but I belive we have to be caution. And I don’t think we will benefit the most from forcing gatekeeping at the moment. Answering the notifications: I’d start with following the current process - put in quarantinee, let @miketheman verify and maintainers ask for details if they want to.

To sum up, I’d suggest:
a) opt-in staging releases allowing enforcing manual approval in PyPI by the maintainer,
b) having staging as a release-level state and exposing it separately in the Index API, so installers won’t see it by default,
c) issuing new PyPI tokens with a default expiry date (I know you can add it manually thanks to macaroons, but how many people do it?),
d) opt-out staging releases for new packages - and let disable it only after the first upload (maybe even after first 30 minutes?).

I see a) as the security mechanism for existing packages without breaking anything as it’s opt-in. The b) is in my opinion the cheaper way to start and we would benefit enough from it, leaving the entirely private staging/gatekeeping for the future. c) would be the simple, forward-thinking feature for the token usage, which will always be present, but we would enforce some control over the risk. It won’t fix nor break existing tokens, but we could teach people that tokens are time-constraind now. The d) is in my opinion a way to popularize staging, teach about possible gateway in the future and introduce some security review window with minimal inconvinience* for existing users. While the most damaging is malware via hacked access and poisoned releases, the most common is malware introduced in new packages. Adding an opt-out staging for new packages, maybe even true gatekeeping for first 30 minutes or so, could give as a way to make abusing PyPI less convinient.

*I’m aware of at least one user who publish relative many packages and it might affect them. Perhaps a per-user feature flag, so a specific user could be excluded from default staging on a request, would be helpful?

3 Likes
13

I incorporated the feedback from this discussion so far into what I had already been writing as the draft. This is very much still a work in progress but I hope it provides some clarity for discussions.

14

Thanks! I did not read the full text yet, but I spotted one thing when looking on APIs: in this proposal, the only control over staging the maintainer has is to request it during uploading, and the staging API is gated by the same authorization mechanism. It has two implications:

  1. the support is dependant on uploading tooling,
  2. it gives no security against leaked access to tokens/CI.

What if we move the staging to be maintainer-controled, index-side feature? The maintainer would control their staging policy in the project configuration in the index, and the tooling would work as always.

I’d love to allow configuring the policy to (optionally) be “only manual approvals”, requireing someone to log in to the PyPI and approve the publishing. Also, I’d suggest here the possibility to separate the tokens scope: I don’t know how it’s possible for Trusted Publishing, but for normal tokens, it would be great to support cavets that limit the token e.g. only for uploading or only for control API, allowing peoples to separates duties and scopes (+ the best also to add token generation with these cavets directly in PyPI UI). This way you could use the uploading token in a CI workflow, but keep the publishing approval token in another system - and only access to both of them would let you successfully hijack the publishing.

As I said before, I’m not a fan of starting with private-only preview. I’d propose scoping the preview-url and refusing serving some packages as a potential improvement for the future. I also feel that cryptography-secure preview URL, except for index refusing serving unscanned packages, makes no sense when the client can just request staged URLs. I’d actually just serve the URLs in the staged-releases key and strip the API for requesting staged releases, leaving it up to the client to parse data from staged-releases or not. My suggestion to clients would be to allow staged releases per project, what would make hiding a malicious staged (transitive) dependency harder. But I’d leave the final decision to clients.

15

I am not a fan of leaving this up to clients. Staged releases should only accessible by the credentials associated with the maintainer until it is marked as released. But before being marked as released it should not be installable by anyone else.

The other suggestions though I agree with. It will take me a bit to get the draft reworked to account for the feedback.

16

This is also how I would want to use staged releases. I want to set something in PyPI that is gated totally independently of GitHub so that even if everything goes wrong on the GitHub side there can be no release without authorisation on the PyPI side. I always assumed that staged release would work this way but it is not clear from the PEP draft so I’m going to say what I would want explicitly here.

Ideally there would be one or more toggles in PyPI like “Secure publishing: enabled” and then:

  • Any release upload must always be staged (regardless of what the upload client says).
  • Staged releases cannot be published without a separate PyPI-side maintainer approval (no fail open for this).
  • A maintainer can download and inspect the staged files and see e.g. hashes of them before approving/rejecting (I don’t see why any special credentials should be needed for this and would prefer not to manage any tokens for it).
  • Once the release is approved it is permanently frozen (no further files can ever be added).

I would also want the pypa/gh-action-pypi-publish GitHub action to wait while the staging/approval happens so that subsequent steps in the release workflow are also gated on the PyPI-side approval/rejection.

2 Likes
17

It feels a bit to me like we’re conflating two different ideas of “staging” here and that’s causing some cross talking.

PEP 694 has a concept of staged releases, but that’s more closely tied to extending the atomic transaction across multiple files, allowing groups of files to be committed together. As a consequence of that, it does allow that “transaction” to be somewhat long lived which does allow testing and the like, but there’s nothing in the feature that requires that transaction to live for more than a few seconds.

What this thread seems to be asking for is something more akin to dependency cooldowns, but implemented server side instead of client side. I think that’s a distinct idea from 694’s concept of staging, because it’s something you’d want to occur after the project has been published and you’d want there to be some minimum time frame associated with it to give scanners and such time to run and inspect them.

Personally, I don’t have any real problem with the idea of a server side dependency cooldown in the abstract, but I have a feeling that trying to introduce it into the ecosystem now after decades of the current behavior would be very confusing for people. As far as I know none of the clients have implemented an on-by-default dependency cooldown (for similar reasons) and they would have an easier time communicating that change to their users and giving their users a way to bypass it than the server side would.

It’s been a good long while now, but many years ago I ended up having to disable pip’s caching of /simple/$project/ because it was confusing people that pip install foo && twine upload && pip install foo didn’t find the new release [1].

In any case, the trickiest question (to me at least) isn’t the technical question of how to implement this sort of “server side dependency cooldown / auto quarantine” feature, but rather how to actually introduce it in a way that isn’t going to cause a lot of breakage and confusion throughout the ecosystem-- or whether we think it’s an important enough feature to deal with that breakage and confusion.

  1. My understanding is there is some desire to re-enable that, so maybe it’s less of a concern today? ↩︎

4 Likes
18

My understanding is that it is currently possible to quarantine both at the project level and the release level. I believe the FAQ quote above implies the same thing, but I haven’t dug into the warehouse code deeply enough yet to verify.

If that’s so, then it might not be a big lift to add the ability to quarantine individual artifacts.

19

I do agree here, I do not think the technical side of this is the hard part. I think it is how do we teach this and ensure that it keeps the ecosystem running smoothly while at the same time providing defense in depth by adding another layer at which malicious actors are able to be stopped before having broader impact to the ecosystem. This is why I have a time boxed window in my mind for scanning. It will not catch everything but it will catch enough to have a positive impact on reducing attacks such as credential harvesters if we can identify them through scanners quickly enough.

1 Like
20

One thing I don’t understand about auto quarantine or default cooldown systems, is how they actually provide safety. Surely, if everyone is subjected to the same cooldown/quarantine period, no-one will be using the package until after the cooldown, and so no-one will spot that it’s malicious. So all we seem to be doing is delaying the point at which someone tries the package out?

2 Likes
21

Yes, I think we have to define the scope and naming as there is definitely a confusion, and I’m not fully convinced of the benefits of the current proposal.

Good catch, I thought this feature wasn’t implemented yet, but it changed month ago :smiley:

I’m afraid the current suggestion still has this flaw. After reading the current draft, I still keep my point that introducing a security gate should be done with the knowledge that we do have scanners that will use this delayed window. Private-only delayed publishing will act in the opposite way.

To the point on what we are trying to achieve, I’ll stand that the biggest security win at the moment we could achieve is by introducing staged releases as a maintainer-controlled, “public” publishing policy: the maintainer can require manual approval for publishing to the index, but the release files are immediately available for everyone to look inside and report. I see this also as a perfect way to prepare the community for the mandatory security gateway in the next iteration.

I do believe in the security gate as a whole, including private-only scanning - and this is why I suggest mandatory staging at the beginning - but I see this as a long-time project with multiple moving parts. First of all, the success of such gating depends on the time necessary for receiving meaningful reports. I think continuing to design this way should start by getting the data on how much time it currently takes to receive such reports. I presented my statistics, but it’s just a part of the flow and I’d not say they look like justifying the prioritization of the security gateway investment.

1 Like