Pre-Pep: Staged Releases separated from PEP-694

Packaging Standards
22

I want to be very precise about the terminology I’m using, which I’ve also been careful with in PEP 694. I think this will explain why I prefer to use the term “quarantine” for this particular feature rather than “staging”.

In 694, “staging” refers to an addressable pre-release container of release artifacts. My (rough) analogy is like staging a house you’re selling: you’re setting it up with furniture and knickknacks so someone can get a sense of what it will look like before they’ve bought it. 694’s stages are like that: you can see what a release will look like before it’s published. It’s “addressable” because it means you could point your installer at the stage and do live testing of the release, again before it’s published. You can also modify the contents before it’s published (“hmm, that couch would look better over there”).

The feature being discussed here isn’t a stage in that sense, hence my preference for the term “quarantine”. The scope of the quarantine isn’t important, but the fact that the artifact isn’t installable (or discoverable) is important.

Agreed, although s/staging/quarantine. Cooldowns are an important piece of the puzzle, but I agree that they are installer-facing, and thus under direct user control.

Staged or not (in the 694) sense, having an index scan uploads for known vulnerabilities is a very useful feature, but it’s separate from either the uploader or installer. It’s also an index-specific feature and thus may not need to be defined in a PEP. An index (like PyPI) could just say “there is an inherent delay between uploading an artifact and its (implicit) publishing of 5 minutes so that our index can scan it.” If there are no API or interoperability concerns, then it might not need a PEP.

What exactly is “the gatekeeping idea”? Is it just the scanning delay I described above?

First, in 694, the ability to create a stage as part of the multi-artifact upload process is not a requirement of all indexes. I wouldn’t be surprised if only PyPI implements staging in the 694 sense.

Second, I don’t see a way to implement staging with the implicit immediate publishing process of the legacy upload mechanism without some kind of new API. And if you’re going to implement a new API, then I think 694 is the right way to go. I’m open to suggestions for how to make the current legacy mechanism work with 694’s definition of stages, but I can’t see it. If we’ll need to add APIs, then do we want multiple likely different APIs? Probably not.

I can imagine an auto-quarantine for individual files uploaded using the legacy mechanism though and I think that would be a really useful feature for PyPI. I’d want to have a plan for the scanning side of the equation though, in order to make auto-quarantine useful.

23

Hmm, I feel scanning for vulnerabilities is actually yet another corner - weren’t we speaking about detecting malware? I think it’s a bit different thing, e.g. no need to hold a release because there is a vulnerability.

I agree on that, but I again feel confused about the scope. If we want to delay 5 minutes for malware detection, I defer again to my question about scanning sources and their quality. But I totally agree this is an index feature without need for PEP.

Exactly, what also was previously refered also as “security gate”.

Fully agree, and to be clear - I was thinking about PyPI as the primary consumer.

I think I wasn’t precise here. When I said about “being able to install”, the only thing I meant in this context was “the release file the index will later serve is public”. I’m aware this is a simplified situation and won’t fully work if the release depends on held dependencies, but my understanding was that we think about a simplified solution.

+1

I think we really need to define the scope because we are mixing a few things that can, but do not have to, be integrated into one solution (and even if I try to separate them, I’m still guilty of trying to solve multiple problems at once). I was focused on “staging”, and thus pushing into maintainer control and public visibility, but if the focus is on integrating scanning into the publishing process, then I see no point in messing up too much with staged publishing.

(the only digression: as already said, I see a publishing policy requiring manual approval as also significant feature, also index-specific and likely not requiering PEP - but maybe it should be a separate discussion topic?)

What do you mean exactly by “auto-quarantine”? Do you mean the delay for scanning, or automatically putting the release into quarantine after qualified reports? It confuses me because PyPI uses this term for the second (and it already exists), and my vision about any pre-publishing scanning is that puting a release into quarantine would be the possible outcome of it.

24

Thanks for publishing this @cjames23 - I’ve read through it once but definitely need to digest and compare against 694 a bit more, so take my questions and comments as fumblings in the semi-dark.

I now think you are effectively using “staging” in the same sense as 694, to some common extent, if different around the edges. I think your PEP is also allowing for implicit creation of a stage, which would also allow for atomic publishing.

A client requests that an uploaded file enter a staged release, rather than being published immediately, by including an additional form field staged with the value true in the legacy multipart/form-data upload request. All files uploaded with staged=true for the same normalized project name and version join the same staged release.

So this isn’t just a per-artifact construct; you really are allowing for multiple artifacts to now live in a common staging area. That’s a change in behavior that 694 makes explicit, but this PEP adds implicitly. A related question: what happens if I have 5 artifacts for a particular name-version pair, and forget to add staged=true for one of them? I assume the one with the missing key would get published immediately and not to a stage. What happens when the stage gets published? Is it possible to upload two duplicate artifacts, one to the stage and one not? How would that conflict be resolved?

694 explicitly wants to allow fully private releases, for testing and embargo purposes. This PEP doesn’t allow that IIUC. This PEP also isn’t explicit about some behavior, like, what if an artifact in the stage is corrupt? 694 explicitly allows mutability (overwrites, deletes) of artifacts in a stage, and then once published, all artifacts become immutable just like they are with the legacy upload.

The security gate described in this PEP is really an index-specific policy. The PEP could be clearer about the aspects which are protocol (and thus interop) requirements, and which are index-specific policy decisions for which PyPI or other indexes could make different decisions. An example of that would be this PEP’s fail-open timeout. PyPI could be 5 minutes, Artifactory could be 15 minutes, a GitLab package registry could be immediate. How would clients discover the policy and know how to drive the stage?

Another thought occurred to me as I read your PEP. We probably could turn this “inside out” so to speak. Meaning, 694 defines extensible upload mechanisms, with a default that every index must provide. I could almost see legacy upload as one of those mechanisms. Meaning, if I used 694 to create a stage and then used the stage token in a legacy upload, you could merge these ideas in a different way. You wouldn’t need two new endpoints, and since you are changing the legacy payload anyway (staged=true), it seems to me a 6/half-dozen difference[1].

  1. mostly, there’s still the implicit stage creation aspect of your PEP ↩︎

25

I personally want us to be more careful about requiring web UI login for controlling certain behaviors. We already have a number of things that can only be done in the web UI (e.g. create a project name in an org or move a project to an org) and that causes a lot of scaling problems for enterprises with zillions of packages. We’ve talked elsewhere about scriptable (REST) APIs and there are are some technical hurdles to get past first, but it would be super inconvenient if more functionality is locked behind a web UI login only.

There are important use cases for it though. Public scanning of staged release is at odds with private pre-release testing. I think we can probably do both if we think about it more. Plus, we may not need to make all staged releases public, e.g. if the index itself will security scan or allow for trusted partner clients to access staged releases for scanning. We’d have to work out some permissions with the latter though. An enterprise that’s doing its own scanning and wants embargoed atomic release might not be happy allowing third parties, even trusted ones, to scan their uploads before publishing.

26

What are we trying to solve with workflows like this though? If I’m a bad actor, can’t I just never enable secure publishing? Secure publishing might be useful for a good actor without the resources to do their own scanning, and wants some peace of mind that third parties will do it before their users get their hands on new releases.

There’s another gap that occurs to me: not all security issues are discoverable at the time of upload. What do you do about post-upload dependency attacks? I release foo 1.0 that depends on bar < 2. Today bar 1.9 is safe, but tomorrow a malicious actor uploads bar 1.10 with an attack. foo 1.0 hasn’t changed but is now vulnerable. It’s already been published so it can’t be scanned.

27

When I wrote that I was thinking that new uploads would automatically get the quarantine flag and the effects would be just the same as the post-publishing quarantine. Only after whatever scans occur give the artifact/release a clean bill of health would the auto-quarantine flag be removed. It could still be applied after the fact for legitimate malware reports.

28

I do not want to put any words into your mouth here, but as I read this it would imply that the belief would be that the only way to discover malicious packages is to have someone become a victim of the attack. There are plenty of ways where heuristic based scanners can apply here. Let’s take Miasma as an example, which used .pth files to download bun and then run a credential harvester javascript file. Where the previous versions of the infected libraries did not contain a pth file at all. So we could at the very least here use those types of heuristics and do checks for known attack styles. Having an auto-quarantine with a heuristic based scanner or even something like socket.dev, if Mike and others are open to it and want to try to get the OSS offering from them, would then catch these without putting any users at risk.

1 Like
29

I will have to go through the rest of your response and try to answer your questions but I think this one is maybe critical to answer. I think there are multiple forms of staging and that they can all use the same underlying mechanisms. What this boils down to is 0…N number of artifacts associated with a given package version are uploaded whether Upload 2.0 API or legacy. Those artifacts are then put into a staged state which can be explicit or implicit. I would argue that we can make going into staging implicit but moving out of staging can be explicit or implicit depending on the type of staging happening. I would lean into explicit and state that even with scanning where we say we are staging this until either the scanner completes or a time window has elapsed.
(The reason for the time window here is to not be completely disruptive and accept that no signal should be treated the same as a good signal until proven otherwise.) Once that happens a maintainer will need to explicitly take an action to move the artifacts into an available state. I do not think this needs to be a UI feature necessarily, but I did lean into Kamil’s idea of a separate token so that it makes compromising the process mean that both credentials have been compromised.

30

It would seem better if adding a cooldown/quarantine period was the recommended security default (which appears to be the case for “public package ecosystems” like PyPI and npm) for PyPI to perform this task rather than requiring all users to configure this (which would otherwise be trap for novices). How quickly do uploads on PyPI go from the HTTP success response sent back the uploader to being widely available (seconds? minutes?)?

1 Like
31

That’s back to the “index policy vs standard” question, though. If an index wants to apply scans before making an upload available, that’s index policy, and doesn’t need a standard. We’re not going to require scanners for all index implementations.

Maybe there’s an argument for some way for an uploader to say “this release is done now”, which indexes could use (or not, based on their policy) to take various actions:

  • Run scanners on the release as a whole
  • Un-quarantine the release
  • Make the release immutable

But if the way to do that is via the index UI, that’s an index UI matter, and again not a matter of standards. It’s only worth standardising if it’s part of the upload API, and that means somehow modifying the API to include a concept of a “release” that’s made up of multiple uploads. And isn’t that just PEP 694?

I’m not against PyPI implementing this stuff, I just don’t think there’s a need for a separate standard - and I think any standardisation energy is better spent on PEP 694.

2 Likes
32

I’m thinking of the possibility that an established project might be compromised in some way rather than a new project that is created by a bad actor. As I understand it the recent Shai Hulud worm attacks basically work like this:

  • A GitHub Actions CI job for established project A is compromised in some way.
  • The compromised CI job is used (either directly or via some secret extraction) to upload malicious versions of A to npm/PyPI.
  • Project B pulls in the malicious version of A in its own CI job
  • And so on (hence “worm”)…

The step that I want to break is that a compromise of the CI job should not lead to a published release on PyPI. This is part of what I meant by:

It is very easy to create vulnerable GitHub Actions jobs. I have done it myself more than once and was just lucky that no malicious person discovered before I did. I think many open source projects are just lucky that no one has tried particularly hard yet to hack their CI given how easy it is to screw this up.

For anyone concerned about this I definitely recommend using @woodruffw’s zizmor to check any GitHub Actions workflows (this would have prevented the mistakes that I previously made) and being as generally strict as possible. Note also that if there is a vulnerability that can be triggered by a malicious PR then an attacker can open a PR against any old branches in the repo so it is not enough to secure the workflows only on the main branch (delete the old branches and stop creating branches in the main repo).

Apart from vulnerable CI jobs there is the question of someone’s GitHub account being compromised. A typical project that has many contributors will likely have many more people with push access to its GitHub repo than with push access to PyPI. If you have a trusted publishing workflow then that is an automated process that can potentially be triggered by that larger number of GitHub accounts. There are lots of knobs in GitHub to control these things but it is not trivial to audit all of the repo control settings and workflows and be confident that only the intended accounts can trigger/affect a release. It is much easier to audit the project configuration in PyPI where there are far fewer settings and accounts to check.

Also separately from security issues it is possible to have a bug in the release scripts or for someone to trigger a release workflow accidentally or to have upload fail part way through and so on. I would just prefer to be able to check the whole thing before the release goes live and I’d be happier knowing that the final gate for that is on the PyPI side where nothing that happens on GitHub can mess it up. These are actually the original reasons I wanted staged releases although these days security considerations are definitely the primary concern.

1 Like
33

This is valuable input. I started at PEP with the idea in my mind that if enough of the community and trusted delegates like you and Barry believe this can just be something that gets implemented as a PR in Warehouses then that is an easy enough transition without going all the way down either road. I do not think the discussion is wasted because either way it ends up refining the design here to something that the community finds as an agreeable one.

1 Like
34

Effectively instantly.

The flipside is that dependency cooldowns on the client side allows end users to easily bypass the cooldown when required-- for instance to upgrade a dependency when a critical security vulnerability is found.

It’s not clear to me how you’d do that for a server side cooldown, unless we added something like the “yanked” attribute that indicates the dependency is too new and shouldn’t be used unless some flag is passed-- but if we’re going to do that, then we already have the upload date and installers could just key off of that and default to having a cooldown?

35

But is that not exactly what trusted publishing is? If you don’t want CI to be able to publish your project, don’t use trusted publishing, and if you do, then you need to ensure your CI is trusted.

Are you not saying that you’d prefer it if trusted publishing was actually trusted staging, and publishing remained a manual operation? Because the popularity of trusted publishing suggests otherwise.

1 Like
36

This is why I have been talking about a time boxed window in parts of this discussion, I have been leaning towards a short window but I also have not checked to see what the timing would be with various scanners. I think there is a balance that will need to be found here between both security issues that present here between ensuring that critical vulnerability updates can be made available quickly to the ecosystem and preventing other supply chain attacks from reaching users.

I think a time boxed window for scanning at least gives us that balance. Yes it would potentially allow some supply chain attacks to get through but it would still catch enough that I think it would be worth it.

37

You can publish your project from CI without trusted publishing. In that case you take a token from PyPI, store it in the repo secrets and expose it to the relevant CI job. The downside of doing that is that someone might exploit a CI vulnerability to extract the secret and then they would have a long-lived token that could use to publish from anywhere at any time in future. With trusted publishing there is no long-lived token.

I suppose what you mean here is that isn’t this what pushing from CI to PyPI is? Yes, obviously you should try to secure your CI as much as possible to do this. That fact that you do your best to secure CI does not mean that you should configure all other systems to trust it fully. There is a nonzero chance that your CI gets hacked no matter how much you try to secure it.

Yes, I am saying that I would want trusted staging but I don’t think that the popularity of trusted publishing is relevant for judging whether that would be a popular option except to say that clearly people like automatic releasing from CI. I am saying that if a currently non-existent feature existed I would use it instead of the current trusted publishing feature. I think others (e.g. @kam193) would use it as well. The fact that it is not currently in use by many projects is because it does not exist yet.

Currently there is no distinction but with staged releases there is a separation between these two steps:

  • Transferring the files to PyPI
  • Publishing the release

The principle of least privilege says that CI only needs the authority to perform the transfer step. Many projects might want to give it the authority for both steps and that is fine but this is just a convenience vs risk tradeoff where for me the benefit of the convenience seems insignificant.

Bear in mind that releasing is usually not going to be a fully automatic process. The minimum amount of manual steps that most people would want is that you push a button to initiate an otherwise automatic release process. I’m just saying that at the end of that process I want it to wait for me to push another button that gives the final sign off and I want that second button to be on PyPI and completely independent of CI.

3 Likes
38

Thanks for the explanation, that clarifies things for me.

That’s fair. I will say that I’d prefer not to have to do a 2-step process, because in reality I don’t think I’d be checking anything between the two approvals.

I understand that the attack being defended against here is “someone compromises my CI credentials without me knowing, and releases a malicious copy of one of my packages”. In that case, I wouldn’t approve the PyPI release because I hadn’t done a CI release. But unless we make the second approval mandatory, I suspect the majority of publishers will choose convenience in the 99% case over security for the 1%. I know I’ll probably be one of those people.

I’m not going to object to having the option for a project to configure multiple approvals into the release process. I’m just not sure it’ll have as much impact as people seem to expect. Which is why I’d rather see effort spent on PEP 694, which updates and improves the whole process, for all indexes.

39

I wish it supported GitLab!

1 Like
40

I’m not sure I’ve said it out loud, but I really do appreciate the ideas you’ve brought up here and in the pre-PEP. I’m still churning things over in my mind.

2 Likes
41

I don’t see TP as something separate from “trusted staging”. In 694 – and I think in any proposal to add staging to PyPI – the publishing step still happens using the current TP protocol. Maybe think about it as Trusted Pushing although Trusted Upload might be more accurate.