PEP 761: Deprecating PGP signatures for CPython artifacts

PEPs
21

Indeed, this is why I’ve avoided in the PEP using the security of PGP as a justification for its discontinuance. I don’t think we need to prove or disprove whether PGP is “dead” or insecure to improve the ergonomics of our release workflow.

This would still require management by someone, likely release managers, and mostly for the benefit of downstreams that are likely to want to adopt Sigstore anyway to serve the thousands of PyPI packages that will be providing verification materials in the near-future.

I apologize if the PEP had this tone anywhere as that wasn’t the intention, I don’t subscribe to “new shiny thing”-driven development. I tried to capture the difference in security model in the “Security Considerations” section without duplicating too much of the nitty-gritty, instead linking out to Sigstore’s more authoritative documentation for folks interested in the details.

Thanks for joining this thread and providing feedback!

I’ll ask @woodruffw to publish a release of the Python Sigstore client (edit: Sigstore-Python v3.4.0 is now available on PyPI, thanks William!), I know the Go client is also available which Gentoo might be interested in. edit: I’ve confirmed directly with the Sigstore Go client folks that the client supports offline verification without TUF so should be acceptable to Gentoo’s needs, please report back with any issues.

The final decision on discontinuation timeline rests with the SC, accepting or editing this PEP doesn’t remove the ability to delay discontinuation. To inform the SC decision-making and Sigstore maintainers on what they might work on next, could you answer these questions:

When does Gentoo start packaging a new Python version, betas? In the most aggressive timeline the first 3.14 beta is available mid-May, meaning 7 months before any real “disruption” is experienced. This can be extended if a manual verification process is acceptable for some time (which is what Debian maintainer @stefanor mentioned as their worst-case if Sigstore support can’t be added to uscan fast enough).

Are there any other blockers to Sigstore adoption beyond offline verification? I’m aware you may not have had an opportunity to fully test Sigstore yet, but how soon could you know whether there are blockers or not?

It’s definitely the plan to put a warning on the PGP signature page if this PEP is accepted. I think it’s better to get awareness and feedback earlier, because I would much rather have either verifiers or Sigstore folks working on fixing the integration problems. If you have ideas where this thread can be syndicated to get more feedback into specific difficulties around using Sigstore as a distro that would be very helpful.

Thanks again!

6 Likes
22

a shared CI-managed GPG key is a very different threat model for CPython releases than the current RM-held key model! It might be an acceptable one, but it changes the model from one where only the RM is trusted to sign to one where anybody with access to the key can sign, with those parties being indistinguishable at the identity layer. In practice, I’m also not sure that PGP key rotation practices are mature enough/easy enough to operationalize to prevent the normalization of deviance around key managment

OpenPGP keys can definitely be made to work safely in CI-driven release automation, but it’s not trivial. A large open source software ecosystem I participate in operates this way.

We export a signing-only subkey from a master release key that’s stored encrypted in a location only accessible to our infrastructure admins and rotated twice yearly. The exported subkey is symmetrically encrypted by a secret key that can only be used by isolated jobs in our CI automation. All release activities go through code review, and trigger CI jobs which ultimately sign Git tags and make detached signatures of release artifacts for publication. Public code review provides transparency and accountability, since the release managers don’t directly create and sign tags or anything else themselves (and the CI jobs embed code review authorization metadata in the tag bodies too for traceability).

The key rotation process is thoroughly documented and there are scheduled reminders to do it, with expirations set to a long enough cushion that it’s not time-sensitive and can be performed well in advance. Each new release key is signed, at a minimum, by the preceding release key and by the well-known personal key of the infrastructure admin who generated it (but ideally also has attestations from other administrators who double-checked that work). Exports of public parts of the master release keys along with all relevant attestations are stored in Git and published to a release site for ease of access, along with pushing them to a popular keyserver network (keys.openpgp.org doesn’t distribute the additional attestations, but those can be downloaded from our release site for anyone who wants them). Release tests ensure that the key rotation was performed correctly and didn’t somehow break any of the release automation in unexpected ways.

Yes it means that a small set of trusted infrastructure admins could subvert release automation, but since they manage the systems where the release artifacts are built and published as well as the code review systems where primary copies of all the project’s Git repositories reside, this was basically possible anyway. It also means the project’s release managers don’t need OpenPGP keys of their own, and don’t even need to know how to use tools like GnuPG. And if the project didn’t have enough people to maintain these systems with a firm grounding in the relevant tools and protocols, it wouldn’t really be possible.

I get the impression that CPython doesn’t really have enough sysadmins to maintain project infrastructure at that scale, given they already made the choice to stop hosting their own Git repositories and defect tracker. In that case, moving more parts of the release process to a third party “signing as a service” provider like Sigstore makes sense, even if it means the project itself is giving up more control and having to place their trust in the operators of additional outside resources.

6 Likes
23

The problem with this is there are better options for release signing than pgp even when sticking to “we host everything and don’t rely on an external provider”. People learned from the mistakes of pgp and rsa and (generally speaking) no longer create schemes that can fail in non-obvious ways or that have as much complexity to set up everything properly.

pgp is a relic of the past, and it is an objectively bad option prone to multiple categories of issue. I don’t want to turn this thread into “just bash pgp” but I don’t think we should be entertaining the pgp is a good option in some sense of balance or “well this could work if only you had all these resources set up”, only that it is the status quo that cpython has reasons to move away from, and that the option being moved to has been considered for the ergonomics and threat model.

4 Likes
24

I stated Debian’s position in the other thread, but summarizing again:

We typically start packaging a new cPython versions at the Beta 1 version.

  1. sigstore isn’t currently packaged in Debian. There are a few prerequsites (like tuf).
  2. Debian’s automated upstream signature verification (in uscan) doesn’t support sigstore yet. This should be achievable once it’s in the archive.
  3. Debian’s source package format supports including upstream OpenPGP signatures, but not sigstore yet. This probably won’t happen in a hurry, the dpkg maintainers will probably want to see significant uptake of sigstore.

We’ll have to manually verify new releases. That was the status quo until a week ago. So not a huge loss.

6 Likes
25

From the PEP:

When in offline mode, Sigstore can’t verify whether a signature has been revoked.

Is there any way to actually make this work? See below on how it works with PGP.

This is a similar restriction to PGP key revocations not being detectable during offline verification.

This is false. PGP will detect signatures made using revoked keys, provided that you update the keys first. It is not uncommon to periodically update the keys online, and then use them offline to verify signatures.

For example, Gentoo keeps a fair number of “key packages” that are periodically updated. If a key is compromised, we learn during this update and naturally, all subsequent offline verifications will fail.

Is no equivalent option available for sigstore?

2 Likes
26

That’s exactly how it works in Sigstore too: if you update your trust root first (which is done by default in online mode), you’ll always have timely key material.

My interpretation of the PEP’s point is that neither PGP nor Sigstore (nor any signing mechanism) can discover revocations while offline, not that you can’t check revocations that have already been discovered while online.

3 Likes
27

Do all relevant revocations get fetched or only these relevant to the signature in question? In other words, how do we ensure that the “trust root” we include in Gentoo has all revocations that could be relevant to all releases of CPython?

1 Like
28

All relevant locations get fetched, yeah. The subtlety with Sigstore is that it doesn’t do per-key/cert revocations, because the signing model is entirely built around short-term signing keys rather than long-lived ones. As such the trust root doesn’t include per-key revocation information; it includes revocation/expiry information for the various PKI roots within Sigstore themselves, which are the only things that live longer than the ~15 minute signing window.

(As an analogy, Sigstore’s signing and key integrity model is comparable to the Web PKI’s in the shorter-lived certificate setting: revocations on a per-certificate level don’t scale well, so the trick is to instead reduce the lifetime of the key material itself to below the timeframe in which revocation normally occurs.)

2 Likes
29

Hi!

Seth reached out to us in PEP 761: Deprecating PGP signatures · Issue #1660 · heroku/heroku-buildpack-python · GitHub since we build Python binaries for use with both Heroku’s classic Python buildpack and the upcoming Python Cloud Native Buildpack.

This requires building all non-EOL Python major versions for all currently supported Ubuntu LTS releases, so that end users can use any permutation of Python version with all of Heroku’s supported stacks.

We currently verify the source archives using the PGP signatures, however, would be happy to switch to Sigstore so long as we have an easy way to bootstrap a Sigstore client.

I’m presuming our options would be to either:

  1. Use the distro system Python install to install sigstore · PyPI using pip.
  2. Or, wait for Debian to package Sigstore, Ubuntu to pick up the package, and for us to install it along with the other distro packages we already install.
  3. Or, download/use a standalone client such as GitHub - sigstore/sigstore-go: Go library for Sigstore signing and verification to avoid the system Python dependency.

It seems that (2) might be problematic since we’d also need support on older Ubuntu LTS releases such as Ubuntu 20.04, which I presume might not get any package additions? And for (3), sigstore-go doesn’t ship pre-built binaries, so would require a Go toolchain.

As such, it seems like (1) might be our only/best option?

4 Likes
30

Thank you for providing feedback! So I’ve been digging in to the distro use-case more to find a better solution that:

  • Can verify Sigstore bundles for files
  • Allows for offline verification with minimal manual “online” setup (analog to fetching and storing PGP keys for later verification use). Verification might happen at multiple stages, both for building the package in centralized build infrastructure or when a user is building a package locally from source.
  • Revocation checks and updates to the root of trust can be done periodically online. Root of trust may be distributed via its own package which is updated periodically.
  • Doesn’t require build toolchain (sigstore-go) or a runtime/many dependencies (sigstore-python)

If there are other qualities that are needed for distros, please let me know. I am spending time to engage with Sigstore folks to make sure your use-cases are covered and they get encoded in the PEP and documentation for posterity.

If those all look good, it seems that Cosign (another Sigstore CLI tool) builds and ships binaries for easy installation and also supports offline verification as long as the trust roots are downloaded.

    # Download Cosign
    wget https://github.com/sigstore/cosign/releases/download/v2.4.1/cosign-linux-amd64

    # For offline verification, also need the Root of Trust.
    # Can be grabbed from GitHub:
    # https://github.com/sigstore/root-signing/blob/main/targets/trusted_root.json
    wget https://raw.githubusercontent.com/sigstore/root-signing/refs/heads/main/targets/trusted_root.json

    # Download CPython artifacts
    wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tgz
    wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tgz.sigstore

    ./cosign-linux-amd64 verify-blob \
      --new-bundle-format \
      --certificate-oidc-issuer 'https://accounts.google.com' \
      --certificate-identity 'thomas@python.org' \
      --bundle ./Python-3.13.0.tgz.sigstore \
      # --offline and --trust-root optional for offline verification
      --offline \
      --trusted-root ./trusted_root.json \
      ./Python-3.13.0.tgz

This matches more closely to (3), let me know if this fits well for you. Also cc-ing @mgorny @stefanor and @thesamesam @hroncok for opinions on this.

5 Likes
31

…and a good way to handle revocations. It’s important that “primarily offline” doesn’t imply inherently insecure.

1 Like
32

I hope you don’t mind me asking further, because — to be honest — I find SigStore’s documentation a bit overwhelming, and I’m not sure if I’m missing something. That said, if you think it’d be better to move the discussion around technicalities elsewhere, please let me know.

What I’m wondering about is whether SigStore has a way to indicate potential revocations when the trust root update is done. Let me illustrate on the example of PGP.

We provide “key bundles” in Gentoo, that are used to verify signatures offline. We can also periodically refresh the relevant keys to check them for updates. During this update, we can get a clear indication whether any of the subkeys were revoked — which give us a clear signal that we need to check the signatures made using that key, or perhaps more practically, recheck packages using that key bundle.

Is something similar possible for SigStore? Because the best I was able to figure out so far is for the trust root update to indicate that something has changed (and I suspect this will happen relatively often), and then recheck all signatures on all packages using SigStore.

2 Likes
33

I’m happy to discuss technicals here, but it’s ultimately @sethmlarson’s call IMO since it’s his PEP. If he prefers that we keep it somewhere else, I’d be happy to use another thread/chat/whatever.

The trust root should change relatively infrequently: the only things that should change are routine root updates (which are scheduled on the order of decades). The exceptional case is for key compromise/disclosure, in which case the trust root will be updated to remove (or constrain the validity of) the distrusted key, thereby intentionally breaking verification by breaking the chain of trust to the root.

In other words, the process on trust root changes should look similar to that for handling PGP revocations: if a member of the trust root is compromised, you should perform an audit of all signatures that chain to that root during the window of compromise. One of the nice properties of Sigstore is that signed timeliness is baked into the scheme, so you don’t have to go backwards in time and verify all signatures ever; only those that occurred during the window of compromise.

34

Totally happy to have technical discussion happening here, I suspect many questions that others will inevitably have regarding this transition will get answered ahead of time that way.

Yes, the model you’re looking for is being able to verify signatures without revocations during offline builds but periodically checking the trust root for updates online. I know that Debian/Ubuntu have uscan that checks for upstream updates, is there such a tool in use for Gentoo? Do the revocation checks occur during this part of the process, or elsewhere?

I updated my list to add periodic revocation and root of trust updates online.

2 Likes
35

Currently, the signatures are checked whenever the relevant package is built from source and the user has signature verification enabled (which should be pretty often). Trust root is distributed as a separate package that also has a test phase that verifies whether it is up-to-date. The latter part is pretty much an experiment right now, with the idea that we’d periodically rerun the test phase to verify if we need make a new version. I’d hardly call this perfect but that’s what I came up with on such a short notice.

2 Likes
36

We cannot download binary programs and then run them during the build.

We would need to build cosign in Fedora first.

2 Likes
37

That’s good to know, so you’d need to choose either a Go or Python toolchain, even with Cosign. That specific case was in reply to @edmorley who is building container images so different requirements.

38

The amount of new discussion has quieted down a bit, I have submitted the PEP to the Steering Council for review. Please continue to discuss the PEP here if there are more things to add or more context to provide to the Steering Council.

5 Likes
39

The Steering Council approves PEP-761. :white_check_mark: :balloon:

As someone who personally first used PGP in the modem and BBS era… Thanks for the writeup and discussion!

13 Likes