Hey all, getting the ball rolling on this discussion. My proposal is that for Python 3.14 and onwards we stop providing GPG signatures for artifacts, instead relying entirely on Sigstore bundles to provide authenticity. Existing Python release streams would be unaffected and would continue providing GPG signatures until they are EOL.
CPython has been providing Sigstore bundles for around 2 years now, since Python 3.11. We were on the early side of adopting Sigstore back in 2022 (Sigstore started as a project in March 2021), at this point though Sigstore has been integrated into many software integrity schemes (including GitHub, PyPI, NPM, etc) and so is likely to stick around.
From a technological perspective, Sigstore has the benefit of not requiring release managers to maintain and protect separate long-lived expiring secret keys to maintain the integrity of Python artifacts, instead release managers only need to maintain control of an email account (something release managers already need to do). On the verifier side, Sigstore doesn’t require discovering and fetching keys which might become unavailable, out of date, etc and doesn’t require downloading an artifact to verify its integrity.
It’s likely that most signature verifications are happening in automated scripts such as distributions. Making this move now, it means we’ll have a year before the next stable release to get the word out to everyone still using GPG keys to start using Sigstore (if only to verify checksums that they have pinned, if there is no way to integrate Sigstore directly, yet).
Also note that last year PyPI dropped support for uploading GPG signatures for Python packages, although the justification there of “most signatures aren’t verifiable” isn’t as much the case for CPython beyond a handful of keys being slightly more difficult to discover or use for verification (see: “OpenPGP Public Keys” section in Download Python | Python.org)
Thank you for opening this @sethmlarson! I’m registering my strong support for this idea: IMO Sigstore offers better cryptographic primitives and a better UX for both signers and verifiers, and has development momentum behind it that we’re unlikely to see the PGP/GPG ecosystem regain.
Do we have any way to determine whether this is true and who might be using our GPG signatures? Or is this more of a “turn it off and see who yells” kind of thing? I’m wondering if brownouts of availability on our existing signatures would be useful, or whether this wouldn’t catch most use cases (my assumption is the more common use case is something like “verify the release once before integrating it into $distro”).
As an aside, I think Sigstore Information | Python.org approximates a migration guide but wasn’t written with that explicitly in mind. We might want to think about updating it with a section to address the perspective of “migrating from GPG to Sigstore”.
Likely better to proactively reach out to people first, but I’m not sure who we can contact right away. Maybe I’ll cc @ksurma as one person I know working on Fedora for some thoughts on how distros typically handle verification material.
Agreed, we can add a section for migrating that recommends a tool to use and how to turn GPG-verified material (likely a checksum that was previously verified with GPG) into something that Sigstore can verify (ie sigstore verify ... sha:deadbeef).
I would like to point out that Sigstore is still not a viable alternative to PGP. At least according to Sigstore information:
(Note that in addition to verifying the signature and certificate, this command performs an online verification that there is a corresponding entry in the signature transparency log as well. Offline verification is not yet supported)
And indeed, the linked issue is open for 18 months now. I haven’t even started testing it because of that.
it is written in Python and has a lot of dependencies
including some questionable or problematic ones, like pyOpenSSL of certifi
this makes packaging sigstore for Fedora generally harder, but not impossible
this makes bootstrapping Python more complex (we might need to temporarily disable the verification until we can build sigstore with the new Python version[2][3])
Just +1 to Miro from openSUSE. If there was ever a time when something should be rewritten from Python to some compiled language, sigstore is probably the utility.
Thanks @mgorny@hroncok and @mcepl for providing feedback, this is great information to have so we can prioritize.
For having a compiled binary for Sigstore, I was pointed to the sigstore-go client as an option. In fact, the sigstore-go client already lists offline verification on their list of features. I’ll double-check that that is correct, though!
FWIW, this particular piece of information has been outdated for a while: sigstore-python performs zero online log lookups by default if you’re verifying with the “bundle” format (which is what CPython distributes). It will perform an online lookup by default if you pass “detached” inputs, but these are deprecated and you can also explicitly disable online lookups in this case by passing --offline.
TL;DR: If you pass --offline while calling sigstore verify’s subcommands, you will never perform an online log lookup.
TL;DR: Starting with the next release, --offline will also mean that sigstore-python performs no automatic trust root updates. You can confirm this yourself (once released) by running it in an un-networked environment, e.g. using eatmynetwork
Just FWIW from the Windows side (and probably the macOS side), the GPG sigs are entirely redundant with the native OS embedded signatures we use, and the only reason we still do them is because people shouted last time we stopped.
Their argument IIRC was that they wanted to validate our artifacts automatically and let us know if we had been hacked, and to do it from Linux without needing any specific Windows or macOS logic. This isn’t any particular benefit to us, IMHO, but the shouting stopped when we did the sigs again, and at least for Windows they’ve been automated ever since.
Well, the same argument is valid for sigstore sigs
Could I ask why we can’t just continue to provide GPG sigs as well as sigstore sigs ? Is that really such a burden on the release manager ?
GPG sigs are well known, can easily be verified and have been around for ages, so people know how to use them. This is not (yet) true for sigstore sigs.
This is exactly my feeling too, albeit from a perspective of principle, since this proposal doesn’t affect me in practice whatsoever: the OP has not quantified what the benefit of the proposal is, i.e. how much time/effort/money/frustration/etc. would be saved by not providing GPG signatures.
Personally I’d rather see Python continue to provide GPG and Sigstore signatures in parallel for another whole ten years before considering the possibility of dropping GPG - especially if no problems with GPG have actually been demonstrated. Anything less than that, I’d argue, is not enough time to find out whether some new technology (Sigstore) stands the test of time. And at a glance, to me, Sigstore (which I’d never heard of before reading this thread) looks similar to many other new-fangled technologies: a centralisation, a silo, a SPOF, a place where someone with a conflict of interest might easily be able to cause trouble.
As was mentioned, Sigstore has all of the development momentum behind it, evidenced by how many ecosystems are adopting the technology right now. Some ecosystems, PyPI among them, are removing support for GPG for a variety of reasons some which do and some don’t apply to CPython. I am doubtful GPG will see that sort of momentum return this late in its lifespan.
Long-lived GPG keys in practice means decisions we make today affect release managers for a minimum of 7 years. 1 year of pre-releases, then at least 2 consecutive yearly releases with a total of 5 years of security releases after. So if we intend to support GPG keys for “10 more years” as was mentioned, that means we’d drop them in 3 years which is not that different than dropping them for 3.14 in a years time.
We can always add back, but we can’t take away. Like @steve.dower mentioned in his example, we can add back GPG signatures if we get it wrong and there’s actual blockers for adopting Sigstore signatures but once we commit to there being GPG signatures for a release we likely wouldn’t want to remove that promise.
GPG is a non-zero amount of work to maintain as a signer, compared to Sigstore which is effectively zero (you have to click a button to OAuth sign-in once per release) multiplied by ~4 release managers at any one time. The upcoming release manager currently doesn’t have a GPG key or any experience with the technology, so this felt like a good time to propose this change. Myself personally I have a GPG key, but I wouldn’t recommend anyone create or use one these days.
Without a distribution attempting to use Sigstore we won’t discover and fix the blockers for adopting this technology. Unless we can convince a distribution to voluntarily try to use Sigstore and our signatures, these sorts of threads are the only mechanism we have to gather feedback short of removing GPG and seeing what happens. This thread already showed that there were misunderstandings about the current adoptability of Sigstore (ie offline verification).
I would take issue with the statement that GPG signatures “can easily be verified” and “people know how to use them”.
I am the sort of person who has two Gemalto OpenPGP hard tokens in my desk drawer in front of me even now, and yet every time I have attempted to productionize a verification pipeline I have managed to set up the keychain so that the gpg command line would somehow verify that the signature was valid (i.e. was signed by the key it said it was) but not verify the correct WOT relationship that I was trying to depend upon from the strong set. One of these even had a unit test for this behavior that was set up wrong, and managed to be in production for a little while, verifying nothing.
If you want to read about the problems with PGP and GnuPG, this article is forceful in its condemnation:
I am not really a stakeholder here and I don’t need to advocate strongly against PGP. If y’all decide they’re necessary for some of your downstreams I can easily continue ignoring them for the next decade as I have for the previous ones. But per Seth’s example that this is going to consume a release manager’s time, the use of PGP signatures on Python is also a social issue, not just a technical one. Coders who look at the Python build process and run across those signatures will assume that they’re a good idea, and given Python’s extremely broad popularity, this could mean that dozens or hundreds of people could spend dozens or hundreds of hours of each of their lives, wasting time puzzling out the massive historical complexity of the GnuPG command line, rather than just using something like sigstore or sigsum to solve their problem more expeditiously.
So I really do think getting rid of them would be a good thing.
It was just pointed out to me that Debian hasn’t been using GPG to verify upstream tarballs, and I am about to fix that.
As to sigstore, we’d need to support it in our uscan tool, to have automated verification happen. To include the signature in our source packages, as PGP signatures are, we’d need to modify dpkg-source, and possibly the archive, to support storing them. It’s hard to motivate that change for a single upstream. I know there are other ecosystems supporting it, but I’ve never run into any doing so actively.
So… we can use GPG and really should. I’ll do that. If it were dropped, we’d rely on local manual verification (probably forgotten), rather than being able to migrate completely to sigstore.
Is there a reason why GPG signatures weren’t being used by Debian in the past, despite being available?
Agreed that it’s tough to justify for a single upstream, however I know that in the near future there will be multiple thousands of projects with Sigstore verification materials on PyPI thanks to the work of @woodruffw on PEP 740. I know that Debian/Ubuntu package projects from PyPI and for many of these projects there aren’t any GPG signatures available, would this volume of projects and viable verification materials justify adopting Sigstore in uscan?
The package maintainers never got around to setting them up. Or didn’t consider them to be important.
I’ll file bugs requesting the support and we’ll see how things go. I think we’re broadly in agreement with the goals here. It’s probably mostly a matter of getting the right tooling packaged and convincing people that sigstore is worth the effort of implementation in Debian package tooling.
