PEP 458: Secure PyPI downloads with package signing

Related to the earlier discussion about Transparency Logs (like Certificate Transparency), some people on the TUF team wrote a blog post comparing TUF with Transparency logs and discussing the potential combination of these solutions. The post is available here. Feedback and question are welcome!

3 Likes

I think having this be follow-up work makes sense.

That being said, having some collaboration/communication during the contracted period to ensure that we don’t make things too burdensome for people working on back-signing also makes sense to me. I think the prior work on 2FA/API tokens will serve as a reference for collaboration here: we’ll make an effort to break things into individual feature PRs, tie them to individual issues, and get feedback/collaboration ideas along the way.

3 Likes

I replaced the table image (pep-0458-1.png) with an actual table in secure-systems-lab/peps/#73 and renamed pep-0458-2.png -> pep-0458-1.png, which confused the pep converter. Apparently this was a known issue, and I think I found a fix for it:

2 Likes

It looks like discussion about the actual meat and potatoes of this PEP has petered out. Unless someone has an objection, I intend to accept this PEP on Friday.

8 Likes

I’ve not been super active here, but concerns were raised above about some of the language and I still think there are problems with it. As I understand it, this PEP deals with the threat model of “malicious actors breach (at least) the storage infrastructure of PyPI”, and wants to mitigate that by ensuring that if they try to manipulate artifacts in that storage, it will be detectable and either PyPI could/would refuse to serve such manipulated artifacts, or other tooling in the packaging ecosystem could/would refuse to accept or use such artifacts.

But this is not what probably 99% of people think of as “package signing” – the threat model implied by that wording is not a compromise of PyPI’s storage, but a compromise of an individual package maintainer’s account, and this PEP explicitly adds no new mitigations for that threat.

Given the amount of “Python 2 is still supported!” misinformation running around right now as a result of confusion over the poorly-communicated release mechanics for the final version of CPython 2.7, I don’t think it’s possible to just shrug and say we can’t control how other people will talk about this. We can foresee right now a very obvious way that people are going to misinterpret the PEP, and we can prevent that. We should prevent that. Title it to be about “storage infrastructure integrity” or something that more closely explains what exactly is being done here, because “package signing” is guaranteed to be read as meaning something else by vast numbers of non-engaged people who’ll hear about this.

2 Likes

getting rid of --generate-hashes --require-hashes and just using something like echo “–tl-root=${pip get-tl-root}” >> requirements.txt would save me precious time

I’m aware the TUF doesn’t preclude PKI-free TL put it doesn’t materially help me with my workflow.

I mean TUF isn’t bad and it doesn’t prevent TL I’m happy to wait till TUF lands and then bring up PKI-free TL later

1 Like

I agree, we want to make sure that the title is as clear as possible. That being said, we had a lot of discussion about how to fit a clear title into 44 characters. The hope is that “secure pypi downloads” would help clarify what we mean by “package signing” while making it clear that the solution involves adding metadata and signatures to packages.

I still think that’s optimistic – when people see “secure PyPI downloads with package signing”, I’m pretty sure the thing they’re going to think is “package authors are being required to sign their packages, and that will make the downloads secure”. Not “PyPI is adding integrity checking to its storage infrastructure”.

So, again, I really think the words “package signing” should not appear in the title of the PEP at all.

While I recognize that the title of the PEP could possibly be improved upon, it appears that nobody has any concerns with the actual content of the PEP. Given that, I’m happy to provisionally accept the PEP at this point, if someone comes up with a better wording for the PEP title I think it would be fine to adjust it, but I also think we do not need to delay the PEP any further while that potentially happens.

I won’t be able to update the PEP repository myself at the moment, but I’m sure someone can, so feel free to consider this an official announcement.

2 Likes

The package signing we’re doing is exactly the same package signing that Fedora/RHEL/etc do: the distribution platform (koji in Fedora’s case, PyPI in ours) does the signing, not the individual package authors (the submission of builds to the build system is authenticated via regular account management systems, not by signing keys).

It’s still package signing, it’s just not package signing by individual publishers (that’s part of PEP 480).

Some folks may have an overly narrow view of “package signing” - that just means PEP 458 may be an opportunity for them to learn something new, it doesn’t mean that we should redefine terms to fit the expectations of people that think that publisher signing is the only kind of signing that exists.

Edit: reworded slightly to be less snarky :slight_smile:

1 Like

Done: https://github.com/python/peps/pull/1306/files

Distros sign packages that are built on the distro’s buildsystem, asserting that the binary artifact corresponds to the source. That’s quite different from what PyPI does.

Of course, what PEP 458 proposes still is signing.

1 Like

The most direct equivalent is between PEP 458’s sdist metadata and Fedora’s SRPM repos, where koji’s signature is saying “this is the source that was uploaded for building”.

I agree the assertions for wheels and RPMs aren’t exactly the same (PyPI makes the same “this is what was uploaded” assertion for wheels as it does for sdists, whereas koji makes the stronger assertion that “this RPM was built from this SRPM”)

Considering this question further, I realised there is a term that covers package signing that is handled by the distribution platform, while excluding publisher signing of individual packages: “signed repository metadata”.

If we tweaked the title and wording of PEP 458 to use that phrase, then I think @ubernostrum is right that it would reduce the chances of people getting confused and expecting to see end-to-end publisher package signing introduced right now, rather than that being considered as part of the follow on work in PEP 480 to address potential attacks on PyPI itself (rather than the attacks on mirrors and download channels that PEP 458 addresses)

1 Like

PR making that change: https://github.com/python/peps/pull/1308/files

The PEP also tidies up a few remaining places that hadn’t been reworded since the PEP 458/480 split between the core repository metadata signing capability (PEP 458) and the more secure, but also higher friction, delegation of package metadata signing authority to individual publisher keys (PEP 480).

Doesn’t Debian allow maintainers to upload binary packages that the maintainer built on their personal system?

Yes, but we are moving away from it. Already for the next release packages are not allowed into the Testing repository unless they were built on a buildd. There are reasons to allow maintainer built binaries, such a bootstrapping, but they can (and will be) rebuilt before making it into a release.

On a related note, Debian also publishes a maintainer keyring that can be used to verify developer signatures on all uploads (source or source + binary), so people can verify what’s in the archive is what they uploaded.

We’ve also been expanding coverage of verifying upstream signatures when they are provided, so developers can verify that the upstream code they have is what the upstream released.

All of that is on top of the signed packages file that is used automatically on end-user systems to verify that the correct (unmodified) package is being installed on that system.

2 Likes

Thanks @dstufft for your decision, and thanks to everyone who commented here or in other fora. I’m not a PEP author and I appreciate all the work and improvements that have gone into this proposal since its start 6+ years ago.

There’s now a Zulip chat stream where the PEP authors and the implementor (@woodruffw) will be discussing implementation details, timeline, and so on.

2 Likes

The PR updating the PEP title to be “Secure PyPI downloads with signed repository metadata” has been merged, and will change on https://www.python.org/dev/peps/pep-0458/ with the next content update (usually within the hour).

3 Likes

For those following along, be aware when reading the PEP that it is still displayed with the incorrect image.

@lukpueh submitted a PR to address the underlying issue: https://github.com/python/pythondotorg/pull/1556