PEP 458 already includes the requirement to allow bulk signing of existing packages:
PEPs are allowed to leave the details of some aspects as implementation details, and I think that’s appropriate in this case, as the best way to do the backsigning will depend on the exact mechanics of key rotation.
Thanks, Nick! Do you also consider the compression issue solved (i.e., PyPI can reuse any of the available compression mechanisms it employs), or would you like a bit more clarification?
From the perspective of a PyPI administrator, “compressed versions of all metadata SHOULD also be made available to clients via the existing Warehouse mechanisms for HTTP compression.” seems clear and reasonable to me on the compression front.
@mnm678 Were there other changes you wanted to make, particularly with regard to the figures / tables?
Yes, as @joshuagl noticed the figure on github does not match the one that renders in the pep. This seems to be due to a bug in the build script, but I haven’t had a chance to track that down. Perhaps a PEP admin would be able to help (a manual rebuild might resolve the issue). @ncoghlan @dstufft do you know a good contact for this?
Related to the earlier discussion about Transparency Logs (like Certificate Transparency), some people on the TUF team wrote a blog post comparing TUF with Transparency logs and discussing the potential combination of these solutions. The post is available here. Feedback and question are welcome!
I think having this be follow-up work makes sense.
That being said, having some collaboration/communication during the contracted period to ensure that we don’t make things too burdensome for people working on back-signing also makes sense to me. I think the prior work on 2FA/API tokens will serve as a reference for collaboration here: we’ll make an effort to break things into individual feature PRs, tie them to individual issues, and get feedback/collaboration ideas along the way.
I replaced the table image (pep-0458-1.png) with an actual table in secure-systems-lab/peps/#73 and renamed pep-0458-2.png -> pep-0458-1.png, which confused the pep converter. Apparently this was a known issue, and I think I found a fix for it:
It looks like discussion about the actual meat and potatoes of this PEP has petered out. Unless someone has an objection, I intend to accept this PEP on Friday.
I’ve not been super active here, but concerns were raised above about some of the language and I still think there are problems with it. As I understand it, this PEP deals with the threat model of “malicious actors breach (at least) the storage infrastructure of PyPI”, and wants to mitigate that by ensuring that if they try to manipulate artifacts in that storage, it will be detectable and either PyPI could/would refuse to serve such manipulated artifacts, or other tooling in the packaging ecosystem could/would refuse to accept or use such artifacts.
But this is not what probably 99% of people think of as “package signing” – the threat model implied by that wording is not a compromise of PyPI’s storage, but a compromise of an individual package maintainer’s account, and this PEP explicitly adds no new mitigations for that threat.
Given the amount of “Python 2 is still supported!” misinformation running around right now as a result of confusion over the poorly-communicated release mechanics for the final version of CPython 2.7, I don’t think it’s possible to just shrug and say we can’t control how other people will talk about this. We can foresee right now a very obvious way that people are going to misinterpret the PEP, and we can prevent that. We should prevent that. Title it to be about “storage infrastructure integrity” or something that more closely explains what exactly is being done here, because “package signing” is guaranteed to be read as meaning something else by vast numbers of non-engaged people who’ll hear about this.
getting rid of --generate-hashes --require-hashes and just using something like echo “–tl-root=${pip get-tl-root}” >> requirements.txt would save me precious time
I’m aware the TUF doesn’t preclude PKI-free TL put it doesn’t materially help me with my workflow.
I mean TUF isn’t bad and it doesn’t prevent TL I’m happy to wait till TUF lands and then bring up PKI-free TL later
I agree, we want to make sure that the title is as clear as possible. That being said, we had a lot of discussion about how to fit a clear title into 44 characters. The hope is that “secure pypi downloads” would help clarify what we mean by “package signing” while making it clear that the solution involves adding metadata and signatures to packages.
I still think that’s optimistic – when people see “secure PyPI downloads with package signing”, I’m pretty sure the thing they’re going to think is “package authors are being required to sign their packages, and that will make the downloads secure”. Not “PyPI is adding integrity checking to its storage infrastructure”.
So, again, I really think the words “package signing” should not appear in the title of the PEP at all.
While I recognize that the title of the PEP could possibly be improved upon, it appears that nobody has any concerns with the actual content of the PEP. Given that, I’m happy to provisionally accept the PEP at this point, if someone comes up with a better wording for the PEP title I think it would be fine to adjust it, but I also think we do not need to delay the PEP any further while that potentially happens.
I won’t be able to update the PEP repository myself at the moment, but I’m sure someone can, so feel free to consider this an official announcement.
The package signing we’re doing is exactly the same package signing that Fedora/RHEL/etc do: the distribution platform (koji in Fedora’s case, PyPI in ours) does the signing, not the individual package authors (the submission of builds to the build system is authenticated via regular account management systems, not by signing keys).
It’s still package signing, it’s just not package signing by individual publishers (that’s part of PEP 480).
Some folks may have an overly narrow view of “package signing” - that just means PEP 458 may be an opportunity for them to learn something new, it doesn’t mean that we should redefine terms to fit the expectations of people that think that publisher signing is the only kind of signing that exists.
Edit: reworded slightly to be less snarky 
Distros sign packages that are built on the distro’s buildsystem, asserting that the binary artifact corresponds to the source. That’s quite different from what PyPI does.
Of course, what PEP 458 proposes still is signing.
The most direct equivalent is between PEP 458’s sdist metadata and Fedora’s SRPM repos, where koji’s signature is saying “this is the source that was uploaded for building”.
I agree the assertions for wheels and RPMs aren’t exactly the same (PyPI makes the same “this is what was uploaded” assertion for wheels as it does for sdists, whereas koji makes the stronger assertion that “this RPM was built from this SRPM”)
Considering this question further, I realised there is a term that covers package signing that is handled by the distribution platform, while excluding publisher signing of individual packages: “signed repository metadata”.
If we tweaked the title and wording of PEP 458 to use that phrase, then I think @ubernostrum is right that it would reduce the chances of people getting confused and expecting to see end-to-end publisher package signing introduced right now, rather than that being considered as part of the follow on work in PEP 480 to address potential attacks on PyPI itself (rather than the attacks on mirrors and download channels that PEP 458 addresses)
PR making that change: https://github.com/python/peps/pull/1308/files
The PEP also tidies up a few remaining places that hadn’t been reworded since the PEP 458/480 split between the core repository metadata signing capability (PEP 458) and the more secure, but also higher friction, delegation of package metadata signing authority to individual publisher keys (PEP 480).
