PEP 458: Secure PyPI downloads with package signing

PEPs
#111

I’ve not been super active here, but concerns were raised above about some of the language and I still think there are problems with it. As I understand it, this PEP deals with the threat model of “malicious actors breach (at least) the storage infrastructure of PyPI”, and wants to mitigate that by ensuring that if they try to manipulate artifacts in that storage, it will be detectable and either PyPI could/would refuse to serve such manipulated artifacts, or other tooling in the packaging ecosystem could/would refuse to accept or use such artifacts.

But this is not what probably 99% of people think of as “package signing” – the threat model implied by that wording is not a compromise of PyPI’s storage, but a compromise of an individual package maintainer’s account, and this PEP explicitly adds no new mitigations for that threat.

Given the amount of “Python 2 is still supported!” misinformation running around right now as a result of confusion over the poorly-communicated release mechanics for the final version of CPython 2.7, I don’t think it’s possible to just shrug and say we can’t control how other people will talk about this. We can foresee right now a very obvious way that people are going to misinterpret the PEP, and we can prevent that. We should prevent that. Title it to be about “storage infrastructure integrity” or something that more closely explains what exactly is being done here, because “package signing” is guaranteed to be read as meaning something else by vast numbers of non-engaged people who’ll hear about this.

2 Likes
#112

getting rid of --generate-hashes --require-hashes and just using something like echo “–tl-root=${pip get-tl-root}” >> requirements.txt would save me precious time

I’m aware the TUF doesn’t preclude PKI-free TL put it doesn’t materially help me with my workflow.

I mean TUF isn’t bad and it doesn’t prevent TL I’m happy to wait till TUF lands and then bring up PKI-free TL later

1 Like
#113

I agree, we want to make sure that the title is as clear as possible. That being said, we had a lot of discussion about how to fit a clear title into 44 characters. The hope is that “secure pypi downloads” would help clarify what we mean by “package signing” while making it clear that the solution involves adding metadata and signatures to packages.

#114

I still think that’s optimistic – when people see “secure PyPI downloads with package signing”, I’m pretty sure the thing they’re going to think is “package authors are being required to sign their packages, and that will make the downloads secure”. Not “PyPI is adding integrity checking to its storage infrastructure”.

So, again, I really think the words “package signing” should not appear in the title of the PEP at all.

#115

While I recognize that the title of the PEP could possibly be improved upon, it appears that nobody has any concerns with the actual content of the PEP. Given that, I’m happy to provisionally accept the PEP at this point, if someone comes up with a better wording for the PEP title I think it would be fine to adjust it, but I also think we do not need to delay the PEP any further while that potentially happens.

I won’t be able to update the PEP repository myself at the moment, but I’m sure someone can, so feel free to consider this an official announcement.

2 Likes
#116

The package signing we’re doing is exactly the same package signing that Fedora/RHEL/etc do: the distribution platform (koji in Fedora’s case, PyPI in ours) does the signing, not the individual package authors (the submission of builds to the build system is authenticated via regular account management systems, not by signing keys).

It’s still package signing, it’s just not package signing by individual publishers (that’s part of PEP 480).

Some folks may have an overly narrow view of “package signing” - that just means PEP 458 may be an opportunity for them to learn something new, it doesn’t mean that we should redefine terms to fit the expectations of people that think that publisher signing is the only kind of signing that exists.

Edit: reworded slightly to be less snarky :slight_smile:

1 Like
#117

Done: https://github.com/python/peps/pull/1306/files

#118

Distros sign packages that are built on the distro’s buildsystem, asserting that the binary artifact corresponds to the source. That’s quite different from what PyPI does.

Of course, what PEP 458 proposes still is signing.

2 Likes
#119

The most direct equivalent is between PEP 458’s sdist metadata and Fedora’s SRPM repos, where koji’s signature is saying “this is the source that was uploaded for building”.

I agree the assertions for wheels and RPMs aren’t exactly the same (PyPI makes the same “this is what was uploaded” assertion for wheels as it does for sdists, whereas koji makes the stronger assertion that “this RPM was built from this SRPM”)

1 Like
#120

Considering this question further, I realised there is a term that covers package signing that is handled by the distribution platform, while excluding publisher signing of individual packages: “signed repository metadata”.

If we tweaked the title and wording of PEP 458 to use that phrase, then I think @ubernostrum is right that it would reduce the chances of people getting confused and expecting to see end-to-end publisher package signing introduced right now, rather than that being considered as part of the follow on work in PEP 480 to address potential attacks on PyPI itself (rather than the attacks on mirrors and download channels that PEP 458 addresses)

2 Likes
#121

PR making that change: https://github.com/python/peps/pull/1308/files

The PEP also tidies up a few remaining places that hadn’t been reworded since the PEP 458/480 split between the core repository metadata signing capability (PEP 458) and the more secure, but also higher friction, delegation of package metadata signing authority to individual publisher keys (PEP 480).

1 Like
#122

Doesn’t Debian allow maintainers to upload binary packages that the maintainer built on their personal system?

1 Like
#123

Yes, but we are moving away from it. Already for the next release packages are not allowed into the Testing repository unless they were built on a buildd. There are reasons to allow maintainer built binaries, such a bootstrapping, but they can (and will be) rebuilt before making it into a release.

On a related note, Debian also publishes a maintainer keyring that can be used to verify developer signatures on all uploads (source or source + binary), so people can verify what’s in the archive is what they uploaded.

We’ve also been expanding coverage of verifying upstream signatures when they are provided, so developers can verify that the upstream code they have is what the upstream released.

All of that is on top of the signed packages file that is used automatically on end-user systems to verify that the correct (unmodified) package is being installed on that system.

3 Likes
#124

Thanks @dstufft for your decision, and thanks to everyone who commented here or in other fora. I’m not a PEP author and I appreciate all the work and improvements that have gone into this proposal since its start 6+ years ago.

There’s now a Zulip chat stream where the PEP authors and the implementor (@woodruffw) will be discussing implementation details, timeline, and so on.

3 Likes
#125

The PR updating the PEP title to be “Secure PyPI downloads with signed repository metadata” has been merged, and will change on https://www.python.org/dev/peps/pep-0458/ with the next content update (usually within the hour).

4 Likes
#126

For those following along, be aware when reading the PEP that it is still displayed with the incorrect image.

@lukpueh submitted a PR to address the underlying issue: https://github.com/python/pythondotorg/pull/1556

1 Like
#127

Sorry for the necrobump here, but I figure this thread is a good place to track errata as we continue the PEP 458 implementation on Warehouse.

In particular the following (not exhaustive) will need to be updated in the PEP:

  • The delegated bin (i.e. bin-n) metadata will not expire every 24 hours. Instead, it will expire every 7 days.

  • The Warehouse implementation will not be using any hash functions in the SHA-2 or SHA-3 families. Instead, it will use BLAKE2, since Warehouse already uses BLAKE2 as its hash function for uploaded files.

I’ll continue to keep this thread updated as more errata (and clarifications) occur.

5 Likes
#128

And to clarify the above: the BLAKE2 that Warehouse uses is BLAKE2b-256, which is the shorter digest form of the default BLAKE2b. That’s distinct from BLAKE2s, which is also 256 bits but uses a slightly different algorithm.

#129

@jku has some related work regarding pip and is seeking feedback: https://github.com/pypa/pip/issues/8585 and https://github.com/pypa/pip/pull/9041 .

1 Like
#130

What might make sense is an actual new section documenting the client download process as that’s currently not specified in the PEP-458… I’m not sure if an addition this large is in the spirit of “errata and corrections”.

Other smaller items (some of these might be covered by the above item already):

  • index files are currently only mentioned in this sentence: snapshot process SHOULD also include any new or updated hashes of simple index pages in the relevant bin-n metadata : I don’t think a SHOULD is enough here.
  • TUF metadata API endpoint location (or discovery method) should be defined
  • method of translating a distribution URL into a TUF target name should be defined (i.e. how does client download a distribution file once it has found the URL it wants in the index file)
  • method of translating package name to TUF target name should be defined (i.e. how does a client download the relevant index file when it has a name of package)
1 Like