PEP 458: Secure PyPI downloads with package signing

PEPs
122

Doesn’t Debian allow maintainers to upload binary packages that the maintainer built on their personal system?

1 Like
123

Yes, but we are moving away from it. Already for the next release packages are not allowed into the Testing repository unless they were built on a buildd. There are reasons to allow maintainer built binaries, such a bootstrapping, but they can (and will be) rebuilt before making it into a release.

On a related note, Debian also publishes a maintainer keyring that can be used to verify developer signatures on all uploads (source or source + binary), so people can verify what’s in the archive is what they uploaded.

We’ve also been expanding coverage of verifying upstream signatures when they are provided, so developers can verify that the upstream code they have is what the upstream released.

All of that is on top of the signed packages file that is used automatically on end-user systems to verify that the correct (unmodified) package is being installed on that system.

3 Likes
124

Thanks @dstufft for your decision, and thanks to everyone who commented here or in other fora. I’m not a PEP author and I appreciate all the work and improvements that have gone into this proposal since its start 6+ years ago.

There’s now a Zulip chat stream where the PEP authors and the implementor (@woodruffw) will be discussing implementation details, timeline, and so on.

4 Likes
125

The PR updating the PEP title to be “Secure PyPI downloads with signed repository metadata” has been merged, and will change on https://www.python.org/dev/peps/pep-0458/ with the next content update (usually within the hour).

4 Likes
126

For those following along, be aware when reading the PEP that it is still displayed with the incorrect image.

@lukpueh submitted a PR to address the underlying issue: https://github.com/python/pythondotorg/pull/1556

1 Like
127

Sorry for the necrobump here, but I figure this thread is a good place to track errata as we continue the PEP 458 implementation on Warehouse.

In particular the following (not exhaustive) will need to be updated in the PEP:

  • The delegated bin (i.e. bin-n) metadata will not expire every 24 hours. Instead, it will expire every 7 days.

  • The Warehouse implementation will not be using any hash functions in the SHA-2 or SHA-3 families. Instead, it will use BLAKE2, since Warehouse already uses BLAKE2 as its hash function for uploaded files.

I’ll continue to keep this thread updated as more errata (and clarifications) occur.

5 Likes
128

And to clarify the above: the BLAKE2 that Warehouse uses is BLAKE2b-256, which is the shorter digest form of the default BLAKE2b. That’s distinct from BLAKE2s, which is also 256 bits but uses a slightly different algorithm.

129

@jku has some related work regarding pip and is seeking feedback: https://github.com/pypa/pip/issues/8585 and https://github.com/pypa/pip/pull/9041 .

1 Like
130

What might make sense is an actual new section documenting the client download process as that’s currently not specified in the PEP-458… I’m not sure if an addition this large is in the spirit of “errata and corrections”.

Other smaller items (some of these might be covered by the above item already):

  • index files are currently only mentioned in this sentence: snapshot process SHOULD also include any new or updated hashes of simple index pages in the relevant bin-n metadata : I don’t think a SHOULD is enough here.
  • TUF metadata API endpoint location (or discovery method) should be defined
  • method of translating a distribution URL into a TUF target name should be defined (i.e. how does client download a distribution file once it has found the URL it wants in the index file)
  • method of translating package name to TUF target name should be defined (i.e. how does a client download the relevant index file when it has a name of package)
1 Like
131

This is maybe a minor thing, but as I was rereading PEP 458 this week, I noticed that the “Integrating PyPI with TUF” section was a little confusing – it refers to the client-side work as “first” and the PyPI-side work as “second” but that’s the opposite of what we’re doing. I’d like to suggest this rewording. @dustin @dstufft @JustinCappos @joshuagl @jku @mnm678 and other interested folks: is this clarification ok? If so then I’ll submit it as a PR to the official PEPs repo.

1 Like
132

Looks good to me! Others?

133

I think this makes sense, I had the same impression when I read that section.

134

Thanks for authoring the change, it looks good to me.

135

Thanks all. Submitted PR.

136

Here’s a list of PEP 458 amendment candidates, which surfaced while working on the PEP 458 design document:

  • The ‘bins’ role is no longer needed. ‘targets’ can delegate directly to ‘bin-N’ roles (details).

  • A single shared online key is used to sign all non-root metadata, most notably ‘targets’ too (details).
    (This also means that there are no separate online keys for different roles, and no signature thresholds for non-root roles, and no one-time keys that may be discarded after signing, as originally suggested in the PEP.)

  • ‘bin-N’ can expire as rarely as targets (once per year). It is enough if ‘timestamp’ makes sure that the repo metadata is fresh. ‘bin-N’ expiry cannot provide an expiration fallback for the case, where ‘timestamp’ is compromised, because they both use the same online key.

  • It is not a hard requirement to serve target files with the sha256 content hash digest in their filename, it is also okay to serve them with the blake2b-256 digest in the filepath, which is the current practice.

  • The PEP should clarify that, although ‘timestamp’ metadata must be published without version identifier in its name, a new version of ‘timestamp’ cannot just override a previous version. It needs to remain accessible to remove obsolete metadata until after the removal.

  • The PEP should clarify for how long index files and distribution files need to be preserved also wrt PEP 503/691.

  • Minor nit: in “Managing offline keys” section, the PEP says:

    […] the root keys MAY not be generated during the offline key ceremony […]

    but should say:

    […] the root signatures MAY not be generated during the offline key ceremony […]

4 Likes