Prerequisites & vetoes -- improving packaging security

TL;DR: Go to “What’s next” and if you want to veto those ideas, or say “but before that we need to do foo”, please speak up within the next week.

Background: see the roundup I just posted of current and upcoming funding for PyPI/packaging security via the Packaging WG.

PEP 458 (“Surviving a Compromise of PyPI”) and PEP 480 (“Surviving a Compromise of PyPI: The Maximum Security Model”) are both Deferred pending funding.

What’s next:

A coauthor of those deferred PEPs 458 and 480, @JustinCappos at New York University, wants to seek funding to better secure Python’s developer-to-user package pipeline. I’m doing some grantwriting (working on the proposal text) and I want to make sure that we’re only proposing to do stuff the community wants, and that we aren’t missing prerequisite tasks.

If you want to veto these ideas, or say “but before that we need to do foo”, please speak up within the next week. (We’re working on a proposal that’s due in a few weeks.)

  1. Finish overhauling the pip dependency resolver. (Justin was one of @pradyunsg’s GSoC mentors on this project a few years ago.)
  2. Secure developer-to-user package pipeline with cryptographically signed metadata: end-to-end signing. Once the Facebook-funded work on cryptographic package signing is in place, this proposal would involve adding support for generating and verifying these signatures through all the parts of the Python packaging and distribution toolchain – possibly using TUF and in-toto, possibly using some other framework.

This would be a multi-year project, with probably enough money to pay the equivalent of 1 full-time developer for a few years. If the proposal’s accepted we’ll find out around April 2020.

@dstufft and other experts: would this be okay? If so, leave a heart on this post; if not, please leave a comment saying why, preferably by 29 August.

1 Like

Sorry I didn’t see this sooner.

I’m checking with some of my colleagues first, because honestly I’m not sure, but we (Microsoft) are very keen to have a signing system that is able to work with our current processes.

In particular, having a way for wheels distributed on Windows (and macOS?) to be signed using commercially provided Authenticode keys would be ideal, as this would align with the processes used by all existing software vendors distributing EXEs, DLLs, MSIs, MSIXs, CABs, PS1s and other signed software.

My current fear is that a system that is tied directly to PyPI would prevent/exclude these approaches. Also, the current proposed constraint (in the linked PEP) that any signing solution be pure-Python is incredibly restrictive.

I would like to see the tool support be extensible to support private distribution sources (e.g. you need to install/configure another package before you get verification).

But, as I said, I’m talking with colleagues who know this area better than I do. If they come back and say that we’re happy to follow a TUF-like model where PyPI holds all the public keys (rather than following the trusted CA model) and RSA is not a strict requirement, I wil happily withdraw :slight_smile:

PSF has published a Request for Information seeking software developers to add these features to Warehouse:

  • Verifiable cryptographic signing of artifacts (PEP 458/TUF or similar)
  • Technical infrastructure and methods for automated detection of malicious package uploads

We’d like for potential contractors & other experts to keep discussion at the Q4 RFI Discourse category, especially on these questions:

Please comment by September 18th. That’s when the RFI ends.

Then, the Request for Proposals period will be September 23-October 16. Then we aim to start work in December. (Timeline details are in RFI.)

1 Like