Which cryptographic signing approach?

PSF PyPI Q4 RFI
1

We want verifiable cryptographic signing of artifacts.

PEP 458 (“Surviving a Compromise of PyPI”) and PEP 480 (“Surviving a Compromise of PyPI: The Maximum Security Model”) were authored to propose implementation of The Update Framework on PyPI to allow end users to verify the integrity of packages downloaded from PyPI. Both PEPs were Deferred pending funding.

These PEPs offer different levels of security; which (if either) should we implement during this particular funded project? Which one has more appropriate operational efficacy? Should we use TUF or another approach?

Reminder: this RFI period ends September 18. Please comment by then.

2

We would be happy to help with the TUF aspects of this. We could help train anyone on TUF (as needed). We have a lot of documentation, resources, and multiple versions of TUF reference code that are used in different production systems. We can also provide developer resources to help review any changes or patches, if needed.

From our standpoint, we’re love to work with someone who knows the PyPI codebase well. TUF integration has not been difficult for our other adoptions in the past ( https://theupdateframework.github.io/adoptions.html ). The biggest thing is understanding how and where to add a few library calls to TUF in PyPI’s codebase.

3

I don’t want to start a debate about TUF and what it can or can’t do (I’ve already been involved/spectated enough of those over the years), but I do want to clarify what the intent of the project is.

The title says “cryptographic signing approach”, and Sumana’s one-liner says “we want verifiable cryptographic signing of artifacts”. A number of other links say “TUF-like”.

I would like to clarify whether the intent is for publishers to be able to prove the provenance of the contents of their uploads as having come from them, or if we merely want to confirm that the contents of the package has not changed since being uploaded to PyPI? (I say “merely” because a publisher signed package has verifiably not changed since before being uploaded, assuming you trust the publisher.)

It would seem there are a number of simpler options (particularly from the Warehouse and wheel metadata POV) if publisher-provided signing is the goal, while maintaining accessible-ness if PyPI were to offer publishers free certificates that are trusted only by pip (which would essentially be what TUF offers anyway). At the very least, I see (and have) a need for verification tools that work without pypi.org being involved at all, and in highly secured environments where “just deploy X” may not be possible.

So at a high level, I’d like to see an approach like this:

  • pip gains ability to verify artifacts pre-extract through an extensible model (i.e. users can configure the use of a specific extension to do this)
  • the default extension for verification uses pure-Python only, verifies the RECORD file in wheels (as vaguely described in PEP 427), only checks against PyPI-signed signatures, and probably only warns by default when a signature is missing/unverifiable
  • PyPI offers publishers a downloadable certificate tied to their login name
  • wheel learns how to sign RECORD files in a way that pip’s default verifier can verify using certificates as provided by PyPI

All of these will have to exist regardless, so the fundamental difference from TUF seems to be a much simpler implementation on PyPI itself, with more responsibility required from publishers.

I deliberately did not specify the type of certificates above, because I honestly don’t care that much what default is used. What I care about is that this allows me to circumvent the signing process in a controlled environment and add these additional steps myself:

  • implement, deploy and configure a custom verification extension to pip on my own systems (or publicly)
  • implement and use a custom wheel signing tool that generates signatures verifiable by my extension

The reason I would want this is to be able to do things like using never-before-shared certificates or using the operating system’s CA collection to establish trust rather than pip itself. Perhaps I want to use a non-portable API for verification, or use a private system for timestamps or revocation.

By implementing this as tooling extensions rather than PyPI extensions, many more scenarios are made available without users having to resort to forking common tools.

But to go back to my original clarification, if the project is only to establish verification from PyPI ingestion to download, then a solution that is entirely internal to PyPI (involving no user effort) is the way to go.

2 Likes
4

@steve.dower The current scope is to ensure that artifacts have not changed once they are uploaded to PyPI, though this RFI is a chance for us to explore other options.

The funding is not sufficient to cover implementing any additional functionality in tools like twine and pip or libraries like setuptools, so whatever direction is taken should be implementable in PyPI initially and made available for client tools to adopt.

What you describe above is probably an ideal scenario, with publishers taking responsibility up front when uploading new releases. As a first step something internal to PyPI creates the biggest impact (verifying that a dependency hasn’t changed due to an attack on PyPI itself or any intermediate infrastructure) for the multitude of projects that do not have maintainers that will do any signing before upload, which is why it has been initially targeted.

1 Like
5

IMHO, the minimum security model proposed in PEP 458 should be doable within 3-6 months of full-time work by a small team of dedicated developers. This includes 1 month of reading about and experimenting with TUF. IIRC, Docker took around the same time to implement Docker Content Trust (server) and Notary (client), their version of TUF. The Datadog TUF and in-toto integration was more complicated because it offers much stronger security guarantees, and it took around 1-2x as long.

There is no signing from developers here, so there is nothing to require on their part, because PyPI would basically sign all packages. pip would be lightly modified to transparently verify packages using TUF. The download code should be no more complicated than this Datadog-specific downloader.

IMO, the most complicated pieces would be in decreasing order of difficulty:

  1. Designing the initialization and rotation of the root TUF keys. This is a time-consuming, offline ceremony that involves careful generation and backing up of keys in a secure environment. An important consideration is to make sure that it can survive changes of personnel. The good news is that large parts of it can be scripted / automated, even though it is run manually by humans. It should be done once every few years (e.g., 1 year), or whenever there is a key compromise.
  2. Writing the PyPI code to sign packages on demand using TUF. This code should include garbage collection of expired metadata.

The code above, which I have a lot of experience with, needs to be written at most once. The TUF reference implementation is already in Python, so a lot of code is already available. I expect that we would run into small bugs involving corner cases. For example, I expect that we will find unexpected bugs when writing many different versions of metadata for a large enough number of packages or at a fast enough rate, or run into unexpected network issues when downloading packages in production environments we had not anticipated before. So, this will needs lots of testing, and some fixing, that will naturally come from beta-testing. After this, it should be in largely maintenance mode, until such time as PEP 480 is ready to be implemented.

I am happy to discuss more details. This timeline should not be considered a strict one: I am deliberately using a conservative estimate to make sure that we can account for unexpected hiccups. I know TUF looks intimidating, but it really isn’t once you have set up the one-time work, and it offers strong security guarantees that other systems simply do not provide (e.g., transparent key rotation for uncompromised users after PyPI itself has been compromised but recovered from). One thing we should do is to introduce security gradually in pieces, not all at once. Again, the TUF team and I are happy to consult and volunteer as much as we can.

6

Thanks for the feedback Trishank. One thing to mention about the timeline here is that Docker also implemented TUF in Go during the 3ish months it took them (along with adding Docker Content Trust). I’m optimistic that reusing the Python TUF implementation (with help from us) would be much less time consuming.

Trishank (hopefully) already found most of the bugs in the Python code when he helped to integrate it and in-toto into Datadog…

1 Like
7

We tried to post our proposed approach, however new users are apparently limited to posting 2 links at a time.
:crazy_face:

We have posted a gist of our eigentrust proposal and an example of a trust repository that we are currently using internally.

8

Does anyone have questions about the TUF proposal, discussed in PEPs 458 and 480?

9

I’m very excited to see this RFI. I stumbled across the TUF PEPs a couple of months back whilst investigating areas to contribute and would very much like to contribute to the effort to realise them.

I’m in favour of using TUF with PyPI, as opposed to another approach. TUF is a peer-reviewed and widely deployed system. Alternatives which have been suggested (based on a non-exhaustive search of the distutils-sig mailing list archive) seem to offer minimal improvements to the current security, awkward usability, or both. Furthermore, designing and implementing a new mechanism seems infeasible in the time frame suggested by the RFI.

Consensus in the past seems to have been in favour of implementing PEP 458 first and using the experience of doing so to refine PEP 480 (around which there seemed to be several open questions, not least of all around the impact on user/developer experience). This seems like the most prudent course of action - implementing PEP 458 offers concrete security improvement, develops familiarity with TUF and the operational changes in PyPI administration it requires, integrates the required dependencies into Warehouse, pip, etc and will help uncover any potential issues in the reference implementation.

Based on Trishank’s comment, it seems that the RFI proposal’s estimated time/budget may not be enough to complete the implementation, particularly if the PEP needs to be refined/edited and reviewed first. With that in mind I would like reiterate my above interest in contributing to this effort. Whether that’s joining a team established through the RFI to work on cryptographic signing, working on any outstanding issues and tool integration after the work under this RFI is complete or perhaps even allowing more of the funding available to this RFI to be focused on the malicious package detection by having myself (and potentially 1 other engineer in my team) work on the cryptographic signing aspect of this RFI we are committed to contributing however we can.

Having re-read through the PEPs I’m wondering if PEP 458 will need re-working/refining ahead of the implementation of this project? It seems like there was some work done to refine it after the last round of feedback, but I haven’t found evidence that the edits were completed/reviewed.

One question on the PEP specifically is that Ed25519 is suggested as the signature scheme because it has a pure Python implementation however, the module hasn’t seen active development since 2013 and comes with warnings that “This code is not safe for use with secret data” and is susceptible to side-channel attacks. Is this module/signature scheme still the right choice? If not, which module and scheme are appropriate for Warehouse? PyNaCl? If so, is there a more appropriate signature scheme to use (perhaps not, the small key sizes of Ed25519 seem valuable when it comes to future work such as PEP 480)?

1 Like
10

One issue is that quite a lot of the external links in PEP 458 are dead or outdated: [2], [13], [22], [30], [16], [40]

1 Like
11

Thanks for pointing this out. Here’s a list of updated and in some cases more permanent references:

[2] Survivable Key Compromise in Software Update Systems

[5] Attacks on software repositories
The original link is still good, but the authors of TUF keep a more up-to-date and broader collection at github.com/in-toto/supply-chain-compromises (see “Publishing Infrastructure” type compromises).

[13] A look in the mirror: attacks on package managers

[16] TUF specification

[22] NIST Recommendation for Key Management, Part 1, Revision 4
Not sure which revision the original link pointed to. This is the latest now.

[24] This used to point to pycrypto which is not used anymore in the TUF reference implementation. Instead it uses cryptography and PyNaCl, both optionally, and ed25519 for a minimal pure Python installation. On a side note, the TUF team is also working on support for OpenPGP with gnupg (#174), HSM signing with PyKCS11 (#170), and SPHINCS + with PySPX (#169).

[27] TUF updater client (at latest stable release, i.e. v0.11.1)

[30] TUF metadata documentation (v0.11.1)

[31] TUF repository tool documentation (v0.11.1)

[33] TUF “lazy bin walk”
The original link points to a closed GitHub issue, which points to a deprecated google document, which points back to PEP 458. I changed [33] above to point to the corresponding part in the TUF documentation (v0.11.1).

[40] This used to point to minilock, proposing an easy-to-use key management solution for developers (relevant for PEP480). I suppose today’s suggestion would be YubiKey.

1 Like
12

With reference to the allotted time and effort, NYU would be happy to contribute developer time from our side (at no cost) to help the winning proposal integrate TUF into Warehouse.

@lukpueh, would you like to chime in on what assistance you can provide to the effort?

13

Joshua, thanks very much for your commitment, I think it will go a long way. Provided that everyone has familiarized themselves with TUF and PEP 458, I think if we all hunker down and spend 1-2 weeks reviewing and editing the PEP so that a straightforward implementation is much clearer. This might significantly shave down implementation time.

As for ed25519, we can easily use the mature pynacl library.

14

I will also help here however much I can.

Justin, I think another thing we should do right now is to update PEP 458 to:

  1. Fix broken links.
  2. Update any outdated assumptions.
  3. Discuss offline and online key ceremony / management much more explicitly.
  4. Discuss the easiest implementation in terms of code.
15

I’d be happy to help wherever I can, be it as an adviser and implementer of adjustments on the TUF-side (if needed) , but also as actually working on the integration with warehouse, or both.

I agree with Trishank that updating PEP458 and creating an implementation plan would be a good way to start right away. This might also help interested developers to make a bid for the job.

I have noticed that PEP 458 on python.org does not seem to match any of the revisions of PEP 458 on GitHub. Does anyone know, which document should be used as base to propose patches?

16

I’m not entirely opposed to TUF and can see the benefits to an incremental approach, but without end-to-end signing it looks like there is significant work for some modest initial benefits.

In its favor, I see a few benefits to Non-E2E TUF, first the ability to cryptographically validate packages have not been tampered with in transit from PyPI to disk. It prevents tampering by CDNs and mirrors and it looks like it can enable a recovery from a future compromise. As long as root.json is bundled with the OS distribution of pip, it looks like TUF removes the need for certificate pinning, otherwise I’m not sure how root.json gets initialized. Finally, while TUF has some complexity, most of it looks like it can be hidden from developers and end-users.

On the other hand, as far as I can tell Non-E2E TUF does not help if PyPI has been compromised before the TUF deployment. It does not help if the person generating the timestamp, snapshot and bin signing keys is compromised and it does not prevent PyPI from being legally compelled to modify packages, though it may help recover from some of these issues if they are discovered. It is unclear to me if Non-E2E TUF can prevent tampering with new uploads while an upload server is actively compromised.

Beyond TUF itself, in-toto was mentioned as part of the goal state and after reviewing the spec, I have a few concerns.

First, I can see the benefit of in-toto for audits and for scripts that are not compiled, but I’m having trouble seeing how it secures the build process from a compromise without some sort of byzantine CI. Further it is unclear how the root.layout would specify a set of possible valid signing keys that could be used by multiple CI build servers. Edit: Although the example refers to the test step, it looks like this is handled with the threshold field being set for builds.

While perhaps it works well for enterprises, the packaging trust delegation in root.layout flows backwards from common opensource practices where OS packagers trust developers, not the other way around. Would developers need to add the PyPI, Conda, Debian, RedHat, etc public signing keys into the package root.layout? How would key updates for this be handled? or is packaging trust delegation just for running python ./setup.py bdist_wheel on the CI server?

I’m unclear on the use of git tag -s in the fetch-upstream example. Is it supposed to be git tag -v?

Tooling can help, and maybe some of the files needed for in-toto can be generated automatically, but creating a root.layout seems significant for non-enterprise developers wanting to upload packages to PyPI, much less debugging.

It seems reasonable that the root.json can establish the initial set of trusted keys, it is unclear how TUF or in-toto would establish that a developers public key is associated with a package. Would PyPI provide this validation? How would developer key rotation be handled?

While I understand the enterprise use case for in-toto, if the person who writes, builds and uploads the package to PyPI is all the same person in most use cases, what is the benefit of in-toto over something like minisign?

1 Like
17

Crypto that’s not in the standard library tends to be difficult to use in places like US government institutions, which have regulations around using cryptography.

1 Like
18

What alternatives are available in the standard library?

19

Thanks for raising these concerns. Let me try to reply inline…

[…] root.json is bundled with the OS distribution of pip, it looks like TUF removes the need for certificate pinning, otherwise I’m not sure how root.json gets initialized.

Obtaining the inital root.json is not in the scope of the TUF spec. But bundling with the OS distribution of pip, as you say, is a reasonable approach and recommended in PEP458 (see “How to Establish Initial Trust in the PyPI Root Keys”).

Finally, while TUF has some complexity, most of it looks like it can be hidden from developers and end-users.

This is the goal. I am happy to help reducing complexity on the TUF side.

On the other hand, as far as I can tell Non-E2E TUF does not help if PyPI has been compromised before the TUF deployment. It does not help if the person generating the timestamp, snapshot and bin signing keys is compromised and it does not prevent PyPI from being legally compelled to modify packages, though it may help recover from some of these issues if they are discovered. It is unclear to me if Non-E2E TUF can prevent tampering with new uploads while an upload server is actively compromised.

This is correct. In conversations with Ernest at the PyCon '19 sprints we talked about ways of highlighting back-signed packages (see item “Back-sign existing packages”).

Beyond TUF itself, in-toto was mentioned as part of the goal state and after reviewing the spec, I have a few concerns.

I think we are getting ahead of ourselves by discussing the details of in-toto, but I’ll try to reply…

First, I can see the benefit of in-toto for audits and for scripts that are not compiled, but I’m having trouble seeing how it secures the build process from a compromise without some sort of byzantine CI. Further it is unclear how the root.layout would specify a set of possible valid signing keys that could be used by multiple CI build servers.

The answer might be a combination of thresholds for in-toto build attestations and reproducible builds. The in-toto team is working closely with the reproducible builds community to explore this avenue (see e.g. community reports for Aug, Jul, Jun).

While perhaps it works well for enterprises, the packaging trust delegation in root.layout flows backwards from common opensource practices where OS packagers trust developers […].

This is exactly what in-toto sublayouts are designed to do. A downstream OS packager does not have to be concerned with all the steps that happened upstream, or any of the upstream developer keys, instead downstream packager considers upstream as a black box (e.g. called fetch-upstream) and trusts the upstream project owner, to make sure that said black box contains a in-toto verifiable suppy chain (of which the first step might be tagging a signed upstream release hence the git tag -s in the example).

It seems reasonable that the root.json can establish the initial set of trusted keys, it is unclear how TUF or in-toto would establish that a developers public key is associated with a package. Would PyPI provide this validation? How would developer key rotation be handled?

How or why trust is established is not in the scope of TUF or in-toto. Both protocols “only” provide mechanisms, to define, communicate and verify trust relationships. PEP 480 recommends a trust on first upload scheme (see End-to-End Signing).

While I understand the enterprise use case for in-toto, if the person who writes, builds and uploads the package to PyPI is all the same person in most use cases, what is the benefit of in-toto over something like minisign ?

Only a mechanism like in-toto allows us to verify that a package was meant to be written built and uploaded by a single person, and that that person indeed did all those things.

1 Like
20

None, the Python standard library does not expose asymmetric signature verification facilities. There is only HMAC, which is message authentication with symmetric keys.

As maintainer of hashlib, hmac, and ssl module I’m open to add a high level signature verification API, but really just signature verification. Key generation and signing should be delegated to PyCA cryptography. The implementation must also be based on OpenSSL API. I don’t want to have to deal with maintenance and legal issues caused by another cryptographic library.

I also would prefer algorithms that are standardized by IETF and approved by NIST. NIST SP 800-56A is probably too limited, but the current draft of SP 800-186 includes fancy new algorithms like Curve25519 and EdDSA. In my humble opinion RFC 8032’s Ed25519 would be a good choice here. The algorithm also supports context specific signature and prehashing (although PH has some caveats).

4 Likes