TL;DR: PEP 458 implementation is ongoing and we’re working to make the changes easier to review. We would appreciate feedback or alternative suggestions on how to make review of these changes easier (the final two bullets of this post).
Hello from PEP 458 implementers! We wanted to provide an update on what’s happened since “PEP 458 – Secure PyPI downloads with signed repository metadata” was accepted and what we are planning next to help deliver this important PEP.
Over the past two years many folks have been working constantly to enable the integration in a way that delivers the full benefits of the PEP, while striving for minimal impact on PyPI operations and experience. That work includes:
- William designed the psf-tuf-runbook for implementing the security policies from the PEP for offline keys, including a fully-remote signing ceremony.
- William did an initial draft implementation for initialising a TUF repository in Warehouse (warehouse#7488).
- Jussi did an initial draft implementation for integrating TUF into pip (pip#9041) (and some follow-on design work which didn’t make it to PR’s).
- With significant inspiration from the warehouse and pip drafts, the python-tuf team set about improving python-tuf to be more suitable for PEP 458 (and PEP 480), leading to python-tuf 1.0. This release completely overhauled the python-tuf API to make it more ergonomic and provides a much slimmer and less opinionated client implementation for better integration into pip.
- Kairo ported William’s draft initialisation PR to the new python-tuf API and implemented extensive tests (warehouse#10870).
Feedback on the warehouse PRs has been positive, yet no formal review or acceptance of changes has occurred. We fully empathise with why they have not.
This is a complex series of changes which is difficult to review, requiring considerable time and effort. To that end, we have been trying to come up with proposals for how to ease the review. We have started two efforts:
Produce a design document: PEP 458 is ~39 A4 pages when printed and contains a mixture of TUF background, comprehensive problem statement, and architectural documentation. We are working to extract the technical details into a design document, update that based on existing work, and fill any gaps. We hope this will provide a concise document for reviewers to be able to determine whether a PR is implementing what is expected (this should give the implementers an easier time too).
- Aside: would it make sense for this document to be an appendix to the PEP?
Complexity reduction: the PR in warehouse#10870 implements an architecture that has two levels of abstraction over TUF. This increases lines of code and complexity, because the distinction between generic TUF repository operations and application specific logic is not as clear-cut as originally thought. We are working to consolidate all TUF management code into one class and initial signs are very promising (one file less and several hundred fewer lines of code).
Once the above two efforts are completed and we have new/updated PRs ready for review we have wondered whether the following might help progress the review/integration?
- Would it make sense to have a feature branch for the PEP 458 integration where PRs can be reviewed and merged in a more granular fashion without having to worry about whether merging them will break PyPI?
- Would it be beneficial to schedule PR review meetings so that the implementers can talk through more complex aspects of the code?
Thanks for reading and we look forward to continued collaboration on PEP 458 and future supply chain security enhancements!
Kairo Franciso de Araujo