TL;DR: PEP 458 implementation is ongoing and we’re working to make the changes easier to review. We would appreciate feedback or alternative suggestions on how to make review of these changes easier (the final two bullets of this post).
Over the past two years many folks have been working constantly to enable the integration in a way that delivers the full benefits of the PEP, while striving for minimal impact on PyPI operations and experience. That work includes:
William designed the psf-tuf-runbook for implementing the security policies from the PEP for offline keys, including a fully-remote signing ceremony.
William did an initial draft implementation for initialising a TUF repository in Warehouse (warehouse#7488).
Jussi did an initial draft implementation for integrating TUF into pip (pip#9041) (and some follow-on design work which didn’t make it to PR’s).
With significant inspiration from the warehouse and pip drafts, the python-tuf team set about improving python-tuf to be more suitable for PEP 458 (and PEP 480), leading to python-tuf 1.0. This release completely overhauled the python-tuf API to make it more ergonomic and provides a much slimmer and less opinionated client implementation for better integration into pip.
Kairo ported William’s draft initialisation PR to the new python-tuf API and implemented extensive tests (warehouse#10870).
Feedback on the warehouse PRs has been positive, yet no formal review or acceptance of changes has occurred. We fully empathise with why they have not.
This is a complex series of changes which is difficult to review, requiring considerable time and effort. To that end, we have been trying to come up with proposals for how to ease the review. We have started two efforts:
Produce a design document: PEP 458 is ~39 A4 pages when printed and contains a mixture of TUF background, comprehensive problem statement, and architectural documentation. We are working to extract the technical details into a design document, update that based on existing work, and fill any gaps. We hope this will provide a concise document for reviewers to be able to determine whether a PR is implementing what is expected (this should give the implementers an easier time too).
Aside: would it make sense for this document to be an appendix to the PEP?
Complexity reduction: the PR in warehouse#10870 implements an architecture that has two levels of abstraction over TUF. This increases lines of code and complexity, because the distinction between generic TUF repository operations and application specific logic is not as clear-cut as originally thought. We are working to consolidate all TUF management code into one class and initial signs are very promising (one file less and several hundred fewer lines of code).
Once the above two efforts are completed and we have new/updated PRs ready for review we have wondered whether the following might help progress the review/integration?
Would it make sense to have a feature branch for the PEP 458 integration where PRs can be reviewed and merged in a more granular fashion without having to worry about whether merging them will break PyPI?
Would it be beneficial to schedule PR review meetings so that the implementers can talk through more complex aspects of the code?
Thanks for reading and we look forward to continued collaboration on PEP 458 and future supply chain security enhancements!
Kairo Franciso de Araujo
Lukas Pühringer
Joshua Lock
From a PEP editor perspective, I would advise that instead of making it an appendix to the PEP itself, which would make the PEP even longer and harder to navigate, you could make it a separate reST file in the pep-0458 subdirectory and link it from the PEP using the :doc: or :ref:; with the new PEP infrastructure, that should work perfectly fine. That’s what I’ll be doing with several appendices and ancillary sections of my PEP 639, to greatly shorten its length to just the key material and make it more easily navigable.
You could also shorten the PEP a bit by moving from footnotes to regular links (either inline or referenced from the ref section), which would reduce the length of the PEP by a page or two without changing the normative content. You could also pull more of the non-core PEP material into linked ancillary documents, but as the PEP is already accepted, that would only be a last resort.
I don’t want to make any promises on a timeline here that you should hold me to strictly, but I’m planning to review the state of all of this, and either review the current PR or come up with concrete next steps that we’d like to see to move forward with this work.
I hope to do that this week sometime, but in either way I’ll try to get it done ASAP.
Thank you Donald! We will definitely appreciate your review and feedback.
Please don’t spend too much time with the PR until we’ve had chance to finish our effort to remove an abstraction layer and reduce the complexity/LOC, as mentioned in the PR discussion.
FYI: two of us are at EuroPython this week, so the efforts I outlined will be slow moving until after the conference.
TL;DR: PEP 458 and its implementation in Warehouse have been streamlined and described in a succinct design document. In order to further simplify Warehouse implementation, focus has shifted from a deep integration within Warehouse, to the development of an autonomous repository service (RSTUF), which can be interfaced by Warehouse. A rough ETA for this is the end of February.
—
Hello again from the PEP 458 implementers! In our initial post on this thread we have reported our ongoing integration efforts of TUF into Warehouse. These efforts have produced a succinct but comprehensive design document and streamlined the existing PR:
The design document provides clear implementation instructions that can be easily validated against PEP 458 and the TUF specification by people familiar with these resources, and also be used by adopters to validate the implementation correctness without in-depth background knowledge. Detailing the routines prescribed by the PEP allowed adopting a few major simplifications, such as requiring only one instead of two offline roles and re-using a single online key for all other roles.
Despite complexity reduction on multiple levels, a deep integration of PEP 458 into Warehouse remains a drastic change to the Warehouse code base, which poses a challenge for Warehouse maintainers, both in terms of initial review and in ongoing maintenance effort. This is why we have shifted our focus to the implementation of a stand-alone Repository Service for TUF (RSTUF).
RSTUF is a collection of services to deploy a scalable and distributed TUF repository, as described in, but not limited to, the PEP 458 setup. It consists of three main components:
A management CLI, which can be used to bootstrap TUFs root of trust in an offline key ceremony.
A REST API, which can be interfaced by a content repository such as PyPI, e.g. to register newly added packages.
A worker service, which listens on events, such as package additions, and maintains a consistent TUF metadata repository in a scalable way.
Our next steps will focus on a stable release of RSTUF (see roadmap) followed by a PR for Warehouse development environment (as current PR), which uses RSTUF to implement PEP 458. A rough ETA for this is the end of February.
Stay tuned!
Kairo Franciso de Araujo
Joshua Lock
Lukas Pühringer
