New packaging security funding & NYU

Packaging
1

Good news!

New York University – specifically Professor Justin Cappos – and I have successfully asked the US National Science Foundation for a grant to improve Python packaging security. The NSF is awarding NYU $800,000 over two years – from mid-2021 to mid-2023 – to further improve the pip dependency resolver and to integrate The Update Framework further into the packaging toolchain.

https://nsf.gov/awardsearch/showAward?AWD_ID=2054692&HistoricalAwards=false

What we’re planning to do

NYU researchers and developers will

  • Further assess and improve pip’s dependency resolver, following up on the work done in 2020 and making ResolveLib more reusable by other tools in the packaging ecology
  • Secure the PyPI-to-user pipeline by integrating TUF support for signed packages throughout PyPI’s clients (we’re targeting conda, pip, and bandersnatch initially)

And I’ll be part of this work, paid to work on this part-time, doing some outreach, coordination, project management, and similar.

This is a plan and subject to change.

What this means in the short term

More of the same, in a lot of ways. You already see Justin’s Secure Systems Lab doing stuff like PEP 458 and helping with the TUF rollout on PyPI. And: Once the funding from CZI and MOSS ran out I started spending way less time on packaging-related stuff, but I’ll return to higher levels of active attention and participation once the NSF-funded work kicks off. With NYU folks, I’ll be helping nudge PEP 480 along so we can come to resolution on it.

I will of course keep folks apprised of progress through this and similar public channels. Since this wasn’t funded through the Packaging Working Group of the PSF, I won’t be indexing updates on that wiki page but I’ll find some other public place to archive them.

Why NYU & NSF?

Justin and other folks in his lab have been pushing forward on Python packaging security for several years, including authoring PEP 458 way back in 2013. He’s used to applying for NSF grants and it was great to work together with him on this and learn the ropes a bit. NYU will of course collaborate with the PyPA, the PSF, and other stakeholders as we move forward.

The National Science Foundation is an arm of the United States government that funds advances in science and engineering – see their About page for more. This funding is for a “Transition to Practice” project in the NSF’s “Secure & Trustworthy Cyberspace” program.

The future is bright for the PSF to participate more directly as a co-proposer on these kinds of proposals in the future, and Justin is working to help with that!

Thanks

Thanks to the NSF for the award, to Justin for leading this proposal and NYU for supporting it, and to y’all for supporting improvements in Python packaging security!

Please reply with any questions.

Best,

Sumana

(some relevant threads: PEP 458: Secure PyPI downloads with package signing , Prerequisites & vetoes -- improving packaging security , RFC: improving pip security with package signing (PEP-458) , Current/upcoming funding for PyPI/packaging security (roundup FYI) )

28 Likes
2

ron-paul

1 Like
3

Amazing, great news!

1 Like
4

Very cool!

1 Like
5

Excellent news! Look forward to the collaboration.

1 Like
6

Congratulations and huzzah!

7

Hey @sumanah, I have a question. Will this work include extracting the pip dependency resolver logic to resolvelib, so that other tools can re-use it, or just improving resolvelib in general? Thanks :blush:

8

Can’t answer about the scope, but if pip’s resolver is extracted, it won’t become a part of resolvelib, but a separate library (that depends on resolvelib and a lot of other stuff). pip’s resolution logic is not useful for a lot of resolvelib users.

2 Likes
9

Right, it doesn’t make much difference anyway, as long as we can re-use this logic for other projects.

10

Great news, well done you and Justin, and thanks to the NSF for the funding.

I’m not clear why the funding source would affect the relevance for putting updates about packaging on the packaging WG page: that page is a volunteer page - its a place for people spending time on packaging to collaborate, and I think your work certainly fits; @dstufft when working full time at HP, with a directive there to work on pypi and packaging was still generating updates to that page as far as I can tell - so there is precedent.