Prerequisites & vetoes -- improving packaging security

Packaging
1

TL;DR: Go to “What’s next” and if you want to veto those ideas, or say “but before that we need to do foo”, please speak up within the next week.

Background: see the roundup I just posted of current and upcoming funding for PyPI/packaging security via the Packaging WG.

PEP 458 (“Surviving a Compromise of PyPI”) and PEP 480 (“Surviving a Compromise of PyPI: The Maximum Security Model”) are both Deferred pending funding.

What’s next:

A coauthor of those deferred PEPs 458 and 480, @JustinCappos at New York University, wants to seek funding to better secure Python’s developer-to-user package pipeline. I’m doing some grantwriting (working on the proposal text) and I want to make sure that we’re only proposing to do stuff the community wants, and that we aren’t missing prerequisite tasks.

If you want to veto these ideas, or say “but before that we need to do foo”, please speak up within the next week. (We’re working on a proposal that’s due in a few weeks.)

  1. Finish overhauling the pip dependency resolver. (Justin was one of @pradyunsg’s GSoC mentors on this project a few years ago.)
  2. Secure developer-to-user package pipeline with cryptographically signed metadata: end-to-end signing. Once the Facebook-funded work on cryptographic package signing is in place, this proposal would involve adding support for generating and verifying these signatures through all the parts of the Python packaging and distribution toolchain – possibly using TUF and in-toto, possibly using some other framework.

This would be a multi-year project, with probably enough money to pay the equivalent of 1 full-time developer for a few years. If the proposal’s accepted we’ll find out around April 2020.

@dstufft and other experts: would this be okay? If so, leave a heart on this post; if not, please leave a comment saying why, preferably by 29 August.

1 Like
2

Sorry I didn’t see this sooner.

I’m checking with some of my colleagues first, because honestly I’m not sure, but we (Microsoft) are very keen to have a signing system that is able to work with our current processes.

In particular, having a way for wheels distributed on Windows (and macOS?) to be signed using commercially provided Authenticode keys would be ideal, as this would align with the processes used by all existing software vendors distributing EXEs, DLLs, MSIs, MSIXs, CABs, PS1s and other signed software.

My current fear is that a system that is tied directly to PyPI would prevent/exclude these approaches. Also, the current proposed constraint (in the linked PEP) that any signing solution be pure-Python is incredibly restrictive.

I would like to see the tool support be extensible to support private distribution sources (e.g. you need to install/configure another package before you get verification).

But, as I said, I’m talking with colleagues who know this area better than I do. If they come back and say that we’re happy to follow a TUF-like model where PyPI holds all the public keys (rather than following the trusted CA model) and RSA is not a strict requirement, I wil happily withdraw :slight_smile:

3

PSF has published a Request for Information seeking software developers to add these features to Warehouse:

  • Verifiable cryptographic signing of artifacts (PEP 458/TUF or similar)
  • Technical infrastructure and methods for automated detection of malicious package uploads

We’d like for potential contractors & other experts to keep discussion at the Q4 RFI Discourse category, especially on these questions:

Please comment by September 18th. That’s when the RFI ends.

Then, the Request for Proposals period will be September 23-October 16. Then we aim to start work in December. (Timeline details are in RFI.)

1 Like
4

Reminder: the Request for Proposals for these security improvements is now open and contractors who want to submit proposals need to do that by October 21st, 2 weeks from now.

5

G’day!

These are still my top two security problems I encounter almost every day that have been widely known issues for years.

  • don’t break peoples systems with sudo pip install being allowed, and defaulting to system packages.
  • don’t recommend people store plain text passwords for uploading packages.

Since python packaging is already inaccessible to most python programmers, I hope the “ease of use” factor of these “advanced security features” are addressed so that we don’t end up with an even harder system to use.

Is taking money from Facebook ethical according to members of the PSF?

6

We’re working on that, independently, from this funded project. Please see the latest posts on #4575 in pip’s issue tracker.

1 Like
7

Hi and thanks for sharing your thoughts!

You said that “Since python packaging is already inaccessible to most python programmers” - I’ll agree that there have been a lot of usability problems in the past and some still persist. I would like to systematically understand people’s current difficulties with packaging and systematically address them - that’s why I have been asking funders for money https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Improve_user_experience_of_packaging to do UX research on this topic (following on the UX research and design that Nicole Harris has been doing on Warehouse over the last few years).

I know a lot of Python programmers run into problems figuring out how to package their work. However, just a week ago I led a sprint night where I pointed about a dozen people (many of them novice Python programmers) to learn how to package for the first time by working through https://packaging.python.org/tutorials/packaging-projects/ and they were able to follow it without difficulty. I know the documentation and guidance at https://packaging.python.org/ doesn’t come up as high in search results as some older docs/blog posts/Stack Overflow answers, and doesn’t cover every circumstance and use case, but I do want to point out this counterexample.

You said: “I hope the ‘ease of use’ factor of these ‘advanced security features’ are addressed so that we don’t end up with an even harder system to use”. It sounds like you have specific ease of use concerns with the current toolchain - have you already filed them at
https://github.com/pypa/packaging-problems/issues ? I would appreciate if you could. And yes, as written in the RFP, the backend developer “will have support from an additional contractor focused on user interface and user experience design” who will be key in ensuring ease of use.

You ask an important ethical question. My short initial answer is: yes, in the case of this particular gift (and note that I personally do not use Facebook, as a user), but I need to take care of some family stuff and will respond on this point more later.

1 Like
8

You asked:

Is taking money from Facebook ethical according to members of the PSF?

(I am speaking here as an individual; I am not a Python Software Foundation staffer or board member, and am not speaking for the PSF.)

I presume you are asking this question because this particular work is being funded by a gift from Facebook. So there are a few parts to your question as I see it: is it ethical for the PSF to accept any money at all from Facebook, and is it ethical for the PSF to accept a gift from them to fund this particular work?

First, on whether the PSF should accept a donation from Facebook:

The PSF has a Sponsor Working Group that screens organizations that apply to sponsor the PSF (see Sponsor the PSF | Python Software Foundation ). And the PSF is already a recipient of Facebook money via sponsorship Python Software Foundation Sponsors | Python Software Foundation , and you can see in PSF Board Resolutions | Python Software Foundation that on 16 October 2018 that this WG approved Facebook as a sponsor:

RESOLVED, that the Python Software Foundation Sponsor Working Group approve the sponsor application from Facebook at the Principal Level.

Facebook has also funded the PSF by sponsoring PyCon; see PyCon US and About PyCon Sponsors | PyCon 2019 in Cleveland, Ohio and About PyCon Sponsors | PyCon 2018 in Cleveland, Ohio … and so on. They have sponsored every PyCon North America since 2012, as far as I can tell. And that’s just PyCon NA; I haven’t taken the time to check the other regional cons.

So I suggest that if you want to check with the PSF membership about overturning this decision by the Board and by the Sponsor WG, and about refusing future PyCon sponsorship by Facebook, it would make sense to ask that question in a different venue, such as a different discuss.python.org thread outside the Packaging category. Because if the PSF membership does not the PSF want to take any money at all from Facebook, that’s going to affect far more than this one project.

Next: is it ethical for the PSF to accept a gift from Facebook to fund this particular work?

I believe it is. This is work that the Python packaging maintainers already wanted to implement (and had been seeking funding for); we are not making new, previously unwanted architectural or feature choices at the funder’s behest. The work will be done in the open, publicly, and the process and the finished features will be equally accessible to people inside and outside Facebook. We aren’t adding any Facebook-specific integrations. And however/whether we credit Facebook as a sponsor on PyPI, I am confident that it’ll be unobstrusive and that we’re not going to add Facebook advertising to the PyPI user flow.

I hope this addresses your concerns.

As I said, I personally don’t use Facebook. And I’m very unhappy with many of the company’s choices. (Again, I’m speaking for myself as an individual here.) I know that the Packaging Working Group would welcome introductions to other companies, nonprofits, governments, academic institutions, and so on who will step up to fund specific projects within the list of fundable packaging improvements at Fundable Packaging Improvements - PSF Wiki .

4 Likes
9

Thanks for taking the time to make detailed responses.

Plain text password issue: https://github.com/pypa/packaging.python.org/issues/297

10

@steve.dower I’m not sure whether your concern has been addressed elsewhere? (I haven’t been keeping up to that extent; apologies!) I think TUF and PEP 458 concerns should now go in PEP 458: Surviving a Compromise of PyPI – heads-up @dstufft and @trishankatdatadog to reply, preferably in that thread.

2 Likes
11

It’s been defined as out of scope, so I’m going ahead with custom private solutions for our needs. Adding TUF on PyPI won’t impact any of my recommendations I provide for people interested in securing their setups.

1 Like