PyPI security work: multifactor auth progress & help needed

More scope creep: can we just upload zip files instead of two different file types for .tar.gz and .zip via .whl? :wink: (I know this isn’t going to happen, but I can dream …).

2 Likes

The code already pokes inside zip/whl/egg files as you’ve noticed (zip contents validation). PR created, but I do agree with Brett, long term we should pick a single archive format and only support that on PyPI. An interim period where we auto-convert from the other formats to the desired one would make sense.

If we care about storage and network bandwidth costs zip and .tar.gz are both less than ideal. But a single format is a larger conversation. #dream

There were some long threads about file formats on PyPI back in 2016:

That eventually led to PEP 527. The biggest change in PEP 527 was to slim down PyPI’s support matrix from a dozen+ file formats to just 3: wheel, .tar.gz sdist, and .zip sdist. There was a lot of debate about whether to reduce that further to just one format of sdist, and if so, whether it should be .tar.gz or .zip. @dstufft’s original PEP draft made .tar.gz the only support format, IIRC based on it being by far the most common in actual usage, but it looks like the final draft relaxed that.

Then in PEP 517, we needed to specify a generic interface for producing sdists, and we made it use .tar.gz, basically because that’s what @dstufft liked better: https://www.python.org/dev/peps/pep-0517/#build-sdist

I think our options going forward are:

  • Continue to support both .zip and .tar.gz sdists on PyPI (status quo).
  • Drop support for .zip sdists. This would be pretty simple: we’d stop allowing .zip uploads, and that’s basically it. We’d be committed to handling both .zip and .tar.gz everywhere forever, for wheels and sdists respectively. This obviously isn’t a prohibitive cost (we do it now), but is some cost.
  • Drop support for .tar.gz sdists. This would be more involved: we’d have to first update PEP 517, and change setuptools to default to generating .zip sdists. (And ideally make the same change to other build backends, like distutils and flit.) And after that was deployed for a bit and people adapted to .zip sdists being common, then we could stop allowing .tar.gz uploads. The upside would be that .zip is somewhat easier to work with programmatically (since it supports random access), and that it matches wheels. And the downside of course is the transition costs.

(It’s an interesting side-effect of using discourse, that suddenly these conversations are drawing in folks who didn’t follow this history!)

2 Likes

I made a PR for this. I don’t have anything beyond normal-peon access to the warehouse repository so someone else will have to assign/review/merge if desired. This was my first time touching the warehouse codebase. “such docker. oh my.”

Thank you @gpshead for your PR!

Hi, Python packaging colleagues!

We continue to work towards our first goal: support for two-factor authentication on PyPI via TOTP and U2F/FIDO. William and Nicole are continuing their development and design work as I mentioned in the last update, with additional work by Mark Mossberg at Trail of Bits, plus Ernest, Dustin, and Donald advising and reviewing.

We are working out our rollout plans for multifactor auth, and so we don’t yet have an estimate for when we’ll deliver that and when we’ll start the API keys or audit trail work. But the existing work-in-progress PR for MFA is ready for you to try out and play with now, and we’ll have more for you to try out next month at the PyCon sprints.

Want to help?

Thank you @gpshead for your PR to validate whether uploaded packages ending in tar.gz are actually tarballs!

And please speak up in this topic if you’re planning to come to sprints at PyCon North America, May 6th-9th, so we can plan tasks.

We’ll send another progress report around mid-month. That’s also when PSF aims to announce another Request For Information for Warehouse security improvements: “highly requested security features in PyPI such as cryptographic signing and verification of files uploaded and installed from the index” (possibly using TUF).

Thanks to the Open Technology Fund for funding this work!

Sumana Harihareswara, Warehouse project manager

4 Likes

Installation and setup was a breeze. I’ll be there for four days of sprints and would love to contribute. I am the architect and former security lead on a Python based MFA API. I hope I’ll be able to lend some SME help along with knocking out tickets. :monkey_face:

1 Like

How does the plan to implement TUF (and maybe webauthn) dovetail with this work?

What issues need more attention from volunteers with which skills?

1 Like

What’s the GH Issue # for the API keys work; is the system being designed so that I can create a per-package key (so that I’m not delegating all privileges to my CI builds and CI build systems)?

We’re excited to help with the encryption support aspect whatever is decided. Just point us at the best way to dive in. I can’t personally make it but I can check to see if someone from our team can attend PyCon if being there in person would be a major help…

2 Likes

Yeap, same here, ready to help with the PyPI + TUF effort, just let me know

Cc @pradyunsg

1 Like

Lukas from the TUF team at NYU here. I will be at the PyCon sprints next week, and I’m looking forward to discuss and work on PyPI + TUF integration.

2 Likes

Summary: Work continues on Milestone 1, Security Feature Development, and specifically on the Multi-Factor Authentication task. TOTP-based 2FA is about to roll out for everyone, and we’re working on WebAuthN (e.g., Yubikeys).

In April and in the first week of May, the team finalized the backend and user experience for 2FA and planned and started user tests. On May 2nd, we began advertising the test to our users and requesting feedback. That test has already found issues which we have fixed or prioritized to fix later.

We planned for the test to go till May 20th. We decided to require email verification before activating 2FA, and that’s underway. Once we finish that, we’ll be turning on the optional 2FA login feature for current and future accounts on pypi.org (it’s already on for most existing accounts on Test PyPI, and we’ll turn it on for all current and future accounts there, too.) There are some UI issues that we should fix in the medium term, but I’ve decided it’s ok to roll out the feature and end the beta before fixing those.

Thanks to everyone we spoke with at the PyCon sprints and who worked on Warehouse and other packaging projects, including by testing two-factor auth, learning to package for the first time, and reviewing open pull requests! And thanks to volunteer contributors lukpueh, MattIPv4, hugovk, vinayak-mehta, HonzaKral, alex, alexwlchan, ppiyakk2, ofek, theb10n707, jamadden, ALDamico, and DavidBord for pull requests!

Our backend development contractors, in particular William Woodruff but also his colleagues at Trail of Bits, finished their TOTP-based multi-factor authentication pull request and responded to reviews from community maintainers, who approved and merged it. They then began work on WebAuthN-based multi-factor authentication, which is in progress and will let you use, for instance, Yubikeys for your second factor. So far that has included a fix to an upstream library – thank you, Duo!

The Frontend and UX contractor, Nicole Harris, finalized the review for the user experience for TOTP-based multi-factor authentication and started to define PyPI’s manual account recovery process, and is working on improving the WebAuthn authentication and provisioning user experience.

The project manager (me) also ran sprints at PyCon (see details at https://wiki.python.org/psf/PackagingSprints and http://bit.ly/pypa2019 ), and expanded our test period publicity in mid-May to ensure we reach users in important technological categories (e.g., who have slow internet connections).

In case you’re curious what issues and bugs we’ve found so far, check out some examples:

Next steps! Check out the OTF security work milestone on GitHub.

  1. Finish WebAuthN.
  2. API keys including adding scoping for users & projects. Heads up @westurner. :slight_smile:
  3. Then that will make adding audit trails/logs easier (reusing scoping and what any given token is being used for).

And we’ll probably be able to parallelize a bit and have Nicole start on Milestone 2 (Accessibility and internationalization development) before we’re quite finished shipping all 3 of those.

As a reminder, TUF and cryptographic signing is NOT in scope for this current project, and will only start after we’re done with the current project. The TUF GitHub issue and PyCon sprint notes are a good place to comment if you want to talk about TUF!

We’re looking forward to continuing to ship components of our project as we progress. And, as always, you can read our notes at https://wiki.python.org/psf/PackagingWG .

Thanks again to the Open Technology Fund for making this work possible!

3 Likes

Thanks for implementing 2FA! Is it possible to add more than one TOTP app, or to have some kind of recovery codes? I’m worried that losing my phone would lock me out of my account.

Hi, @jks – thanks for your questions!

We haven’t yet implemented recovery codes – here’s the GitHub issue for that – and it’s something we plan to do, but it’s not as urgent as making progress on WebAuthn and the other OTF-funded milestones. We’re trying to maximize how much progress we make on core features (WebAuthn, API keys, the audit log, and so on) using the OTF funds.

If you lose your phone, just like if you lose/forget your password, you can ask for a manual account reset/recovery. And that is a reason why we’ve mandated that you have to verify an email address on your account before you can turn on 2FA. :slight_smile: We’re also polishing our manual account recovery policy – once we’ve firmed it up more, we’ll put that in the FAQ, and once we have recovery codes implemented, that’ll be in the FAQ as well.

2 Likes

Ok, good to hear that there is a recovery process. In that case I completely agree that the other features are more urgent.

1 Like

Hey @jks!

To follow up on the TOTP question: we currently plan to support multiple WebAuthn keys, but not multiple TOTP applications.

The reasons for this are mostly technical and not contractual: because TOTP is fully symmetric and the server-side has no notion of client/application identity, the process of validating a TOTP code for N provisioned TOTP devices would require N (actually N * 2 in the worst case, because of valid windows) checks. Those checks aren’t particularly intensive, but they add complexity to an otherwise relatively simple verification process. Similarly from a provisioning/deprovisioning perspective: because TOTP applications aren’t identifiable, we’d be reliant on users to accurately tag/label each device within account management. That introduces additional state/deconflict work and potentials for user error (e.g., incorrectly naming and then deleting the wrong TOTP key). Ultimately, we can avoid all of that while also mitigating the lost/deleted application scenario by (1) adding WebAuthn as a parallel 2FA method, and (2) allowing users to provision multiple WebAuthn credentials (since identity is baked into the WebAuthn scheme). Both of those tasks are on the roadmap and are in progress in https://github.com/pypa/warehouse/pull/5795.

Hope that answers your questions! Please feel free to ping me if you have any others/any other feedback/ideas :smile:

2 Likes

Thanks, @woodruffw!

Work continues on WebAuthn support. And! We launched TOTP-based 2FA across PyPI & test PyPI yesterday. :tada:

1 Like

Summary: Work continues on the Multi-Factor Authentication task within Milestone 1, Security Feature Development. TOTP-based two-factor auth has rolled out for everyone, and we’re close to shipping the beta of WebAuthN support (e.g., Yubikeys) for y’all to test. Here’s a rough recording of TOTP and WebAuthn in action (activation, login, deactivation, etc.). And soon we’ll start accessibility work as well.

We had a short planning meeting today; notes are on the wiki.

Security: Yes, you might be able to start wielding your Yuibkeys and similar access tokens for 2FA on PyPI as early as next week! I’ll be announcing rollout dates as soon as we have them; we may do a beta similar to what we did for TOTP, or we may just push it live to everyone but with a badge cautioning that WebAuthn is a beta feature. We don’t have a proper feature flag system so user access to the private beta was a bit of a pain.

Accessibility: Nicole (our UX and frontend expert who’s been creating and improving 2FA UIs) has time available right now, so once we’ve finished WebAuthn, we’re going to temporarily switch to accessibility work – the next OTF-funded milestone – to give her tasks to do. Volunteer Mattias Östblom, a front end developer working at Axess Lab, did a light accessibility review of PyPI about a year ago, and practically all those issues still need fixing (if you’re reading this, feel free to dive in), but the first step in our funded accessibility work is going to be a full proper audit.

Thanks to volunteers MattIPv4, alanbato, Ernest W. Durbin III, Dustin Ingram, and Donald Stufft in particular for writing and reviewing Warehouse code in the past few weeks! (Including on cross-origin requests, a clearer security policy, rate-limiting TOTP submission, and a fix to 2FA notification.)

You can help too!

Thanks to OTF for their support for the PyPI & Warehouse work!

1 Like

To quote the blog post:

To further increase the security of Python package downloads, we’re adding a new beta feature to the Python Package Index: WebAuthn support for U2F compatible hardware security keys as a two-factor authentication (2FA) login security method. This is thanks to a grant from the Open Technology Fund, coordinated by the Packaging Working Group of the Python Software Foundation.

Starting today, PyPI also supports (in beta) WebAuthn (U2F compatible) security keys for a second login factor. A security key (also known as a universal second factor, or U2F compatible key) is hardware device that communicates via USB, NFC, or Bluetooth. Popular keys include Yubikey, Google Titan and Thetis. PyPI supports any FIDO U2F compatible key and follows the WebAuthn standard. Users who have set up this second factor will be prompted to use their key (usually by inserting it into a USB port and pressing a button) when logging in. (This feature requires JavaScript.)

a screenshot of the 2FA setup screen, with documentation and a place to name your key

We need your help testing this while it’s in beta. Later this week I’ll publicize it to some more communities, and then in maybe 10 days, assuming we can quickly fix all the urgent bugs we find, we’ll remove the “beta” badge.

During this testing period, if things go awry, there’s a chance we will need to wipe tokens from users’ accounts, so if you choose to try it, please be forewarned. That’s why you have to have a PyPI-verified email address on your user account before trying the feature, to make potential account recovery smoother.

Thanks to the Open Technology Fund for funding this work. The list of our progress reports is at the Packaging Working Group’s wiki page.

1 Like