PyPI security work: multifactor auth progress & help needed

Packaging
#12

How does the plan to implement TUF (and maybe webauthn) dovetail with this work?

What issues need more attention from volunteers with which skills?

2 Likes
#13

What’s the GH Issue # for the API keys work; is the system being designed so that I can create a per-package key (so that I’m not delegating all privileges to my CI builds and CI build systems)?

#14

We’re excited to help with the encryption support aspect whatever is decided. Just point us at the best way to dive in. I can’t personally make it but I can check to see if someone from our team can attend PyCon if being there in person would be a major help…

3 Likes
#15

Yeap, same here, ready to help with the PyPI + TUF effort, just let me know

Cc @pradyunsg

1 Like
#16

Lukas from the TUF team at NYU here. I will be at the PyCon sprints next week, and I’m looking forward to discuss and work on PyPI + TUF integration.

2 Likes
#17

Summary: Work continues on Milestone 1, Security Feature Development, and specifically on the Multi-Factor Authentication task. TOTP-based 2FA is about to roll out for everyone, and we’re working on WebAuthN (e.g., Yubikeys).

In April and in the first week of May, the team finalized the backend and user experience for 2FA and planned and started user tests. On May 2nd, we began advertising the test to our users and requesting feedback. That test has already found issues which we have fixed or prioritized to fix later.

We planned for the test to go till May 20th. We decided to require email verification before activating 2FA, and that’s underway. Once we finish that, we’ll be turning on the optional 2FA login feature for current and future accounts on pypi.org (it’s already on for most existing accounts on Test PyPI, and we’ll turn it on for all current and future accounts there, too.) There are some UI issues that we should fix in the medium term, but I’ve decided it’s ok to roll out the feature and end the beta before fixing those.

Thanks to everyone we spoke with at the PyCon sprints and who worked on Warehouse and other packaging projects, including by testing two-factor auth, learning to package for the first time, and reviewing open pull requests! And thanks to volunteer contributors lukpueh, MattIPv4, hugovk, vinayak-mehta, HonzaKral, alex, alexwlchan, ppiyakk2, ofek, theb10n707, jamadden, ALDamico, and DavidBord for pull requests!

Our backend development contractors, in particular William Woodruff but also his colleagues at Trail of Bits, finished their TOTP-based multi-factor authentication pull request and responded to reviews from community maintainers, who approved and merged it. They then began work on WebAuthN-based multi-factor authentication, which is in progress and will let you use, for instance, Yubikeys for your second factor. So far that has included a fix to an upstream library – thank you, Duo!

The Frontend and UX contractor, Nicole Harris, finalized the review for the user experience for TOTP-based multi-factor authentication and started to define PyPI’s manual account recovery process, and is working on improving the WebAuthn authentication and provisioning user experience.

The project manager (me) also ran sprints at PyCon (see details at https://wiki.python.org/psf/PackagingSprints and http://bit.ly/pypa2019 ), and expanded our test period publicity in mid-May to ensure we reach users in important technological categories (e.g., who have slow internet connections).

In case you’re curious what issues and bugs we’ve found so far, check out some examples:

Next steps! Check out the OTF security work milestone on GitHub.

  1. Finish WebAuthN.
  2. API keys including adding scoping for users & projects. Heads up @westurner. :slight_smile:
  3. Then that will make adding audit trails/logs easier (reusing scoping and what any given token is being used for).

And we’ll probably be able to parallelize a bit and have Nicole start on Milestone 2 (Accessibility and internationalization development) before we’re quite finished shipping all 3 of those.

As a reminder, TUF and cryptographic signing is NOT in scope for this current project, and will only start after we’re done with the current project. The TUF GitHub issue and PyCon sprint notes are a good place to comment if you want to talk about TUF!

We’re looking forward to continuing to ship components of our project as we progress. And, as always, you can read our notes at https://wiki.python.org/psf/PackagingWG .

Thanks again to the Open Technology Fund for making this work possible!

3 Likes
#18

Thanks for implementing 2FA! Is it possible to add more than one TOTP app, or to have some kind of recovery codes? I’m worried that losing my phone would lock me out of my account.

#19

Hi, @jks – thanks for your questions!

We haven’t yet implemented recovery codes – here’s the GitHub issue for that – and it’s something we plan to do, but it’s not as urgent as making progress on WebAuthn and the other OTF-funded milestones. We’re trying to maximize how much progress we make on core features (WebAuthn, API keys, the audit log, and so on) using the OTF funds.

If you lose your phone, just like if you lose/forget your password, you can ask for a manual account reset/recovery. And that is a reason why we’ve mandated that you have to verify an email address on your account before you can turn on 2FA. :slight_smile: We’re also polishing our manual account recovery policy – once we’ve firmed it up more, we’ll put that in the FAQ, and once we have recovery codes implemented, that’ll be in the FAQ as well.

3 Likes
#20

Ok, good to hear that there is a recovery process. In that case I completely agree that the other features are more urgent.

2 Likes
#21

Hey @jks!

To follow up on the TOTP question: we currently plan to support multiple WebAuthn keys, but not multiple TOTP applications.

The reasons for this are mostly technical and not contractual: because TOTP is fully symmetric and the server-side has no notion of client/application identity, the process of validating a TOTP code for N provisioned TOTP devices would require N (actually N * 2 in the worst case, because of valid windows) checks. Those checks aren’t particularly intensive, but they add complexity to an otherwise relatively simple verification process. Similarly from a provisioning/deprovisioning perspective: because TOTP applications aren’t identifiable, we’d be reliant on users to accurately tag/label each device within account management. That introduces additional state/deconflict work and potentials for user error (e.g., incorrectly naming and then deleting the wrong TOTP key). Ultimately, we can avoid all of that while also mitigating the lost/deleted application scenario by (1) adding WebAuthn as a parallel 2FA method, and (2) allowing users to provision multiple WebAuthn credentials (since identity is baked into the WebAuthn scheme). Both of those tasks are on the roadmap and are in progress in https://github.com/pypa/warehouse/pull/5795.

Hope that answers your questions! Please feel free to ping me if you have any others/any other feedback/ideas :smile:

2 Likes
#22

Thanks, @woodruffw!

Work continues on WebAuthn support. And! We launched TOTP-based 2FA across PyPI & test PyPI yesterday. :tada:

2 Likes
Can we increase the 3-consecutive-post limit?
#23

Summary: Work continues on the Multi-Factor Authentication task within Milestone 1, Security Feature Development. TOTP-based two-factor auth has rolled out for everyone, and we’re close to shipping the beta of WebAuthN support (e.g., Yubikeys) for y’all to test. Here’s a rough recording of TOTP and WebAuthn in action (activation, login, deactivation, etc.). And soon we’ll start accessibility work as well.

We had a short planning meeting today; notes are on the wiki.

Security: Yes, you might be able to start wielding your Yuibkeys and similar access tokens for 2FA on PyPI as early as next week! I’ll be announcing rollout dates as soon as we have them; we may do a beta similar to what we did for TOTP, or we may just push it live to everyone but with a badge cautioning that WebAuthn is a beta feature. We don’t have a proper feature flag system so user access to the private beta was a bit of a pain.

Accessibility: Nicole (our UX and frontend expert who’s been creating and improving 2FA UIs) has time available right now, so once we’ve finished WebAuthn, we’re going to temporarily switch to accessibility work – the next OTF-funded milestone – to give her tasks to do. Volunteer Mattias Östblom, a front end developer working at Axess Lab, did a light accessibility review of PyPI about a year ago, and practically all those issues still need fixing (if you’re reading this, feel free to dive in), but the first step in our funded accessibility work is going to be a full proper audit.

Thanks to volunteers MattIPv4, alanbato, Ernest W. Durbin III, Dustin Ingram, and Donald Stufft in particular for writing and reviewing Warehouse code in the past few weeks! (Including on cross-origin requests, a clearer security policy, rate-limiting TOTP submission, and a fix to 2FA notification.)

You can help too!

Thanks to OTF for their support for the PyPI & Warehouse work!

2 Likes
#24

To quote the blog post:

To further increase the security of Python package downloads, we’re adding a new beta feature to the Python Package Index: WebAuthn support for U2F compatible hardware security keys as a two-factor authentication (2FA) login security method. This is thanks to a grant from the Open Technology Fund, coordinated by the Packaging Working Group of the Python Software Foundation.
…

Starting today, PyPI also supports (in beta) WebAuthn (U2F compatible) security keys for a second login factor. A security key (also known as a universal second factor, or U2F compatible key) is hardware device that communicates via USB, NFC, or Bluetooth. Popular keys include Yubikey, Google Titan and Thetis. PyPI supports any FIDO U2F compatible key and follows the WebAuthn standard. Users who have set up this second factor will be prompted to use their key (usually by inserting it into a USB port and pressing a button) when logging in. (This feature requires JavaScript.)

a screenshot of the 2FA setup screen, with documentation and a place to name your key

We need your help testing this while it’s in beta. Later this week I’ll publicize it to some more communities, and then in maybe 10 days, assuming we can quickly fix all the urgent bugs we find, we’ll remove the “beta” badge.

During this testing period, if things go awry, there’s a chance we will need to wipe tokens from users’ accounts, so if you choose to try it, please be forewarned. That’s why you have to have a PyPI-verified email address on your user account before trying the feature, to make potential account recovery smoother.

Thanks to the Open Technology Fund for funding this work. The list of our progress reports is at the Packaging Working Group’s wiki page.

1 Like
#25

Thanks for the regular updates @sumanah!

Is anyone giving out USB-A WebAuthn keys? I don’t have one and it’d be nice to be able to secure pip’s PyPI package with 2FA instead of, like, just my password.

#26

Use TOTP on your phone. Like FreeOTP+

#27

Summary: In June, to use different people’s time well, we parallelized our work a bit. The multifactor auth task within Milestone 1, Security Feature Development is getting a lot closer to done – we shipped the beta of WebAuthn support and are actively seeking out test subjects and fixing bugs based on their input – but we also started work on API keys and on accessibility.

2FA: Users can now use U2F keys to better secure their accounts! Yay WebAuthn! As the beta proceeds (and thanks to @nlhkabu for the beta badge and FAQ), we’ve been fixing issues – the funded contractors @nlhkabu and @woodruffw, as well as @EWDurbin at the PSF with review help from @dustin and @dstufft, so thanks to them and to the folks who filed issues! Some 2FA improvements from the past month:

Testing: @nlhkabu and I have been seeking out users from a variety of backgrounds during the beta. I’ve been asking package maintainers to test the beta and file issues. She’s started a fresh round of user tests with more novice Python users to validate our current 2FA design, and has started collating responses to turn into GitHub issues.

Accessibility: Also last month, @woodruffw audited Warehouse’s accessibility. We’re actually a lot more solid there than I worried we’d be! A lot of credit there is due to @nlhkabu, who committed our current accessibility guidelines for developers three years ago yesterday. But we still found things to fix, and started to address them through both research and implementation (annotating the search form correctly, increasing link visibility, fixing a tabindex).

API keys: We had a chat to make some design decisions on scoped API keys, and Will’s made substantial progress on a work-in-progress PR (not ready for review yet). To quote Will’s summary:

We’re going to work with macaroons from the very beginning, and not go with dumb API keys as I proposed above…

In order to minimize the amount of time spent on implementation, I intend to deliver a [proof of concept] version without constraints or a caveat language. This deliverable will meet the requirements of the [Statement of Work] (allowing users to replace their username/password with a single token for upload only), and will serve as the foundation for future iterations. Upload-only enforcement will be handled by route whitelisting and a version identifier within the macaroon, preventing future iterations from inadvertently creating “god” tokens.

And, as project manager, I led a planning meeting, coordinated volunteers and contractors for testing, code contribution, security design review and code review, reached out to external communities for further testing, planned issues and milestones for upcoming grant-funded work, reviewed pull requests, triaged feature requests that are out of scope, and added documentation.

Thanks to volunteers ppiyakk2, @trishankatdatadog, robindboer, @yeraydiazdiaz, @alex_Gaynor, minho42, @dustin, and @dstufft in particular for writing & reviewing Warehouse code in the past few weeks! (Including fixes to email validation, window scrolling, the 2FA token form field, and Docker image cleaning.)

Next: In July we aim to finish this round of user tests, fix resulting bugs, and declare WebAuthn support out of beta, and thus complete the MFA task within Milestone 1. We also aim in July to make further progress on accessibility, and on API keys, but I don’t know whether either of those will be complete in July.

Please help us out:

Sorry that this was for a month rather than a fortnight; next summary update will be in more like 2 weeks.

Thanks to Open Tech Fund for their support for the PyPI & Warehouse work!

#28

@sumanah It was a pleasure hacking on Warehouse with friends and yourself! Let me know how else I can help…

1 Like
#29

Summary: In the last two weeks, we’ve made major progress on the multifactor auth milestone and substantial progress on accessibility. Adventurous developers can test out user-scoped and project-scoped API keys for Warehouse right now, and PyPI and Test PyPI are already more accessible.

(@nlhkabu and @woodruffw have been collaborating on all this.)

2FA: Our WebAuthn/U2F key support for two-factor auth is still in beta. That’s partly because Nicole ran several user tests and found user experience confusion that she’s now fixed. And it’s because we still need to fix an Edge issue, a Chrome issue, and an accessibility issue. I am looking forward to sending the “out of beta!” mail to the announcement list but I don’t have an estimate.

Also, check out the new FAQ: we now “Allow anonymous WebAuthn attestation”: “allows users to use TouchID and other methods that don’t use separate public keys.” Thanks, Will!

Uploading to PyPI with an API key: Will’s pull request is ready for you to test it, if you’re adventurous and you’re comfortable setting up a Warehouse developer environment on your computer! In case you haven’t tested this kind of feature locally before, here’s how.

Accessibility: Check out the milestone! Nicole’s already fixed several issues so our sites are already friendlier to people who use screenreaders, and to color-blind and color-impaired users.

Also a shout-out to Nicole for fixing this image rendering issue.

Next up: finishing up the WebAuthn beta, improving the upload API keys feature to the point where we can release it as a beta on both PyPI sites, and finishing up the accessibility milestone. We still have some security work to come, specifically the audit log. And then the localization work.

Thanks to volunteers @gpshead, @EWDurbin, @dstufft, @dustin, @alex_Gaynor, sayanarijit, and @yeraydiazdiaz in particular for writing and reviewing Warehouse code in the past two weeks!

Please help us out: please spread the word in your work and developer circles that two-factor auth is available on PyPI. Lots of people don’t know yet! Example:

Thanks again to the Open Technology Fund for making this work possible!

3 Likes
#30

I saw that Tidelift mentioned this thread saying we’d listed some 2FA improvement GitHub issues that need help. So here are two:

Also, hey package maintainers & owners: what should owners be able to see in an audit log that maintainers shouldn’t be able to see? We’re working on that feature pretty soon.

#31

You can now use API tokens to upload packages to PyPI and Test PyPI! Warning that this is a beta feature. More details on the wiki and PyPI’s help section.

This is a first step to enforcing that Users with Two-Factor Authentication enabled will require API Tokens to upload, rather than just their password sans second factor.

Once the beta period for API Tokens is complete, we will notify parties with Two-Factor Authentication enabled that uploads for their projects will require API Tokens. After a suitable waiting period we will then begin to enforce this restriction and include a notice in the error message returned to clients.

Please do test this while it’s in beta. Lead developer William Woodruff says, “Our current auth-policy is drop-in compatible with Twine and distutils. When using a token, your “username” will be @token and your “password” will be the token itself.” So, if your token is pypi:Ab9GpH-H5y your command will be:

twine upload --repository-url https://test.pypi.org/legacy/  -u @token -p pypi:Ab9GpH-H5y dist/*

(but actual tokens are 160+ characters long).

API tokens also support scoping! To ensure that your newly created tokens only contain the minimum permissions they require, make sure to select the package you’d like to use them with on the creation page. By default, newly created tokens will have “user” scope, meaning that they’ll behave exactly like your password. Once created, the permissions associated with a token cannot change, the token can only be revoked.

We’d particularly like testing from:

  • Orgs that automate uploads using continuous integration
  • People who save PyPI credentials in a .pypirc file
  • Windows users
  • People on mobile devices
  • People on very slow connections
  • Organizations where users share an auth token within a group
  • 4+ maintainers or owners for one project
  • Use an unusual TOTP app or U2F token
  • Usually block cookies and JavaScript (note that you can’t set up a U2F key without JavaScript)
  • Maintain 20+ projects
  • Created PyPI account 6+ years ago
5 Likes