2FA usability on PyPI and with packaging tools

[LtWorf (Salvo 'LtWorf' Tomaselli)]

I am slowly moving from github to codeberg these days. I moved all the small
projects and I am kind of stuck with the bigger ones. But certainly new
projects I’d start on codeberg and not on github.

[dustin (Dustin Ingram)]

@LtWorf

The link seems specific to projects on github.

Does the method work in general?

PyPI currently only supports GitHub since they were one of the first providers that had the necessary functionality to make this possible, however it is based on open standards and we have plans to add additional providers in the future (Google Cloud, ActiveState, GitLab, CircleCI and Cirrus CI are in the works.)

@pitrou

Also, this assumes that people want to automate project creation, which seems a bit weird and overkill.

The goal is to make “project creation” (or publishing the first release) happen exactly the same way as publishing additional releases, and not require special steps, workflows, tokens, or interfaces just because it’s the first release.

[pitrou (Antoine Pitrou)]

This seems to be entirely because PyPI does not allow creating a project without pushing a first release (which IMO is a wart, not a feature). Otherwise, people would just push the “Create project” button somewhere on the PyPI Web UI

[fungi]

This seems to be entirely because PyPI does not allow creating a
project without pushing a first release (which IMO is a wart, not
a feature). Otherwise, people would just push the “Create project”
button somewhere on the PyPI Web UI

My understanding is that the “feature” of being able to explicitly
create empty projects in Cheeseshop was intentionally omitted in
Warehouse’s design because people were using it to create projects
to which they never pushed any packages, and this new workflow sends
a clear signal that such package name “squatting” is discouraged.

[jamestwebber (James Webber)]

If I could create packages the way I create GitHub repos that I never follow up on, PyPI would be in trouble…

[pitrou (Antoine Pitrou)]

Or there could just be a delay after which a project without releases is deleted.

I also don’t understand how this relates to namesquatting. People who want to namesquat popular names can upload empty releases anyway…

[ofek (Ofek Lev)]

The extra burden I would assume discourages 99.9% of cases, similar to CAPTCHA.

[dstufft (Donald Stufft)]

@fungi

My understanding is that the “feature” of being able to explicitly
create empty projects in Cheeseshop was intentionally omitted in
Warehouse’s design because people were using it to create projects
to which they never pushed any packages, and this new workflow sends
a clear signal that such package name “squatting” is discouraged.

Well there was never any support for creating empty projects really. The only way to create a project was always to create a release.

Previously you had two options, you could use register to create a metadata only release, or upload to upload an artifact and create a release. We dropped the register command, requiring an artifact to be uploaded to create a new release.

Many people used register to create projects with a zero version, and then would delete the version after, but it wasn’t really fundamentally different than what we have now, except you also need to upload an empty artifact.

[dstufft (Donald Stufft)]

I don’t think there’s any reason we can’t create a “Create Project” button in the UI.

We basically already have this, but it’s scoped to just trusted publishing, we call it “Pending Publishers”.

With a Pending Publisher, you’re basically registering the intent to publish something using trusted publisher, but you’re not actually registering or claiming the name. Rather the name remains unclaimed until you actually release something with that Pending Publisher, at which point it tries to create the project (and fails if someone else already registered the project in the interim).

So you could image something like that, where you can create a Project scoped API token that is scoped to a pending project, that converts to a real project on first use, and if someone else uses the name in the interim then it fails.

[kknechtel (Karl Knechtel)]

@dstufft

With a Pending Publisher, you’re basically registering the intent to publish something using trusted publisher… you could image something like that, where you can create a Project scoped API token that is scoped to a pending project, that converts to a real project on first use, and if someone else uses the name in the interim then it fails.

Perhaps I’m missing something. What problem does it solve to acquire an API token in advance, if there’s a risk that it becomes invalid before first use? Where is the intent being “registered”? If that registration doesn’t actually include a name registration or claim, then who is expected to care about the intent, and why?

[effigies (Chris Markiewicz)]

The point is to limit damage if a token is compromised. If you have to create a global token to upload a new package, then someone who acquires that token can upload unauthorized packages to any project you have permissions on. Creating tokens for as-yet-unpublished projects eliminates this use case that still requires global tokens.

[dstufft (Donald Stufft)]

@kknechtel

Perhaps I’m missing something. What problem does it solve to acquire an API token in advance, if there’s a risk that it becomes invalid before first use? Where is the intent being “registered”? If that registration doesn’t actually include a name registration or claim, then who is expected to care about the intent, and why?

Currently you can either register a “user scoped” token, which can upload to any project that the user has access to or upload new projects OR a “project scoped” token, which can upload to the projects that the user has elected.

Because project scoped tokens encode the actual primary key of the project within the token (to prevent rebinding attacks where someone creates a token, deletes the project, then the project is recreated), you can’t currently create a project scoped token for a token that doesn’t yet exist.

This creates a gap where if you want to create a new project, you can’t use a project scoped token, and are forced to use a user scoped token.

There are of course options for how this could work. We could just create a form that lets you create a fully fledged project with just a web form, which would then allow the user to create an upload token. This has the problem (or well, trade off) that we make name squatting a little bit easier.

Maybe that’s ultimately OK, but it’s a different choice then we made for Trusted Publisher, where instead of a Create Project button, we let people create a Pending Publisher, that registered your intent to create a project with a given trusted publisher, but didn’t actually do it until the first upload. The rough idea I outlined creates a mechanism similar to what we have for Pending Publisher, but for project scoped API tokens.

It would be easier to just make a Create Project form, so there is that benefit.

[kknechtel (Karl Knechtel)]

@dstufft

Because project scoped tokens encode the actual primary key of the project within the token (to prevent rebinding attacks where someone creates a token, deletes the project, then the project is recreated), you can’t currently create a project scoped token for a token that doesn’t yet exist.

Under this proposal, would they encode something else, then? (I’m assuming you refer to a database table in the Warehouse implementation, and that it’s primary-keyed on something other than simply the project name… ?)

[oscarbenjamin]

@dstufft

Currently you can either register a “user scoped” token, which can upload to any project that the user has access to or upload new projects OR a “project scoped” token, which can upload to the projects that the user has elected.

Because project scoped tokens encode the actual primary key of the project within the token (to prevent rebinding attacks where someone creates a token, deletes the project, then the project is recreated), you can’t currently create a project scoped token for a token that doesn’t yet exist.

Could it be possible to have a kind of token whose only scope is to allow creating a new project and pushing files for its first release?

[dstufft (Donald Stufft)]

@oscarbenjamin

Could it be possible to have a kind of token whose only scope is to allow creating a new project and pushing files for its first release?

There’s no technical reason we couldn’t do that.

Basically our API tokens work by allowing us (or anyone who posses the API token actually!) to append any infinite number of restrictions onto the tokens, and all of the restrictions must be satisfied in order for the token to be valid. The main constraint about these rules that can be appended is that they must only ever restrict what the token can do, they can’t expand it.

So anything you can write in a language of “restrict this token to only be valid when …” you can pretty much do.

In this case, you’d add a restriction that it can only be used for project creation, so if the project already exists, it can’t work.

That would require a little refactoring work in our actual permission checking code because we currently don’t check permissions until we’ve either retrieved the existing project or created a new project, because our permissions are currently of the form of “has permission to do X on Y”, but I don’t think it would be terribly hard.

I’m not sure if that would be better or worse than the idea of being able to make a project scoped token for a project that hasn’t been registered yet. The main practical difference I guess would be the one time nature, whether it could only be used for a new project, or whether it ends up functioning like a regular project scoped token after.

Well I guess the other practical difference is that you could use the “only allow new projects” token as your default token in twine, which then gets overriden for existing projects with project scoped tokens.

[sirosen (Stephen Rosen)]

As someone who has been converting to the currently available feature for trusted publishing, I would like to be allowed to create an empty project (even if it has limits, like a 24 hour lifetime).

Right now, I need to follow two slightly different UI flows on pypi for new vs existing projects. It’s a minor thing, but yet another bit of friction which supports the idea of empty projects.

---

All of this gets a little bit away from the original topic of 2FA though.

The previously suggested user experience for a CLI-driven upload seems somewhat reasonable to me, in which you are promoted to go to a browser to verify. I think it could even be refined to be simpler and reuse whatever mechanism is used by the pypi frontend for 2FA: prompt the user to enter their 2FA code directly into the CLI tool doing the upload.

i.e. A good user experience might be

• I have one (global) pypi token in pypirc
• I have enabled 2FA
• I use twine upload to manually publish
• Every time I do do, twine prompts me for my 2FA code

I’m not sure what would need to be added to pypi (anything? probably some slightly new non-password based API?), but this seems like a nice solution.

I grant that those global tokens are seen as undesirable, but they are quite a lot easier to get started with.

Is this proposal unreasonable? Unsafe? Requires a PEP?