Protecting component package names on PyPI before a project is ready to publish

Packaging
,
1

I’m building an open source game engine in pure Python, with Vulkan desktop rendering and
WebGPU export so games run in the browser. The site and demos are up and I’m putting together
the infrastructure for early dev previews. First public preview is about a month out, but a
real release is more like 6 to 12 months away.

The project is split up the usual way into projectname, projectname-core,
projectname-graphics, projectname-editor and a few more. Right now they come off a private
index as 0.0.0.dev### builds, and they pin each other with exact == versions so that an
install is always one matched set, excluding some optional components.

While getting ready for the preview I realised this creates a dependency confusion problem.
My index doesn’t mirror the rest of PyPI, so anyone installing needs PyPI configured for the
other dependencies. Once both indexes are in play pip picks by version across all of them
rather than preferring mine, and none of my component names exist on PyPI yet. So anyone
could register one of those names, publish a higher version than my real builds, and win the
resolution outright, even for a user who pointed the top level package at my index correctly.
I can tell people to use --index-url instead of --extra-index-url, but I can’t control other
people’s configuration. Whoever wins that resolution ends up running their code on my users’
machines, which is the part that actually worries me.

Pending publishers looked like the answer at first, but they don’t reserve a name until the
first actual upload, so they’re no help before you have something to upload.

That leaves uploading minimal placeholder releases under each name, which is what I’d assumed
people do. But PEP 541 lists “package has no functionality or is empty” as name squatting,
and that’s a fair description of a placeholder.

So is there an accepted way to hold component names before you have something real to
publish? And if placeholders are fine, what makes one fine rather than squatting? A module
that actually does something, like printing where to get the real builds? A link back to the
active project with regular dev pre-releases?

I don’t really feel comfortable pushing code out to something as permanent as PyPI until I
have proper infrastructure to ensure quality and security, so I’m in a bit of a bind. This
must come up fairly often for projects that pick up users before they’re ready for a
permanent PyPI release.

Any practical advice would be appreciated.

2

I don’t think the name you chose can satisfy both of the following:

  • It is currently available on PyPI
  • It is so common that the name being taken within a month is a legitimate concern

Anyway, a package with a version number and your preliminary code, regardless of the number of bugs, is not squatting.

3

That makes sense, thanks.

I’m still stuck on the practical side though. It isn’t the bug count that puts me off, it’s that I don’t have the review and release process yet that I’d want behind anything going somewhere permanent. I could probably carve out one or two components that are genuinely ready and publish those properly, but for one dev on a codebase this size that’s a fair bit of work, and it only covers the names I manage to ship.

On the name being taken, agreed that wouldn’t happen by accident. What I have in mind is bulk automation rather than anyone deciding they want this particular name. After the dependency confusion research in 2021 a single actor uploaded a few thousand scraped names across PyPI and npm and PyPI ended up pulling over 3,500 of them. Those were mostly proof of concept, but the same automation with a payload behind it would have worked just as well.

I’d expect AI to make this worse. An attacker can scrape new projects and generate the names most likely to be used, and mine follow a predictable projectname-thing pattern, so projectname-audio or projectname-physics are easy guesses. Publishing real code doesn’t help there, because those aren’t components I have.

4

Could someone cite a source on best practices from pypi.org from something like this?

I don’t want to hijack this thread, but at an organizational level, we may release more than a few number of packages that was developed internally. OP has the right idea, the worry is dependency confusion attacks for malicious actors with mined or privileged knowledge.

5

If it’s just quality and security, upload the 0.0.0.dev123 code. No one should expect that to be any good, or secure, and it should be OK on PyPI as long as it isn’t outright malicious.
As for permanence, you can delete a release later. (Though you don’t really need to – if you’ve flagged it as not ready, no one should blame you for any bugs.)

If their uploads are malicious or mass-registered, they’d be violating acceptable use policy and it should be relatively straightforward to get them taken down.
There can be moderation delays though, so ideally have these projects registered before announcing a release date – even if that means uploading projectname-physics 0.0.0dev0 that only implements collision with ground at y=0.

2 Likes
6

There are options for publishing initial releases without permanently affecting future release listings. For example:

  • yank the initial releases after publishing them
  • if you put a “Python-Requires: <1.0” in the package metadata for the preliminary release, most clients will ignore the release entirely

Publish-and-yank is the approach I’ve used for this problem myself. It’s especially appropriate when the placeholder packages have real metadata pointing to the actual source repository.

2 Likes
7

As a long term option it would be PEP 752 – Implicit namespaces for package repositories | peps.python.org as the solution to avoid namesquatting and dependency confusion attacks.

2 Likes
8

You can also create a PyPI org, and reserve names in the org through the web UI (there’s no API for that). No upload needed, but it can be a useful workflow. Uploading a “zero version” empty pre-release sdist also does the trick and that can be automated.

9

Thanks for the feedback. I’ll setup an organization as we wait for PEP 752 to make it’s way to PyPI, that’s the real long term solution. In the meantime, I’ll try releasing stub components with proper metadata, documentation for the actual code that will go there and some minimal code that lets it get imported by the core components. That way it has a purpose (documentation for in progress api + satisfy requirements/imports). I just really don’t want a cybersecurity incident at launch, that’d kill the project real quick.

@encukou I didn’t know about the delete option, good to know. I also have some licensing concerns though. I’m releasing the engine under AGPLv3 with game distribution exception (engine stays agpl but can distribute with closed source game, so long as attribution is given at load or somewhere static) so I have to make sure all of the code is compatible with that. I’m at well over a quarter million lines of python code at this point and some of my early prototyping code isn’t compatible with agpl. Open source is hard :smiling_face_with_tear:

1 Like
10

It’s important to actually document that the code will be there or provide some minimal functionality - empty packages are considered violating the ToS and are regularly removed.

1 Like
11

On PyPI? Is that actually true in general? Sure, perhaps for a name conflict that reaches the PEP 541 stage.

12

https://peps.python.org/pep-0541/#invalid-projects lists the following as an example of “invalid” project:

project is name squatting (package has no functionality or is empty);

If what you’re trying to do is make sure you have the names registered, what Protecting component package names on PyPI before a project is ready to publish - #5 by encukou and Protecting component package names on PyPI before a project is ready to publish - #6 by ncoghlan stated is what I’d recommend.

1 Like