Namespace support in pypi

[dstufft (Donald Stufft)]

As far as I am aware there has been no progress. It would require someone to write and champion a PEP, and then someone to implement it assuming it got accepted.

[ncoghlan (Alyssa Coghlan)]

Addding a belated note on the “private package” problem: Linux distros face a variant of this with system API binding packages that are installed directly into the system Python by a primarily C/C++ focused build process instead of being published as regular Python packages.

I think any PEP should put this into the “No need to solve” category though, for the following reasons:

1. These are open source projects, so they (or a consuming distro) are free to make an sdist and publish it to PyPI (best resolution)
2. Distros that allow for multiple Python stacks or fully support venvs will likely want the sdist anyway, so the bindings can be used outside the system Python (encourages the best resolution)
3. When there are technical barriers to the best resolution, actual namesquatting is a defensible interim measure given the availability of PEP 541 to address disputes (e.g. I’ve held the “rpm”, “dnf” and “solv” names on PyPI for years, and relatively recently allowed a team from Red Hat access to the last to publish real libsolv bindings).
   4.The namesquatting workaround could be made more systematic (similar to the blocking of stdlib names), but any prefix based name reservation would only apply to future explicitly distro-controlled packages and, for that purpose, distros fall under the same design category as “largeco” in NJS’s write-up.

[encukou (Petr Viktorin)]

@ncoghlan

When there are technical barriers to the best resolution, actual namesquatting is a defensible interim measure given the availability of PEP 541 to address disputes (e.g. I’ve held the “rpm”, “dnf” and “solv” names on PyPI for years, and relatively recently allowed a team from Red Hat access to the last to publish real libsolv bindings).

See proposal for name reservations: PyPI as a Project repository vs. Name registry (a.k.a. PyPI namesquatting, e.g. for Fedora packages) - #12 by encukou

@ncoghlan

The namesquatting workaround could be made more systematic (similar to the blocking of stdlib names), but any prefix based name reservation would only apply to future explicitly distro-controlled packages and, for that purpose, distros fall under the same design category as “largeco” in NJS’s write-up.

I don’t think distros need prefix name reservation. Who’d be the gatekeeper? If someone random contributor wants to write a Fedora-specific tool and name it fedora-foobar, they should go right ahead. (Though of course, just foobar would be a better name—if it’s good, others might want to adopt it.)

[sumanah (Sumana Harihareswara)]

@sumanah

Thanks for the summary, @njs!

• Comment: IMO this doesn’t seem very promising right now, because we don’t have good ways to manage the resulting conflicts at the Python import level. Maybe it’s worth revisiting after we have a robust resolver and Conflicts metadata?

For those who haven’t been following it, here’s the GitHub issue about planning the rollout of the new pip resolver .

#988 is now closed as pip 20.3 has the new resolver on by default.

[dblack (David Black)]

Hi all, in light of Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies | by Alex Birsan | Feb, 2021 | Medium is now an appropriate time to revisit this & Improving risks and consequences against typosquatting on pypi - #12 by PoluX ?

[uranusjr (Tzu-ping Chung)]

IMO this is not (directly) related to the issue. As you can read from the article, npm (which does support namespacing) is still affected by the issue. On the flip side, the issue can be quite easily prevented by registering placeholder packages on PyPI even without namespacing, and I believe the PyPI team accept requests to reserve names without actually publishing any releases as well. Namespacing can make prevention a bit easier, but can’t help much, nor is necessary to avoid it from happening.

[dblack (David Black)]

Agreed. Although it may make it easier to make a clear distinction between what is <company A’s> and what isn’t.

and I believe the PyPI team accept requests to reserve names without actually publishing any releases as well

Is there a sanctioned/“official” means of doing this?

[kpfleming (Kevin P. Fleming)]

@uranusjr

As you can read from the article, npm (which does support namespacing) is still affected by the issue

This is true only when package publishers fail to make use of the namespace support. If PayPal’s auth-paypal package had been named @paypal/auth, with PayPal in control of the @paypal scope, they would not have been attackable using this technique.

Solving this problem requires both namespace support, and awareness of that support’s existence and its importance.

[uranusjr (Tzu-ping Chung)]

You can squat a package name without namespacing as well (at least on PyPI); if auth-paypal were a Python package, they could claim that name on PyPI with a stub package, or ask for PyPI admins to give it to them. Namespacing makes the process much easier, but it is not required.

p.s. To be clear, I’m fully supportive of having namespace support in Python packaging. But it should not be presented as a requirement incorrectly.

[encukou (Petr Viktorin)]

Based on my discussion with them, PyPI maintainers prefer you sending a list of names to block to admin@pypi.org.
Name squatting is against PyPI policy.

[kpfleming (Kevin P. Fleming)]

In order to use the blocking/squatting method, we’d have to send a list of approximately 1,000 internal packages, and we’d have to update the list daily. To ensure safety, we’d have to delay publication of a new internal package until after the block/squat has been put in place.

So while it is true that namespace support is not required, I can’t imagine it being practical to handle enterprise package protection without it.

[pradyunsg (Pradyun Gedam)]

Speaking in a personal capacity, this would be a great thing to add to fundable-packaging-improvements/FUNDABLES.md at master · psf/fundable-packaging-improvements · GitHub. There’s consensus that we want this in some form, and there’s at least 2 “good models” for how this could work.

This is definitely a good candidate for “targeted funding to do this” style grants, especially given that some of the organisations that care about this might also be willing to fund this work.

And, if you’re reading this as an organisation interested in moving this forward (through funding), I suggest dropping an email to packaging-wg@python.org that you’re interested in funding this.