At my company, we have quite a number of internal packages that use a common namespace. We did this to assure that our internal code does not clash with public packages on PyPI.
(There is actually a package with that prefix on PyPI, but we are highly unlikely to ever use it and the project appears dead)
As this was started like 10 years ago, all packages use the pkg_resources approach for namespace packages, but pkg_resources is deprecated as an API.
Therefore, I wanted to start slowly migrating all those packages that are still in use, but noted this remark in the docs:
To migrate an existing package, all packages sharing the namespace must be migrated simultaneously.
That is a hard task, given that the project I am working on alone has 39 of those packages as a dependency and there are around 30 that are used by other projects only. Also, there is no Conflicts dependency in Python’s package metadata, so I can’t make the new-style packages incompatible with the old style.
I assume that mixing an old-style package into an environment that uses the new style would break all the new-style package (as the __init__.py with the pkg_resources hook would hide the native packages)?
Is there somebody around here who already went through the migration and can share his experiences? @jaraco comes to mind, as he has a similar approach for his packages, sharing the jaraco.* namespace?
See this comment which points to some good resources for namespace packages. In particular, the sample-namespace-packages project has a table that’s been meticulously constructed to show the compatibility options.
There are many factors at play. I’ve found that even that table is inadequate to capture all of the nuances. For example, it focuses primarily on installing the namespace package to a single directory on the sys.path. If some packages are installed to separate directories, all bets are off. Also, the results depend on how the packages are installed. If installed by pip, the behaviors are pretty well documented, but your mileage may vary if using another installer (especially a deprecated one or a bespoke one; uv is probably okay).
One nice thing about pip is it will omit the __init__.py file for pkg_resources-declared namespaces, allowing pkgutil-style packages to co-exist (in some cases).
According to that table, packages with different modes may be able to co-exist as long as you’re not using editable installs. It’s also conceivable that modern editable installs are more robust than legacy ones.
The way I did the migration for jaraco packages, I first migrated from pkg_resources to pkgutil in 2019 (jaraco.functools) and then two years later made the transition to native namespace packages, after dropping support for Python 2.
I had very little issue with this approach, as I performed the change on all jaraco.* projects at once, and I don’t recall any issues, but that’s also probably because these projects have modest adoption and where possible, I encouraged to use the latest version (not pin dependencies).
So you may have better luck with a transitional approach, first from pkg_resources to pkgutil then to native namespace packages. If I were you, I’d try to migrate to native namespace packages first with one package and see how it goes and what breaks.
Thank you so much for the pointers. This looks like we can actually mix pkg_resources and PEP 420, so it should be possible to stagger the migration. Thanks! Will try it out soon.