Confused about interaction between pkg_resources namespace packages, pth files, and wheels

At work, I’ve inherited a collection of packages in the lazr namespace. These are fairly old, and use the pkg_resources style (actually they currently follow the old and bad recommendation to try pkg_resources and fall back to pkgutil, which I should probably fix, but I don’t think that’s the issue here). I’d love to migrate to a newer approach, but the Python Packaging User Guide says it isn’t advisable to try to migrate an existing package.

Historically we’ve only uploaded sdists of these packages to PyPI, but I’d like to start using pyproject-build and uploading wheels as well. However, when I try this with lazr.restfulclient, I get some results which seem concerning for potential compatibility of the resulting wheel:

* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Getting dependencies for wheel...
running egg_info
writing src/lazr.restfulclient.egg-info/PKG-INFO
writing dependency_links to src/lazr.restfulclient.egg-info/dependency_links.txt
writing namespace_packages to src/lazr.restfulclient.egg-info/namespace_packages.txt
writing requirements to src/lazr.restfulclient.egg-info/requires.txt
writing top-level names to src/lazr.restfulclient.egg-info/top_level.txt
reading manifest file 'src/lazr.restfulclient.egg-info/SOURCES.txt'
adding license file 'COPYING.txt'
writing manifest file 'src/lazr.restfulclient.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/lazr
copying src/lazr/__init__.py -> build/lib/lazr
creating build/lib/lazr/restfulclient
copying src/lazr/restfulclient/_browser.py -> build/lib/lazr/restfulclient
copying src/lazr/restfulclient/errors.py -> build/lib/lazr/restfulclient
copying src/lazr/restfulclient/resource.py -> build/lib/lazr/restfulclient
copying src/lazr/restfulclient/__init__.py -> build/lib/lazr/restfulclient
copying src/lazr/restfulclient/_json.py -> build/lib/lazr/restfulclient
creating build/lib/lazr/restfulclient/tests
copying src/lazr/restfulclient/tests/test_oauth.py -> build/lib/lazr/restfulclient/tests
copying src/lazr/restfulclient/tests/test_atomicfilecache.py -> build/lib/lazr/restfulclient/tests
copying src/lazr/restfulclient/tests/test_error.py -> build/lib/lazr/restfulclient/tests
copying src/lazr/restfulclient/tests/example.py -> build/lib/lazr/restfulclient/tests
copying src/lazr/restfulclient/tests/__init__.py -> build/lib/lazr/restfulclient/tests
copying src/lazr/restfulclient/tests/test_docs.py -> build/lib/lazr/restfulclient/tests
creating build/lib/lazr/restfulclient/authorize
copying src/lazr/restfulclient/authorize/oauth.py -> build/lib/lazr/restfulclient/authorize
copying src/lazr/restfulclient/authorize/__init__.py -> build/lib/lazr/restfulclient/authorize
running egg_info
writing src/lazr.restfulclient.egg-info/PKG-INFO
writing dependency_links to src/lazr.restfulclient.egg-info/dependency_links.txt
writing namespace_packages to src/lazr.restfulclient.egg-info/namespace_packages.txt
writing requirements to src/lazr.restfulclient.egg-info/requires.txt
writing top-level names to src/lazr.restfulclient.egg-info/top_level.txt
reading manifest file 'src/lazr.restfulclient.egg-info/SOURCES.txt'
adding license file 'COPYING.txt'
writing manifest file 'src/lazr.restfulclient.egg-info/SOURCES.txt'
creating build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/Makefile -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/NEWS.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/authorizer.standalone.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/caching.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/collections.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/entries.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/hosted-files.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/index.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/operations.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/retry.standalone.rst -> build/lib/lazr/restfulclient/docs
copying src/lazr/restfulclient/docs/toplevel.rst -> build/lib/lazr/restfulclient/docs
/tmp/build-env-qwg9p_ta/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
Skipping installation of build/bdist.linux-x86_64/wheel/lazr/__init__.py (namespace package)
copying lazr/restfulclient/_browser.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient
copying lazr/restfulclient/errors.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient
copying lazr/restfulclient/resource.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient
copying lazr/restfulclient/__init__.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient
copying lazr/restfulclient/_json.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient
copying lazr/restfulclient/docs/collections.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/operations.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/authorizer.standalone.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/entries.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/NEWS.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/Makefile -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/toplevel.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/retry.standalone.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/index.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/hosted-files.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/docs/caching.rst -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/docs
copying lazr/restfulclient/tests/test_oauth.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/tests
copying lazr/restfulclient/tests/test_atomicfilecache.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/tests
copying lazr/restfulclient/tests/test_error.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/tests
copying lazr/restfulclient/tests/example.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/tests
copying lazr/restfulclient/tests/__init__.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/tests
copying lazr/restfulclient/tests/test_docs.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/tests
copying lazr/restfulclient/authorize/oauth.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/authorize
copying lazr/restfulclient/authorize/__init__.py -> build/bdist.linux-x86_64/wheel/lazr/restfulclient/authorize
running install_egg_info
Copying src/lazr.restfulclient.egg-info to build/bdist.linux-x86_64/wheel/lazr.restfulclient-0.14.5-py3.8.egg-info
Installing build/bdist.linux-x86_64/wheel/lazr.restfulclient-0.14.5-py3.8-nspkg.pth
running install_scripts
adding license file "COPYING.txt" (matched pattern "COPYING*")
creating build/bdist.linux-x86_64/wheel/lazr.restfulclient-0.14.5.dist-info/WHEEL
creating '/home/cjwatson/src/canonical/lazr.restfulclient/lazr.restfulclient/dist/tmpo_vwcvbj/lazr.restfulclient-0.14.5-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'lazr.restfulclient-0.14.5-py3.8-nspkg.pth'
adding 'lazr/restfulclient/__init__.py'
adding 'lazr/restfulclient/_browser.py'
adding 'lazr/restfulclient/_json.py'
adding 'lazr/restfulclient/errors.py'
adding 'lazr/restfulclient/resource.py'
adding 'lazr/restfulclient/authorize/__init__.py'
adding 'lazr/restfulclient/authorize/oauth.py'
adding 'lazr/restfulclient/docs/Makefile'
adding 'lazr/restfulclient/docs/NEWS.rst'
adding 'lazr/restfulclient/docs/authorizer.standalone.rst'
adding 'lazr/restfulclient/docs/caching.rst'
adding 'lazr/restfulclient/docs/collections.rst'
adding 'lazr/restfulclient/docs/entries.rst'
adding 'lazr/restfulclient/docs/hosted-files.rst'
adding 'lazr/restfulclient/docs/index.rst'
adding 'lazr/restfulclient/docs/operations.rst'
adding 'lazr/restfulclient/docs/retry.standalone.rst'
adding 'lazr/restfulclient/docs/toplevel.rst'
adding 'lazr/restfulclient/tests/__init__.py'
adding 'lazr/restfulclient/tests/example.py'
adding 'lazr/restfulclient/tests/test_atomicfilecache.py'
adding 'lazr/restfulclient/tests/test_docs.py'
adding 'lazr/restfulclient/tests/test_error.py'
adding 'lazr/restfulclient/tests/test_oauth.py'
adding 'lazr.restfulclient-0.14.5.dist-info/COPYING.txt'
adding 'lazr.restfulclient-0.14.5.dist-info/METADATA'
adding 'lazr.restfulclient-0.14.5.dist-info/WHEEL'
adding 'lazr.restfulclient-0.14.5.dist-info/namespace_packages.txt'
adding 'lazr.restfulclient-0.14.5.dist-info/top_level.txt'
adding 'lazr.restfulclient-0.14.5.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel

So this is creating a wheel that’s just labelled py3, but it includes a -py3.8-nspkg.pth file. This seems weird. It does seem to work across various Python versions (even if I make the wheel universal and try 2.7), I guess because the actual content of the .pth file isn’t very version-dependent; but is it guaranteed to work, and do I have any practical alternative that would be cleaner?

That should be fine, as the name of a .pth file isn’t important, except to ensure uniqueness. I assume the -py3.8 is some sort of thing pkg_resources adds.

To get to something cleaner would probably mean upgrading the build stuff to a more up to date approach, but as you say that’s a reasonable amount of work, so it may be better to do that as a separate exercise.

1 Like

OK, fair enough then. Thanks for answering!