Use cases for the `platstdlib` path in `sysconfig`?

Core Development
1

I noticed recently that the values that sysconfig reports for the platstdlib paths have been wrong for a long time (and have potentially never been correct): sysconfig values for `platstdlib` do not appear to be accurate · Issue #119845 · python/cpython · GitHub

Given that we haven’t previously received any bug reports about this, @gpshead suggested we may want to deprecate and remove the platstdlib entries rather than fixing them.

Does anyone have concrete use cases for the platstdlib paths in sysconfig?

2

Here’s matches for "\bplatstdlib\b" in the top 8k PyPI packages (downloaded 2024-04-14):

Found 55 matching lines in 22 projects 
❯ python3 ~/github/misc/cpython/search_pypi_top.py -q . "\bplatstdlib\b"
./ddtrace-2.8.0.tar.gz: ddtrace-2.8.0/ddtrace/internal/packages.py: platstdlib_path = Path(sysconfig.get_path("platstdlib")).resolve()
./ddtrace-2.8.0.tar.gz: ddtrace-2.8.0/ddtrace/profiling/exporter/pprof.pyx: platstdlib_path = sysconfig.get_path("platstdlib")
./ddtrace-2.8.0.tar.gz: ddtrace-2.8.0/ddtrace/profiling/exporter/pprof.pyx: "name": "platstdlib",
./ddtrace-2.8.0.tar.gz: ddtrace-2.8.0/tests/profiling/exporter/test_pprof.py: {"kind": "standard library", "name": "platstdlib", "version": platform.python_version()},
./debugpy-1.8.1.zip: debugpy-1.8.1/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_filtering.py: for path_name in set(('stdlib', 'platstdlib', 'purelib', 'platlib')) & set(sysconfig.get_path_names()):
./clvm_rs-0.6.1.tar.gz: clvm_rs-0.6.1/venv/lib/python3.7/site-packages/pip/_internal/req/req_uninstall.py: for p in {sysconfig.get_path("stdlib"), sysconfig.get_path("platstdlib")}
./astroid-3.1.0.tar.gz: astroid-3.1.0/astroid/modutils.py: # TODO: Adding `platstdlib` is a fix for a workaround in virtualenv. At some point we should
./astroid-3.1.0.tar.gz: astroid-3.1.0/astroid/modutils.py: STD_LIB_DIRS = {sysconfig.get_path("stdlib"), sysconfig.get_path("platstdlib")}
./astroid-3.1.0.tar.gz: astroid-3.1.0/astroid/modutils.py: STD_LIB_DIRS.add(str(Path(sysconfig.get_path("platstdlib")).parent / "lib_pypy"))
./astroid-3.1.0.tar.gz: astroid-3.1.0/astroid/modutils.py: str(Path(sysconfig.get_path("platstdlib")).parent / "lib-python/3")
./autoflake-2.3.1.tar.gz: autoflake-2.3.1/autoflake.py: path_names = ("stdlib", "platstdlib")
./loguru-0.7.2.tar.gz: loguru-0.7.2/loguru/_better_exceptions.py: names = ["stdlib", "platstdlib", "platlib", "purelib"]
./mod_wsgi-5.0.0.tar.gz: mod_wsgi-5.0.0/configure: stdout.write(sysconfig.get_path("platstdlib") +"/config")'`
./mod_wsgi-5.0.0.tar.gz: mod_wsgi-5.0.0/configure.ac: stdout.write(sysconfig.get_path("platstdlib") +"/config")'`
./mod_wsgi-5.0.0.tar.gz: mod_wsgi-5.0.0/setup.py: PYTHON_CFGDIR =  get_python_lib('platstdlib') + '/config'
./hupper-1.12.1.tar.gz: hupper-1.12.1/src/hupper/worker.py: for name in {'stdlib', 'platstdlib', 'platlib', 'purelib'}:
./pdm-2.14.0.tar.gz: pdm-2.14.0/src/pdm/pep582/sitecustomize.py: "platstdlib": "{pep582_base}/lib",
./pdm-2.14.0.tar.gz: pdm-2.14.0/src/pdm/utils.py: "platstdlib": "{pep582_base}/lib",
./pdoc-14.4.0.tar.gz: pdoc-14.4.0/pdoc/extract.py: #    filename.startswith(sysconfig.get_path("platstdlib"))
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_internal/req/req_uninstall.py: sysconfig.get_path("platstdlib")}
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {platbase}/lib/python{py_version_short}
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {base}/lib/python
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {base}/Lib
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {base}/Lib
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {userbase}/lib/python{py_version_short}
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {userbase}/Python{py_version_nodot}
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {userbase}/lib/python{py_version_short}
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.cfg: platstdlib = {userbase}/lib/python
./pex-2.3.1.tar.gz: pex-2.3.1/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.py: vars['BINLIBDEST'] = get_path('platstdlib')
./pycln-2.4.0.tar.gz: pycln-2.4.0/pycln/utils/pathu.py: {sysconfig.get_path("platstdlib"), sysconfig.get_path("stdlib")}
./pydevd-3.0.3.tar.gz: pydevd-3.0.3/_pydevd_bundle/pydevd_filtering.py: for path_name in set(('stdlib', 'platstdlib', 'purelib', 'platlib')) & set(sysconfig.get_path_names()):
./pigar-2.1.4.tar.gz: pigar-2.1.4/pigar/_vendor/pip/_internal/req/req_uninstall.py: for p in {sysconfig.get_path("stdlib"), sysconfig.get_path("platstdlib")}
./ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/__main__.py: for path_name in {'stdlib', 'platstdlib', 'purelib', 'platlib'} & set(
./ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_filtering.py: for path_name in set(('stdlib', 'platstdlib', 'purelib', 'platlib')) & set(sysconfig.get_path_names()):
./pip-24.0.tar.gz: pip-24.0/src/pip/_internal/req/req_uninstall.py: for p in {sysconfig.get_path("stdlib"), sysconfig.get_path("platstdlib")}
./pipenv-2023.12.1.tar.gz: pipenv-2023.12.1/pipenv/environment.py: 'platstdlib': '/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7',
./pipenv-2023.12.1.tar.gz: pipenv-2023.12.1/pipenv/environment.py: for key in ("purelib", "platlib", "stdlib", "platstdlib")
./pipenv-2023.12.1.tar.gz: pipenv-2023.12.1/pipenv/environment.py: "platstdlib",
./pipenv-2023.12.1.tar.gz: pipenv-2023.12.1/pipenv/environment.py: for key in ("platlib", "platstdlib", "stdlib"):
./pipenv-2023.12.1.tar.gz: pipenv-2023.12.1/pipenv/patched/pip/_internal/req/req_uninstall.py: for p in {sysconfig.get_path("stdlib"), sysconfig.get_path("platstdlib")}
./pyLDAvis-3.4.1.tar.gz: pyLDAvis-3.4.1/pyLDAvis/lib/python3.11/site-packages/pip/_internal/req/req_uninstall.py: for p in {sysconfig.get_path("stdlib"), sysconfig.get_path("platstdlib")}
./pyLDAvis-3.4.1.tar.gz: pyLDAvis-3.4.1/pyLDAvis/lib/python3.11/site-packages/setuptools/_distutils/command/_framework_compat.py: platstdlib='{platbase}/{platlibdir}/python{py_version_short}',
./robocorp_log-2.9.2.tar.gz: robocorp_log-2.9.2/src/robocorp/log/_rewrite_filtering.py: for path_name in set(("stdlib", "platstdlib", "purelib", "platlib")) & set(
./scikit_build_core-0.8.2.tar.gz: scikit_build_core-0.8.2/src/scikit_build_core/resources/find_python/FindPython.cmake: or else ``sysconfig.get_path('platstdlib')``.
./scikit_build_core-0.8.2.tar.gz: scikit_build_core-0.8.2/src/scikit_build_core/resources/find_python/FindPython3.cmake: or else ``sysconfig.get_path('platstdlib')``.
./scikit_build_core-0.8.2.tar.gz: scikit_build_core-0.8.2/src/scikit_build_core/resources/find_python/FindPython/Support.cmake: "import sys\ntry:\n   from distutils import sysconfig\n   sys.stdout.write(';'.join([sysconfig.get_python_lib(plat_specific=False,standard_lib=True),sysconfig.get_python_lib(plat_specific=True,standard_lib=True),sysconfig.get_python_lib(plat_specific=False,standard_lib=False),sysconfig.get_python_lib(plat_specific=True,standard_lib=False)]))\nexcept Exception:\n   import sysconfig\n   sys.stdout.write(';'.join([sysconfig.get_path('stdlib'),sysconfig.get_path('platstdlib'),sysconfig.get_path('purelib'),sysconfig.get_path('platlib')]))"
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/src/virtualenv/create/describe.py: self._stdlib_platform = Path(self.interpreter.sysconfig_path("platstdlib", config_var=self._config_vars))
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/src/virtualenv/discovery/py_info.py: self.system_stdlib_platform = self.sysconfig_path("platstdlib", confs)
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/tests/unit/discovery/py_info/test_py_info.py: "platstdlib": "{platbase}/lib/python{py_version_short}",
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/tests/unit/discovery/py_info/test_py_info.py: "platstdlib": "{platbase}/{platlibdir}/python{py_version_short}",
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/tests/unit/discovery/py_info/test_py_info.py: "platstdlib": "{base}/lib/python",
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/tests/unit/discovery/py_info/test_py_info.py: "platstdlib": "{base}/Lib",
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/tests/unit/discovery/py_info/test_py_info.py: "platstdlib": "{platbase}/{platlibdir}/python{py_version_short}",
./virtualenv-20.25.1.tar.gz: virtualenv-20.25.1/tests/unit/discovery/py_info/test_py_info.py: "platstdlib": "{platbase}/{platlibdir}/python{py_version_short}",
./setuptools-69.5.1.tar.gz: setuptools-69.5.1/setuptools/_distutils/command/_framework_compat.py: platstdlib='{platbase}/{platlibdir}/python{py_version_short}',

Time: 0:00:58.930139
Found 55 matching lines in 22 projects
3

I honestly don’t even know what it’s meant to be for. I got a hint somewhere that it’s for the platform-specific parts of the stdlib (i.e. extension modules), but haven’t really noticed it being used that way.

1 Like
4

Yeah, I think it’s meant to be purely descriptive of how the stdlib is laid out, acknowledging that the pure Python files and the extension modules may be stored separately. I genuinely don’t know why anything would ever both care about that location and rely on sysconfig for the info:

  • the interpreter itself works out this value in getpath.py, it doesn’t get it from sysconfig
  • if distro packagers want to modify it, they’ll do that in getpath.py, not sysconfig
  • anything outside the interpreter core would have to get the info from sysconfig, since that’s the only public API, but what would be the use cases? Maybe to exclude the stdlib from a sys.path scan of some kind?

If getpath.py had a convenient way to export this path for sysconfig to pick up so we weren’t saving it in two places that would be one thing, but it doesn’t, so the info gets entered in two places instead.

I still need to go through @hugovk’s full list, but the first few I looked at were just expected values captured for testing purposes.

5

This is the general model, though. sysconfig is meant as a Python-accessible reflection of what we really do, it’s not the source of anything.

Initialization configs are now readable at runtime,[1] though, so sysconfig could read some live values rather than the hard-coded templates. Or initialization could copy paths into sysconfig like it already does for copying settings into sys.

  1. And writable… though that’s generally considered an unfortunate misuse. The config structs should be considered to belong to the host, not the runtime. ↩︎

1 Like
6

And that would eliminate the current maintainability issue.

If the layouts were all defined at the end of getpath.py using the relevant variables rather than hard coding relative paths the way sysconfig currently does, the C code would just need to transfer those references, it wouldn’t need to reformat them.

We may want to stash them in a private sys variable for sysconfig to read to avoid having to eagerly load sysconfig, though.

1 Like