What information is useful to know statically about an interpreter?

steve.dower · April 11, 2023, 9:46pm

Nope, the ABI and CRT semantics are different. Windows never used a d in the ABI tag though, because it had the suffix first, but since it was officially taken away we can’t switch to it now either.

FFY00 · May 2, 2023, 6:52pm

FYI, I am proposing a new sysconfig API, which plays a bit with this.

The current focus is adding a new API, but in a next step I’d like to propose shipping a static file detailing the build, etc., details as part of the Python installation. And afterwards, allow the new sysconfig API to use it as a data source (instead of relying on the Makefile variables).

github.com/python/cpython

New sysconfig API 🤞 (meta-issue)

opened 06:48PM - 12 Apr 23 UTC

FFY00

type-feature topic-sysconfig

# Feature or enhancement Provide new `syconfig` API that is aligned to how th…ings have evolved. Fixes GH-99942 Fixes GH-99560  # Pitch The `sysconfig` module was added in Python 3.2, over 10 years ago, and though there were some minor changes and additions, its API stayed mostly the same. Since then things have evolved, with new platform support (eg. WASM) and features being added, and we are now at a point where the current design is not great and has been the source of several issues, especially since the `distutils` deprecation (see GH-99942 for eg.). Changing the existing API to better represent modern Python installations would cause a lot of breakage on existing code. Considering that, on top of several of its shortcomings, we think the best step at the moment would be to introduce a new API and mark part of the existing one as either deprecated, or pending deprecation, depending how development goes and on feedback from the impacted parties. Here are the main points we'd like to improve on: 1) Users currently rely on [`sysconfig.get_config_vars`](https://docs.python.org/3/library/sysconfig.html) for **a lot** of things On Posix, `sysconfig.get_config_vars` exports all the variables present in the installation's `Makefile`, and on Windows, it exports a handful of selected variables. https://github.com/python/cpython/blob/52f96d3ea39fea4c16e26b7b10bd2db09726bd7c/Lib/sysconfig.py#L540-L556 There are several issues with this, the main being that the exported variables are not documented, so there are no guarantees about what is information is gonna be available, or the variable names. While this is a good escape-hatch mechanism, it is a very poor option for the intended interface for key information. We should be exporting that information in a proper API. 2) The installation paths model is outdated There are a couple of issues and things to improve here here: - A scheme should be able to not provide non-critial paths (`scripts` for eg.) - We should introduce the concept of writable and read-only paths (users should not install to any path in the installed base, like `include`, for eg.) - We should introduce the concept of active location schemes (eg. `posix_user` should not be available on virtual environments) - This would extend to vendor schemes outside the default environment, if we ever get around to standardize that **Discussion on this should go to GH-103482** 3) Cross-compilation support is very poor Cross-compilation at the moment is done by patching several aspects of Python, with the `sysconfig` module being the main thing. We provide several undocumented features to support this, like the allowing the project base path and sysconfig data module name to be overwritten via the `_PYTHON_PROJECT_BASE` and `_PYTHON_SYSCONFIGDATA_NAME` environment variables, respectively. We believe one of the reasons that plays into users choosing to support cross-compilation by monkey patching Python, instead of directly adding cross-compilation support to the 3rd party building code, is because with the current `sysconfig` design, a lot of custom code would have to be added to properly support cross builds. Our proposal to solve this is to make the new API standalone, meaning it would provide all the information users may need for binary interfacing (eg. building extension modules) and similar applications. On top of this, we'd shift the design from using module functions to data holder objects, meaning all the relevant details of the Python installation would be represented by just a couple objects. Users that adopt this new API should then be able to support the cross build by simply changing the data objects for ones that represent the target installation. This has the drawback that we'd be re-exporting information already present in other places (eg. `sys.version`, `sys.implementation`), but we believe it is worth. ###### Future work *In a future step* we would also like to get Python to provide a static file detailing all the information required by the new `sysconfig` API. With this, we could then allow the new API to use an external installation/interpreter config file as the data source, making cross builds even simpler to support. See the discussion at https://discuss.python.org/t/what-information-is-useful-to-know-statically-about-an-interpreter/25563 for examples of users that would benefit from this feature. ## Migration plan To help migration, and to make these improvements available on older Python versions, we plan to provide a backport package on PyPI, similarly with we already do with [`importlib_metadata`](https://github.com/python/importlib_metadata) (for [`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html)), [`importlib_resources`](https://github.com/python/importlib_resources) (for [`importlib.resources`](https://docs.python.org/3/library/importlib.resources.html)), and [`typing_extensions`](https://github.com/python/typing_extensions) (for [`typing`](https://docs.python.org/3/library/typing.html)). ## Sub-issues The proposal is quite big, and may require extensive discussions with different parties, so I am splitting it into a couple separate issues. - Install paths (GH-103481) - Build information - Compilation (GH-103482) - Compatibility (GH-103483) # Previous discussion This has been mentioned in a couple places, here are some of the most relevant: https://discuss.python.org/t/building-extensions-modules-in-a-post-distutils-world/23938 https://discuss.python.org/t/sysconfig-should-provide-more-information-to-help-packaging/22950 https://discuss.python.org/t/pep-582-python-local-packages-directory/963/391 https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/28 https://github.com/python/cpython/issues/102522#issuecomment-1497727808 https://github.com/PyO3/maturin/blob/5d5b96a9974eac26b8cdf601cd2faf64f4999de9/src/python_interpreter/sysconfig-freebsd.json

FFY00 · August 15, 2023, 8:18pm

Hey, I opened an issue to propose shipping an installation-specific description file, which should help with some of the issues discussed here.

Please note this proposal is only for an installation description file, and not an environment.

Here’s the issue if anyone wants to add more to it and/or follow the progress.

github.com/python/cpython

Install a static installation description file as part of the Python installation

opened 11:30PM - 14 Aug 23 UTC

FFY00

type-feature

# Feature or enhancement Ship a static file that describes various aspects of the installation as part of the Python installation. # Pitch Shipping such a file would make it much easier to introspect a Python installation without having to run the interpreter. There are many use-cases for this, but some key would be for eg. Python launchers, and cross-compilation tooling. Information we could provide: - Python language version - Python implementation (inc. version) - Interpreter path - Standard library path - Interpreter architecture - Python ABI - Tag ([PEP 3149](https://peps.python.org/pep-3149) on Posix) - Build flags (debug, trace refs) - Static `libpython` (available?) - Name - Location - Shared `libpython` (available?) - Name (standard library) - Major-only pin library name (eg. `libpython3.so`) - Location - C API - Hex version - Include paths - Required C flags - Supported native module file extensions - Interpreter-specific - Stable ABI (incomplete table, just an initial proposal) Note: This issue specifically targets a descriptor file for the Python **installation**, not a Python **environment**, so paths are out of scope. # Previous discussion https://discuss.python.org/t/what-information-is-useful-to-know-statically-about-an-interpreter/25563. (cc @brettcannon)