PEP 741: Python Configuration C API (second version)

[vstinner (Victor Stinner)]

@eric.snow

• who are we trying to help with this API?
• how did they let us know their needs (and that the status quo wasn’t good enough)?

I’m collecting feedback from users: see previous comments.

@eric.snow

• why does this need to be part of the limited API?

There are use cases to embed Python with the stable ABI. The main use case is to select which Python version is select which Python version is used when running an application, without having to rebuild the application.

@eric.snow

what is the purpose of exposing config values after initialization has finished?

Reading the current configuration at runtime was asked multiple times by Marc-André Lemburg.

Marc-André Lemburg wrote (emphasis is mine):

I also could not find a public PyConfig API for reading the current config. Why is that ?
There’s lots of useful information in the PyConfig struct, which extensions could use as well and won’t be available elsewhere anymore once the global vars are gone.

Marc-André Lemburg also wrote (emphasis is mine):

@malemburg

FWIW and as a real use case: For eGenix PyRun, I need a way to access some of the Python config settings in Python and after having initialized the interpreter (essentially, all settings which can be set from the Python command line or via env vars). Both for reading and writing.

Before 3.11, I could do this via accessing the global config variables from C. Since 3.11, this no longer works.

Marc-André also asked to have a way to modify them at runtime, but this feature is not part of the PEP scope. When it was discussed later, it was said that it’s better to modify Python attributes such as sys.path, rather than the “configuration” (such as config.module_search_paths).

Marc-André wants to update config.verbose but again, it’s not part of the PEP scope. I didn’t see other people asking for such feature so far. By the way, sys.flags.verbose is read-only.

@eric.snow

each phase should have a distinct config this is used exclusively as input to the corresponding init function

PyInitConfig API can be used for that, no? It’s an opaque structure, we have a lot of freedom to decide what it contains and how it behaves.

If you want PyInitConfig_SetInt() to behave differently depending if you initialize the Python main interpreter, or a sub-interpreter, we can have different “Create” functions, such as:

• PyInitConfig_CreatePythonInterpreter().
• PyInitConfig_CreateIsolatedInterpreter().

For example, we can say that setting a preconfiguration option, such as allocator, would fail with an error.

@eric.snow

there isn’t a strict 1-to-1 mapping between config fields and runtime state fields

PyConfig_Get() does not only read PyInterpeterState.config, but also sys attributes: there is no 1to1 mapping. It’s specified in the latest version of PEP 741 (PyConfig_Get() spec).

@eric.snow

once an init phase finishes, that config becomes irrelevant; the runtime never uses it

Many PyInterpeterState.config members are used at runtime and there is no other copy. Even if the implementation changes to remove that member, PyConfig_Get() would still remain relevant, no?

@eric.snow

we would keep a read-only copy of the config around only for diagnostic purposes (as a snapshot of how the runtime was initialized)

What would be your use case for that? With no concrete use case, there risk is that it would only waste memory. Which API would give access to it?

Currently, we don’t keep copies of PyConfig.module_search_paths or sys.path when they are modified.

@eric.snow

many config values in PyConfig are relevant for the global runtime but not for individual interpreters (we shouldn’t store a PyConfig on each PyInterpreterState)

PyInitConfig is opaque and so is a better API, since some members can be removed/ignored for sub-interpreters.

Note: PEP 741 “PyInitConfig” API does not deprecate PEP 587 PyConfig API.