PEP 741: Python Configuration C API (second version)

[vstinner (Victor Stinner)]

Hi,

I wrote a major update of my PEP 741 “Python Configuration C API (second version)” to address most, if not all, requests in the previous discussion.

=> Read the updated PEP 741 <=

See the Commit of the PEP 741 update.

Changes (incomplete list):

• PyInitConfig_SetStr() and PyInitConfig_SetStrList() now expect UTF-8 encoded string, instead of strings encoded to the locale encoding.

• Add PyInitConfig_SetStrLocale() and PyInitConfig_SetStrLocaleList() functions which expect null-terminated strings encoded to the locale encoding.

• Calling PyInitConfig_SetStrLocale() and PyInitConfig_SetStrLocaleList() now requires first to preinitialize Python.

• Setting a preconfiguration option, such as "allocator", after Python preconfiguration now fails with an error.

• Add functions to get PyInitConfig options:

  • PyInitConfig_GetInt()
  • PyInitConfig_GetStr()
  • PyInitConfig_GetWStr()
  • PyInitConfig_GetStrList()
  • PyInitConfig_GetWStrList()
  • PyInitConfig_FreeStrList()
  • PyInitConfig_FreeWStrList()

• Remove PyInitConfig_GetExitCode() and PyInitConfig_Exception() functions: simplify error handling.

• Document that PyConfig_Get() cannot be called before Python initialization, nor after Python finalization.

• PyConfig_Get(): Document which values are read from the sys module.

• Remove support for custom configuration options. I’m no longer convinced that third-party project need/want pass their custom options. Apparently, Python is the main and only consumer of Python configuration options. I prefer to make the API as small as possible for now.

• Add PyConfig_Keys() function: get all configuration option names as a tuple. I plan to sort them for convenience.

• Deprecating and removing configuration options is out of the scope of the PEP and should be discussed on a case by case basis.

• Give multiple examples of applications, libraries and utilities embedding Python.

• Quote developers asking for a stable ABI.

---

I was asked to fully remove the Python preinitialization. Well, I almost did it: only PyInitConfig_SetStrLocale() and PyInitConfig_SetStrLocaleList() need it. If you use other APIs using UTF-8, you should no longer have to bother with the preinitialization. Well, if you only speak ASCII or UTF-8, you’re good!

PyInitConfig_SetStrLocale() is the only correct way to pass a string encoded to the locale encoding. For example, to pass main() char **argv arguments to set the configuration option "argv".

[steve.dower]

Many of these changes look great, nice work.

Unless we find a way to completely remove preinitialization (which I still hope we can), I’d like to see it explicitly marked as optional. Possibly the best way to do that is to change it into a “set current locale” or “infer current locale” function instead, and then require it be called before the Set*Locale functions.

The quotes under “Usage of a stable ABI” are mostly asking for the API to remain unchanged during bugfix releases. I think the title of the section should be changed to reflect this, and the quotes should not be used to claim that there’s support for adding anything to the Stable ABI.

I’m still not entirely convinced by the native “Get” functions (other than PyConfig_Get). The struct can be write-only - there’s no way to end up with one that you haven’t filled in yourself (other than the “InitializePythonConfig” which I’ll get to later… short version: the contents of that struct might be usefully accessible, but should not be in any way stable, even between bugfix versions).

PyConfig_Get doesn’t need to say where the values are coming from. Like @eric.snow, I think the config should be used to initialise and then any values that need to be kept should be moved into *State structures (including all the ones mentioned in the PEP - the fact that they are read/written directly to the Config struct should be a bug, and we should just move them to fields on InterpreterState). Having PyConfig_Get read from the sys module wherever it can is perfect - it also means one day we can move those options to be stored somewhere else without having to change any APIs (provided we update the existing ones).

The PyConfig_Get description should say that you need to be holding the GIL, and that it will read settings from the current active [sub]interpreter.

Regarding Configuration Options, I don’t like having exhaustive lists of options like this in the PEP. People are too likely to read it as documentation and not realise that it’s not up to date (the headers we have now help, but still better to tweak the phrasing in the text). For example, we could list a few as examples, maybe a few counter-examples (options we shouldn’t add), and then set a policy for adding/removing them (e.g. can only be added in a new feature release; requires normal deprecation period/stable ABI deprecation period for removal or change of semantics). Then create a doc page that lists them all. This is how we handle audit events.

Can we just move “coerce C locale” out of the library entirely? That seems like a perfect candidate for something that we want python.c to do, but really shouldn’t be doing at all when embedded. (The legacy Windows FS encoding should probably be completely removed by now, I’ve not heard of anyone needing to use it. It shouldn’t need to be in preinitialization anyway - it affects how bytes objects are decoded by fsdecode, and we shouldn’t be using them at all during initialization.)

Py_ExitWithInitConfig exits the entire process. We shouldn’t provide embedders this API. They can exit their own process when they’re ready.

The examples are great! Realistic, sufficient, reusable. But I think they illustrate that reading back from the PyInitConfig object isn’t important for the intended use, which is to put settings into the config.

I still don’t want to “stabilise” all the inferencing we do to create the Python config, and would rather see that totally removed (and if not totally, then definitely removed from the stable ABI). Users who want stability need to set those options themselves, and if they want to match a normal Python interactive environment, they can read the environment variables themselves.

And I still don’t think any of this belongs in the limited API/stable ABI. That’s an ongoing discussion elsewhere, so no need to focus on it here. But I want to make sure my objection is registered, and I’ll continue to be -1 on this entire proposal until I’m either persuaded that (a) we don’t need to keep the stable ABI stable, (b) that we won’t change any aspect of initialization/runtime into the future, or (c) it’s updated to not add anything to the stable ABI.

[vstinner (Victor Stinner)]

About the PyInitConfig_AddModule() function, see C-API for initializing statically linked extension modules discussion which asks for such API.

[milianw (Milian Wolff)]

Hello all, Victor asked me to share my past experience with the stable Python API wrt. integration of Python as a scripting language/console into a C++ GUI application. So here goes my reply that I already sent to him by email previously:

A bit of background first: Our application is a large complex C++ code base
with lots of dependencies targeting all three major desktop platforms. We use
Python both externally (for headless data processing, e.g. UMAP) as well as
internally for an integrated python console that allows biologists to run
python scripts with access to some API that allows automation of tasks in the
C++ GUI.

Originally, we hoped to be able to use the stable python ABI to allow
biologists to “bring your own python”. The idea was that they probably have a
custom set of python libraries and code that they would like to continue
using. Our integrated API - so we thought - was a tiny addition that should
work with any Python out there, so we used the stable ABI.

This turned out to be a dead end, and I believe we can (should?) now use the
non-stable ABI of python. Allowing end users to BYO Python caused far too much
setup problems and support issues for us that it was not worth it in the end.
Instead, we now rather want to ship a custom Python with a custom prefix that
they can pip install custom libraries into as needed.

The problems we faced are not directly related to the stable ABI - quite the
contrary. Rather, it was due to thirdparty python libraries that we shipped
which themselves are not compatible across python version increments. E.g. for
the integrated console we use qtconsole/jupyter, which worked in an archaic
version with python 3.9 but requires newer versions for python 3.11+.

The ton of dependencies pulled in by UMAP was even worse, with numba and
pydnndescent and llvmlite often taking months to support newer python
versions.

Finally, we faced tons of issues when we tried to combine different python
prefixes - e.g. the user has Python 3.X and we ship Python 3.Y, both having a
prefix of their own with some PIP packages installed. This often led to
incompatibilities, when e.g. some package dropped (or added) a third party
dependency which was then picked up in another prefix.

Overall, I can safely say that integrating Python as a scripting language into
an application in a meaningful way (i.e. not just the interpreter, but also
some thirdparty/PIP packages) is probably one of most time draining
experiences I have had in my professional career.

[pmp-p (Paul P.)]

I cannot agree more, it is the same story everywhere/everytime CPython must be embedded. I maintened a runtime+ecosystem for Android 4.4+ for some time ( in order more comfortably use Panda3D standalone than with Kivy) ,
patching CPython and making a CI for it was ok.

But i had to give up because now each year i had to recompile every known modules often : this is not sustainable for one individual ( and sept 2019 marked the end of hobby time spent on that for me for obvious reaons anyway).

So i dropped the 4 android arch to only go webassembly ( emscripten )

But same (hard and boring) problem always : have to rebuild numerous packages that are commonly used with 2D/3D framework …

For example, i add to PR to numpy vendored-meson fix for numpy just yesterday, just to build on 3.13.

Except for ONE, Harfang3d this one i did not rebuild since 3.11 initial port … Guess why ? it is a limited api - abi3 module !

Even when people care about PEP, they may not know enough or have samples about limited api/abi to do anything durable with it.

That could sometimes be mistaken for some kind of hostility against third party modules maintener just because you often hear “Don’t use that it is a binary module. We only support pure python modules.” or " Yes you can have binary modules, but this one will be hell to build/maintain because it does not support the only CPython that supports your platform".

Well pure python modules are not performant at all. and also have porting problem of their own that only pop up at runtime

c-api modules need constant care ( eg all/parts of scientific stack each year see pyodide/kivy had to do for getting around )

cffi modules need (too) complex wrappers and are runtime minefields.

Limited api abi3 are fresh air, fast and portable.
and associated with a stable config runtime it would be just perfect way better !

Thanks a lot to everyone trying helping the C-API Working group to get around the problem and improve !

[vstinner (Victor Stinner)]

If PEP 741 is accepted, I plan to attempt to implement it on Python 3.8-3.12 in the experimental deadparrot project: a compatibility layer for the C API implemented as a shared library. It might help to increase the adoption of such API if it’s available on old Python versions as well.

[vstinner (Victor Stinner)]

@eric.snow

• who are we trying to help with this API?
• how did they let us know their needs (and that the status quo wasn’t good enough)?

I’m collecting feedback from users: see previous comments.

@eric.snow

• why does this need to be part of the limited API?

There are use cases to embed Python with the stable ABI. The main use case is to select which Python version is select which Python version is used when running an application, without having to rebuild the application.

@eric.snow

what is the purpose of exposing config values after initialization has finished?

Reading the current configuration at runtime was asked multiple times by Marc-André Lemburg.

Marc-André Lemburg wrote (emphasis is mine):

I also could not find a public PyConfig API for reading the current config. Why is that ?
There’s lots of useful information in the PyConfig struct, which extensions could use as well and won’t be available elsewhere anymore once the global vars are gone.

Marc-André Lemburg also wrote (emphasis is mine):

@malemburg

FWIW and as a real use case: For eGenix PyRun, I need a way to access some of the Python config settings in Python and after having initialized the interpreter (essentially, all settings which can be set from the Python command line or via env vars). Both for reading and writing.

Before 3.11, I could do this via accessing the global config variables from C. Since 3.11, this no longer works.

Marc-André also asked to have a way to modify them at runtime, but this feature is not part of the PEP scope. When it was discussed later, it was said that it’s better to modify Python attributes such as sys.path, rather than the “configuration” (such as config.module_search_paths).

Marc-André wants to update config.verbose but again, it’s not part of the PEP scope. I didn’t see other people asking for such feature so far. By the way, sys.flags.verbose is read-only.

@eric.snow

each phase should have a distinct config this is used exclusively as input to the corresponding init function

PyInitConfig API can be used for that, no? It’s an opaque structure, we have a lot of freedom to decide what it contains and how it behaves.

If you want PyInitConfig_SetInt() to behave differently depending if you initialize the Python main interpreter, or a sub-interpreter, we can have different “Create” functions, such as:

• PyInitConfig_CreatePythonInterpreter().
• PyInitConfig_CreateIsolatedInterpreter().

For example, we can say that setting a preconfiguration option, such as allocator, would fail with an error.

@eric.snow

there isn’t a strict 1-to-1 mapping between config fields and runtime state fields

PyConfig_Get() does not only read PyInterpeterState.config, but also sys attributes: there is no 1to1 mapping. It’s specified in the latest version of PEP 741 (PyConfig_Get() spec).

@eric.snow

once an init phase finishes, that config becomes irrelevant; the runtime never uses it

Many PyInterpeterState.config members are used at runtime and there is no other copy. Even if the implementation changes to remove that member, PyConfig_Get() would still remain relevant, no?

@eric.snow

we would keep a read-only copy of the config around only for diagnostic purposes (as a snapshot of how the runtime was initialized)

What would be your use case for that? With no concrete use case, there risk is that it would only waste memory. Which API would give access to it?

Currently, we don’t keep copies of PyConfig.module_search_paths or sys.path when they are modified.

@eric.snow

many config values in PyConfig are relevant for the global runtime but not for individual interpreters (we shouldn’t store a PyConfig on each PyInterpreterState)

PyInitConfig is opaque and so is a better API, since some members can be removed/ignored for sub-interpreters.

Note: PEP 741 “PyInitConfig” API does not deprecate PEP 587 PyConfig API.

[vstinner (Victor Stinner)]

@steve.dower

Unless we find a way to completely remove preinitialization (which I still hope we can), I’d like to see it explicitly marked as optional. Possibly the best way to do that is to change it into a “set current locale” or “infer current locale” function instead, and then require it be called before the Set*Locale functions.

It’s now mostly optional, only the 2 “Locale” functions need it. If you avoid these functions, the preconfiguration is implicit and only done when Python is initialized at once by Py_InitializeFromInitConfig().

Py_PreInitializeFromInitConfig() must be called before you can call PyInitConfig_SetStrLocale(). Py_PreInitializeFromInitConfig() is responsible to configure the LC_CTYPE locale if you use PyInitConfig_CreatePython(). If you use PyInitConfig_CreateIsolated(), the LC_CTYPE locale is left unchanged: "configure_locale" option is 0 by default in the “isolated” configuration.

@steve.dower

Can we just move “coerce C locale” out of the library entirely?

I don’t think so. The current implementation cannot set temporarily the LC_CTYPE locale just during a PyInitConfig_SetStrLocale() call and then restore the locale. It would lead to mojibake and other encoding issues. This code is complicated and fragile. Just use an isolate configuration to leave LC_CTYPE unchanged.

PEP 587 added the “Python Configuration” since there are use cases to write a “custom” Python which behaves like the regular Python, but with a different default configuration.

@steve.dower

I’m still not entirely convinced by the native “Get” functions (other than PyConfig_Get).

This feature was asked by @pitrou:

@pitrou

Finally, it seems that PyConfig_Get and PyConfig_GetInt will only work if interpreter initialization was successful, since they set an exception on error? This means it’s not possible to inspect the current configuration while building it.

---

@steve.dower

The PyConfig_Get description should say that you need to be holding the GIL, and that it will read settings from the current active [sub]interpreter.

Oh sure, I will clarify these points in the doc in my next PEP update.

@steve.dower

Regarding Configuration Options, I don’t like having exhaustive lists of options like this in the PEP. People are too likely to read it as documentation and not realise that it’s not up to date (the headers we have now help, but still better to tweak the phrasing in the text).

Some PEPs now have a header saying that it’s a “historical document” and that the implementation is now the reference. I added the list since it’s uneasy to parse PyConfig and PyPreConfig members from the documentation, and the format is different (PEP 741 uses strings).

@steve.dower

Py_ExitWithInitConfig exits the entire process. We shouldn’t provide embedders this API. They can exit their own process when they’re ready.

Hum. The function made more sense in the first version of the PEP when PyInitConfig contained an “exit code”. I will remove the function, you can just use the other functions:

printf("ERROR: %s\n", PyInitConfig_GetError(config));
PyInitConfig_Free(config);
exit(1);

[steve.dower]

@vstinner

@steve.dower

Can we just move “coerce C locale” out of the library entirely?

I don’t think so. The current implementation cannot set temporarily the LC_CTYPE locale just during a PyInitConfig_SetStrLocale() call and then restore the locale. It would lead to mojibake and other encoding issues.

I meant that if you want to embed Python and coerce the locale, then you have to coerce it yourself, and Python won’t do it for you. (And we do it in python.c, so that anyone just running python3 will get coercion still.) Otherwise, you should only use UTF-8 APIs.

As you point out, modifying your entire process’s state is very fragile. We should avoid doing it to embedders, and encourage them to do it themselves. We can still do it in our python3 executable, but not during initialization.

[ronaldoussoren]

For context: py2app currently still uses the old initialisation API (Py_Initialize and related APIs), which has been pretty stable over the years other than needing some adjustments for the python 2 to python 3 transition. The stub executable included with py2app works with all python versions from 2.7 onwards (for now) by using dlsym to look for API symbols.

I’m currently (slowly) working on transitioning to the new PyConfig API which is a lot nicer than the old API, but sadly means that the stub executable will have to be rebuild for every python version. Not too bad, but needs some extra work to get right.

End of “context”.

The proposed API looks like a step forward by once again enabling having a single executable that works with multiple python versions going forward. It will also make it possible to simplify the stub executable by removing most hard coding knowledge about configuration options (and move that knowlegde to the python code that generates app bundles).

One thing I’d like to see is a way to either recover from specifying an incorrect config option for PyInitConfig_Set...(), or to check if an option is valid. That would make it easier in the future to adjust to changing/new configuration options. That’s not a big issue for me though, I expect that I’ll be using a config file in the app bundle to control most options and that config file will be generated by python code that’s easily adjusted as needed.

What is the use case for the introspection API in https://peps.python.org/pep-0741/#get-current-configuration?

[vstinner (Victor Stinner)]

@steve.dower

I meant that if you want to embed Python and coerce the locale, then you have to coerce it yourself, and Python won’t do it for you.

It seems like we are talking past each others. That’s exactly what the “isolated” configuration, PyInitConfig_CreateIsolated(), does.

[steve.dower]

We’re definitely talking past each other. I’m talking about _Py_CoerceLegacyLocale, which is currently called automatically during preinitialization. It gets called regardless of which default settings you choose (isolated or “Python”).

Perhaps you’re planning on changing that? But I don’t see it in the PEP, which currently lists “coerce_c_locale” as a preconfiguration option that cannot be set after preinitialization.

I’m proposing that it should not be an option at all. We should move _Py_CoerceLegacyLocale into python.c, or at least make it a public API that has to be called manually, and then we call it from python.c (unless the environment says not to).

But it should never be called as part of initialization or preinitialization. Right now, it is, and right now, PEP 741 is not proposing to change that.

[vstinner (Victor Stinner)]

@steve.dower

We’re definitely talking past each other. I’m talking about _Py_CoerceLegacyLocale, which is currently called automatically during preinitialization. It gets called regardless of which default settings you choose (isolated or “Python”).

Aha. Let me check.

In short, the isolated configuration does not “coerce” the LC_CTYPE locale, unless you explicitly opt-in for this feature.

I replaced Programs/_testembed.c with:

#ifndef Py_BUILD_CORE
#  define Py_BUILD_CORE
#endif
#include <Python.h>
#include "pycore_runtime.h"   // _PyRuntime

int main()
{
    PyConfig config;
    PyConfig_InitIsolatedConfig(&config);

    PyStatus status = Py_InitializeFromConfig(&config);
    PyConfig_Clear(&config);
    if (PyStatus_Exception(status)) {
        Py_ExitStatusException(status);
    }

    printf("coerce_c_locale? %i\n", _PyRuntime.preconfig.coerce_c_locale);

    return Py_RunMain();
}

I also added a printf():

diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 5e5db98481..a5a8bfd189 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -278,6 +278,7 @@ _coerce_default_locale_settings(int warn, const _LocaleCoercionTarget *target)
 int
 _Py_CoerceLegacyLocale(int warn)
 {
+printf("CALL _Py_CoerceLegacyLocale\n");
     int coerced = 0;
 #ifdef PY_COERCE_C_LOCALE
     char *oldloc = NULL;

Python calls _Py_CoerceLegacyLocale() if the LC_CTYPE locale is C (it even calls it twice):

$ env -i ./python -c pass
CALL _Py_CoerceLegacyLocale
CALL _Py_CoerceLegacyLocale

Programs/_testembed doesn’t:

$ env -i ./Programs/_testembed 
coerce_c_locale? 0
Python 3.13.0a3+ (heads/python_finalization_error2-dirty:30bd9c94f1, Feb 15 2024, 10:02:13) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

_Py_CoerceLegacyLocale() is not called and effective preconfig.coerce_c_locale is 0.

Extract of _PyPreConfig_Write() in Python/preconfig.c:

    if (config.configure_locale) {
        if (config.coerce_c_locale) {
            if (!_Py_CoerceLegacyLocale(config.coerce_c_locale_warn)) {
                /* C locale not coerced */
                config.coerce_c_locale = 0;
            }
        }

        /* Set LC_CTYPE to the user preferred locale */
        _Py_SetLocaleFromEnv(LC_CTYPE);
    }

Another extract of Python/preconfig.c:

static void
preconfig_init_coerce_c_locale(PyPreConfig *config)
{
    if (!config->configure_locale) {
        config->coerce_c_locale = 0;
        config->coerce_c_locale_warn = 0;
        return;
    }
    ...
}

In short, if configure_locale is 0, which is the case for the isolated preconfiguration, the LC_CTYPE locale is not coerced.

If Python changes the LC_CTYPE locale with the isolated configuration, it’s a bug which must be fixed.

[vstinner (Victor Stinner)]

@steve.dower

I’m proposing that it should not be an option at all. We should move _Py_CoerceLegacyLocale into python.c, or at least make it a public API that has to be called manually, and then we call it from python.c (unless the environment says not to).

PEP 587 and PEP 741 have two configurations:

• “Isolated” to embed Python.
• “Python” to create a custom Python program.

I don’t see why we should drop this “Python” configuration from PEP 741. Having a single API for Python itself and in the public C API makes the implementation simpler: less code to maintain. Also, there are different use cases to write a custom Python.

If there is good reason to disallow configuring the LC_CTYPE locale and/or configuring C streams (such as stdio) in the “Isolated” configuration, I can consider changing PEP 741 PyInitConfig_SetInt() to fail when attempting to set these preconfiguration options (in an isolated configuration). But I don’t think that it’s harmful to let users do whatever they want, we are between consenting adults here, no? For me, what matters the most are only default values of each of these two configurations.

[steve.dower]

@vstinner

I don’t see why we should drop this “Python” configuration from PEP 741. Having a single API for Python itself and in the public C API makes the implementation simpler: less code to maintain. Also, there are different use cases to write a custom Python.

Because you are proposing to make the behaviour of these APIs stable for a long period of time.

I’m okay with that for an isolated configuration, because it means we will stabilise the use of config settings on a struct.^[1]

I’m not okay with that for a configuration that relies on environment variables, path searching, registry searching, and so on. Claiming that is stable is laughable.

If someone is writing a custom Python (e.g. me, I do this), they can and will have to adapt to each version anyway. Nothing is gained from telling them they don’t have to worry about each version separately - they will know that’s not true and will laugh at us. The usual guarantee about not changing within minor releases is entirely sufficient for this case.

---

1. It doesn’t matter whether the user sets stuff directly or through an opaque handle. The point is, if the calling code doesn’t set it, it is set to a constant. ↩︎

[encukou (Petr Viktorin)]

I was asked to fully remove the Python preinitialization. Well, I almost did it: only PyInitConfig_SetStrLocale() and PyInitConfig_SetStrLocaleList() need it.

Do they need it right away, though? Perhaps they can set a flag that this string should be decoded when it’s time?
(That would mean the value can’t be read by PyInitConfig_GetStr, and the PEP might need a new PyInitConfig_GetStrLocale for them.)

---

What is sys_path_0?

---

If I set "dev_mode" to 1, will a subsequent PyInitConfig_GetInt(..., "faulthandler", ...) return 1?
If I explicitly set "dev_mode" to 1 and “faulthandler” to 0, what will happen? Does the order matter?

Generally, should I think of the Set functions as setting a single member of a struct, or possibly doing a more complex operation?

---

PyConfig_Get and PyConfig_Keys form the skeleton of a Mapping interface. Why not expose one in sys, instead of sys.get_config(name: str)?

---

@steve.dower

make the behaviour of these APIs stable for a long period of time.

ABI stability doesn’t mean that a function can’t start failing once a future version of Python makes its behaviour obsolete. IMO, the API proposed here is as stable as, for example, importing a stdlib module and using getattr on it.

If a removed option breaks users’ code, with the API proposed here they can make their code compatible with both old and new Python. The lack of “TrySet” or “ClearError” API makes it cumbersome – on error they need to start over with a fresh PyInitConfig – but it’s doable.
When removing options, PEP-387 should apply, and there should be a way to report deprecation warnings (e.g. store the warning and emit in right after initialization). But I don’t think we need to be stricter than PEP-387. (We could make PEP-387 itself stricter, though.)

[pitrou (Antoine Pitrou)]

@steve.dower

[1]I’m not okay with that for a configuration that relies on environment variables, path searching, registry searching, and so on. Claiming that is stable is laughable.

This does not help. There are two aspects to stability here:

1. stability of configuration options and their respective presets
2. stability of the programmatic means to access those options at runtimes (in other words, APIs)

It’s pretty clear to me that @vstinner 's proposal aims at stability #2 and does not claim to freeze the actual options and presets themselves until the end of times.

You may think that #2 is useless without #1, but other people in this thread disagree. Let’s stop arguing endlessly against this?

PS: we already expose stable APIs with changing semantics, and that’s a feature.

[vstinner (Victor Stinner)]

@encukou

Do they need it right away, though? Perhaps they can set a flag that this string should be decoded when it’s time?
(That would mean the value can’t be read by PyInitConfig_GetStr, and the PEP might need a new PyInitConfig_GetStrLocale for them.)

I don’t get which problem you are trying to solve here. If you call PyInitConfig_GetStrLocale(), Python must be preinitialized. How is it a big deal if it’s during PyInitConfig_GetStrLocale() call, or at Py_InitializeFromInitConfig()? During it during PyInitConfig_GetStrLocale() call avoids mojibake (and makes the implementation simpler.)

@encukou

What is sys_path_0?

Oh, I don’t know It comes from PyConfig and was added to Python 3.13. It’s set in pymain_run_python() of Modules/main.c and used by init_interp_main() of Python/pylifecycle.c. It seems to be a recent change. Maybe it should not be exposed, I don’t know.

@encukou

If I set "dev_mode" to 1, will a subsequent PyInitConfig_GetInt(..., "faulthandler", ...) return 1?
If I explicitly set "dev_mode" to 1 and “faulthandler” to 0, what will happen? Does the order matter?

Ah. That’s a tricky question. If a configuration option changes other options, IMO InitConfig should not attempt to emulate or implement this logic. It should only be used to feed PyConfig which implements the final logic, in _PyConfig_Read(). Setting “dev_mode” should not change read “faulthandler” value (which stays in its initial value, unless it’s explicitly set).

“Reading” the configuration is a complex and slow operation. It can read configuration files, environment variables, implement multiple “default values”, etc. It should not be run at each “PyInitConfig_Get” operation.

This specific example is also tricky, it depends on the selected config. In the isolated configuration, setting dev_mode doesn’t set faulthandler.

_PyConfig_Read() logic:

    /* default values */
    if (config->dev_mode) {
        if (config->faulthandler < 0) {
            config->faulthandler = 1;
        }
    }
    if (config->faulthandler < 0) {
        config->faulthandler = 0;
    }

• Isolated config: dev_mode=0, faulthandler=0.
• Python config: dev_mode=0, faulthandler=-1.

This complexity also comes indirectly from the legacy API to configure Python initialization, especially supporting global configuration variables. In fact, Python has 3 configurations:

• Legacy config: legacy Py_Initialize() API
• Python config: PEP 587
• Isolated config: PEP 587

The legacy config behaves differently than the two others to support global configuration variables and other ways to config the init…

[vstinner (Victor Stinner)]

@encukou

PyConfig_Get and PyConfig_Keys form the skeleton of a Mapping interface. Why not expose one in sys, instead of sys.get_config(name: str)?

Which API do you propose?

Note: I’m not convinced that PyConfig_Keys() is strictly needed.

[encukou (Petr Viktorin)]

@vstinner

How is it a big deal if it’s during PyInitConfig_GetStrLocale() call, or at Py_InitializeFromInitConfig()? During it during PyInitConfig_GetStrLocale() call avoids mojibake (and makes the implementation simpler.)

I’m trying to understand what the API means to the users, and how we can simplify their mental model.
This is hard for me – there’s a pile of options where no one knows all the details about how they’re set and what they all mean. I’m impressed that the initialization now has only two phases – preinit and init. It took a lot of work to get there. But still, should we stop at number two for the long-term-stable API? What if we need 3 in the future? Or can we get it down to one?

For example, maybe PyInitConfig_SetStrLocale could look like this:

int PyInitConfig_SetStrLocale(PyInitConfig *config, const char *name, const char *value):

Set a string configuration option from a null-terminated bytes string encoded in the locale encoding. The string is copied.
This function initializes the locale encoding. The following options cannot be set after this function is called:

• allocator
• coerce_c_locale
• coerce_c_locale_warn
• configure_locale
• legacy_windows_fs_encoding
• utf8_mode

we could remove preinitialization as an API concept, and set a precedent for any new “phase” we’d need to add in the future – it “locks in” some of the options.

But the implementation would be more complicated in this case: preinitialization affects global state, and we don’t want PyInitConfig_SetStrLocale to do that. Storing the original un-encoded bytes until initialization seems like a possible solution.
(edit: and of course, with this solution it wouldn’t need to “lock in” any options)
I’m not sure it’d be worth it. I’m trying to understand the trade-offs.

@vstinner

Oh, I don’t know It comes from PyConfig and was added to Python 3.13. It’s set in pymain_run_python() of Modules/main.c and used by init_interp_main() of Python/pylifecycle.c. It seems to be a recent change. Maybe it should not be exposed, I don’t know.

Ah! Thanks. I followed up in #109853.

@vstinner

Ah. That’s a tricky question. If a configuration option changes other options, IMO InitConfig should not attempt to emulate or implement this logic.

It is indeed a tricky question.
Your answer seems fine. Conceptually (talking about semantics, not implementation):

• setting the config writes a value to a struct.
• getting using PyInitConfig_Get retrieves that config value, unchanged. (And if we assume that only a single piece of code calls PyInitConfig_Set*, then Get isn’t all that useful.)
• reading at runtime (PyConfig_Get) computes the effective value, merging all the various inputs (including changes done at runtime in Python code, and, in non-isolated mode, external state). Presumably the effective values would all be computed at initialization time, to make sure the order of reads doesn’t matter.

But since this is a PEP, we should think about alternatives, if only to put them in rejected ideas. For example:

• setting a value sets all the dependent values as well. Options that only affect multiple “smaller” options (like dev mode) might actually not be stored at all. The order of set calls matters.
• getting using PyInitConfig_Get or PyConfig_Get gets the current effective value. This would imply that you’re not guaranteed to get the same the value back (for example, in a future Python, a legacy option is now no-op and reading it always returns a constant)

I don’t think this is the better option, all things considered. But IMO the PEP should explicitly say that this is not how things will work, to avoid confusion.

@vstinner

Which [mapping] API do you propose?

sys.config, an object with the Mapping interface (__getitem__, __iter__, __len__, __contains__, keys, items, values, get, __eq__, and __ne__).

Alternatively, API that could enable a third-party project to expose such an object – that is, expose PyConfig_Keys in sys.

FWIW, the proposed PyConfig_Keys is C API without a Python equivalent, I don’t think we should do that: either it shouldn’t be in C API or it should be in sys.