Adding char* based APIs for Unix

[ncoghlan (Alyssa Coghlan)]

@steve.dower

Shouldn’t Py_Main be the part that handles command line arguments? Since arguments are only really useful for our main entry point anyway, whereas embedders should do their own configuration (perhaps from their own command line)?

There are two major kinds of embedding scenario:

• Applications embedding Python as a scripting engine (think Blender, Maya, etc)
• Applications that are providing alternatives to the standard CPython CLI (think the isolated-by-default system-python idea in PEP 432, alternative REPLs, the bootstrapping no-import-system binary used to freeze importlib, etc)

Apps in the second category currently have a hard time correctly emulating CPython’s argument and environment handling, since a large chunk of it is hidden inside Py_Main. Even if you call Py_Initialize and then Py_Main, it isn’t quite the same thing, since not everything gets reconfigured after Py_Main has had a chance to look at the command line arguments.

So if we expose our command line processing machinery directly, then embedding applications can use that part of the system like a support library, rather than having to try to emulate it themselves.

[vstinner (Victor Stinner)]

@ncoghlan

Even if you call Py_Initialize and then Py_Main , it isn’t quite the same thing, since not everything gets reconfigured after Py_Main has had a chance to look at the command line arguments.

Yeah, right now, almost all the new “configuration” read by Py_Main() is ignored and will not be applied. Python keeps the “old” configuration read by Py_Initialize().

The problem is that many Python objects are kept alive between Py_Initialize() and Py_Main(). For this reason, it’s not possible to change the memory allocator in such case.

I would suggest to deprecate calling Py_Initialize() before Py_Main().

[vstinner (Victor Stinner)]

@vstinner

I would suggest to deprecate calling Py_Initialize() before Py_Main().

I created Issue 36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()? - Python tracker to discuss this issue.

[vstinner (Victor Stinner)]

@vstinner

I created https://bugs.python.org/issue36142 which adds a new internal _PyArgv structure to be able to pass command line arguments as bytes (char*) or Unicode (wchar_t*). It’s an helper to factorize the code. My change also rework Python initialization to be able to initialize _PyPreConfig and _PyCoreConfig structures (list of parameters) from command line arguments, whereas previously command line arguments were only handled in main.c (somehow private API).

This PR is a small step towards new Python initialization APIs which would accept command line arguments. (Py_Main() cannot be used to embed Python into an application.)

FYI I finished (merged) the implementation and I closed the issue We can now discuss in Issue 36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()? - Python tracker if we need to add a new Py_InitializeFromArgv() API.

---

Oh wow, Discourse disallowed me to post a “4th reply in a row”, even if I replied to 4 different messages. It forced me to edit a previous message… So here is my 4th message:

@methane

Strictly speaking, vim should use Py_DecodeLocale instead of mbstowcs here, if I understand the API correctly. But I’m not sure.
This is one example of how embedding Python correctly on Unix is difficult.

Even Py_DecodeLocale() can lead to mojibake since Python 3.7 See he issue that I just created:
https://bugs.python.org/issue36202

[vstinner (Victor Stinner)]

Most or even all issues described in this discussion are solved by my PEP 587: https://www.python.org/dev/peps/pep-0587/ PyConfig_SetBytesString() and PyConfig_SetBytesArgv() decode bytes string for you, and the PEP adds a new “preinitialization” phase with a dedicated PyPreConfig configuration to configure the LC_CTYPE locale. The PEP allows to parse argv as command line arguments. It implements the 2 use cases described by Nick Coghlan with 2 separated default configuration: “Python Configuration” and “Isolated Configuration”.

[methane (Inada Naoki)]

Thank you for adding bytes based APIs, and congrats for your PEP 587.
It looks very tough job.

[vstinner (Victor Stinner)]

You’re welcome. And you’re right, it was a very tough job The implementation took me longer than one year.