Builtin function input writes its prompt to sys.stderr and not to sys.stdout

The input documentation says that the prompt is sent to sys.stdout, but it is in sys.stderr. Is this a documentation bug or a software bug, and where should it be posted ?

c:\>cat input_writes_to_stderr.py
import sys
answer=input('type something:')
print('stderr',answer, file=sys.stderr)
print('stdout',answer, file=sys.stdout)

c:\>python input_writes_to_stderr.py 2> err.txt
No prompt appears, so I write this
stdout No prompt appears, so I write this

c:\>cat err.txt
type something:stderr No prompt appears, so I write this

c:\>

The documentation could use more details. The input() function in CPython uses sys.stdin and sys.stdout if either one is a different OS file descriptor from C stdin and stdout, or if either one isn’t a tty according to isatty().

Otherwise, input() calls PyOS_Readline(), which depends on the registered handler function, PyOS_ReadlineFunctionPointer(). The default handler is PyOS_StdioReadline(), which, for whatever reason, writes the prompt to stderr instead of stdout.

The PyOS_Readline() call will be handled by the readline module if it’s available and imported. It writes the prompt to stdout, not stderr – at least if it’s based on GNU Readline. By default readline is imported for interactive use, but it has to be imported manually in a script. You can check this in a POSIX system by redirecting just stderr to a file. For example:

$ python -c "import readline; print(input('prompt: '))" 2>err.txt
prompt: spam
spam
$ stat -c "size: %s" err.txt
size: 0

Out of the box, the readline module is not available in Windows. There’s a third-party pyreadline module, but it’s kind of buggy and hasn’t been maintained in a while.

Using GitHub “blame” one can follow the fprintf(stderr, "%s", prompt); back to 1993!

1 Like

Thanks for your elaborate answer, Eryk. With your input I’ve found this is very old issue 1927, that stalled many years ago. Code from 1993 is written in stone, so that will never get fixed anymore without major impact.
But it only fails on windows, and is indeed working as expected on Linux, even without readline module.

The simple solution for me is to print the prompt in a separate command, and refer to this post. :slight_smile:

If the readline module isn’t imported, PyOS_StdioReadline() is used by default, which prints the prompt to stderr:

$ python -c "print(input('prompt: '))" 2>err.txt
spam
spam
$ cat err.txt 
prompt: 

I wouldn’t expect the prompt to ever be written to stderr. It’s normal UI text, not auxiliary or error text.

Maybe we should take this up in that ancient issue, 1927? The conclusion there was that we should always write the prompt to stderr. I don’t recall why – on the surface, stdout makes more sense. But the precedent of bash prompting to stderr was brought up.

Maybe we should take guidance from where the “>>>” prompt goes. Does it go to stdout or to stderr? Arguably input() should use whatever that uses.

The REPL uses PyOS_Readline(), which input() also uses when sys.stdin and sys.stdout are tty files and the same OS files as C stdin and stdout. The REPL runs interactively, in which case the readline module is always imported, if it’s available. In a script or -c command, readline has to be imported manually.

I guess the historic reason is that you may want to capture just the user input for later reapplying this to an interactive program, i.e. an answer file. Having the prompts on stdout would make this difficult.

1 Like

You are right, but I was right too … in interactive mode, it is different. I have only 3.8, but that won’t make a difference, I think:

$python3.8 -c "print(input('prompt: '))" 2>err.txt
spam
spam
$ python3.8  2>err.txt
>>> print(input('prompt: '))
prompt: spam
spam
>>> 

While in windows, it remains stderr.

C:\>python 2> %TEMP%\err.log
print(input('prompt: '))
spam
spam
exit()
C:\>

So @guido I the “>>>” behaves exactly the same as prompt, but not consistently on Linux.

Interactive mode implicitly imports the readline module:

$ python -q
>>> import sys
>>> 'readline' in sys.modules
True

The readline module isn’t available by default in Windows. Since pyreadline seems to not be maintained anymore, a dependable readline module isn’t currently available in Windows.

1 Like

Yah. There’s longstanding tension (not just in Python) between prompting
to stdout (where you might expect the user’s eyes to be directed) and
polluting stdout with interactive junk. And not just polluting with
interactive junk, but also supporting interaction in a pipeline:

blah | blah | interactive-thing | blah

If you run that on a terminal and prompt to stderr the user can see the
prompts. Of course input() reads from stdin, which breaks that
particualr model.

The cleaner model for the above is the interact via a direct open of
/dev/tty (on UNIX, if there is a /dev/tty). Caveats abound :frowning:

Cheers,
Cameron Simpson cs@cskk.id.au

That would be for me a reason to just explain this in the documentation, and close this thing up. The same explanantion goes for >>>, e.g. this question, it is logical that interactive does nothing different. So in interactive mode, you would do python > file.out and see welcome string, “>>>” and input prompt on screen. But then again, just typing a varable name, puts its contents to stdout, that is then again incompatible

As Guido suggested, builtin input() should implement the same behavior as the REPL. The REPL calls PyOS_Readline(), which uses PyOS_StdioReadline() if either C stdin or stdout isn’t a tty, which in turn writes the prompt to stderr. Builtin input(), on the other hand, calls PyOS_Readline() only when either sys.stdin or sys.stdout isn’t a tty or when either is redirected from C stdin or stdout. Otherwise it always writes the prompt to sys.stdout. To be consistent with the REPL, it should write the prompt to sys.stderr when either sys.stdin or sys.stdout isn’t a tty. Here’s an example of the behavior that’s inconsistent with the REPL:

$ python -c "print(input('prompt: '))" >out.txt
spam
$ cat out.txt
prompt: spam

The tty check in input() could be split into separate std (i.e. standard file) and tty values. The tty value can then be reused to determine whether the prompt should be written to sys.stdout or sys.stderr. Also, I think if there’s no file descriptor associated with a file, the value of tty should come from its isatty() method. For example:

    /* Only use PyOS_Readline() if sys.stdin and sys.stdout
       are the same as C stdin and stdout and interactive. */
    tmp = _PyObject_CallMethodIdNoArgs(fin, &PyId_fileno);
    if (tmp == NULL) {
        PyErr_Clear();
        std = 0;
        tmp = _PyObject_CallMethodIdNoArgs(fin, &PyId_isatty);
        if (tmp == NULL) {
            PyErr_Clear();
            tty = 0;
        }
        else {
            tty = PyLong_AsLong(tmp);
            Py_DECREF(tmp);
            if (tty < 0 && PyErr_Occurred())
                return NULL;
        }
    }
    else {
        fd = PyLong_AsLong(tmp);
        Py_DECREF(tmp);
        if (fd < 0 && PyErr_Occurred())
            return NULL;
        std = fd == fileno(stdin);
        tty = isatty(fd);
    }
    if (std && tty) {
        tmp = _PyObject_CallMethodIdNoArgs(fout, &PyId_fileno);
        if (tmp == NULL) {
            PyErr_Clear();
            std = 0;
            tmp = _PyObject_CallMethodIdNoArgs(fout, &PyId_isatty);
            if (tmp == NULL) {
                PyErr_Clear();
                tty = 0;
            }
            else {
                tty = PyLong_AsLong(tmp);
                Py_DECREF(tmp);
                if (tty < 0 && PyErr_Occurred())
                    return NULL;
            }
        }
        else {
            fd = PyLong_AsLong(tmp);
            Py_DECREF(tmp);
            if (fd < 0 && PyErr_Occurred())
                return NULL;
            std = fd == fileno(stdout);
            tty = isatty(fd);
        }
    }

    if (std && tty) {
    
        // implement PyOS_Readline() call

    }

    /* Fallback for non-standard files or non-interactive use */
    /* For non-interactive use, prompt to stderr instead of stdout. */
    if (!tty) {
        fout = ferr;
    }
    if (prompt != NULL) {
        if (PyFile_WriteObject(prompt, fout, Py_PRINT_RAW) != 0)
            return NULL;
    }
    tmp = _PyObject_CallMethodIdNoArgs(fout, &PyId_flush);
    if (tmp == NULL)
        PyErr_Clear();
    else
        Py_DECREF(tmp);
    return PyFile_GetLine(fin, -1);

For Windows, we could combine the prompt and line-reading operations into a single _PyOS_WinStdioReadline() call. If legacy mode is disabled, and sys_stdin and sys_stdout are console files, this call writes the prompt to stdout via WriteConsoleW() and reads from stdin via ReadConsoleW(). Otherwise it chains to _PyOS_StdioReadline(), which writes the prompt to stderr and reads via fgets(). Define PyOS_StdioReadline as a platform-dependent macro, so the implementation of PyOS_Readline() can remain the same.

Other than moving all of the Windows-only code out of _PyOS_StdioReadline(), the most significant change would be that stdout will be used whenever stdin and stdout are console files. Thus if just stderr is redirected to a file or pipe, UI prompts will still be displayed in the console. This agrees with POSIX systems, if the readline module is available and imported. The ReadConsoleW() call is analogous to using the readline module. The console’s line-input mode implements a basic line editor that supports command history and aliases.

Example implementation:

#ifdef MS_WINDOWS
extern char _get_console_type(HANDLE handle);

/* Readline implementation using ReadConsoleW */

char *
_PyOS_WinStdioReadline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt)
{
    HANDLE hStdIn = _Py_get_osfhandle_noraise(fileno(sys_stdin));
    HANDLE hStdOut = _Py_get_osfhandle_noraise(fileno(sys_stdout));

    if (Py_LegacyWindowsStdioFlag ||
        _get_console_type(hStdIn) != 'r' ||
        _get_console_type(hStdOut) != 'w') {
        return _PyOS_StdioReadline(sys_stdin, sys_stdout, prompt);
    }

    PyThreadState *tstate = _PyOS_ReadlineTState;
    assert(tstate != NULL);

    int err = 0;
    char *buf = NULL;

    static wchar_t wbuf_local[16 * 1024];
    wchar_t *wbuf = wbuf_local;
    DWORD wlen = sizeof(wbuf_local) / sizeof(wbuf_local[0]);
    DWORD total_read = 0;
    DWORD n;

    fflush(sys_stdout);

    if (prompt) {
        DWORD pwlen = MultiByteToWideChar(CP_UTF8, 0, prompt, -1, wbuf, wlen);
        if (pwlen == 0) {
            if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {
                err = GetLastError();
                goto exit;
            }
            wlen = MultiByteToWideChar(CP_UTF8, 0, prompt, -1, NULL, 0);
            wbuf = PyMem_RawMalloc(wlen * sizeof(wchar_t));
            if (wbuf == NULL) {
                err = ERROR_NOT_ENOUGH_MEMORY;
                goto exit;
            }
            pwlen = MultiByteToWideChar(CP_UTF8, 0, prompt, -1, wbuf, wlen);
            if (pwlen == 0) {
                err = GetLastError();
                goto exit;
            }
        }
        /* pwlen includes the null terminator. */
        if (!WriteConsoleW(hStdOut, wbuf, pwlen - 1, &n, NULL)) {
            err = GetLastError();
            goto exit;
        }
    }

    while (1) {
        if (PyOS_InputHook != NULL) {
            (void)(PyOS_InputHook)();
        }
        n = (DWORD)-1;
        BOOL res = ReadConsoleW(hStdIn, &wbuf[total_read],
                        wlen - total_read - 1, &n, NULL);
        if (!res) {
            err = GetLastError();
            goto exit;
        }
        if (n == (DWORD)-1 && GetLastError() == ERROR_OPERATION_ABORTED) {
            err = ERROR_OPERATION_ABORTED;
            goto exit;
        }
        if (n == 0) {
            if (GetLastError() != ERROR_OPERATION_ABORTED) {
                err = GetLastError();
                goto exit;
            }
            HANDLE hInterruptEvent = _PyOS_SigintEvent();
            if (WaitForSingleObjectEx(hInterruptEvent, 100, FALSE)
                    == WAIT_OBJECT_0) {
                int s;
                ResetEvent(hInterruptEvent);
                PyEval_RestoreThread(tstate);
                s = PyErr_CheckSignals();
                PyEval_SaveThread();
                if (s < 0) {
                    goto exit;
                }
            }
            continue;
        }

        total_read += n;
        if (total_read == 0 || wbuf[total_read - 1] == L'\n') {
            break;
        }

        wlen += 1024;
        wchar_t *tmp = PyMem_RawRealloc(wbuf == wbuf_local ? NULL : wbuf,
                            wlen * sizeof(wchar_t));
        if (tmp == NULL) {
            err = ERROR_NOT_ENOUGH_MEMORY;
            goto exit;
        }
        if (wbuf == wbuf_local) {
            wbuf[total_read] = L'\0';
            wcscpy_s(tmp, wlen, wbuf);
        } 
        wbuf = tmp;
    }

    /* Input that begins with Ctrl+Z is handled as EOF. */
    if (total_read > 0 && wbuf[0] == L'\x1a') {
        total_read = 0;
    }

    DWORD u8len;
    if (total_read == 0) {
        u8len = 0;
    } else {
        u8len = WideCharToMultiByte(CP_UTF8, 0, wbuf, total_read,
                    NULL, 0, NULL, NULL);
        if (u8len == 0) {
            err = GetLastError();
            goto exit;
        }
    }

    buf = PyMem_RawMalloc(u8len + 1);
    if (buf == NULL) {
        err = ERROR_NOT_ENOUGH_MEMORY;
        goto exit;
    }

    if (u8len > 0) {
        u8len = WideCharToMultiByte(CP_UTF8, 0, wbuf, total_read,
                    buf, u8len, NULL, NULL);
    }

    buf[u8len] = '\0';

exit:
    if (wbuf != wbuf_local) {
        PyMem_RawFree(wbuf);
    }

    if (err) {
        PyEval_RestoreThread(tstate);
        if (err == ERROR_NOT_ENOUGH_MEMORY) {
            PyErr_NoMemory();
        } else {
            PyErr_SetFromWindowsErr(err);
        }
        PyEval_SaveThread();
    }

    return buf;
}
#endif /* MS_WINDOWS */

#ifdef MS_WINDOWS
#define PyOS_StdioReadline _PyOS_WinStdioReadline
#else
#define PyOS_StdioReadline _PyOS_StdioReadline
#endif

This implementation also fixes the bug when SIGINT or CTRL_C_EVENT is ignored, but I hope to find a better solution. Typing Ctrl+C cancels a read in the console itself, so even if the CTRL_C_EVENT is ignored by the process, whatever the user typed beforehand is lost. The display doesn’t change, so it misleadingly looks like the previous input will be read, but only subsequently typed text will be read.

The current implementation of PyOS_Readline() just immediately returns an empty string after an ignored SIGINT, instead of continuing the read loop. This signifies EOF, so the REPL exits and input() raises EOFError, which partially defeats the point of ignoring SIGINT. The above implementation is at least better than that since it continues the read loop. Maybe the UI can be improved by using the pInputControl parameter of ReadConsoleW().

1 Like

IDLE’s Shell colorizes user-code sys.stdout and sys.stderr output differently (and handles shell prompts differently from both). At least on Windows and Mac, input() prompts get sys.stdout colors, so they must be going there. Eryk explained this for Windows; I don’t know about Mac. Going to sys.stderr for a distinct color might be better (IDLE could add a masking ‘def input’ to make this happen).

After sys.stderr=None, calling input() results in RuntimeError: input(): lost sys.stderr. (IDLE re-routes the traceback to sys.stdout.) So sys.stderr is accessed by the call. In any case, any change to input should consider possible effects on code.InteractiveInterpreter and derivatives.

Terry, the change I proposed for input() was to make it more closely match how the builtin REPL works in POSIX. The builtin REPL calls PyOS_Readline() with C stdin and stdout. However, if both aren’t interactive (tty) files, PyOS_Readline() always writes the prompt to C stderr.

I modified input() to write the prompt to sys.stderr if both sys.stdin and sys.stdout aren’t interactive. The modified implementation still writes the prompt to sys.stdout if both sys.stdin and sys.stdout are interactive but either is a different file from C stdin or stdout.

In IDLE’s shell, sys.stdin and sys.stdout are both interactive, via their high-level isatty() methods, and they’re unrelated to C stdin and stdout. Thus, under IDLE, the proposed implementation of input() would continue to write the prompt to sys.stdout.

After sys.stderr=None, calling input() results in RuntimeError: input(): lost sys.stderr . (IDLE re-routes the traceback to sys.stdout.) So sys.stderr is accessed by the call.

Currently input() only uses sys.stderr to call sys.stderr.flush(). This is because PyOS_Readline() may be called, which may write the prompt to C stderr.

If the prompt from input() is going to keep being written to sys.stderr, can the documentation be updated to reflect this?

Sure! Please consider submitting a PR with the proposed changes you would like to see.