While working on the “run” function on PyEM (similar to pipenv run and poetry run) I accidentally found that the command history on Windows sometimes does not work.
Here’s a minimal reproducible example:
>>> import os, subprocess, sys
>>> sys.executable # A Python 3.8 venv created for testing.
'C:\\Users\\uranusjr\\Downloads\\foo\\Scripts\\python.exe'
>>> os.getpid()
6872
>>> # ["os.getpid()" if you press up here]
>>> subprocess.run([os.path.join(sys.base_prefix, 'python.exe'), '-q'])
>>> import os # Now we are in the inner interpreter.
>>> os.getpid()
13304
>>> # [Can still get "os.getpid()" by pressing up]
>>> ^Z
CompletedProcess(args=['C:\\Users\\uranusjr\\AppData\\Local\\Programs\\Python\\Python38\\python.exe', '-q'], returncode=0)
>>> # Now we're back to the outer interpreter
>>> subprocess.run([sys.executable, '-q'])
>>> import os # New inner interpreter.
>>> os.getpwd()
3020
>>> # [Now pressing up does nothing!]
I tried some combinations, with the following observations:
This happens regardless of the outer interpreter. The outer interpreter is a venvlauncher in the example, but I can reproduce with a base interpreter or venv befoew bpo-34977 implementation as well.
This only happens when the inner interpreter (the one run via subprocess) is a redirector. Both py.exe or venvlauncher has the same problem. An actual interpreter run via subprocess can still get history correctly.
I am unfamiliar with either how the command history is set up on Windows (the documentation is scarce), or how the redirector forwards calls using Windows API. Is the redirector not correctly forwarding inputs? (But left and right arrow keys work fine.) Is this due to how history is set up that the intermediate redirect confuses it? I am honestly quite stumped by this.
My guess is that the history is stored against the filename of the process that is directly connected to the console, and that the STARTF_USESTDHANDLES approach we use in launcher.c doesn’t properly communicate this. I’m not sure exactly what to do about it though…
@eryksun is the real expert here - perhaps we don’t have to duplicate standard handles and should only duplicate file/pipes?
I’m also trying to find a conhost.exe expert at work, but nobody is owning up to it yet
By default, Windows Python relies on the console’s cooked read (i.e. ReadFile or ReadConsoleW with line-input and echo-input modes enabled, and typically also processed-input mode). A cooked read implements command editing, aliases, and history in the console session server, conhost.exe. (ReadConsoleW does support a limited input-control hook for client processes, which is how cmd.exe implements tab completion, but Python has no use for it.) A console session stores command history for each attached client process in a history buffer. If you run python.exe via py.exe from cmd.exe, then 3 history buffers are consumed. Unfortunately there’s a fixed number of buffers. I think the default is just 4. You can increase the default in the Defaults->Options dialog, or increase it just for the current console instance (based on initial window title or shell link) in the Properties->Options dialog. Personally, I increase the default to 32 history buffers, with 512 commands per buffer.
Other languages such as PowerShell have moved away from the console’s built-in cooked read in order to gain more control over the UI and better cross-platform compatibility. This requires using a low-level read (i.e. ReadConsoleInputW) and a readline library, such as PowerShell’s PSReadLine. There’s a ctypes-based pyreadline module available for Python, but it’s not actively maintained. Maybe some day Windows Python will support the standard library’s readline module, linked with a readline(-ish) library such as WinEditLine.
Thanks for the super detailed response @eryksun! That explains everything… I would never have realised this is some kind of system constraint I’ve successfully worked around the issue by increasing the the number of buffers as you suggested.
Now I’ll need to figure out why Windows Terminal isn’t picking up the new default… brb rebooting Windows
Windows Terminal uses a modified build of conhost.exe that’s named openconsole.exe. I used procmon to monitor the registry keys it accesses at startup, and it appears the user’s console settings in “HKCU\Console” are never accessed, so openconsole.exe must use hard-coded defaults. You should still be able to use the console API to update the history settings dynamically. For example:
import ctypes
import collections
from ctypes import wintypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
HISTORY_NO_DUP_FLAG = 1
class CONSOLE_HISTORY_INFO(ctypes.Structure):
_fields_ = (('cbSize', wintypes.UINT),
('HistoryBufferSize', wintypes.UINT),
('NumberOfHistoryBuffers', wintypes.UINT),
('dwFlags', wintypes.DWORD))
def __init__(self, *args, **kwds):
super().__init__(ctypes.sizeof(self), *args, **kwds)
ConsoleHistoryInfo = collections.namedtuple('ConsoleHistoryInfo',
'bufsize nbuf flags')
def get_console_history_info():
info = CONSOLE_HISTORY_INFO()
if not kernel32.GetConsoleHistoryInfo(ctypes.byref(info)):
raise ctypes.WinError(ctypes.get_last_error())
return ConsoleHistoryInfo(info.HistoryBufferSize,
info.NumberOfHistoryBuffers, info.dwFlags)
def set_console_history_info(bufsize=512, nbuf=32,
flags=HISTORY_NO_DUP_FLAG):
info = CONSOLE_HISTORY_INFO(bufsize, nbuf, flags)
if not kernel32.SetConsoleHistoryInfo(ctypes.byref(info)):
raise ctypes.WinError(ctypes.get_last_error())
Set the size to 512 commands with 32 buffers, and filter out duplicate commands:
I ran into a similar problem when using xonsh and py. I ended up employing Eryk’s workaround in my shell startup to work around the issue. As a result, it’s available for general-purpose use in jaraco.windows.api.console.get/set_history_info.
I’ve successfully worked around the issue by increasing the the number of buffers as you suggested.