# Better PyREPL history file format?

**URL:** <https://discuss.python.org/t/better-pyrepl-history-file-format/87918>\
**Category:** Ideas\
**Created:** [April 11, 2025, 4:55am UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918 "2025-04-11T04:55:02Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![skirpichev](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/skirpichev/32/10996_2.png) [@skirpichev](https://discuss.python.org/u/skirpichev)\
**Post date:** [April 11, 2025, 4:55am UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/1 "2025-04-11T04:55:02Z")

</div>

New REPL uses same old plain-text format for storing history entries. But basically it has only one positive pros: it’s simple. Cons: it’s simple! It can’t keep anything else, but commands, no any other meta-information (e.g. timestamps).

One example of it’s current limitations is support for multiple repl, running concurrently. When all repl will be successfully finished, you end with a history from the _last_ repl you quit. Everything else will be lost.

This is something that could be mitigated by using separate `PYTHON_HISTORY` environment variables, pointing to individual (per running interpreter) files. Such history files then could be merged, though I can’t imagine anything better than just plain concatenation of all commands (well, perhaps you could find a common preamble for all saved command histories) to the single file. I doubt someone did something like this.

The IPython REPL is much better there. Here is an example (repl 1 and 2 running concurrently with repl 1 opened first):

```python
$ ipython
In [1]: # repl 1
In [2]: 111
Out[2]: 111
In [3]: 333
Out[3]: 333
In [4]: %history
# repl 1
111
333
%history
In [5]: # quit repl 1
In [6]:                                            
$ ipython
In [1]: # repl 2
In [2]: 222
Out[2]: 222
In [3]: 444
Out[3]: 444
In [4]: %history
# repl 2
222
444
%history
In [5]: # quit repl 2
In [6]:                                           
$ ipython
In [1]: # repl 3
In [2]: %history -l 11
# repl 1
111
333
%history
# quit repl 1
# repl 2
222
444
%history
# quit repl 2
# repl 3

```

Another issue with the current repl is that [it’s history lost if the interpreter crashed](https://github.com/python/cpython/issues/127495), because history file updated only at the end of session. This is something that never happened with IPython: statements are saved immediately _before_ execution and are available for all repl, which you will run later.

We can extend old plain-text format, e.g. like the bash does (put timestamps in comments). This (with some care) will allow to automate history merging.

Though, why not adopt IPython approach (maybe even re-use [it’s code for history support](https://github.com/ipython/ipython/blob/f3e796c8e82595990870218c2e25d7cd1fdfac8f/IPython/core/history.py))? The stdlib has sqlite3 module (plain-text format support could be also kept for compatibility and/or as a fallback), and we can provide a cli interface to work with history file by plain-text tools. (I wonder how common something like `grep <foo> ~/.python_history` for people?)

Is that a sane feature proposal or too complex for the default CPython shell?

---

<div class="post-metadata">

**Author:** ![sirosen](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sirosen/32/8478_2.png) [@sirosen](https://discuss.python.org/u/sirosen)\
**Post date:** [April 11, 2025, 12:12pm UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/2 "2025-04-11T12:12:05Z")

</div>

I’m cautiously supportive of these ideas. My main question is how many user tickets or forum questions there have been around these issues? It would seem like a shame to implement significant additional complexity for relatively little benefit.

In particular, I’m not sure how important being more graceful on crashes is, as crashes of the interpreter are relatively rare. If crashes are considered unimportant to handle better, then I’m not sure a change in file format is justified.

Here’s a scenario which I think is worth considering:  
If I accidentally enter sensitive information in a REPL, how do I remediate it?  
e.g.

```python
>>> # a long and productive REPL session
...
>>> # accidental copy paste
>>> x = "the quick brown medical record number: 987654"
>>> quit
# now what?

```

With the current flat history, you can edit or delete it. If the format changes to sqlite, editing becomes much harder. For many users, deleting it would be their only option in such a case.

---

<div class="post-metadata">

**Author:** ![skirpichev](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/skirpichev/32/10996_2.png) [@skirpichev](https://discuss.python.org/u/skirpichev)\
**Post date:** [April 11, 2025, 2:41pm UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/3 "2025-04-11T14:41:59Z")

</div>

> [@sirosen](#):
>
> My main question is how many user tickets or forum questions there have been around these issues? It would seem like a shame to implement significant additional complexity for relatively little benefit.

Good question. I suspect not too much, but the reason is: the plain CPython shell is less popular for day-to-day work, when there are enhanced, developed and stable alternatives like the IPython. Perhaps, the new REPL _might_ change this someday.

BTW, I’m not sure that required changes are significant (c.f. support for syntax highlighting).

> [@sirosen](#):
>
> crashes of the interpreter are relatively rare.

That depends on your workflow, but in general — yes, I think you are right. That’s why mentioned issue was a secondary example of limitations in the history handling of the PyREPL.

Edit: and it’s possible to solve this with plain-text history file format.

> [@sirosen](#):
>
> If I accidentally enter sensitive information in a REPL, how do I remediate it?

This one is seems easy: you can remove entries from the SQL tables just like you can remove line from the plain-text file. The `readline` module has interfaces to remove/replace history items, we can provide same API in the new REPL. I’m not sure how popular this scenario is — the IPython `%history` magic has no simple option to do that (though, it’s possible).

---

<div class="post-metadata">

**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\
**Post date:** [April 11, 2025, 2:48pm UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/4 "2025-04-11T14:48:24Z")

</div>

> [@skirpichev](#):
>
> This one is seems easy: you can remove entries from the SQL tables just like you can remove line from the plain-text file.

It’s considerably more complicated, if only because editing plain-text is dead easy.

That said, “nuke your whole history file” is also an acceptable solution to that problem, IMO.

---

<div class="post-metadata">

**Author:** ![xitop](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xitop/32/13242_2.png) [@xitop](https://discuss.python.org/u/xitop)\
**Post date:** [April 11, 2025, 2:51pm UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/5 "2025-04-11T14:51:50Z")

</div>

Saving the history list at exit is a good thing IMO, because it makes possible to work with the history list in memory. It’s a pity that the REPL currently doesn’t do that. It is just appending the last line even if it is a duplicate. I would prefer if only the last occurence was saved. A way to limit the history size would be appreciated too. I find anything above say 25 distinct lines not really useful. However none of these nice to have features is depending on the history file format.

---

<div class="post-metadata">

**Author:** ![barry](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barry/32/42_2.png) [@barry](https://discuss.python.org/u/barry)\
**Post date:** [April 11, 2025, 4:33pm UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/6 "2025-04-11T16:33:00Z")

</div>

Maybe related. I’ve _long_ had code (based on the [example in the docs](https://docs.python.org/3/library/readline.html#example)) in my `$PYTHONSTARTUP` file to put the history in a different location.

I was debugging a different problem with the CPython test suite and it led me to a bug in my startup file. After fixing that, I also noticed that Python wasn’t actually saving the history correctly, at least AFAICT, because I couldn’t just scroll up and back after restarting the interactive interpreter. I ended up just deciding that it didn’t matter where my history file is saved, so I just deleted the code and moved on, but I wonder if it was related to the new repl.

Point being, whatever we do with the new repl’s history, let’s make sure we keep in mind that folks do want to store their repl history elsewhere, and perhaps make it easier by adding something like a `$PYTHONREPLHISTORYFILE` environment variable, rather than requiring users to write some obscure readline code.

---

<div class="post-metadata">

**Author:** ![ambv](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ambv/32/25884_2.png) [@ambv](https://discuss.python.org/u/ambv)\
**Post date:** [April 16, 2025, 10:17am UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/7 "2025-04-16T10:17:26Z")

</div>

> [@barry](#):
>
> Point being, whatever we do with the new repl’s history, let’s make sure we keep in mind that folks do want to store their repl history elsewhere, and perhaps make it easier by adding something like a `$PYTHONREPLHISTORYFILE` environment variable, rather than requiring users to write some obscure readline code.

[This is already the case, Barry](https://docs.python.org/3/using/cmdline.html#envvar-PYTHON_HISTORY).

---

<div class="post-metadata">

**Author:** ![barry](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barry/32/42_2.png) [@barry](https://discuss.python.org/u/barry)\
**Post date:** [April 16, 2025, 5:35pm UTC](https://discuss.python.org/t/better-pyrepl-history-file-format/87918/8 "2025-04-16T17:35:50Z")

</div>

Awesome! It probably makes sense then to update the [readline example](https://docs.python.org/3/library/readline.html#example) to something more useful, or at least add a note that this is no longer necessary to get the stated functionality, and is just an example.