IDLE/REPL: add feature from PEP 657 Fine-Grained Error Locations in Tracebacks

Ideas
1

I just installed 3.11.0 and love the new feature PEP 657 – Include Fine Grained Error Locations in Tracebacks! But it seems that it doesn’t work IDLE nor the shell REPL:

$ python3.11
Python 3.11.0 (main, Oct 26 2022, 11:33:21) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = {'a': {'b': None}}
>>> x['a']['b']['c']['d'] = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not subscriptable

Compare to use in a file, as in the example:

$ cat test.py
x = {'a': {'b': None}}
x['a']['b']['c']['d'] = 1
print(x['a']['b']['c']['d'])

$ python3.11 test.py
Traceback (most recent call last):
  File "/data/shared/lusers/enneking/test.py", line 4 in <module>
    x['a']['b']['c']['d'] = 1
    ~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable

Can it be added to IDLE and the shell REPL?

(NB: The REPL is not discussed in the PEP nor the main discourse thread for PEP 657 (now closed).)

4 Likes
2

Thanks for your message.

This is not a problem of the PEP but how the REPR works. That’s why it was never discussed in the PEP discussion thread or the PEP itself.

The reason this is not supported at the time in the REPL is because the source is not available at the time of the exception. We would need to implement some bigger changes all around to make it available when that happens. Notice that even before 3.11 cpython doesn’t show source information in trace backs.

We played a bit with the idea and tried to propagate source around in code objects and the like but we didn’t like the compromises and the balance of advatanges to maintenance cost, so we never went forward.

In general a proper solution I think is to decouple the repr from the parser but that’s another story.

So thanks a lot for the proposal, I will give it another go but is unlikely to happen soon unfortunately :frowning:

2 Likes
3

Issue IDLE: make use of extended error info. · Issue #91312 · python/cpython · GitHub is about utilizing enhanced error location data when printing tracebacks in IDLE. For SyntaxErrors, color slices instead of just one character. For other errors, perhaps use separate colors for ~ and ^, though maybe not immediately.

@pablogsal It seems that py -c "None.a" treats the command like interactive input.

Was there any fine-grained markup for non syntax errors, such as this, in 3.10? If so, this TypeError is not one of them; can you give me (or point me to) an example I could use for testing in 3.10?

4

Note that this problem is not just in REPL but also when the code comes from stdin or the -c option (as mentioned):

$ echo 'assert False' > exception.py

Here we see the code:

$ python3 exception.py
Traceback (most recent call last):
  File "/home/vbrozik/tmp/exception.py", line 1, in <module>
    assert False
AssertionError

Here we do not see it:

$ python3 <exception.py
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

$ python3 -c "$(< exception.py)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError
1 Like
5

Thanks for explaining!

6

This is interesting. I didn’t realize the enhanced error messages didn’t work in REPLs. I wrote up a workaround for this that enables fine grained error locations in REPLs by routing all input through a file. It’s not the most efficient thing in the world but when in a REPL, I’m not too concerned about a tiny amount of delay on enter and am willing to pay the cost for better errors.

import code
import tempfile
from pathlib import Path


class FineGrainedErrorLocationsConsole(code.InteractiveConsole):
    def interact(self, banner=None, exitmsg=None):
        with tempfile.TemporaryDirectory() as dir_path:
            self.trick = Path(dir_path) / "ortreat.py"
            super().interact(banner, exitmsg)

    def runsource(self, source, filename="<input>", symbol="single"):
        self.filename = filename
        self.trick.write_text(source)
        return super().runsource(source, self.trick, symbol)

    def write(self, data) -> None:
        data = data.replace(str(self.trick), self.filename)
        return super().write(data)

if __name__ == "__main__":
    FineGrainedErrorLocationsConsole().interact()
1 Like
7

Have you tested FGELC? Do you have a github user name and have you signed the PSF Contributor Agreement, so I can credit you in case I use this code in IDLE?

8

Would not it be preferable to resolve the problem for all the cases (stdin, -c, REPL, IDLE) instead of just IDLE?

9

exec has the same problem also as its input ‘file’ is ‘’, and I would like a universal solution to interactive input errors.

IDLE uses compile and exec for both interactive and edited file input. The exec is within a try-except that catches and displays user code exceptions separately from IDLE code exceptions. In the file case, it supplies the file name to compile. Its current custom exception handing does not get the location indicators, but it could, regardless of what is done for interactive input.

========================== RESTART: F:\dev\tem\tem.py ==========================
Traceback (most recent call last):
  File "F:\dev\tem\tem.py", line 1, in <module>
    a=None; b={'c': 3}; b[a]
KeyError: None
>>> import sys
>>> sys.excepthook(sys.last_type, sys.last_value, sys.last_traceback)
Traceback (most recent call last):
  File "C:\Programs\Python312\Lib\idlelib\run.py", line 578, in runcode
    exec(code, self.locals)
  File "F:\dev\tem\tem.py", line 1, in <module>
    a=None; b={'c': 3}; b[a]
                        ~^^^
KeyError: None

IDLE currently used functions in the traceback module. One issue is that IDLE’s exception printer does not know whether the exception is from file or interactive input. Another is that I would want to highlight the offending code instead of leaving the symbol line.

1 Like
10

I’m not sure what “FGELC” is. My github username is TheTripleV and I just signed the agreement.
I realized the version of the code above doesn’t handle exceptions in previous inputs such as

>>> x = {'a': {'b': None}}
>>> def q():
...   x['a']['b']['c']['d'] = 1
>>> q()

Overwriting the same file over and over again doesn’t work so now I create one per input.

import code
import tempfile
import os
import re


class FineGrainedErrorLocationsConsole(code.InteractiveConsole):
    def __init__(self, locals=None, filename="<console>"):
        self.count = 0
        self.filenames = [None]
        super().__init__(locals, filename)

    def interact(self, banner=None, exitmsg=None):
        with tempfile.TemporaryDirectory() as dir_path:
            self.basepath = os.path.join(dir_path, "halloween")
            super().interact(banner, exitmsg)

    def resetbuffer(self):
        self.count += 1
        return super().resetbuffer()

    def runsource(self, source, filename="<input>", symbol="single"):
        self.filenames.append(filename)
        with open(self.basepath + str(self.count), "w") as temp:
            temp.write(source)
            temp.flush()
        return super().runsource(source, self.basepath + str(self.count), symbol)

    def write(self, data):
        data = re.sub(
            rf'"{re.escape(self.basepath)}(\d+)"',
            lambda m: f'"{self.filenames[int(m.group(1))]}", >>> {m.group(1)}',
            data,
        )
        return super().write(data)


if __name__ == "__main__":
    FineGrainedErrorLocationsConsole().interact()

The output of the above first snippet in this message is now

>>> x = {"a": {"b": None}}
>>> def q():
...   x["a"]["b"]["c"]["d"] = 1
... 
>>> q()
Traceback (most recent call last):
  File "<console>", >>> 3, line 1, in <module>
    q()
  File "<console>", >>> 2, line 2, in q
    x["a"]["b"]["c"]["d"] = 1
    ~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable

Alongside the line number, the input number is printed as I feel that’s more relevant to a repl.

11

FGELC = your FineGrainErrorLocationConsole class.

IDLE currently gets better tracebacks by stuffing input lines into the LRU cache in the linecache module. This is used by the traceback printer to get file lines. Compare this REPL traceback

>>> 1/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

to IDLE’s, which has the erroneous code line.

>>> 1/0
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    1/0
ZeroDivisionError: division by zero

Note that IDLE, unlike REPL, uses sequentially numbered pseudofile names for each statement rather than the same one for each statement. The linecache maps (filename, lineno) pairs to code lines. So if statement 33 calls a function defined in statement 22, which calls a function defined in statement 11, all the needed lines are available. While the toy example above does not really need 1/0 repeated, realistic examples do need the specific code line.

If a file line needed by the traceback is not in the cache, the file is read. Reusing the same filename means that (‘’, 1), for instance, once mapped, never rereads the file! I suspect that your class would work if it did the same as IDLE instead of writing disk files. IDLE’s linecache code is scattered within idlelib.pyshell.
(Improvements are likely possible and tests are needed.)

1 Like
12

Yes, but is also one order of magnitude more complicated to solve in all cases (because of how the REPL is implemented). And they are roughly independent.