IDLE IDE A Bit Too Slow

Hello,

when testing the following test script on both the IDLE and PyCharm IDEs, it became readily apparent that the IDLE IDE is extremely slow. Can it be improved?

Here are my test results:

  1. PyCharm … ~20 s
  2. IDLE … ~252 s

The IDLE IDE is slower than PyCharm by a factor of ~12.6.

Here is the test script:

import _thread as thread
stdoutmutex = thread.allocate_lock()
exitmutexes = [thread.allocate_lock() for i in range(10)]

countera = 0

def counter(myId, count):

    global countera

    for i in range(count):

        stdoutmutex.acquire()
        print(f'[{myId}] => {i}, count = {countera}')
        countera = countera + 1
        stdoutmutex.release()

    exitmutexes[myId].acquire() # signal main thread

try:

    for i in range(10):
        thread.start_new_thread(counter, (i, 100))

    for mutex in exitmutexes:
        while not mutex.locked(): pass
        
except KeyboardInterrupt:
    pass

print('Main thread exiting.')

I think the idea is a bit vague. I’m sure we’d all like it faster. Have any specific ideas to get there?

I believe the first thing that needs to be precised is: slow in doing what? I suppose that these results show a slowness just in printing to the IDE’s console.

Hello,

thank you for responding to my query.

The results are a bit surprising since PyCharm uses the IDLE interpreter as its Python interpreter. Both IDEs were tested on the same computer and hence have the same system clock. So, not really sure why there is a substantially greater delay when using the IDLE IDE. Is it only when it comes to the print statements as @zuo stated in his response that affects the latency? There are after all 1000 print statements that are produced when running the test script.

Update
I reran the script this morning. The IDLE execution time remained the same. On PyCharm, the execution time was ~8 s. This would make the IDLE / PyCharm factor equal to: 252 / 8 = 31.25!

IDLE is not an interpreter. It’s an interactive REPL that uses the CPython executable as its interpreter. IDLE isn’t meant to be a tool for running scripts; the first thing I would try is to run the script with python directly.

To be precise: my statement was just a hypothesis, I am not sure it is true.

You may want to verify it by trying the same comparison you did (PyCharm vs. IDLE) with a slight modification to the script: removing the print(...) line.

1 Like

Hello,

thank you very much for the suggestion. Yes, I have verified that it is the print statements that are the root cause for the latency. When I replaced the print statement with this:

t = 2 ** 10
g = myId + count

The output latency was nearly instantaneous. As a verification that it was looping 1000 times, I added the following print statement as the last statement in the script:

print('The value of countera is: ', countera)

What was printed was:

The value of countera is:  1000

On average, a print statement in the IDLE IDE takes ~250 ms whereas in Pycharm it takes 8 ms.

1 Like

Would be interesting to find where the extra latency / time is, what gets that much slower? That large a delta tends to be indicative of something going fairly sideways and that fixing it may improve user experience generally. Needs someone who wants to dive in / investigate.

It is also possible that the PyCharm’s UI toolkit (Java-based) is inherently faster than the IDLE’s one (which is tkinter, Tcl/Tk-based).

@tjreedy Sorry if I tag you unnecessarily here, but I do it because I suppose you may be interested in this topic :slight_smile: (from the dev point of view).

Idle is (completly?) pure python build on top of tkinter. To make this work, sys.stdout is redirected by overwriting it with a custom class:

>>> import sys
>>> sys.stdout
<idlelib.run.StdOutputFile object at 0x000002814E3E7EE0>

You can lookup the source code for this class here: cpython/Lib/idlelib/run.py at 58a4357e29a15135e6fd99f320c60f8ea0472d27 · python/cpython · GitHub

This means that IO redirection happens from within Python. PyCharm instead does IO redirection out-of-process and at C level, which is faster.

In the end, this is a cost of IDLE being pure-python and while it might be possible to reduce this a bit, it’s probably not worth the effort. Don’t run STDIO-bound performance critical programs within IDLE

1 Like

Just did a quick test with for i in range(1000): print(i) in IDLE compared to the terminal: Yes, it’s absolutely the text printing that IDLE is doing slowly. This is likely a Tcl/Tk issue that can’t be fixed.

The workaround is to replace the print() calls by writing to a file. Use the tail -f command on the file if you need live updates instead of reviewing everything after your Python program has finished.

1 Like

Jan, thank you for tagging me. I try to watch this channel for IDLE items, but I don’t look every day.

Anyone, feel free to ‘at’ me, tjreedy here on IDLE posts.

The extra time for printing is mostly due to running short strings through the socket connecting the user code execution process to the IDLE gui process. Prompted by a similar stackoverflow.com python-IDLE question, I investigated and then added the following to the IDLE doc in Running user code

Sending print output from one process to a text widget in another is slower than printing to a system terminal in the same process. This has the most effect when printing multiple arguments, as the string for each argument, each separator, the newline are sent separately. For development, this is usually not a problem, but if one wants to print faster in IDLE, format and join together everything one wants displayed together and then print a single string. Both format strings and str.join() can help combine fields and lines.

Applied to the initial post,

out = []
...
out.append(f'...')
...
print('\n'.join(out)

As Clint said above, IDLE is mostly for developing scripts (+ learning, me) rather than production runs.

2 Likes