Harden ttk/tkinter GUI tests against display scaling (DPI) · Issue #153716 · python/cpython · GitHub is a good example.
When Skip asked how to configure Python for Tcl 9.0 on Ubuntu, I installed Tcl/Tk 9.0 and redirected the question to AI. It given me the reciepe and built Python with Tcl/Tk 9.0 in a new worktree. For curiosity, I ran the tests and spot few failures. I asked AI to look at them (the tests were passed on CI and buildbots). It ran them several times, asked me to install xvfb, and discovered that they are flaky on the real X server, but are passed on the fake X server, used for automatic testing, what matters. This solved my question. It also spot a test that passed on the real X server, and always failed on the fake X server, and this is where the journey began.
It tried several options and discovered that the result depends on the DPI. It proposed solution: make the assertion more lenient, accept several possible results (depending on platform), or even simply assert that the result is not empty. This would fixed the test failure. But it missed the root problem. The purpose of the test was to simultate mouse click on the tab in the Notebook widget and check that the corresponding child was selected. The result of identify() was tested only to ensure that we click on the tab before clicking. That was the intention. But subsequent edits changed the test. On Mac, the point to click was outside the tab. identify() returned an empty string, but it was accepted as correct result to make the test pass. The following mouse click did not have any effect. The test passed because the child was already selected. The tes tested nothing – the mouse click was no-op on other platforms too. After pointing on this and instructing how to solve it (find a real point on the tab, select other tab before click, so it will no longer be no-op), the AI was able to write the code, and, after additional instructions, to find and harden other similar spots (some of them failed for even higher DPI, not tested before).
This shows that AI is well capable to solve problems if it recognizes a pattern, and it can apply that pattern to find and fix similar issues. But it can solve wrong problem, propose solution that only cures symptoms. So it always needs a supervisor which is able to understand what it does and is knowledgable enough to see the whole picture. The cases like today occurr constantly.