I was using Astra to work on optimizing CPython when the JIT is not enabled. I found several small improvements, and I plan to submit PRs for them.
Throughout the optimization work, the instability of pyperformance was a major obstacle. Even when a change is completely unrelated to the affected code, simply causing a particular function to cross some boundary can result in a significant performance regression in one of the benchmarks.
Codex found that even whether .pyc files were cleaned before running the PGO profiling could cause performance variations large enough to obscure the gains from an optimization.
I had Codex itself summarize the lessons learned from this optimization effort into a Skill. Using this Skill may help AI agents make better use of benchmarks when other people try to optimize CPython with an AI agent.
Of course, simply reading this Skill yourself can also be informative. It would be worth reading alongside Victor’s “My journey to stable benchmark.”