Making performance work easier in CPython

Core Development
1

After chatting with folks at PyCon US, EuroPython and the sprint last week, I wanted to start a broader discussion about our benchmarking infrastructure and tooling. There are a few things I’d like to work on and would love input from folks.

Rehoming bench_runner into the Python org

I’d like to move bench_runner into the Python org on GitHub.

For those unfamiliar, bench_runner is what we use to run pyperformance through GitHub Actions on self-hosted runners. I’ve been using it for doesjitgobrrr.com for the better part of a year, and I know Meta and Arm have it configured as well.

This tool currently lives in the Faster CPython org, but I think it falls pretty squarely into Python org territory. I’d like to take on more of the maintenance and cleanup, including making the setup documentation easier to follow. I’ve documented my setup on my personal website to some degree, but it’d be good to bring that into the Devguide too.

I chatted with @mdroettboom about the move earlier this year, and he was supportive.

Our benchmark suite and workflow

This is a bigger and, I expect, more contentious topic, but there are a couple of things I’ve been thinking about:

  • Our benchmark mix: Small, focused benchmarks are useful, but I think we’d benefit from revisiting the balance of those and larger workloads. I’m also interested in how families of related benchmarks contribute to the overall geometric mean. For example, are we giving the various async-tree benchmarks the right collective weight? I also know anecdotally that there are benchmarks folks intentionally ignore because they don’t think they’re useful or good benchmarks. It’d be good to hear which ones and why.

  • Workloads: I think it would be valuable to have more recognizable macrobenchmarks: a modern web application, a data analysis workflow, an ML workload, etc. Having some agreed-upon workloads that we benchmark regularly across machines could help make performance improvements more tangible for people outside the project. It’s easier to connect a result to your own work when you recognize the application or workflow being measured. Richards, for example, probably doesn’t mean much to most end users.

  • Nightly runs: The nightly runs for doesjitgobrrr.com have been really useful for catching regressions and seeing whether optimizations help (see @kj0’s blog about this too). I think it could be valuable to build on this and make day-over-day performance tracking easier to follow for CPython more broadly (doespythongobrrr?). Maybe that eventually includes adapting the site into something that could replace speed.python.org.

A perf bot

There are a lot of times in PRs where we ask whether a change is worth it or what the performance delta actually is. Getting that answer can be slow and requires access to a benchmarking setup, or finding someone who has it.

I’d love something like a !perf command or a label on a PR that core team members could use to trigger benchmarks and get results compared to main. We’d probably need a subset of benchmarks or a more targeted run to make the turnaround useful during review. Details TBD, but something faster than two hours would be awesome.

So, a few questions to kick this off:

  • Are you supportive of some or all of these ideas?
  • For folks working on performance, what would help you get useful data faster?
  • Which workloads should we be paying more attention to?
14 Likes
2

Yes, let’s do it.

2 Likes
3

Benchmarking can be tricky to do. Two common issues are binary layout noise and noise on the benchmarking machine. For the layout noise I have no good suggestions, but for noisy systems (and almost all are) it would be great if the benchmarking tool (e.g. pyperf or something else) supports interleaving (at a fine level, e.g. 40 or more switches between two binaries). This gives more accurate results in less time.

4

Yes, I’d really like to see this. Microbenchmark can be quite sensitive. Sometimes a tiny change makes it look much better, but barely affects real-world workloads.

I think a web application benchmark would be a good choice, something like SPECjbb for Java. Python gets used a lot in AI, but I’m not sure ML workloads would tell us much about CPython’s own performance, since most of the bottlenecks might be on GPU or in PyTorch.

5

Very supportive overall.

I would really like one command to re-generate any metric CPython tracks over time. Currently I juggle a mix of pyperformance, pyperf microbenchmarks, and periodically memory.python.org runs and how to get to the benchmark code so I can run it with local system tools is always difficult. The complexity has made it harder for me to investigate when I spot a regression or proactively check things I think my change will effect.

There are additional metrics such as code size (gh-157407), individual test case run time, etc. which would be nice to gather metrics on over time.

To be clear here: The tool doesn’t need to support every setup, etc. Error if a tool, such as uv, or library is not installed but required. Opinionated set of core tools works for me.