About speed.python.org and aarch64

Hello all,

Diego from Arm Ltd here. I’m assessing the status of CPython and its ecosystem on aarch64 and during my investigation I bumped into speed.python.org and pyperformance package.

I really like integration between pyperformance and codespeed and I was able to setup an internal instance within Arm with aarch64 results.

Being Arm architecture present on all mayor cloud providers and Python a critical programming language for cloud workflows, it would be great if we can integrate aarch64 performance results on speed.python.org.

Pointless to say that we are more than happy to help out with both development and by providing infrastructure (in some form) to achieve this goal.

Cheers,

Diego

6 Likes

Hi Diego,

Happy New Year! I had in the back of my mind to write you about how ARM could help Python, and I got overwhelmed with other things on my TODO list. It’s great that you managed to set up an ARM instance of speed.python.org. Is there a chance you can make that public? It doesn’t have to be on a *.python.org URL for now. I’d like to see if we can learn something by comparing the ARM results to the x64 results.

I’d also like to point you to Mike Droettboom’s efforts, which you can follow here: GitHub - faster-cpython/benchmarking-public: A public mirror of our benchmarking runner repository (and also to some extent in GitHub - faster-cpython/ideas).

2 Likes

Guido, happy new year too!

Apart from the performance point of view, is there anything else you have in mind we could help with? I just want to gather feedback on this.

I will try my best to make it happen. What’s the rationale behind having a separate instance instead of integrating directly to speed.python.org?

That’s useful indeed, thanks for the pointers!

Mostly just that speed.python.org has exactly one maintainer and he’s not got a lot of time for it, so anything that would require coordinating with him would just be extra red tape that I’d like to save you from.

1 Like

Guido, understood, thanks for explaining. I see what I can do and update here as soon as I have news (it might take some time)

1 Like

Just a quick update. Things are progressing well but before making anything public, we’ve been running some internal CI infrastructure to cross validate some data.
I will be off for the next three weeks but as soon as I come back, I’ll continue working on this.

3 Likes

Hello everyone, here I am with some exciting news :slight_smile:

First of all I want to apologise the delay: I’ve been busy with work, EuroPython 2023 (organising it, sponsor it and presenting) and I took annual/parental leaves on the way. So the whole thing took more than expected.

I’m happy to announce that I was able to replicate the benchmark infrastructure that shows CPython performance across Arm CPUs: Neoverse-N1, Neoverse-N2, and Neoverse-V1.

The full address is https://speed.python.arm.com/. We are tracking the following branches:

  • 3.11
  • 3.12
  • main

Older branches should be more stable in terms of metrics so we preferred to focus on newer ones. If you think we need to include older, let us know and we will do it.

We’ve had this infra in place for more than a month now, (since August 15th) because we needed to solve a few of issues with the infrastructure. Until September 14th commits benchmarked on machines might be different depending on the machine: this because the machines were available at different times (with a few hours difference) hence picked the latest commit of the branch at the time of execution of the job.

Since September 14th you should see more consistent data: we first pick the commit and then we schedule the job on different machines with this specific commit. Data might be uploaded at different times but when it arrives it is consistent with other machines.

The web server and the database are running on an aarch64 machine sponsored by the Works on Arm project and the benchmarks are running on physical machines behind the Arm firewalls. Jobs are managed by our own Jenkins instance.

I think that’s it for now, have a look and please share your thoughts.

10 Likes

Hi Diego,

Thanks for doing this!

I was glancing at the timelines, and the graph for the coverage benchmark struck me: CPython on aarch64 : Timeline

Between August 19 and August 27 the time this took shot up from 100ms to 800ms. Almost everything else was pretty stable. I didn’t see a similar jump for this benchmark on speed.python.org: Python Speed Center : Timeline

1 Like