Community perspectives on the JIT: experiences, expectations, and concerns

To set some expectations for performance, I have seen a lot of talk by internet users that PyPy is 2x faster on pyperformance (not from the PyPy devs themselves). I personally think this is people trying to napkin math the PyPy benchmark numbers into pyperformance numbers and guesstimating that. I personally think this is a bad practice, as benchmark numbers are not comparable across benchmark suites.

I would like to state that while PyPy is likely >2x faster on their benchmark suite. PyPy 3.11 is a very respectable 50% faster on macOS Apple Silicon and 80-90% faster on x86-64 Linux on pyperformance than 3.15 alpha from my own actual pyperformance runs.

This is not because PyPy is bad, but that pyperformance’ async benchmarks are notoriously hard for JITs, every JIT I’ve tried on that (PyPy, GraalPy, CPython 3.15) falls over on the async benchmarks.

Aside from that, CPython cannot break C API compatibility and do what PyPy does with collections, we also cannot abandon reference counting in the interpreter because people rely on immediate reclamation. Coupled with opaque C-level finalizers for many objects, CPython has to treat nearly every object deallocation site as a call that might call eval or something catastrophic. Even after a refcount elimination pass, there are still some sites that cannot be removed to preserve immediate reclamation. This is why I’m suggesting that 20% pyperformance geometric mean (with an up to 200% speedup range) is a good target to have considering all the constraints. I don’t mean we should require 20% for accepting the PEP, but it should be what we angle our expectations around.

JFYI: I have the utmost respect for PyPy. I have also shared the individual benchmark numbers with the PyPy devs to help them speed up PyPy so they can beat us even more :slight_smile: .

18 Likes