PEP Talk
Hi,
The following is co-written with Aaron Patterson but written mostly in the first person. It represents only our personal viewpoints, not those of our employer or other CRuby developers.
My name is Max. I work on ZJIT at Shopify. I used to work on Cinder at Facebook and before that on Skybison at Facebook. I thought it would be helpful to give perspective about how the YJIT/ZJIT team addressed the following concerns from the CRuby perspective (though keep in mind that CRuby’s development process isn’t as formal as CPython’s, so we don’t have exactly the equivalent of PEPs):
Maintenance is a real concern for a subsystem of this size and complexity, and the PEP should set out a clear plan for how the JIT will be sustained and maintained over the long term and how it will impact maintainers and contributors that don’t directly contribute to the JIT.
Maintenance can have multiple facets. For example the host language choice (in our case Rust baked inside of a C codebase), the compiler design (YJIT is LBBV, ZJIT is a more traditional SSA CFG method based compiler).
Here is some brief history and some of the things we’ve found so far:
- Initial versions of UJIT/YJIT were written in C
- Writing a compiler in C is hard™
- Rust makes it easier to avoid UB etc in the compiler itself
- Rust code is easier to organize than C
- Rust has nice data structures that you have to DIY in C
- Rust is a maintenance burden on the core CRuby team because they are mostly familiar with C and Ruby
- Rust is a maintenance burden because of the requirement that CRuby be buildable and packageable on stock Debian; we cannot use Cargo to install dependencies or build the JITs.
- This has knock-on effects such as some of the implementation details being slightly less idiomatic than in the wild
- YJIT used a newer-research-based architecture (LBBV) whereas ZJIT intentionally moved toward a more standard/well-known SSA-CFG-based architecture to encourage more community contribution and ownership
- CRuby configure/build complexity. CRuby has the requirement that it only depends on a C compiler, so we’ve had to introduce build complexity in order to make YJIT/ZJIT optional
We’ve found the awkward build requirements to be not quite a “one time cost”, but very close to it—more O(1) than O(n). We’re not frequently mucking with build configuration things while doing JIT development. Additionally, since the JIT is something that can optionally be enabled at compile time, it forced us to design the JIT integration such that it could be removed from the codebase at any time (since it can be removed via build flags).
Despite some of the negative points listed above, we’ve found the added complexity to be worthwhile. We’ve found the Rust code to be easier to maintain, and the more “traditional” design of ZJIT to attract more external contributors.
Last, there is corporate sponsorship. I have seen that corporate sponsorship can accelerate and focus a JIT project (maybe makes it more cathedral than bazaar) and also can run the risk of abandonment or slowdown if the corporate sponsor loses interest. It cuts both ways.
How it will keep compatibility with existing CPython features and tooling. How the JIT interacts with, and what it guarantees for, the capabilities CPython already supports. This includes things such as free-threading, profilers, and debuggers, but the PEP should treat this broadly and with nuance rather than as a fixed checklist.
We haven’t explicitly documented this for YJIT/ZJIT, but the informal requirement is that there is no difference between observed interpreter behavior and behavior when the JIT compiler is enabled. In some cases the interpreter plays fast and loose with things like frame pushes (e.g. what is visible in a backtrace, or a profiler), and sometimes the JIT compiler does too, but usually in the same ways.
Other compatibility things we have noticed:
- Non-public C API stuff – the ZJIT team has decided not to care to keep exact (accidental?) interpreter compatibility if C extensions do naughty things
- We may delete allocations that are potentially countable by Ruby-visible allocator APIs
- We currently keep the same exact backtraces as the interpreter but may in the future choose to occasionally elide a frame (this will have to be discussed and formalized)
- We support sampling profilers and tracing and all manner of strange reflection for local variables
- Bytecode instructions are the API boundary between VM and JITs and and we have had some friction there: some Ruby tests assert that methods compile, or don’t side-exit into the interpreter, etc. So sometimes the JIT team plays catch-up re-enabling tests and following up with an implementation of the opcode, and there is the possibility that this complication has a (small?) chilling effect on the development of new or existing opcodes in the interpreter.
Additionally, the way we have structured YJIT and ZJIT is that they can choose to opt-in or opt-out for potentially frustrating combinations of features such as JIT+Ractor or JIT+Box; when the VM enables/makes use of a feature that the JIT does not yet support, it bails out of existing code and does not attempt to compile if it has interactions with new code. This lets us have a more gradual development process rather than tackle everything head-on. I think there is at least a little bit of this kind of implementation strategy in the existing CPython JIT.
Clear, measurable success metrics and timelines. What the project is aiming for and by when, for example performance targets, platform coverage, and memory overhead.
Prior to YJIT, CRuby had MJIT which emitted C and shelled out to a C compiler to get platform-native code. MJIT committers introduced this style of JIT because they initially had a requirement that the JIT should support most/all target architectures; relying on an existing, mature C compiler helped here. They found that trying to write a very platform-generic compiler by relying on existing tools is very difficult and eventually wound it down.
I can’t say what is best for CPython, but Shopify initially developed YJIT in a fork of CRuby and when the team was able to demonstrate a substantial speed improvement it was a much easier case for upstreaming to CRuby (including getting Matz, the BDFL, to relax the all-architecture requirements). ZJIT followed on YJIT’s coattails because there was trust in the team.
Relationship to other JIT compilers. Whether the design is intended to provide general infrastructure that other efforts could build on, and whether it is expected to be compatible or incompatible with third-party JIT implementations such as CinderX, Numba, and PyTorch or any other 3rd party JIT.
We don’t have a public API for building a 3rd party JIT, but CRuby does have a small set of well-understood touchpoints inside the VM that allow JITs to hook in: method call counters, method invalidation callbacks, redefinition of “constants”, tracing APIs (TracePoint), etc. This allows both YJIT and ZJIT to both intercept frequently called methods and bail out when other code causes JIT-unfavorable conditions.
Additionally, CRuby has support for building a VM with both YJIT and ZJIT compiled in, but only allows one to ever be enabled in a given process. Perhaps this is a temporary limitation; the JIT developers have not discussed it further.
CPython has at least some support for compiler infrastructure: CinderX works as a native extension. Other runtimes offer support for fuller interfaces, such as OpenJDK’s JVMCI (C1+C2+Graal).
Thanks for reading this long post. Hopefully it is somewhat illuminating. I am happy to answer questions and field comments.
Best,
Max & Aaron
cc @corona10, @kj0, @mpage