Building the JIT with pre-built stencils

Core Development
1

Hello. As a follow-up from PEP 774: Removing the LLVM requirement for JIT builds - #36 by hroncok I’d like to be able to build CPython with pre-built JIT stencils.

Pretty much:

  1. Build the stencils on a platform with the required LLVM version (say, on Fedora).
  2. Save the stencils (per architecture, per build type (optimized/debug)).
  3. Use the stencils from above on a different platform without the required LLVM version (say, on RHEL) by dropping them to the out of tree build directory.

I was able to do this with hacks. Notably:

  • The make dependency hierarchy insisted on rebuilding jit_stencils.h. I was able to hack around that by invoking make with JIT_DEPS= (empty) (including make install).
  • The PGO task deletes the jit_stencils.h file mid build. I was able to hack around that by sed -i '/rm -f jit_stencils.h/d' Makefile after running ./configure.

I was wondering if eliminating the need for those hacks could be a supported use case. E.g. something like --with-jit-stencil=my-file.h which would copy it instead of trying to rebuild it?

Thanks.

5 Likes
PEP 774: Removing the LLVM requirement for JIT builds
2

This should be pretty easy to add to the prototype but I’m curious about what your use case is? More specifically, I’m interested in why you’d want to force using the stencils.

1 Like
4

Deleted the post above because I may have misunderstood, sorry folks!

1 Like
5
  • In RHEL, we keep one Python version for a very long time.
  • On the other hand, LLVM keeps getting rebased.

As a conclusion, we cannot keep a build time requirement on a specific LLVM version (for ~10 years).

Similarly, sometimes in Fedora, a new LLVM version is not available in the oldest supported Fedora (but we want to add a new alpha version of Python).

When that Python becomes older, the old LLVM version might no longer be available on the newest supported version of Fedora.

For now, as the is JIT experimental, we keep building it only in (new enough) Fedora with the required LLVm version. But we would like to keep building it for other platforms and using the prebuilt stencils seemed like a good solution.

I want to force using the stencils because if I don’t, the build machinery tries to rebuild them (and fails, without the necessary LLVM version).

1 Like
6

In fact, I think your summary was good enough. Minor changes:

We want to build the JIT without the clang/LLVM build requirement on (too old or too new) Fedora/RHEL systems. One way to do that would be to prebuild the stencils at the distributor level (say, a specific Fedora version), then pass it around for various other Fedora/RHEL versions to use when we build Python there (also at the distributor level).

3 Likes
7

Thanks for the details. One concern I have is around long-term correctness. If you’re using pre-generated stencils with an older Python version and we backport a change that changes how some bytecode is specialized, the stencils could become subtly stale. Since the build and tests might not catch this, it could silently introduce bugs or crashes.

In my reference implementation, I have CI rerun to build and assert that the stencils haven’t changed (and of course, we are mandating a specific LLVM version at this point) to avoid this issue.

I’d be interested in how you plan on catching this kind of drift in your workflow.

2 Likes
8

We always plan to regenerate the stencils when we update Python (or add a patch related to JIT).

If the stencils are always generated bit-by-bit identically, we could also assert that (and know that we need to save new ones when they change).

1 Like
9

Makes sense. Coincidentally, during the PyCon US sprint, Brandt recently added a --output-dir flag in the build script to fix a Windows CI issue. I haven’t tested this myself, but I don’t think you need to have LLVM installed to check if the stencils need to be regenerated. If you place your stencils in this directory and pass in this new flag, this might do what you’re after.

Want to give that a try and report back? If this doesn’t work for some reason, I’d be happy to explore options for either modifying the supported flags or introducing a new flag, as you mentioned above.

2 Likes
10

Thanks for the pointer. I am not sure how to use this via configure+make… ?

Happy to try it out. However, we are starting the Python 3.14 mass rebuild in Fedora today, so I won’t get to it this week.

11

So, you should be able to validate this by running the JIT build script directly, e.g.:

./python.exe Tools/jit/build.py <triple> --output-dir ./test --pyconfig-dir . .

If you point --output-dir to the directory containing your pre-generated stencils and pass that into the script, it should let us confirm whether this works as expected.

That said, we’d still need to make this configurable via configure to support this workflow cleanly, since --output-dir is currently hardcoded.

12

Finally, I got back to this. As of now, it seems that I can successfully do:

./configure --enable-experimental-jit=yes-off --enable-optimizations
cp -a .../Python-3.15.0a2-x86_64-optimized-jit_stencils.h jit_stencils-x86_64-unknown-linux-gnu.h
sed -i '/rm -f jit_stencils\*.h/d' Makefile
make ...

And I get a successful build with JIT using pregenerated stencils. No hacks with JIT_DEPS= seem to be necessary. However, I still need to remove this line:

When I build with --enable-optimizations and don’t remove that line from the clean-retain-profile make target, the build deletes my stencils mid-run and then fails with Can't find clang-21!. I wonder if the stencils need to actually be removed during this task. If they are the same before PGO and after PGO, perhaps not? (Will verify this assumption.)

13

Yeah that’s a bug in the PGO script I think. We should remove that.

1 Like
14

When doing a JIT build in WSL, I see the

JIT support for x86_64-unknown-linux-gnu is still experimental!

message twice in a full rebuild.
Maybe that’s a hint that the stencils are built twice?

At least on Windows I know for sure, that, in case of a PGO build the stencils are built twice:
First they are generated into the PGInstrument folder and then again in the PGUpdate folder.
They’re absouletely the same, though …

Update: or at least they are tried to be generated twice at the same location on Linux, and the second time the hash - shortcut would kick in so at least they aren’t generated another time there - but on Windows they are, because it’s two different folders …

15

Verified, they are the same. I’ll propose the removal of that line from the Makefile.

16

I opened gh-141808: Don't remove the JIT stencils when building with PGO by hroncok · Pull Request #141809 · python/cpython · GitHub as a draft for now.

17

Hello, we’ve been doing this on Fedora for each release of Python 3.14 and 3.15 so far. We generate stencils on the newest system with llvm 21.1.8 (currently) and store them for platforms without that version available. On platforms where llvm 21 is available, we generate the stencils and check that it’s the same as the stored copy. This has worked so far, but not for 3.15.0a8 release where the file generated on Fedora 45 doesn’t match the one generated on Fedora 43 for aarch64 debug build:

+ diff -u /builddir/build/SOURCES/Python-3.15.0a8-aarch64-debug-jit_stencils.h build/debug/jit_stencils-aarch64-unknown-linux-gnu.h
--- /builddir/build/SOURCES/Python-3.15.0a8-aarch64-debug-jit_stencils.h	2026-04-08 00:00:00.000000000 +0000
+++ build/debug/jit_stencils-aarch64-unknown-linux-gnu.h	2026-04-08 12:41:11.984291756 +0000
@@ -67367,7 +67367,7 @@
     // a4: cb282ec8      sub     x8, x22, w8, uxth #3
     // a8: f100053f      cmp     x9, #0x1
     // ac: 928000e9      mov     x9, #-0x8               // =-8
-    // b0: 9a9f1129      csel    x9, x9, xzr, ne
+    // b0: 9a8903e9      csel    x9, xzr, x9, eq
     // b4: f869691b      ldr     x27, [x8, x9]
     // b8: 2a3b03e8      mvn     w8, w27
     // bc: f240051f      tst     x8, #0x3
@@ -67810,7 +67810,7 @@
         0x60, 0x00, 0x00, 0x54, 0x3f, 0x05, 0x00, 0xf1,
         0xe0, 0x18, 0x00, 0x54, 0xc8, 0x2e, 0x28, 0xcb,
         0x3f, 0x05, 0x00, 0xf1, 0xe9, 0x00, 0x80, 0x92,
-        0x29, 0x11, 0x9f, 0x9a, 0x1b, 0x69, 0x69, 0xf8,
+        0xe9, 0x03, 0x89, 0x9a, 0x1b, 0x69, 0x69, 0xf8,
         0xe8, 0x03, 0x3b, 0x2a, 0x1f, 0x05, 0x40, 0xf2,
         0xe0, 0x12, 0x00, 0x54, 0xa8, 0x22, 0x40, 0xf9,
         0xe8, 0x13, 0x00, 0xb5, 0x68, 0xfa, 0x7f, 0x92,

Is this possible (and expected) to happen regularly? I have checked that our llvm versions on the respective systems differ slightly, there are some patches backported from 22, which I suspect could be the culprit. Do you have ideas if the difference here is semantically relevant?
On x86_64 build the files are identical.

1 Like
18

@diegor do you know anything about this?

19

This instruction that is changed is a conditional select: https://developer.arm.com/documentation/dui0801/h/A64-General-Instructions/CSEL

By the look of it, they are semantically the same. The csel stores the first register into x9 if the condition is True, otherwise it stores the second register.
In this case the condition comes from a8: f100053f cmp x9, #0x1

In both cases the same value ends up in x9

Why the compiler decides to do this, it’s a mystery to me :slight_smile:

2 Likes
20

We have another problem now – the digest no longer matches when we updated autoconf. I’ve opened Add a way to ignore JIT stencil digest · Issue #148646 · python/cpython · GitHub for that.