# PEP 744: JIT Compilation **URL:** \ **Category:** PEPs\ **Created:** [April 12, 2024, 12:33am UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756 "2024-04-12T00:33:50Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![brandtbucher](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brandtbucher/32/22862_2.png) [@brandtbucher](https://discuss.python.org/u/brandtbucher)\ **Post date:** [April 12, 2024, 12:33am UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/1 "2024-04-12T00:33:50Z")
[PEP 744](https://peps.python.org/pep-0744) is an _informational_ PEP answering many common questions about CPython 3.13’s new experimental JIT compiler. My main goal for this PEP is to build community consensus around the specific criteria that the JIT should meet in order to become a permanent, non-experimental part of CPython. The “Specification” section lists three basic requirements as a starting point, but I expect that more will be added as a result of our discussions here: > The JIT will become non-experimental once all of the following conditions are met: > > 1. It provides a meaningful performance improvement for at least one popular platform (realistically, on the order of 5%). > > 2. It can be built, distributed, and deployed with minimal disruption. > > 3. The Steering Council, upon request, has determined that it would provide more value to the community if enabled than if disabled (considering tradeoffs such as maintenance burden, memory usage, or the feasibility of alternate designs). If you have any questions or concerns that are not addressed in the PEP or its linked materials, please let me know and I’ll be happy to add them. ---
**Author:** ![NeilGirdhar](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/neilgirdhar/32/5776_2.png) [@NeilGirdhar](https://discuss.python.org/u/NeilGirdhar)\ **Post date:** [April 12, 2024, 5:15am UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/2 "2024-04-12T05:15:55Z")
Would it make sense to contrast the 3.13 JIT with PyPy’s JIT? ---
**Author:** ![malemburg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/malemburg/32/50_2.png) [@malemburg](https://discuss.python.org/u/malemburg)\ **Post date:** [April 12, 2024, 8:39am UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/3 "2024-04-12T08:39:54Z")
Thank you for the write-up, Brandt. This is an exciting development 😄 A few questions: - _LLVM_: I am a bit confused about the statements related to LLVM. In the “Rational” section you write using LLVM would incur significant overhead and a heavy dependency. Then in the “Support” section you write that some platforms may not receive JIT support due to LLVM not having good support for them. Could you clarify whether you are using LLVM of not ? Other comments in the PEP suggest that you are using LLVM. If that’s the case, how do you manage to keep the overhead low ? (In that case, it may also make sense to change/update the “Rational” section to avoid such questions) - _Patents_: Are you aware of any patents on the copy-and-patch approach or other advanced techniques you are using in the JIT, which may encumber the use of a JIT enabled CPython ? - _RAM usage_: Does the JIT limit itself to using a certain percentage of available RAM (as a measure against DOS style attacks or bugs in the code) ? How quickly are less used JIT compiled parts garbage collected ? Can the JIT GC be controlled programmatically (e.g. by sys module calls) ? - _Dependencies_: Assuming you are using LLVM, is it still possible to build CPython without JIT support on platforms where LLVM is not available/supported ? ---
**Author:** ![mdroettboom](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mdroettboom/32/6774_2.png) [@mdroettboom](https://discuss.python.org/u/mdroettboom)\ **Post date:** [April 12, 2024, 2:05pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/4 "2024-04-12T14:05:56Z")
_LLVM_: A typical approach to creating a JIT is to use LLVM at runtime. This is the “cost of introducing heavy runtime dependencies” mentioned in the Rationale that we want to avoid. The copy-and-patch approach is novel in that it only introduces the LLVM dependency at build time, where the overhead is less important since it only affects CPython developers, not Python users. I think some judicious use of “runtime” and “build time” to the doc will hopefully clear up this confusion. _Patents_: We aren’t aware of any patents, and the copy-and-patch paper on which this is based is open access (which isn’t the same thing, but helps). However, I am not a lawyer – is there a typical process that happens when this is a concern? _Dependencies_: There are no plans to remove the ability to build CPython without the JIT on any platform. It is likely that the default build will remain “without JIT”, even after the default binaries on supported platforms become “with JIT”, just as PGO and LTO are today. _RAM usage_: I’ll leave this for Brandt to answer. ---
**Author:** ![smontanaro](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/smontanaro/32/1389_2.png) [@smontanaro](https://discuss.python.org/u/smontanaro)\ **Post date:** [April 12, 2024, 2:38pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/5 "2024-04-12T14:38:31Z")
Brandt, Thanks for the PEP (still reading it). I did watch your talk this morning. As someone who hasn’t really done much with the virtual machine in a couple decades, Knowing know 3.11 & 3.12 laid the groundwork for the JIT work was quite useful. I’m guessing most people won’t be familiar with the copy-and-patch technique, so it might be a good idea to add those Lua references to the PEP bibliography. ---
**Author:** ![smontanaro](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/smontanaro/32/1389_2.png) [@smontanaro](https://discuss.python.org/u/smontanaro)\ **Post date:** [April 12, 2024, 2:42pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/6 "2024-04-12T14:42:59Z")
Note that I stopped reading the PEP when I encountered the link to your talk, so missed the two links right after it. I still think an explicit bibliography makes sense. ---
**Author:** ![brandtbucher](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brandtbucher/32/22862_2.png) [@brandtbucher](https://discuss.python.org/u/brandtbucher)\ **Post date:** [April 12, 2024, 5:40pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/7 "2024-04-12T17:40:02Z")
> [@NeilGirdhar](#): > > Would it make sense to contrast the 3.13 JIT with PyPy’s JIT? Maybe. Despite the maybe-overly-broad title, I’ve tried to keep the PEP focused on this implementation specifically, rather than making it “a survey of all of the Python JITs”. What sort of information are you hoping to get from a section contrasting the two JITs? While I have had short conversations with a couple of PyPy devs on different occasions (including brief tours of parts of their code), I’m far from an expert on PyPy’s JIT backend. So I would either need to do a lot deeper dive into the source code, or provide a perhaps-not-detailed-enough summary based on my own incomplete understanding from various papers, blog posts, and other [resources](https://rpython.readthedocs.io/en/latest/jit/index.html) available online. ---
**Author:** ![brandtbucher](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brandtbucher/32/22862_2.png) [@brandtbucher](https://discuss.python.org/u/brandtbucher)\ **Post date:** [April 12, 2024, 6:03pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/8 "2024-04-12T18:03:25Z")
Mostly echoing @mdroettboom’s excellent answers: > [@malemburg](#): > > Thank you for the write-up, Brandt. This is an exciting development 😄 > > A few questions: > > - _LLVM_: I am a bit confused about the statements related to LLVM. In the “Rational” section you write using LLVM would incur significant overhead and a heavy dependency. Then in the “Support” section you write that some platforms may not receive JIT support due to LLVM not having good support for them. Could you clarify whether you are using LLVM of not ? Other comments in the PEP suggest that you are using LLVM. If that’s the case, how do you manage to keep the overhead low ? (In that case, it may also make sense to change/update the “Rational” section to avoid such questions) LLVM is used _at build time_ to compile individual micro-op instructions into mostly-opaque blobs of machine code. These blobs are dumped into a header file, which is used to build CPython itself. At runtime, we compile a sequence of micro-ops by “simply” copying the machine code for each one, almost verbatim. So instead of firing up LLVM, it’s “just” a `memcpy` and a couple of loops to fix up parts of the code. [Implementation here, in C.](https://github.com/python/cpython/blob/4ad8f090cce03c24fd4279ec8198a099b2d0cf97/Python/jit.c#L368-L369) The PEP could probably be updated to make the run-time/build-time distinction clearer. > [@malemburg](#): > > - _Patents_: Are you aware of any patents on the copy-and-patch approach or other advanced techniques you are using in the JIT, which may encumber the use of a JIT enabled CPython ? I’m not aware of any patents on the approaches used here. > [@malemburg](#): > > - _RAM usage_: Does the JIT limit itself to using a certain percentage of available RAM (as a measure against DOS style attacks or bugs in the code) ? How quickly are less used JIT compiled parts garbage collected ? Can the JIT GC be controlled programmatically (e.g. by sys module calls) ? No, when their refcount hits zero, and no. 🙂 …at least, that’s the state right now. The current memory allocation/freeing scheme can be summed up as “I got it working a while back and haven’t touched it since”. All of the points you make above are things I want to do, but simply haven’t had the time to work on yet. One main thing to keep in mind is that I’d like to avoid introducing any “official” JIT APIs (or encouraging users to tweak them) while it’s still experimental. I could definitely see tunable controls for enabling/disabling the JIT, controlling max memory, and more control over GC of cold traces in the future, though, exposed through the `sys` module, environment variables, etc. The good news is that all of the things you ask for should be easy to do. We already maintain a doubly-linked list of all JIT code (for unrelated correctness reasons) and have ways to turn the JIT on and off at runtime (for testing). > [@malemburg](#): > > - _Dependencies_: Assuming you are using LLVM, is it still possible to build CPython without JIT support on platforms where LLVM is not available/supported ? Yep, and it always will be (even on platforms where LLVM is available). ---
**Author:** ![NeilGirdhar](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/neilgirdhar/32/5776_2.png) [@NeilGirdhar](https://discuss.python.org/u/NeilGirdhar)\ **Post date:** [April 12, 2024, 6:17pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/9 "2024-04-12T18:17:22Z")
> [@brandtbucher](#): > > So I would either need to do a lot deeper dive into the source code, or provide a perhaps-not-detailed-enough summary based on my own incomplete understanding from various papers, blog posts, and other [resources](https://rpython.readthedocs.io/en/latest/jit/index.html) available online. I was just curious if there were any fundamental design differences. No need for a detailed comparison! ---
**Author:** ![kj0](https://avatars.discourse-cdn.com/v4/letter/k/db5fbb/32.png) [@kj0](https://discuss.python.org/u/kj0)\ **Post date:** [April 12, 2024, 7:22pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/10 "2024-04-12T19:22:01Z")
> - _Patents_: Are you aware of any patents on the copy-and-patch approach or other advanced techniques you are using in the JIT, which may encumber the use of a JIT enabled CPython ? IIRC, the trace trees technique or tracing JITs in general (I forgot which one) are patented by UC Irvine/ Andreas Gal and Michael Franz. But IIUC, they allow open source code to use it without getting sued. I am not a lawyer so please don’t take my word on this and actually consult a real expert. The US Patent number is `US 8,769,511 B2`. ---
**Author:** ![malemburg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/malemburg/32/50_2.png) [@malemburg](https://discuss.python.org/u/malemburg)\ **Post date:** [April 14, 2024, 11:20am UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/11 "2024-04-14T11:20:43Z")
Thanks for your answers, @mdroettboom and @brandtbucher > [@mdroettboom](#): > > _Patents_: We aren’t aware of any patents, and the copy-and-patch paper on which this is based is open access (which isn’t the same thing, but helps). However, I am not a lawyer – is there a typical process that happens when this is a concern? For companies using CPython this could be a concern (simply to avoid the risk of being sued for patent infringement) and it’s good to collect such knowledge, since good patent research is hard and expensive. I did a quick search on Google Patents, but the only patent which came up is this one: > **[US8099729B2 - Method and device for creating and using pre-internalized...](https://patents.google.com/patent/US8099729B2/en)** > > A device ( 45 ) receives new program files ( 46 ) and uses pre-internalized images to avoid having to internalize a program file every time that program execution occurs. In one embodiment, a software Virtual Machine ( 50 ) in the device... which only appears remotely related and is expired by now. Which doesn’t mean that there may be other patents or pending ones related to the used JIT techniques. The paper was written by Stanford scientists, so it wouldn’t be surprising to find that a patent application is underway. > [@kj0](#): > > IIRC, the trace trees technique or tracing JITs in general (I forgot which one) are patented by UC Irvine/ Andreas Gal and Michael Franz. But IIUC, they allow open source code to use it without getting sued. I am not a lawyer so please don’t take my word on this and actually consult a real expert. > > The US Patent number is `US 8,769,511 B2`. Thanks, @kj0 for the added information. Here’s the link to the patent: [US8769511B2 - Dynamic incremental compiler and method - Google Patents](https://patents.google.com/patent/US8769511B2/en) I tried to find our whether the UC has a policy of allowing open source projects using such patents, but wasn’t successful. This may actually be something the PSF could get involved in for the benefit of our (commercial) users, by setting up a patent license pool to which PSF sponsors would then get access. > [@brandtbucher](#): > > LLVM is used _at build time_ to compile individual micro-op instructions into mostly-opaque blobs of machine code. These blobs are dumped into a header file, which is used to build CPython itself. Thanks for clearing this up. So the comment in the “Rational” section was about using LLVM at runtime. You do use LLVM at build time to compile the templates, which are then copy-and-patched at runtime with the needed details, but not at runtime. BTW: I think it would make the PEP easier to follow, if you’d add a sentence explaining what _copy-and-patch compilation_ is all about, e.g. based on [the one you find on Wikipedia](https://en.wikipedia.org/wiki/Copy-and-patch): “Copy-and-patch compilation is a simple compiler technique that uses pre-written machine code fragments that are then patched to insert memory addresses, register addresses, constants and other parameters to produce executable code.” > [@brandtbucher](#): > > _RAM usage:_ All of the points you make above are things I want to do, but simply haven’t had the time to work on yet. Great, so these things are on the radar. I was just wondering, since I still remember how [Armin’s psyco](https://en.wikipedia.org/wiki/Psyco) used to have issues with significant RAM usage due to the JIT keeping too many traces in memory. > [@brandtbucher](#): > > > [@malemburg](#): > > > > - _Dependencies_: Assuming you are using LLVM, is it still possible to build CPython without JIT support on platforms where LLVM is not available/supported ? > > Yep, and it always will be Great. Thanks for the confirmation. ---
**Author:** ![daniele](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@daniele](https://discuss.python.org/u/daniele)\ **Post date:** [April 14, 2024, 1:18pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/12 "2024-04-14T13:18:34Z")
> [@malemburg](#): > > For companies using CPython this could be a concern (simply to avoid the risk of being sued for patent infringement) and it’s good to collect such knowledge, since good patent research is hard and expensive. Why is the JIT any more of a concern patent wise than any other part of CPython? ---
**Author:** ![malemburg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/malemburg/32/50_2.png) [@malemburg](https://discuss.python.org/u/malemburg)\ **Post date:** [April 14, 2024, 1:46pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/13 "2024-04-14T13:46:14Z")
> [@daniele](#): > > Why is the JIT any more of a concern patent wise than any other part of CPython? It is not, but since JIT technology has seen quite a few patents in recent years (esp. due to the Java and JS VMs heavily using JITs), and because the copy-and-patch technology sounds like a rather new and clever idea, this triggered my question 🙂 ---
**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\ **Post date:** [April 14, 2024, 4:19pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/14 "2024-04-14T16:19:13Z")
> [@malemburg](#): > > For companies using CPython this could be a concern (simply to avoid the risk of being sued for patent infringement) and it’s good to collect such knowledge, since good patent research is hard and expensive. “Hard, expensive, and valuable to corporations” sounds exactly like the type of work that corporations need to pony up for. 😉 Maybe some group of them would be willing to dedicate funds for such a thing. ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [April 15, 2024, 10:36am UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/15 "2024-04-15T10:36:08Z")
Could you perhaps mention [the slides](https://github.com/brandtbucher/brandtbucher/blob/master/2023/10/10/a_jit_compiler_for_cpython.pdf) in the PEP text, and make sure that PDF file is hosted in the PEPs repo itself, rather than in a personal GH repo that might become unavailable at some point? ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [April 15, 2024, 10:51am UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/16 "2024-04-15T10:51:42Z")
General question: since half of the tasks in [Improving JIT code quality · Issue #115802 · python/cpython · GitHub](https://github.com/python/cpython/issues/115802) seem already done on main platforms, by how much do you envision the remaining tasks to improve performance? It seems there is a general contradiction between working on tiny unit of works (micro-ops) and using a JIT compilation scheme that doesn’t optimize accross micro-ops. Is it possible to break out of this contradiction while still benefitting from the advantages of the copy-and-patch approach? ---
**Author:** ![diegor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/diegor/32/34816_2.png) [@diegor](https://discuss.python.org/u/diegor)\ **Post date:** [April 15, 2024, 3:40pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/17 "2024-04-15T15:40:06Z")
We are now at a point where a code execution can take different paths. Tier-1, tier-2, JIT… I found [this explanation](https://docs.python.org/3.13/whatsnew/3.13.html#whatsnew313-jit-compiler) easy to understand and I took the liberty to create a diagram from it. ![Screenshot 2024-04-15 at 16.34.28](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/d/7/d79191c9260bb75c4c21ecf5722757213f558131.png) **UPDATE: the above diagram is incorrect. Please scroll down to find the fixed version.** Maybe spend a few words (or even a simple diagram) to explain where the JIT fits in the whole picture. ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [April 15, 2024, 4:00pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/18 "2024-04-15T16:00:19Z")
Thanks for that diagram. I think there’s one decision point misplaced: the “hot enough?” question is only asked when _either_ the JIT is enabled, _or_ when the “uops” option is selected (via command line or env var). ---
**Author:** ![smontanaro](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/smontanaro/32/1389_2.png) [@smontanaro](https://discuss.python.org/u/smontanaro)\ **Post date:** [April 15, 2024, 4:39pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/19 "2024-04-15T16:39:35Z")
I’m confused by the use of the terms “tier 1” and “tier 2” in this diagram. I thought those terms referred to the level of support for different platforms. That doesn’t seem to be how they are being used in this diagram. Does a tier 1 platform become tier 2 part of the way through the decision bits related to JIT execution? ---
**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\ **Post date:** [April 15, 2024, 4:40pm UTC](https://discuss.python.org/t/pep-744-jit-compilation/50756/20 "2024-04-15T16:40:56Z")
It’s a totally different set of tiers: the tiers of optimization. There are only so many synonyms for “level”. [Next page](https://discuss.python.org/t/pep-744-jit-compilation/50756.md?page=2)