(Posting this as myself, not as a Steering Council member.)
Thanks for this PEP, I enjoyed reading it, and I have lots of thoughts on the JIT design.
However, this really feels like (at least) two PEPs, and I think it should be split up. I think I understand you want to show that the criteria for supported status are attainable even if we aren’t particularly close right now, but honestly, I don’t think that should matter for the simple goal of setting acceptable criteria. I also don’t think it’s a good idea to mix the two fairly disparate discussions, about the criteria and about how to get there. Fundamentally, the discussion about the minimum expectations for a supported JIT is distinct from how to get there, especially as we have different proposals for how to get there.
We can decide on the criteria for declaring the JIT supported without deciding on how to get there, and we can decide on a different approach for the JIT (like tracing versus method JIT) without it affecting the criteria at all.
The criteria for supported status as laid out in the PEP look pretty good to me. I think the strong requirement on free-threading support, the JIT distribution story, the maintenance and tooling and platform support, all of that is important, and I don’t disagree with anything mentioned in the PEP about those.
The timeline given in the PEP is, I think, a little out of place. It combines the performance goals with the implementation paths for the new JIT ideas, and I can’t help but point out that past predictions for performance progress have rarely panned out precisely as predicted. I don’t think we should expect a firm timeline for JIT development. It’s just not how development works in an open-source project, especially not one that cares about backward compatibility.
I think the PEP should encapsulate the expectations the community has for the JIT’s benefits and costs, in the form of +X% on benchmarks Y and Z, no more than -U% penalty on startup/memory according to benchmarks V and W. Those goals should be realistic, but they don’t have to be certain, and they don’t have to come with an implementation plan. The purpose is to set some minimum expectations, some goals for the JIT work to aim for. It may very well be that those goals are unattainable at this time, if it turns out that people actually care way more about memory use or startup time or maintainability than they do about performance. Maybe the community wants to see +50% performance before it’ll consider it acceptable to include the JIT in CPython. If that’s the end result of the discussions, then so be it.
I also think the goals should be aspriational, not absolute, and in the end the Steering Council at the time should decide whether the criteria have been sufficiently met.
Personally, I think +20% geometric mean over pyperformance is an excellent goal, although I will point out that pyperformance is far from a rigorous or meticulous collection of benchmarks. (For instance, at the moment, asyncio is quite overrepresented in the result set, a lot of benchmarks run so quickly it’s hard to get accurate results, and there’s a handful of benchmarks that are genuinely bad.) We can’t just hit that goal by changing pyperformance to suit the JIT of course, but I do think we should be open to work on making pyperformance more reflective of real-world programs.
I also think the PEP should be explicit about startup time, warmup time, and memory overhead. Some of that is reflected in pyperformance as it is, but I think they deserve separate, explicit measurements. I don’t think these have to be particularly strict numbers – the JIT is still optional even if it’s supported – but if we don’t track them we don’t know what cost we’re actually paying here.
The other big requirement is, of course, maintainability, or support among the Core team. This is very difficult to quantify and I can see how the talk about the JIT implementation is supposed to address this. Using a stack based JIT makes sense if you look at it from the view point of a CPython developer familiar with the stack based interpreter who needs to learn about the JIT. I do not think that this is important. We should not design our architecture to be most understandable to current Core team members. Instead, we should focus on making whatever design we end up with understandable, and understood by the Core team. We need to make sure enough people work on it and spend time with it. We actually discussed this notion briefly among people working on CinderX, and @mpage and @kddnewton pointed out that attempting to design a JIT to be more easily understood by Core team members makes it less easily understood by others, and makes it harder to apply literature and lessons learned from other JITs to CPython’s.
I will have more thoughts on the JIT implementation discussion, but I need more time to articulate them… In the interest of word count in this post, and given the suggestion to split up the discussion, I’ll hold on to them for now 