It’s also not clear that py-ni is an “official” successor[1]. Given that the Github page hasn’t been updated since December, it’s not clear if that project is abandoned either.
Given all that, it’s not surprising that other efforts aren’t keeping it at front of mind.
as opposed to someone else’s version of the same idea ↩︎
I’ll append this by noting that PyNI uses the HPy discord server, which makes the name change more obvious. But it’s definitely not clear on the HPy site, which claims to be active.
I read something about it. It will be really great. Does it cover all the HPy work?
I will really happy to see it landed in CPython. I will bring some nice shrubberies 
It’s probably worth being realistic about what PyNI actually is. In that it first it isn’t really anything concrete yet. It’s just a broad ambition to add a more JIT/alternative interpreter friendly C API. With HPy as a starting point to learn from.
Obviously it only has benefits to any JIT project if enough people switch to it and enough time passes (which is probably of the order of at least 5-10 years).
So it maybe doesn’t help the JIT project at the rate they would like to progress. But you could imagine gradually worrying less about the performance of the existing C API if enough of the big users had switched to a newer one.
Just thinking out loud here, but if you create a demonstration comparing JIT performance with and without the use of PyNI (or hpy), a substantial performance improvement for the latter might encourage faster adoption. I, of course, speculate about this knowing nothing about how difficult this experiment might be to perform.
No, but the parts of HPy that it doesn’t cover can be (and usually were) added to CPython as normal issues or individual smaller PEPs.
Unfortunately, the benefit comes from “when everyone is using [new interface] we can then change CPython internals in a way that would break [old interface]”. So we genuinely can’t tell unless someone forks CPython, makes all the breaking changes, and shows that it’s faster. Which people actually do all the time, but then struggle to integrate back into the original project.
Would that really be true? Couldn’t you – in theory – convert some key extensions , like numpy, which are heavily involved in specific applications and measure the JIT performance between the two? I’m skeptical that you’d really have to completely convert all of CPython to decide whether to pursue the larger conversion.
In the absence of conversion of a key extension like numpy, could you pick off specific objects within CPython and measure microbenchmark performance changes?
I’m thinking of experiments which serve more like gateway drugs than reimplementing an entire international drug cartel. 
Edit: I should perhaps point out that both hpy and PyNI advertise the ability to work in conjunction with Python.h, so incremental conversion should be possible.
Sure, there was an experiment with using unboxed integers which showed a lot of promise, at the cost of breaking every single extension that hasn’t been rewritten for the new interfaces. The benefits are there, it’s just the cost is not worth it.
This is true, but this incremental conversion is on the extension side, not the runtime side. The runtime can’t break any of the old interface as long as any extension is using it, but extensions can choose whether to use the new or old interfaces. We just don’t get the benefit until they’re all using the new interfaces all the time.
I think this is true and in light of this, perhaps the HPy / PyNI discussion should be split off into a new thread?
Can’t CPython offer two APIs for them for allowing the transition? I realize this is difficult, considering also the fact that the code now is much more complicated because of the nogil, the multiintepreters, the JIT etc. But maybe introducing the PyNI interface as an alternative will solve the problem of using a direct pointer instead of a handle.
+1
This is not a direct response to what you particularly said Steve rather what you said fits well with what I’ve been trying to articulate since the thread was started.
In the past I’ve made the choice to use pypy for a fairly high load service and we had to go through the cost / benefit conversation there. In our particular use the benefit was between 2 and 6 times on the code that mattered to us most but we were very aware of the significant slowdowns in other code paths and had to deal with that through some nasty complications. That difference was enough for us to make a call on a tradeoff but looking at something like 10+/- a few % improvements leans very heavily to a need for nothing to be slower, the benefit just doesn’t feel enough to justify the hassle of the potential downsides (depending on what is affected).
2x speed allows for significant hardware reduction or performance increase or both, 10% just doesn’t do that and if we need to navigate a happy path then it’s definitely not a no brainer to go that way.
None of this is suggesting getting a very small single digit max slow down is an easy thing at all and any improvement is very much appreciated, just that a 5% with no slow down is likely nicer, in my general case, than 10% with some major slowdowns.
Having the benefit of only being a user of extensions I’d buy into needing extensions to update if the benefit was there, I’m far from qualified to know if any changes will impact performance but it does feel like a trade off worth pursuing if it’s not already understood what the possible improvements are.
Yes, but you don’t get the benefit as long as the old API exists (bear in mind that the old API is “dereference the integer without type checking to modify its reference count”, which is entirely incompatible with unboxed integers, which can’t be dereferenced).
So we can let people write code now that will do the right steps in the right way, and those people won’t have to modify their code to adopt a new implementation, but the new implementation can’t be made broadly available until the old API is gone.
I’m repeating myself now. If anyone feels like questioning this issue further, please re-read my earlier posts and then open a new discussion (without tagging me - there are plenty of other people who can explain this).
Adding a followup here for Fedora.
First of all relying on a very specific LLVM version doesn’t seem ideal, usually compiler attributes are checked for specific features (e.g. the tail call interpreter) but JIT seems to be gated on a simple version string.
Moreover making the jit_shim object linked to libpython through Supporting stack unwinding in the JIT compiler · Issue #126910 · python/cpython · GitHub creates the challenge of having to take into account that the final binary is affected on said object’s properties (e.g. linker’s AND-merge), so for the case I stumbled on was the hardware protections that got dropped because of that: JIT shim object drops GNU property notes (CET/BTI/PAC) from output binaries · Issue #149697 · python/cpython · GitHub
I found myself diving deep into JIT code trying to figure out a solution but due to its complexity I’d basically need to familiarize myself with the whole JIT infra to resolve it and basically re-design some things. Maybe not a bad idea per se, but I wouldn’t say the JIT would be considered experimental if it can affect the security or other attributes of the final binary.
Similarly, I managed to break the JIT CI for Mac when working on the Perf profiling code: gh-149800: Split perf trampoline assembly into per-architecture files by stratakis · Pull Request #149894 · python/cpython · GitHub
Which again makes me having to deal with the JIT in unexpected ways, not too big of a nuisance in the specific case but it further supports my point that the JIT cannot be considered so experimental since it’s been quite embedded in some aspects, that unrelated changes can break other things around.
I don’t know what the best way forward should be and it’s not an easy situation to resolve. I’d be happy to dive myself more into JIT and try to see if I can resolve potential issues but a clear path forward would make such a decision easier.
Edit for something I forgot:
I’m currently working on the 2nd part of my proposal Refactor perf trampoline proposal: split the assembly per architecture, auto-generate DWARF unwind data · Issue #149800 · python/cpython · GitHub . Knowing if the JIT stays in core would help determine if I can use some of the JIT infra here or not.
Just a small point about HPy/PyNI — if the default JIT is sufficiently performant, it will start to make sense to rewrite things in Python. This tends to happen with JITs that are a bit further along like YJIT, PyPy, and hotspot to name a few.
This is all to say, the problem of native extensions being black boxes for JITs may take care of itself if the community is sufficiently motivated to replace them with pure-Python versions (or to load a pure-Python version when the JIT is on).
It’s a very common one. However, many times it’s because it’s compared to statically-compiled programming languages, which is unfair IMO: (C)Python was built for different needs/purposes/use-cases, and it works very differently.
CPython never had the luxury of spending 50% of (a huge amount of) time just for the instruction selection phase, fo example (LLMV/Clang): it has to startup quickly, and spend as less time as possible to parse & compile the source code, before starting the real execution.
One possible improvement from this PoV is providing a way (standalone/different tool?) to statically compile what’s possible (from the main source to all its statically-determined dependencies), and generating well-optimized files (.pyx?) directly in native/binary format (x86-64, AArch64, …). Basically, it’s a AoT compiler which takes all time to evaluate the sources, possibile dependencies, with the possibility to generate different code paths according to some guessed key data-types. But it’s a completely new project which requires a lot of effort & investments.
Another thing which is way simpler to implement is caching the generated native code for the associated bytecode fragments by saving them into a proper .pyx files, and load them at runtime (instead of the usual bytecode?). Modern architectures natively support PIC code, so only patching of some addresses might be required.
An idea which I had some time ago is to use one of the available cores in modern processors and use/reserve it to “offline compile” a bytecode once it’s generated, by adding it to a queue. The extra core wakes-up from time to time, and if the queue has some element then it starts compiling the given bytecode and store it into the same place where the JIT usually stores binaries, so that the next time that it’s invoked then it can immediately executed. Paired to the previously mentioned caching, it should be effective on converting all bytecodes to native binaries. Another core might be used for a queue which is dedicated only to “hot” loops/functions, with the sole purpose of deeply evaluating the bytecode to generate much more optimized native code.
A drive-by know-nothing comment: As a Python user I would be deliriously happy with a slightly less dynamic Python, if that was what was required to unlock the performance gains JITs in other contexts are capable of.
from __compiler__ import <list-of-compilation-options>
where the items in the list could be:
annotations_as_types → enforces the use of annotations as types, where deemed appropriate.
static_builtins ->All built-ins are “frozen” within the module and cannot be changed. This would make it possible to generate much more optimised code right at the bytecode level. E.g.: len(s) could generate specific bytecode that immediately calls the corresponding built-in function (or generate bytecode that reads the size field directly from the PyVarObject, if it already knows that s is an internal CPython type that “descends” from this object). By extending the import syntax slightly, one could select only the built-ins to be frozen (leaving all others in the usual built-in namespace):
from __compiler__ import static_builtins[int, float, complex, str, len]
Or to freeze them all except for a few:
from __compiler__ import static_builtins-[file, open]
frozen_globals → “freezes” all global variables at the end of the module’s initialisation. This makes it possible to access them directly using predefined array indices, completely bypassing dictionary hashing and lookup.
i8, i16, i32, i64, u8, u16, u32, u64 → signed and unsigned integer types of a specific size.
s_ascii, s_utf8, s_utf16, s_utf32, c_str → specific string types
single_inheritance → Classes can only be derived from a single base class (and, where applicable, from classes that act solely as interfaces), so as to utilise the classic VMT model, which is simpler and more efficient.
And so on. The possibilities are considerable, and can be further expanded. Obviously, this directive would also override the corresponding command-line or ENVAR parameters specifying the same settings, allowing you to selectively choose, in particular modules, to deviate from the global settings.
I don’t see the problem, as long as the JIT code can be easily removed from CPython.
mypyc does this to a certain degree: you feed it a mypy-checked (in strict mode) python file and it gets compiled into a C extension.
For those kind of performance improvements I think that that’s a reasonable path to consider.
Of course that’s only talking in terms of AoT compilation. I’m not sure how a JIT could leverage this.
Many widely used libraries rely on the dynamic nature of python to provide their functionality. For example, pydantic, sqlalchemy, Django, pandas depend heavily on dynamic features of python. While they frequently optimize their core functionality without this dynamic nature, their APIs are dependent on dynamic features.
While it might be possible to scope the “slightly less dynamic Python” to specific builtins, modules, functions, etc, I question whether this could be achieved in a way with meaningful benefits without precluding the use of libraries that expose APIs that rely on pythons dynamic nature. Would doing this bifurcate the python ecosystem into static/fast vs dynamic/slow? Is it possible to scope in a way that doesn’t overly restrict future development options?
It is certainly an interesting idea to reduce JIT complexity and enable optimizations, but it is such a fundamental change to the language I’m skeptical it would be compatible with most existing python code. I love doing greenfield development, but performance critical greenfield projects are highly unlikely to choose python, even with JIT improvements enabled by scoping/limiting dynamic capabilities.