You’re completely correct here, the pre-PEP does not get us to Rust being ubiquitous in CPython. I think we probably didn’t do a good enough job explaining why the approach is so conservative, and should explain more on a potential path to make Rust ubiquitous, so let me expand on that.
I want to start by echoing @FFY00
We intentionally want to keep the initial test as conservative as possible for a few reasons. Right now, the impact of introducing Rust at all is not fully known, so we want to make sure that it is both easy to back out of the change if it is untenable, and understand what workflows would break when Rust becomes required. One thing this thread has reinforced for me is that CPython is used all over the place, often in places we the core team are not aware of. The only way we can properly evaluate the potential impact of making Rust required is to start warning users we hope to do so and hear about places it would break, and what is needed to change to make Rust required.
As for using Rust outside of extension modules, I think it could be used for experimental, opt-in interpreter features such as the JIT. I want to be cautious of this though because I don’t want the JIT to be in the state where it should become the default but making Rust required is a blocker to that. As everything else, it should be considered if migrating to Rust makes sense for the JIT.
I’d also like to cite an article by Google researchers:
There is a temptation, when introducing a new language, to pursue an aggressive timeline to justify the investment with short-term gains. Such an approach, fixated on overly ambitious goals, or rapid, sweeping change, invariably carries elevated risks of failure and can actively disincentivize future adoption by disrupting roadmaps and competing with other business objectives.
A more powerful strategy, the one that proved so effective for Android, is to treat language adoption as a long-term investment in sustainable, compounding growth that supports other business objectives instead of competing with them. This approach patiently accepts initially lower absolute numbers to provide the necessary time for the new language to establish a foothold, build momentum, and achieve critical mass.
In summary, for Android they found that moving to Rust is a long-term investment, that may not see immediate payoffs, but that moving to Rust is a long term success with significant payoffs. I realize this is a hard sell. But to me it makes a lot sense because there is a lot of learning and investigation to do related to integrating Rust into CPython. We need to spend the early period where Rust is allowed conservatively to build out the experience and support infrastructure for making Rust for CPython succeed.
I could see a path where we start very conservatively with base64, then introduce a Rust version of a more central standard library module like json (as an optional replacement to the C version to start with), then make Rust required. But this is a guess at a plausible path forward, and the final path needs to be informed from experience.
I would push back against this. One of the theses of this pre-PEP is that contributors are not inclined to write complex C code because it is difficult to do, thus projects like a C accelerator are not getting written.
As an example, the performance of the json module has been significantly slower than other implementations for a long time. Yet a re-write in C would be daunting. I think Rust would excel in cases like this.
Another example is actually the base64 module! `base64` module: Link against SIMD library for 10x performance. · Issue #124951 · python/cpython · GitHub
We could use the Rust base64-simd crate for an easier, significant improvement to performance.
Wholeheartedly agree with this, and that’s the goal with base64. We want to start somewhere, to better inform efforts elsewhere. Starting simply and minimally seems like the best path to allowing us to get an understanding without risk.
I think on the contrary we will see the greatest benefits for memory safety from making new code in Rust. A usenix paper found that the majority of vulnerabilities are in new code, as older code is battle-tested. So while I think we should enable replacing C code with Rust code where it makes sense, for example the json module, I think we will still see plenty of wins in new code too.