Pre-PEP: Rust for CPython

[emmatyping (Emma Smith)]

Regarding optional extension modules, here’s a rough timeline I could see and why each step is chosen to be so.

1. The goals of this stage are to get the initial build system changes set up, and start getting experience interfacing with the C code from Rust. Extension modules are a limited interface to the interpreter which minimizes the work to do for interoperability. Ideally we would also start warning users that we plan to eventually make Rust a hard requirement, and if this is a hardship for their environment, to please relate that to us. Only optional extension modules are allowed. Even more of a limitation, they should only be introduced where they would replace a C extension module (this could be things like base64 or json). The latter restriction exists to ensure we are not limiting access to new features to those who have Rust available. I picked the base64 module because it was simple but should exercise enough parts of the build system and C interop code, but we could just as well choose the json module.
2. The goals of this stage are to make progress on improving portability and bootstrapping workflows based on feedback from stage 1. If we don’t gather feedback in stage 1 we could also start doing so here. More modules may be ported here to get even more experience with the C <—> Rust interfacing and overall developer experience.
3. The goals of this stage are to ensure that we have resolved bootstrapping issues and warn of the impending requirement on Rust. Make Rust an opt-out feature of the interpreter. This is an even stronger warning of the up-coming requirement. At this point the vast majority of users should be able to build CPython with Rust enabled and bootstrapping issues should be resolved to greatest extent possible. I would say at this point new extension modules can be in Rust.
4. The goal of this stage is to integrate Rust into core parts of CPython. Rust is required to build CPython. Rust can now be used across the CPython code base.

This is one hypothetical scenario. We’re not going to propose this as the specification in the PEP.

@Jelle

My thoughts are that a proposal that allows Rust only for optional extension modules is likely not useful enough to justify the cost of adding a new language. It can be a first step (in fact, it’s a very reasonable first step!), but that means there must be a plan to expand beyond it just being an optional part of building CPython.

There is simply too much to consider to resolve the bootstrapping and portability problems to have a concrete plan now. I think like PEP 703 did, we could leave an open question with some version of the above roadmap. But I don’t think we have enough information at the moment. The only way to tell what the impact of introducing Rust to CPython is to introduce Rust to CPython. In the above plan we start very conservatively to ensure we don’t break anyone’s workflows until we’re confident we can introduce Rust to core. I wouldn’t want to set hard timelines to any of this because we don’t know now when we should move to the next step in the process. So I think like free-threading, it would be best to start with step 1, then have a follow-up PEP when we are confident we have the information we need to move to the other steps.