Pre-PEP: Rust for CPython

Introduction

We (@emmatyping, @eclips4) propose introducing the Rust programming language to CPython. Rust will initially only be allowed for writing optional extension modules, but eventually will become a required dependency of CPython and allowed to be used throughout the CPython code base (see update here about requiring Rust).

Motivation

Rust has established itself as a popular, memory-safe systems programming language in use by a large number of projects. Memory safety is a property of a programming language which disallows out-of-bounds reads and writes to memory, as well as use-after-free errors. Rust’s safety properties are enforced by an ownership mode for code, which ensures that memory accesses are valid. Rust’s memory safety guarantees have been formally proven by the RustBelt project for code that does not use “unsafe” . By adopting Rust, entire classes of bugs, crashes, and security vulnerabilities can be eliminated from code.

Due to Rust’s ownership model, the language also enforces thread safety: data races are prevented at compile time. With free-threaded Python becoming officially supported and more popular, ensuring the standard library is thread safe becomes critical. Rust’s strong thread safety guarantees would ease reasoning around multi-threaded code in the CPython code base.

Large C/C++ based projects such as the Linux kernel, Android, Firefox, and many others have begun adopting Rust to improve memory safety, and some are already reporting positive results from this approach. Furthermore, Rust has become a popular language for writing 3rd-party extension modules for Python. At the 2025 Language Summit, it was mentioned that 25-33% of 3rd-party Python extension modules use Rust, especially new extension modules. By adopting Rust in CPython itself, we expect to encourage new contributors to extension modules.

CPython has historically encountered numerous bugs and crashes due to invalid memory accesses.We believe that introducing Rust into CPython would reduce the number of such issues by disallowing invalid memory accesses by construction. While there will necessarily be some unsafe operations interacting with CPython’s C API to begin with, implementing the core module logic in safe Rust would greatly reduce the amount of code which could potentially be unsafe.

Rust also provides “zero-cost”, well designed implementations of common data structures such as Vectors, HashMaps, Mutexes, and more. Zero cost in this context means that these data structures allow implementations to use higher-level constructs with the performance of hand-rolled implementations in other languages. The documentation for the Rust standard library covers these data structures very thoroughly. By having these zero-cost, high-level abstractions we expect Rust will make it easier to implement performance-sensitive portions of CPython.

Rust additionally enables principled meta-programming through its macro system. Rust has two types of macros: declarative and procedural. Declarative macros in Rust are hygienic, meaning that they cannot unintentionally capture variables, unlike in C. This means it is much easier to reason about macros in Rust compared to C. Procedural macros are a way to write Rust code which does token transformations on structure definitions, functions, and more. Procedural macros are very powerful, and are used by PyO3 to ergonomically handle things like argument parsing, class definitions, and module definitions.

Finally, Rust has an excellent build system. Rust uses the Cargo package manager, which handles acquiring dependencies and invoking rustc (the Rust compiler driver). Cargo supports vendoring dependencies out of the box, so that any Rust dependencies do not need to be downloaded (see open questions about dependencies). Furthermore, Rust has easy to set up cross-compilation which only requires installing the desired target and ensuring the proper linker is available.

In summary, Rust provides many extremely useful benefits that would improve CPython development. Increasing memory safety would be a significant improvement in of itself, but it is far from the only benefit Rust provides.

Implementation

The integration of Rust would begin by adding Rust-based extension modules to the “Modules/” directory, which contains the C extensions for the Python standard library. The Rust modules would be optional at build time, dependent on if the build environment has Rust available.

Integrating Rust with the CPython C API requires foreign function interface (FFI) definitions in Rust to describe the APIs available. A new crate (a library in Rust terminology) cpython-sys will be introduced to handle these FFI definitions. To automate the process of generating Rust FFI bindings, we use bindgen. Bindgen is not only an official Rust project, but also used ubiquitously throughout the Rust ecosystem to bind to C APIs, including in the Linux and Android projects. Bindgen uses libclang at build time to read C headers and automatically generate Rust FFI bindings for the current target. Unfortunately, due to the use of C macros to define some constants and methods, the cpython-sys crate will also need to replicate a few important APIs like PYOBJECT_HEAD_INIT manually. However these should all be straightforward to replicate and few in number.

With the C API bindings available in Rust, contributors can call the FFI definitions to interact with CPython. This will necessarily introduce some unsafe Rust code. However extension modules should be written such that there is a minimal amount of unsafe at the edges of FFI boundaries.

Eventually safe abstractions to simplify and standardize code like module definitions, function argument parsing, and class definitions could be adopted to reduce the amount of raw FFI code written.

A reference implementation which includes a proof-of-concept _base64 module which uses Rust to provide a speedup to base64 is available.

Distribution

Rust supports all platforms which CPython supports and many more as well. Rust’s tiers are slightly different, and include information on whether host tools (such as rustc and cargo) are provided. Here are all of the PEP 11 platforms and their corresponding tiers for Rust:

Platform Python Tier Rust Tier
aarch64-apple-darwin 1 1 with Host Tools
aarch64-unknown-linux-gnu (gcc, glibc) 1 1 with Host Tools
i686-pc-windows-msvc 1 1 with Host Tools
x86_64-pc-windows-msvc 1 1 with Host Tools
x86_64-unknown-linux-gnu (gcc, glibc) 1 1 with Host Tools
aarch64-unknown-linux-gnu (clang, glibc) 2 1 with Host Tools
wasm32-unknown-wasip1 2 2 without Host Tools
x86_64-apple-darwin 2 2 with Host Tools
x86_64-unknown-linux-gnu (clang, glibc) 2 1 with Host Tools
aarch64-linux-android 3 2 without Host Tools
aarch64-pc-windows-msvc 3 1 with Host Tools
arm64-apple-ios 3 2 without Host Tools
arm64-apple-ios-simulator 3 2 without Host Tools
armv7l-unknown-linux-gnueabihf (Raspberry Pi, gcc) 3 2 with Host Tools
aarch64-unknown-linux-gnu (Raspberry Pi, gcc) 3 1 with Host Tools
powerpc64le-unknown-linux-gnu 3 2 with Host Tools
s390x-unknown-linux-gnu 3 2 with Host Tools
wasm32-unknown-emscripten 3 2 without Host Tools
x86_64-linux-android 3 2 without Host Tools
x86_64-unknown-freebsd 3 2 with Host Tools

In summary, every platform Python supports is supported Rust at tier 2 or higher, and host tools are provided for every platform other than those where Python is already cross-compiled (e.g. WASI and mobile platforms).

Rejected Ideas

Use PyO3 in CPython

While CPython could depend on PyO3 for safe abstractions over CPython C APIs, this may not provide the flexibility desired. If a new API is added to the C API, it would need to be added to PyO3, then the version of PyO3 would need to be updated in CPython. This is a lot of overhead and would slow down development. Using bindgen, new APIs are automatically exposed to Rust.

Keep Rust Always-Optional

Rust could provide many benefits to the development of CPython such as increased memory safety, increased thread safety, and zero-cost data structures. It would be a shame if these benefits were unavailable to the core interpreter implementation permanently.

Open Questions

How should we manage dependencies?

By default cargo will download dependencies which aren’t already cached locally when cargo build is invoked, but perhaps we should vendor these? Cargo has built-in support for vendoring code. We could also cargo fetch to download dependencies at any other part of the build process (such as when running configure).

How to handle binding Rust code to CPython’s C API?

The MVP currently uses bindgen, which requires libclang at build time and a number of other dependencies. We could pre-generate the bindings for all supported platforms, which would remove the build-time requirement on vendoring bindgen and all of its dependencies (including libclang) for those platforms.

When should Rust be allowed in non-optional parts of CPython?

This section is out of date, please see Pre-PEP: Rust for CPython - #117 by emmatyping. Leaving the original for the record.

Given the numerous advantages Rust provides, it would be advantageous to eventually introduce Rust into the core part of CPython, such as the Python string hasher, SipHash. However, requiring Rust is a significant new platform requirement. Therefore, we propose a timeline of:

  1. In Python 3.15, ./configure will start emitting warnings if Rust is not available in the environment. Optional extension modules may start using Rust
  2. In Python 3.16, ./configure will fail if Rust is not available in the environment unless --with-rust=no is passed. This will ensure users are aware of the requirement of Rust on their platform in the next release
  3. In Python 3.17, Python may require Rust to build

We choose this timeline as it gives users several years to ensure that their platform has Rust available (most should) or otherwise plan for the migration. It also ensures that users are aware of the upcoming requirement. We hope to balance allowing time to migrate to Rust with ensuring that Rust can be adopted swiftly for its many benefits.

How to handle bootstrapping Rust and CPython

Making Rust a dependency of CPython would introduce a bootstrapping problem: Rust depends on Python to bootstrap its compiler. There are several workarounds to this:

  1. Rust could always ensure their bootstrap script is compatible with older versions of Python. Then the process is to build an older version of Python, build Rust, then build a newer version of CPython. The bootstrap script is currently compatible with Python 2, so this seems likely to continue to be the case
  2. Rust could use PyPy to bootstrap
  3. Rust could drop their usage of Python in the bootstrap process

I (@emmatyping) plan to reach out to the Rust core team and ask what their thoughts are on this issue.

What about platforms that don’t support Rust?

Rust supports all platforms enumerated in PEP 11, but people run Python on other operating systems and architectures. Reviewing all of the issues labeled OS-unsupported in the CPython issue tracker, we found only a few cases where Rust would not be available:

  1. HPPA/PA-RISC: This is an old architecture from HP with the last released hardware coming out in 2005 and a community of users maintaining a Linux fork. There is no LLVM support for this architecture.
  2. RISC OS: This is a community maintained version of an operating system created by Acorn Computers. There’s no support in Rust for this OS.
  3. PowerPPC OS X: This older OS/architecture combination has a community of users running on PowerBooks and similar. There is no support in Rust for this OS/architecture combination, but Rust has PowerPC support for Linux.
  4. CentOS 6: Rust requires glibc 2.17+, which is only available in Centos 7. However, it is unlikely users on a no-longer-supported Linux distribution will want the latest version of CPython. Even if they do, CPython would have a hard time supporting these platforms anyway.

How should current CPython contributors learn/engage with Rust portions of the code base?

Current contributors may need to interact with the Rust bindings if they modify any C APIs, including internal APIs. This process can be well covered in the devguide, and there are many great resources to learn Rust itself. The Rust book provides a thorough introduction to the Rust programming language. There are many other resources in addition, such as Rust for C++ programmers and the official learning resources Learn Rust - Rust Programming Language.

To ease this process, we can introduce a Rust experts team on GitHub who can be pinged on issues to answer questions about interacting with the API bindings. Furthermore, we can add a Rust tutorial focused on Rust’s usage in CPython to the devguide.

Obviously any extension modules written in Rust will require knowledge of Rust to contribute to.

What about Argument Clinic?

Argument Clinic is a great tool that simplifies the work of anyone writing C functions that require argument processing. We see two possible approaches for implementing it in Rust:

  1. Adapt the existing Argument Clinic to parse Rust comments using the same DSL as in C extensions, and generate Rust function signatures.
  2. Create a Rust procedural macro capable of parsing a similar DSL. This approach might allow it to be used by any third-party package, whereas the C-based Argument Clinic does not guarantee compatibility with third-party extensions.

Using a proc macro would allow for better IDE integration and could become useful to 3rd party extension authors.

Should the CPython Rust crates be supported for 3rd-party use? Should there be a Rust API?

Having canonical Rust crates for bindings to CPython’s C API would be advantageous, but the project is ill-prepared to support usage by 3rd-parties at this time. Thus we propose deferring making Rust code supported for 3rd-party use until a later date.

82 Likes

Isn’t the experience in the linux kernel with adding rust support as a core part more a cautionary tale? At least it looks that way from the outside. There is constant friction between the two different “types of programmers”, causing a lot of quite public disagreement. If rust becomes a requirement for core devs, might that scare away some current maintainers who don’t want to deal with it?

12 Likes

Hello, and thanks for your question!
We will try to make this transition as smooth as possible. We’re planning to write a “migration guide” for CPython contributors to help make their adoption of Rust as easy as possible.
Can this scare current maintainers? Yes.
Will we do everything we can to make sure it doesn’t? Absolutely.

22 Likes

I’m not a core dev nor expert in the internals of CPython, but I wanted to chime in to resonate with the question from @MegaIng , pointing out though that it looks to me (as a Python user) that the community in general is more “approachable” in comparison to what happened during the integration of some Rust in the Linux kernel.

Out of curiosity, what would this mean for both PyO3 and RustPython? Have you reached out to the mantainers of the latter for feedback on how to approach this possibility? If so, what’s their opinion? It looks to me like this possible PEP might benefit greatly from their experience (I don’t know if you guys are part of their team, I’m just assuming you’re not but I apologize if that’s incorrect).

Also, I’m probably jumping the gun with this question, but since also RustPython seems to be implementing the GIL in a similar fashion, would you expect some challenges in applying PEP 703 efficiently for rust-based extensions? I imagine (or rather, hope) that by the time 3.17 is out that CPython will be completely GIL-less by then. Do you expect that this additional feature will be provided smoothly in Rust extensions as well?

Same question applies for the new experimental JIT which should be more stable in future releases.

Finally, if I recall rust applications tend to be a bit “bloated” in binary size, although there are some tricks that can be done at compile time to reduce this - what do these tricks imply on performance I have no idea.

5 Likes

As a general direction, I’d rather see “optional extension modules”[1] living outside the main repo and brought in later in the build/release process. Presumably such modules have no tight core dependency, or they wouldn’t be able to be optional, and so they should be able to build on pre-release runtimes and work fine with released runtimes (as we expect of 3rd party developers).

The bootstrapping position is discussed in the proposed text and determined to be viable. I think this position applies equally well to bootstrapping the modules we theoretically expect people to write that rely on non-core dependencies.

Merging in additional modules as part of our release process is fine. Bundling additional sources and optional build steps into the source tarball is fine (if we think that Linux distros will just ignore us when we say “you should include these modules in your Python runtime”).

For what it’s worth, I believe that the parts of CPython that directly depend on OpenSSL and Tcl/Tk are also fit for this. So don’t see my position as being about Rust itself[2] - it’s about the direction we should be taking with adding new modules to the core runtime.

In short, adding new ways to add new, non-essential modules to the core is counter to the approach we’ve been taking over the last few years. So I’m -1 on adding one.


  1. Also optional regular modules. ↩︎

  2. If you want my position about Rust itself, well… it’s not going to help your PEP at all :wink: ↩︎

19 Likes

As I think many folks know, I’ve been quite involved in these sorts of efforts (starting the effort that became Rust for Linux at the PyCon sprints, migrating pyca/cryptography to Rust, and helping to maintain pyo3). As a result, it will come to no one’s surprise that I’m very supportive of this pre-PEP :slight_smile:

I’m happy to answer any questions folks have about those experiences and lessons learned.

41 Likes

As a Rust fan I think this is super cool! I do wonder if this is too much to figure out in one PEP, when parts of it seems pretty easily separable. I can see how the overall vision fits together, but it’s a lot.

Would it make sense to restrict the initial proposal as just “create the cpython-sys crate and allow optional Rust extensions in the stdlib”? That would allow iteration on making a good generic interface[1] without requiring the SC to make a decision on the long-term plan.


  1. there is plenty of prior art in PyO3, and I think the HPy project has some relevance there too ↩︎

5 Likes

I would feel more comfortable with this if it was dependent on custom json targets stabilizing in Rust. Right now, targetting platforms that Rust itself doesn’t support still requires nightly, and python is used to bootstrap a lot of things in a lot of places.

I don’t think the impact of this is going to be reasonably predictable, and I’d want there to be a stable upstream escape hatch for those in unusual situations.

6 Likes

Doesn’t “C“ in the name of “CPython“ means “C programming language”? If so, shouldn’t the project be eventually renamed when a significant part of it is (re)written in Rust?

/joke, but who knows

13 Likes

I’ll start by saying my opinion doesn’t matter here: the number of lines of C I’ve written that’s in the Python interpreter is of the order of 10, so whatever you decide it’s unlikely to be my personal problem.

I’m pretty convinced of Rust as a nice language to write cool new things in (and of the value of exposing cool those cool new things to be used from Python). I’m less convinced of the benefit of rewriting existing working things in Rust.[1]

So from my point of view this PEP doesn’t offer a lot - it’s largely proposing “rewrite some modules in Rust with a view to soften people up to rewrite more in Rust”. I’d be much more convinced by a PEP that wanted to add something useful to the standard library and had found the best way to do that was to use a Rust library.


Minor comment:

I suspect this isn’t viable for a couple for reasons: PyPy isn’t hugely well maintained right now, and (I believe) PyPy needs a Python interpreter to bootstrap itself so may not solve the problem.


  1. And I think some of the push-pack Rust gets is from when people do this. It’s hard to argue with the benefit of a new and exciting feature but easy to be unimpressed with “what we already have but in Rust”. ↩︎

21 Likes

This example shows that you really want the safe abstractions for this to be useful. Otherwise you’re doing the same kind of tedious, unsafe, error-prone low-level pointer juggling and manual reference cleanup as in C (hello Py_DecRef :cry:).

6 Likes

Snarky note: do we eventually have to rename CPython to CRPython? :slight_smile:

Seriously, I think this is a great development. We all know that a full rewrite in Rust won’t work, but starting to introduce Rust initially for less-essential components, and then gradually letting it take over more essential components sounds like a good plan.

I trust Emma and others to do a great job of guiding the discussion over the exact shape that the plan and the implementation should take.

62 Likes

As a PyO3 maintainer IMO it’d be really nice if we could get rid of the need for the pyo3-ffi crate and instead rely on bindgen bindings maintained by upstream.

Most of the overhead of supporting new Python versions is updating the FFI bindings.

I bet @davidhewitt has opinions about this from the PyO3 maintainer perspective and I’ll defer to him if he disagrees with me about pyo3-ffi.

17 Likes

The specific team involved is T-bootstrap which you can reach out to on zulip at #t-infra/bootstrap

6 Likes

I would strongly advise against that, on account of people inserting an “A” into it :slight_smile:

More seriously though: Rust has been a focus for a lot of controversy and risk, with massive open questions such as the impact of a potential Ken Thompson style compiler hack. Will CPython lose trust by becoming dependent on a single specific compiler that might be subject to such a hack? Currently, CPython can be compiled using multiple different compilers (I believe gcc, clang, and MSVC are all supported - correct me if I’m wrong?), which mitigates the threat by allowing them to check each other. Adding Rust to the mix will mean that, on every platform, the same Rust compiler MUST be used.

13 Likes

I have long liked the idea of doing something like this, and as someone who always introduces memory leaks and segfaults and such any time he writes any kind of C extension code, I welcome the idea of more modern zero-cost abstractions for that[1].

However, one cost I think that has not been mentioned here is the effect that this could have on build times. In my experience, compile times for Rust (and C++) are much slower than for C. On my 2019 Thinkpad T480, from a fresh clone of CPython, I can build the whole thing in 2m50s (with make -j, and admittedly it taxes the machine):

real	2m50.573s
user	14m57.260s
sys	0m42.975s

After the initial build, incremental builds are in seconds. Not sure I know of any projects of comparable size and complexity to compare with, does anyone know if (in the extreme — I know this isn’t the plan) we were to re-write CPython in Rust, how the build times would compare? I really think that the fast iteration time and low overhead from the builds is a great feature of our current build system that I would really hate to give up.

I guess the question is a dual one: can we write Rust in a way that it will not cause build times to explode and if we cannot, is the plan to keep the scope of Rust in CPython to a level where building is still very accessible?

My thinking is that if build times were 2x slower that would be acceptable if it meant a significant reduction in security critical bugs, but I personally wouldn’t love it if it were much slower than that.


  1. On rereading, I realized that this makes it sound like I want zero-cost abstractions to help me introduce more segfaults. If it’s unclear to anyone, I meant “to avoid that”. I have left the original wording because I like to at least initially convey that I am Chaotic Neutral. ↩︎

12 Likes

Clearly it should be “CrustyPython” :smiley:

But seriously, I’m really excited about this proposal.

22 Likes

In my experience incremental Rust builds are also very fast–the initial setup (including downloading and building all the dependencies) can be slow, but it’s able to do fast incremental builds just fine.

Also, there’s a big difference between debug and release mode–building in debug mode is way faster. I would be surprised if this impacted iteration time unless you’re trying to rebuild the world every time.

3 Likes

I recently elected to try learning C++ after an entire professional career of Rust + Python, because I wanted to make a build system that can be bootstrapped in as few jumps as possible (so that it could be invoked from inside other build systems like cargo or pip). Having worked a lot on the pants build tool in python + rust, I have strong feelings about affordances the build system should provide to users—feelings I will try to quell to achieve a productive discussion.

The pants build tool has incorporated rust since Twitter’s “moonshot” rewrite from python-only through 2017-2019—I contributed the current iteration of our python-level interface for cacheable+parallelizable build tasks (Concepts | Pantsbuild), which uses pyo3 to hook up rust async methods as “intrinsics” which can be interchanged with “async def” coroutines (Stu Hood originally proposed and implemented this approach)—we’ve had this interop since before async was stable and before pyo3 existed.

We used rust, so we had to use cargo. Most of a decade later, we haven’t managed to use our own build system (written in rust) to build the rust code at the heart of our system. I claim that one of the major reasons for this failure is that cargo is almost unique among build systems in providing absolutely no structured mechanisms for:
(1) communicating with other package build scripts in the same dependency graph
(2) communicating with the downstream user who invokes cargo (such as a distro packager, or a github actions pipeline)

This failure is especially notable because of the same powerful guarantees cargo ensures for rust-only dependency graphs, as described in OP:

Finally, Rust has an excellent build system. Rust uses the Cargo package manager, which handles acquiring dependencies

cargo unfortunately has no standard mechanism for declaring dependencies downloaded within a build script so that they can be audited or overwritten, except the excessively restricted (non-transitive) and poorly documented links key in Cargo.toml, which requires that a build script link a native library into the resulting executable.

cargo also has no standard conception of a “toolchain”, or even an ABI outside of rustc output. This can and does mean that build scripts will fail because a dependency was built for a slightly different ABI, because again, not only is there no standard interface for downstream build configuration, but there’s not even a standard interface for communicating structured data across build scripts. So rust devs end up doing the natural thing and using somewhere in ~/.cache or ~/.config or elsewhere as undocumented mutable state.

I am relatively confident this isn’t an oversimplification, because bootstrapping the rust compiler itself ends up invoking multiple distinct reimplementations of LLVM target triple parsing, added at different times and never synced up. This is because rustc uses cargo, and cargo does not support structured communication across build scripts.

I proposed some of how I wanted to help improve this situation to NGI Zero at the end of last year. This C++ system I mentioned at the beginning is a competing approach, which would replace cargo instead of attempting incremental reform. I’m still not sure of the “right” answer to this—and I don’t think fixing cargo should be the purview of this PEP anyway.

But I am personally convinced that if CPython were to integrate rust (possibly even just at phase 1, with only external module support), we (CPython and pypa contributors, of which I am only the latter) would necessarily have to figure out a more structured way to thread ABI info through cargo, and potentially even institute a whole structured communication mechanism across the build script dependency graph. I think that will be a lot of hard work and we should prepare for it earlier rather than later.

I am very heartened to read in OP that there are steps being taken to interface with the rustc team to express CPython’s bootstrapping+portability requirements. It sounds like we’re on a good trajectory already to consider the above. I would just urge contributors to consider that pants has not solved cargo’s python packaging difficulties in many years and that it may be worth opening up a greater discussion about cargo affordances in order for cargo (not just rustc) to support CPython’s (and consequently pypa’s) needs.

22 Likes

The best things we can do to keep Rust build UX good for core devs (and other contributors):

Keep a lean dep tree – and be able to rebuild each module individually (lots of parallelism, like we have in C).

Have a easy to use flow that uses cargo check, which does type checking but doesn’t actually compile, since that gives you a very fast devloop.

9 Likes