To expand on this point, I see Rust as a potential usability gain here, much more than the security aspect.
I think the argument to use Rust for CPython here would be much stronger if we could see standard_b64encode implemented in a way that takes advantage of Rust’s RAII to reduce the book-keeping we have to do in the C code right now.
Without a “Rust API” here for writing these extensions, this becomes purely a security argument (that I find fairly unconvincing at some level). If standard_b64encode was returning a Result and we were able to use ? and all this other stuff with a wrapper that did “the right thing” that would be, at least to me, much more interesting
EDIT: though a point against this being easy might be the memory allocation story here… though the error allocation failure paths are about allocation failures in the Python arena, not sure if that means we really have no stack left over. Still think it’s worth proving the point that this has ergonomics improvements, because that feels like a pretty big deal all things considered!
One example of reducing mental bookkeeping is the BorrowedBuffer abstraction in the implementation cpython/Modules/_base64/src/lib.rs at c9deee600d60509c5da6ef538a9b530f7ba12e05 · emmatyping/cpython · GitHub. As mentioned previously, the current example is pretty bare-bones as it is a proof of concept. There is a large room for improvement in ergonomics. But even with raw FFI bindings, it’s possible to have a safe, idiomatic Rust core of an extension module then expose that via unsafe wrappers. And I think even that will improve the safety and utility of writing extensions in CPython.
Yeah I think the BorrowedBuffer is a good example of helping a bit with its Drop.
I guess this would be more convincing to me (random person who has little stake in this beyond poking in CPython internals from time to time but wants to see some idea of this succeed!) if we go a bit further in the barebones interpretation to prove some idea of improved ergonomics, focused entirely on this encode implementation, which includes just enough bookkeeping futzing that would be nice to not see anymore:
let result = unsafe {
PyBytes_FromStringAndSize(ptr::null(), output_len as Py_ssize_t)
};
if result.is_null() {
return ptr::null_mut();
}
instead being something like:
let Ok(result) = PyBytes::from_string_and_size(ptr::null(), output_len as Py_ssize_t) else { return ptr::null_mut() } ;
Or even, if there’s some way to make PyResult-y thing that could collapse the common CPython errors nicely (maybe not a possible thing!):
let result = PyBytes::uninit_from_size(output_len)?
Just like BorrowedBuffer, it feels like there could be some single-field wrapper structs, and smart constructors that could operate in a “Rust API”-y level to work off of results
Similarly in:
let dest_ptr = unsafe { PyBytes_AsString(result) };
if dest_ptr.is_null() {
unsafe {
Py_DecRef(result);
}
return ptr::null_mut();
}
I would have expected we could have some Rust-y wrapper on references that would give us that Py_DecRef call for free just through RAII. And maybe I’m too optimistic of Rust’s compiler toolchain, but if the wrapper was a single field struct, my impression is we would be able to get that for free.
The thing I would assume is that within the extension module we could go full Rust API goodness, and it’s only really at the entry and exit points that one would need to go back to acknowledging CPython’s realities a bit more.
Anyways yeah, I’m very curious what the ‘pie in the sky most of the APIs used in the encoding example have a nice API’ version of the encode port looks like, because jumping from C to Rust could make the maintenance barrier seem way lower. We don’t have to port everything, just enough and enough smoke and mirrors to say “this is what it looks like in practice for this one method”. Because right now beyond the platform support tradeoffs etc, the PoC example is also presenting as generally more code, not less. In my mind’s eye we wouldn’t even have a tradeoff here, and the code would be simpler.
Interactions between Python objects and borrows is rather complicated. I don’t think this thread is a great place to go over detailed API design discussion as that isn’t the goal of the PEP, but I’d be happy to chat in another forum like the Python Discord or via DM. I will say a pie-in-the-sky API will look somewhat like an implementation using PyO3. I can write up such an implementation and share that if people think it would informative.
You would need Rust to build any Rust extension modules you want in your embedded Python. I believe most embedding links to libpython so that would already be built and wouldn’t require Rust.
Thank you, I know nothing of Rust, I see the word ‘dependency’ and I immediately get scared. Python and all the 3rd party modules must load into the host application’s process. In my case, I’m running Python inside AutoCAD for windows. If this is optional, or something that’s not going load some sort of runtime, or something that could cause issues, then great
We do already ship with pip and pymanager, and there’s no denying that Rust/Go have created an expectation for modern languages to include quality standard dev tooling. And it is equally apparent that the best Python tooling for the next decade will be written in Rust[1]. So I don’t see a future where Python also ships with such tools as outside the realm of possibility[2].
So I don’t see why not. This is clearly out of scope for this proposal, which is why I framed it only as a future possibility – one that would require its own difficult discussion and the platform/build support this PEP addresses. So I won’t pursue this further here.
regardless of if it’s still Astral tools like uv and ruff (or ty when it reaches stable) or something else like pyrefly ↩︎
I do believe replacing pip with uv might just be the most widely applauded move Python can make ↩︎
To achieve good ergonomics, we’ll need not only cpython-sys but also another crate built on top of it that provides proper Rust abstractions. (Following naming conventions, it would be called cpython, but that name is already taken.)
Right now, the PEP only covers cpython-sys, but if we expect Rust ergonomics by this PEP, I think some consideration of this additional crate should also be included.
This would be unfortunate for Gentoo. We’re currently one of the few Linux distributions that aim to provide a reasonable working experience for people with older, weaker or more niche hardware that is not supported by Rust; and given that we’re largely talking about volunteers with no corporate backing, there is practically zero chance of ever porting LLVM and Rust to the relevant platforms, let alone providing long-term maintenance needed for keeping them working in projects with such a high rate of code churn.
I do realize that these platforms are not “supported” by CPython right now. Nevertheless, even though there historically were efforts to block building on them, they currently work and require comparatively little maintenance effort to keep them working. Admittedly, the wider Python ecosystem with its Rust adoption puts quite a strain on us and the user experience worsens every few months, we still manage to provide a working setup.
The moment CPython starts requiring Rust, this will no longer be possible. Of course, we will still be able to provide older versions of CPython for a few years, at least until some major package starts requiring the newer Python version.
That said, I do realize that we’re basically obsolete and it’s just a matter of time until some projects pulls the switch and force us to tell our users “sorry, we are no longer able to provide a working system for you”.
I don’t expect to change anything here. Just wanted to share the other perspective.
I hope nobody will mind if my first post as a complete newcomer to the forum (though very far from one with Python or Rust) is letting my designed-to-win-trivia-games brain provide some context, clarifications, etc. for various things across the thread as a whole.
Also, sorry for splitting this across multiple posts but, even after pessimizing all my citations to “Search for …” annotations, it was still claiming I had more than two links in it. (If this posts, then I guess it meant two reply-to-post embeds.)
Finally, if I recall rust applications tend to be a bit “bloated” in binary size, although there are some tricks that can be done at compile time to reduce this - what do these tricks imply on performance I have no idea.
The answer depends on how long ago you remember that from. Rust’s history has been a tale of improving defaults on this front and I don’t know where you draw the line on “bloated”.
For example, prior to Rust 1.28 (Search for “Announcing Rust 1.28” site:blog.rust-lang.org), some platforms embedded a copy of jemalloc but it now defaults to the system allocator.
Rust still statically links its standard library, which is distributed as a precompiled “release + debug symbols” artifact to be shared between release and debug profiles and, for much of its life, there was no integrated support for stripping the resulting binaries. According to the Profiles (Search for “The Cargo Book” “Profiles” site:doc.rust-lang.org) section of The Cargo Book, strip = "none" is still the default setting for the release profile.
If I do a simple cargo new and then cargo build --release the resulting “Hello, World!”, the binary is 463K, which drops to 354K if the debuginfo is stripped.
That remaining size includes things like a statically linked copy of libunwind which wouldn’t be needed if using abort on panic as mentioned by Emma Smith… but I’m not up to speed on how much of that will get stripped out without rebuilding the standard library to ensure it isn’t depending on them.
(See my later mention of how the Rust team are currently prioritizing stabilizing -Zbuild-std as part of letting “remove kernelspace Rust’s dependency on nightly features” shape much of the 2025 roadmap.)
Beyond that, one potentially relevant piece of tooling is dragonfire (amyspark/dragonfire on the FreeDesktop Gitlab) as introduced in Linking and shrinking Rust static libraries: a tale of fire. (Search for “Linking and shrinking Rust static libraries: a tale of fire” site:centricular.com) (Which is concerned with deduplicating the standard library when building Rust-based plugins as static libraries.)
Just declare “CPython” to be referring to the stable ABI exposed rather than the implementation language.
After all, the abi_stable crate for dynamically linking higher-level Rust constructs does it by marshalling through the C ABI.
I guess the question is a dual one: can we write Rust in a way that it will not cause build times to explode and if we cannot, is the plan to keep the scope of Rust in CPython to a level where building is still very accessible?
I don’t see why it needs to be slow.
As laid out in The Rust compiler isn’t slow; we are. (Search for "The Rust compiler isn't slow; we are." site:blog.kodewerx.org), rustc is already faster than compiling C++ with GCC and the reason builds are slow has more to do with how much the Rust ecosystem enjoys the creature comforts of macros and monomorphized generics.
…and they’re working on making it faster still. Aside from “Relink, Don’t Rebuild”, as mentioned by Jubilee, there are two bottlenecks which disproportionately affect incremental rebuilds right now:
First, while there’s parallelism between crates and in the LLVM backend, the rustc frontend is single-threaded. Work is in progress and testable in nightly (Search for “Faster compilation with the parallel front-end in nightly” site:blog.rust-lang.org) for making the frontend multithreaded.
They’re also working on rustc_codegen_cranelift (rust-lang/rustc_codegen_cranelift on GitHub) which is a non-LLVM backend for rustc which makes more Go-like trade-offs for compile-time vs. runtime performance and is intended to eventually become the default for the debug profile.
Second, linking. They’ve been rolling out LLD as a faster default linker on a platform-by-platform basis and it came to Linux in 1.90. (Search for “Announcing Rust 1.90.0” site:blog.rust-lang.org)
Beyond that, mold is faster still (it’s what I use on my system) and wild (davidlattimore/wild on GitHub), yet faster, is being developed with an eye toward becoming default for debug builds alongside rustc_codegen_cranelift. (i.e. Doesn’t cover all the needs of a fully general-purpose linker, but does make debug builds for the majority of them very quick.)
I claim that one of the major reasons for this failure is that cargo is almost unique among build systems in providing absolutely no structured mechanisms for:
(1) communicating with other package build scripts in the same dependency graph
(2) communicating with the downstream user who invokes cargo (such as a distro packager, or a github actions pipeline)
This is a known problem that’s been discussed more or less since v1.0 came out in 2015 but more pressing issues keep jumping ahead of it in the queue.
(eg. The 2025H2 roadmap is prioritizing stabilizing an MVP of -Zbuild-std so that embedded and low-level projects like Rust for Linux (i.e. kernelspace Rust) don’t need to use either a nightly compiler or the secret switch to use API-unstable features on stable channel.)
If you want to search up existing discussions, what was done to incorporate Rust builds into Bazel got mentioned a lot.
It should be noted that, if I understand “GCC Testing Efforts” site:gnu.org correctly, GCC’s support for all platforms would count as “Tier 3, at varying degrees of stubbornness” by Rust testing standards since I don’t see any mention of any of the “Current efforts” entries being integrated to the same “CI on every push and will block merging into main if it fails” degree.
Rust’s approach to Tiers 1 and 2 leans in the direction of “We don’t trust our testing to be sufficient for Continuous Deployment, but we’ll do it as diligently as if we were pushing directly to stable channel”.
EDIT: …and, apparently, there’s also a limit on number of posts for new users so I can’t get it all in without breaking the rule about no substantial edits. I’ll drop an in-reply-to-embed and add a GitHub Gist containing the source for the entire thing as it was before I started making any changes to try to crunch it in.
In total, the posts being replied to, as represented in the auto-updating Discourse permalink URLs, are 4, 9/12, 15, 16, 18, 19, 30, and 38, and a few I forgot to grab URLs for while blockquoting, and the bits which don’t fit include an answer to the concern about Trusting Trust attacks, a clarification about “Rust guarantees that code outside an unsafe {} block is safe”, a mention of #[repr(transparent)], and a few other little things.
If python cannot be built without rust in the future, I believe the difficulties this would bring to bootstrapping are being underestimated. Python is such a widely used programming language that many projects have started to use python during the build stage. For example, glibc and gcc require python to build (https://github.com/fosslinux/live-bootstrap/commit/69fdc27d64ec56ad59b83b99aab0747c9d9f81ed). If python depends on rust, it would mean that all projects using the meson build system would also need rust to bootstrap (I know muon can be a replacement, but muon isn’t 100% compatible with meson). The live-bootstrap project has already completed the bootstrap of python, and it currently requires a total of 11 builds to obtain CPython 3.11.1, including regenerating all generated code (https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst#159python-201). And even when using mrustc to build rust 1.74.0, it still requires 18 builds to obtain rust 1.91, especially since the time required to build rustc is much longer than CPython. Moreover, rust releases a new version every 6 weeks, so this number will grow quickly.
Compilation time is also an issue. Although incremental builds in debug mode may be fast, this is not always possible, for example when doing distribution packaging, when using git bisect, or when debugging bugs that only reproduce in release mode. Currently, a full CPython build is still relatively fast, and I agree that it would be acceptable if the full build time were up to 2x slower.
Regarding the previously mentioned issue with os.fork, I am not sure whether using fork in a single-threaded process is safe. If using fork in a single-threaded process would still break rust’s safety guarantees, then os.fork and multiprocessing.get_context("fork") would become completely unusable, and that would break many third-party libraries that depend on it.
On the other hand, rust occasionally introduces breaking changes outside of editions, for example https://github.com/rust-lang/rust/issues/127343, and the potential impact of such risks on CPython should be considered carefully.
I spoke to some of the Miri maintainers about running PyO3 through Miri a while ago, I believe back then there were limitations due to all the C FFI being opaque to Miri. I vaguely recall the conversation concluded those limitations could be lifted, would just need some work. I’m not aware of anything to make me think that work has been done yet.
At the moment PyO3 has two abstractions, the low level FFI which this pre-PEP proposes generating with bindgen and the high-level abstractions which are totally safe. I’ve wondered about a third level which sits between the two; it would still use unsafe extern “C” ABI and call the C symbols directly, but the types for input & output could encode the possible states, e.g. BorrowedPtr(*mut PyObject) or even Option<NonNull<PyObject>> to force null checking. As long as these are layout identical with the actual C type passed through the FFI, it would just improve type safety without actually introducing any overheads or much “high level” API. There is a lot of scope to experiment here.
I agree fully with this - in particular a huge win is that you don’t need to remember to call Py_Clear / Py_DecRef / Py_XDecRef on every error pathway, RAII abstractions can just solve this for you. Your point here also speaks to what I was musing about in the point above.
Absolutely true that while many core devs may not currently be comfortable in Rust, there is a lot of anectotal evidence that after an initial learning curve many people find Rust relatively easy to feel productive and comfortable in. (Google’s experience with Android strongly supports this, for example.)
Many Python packages are already built in Rust, they’re precompiled and uploaded to PyPI as binary distributions which users can use without any awareness they’re built in Rust. CPython using Rust as an implementation detail would be no different for anyone not building from source.
@mgorny the Python ecosystem user experience is important to me and I’m aware there have been pains as tooling has adopted to Rust support. Gentoo particularly runs into these pains due to so much from-source building and extensive hardware support.
I’m sure I’m not aware of every possible configuration, please always do feel free to ping me / direct me at things and I will do my best to help. I build PyO3 / integrate Python & Rust to empower more people to write software, not to alienate.
On the other hand, rust occasionally introduces breaking changes outside of editions, for example
I don’t think it’s fair to call Rust out for that specifically, given that it’s not a Rust-specific problem and that, as demonstrated in places like graydon2’s retrobootstrapping rust for some reason, my prior mention of which got spilled into the GitHub Gist because “New users can’t…”,
"Modern clang and gcc won’t compile the LLVM used back then (C++ has changed too much – and I tried several CXXFLAGS=-std=c++NN variants!)
Modern gcc won’t even compile the gcc used back then (apparently C as well!)
Modern ocaml won’t compile rustboot (ditto)
While I don’t have numbers, given that Rust’s regression suite became a bottleneck on development before Microsoft started donating Azure time, and that they have Crater (a bot for testing proposed changes against slices of the public crate registry up to and including “all of it”), I suspect Rust introduces breaking changes less than C and C++ do.
There is an alternative Rust compiler implementation in C++, mrustc, that implements just enough Rust to compile the official Rust the compiler without relying on it in any capacity. The official Rust compiler bootstrapped both ways (original chain and mrustc) produce identical binaries, which is sufficient to prove the absence of the Ken Thompson hack.
It should be noted that, if I understand “GCC Testing Efforts” site:gnu.org correctly, GCC’s support for all platforms would count as “Tier 3, at varying degrees of stubbornness” by Rust testing standards since I don’t see any mention of any of the “Current efforts” entries being integrated to the same “CI on every push and will block merging into main if it fails” degree.
Rust’s approach to Tiers 1 and 2 leans in the direction of “We don’t trust our testing to be sufficient for Continuous Deployment, but we’ll do it as diligently as if we were pushing directly to stable channel”.
I don’t really want to derail this thread into a discussion on models of testing, but a similar discussion was had the other week on lobste.rs. I don’t think it’s a accurate summary to say Rust’s ‘testing standards’ just mean ‘Tier 3’ for GCC.