Pre-PEP: Rust for CPython

[davidhewitt]

Thank you @emmatyping @eclips4 for proposing this (and @ngoldbaum for the ping)! Very excited to see this initiative and as a proponent of Python, Rust, and the two together, eager to be involved.

As a longtime PyO3 maintainer I have been thinking about what this might look like for a while. I asked about exactly this kind of possibility at the Language Summit earlier this year to gauge the temperature for anyone wishing to experiment. The response I heard then was that experimentation towards a concrete proposal was welcome (I hadn’t yet found the time to explore myself, so thank you).

I have a number of comments so will try to keep each brief for now and we can expand later if needed.

@emmatyping

[rejected] Use PyO3 in CPython

@pitrou

This example shows that you really want the safe abstractions for this to be useful.

I completely agree with both of these points. Depending on PyO3 as currently implemented within CPython will introduce unwanted friction. PyO3 also supports PyPy and GraalPy, as well as older versions of CPython (currently back to 3.7). Rust for CPython will presumably not need to support anything other than current CPython.

At the same time, CPython will need safe higher-level Rust APIs to get the benefit of Rust. PyO3 has a lot of prior art on the high-level APIs (and hard lessons learned); I think the right approach here will be similar to what attrs did for dataclasses - the Rust APIs implemented by CPython can pick the bits that work best.

@emmatyping

What about platforms that don’t support Rust?

gccrs is an alternative implementation of Rust for GCC backend, which is not yet at feature parity but an important target for Rust for Linux. If CPython was using Rust, I would hope that efforts for non-llvm platforms may be helped by this.

@emmatyping

What about Argument Clinic?

PyO3’s proc macros function a lot like argument clinic - we could potentially reuse parts of their design (and/or implementation); PyO3 might eventually even depend upon any implementation owned by CPython. I would recommend this choice as the more idiomatic way to do codegen in Rust.

@emmatyping

Should the CPython Rust crates be supported for 3rd-party use? Should there be a Rust API?

@ngoldbaum

As a PyO3 maintainer IMO it’d be really nice if we could get rid of the need for the pyo3-ffi crate and instead rely on bindgen bindings maintained by upstream.

Most of the overhead of supporting new Python versions is updating the FFI bindings.

I agree with both of these points; we may want some experience before deciding how CPython would want to commit to supporting these crates. At the same time, having a way to consume in-dev CPython versions immediately in PyO3 would really help with iteration speed for the ecosystem.

PyO3 has precedent of “experimental” features for things not yet stable, one middle ground might be to allow PyO3 to have an experimental feature which switches out pyo3-ffi’s curated FFI bindings for the ones generated by CPython.

@MegaIng

Isn’t the experience in the linux kernel with adding rust support as a core part more a cautionary tale? At least it looks that way from the outside.

There are a couple of takeaways from Rust for Linux that I think are most interesting here:

• Social questions - inevitably not everyone will want to be using \<insert language X here\> instead of \<insert language Y here\>. Some current maintainers might churn from not wanting Rust, and other new maintainers may be attracted by the appeal of using it. The Python community places a lot of emphasis on inclusivity and I’d hope we would welcome everyone’s opinions as valid even if eventually a decision driven by the majority must be taken. (Not implying here whether the majority is for or against exploring Rust support; that is the purpose of having these discussions, after all.)
• Technical opening - Rust for Linux has unsurprisingly become a major strategic focus for the language. I would hope that Rust for CPython would have justification for carrying similar weight in the focus of the Rust project should there be friction where Rust (and cargo etc) do not currently meet CPython’s needs.