Pre-PEP: Rust for CPython

[davidhewitt]

@Eclips4

Can we integrate MIRI into our workflow?

I spoke to some of the Miri maintainers about running PyO3 through Miri a while ago, I believe back then there were limitations due to all the C FFI being opaque to Miri. I vaguely recall the conversation concluded those limitations could be lifted, would just need some work. I’m not aware of anything to make me think that work has been done yet.

@steve.dower

I’d rather see people discussing things like how Rust provides any protection/benefit at all when we have to interop everything with “unsafe” C code at a level below anywhere PyO3 could help (which is only safe because it relies on our public C API, which is the safety barrier with guaranteed semantics that can be mapped into Rust’s semantics).

At the moment PyO3 has two abstractions, the low level FFI which this pre-PEP proposes generating with bindgen and the high-level abstractions which are totally safe. I’ve wondered about a third level which sits between the two; it would still use unsafe extern “C” ABI and call the C symbols directly, but the types for input & output could encode the possible states, e.g. BorrowedPtr(*mut PyObject) or even Option<NonNull<PyObject>> to force null checking. As long as these are layout identical with the actual C type passed through the FFI, it would just improve type safety without actually introducing any overheads or much “high level” API. There is a lot of scope to experiment here.

@Gankra

In practical specific terms, it’s often reported that Rust often makes implicit ownership/lifetime constraints in C APIs explicit and easier to work with. The Rust bindings to C functions can include lifetimes that enforce these contracts (and yes a lot of C APIs map onto lifetimes and ownership well).

I agree fully with this - in particular a huge win is that you don’t need to remember to call Py_Clear / Py_DecRef / Py_XDecRef on every error pathway, RAII abstractions can just solve this for you. Your point here also speaks to what I was musing about in the point above.

@barry

The former is surely a smaller number, and my guess is that even fewer are comfortable in Rust today .

Absolutely true that while many core devs may not currently be comfortable in Rust, there is a lot of anectotal evidence that after an initial learning curve many people find Rust relatively easy to feel productive and comfortable in. (Google’s experience with Android strongly supports this, for example.)

@CEXT-Dan

Thank you, I know nothing of Rust, I see the word ‘dependency’ and I immediately get scared. Python and all the 3rd party modules must load into the host application’s process. In my case, I’m running Python inside AutoCAD for windows. If this is optional, or something that’s not going load some sort of runtime, or something that could cause issues, then great

Many Python packages are already built in Rust, they’re precompiled and uploaded to PyPI as binary distributions which users can use without any awareness they’re built in Rust. CPython using Rust as an implementation detail would be no different for anyone not building from source.

@mgorny

Admittedly, the wider Python ecosystem with its Rust adoption puts quite a strain on us and the user experience worsens every few months, we still manage to provide a working setup.

@mgorny the Python ecosystem user experience is important to me and I’m aware there have been pains as tooling has adopted to Rust support. Gentoo particularly runs into these pains due to so much from-source building and extensive hardware support.

I’m sure I’m not aware of every possible configuration, please always do feel free to ping me / direct me at things and I will do my best to help. I build PyO3 / integrate Python & Rust to empower more people to write software, not to alienate.