Pre-PEP: Rust for CPython

[emmatyping (Emma Smith)]

Note: I’m working on replying to everyone but wanted to get some initial replies out, I appreciate patience with this.

@MegaIng

Isn’t the experience in the linux kernel with adding rust support as a core part more a cautionary tale? At least it looks that way from the outside. There is constant friction between the two different “types of programmers”, causing a lot of quite public disagreement.

I agree with @jacopoabramo on this, I like to think that the Python community will be better about being productive, amicable, and respectful when discussing these issues. So I think the experience will be altogether different.

@jacopoabramo

Out of curiosity, what would this mean for both PyO3 and RustPython? Have you reached out to the mantainers of the latter for feedback on how to approach this possibility? If so, what’s their opinion? It looks to me like this possible PEP might benefit greatly from their experience (I don’t know if you guys are part of their team, I’m just assuming you’re not but I apologize if that’s incorrect).

I reached out to David Hewitt who has now responded in this thread and will contribute to the effort (thank you, David!) . Kirill has reached out to the RustPython team as well!

@jacopoabramo

would you expect some challenges in applying PEP 703 efficiently for rust-based extensions?

On the contrary, it should be much less effort as safe Rust is thread-safe due to the borrow checker. There will need to be some support added to handle integrating attached thread states, but it should be relatively easy.

@jacopoabramo

Same question applies for the new experimental JIT which should be more stable in future releases.

Sorry, I’m not sure if you are asking if Rust extensions will work with the JIT or if Rust can be used to implement it? For the former they should work without issue. For the latter the JIT currently relies on a custom calling convention which is not yet exposed in Rust (but is being discussed to do so last I checked). I don’t suggest moving this code to Rust until it is reasonable to implement in Rust and the necessary calling convention features are available.

@jacopoabramo

Finally, if I recall rust applications tend to be a bit “bloated” in binary size, although there are some tricks that can be done at compile time to reduce this - what do these tricks imply on performance I have no idea.

This is definitely something to be cognizant of, thank you for bringing it up! There are several strategies which will not affect performance, such as abort on panic, which we can explore. We will probably want to abort on panic anyway since unwinding over FFI layers is UB.

@steve.dower

As a general direction, I’d rather see “optional extension modules” living outside the main repo and brought in later in the build/release process. Presumably such modules have no tight core dependency, or they wouldn’t be able to be optional, and so they should be able to build on pre-release runtimes and work fine with released runtimes (as we expect of 3rd party developers).

I think this is an interesting proposal, but orthogonal to the current one. We hope Rust will eventually become required to build CPython so it can be used to improve the CPython core. I would suggest splitting this off into it’s own thread to discuss it further.

@steve.dower

In short, adding new ways to add new, non-essential modules to the core is counter to the approach we’ve been taking over the last few years. So I’m -1 on adding one.

Again I think it is important to highlight that this proposal is more than just _base64, and more than just optional modules. We’d eventually like to make Rust a hard dependency so it can be used to improve the implementation of the Python runtime as well.

@alex_Gaynor

I’m happy to answer any questions folks have about those experiences and lessons learned.

Thanks Alex! We definitely want to approach this carefully and with thought, your expertise will be invaluable!

@jamestwebber

As a Rust fan I think this is super cool! I do wonder if this is too much to figure out in one PEP, when parts of it seems pretty easily separable. I can see how the overall vision fits together, but it’s a lot.

I think we necessarily need to make a plan for long term adoption so we can figure out when Rust can be a required dependency and ensure we plan ahead in advance well enough for it. I don’t want to get anyone caught by surprise when suddenly they need Rust to build CPython when they don’t expect it. I will say that the final PEP will probably be what you propose plus a timeline to make Rust a required dependency. Iterating on ergonomic APIs for Rust is definitely something I’d be working on if cpython-sys is approved.

@mikeshardmind

I would feel more comfortable with this if it was dependent on custom json targets stabilizing in Rust. Right now, targetting platforms that Rust itself doesn’t support still requires nightly, and python is used to bootstrap a lot of things in a lot of places.

Perhaps then we can ensure that releases build with an older nightly Rust to enable such bootstrapping? I expect these cases to be relatively uncommon - Rust supports a large number of platforms.

@da-woods

So from my point of view this PEP doesn’t offer a lot - it’s largely proposing “rewrite some modules in Rust with a view to soften people up to rewrite more in Rust”.

I would restate our goal as “slowly introduce Rust to carefully integrate it and make sure we get things right and give people time to adapt to the significant change.” _base64 is chosen as an example as it is easier to implement, easier to understand, and would only affect performance, so is entirely optional. I think there are several existing modules that could see clear benefits from being written in Rust, eventually. Especially for those that interact with untrusted input such as json, it would be a significant improvement security-wise if we implemented them in a memory safe language. But I also don’t want to rush in and cause breakage. These kinds of changes should be done carefully and when well-motivated.

@da-woods

@emmatyping

Rust could use PyPy to bootstrap

I suspect this isn’t viable for a couple for reasons: PyPy isn’t hugely well maintained right now, and (I believe) PyPy needs a Python interpreter to bootstrap itself so may not solve the problem.

Good point re PyPy requiring some bootstrap Python itself! I hope that the approach of using an older CPython will be workable.

@pitrou

@emmatyping

A reference implementation which includes a proof-of-concept _base64 module which uses Rust to provide a speedup to base64 is available.

This example shows that you really want the safe abstractions for this to be useful. Otherwise you’re doing the same kind of tedious, unsafe, error-prone low-level pointer juggling and manual reference cleanup as in C (hello Py_DecRef ).

I absolutely agree, safe abstractions over things like argument parsing and module creation make PyO3 a joy to use. I hope we can collaborate with the PyO3 maintainers and provide similarly pleasant abstractions in CPython core. It will certainly be a high priority. I do think even in this simple example there are examples of safe abstractions that provide benefits. Kirill wrote up an abstraction over borrowing a Py_buffer that automatically releases the buffer on drop, so that it is impossible to forget to do that and cause a bug: cpython/Modules/_base64/src/lib.rs at c9deee600d60509c5da6ef538a9b530f7ba12e05 · emmatyping/cpython · GitHub

@guido

Seriously, I think this is a great development. We all know that a full rewrite in Rust won’t work, but starting to introduce Rust initially for less-essential components, and then gradually letting it take over more essential components sounds like a good plan.

I trust Emma and others to do a great job of guiding the discussion over the exact shape that the plan and the implementation should take.

Thank you Guido for your trust and words of support, it really means a lot!