Adopt a safer memory allocator

Currently, Python’s memory management is handled by the internal Python memory manager, but there are some security risks:

Python developers have no direct control over memory management, even when manipulating object pointers pointing to heap memory blocks. This can lead to unexpected memory access and corruption.
To avoid memory corruption, developers should not use C library functions such as malloc(), calloc(), realloc(), and free() to manipulate Python objects. This can lead to confusion between the C allocator and the Python memory manager, with serious consequences.
Although Python provides some primitive memory allocation functions such as PyMem_RawMalloc(), PyMem_RawCalloc() and PyMem_RawRealloc(), these functions still have security risks, and improper use by developers may lead to memory access errors.

Therefore, it seems important to adopt a safer memory allocator to improve the security of Python’s memory management. This can not only reduce developers’ troubles when dealing with memory issues, but also provide a more stable and reliable running environment for Python applications.

I would like to introduce to a project - PyMemAllocator-RS. This is a memory allocator implemented in Rust language, specifically designed for the Python interpreter.

The main features of PyMemAllocator-RS are as follows:
High-performance memory management: The Rust language is known for its excellent memory management capabilities. PyMemAllocator-RS takes advantage of Rust to provide a more efficient memory allocation and recycling mechanism for the Python interpreter. This helps improve the overall performance of your Python program.
Thread safety: PyMemAllocator-RS is thread-safe, which means that it can work safely in a multi-threaded environment and avoids memory problems caused by concurrent access. This is very helpful for building highly concurrent Python applications.
Scalability: PyMemAllocator-RS is designed with good scalability and can be customized and optimized according to different application scenarios. This makes it suitable for Python projects of all sizes and types.
Cross-platform support: Like Rust, PyMemAllocator-RS also has excellent cross-platform capabilities and can run on mainstream operating systems such as Windows, macOS, and Linux. This greatly increases its scope of application.

PyMemAllocator-RS is a very promising Rust project that brings more efficient memory management capabilities to the Python interpreter. If interested in this, welcome to visit the PyMemAllocator-RS GitHub repository for more information.

This code defines the PyMemAllocator structure, which is the interface for interacting with Python’s obmalloc memory allocator, and its functionality.

For whom, exactly, are you speaking when you are saying “The Python developer community”?

14 Likes

Does CPython already contain Rust language dependencies? Should it?

Hello,

let’s discard for a moment the “python developer community” representation. My questions to you are:

  • Do you have a working prototype? Meaning CPython compiled with this memory allocator?
  • Can you show the "high performance memory management?
  • What’s the complexity of having Rust as build dependency (which is not present at the moment)?

Also you state:

PyMem_RawMalloc(), PyMem_RawCalloc() and PyMem_RawRealloc(), these functions still have security risks"
Can you expand on that?

This looks like a major change in CPython and if you don’t provide enough datapoints, it might be overlooked.

Now, going back to your “Python developer community calls…” How do you know that? Did you run a survey? Are you representing some sub-community?

7 Likes

That sounds like it’s not yours. But isn’t it?

1 Like

Can’t you publish this on PyPI first?

Dear Python user,

CPython should not have a dependency on Rust.

Sincerely,
Another Python user.

4 Likes

Is this you?

Can you demonstrate performance? The community support you allude to?

Thanks :slight_smile:

Python 3.13 now uses mimalloc allocator by default in Free Threaded build. This allocator provides functions which are missing in the default pymalloc allocator. Is your allocator a drop-in replacement for mimalloc? Does it have the same APIs?

You can also opt-in for mimalloc using:

$ PYTHONMALLOC=mimalloc python3.13

Check which allocator is used with:

>>> import _testinternalcapi; _testinternalcapi.pymem_getallocatorsname()
'mimalloc'

7 Likes

This seems to be an issue that transcends Python and includes other cobbled-together languages like R.

They may intially be built on a relatively homogeneous base such as main parts being written in a compiled language like C and then other parts bootstrapped in the new langauge, be it python or R or whatever.

But it seems the door is open to borrow anything that can be patched in, such as well-known efficient libraries from FORTRAN, or even rewriting or adding functionality for the sake of efficiency.

So what happens if one of these embedded languages fall out of favor or are changed in incompatible ways? What happens if a RUST compiler is not available for every platform Python runs on?

And what if that happens to C? It’s not really a useful hypothetical–in this unlikely future scenario, the people who are around at that time will figure out the best option. As it stands, CPython isn’t written in anything but C (and Python, of course)[1].

For people who are interested in an implementation of the Python language in Rust, I suggest looking at the RustPython project. They’d be happy to have more volunteers.


  1. okay, GitHub tells me there’s 0.8% C++, I don’t know what that’s for ↩︎

1 Like

James,

I am not wedded to any one language and have used pretty much all the ones that have been used as part of one version of python or another. Yes, C and its neighbor C++ are likely to remain available for a very long time and partially not for themselves but their role in being part of other things like parts of Python or libraries in general.

My concern is about requiring too many languages to remain supported. If a module is in use that is not heavily used and requires a working compiler for some language like FORTRAN or Rust, no big deal. The core language can be used even if that module is not available. But if the core requires many such different dependencies, we need a good reason to include them.

To be clear, I have nothing against RUST and have programmed in it. In some ways it may have substantial improvements over a language like C which I have had no need to program in for decades. As you note, there may well be a version of python built mainly on that language instead, or perhaps at some point people will rewrite enough of the C in a way that allows C++ or some other C musical evolution to be the single base for main part of Python.

I’m more concerned about the name. CrustPython doesn’t sound very nice :smile:

7 Likes

Then why should we bother responding to yours?

3 Likes

That could be addressed by making the existing C implementations thread-aware. I don’t see the need for involving another language. And being clear about exactly what the benefits are and how using a different language achieves them.

I just want someone to pay attention to the issue of python memory management. I think the topic has achieved the main purpose of discussion so far. It is not from a development standpoint but only from other people’s identities and standpoints. Everything is just nonsense.

I will try LOGIC based on Rust script to build a C99 implementation, thank you for your comment.

These functions are safe when used correctly, but improper usage by developers can introduce security risks, leading to memory access errors.

The same applies to Rust; security risks can be introduced through improper usage of its language features.

Do you have a CPython fork with the mentioned safe memory allocator, or are we discussing this purely theoretically?

1 Like

@semmyenator You didn’t reply to this question.

mimalloc may become the default memory allocator in future Python versions, so you need to provide similar (same) API than mimalloc to compete with it.

1 Like

I’m still building it, it may take a few days