`Cannot allocate memory` on creating a bunch of `mmap.mmap` objects on Linux

Python Help
1

With the help of newly added trackfd=False option in mmap.mmap constructor, we can create far more mmap.mmap objects beyond the open file limit. But creating a lot of mmap.mmap seems to cause memory errors (“OSError: [Errno 12] Cannot allocate memory”).

Example:

import mmap, os

fn = "/dev/shm/test_file"
fd = os.open(fn, os.O_CREAT | os.O_RDWR | os.O_TRUNC, 0o600)
os.ftruncate(fd, 1)

mmap_objects = []
for i in range(1_000_000):
    mmap_object = mmap.mmap(fd, 1, trackfd=False)
    mmap_objects.append(mmap_object)

I runs the snippet above on a CentOS8 server with 48GB RAM, and it stops around 60k-ish iterations with the insufficient memory error.

By contrast, creation of 1M builtins.object instances can be easily done and a C program that creates 1M mmap pointers can also finish with no error.

I’ve read through cpython/Modules/mmapmodule.c at main · python/cpython · GitHub but didn’t find anything suspicious. Maybe someone here can help to confirm whether this is a bug.

Also confirmed on RHEL 7.

2

I suspect you will have a better chance of an answer on a centos or Fedora forum. Try https://discussion.fedoraproject.org/ they answer centos questions as well as Fedora questions.

4

It seems not restricted within CentOS systems. Also occurs on RHEL 7.

5

Its a kernel config or process config error.

It will also break on rhel 8, 9 and any linux system.

You need to ask the experts for help.

6

Did you try the exact same program in C as the Python script, and the former could successfully create one million mmap pointers?

7

Did you try the exact same program in C as the Python script, and the former could successfully create one million mmap pointers?

I’m afraid I may not have understood your question well, but I’ll try to make my OP clearer.

The C program I used is trying to call mmap function (mmap(2) - Linux manual page) 1M times which will result in 1M pointers and no further processing is done on those pointers. As there is no munmap call, I’m sure that those pointers would never get released until this C program exits.

This C program is obviously not doing the same thing of the Python snippet because it is not creating Python objects. But success in calling it shows that there is no number restriction in living mmap pointers in a single process, or the max number is no less than 1M.

8

Yes, that’s exactly what I asked.

Note that the Python itself is an interpreter, a big C program. Maybe it had previously called a few system calls that changed the later behavior of mmap. Try running strace python an_empty_script.py and see how many mmap and other calls are there. So the states of the simple C program process and the Python script process are very different.
I may be very wrong. I’m just speculating.

9

In this situation is system memory actually exhausted, or is Python hitting some kind of lower limit?

What happens if, instead of appending the mmap objects to a list, you close them in your loop?

10

There are limits imposed by the kernel to stop you breaking the system.
You will need to find these limits and change them to match your application needs. These usually end up being sysctl changes, that you can setup to happen on each boot.

11

Sorry for wasting everybody’s time.

I made a horrible mistake in my test C program that it didn’t check whether the returned pointer is valid. The real number of valid mmap pointers a C program can create is also limited.

In fact, this behavior is related to vm.max_map_count setting which is having a default value of 65530 on my CentOS server. After I changed it to a larger number, both the Python script and the C program run successfully.

3 Likes