What I am looking to do is essentially use shared memory for allocations within an existing Python application (technically a Python application running in a container). Rather than Python ultimately allocating directly from system memory, I would like for any system level allocations (Raw domain) to be taken from pre-allocated shared memory from another a C application that will launch the container.
Note: Host OS is RedHat 7.X
The only way this seems achievable is using Custom Python Memory Allocators as documented here. I see how you can create custom allocators for the raw domain but I’m not sure how this would be set up within the Python application exactly. Can anyone help drive me in the right direction? Does this approach seem possible with attaching to and using shared memory for Python allocations?
Note: I have asked this same question on Stack Overflow. There was a good suggestion to use tracemalloc as a guide for how to do it but I wanted to check here as well to see if anyone had any additional thoughts.
The problem is trying to incorporate a 3rd party Python application (running in container) into an existing C application. The C application controls most of the memory being used on the server by its libraries and so this is also needed for this new application as well.
As noted the only way I could see this working is modifying the Python application to use custom memory allocators that would use shared memory from the C application and manage use of that memory…at least in the Raw domain. I am a beginner with Python and so I thought perhaps that is all that is needed and Python could continue directly managing its own memory from the private heap for the Memory/Object domains.
Right you can certainly limit the memory usage of the container but the desire is for it to use specific memory pre-allocated by the C application. I didn’t see any way to specify what memory for the container to use, just the maximum amount it can use.
The issue with allowing the kernel to allocate memory has to do with this new Python application working with the existing C application. The C application will load/unload different libraries depending on need and it provides pre-allocated memory to a library for it to divide up as necessary…essentially as a large work buffer. Most of the memory on the system is pre-allocated for use in the libraries for performance/optimization reasons.
There is not a lot of system memory left over after the pre-allocation and so the desire is for the new Python application to use the pre-allocated memory similar to the other libraries. That way when it is unloaded then the next application/library will be able to start fresh within the same memory space.
Sounds like you have a embedded system, but its running RHEL?
How limited is the memory?
Anyway it is not practical for a python process to do what you describe. Of course you could rewrite the python memory management, but that’s going to be a non-trivial task.
You can free the memory by having the python process quit, just as you are doing when you unload a library in your custom mechanism.
I guess all your other libraries are C or C++ so that you could replace
malloc with your own version. And you do not use .so’s that you have
not rebuilt around your C-runtime library?
Correct, it’s an embedded system running RHEL. As you noted all of the other libraries are C/C++ and there is actually a memory management library with its own API that the libraries use to get memory from the large pre-allocated memory (avoiding use of malloc). And to my understanding there is no use of .so’s in place of the C-runtime library.
Yeah, as I’ve learned how Python memory management works, I can see this would be non-trivial but in this case it seems like it has to be done (at least for the Raw domain allocators).