In the production environment, the memory leak is a common issue for all the developers. For now, we may have two kinds of way to debug it.
Intrusive way, we need to modify our code, inject some code to monitor the object malloc, and publish/deploy it again. It will use a lot of time in non-python-code process.
Process attach way, we may use the tools like py-spy or other ptrace based way to attach to the process, and dump/analyze the all memory. It’s useful but will be limited in the container. For most circumstance, we limit the privilege of the container which is means that the ptrace based way will be unuseable because of the security issue.
I think maybe we can introduce a new -x option to dump heap and current stack automaticlly when MemoryError has been raised just like the developer did in JVM
My experience debugging memory leaks is that a preiodic dump of the count of each object type provides the information needed to track down where in the code to concentrate effort. Dumping in json format to a file once every few minutes allows a leak trend to be spotted. And you can do this in pure python today.
If you go for the dump of memory you will need to develop tools to process that memory dump. For a unix world I would prefer to have a core file from the process so that I can use gdb etc.
I’m not sure what would make sense in a Windows world.
Yes, the core file or dump it by customized code in pure Python should be the standard ways. But here’s still some limit
As I have said in the original post, injecting some customized code is a intrusive way. I think it’s not suitable for some of the workflow in cooperation circumstance.
Dump a core file need extra setup for the infra. For example we need to setup the ulimit for the host system or setup it for a dockerd or containerd daemon in container/K8S environment. It may need extra privilege request
So I think that should be great if we can support dump the heap and stack automaticlly.
I build these debug features into the apps, so there is no need to inject the code.
If you have a server then add an API to return object use counts for example.
Or start with a command line option to log to a file periodically.
I have tried going through a core dump from a python process and its a slow and tedious job to get any meaningful information out of the dump. Maybe there are tools now that make that easier to get meaningful analytics from the dump.