I’m seeing an unexplainable memory leak, which gets better when I start the program with MALLOC_ARENA_MAX=1(but that doesn’t resolve it)… and it may either get much better or solved when using LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2.
It’s an exporter for Prometheus, but it will be difficult for any interested parties to directly test this (as it requires an installation of dCache).
What basically happens is, when run in --export-mode http:
Now I don’t see any obvious memory leak in my own code (and I asked LLM and it does neither), so I tried to rule out several things:
It’s not the webserver from prometheus_client, cause when I don’t even run that and simply execute collect() in an endless loop, it also happens.
It doesn’t seem to be anything from prometheus_client either, like that that would keep refs on my objects forever, cause I’ve changed collect() to merely call get_raw_data() and then return () so nothing from prometheus_clientshould ever even see my data.
It doesn’t seem to be httpx or paramiko either, since in some tests I’ve replaced them by dummy lambda functions.
The culprit really seems to be something that I .submit(...)because when I change my Future lists to simply be empty (instead of being populated using submit(), there’s no memory leak either.
Now LLMs (Claude) gave me some more or less useful hints (the latter because I simply have no clue about these deeper internals in how Python / glibc do their threads:
As said above using MALLOC_ARENA_MAX=1 already slows down the leak, but resident memory of the process still grows… and it may either get much better or be solved when using LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2… would need to run it longer in order to really tell.
Now the exporter is a long running process… so even if there’d be only a slowly growing leak, that would be bad.
Any ideas what’s happening there or how I can solve this? Or is it some bug in Python (using 3.14.6)/glibc (using 2.42)?
One thing I should add… I’m not sure whether or not the internal webserver used by prometheus_client allows concurrent requests…
What I mean is, I guess it might be difficult to simply tear-down the executors after say 1000 jobs…
At least in my tests all requests were sequential, but in production use, multiple Prometheus servers might scrape the same exporter.
A while ago I debugged a app that seems to have had the “leak” like you are seeing.
I also set MALLOC_ARENA_MAX=1 to keep memory allocation code from leaving arenas only partly allocated. My analysis was that it was not leak it was inefficient use of the areans.
As far as I recall Setting MALLOC_ARENA_MAX=1 forces all threads to use 1 pool of arenas and thus limits unused space in the arenas. Otherwise there is 1 pool of arenas per thread.
What this does is trade locking overhead for memory usage.
Changing the allocation algorithm like you have with jemalloc seems to confirm you are seeing the same thing.
I would assume that after enough run time, a few days or a few weeks the memory will stabilise.
For a long-running process, it can be hard to tell whether you’re seeing a real memory leak or whether the process legitimately needs that much memory — unless you let it run long enough. Could you share more details? For example:
how many CPU cores and how much RAM the machine has
how much memory the process used at startup
how much it used after running for some time
and whether leaving it running eventually triggers an OOM?
I have one server where it runs since ~ 1,5 days … there the memory usage might have “stabilised” at around 900-1000MB.
But that server is for production use and get_raw_data() is typically called “only” every 10s (rather as fast a possible on my local system)… so perhaps it still grows further.
When I run it in one-shot mode (i.e. collect data only once, print metrics) it needs around 127M…
That would be over a factor of 7 that gets lost just because of presumed fragmentation issues with threads. Seems quite a lot.
Should I open an issue in the bug tracker, whether Python could improve something with how it uses/releases memory in that case, so that it works better with glibc’s standard malloc?
@weixlu I run it on two systems now, a production system and my local laptop for testing.
production sys has:
64 cores Xeon Silver 4216 CPU @ 2.10GHz
96 GB mem
laptop has:
16 cores i7-1270P
64 GB mem
when the program has started, but not yet received any scrapes, it takes about 64M (which actually surprises me already a bit…)
after a longer while (1.5d) on the production sys it has ~900-1000GB
OK. We had a similar issue and I asked about it. The tl;dr is that glibc doesn’t return memory to the OS - it holds it for future allocations and can cause fragmented memory leading to an OOM. We addressed this issue by adding calls to malloc_trim if we’re using glibc which tells it to return unused memory.
cpython already has look aside lists for allocations I think, I’m not sure there is much room for a python algorithm change.
glibc malloc is working to trade off performance against percent of arenas used. The default is fast allocations. You already applied one of tuning fixes, have you looked into the other knobs you can tune?
If you were an async app and not threaded you might see better memory usage. If it is I/O bound then you likely do not need threads.
Are you using systemd’s ability to cap memory use for a service?
I have used that to make sure that when there are services sharing a server stay within expected CPU and memory budgets.
I doubt you will given the amount of RAM on your test and production systems.
As you use Prometheus are you using node_exporter to track the memory and CPU use of the server running this app?
You could just watch the app run for a month and see if you afford the memory used. If you are say happy that it caps under 8GiB is it worth your time to optimise it’s memory use?
You could just set a memory limit in systemd and when the process is killed on running out of memory have systemd restart it automatically.
@brass75 It feels as if rather CPython itself should do these calls, if its threading somehow uses memory in a way that causes glibc not to free it reasonably.
@barry-scott At least I haven’t seen such behaviour in C or C++ when using threading. It feels more, that when simply starting many threads, and even re-using the the ThreadPoolExecutor, one shuldn’t need to turn some deeply underlying knobs.
I think both httpx and paramiko do parallelise very well with thread,…
Caping the memory would place a hard limit which I kinda don’t want to… the program is meant for general use amongst sites of the LHC Computing Grid, other sites have much larger dCache instances, which would result in more memory needs.
I doubt you will given the amount of RAM on your test and production systems.
Well it runs now for nearly two days and while memory usage is (too) large it seems to have stabilised…oddly enough it even fell back to 350M (from around 1G)… no idea why… it’s still the same process and AFAICS nothing has really changed.
As you use Prometheus are you using node_exporter to track the memory and CPU use of the server running this app?
I do, but it only tracks overall mem usage… it can’t to single processes. But it should be easy to hacks something together that periodically records the memory usage.
Nevertheless… it feels as if this should work better out of the box, but I’m not sure whether it’s worth opening an issue over in the bug tracker.
It looks like your machines are quite large and have plenty of RAM, so glibc may simply not consider holding onto an extra ~1 GB a big deal — even if you’d rather it didn’t. We all want something that works well out of the box, but memory really is a trade-off: use more memory for better performance, or save memory at some cost to performance.
In your case, since jemalloc already behaves well, I’d just switch to it. jemalloc is an excellent allocator as well. I’m a maintainer of Dragonwell JDK (an OpenJDK distribution), and we’ve replaced glibc’s malloc with jemalloc because it performs better in our workloads.
In your case, since jemalloc already behaves well, I’d just switch to it.
I did… or at least will make it automatically used if available… but it still doesn’t seem the proper solution.
Doing something normal (and I guess with the GIL on it’s way out it may become even more standard to use threads in Python) and simple like submitting many tiny jobs shouldn’t require a Python user to replace the malloc, or call (non-portable) glic functions from within the code.
What metric are you using to determine memory size rss, pss, vsz?
For quick testing, I simply did a ps -o pid,user,%mem,vsz,rss,comm on my process in a while loop and watched how it went up… For the “long” running process on that production server I simply used btop (which I thinks shows rss).
It is my experience that you do have to tune to keep memory usable down especially in processes that use lots of threads.
I tend to use async rather than threading if the code is not CPU
bound because you can avoid lots of the costs of threading. Memory being
one of them.
Same answer than above… aren’t threads meant to become generally usable now in Python?
Be able to use the free-threaded python does not mean you escape the cost of threads.
Do you need that many, or are you just using a default?
One limit that some use is to limit number of threads to number of CPU cores.
I’m seeing docs from web search that say with glibc you will see 8 x num-threads arenas by default. Each arena is 128MiB.
But this would need double checking.
