Any idea why this seems to have a memory leak

Python Help
1

Hey.

When writing a question on stackoverflow I noticed that one variant of my code causes my Python to get OOM killed by the kernel.

This is the code:

#!/usr/bin/python3

import timeit
import argparse


parser = argparse.ArgumentParser(allow_abbrev=False)
parser.add_argument("-n", "--iterations", type=int, default=1)
args = parser.parse_args()



class MyModuleError(Exception):
    pass

def foo():
    try:
        f = 1/0
    except (ValueError, ZeroDivisionError) as e:
        raise MyModuleError from e

def wrapper1():
    try:
        foo()
    except MyModuleError as e:
        t = type(e.__cause__)
        if t is ZeroDivisionError:
            pass
        elif t is OSError:
            pass
        else:
           raise

def wrapper2():
    try:
        foo()
    except MyModuleError as e:
        if isinstance(e.__cause__, ZeroDivisionError):
            pass
        elif isinstance(e.__cause__, OSError):
            pass
        else:
           raise

def wrapper3():
    try:
        foo()
    except MyModuleError as e:
        try:
            raise e.__cause__
        except ZeroDivisionError:
            pass
        except OSError:
            pass
        except:
           raise e    #which should be the "outer" exception



t = timeit.timeit(wrapper1, number=args.iterations)
print(f"wrapper1: {t}")
t = timeit.timeit(wrapper2, number=args.iterations)
print(f"wrapper2: {t}")
t = timeit.timeit(wrapper3, number=args.iterations)
print(f"wrapper3: {t}")

Using a sufficiently high -n (on my 64GB system it’s 100000000) the python process eats up “all” memory until it gets killed.

Is this a bug in CPython (running 3.11.4 from Debian sid) or is there a reason for it?

Thanks,
Chris.

2

An answer may have been given at:

3

Interesting.

The problem is specifically with wrapper3, and specifically using timeit. I could not reproduce a memory leak on my system with the other wrappers at all, nor with simply running wrapper3 repeatedly in a simple loop. Rather than simply leading to a MemoryError or OOM kill, my computer became unresponsive and I had to fight my way to a terminal to kill the process myself.

timeit’s internal logic will temporarily disable garbage collection during the trial. If I try doing that (with a smaller iteration count!) I can clearly see the memory “leak” with wrapper3 (not with the others). So, this somehow creates objects that are not garbage-collected by the ordinary reference-counting system, but are, immediately collected by the cycle-detecting garbage collector.

As noted in the comments on your Stack Overflow question, this cycle is between the exception and the corresponding stack frame; this is why, in 3.x, the name binding in except blocks gets deleted afterward (adding support for chained exceptions caused the reference cycle to appear):

And as we’ve seen here, re-raising the exception defeats that scheme. We can reproduce this more simply:

def wrapper4():
    try:
        foo()
    except MyModuleError as e:
        f = e
1 Like