Effiency drop after loading a huge JSON

Hi guys, a script like this:

import os
import sys
import json
import commands

def gen(i):
s,o = commands.getstatusoutput(’:’)
print i

a=json.load(open(’./big.json’, ‘r’))

for i in range(300):
s=gen(i)

The loop goes very slowly, like 10s, after the json is loaded. But without loading the json, the loop just takes 1 second.

Also, this issue only happens when calling os.system/commands.getstatusoutput/os.popen or something like that. If function gen just does some math work,the problem goes away (have same effiency with or without ‘a’).

I run it on a CentOS7 with 32GB RAM and python 3.8, and with the json loaded, the process consumes 1.8GB RAM. No swap is activated.

So I wonder why the loop slows down and how can I make is faster?

You may want to try temporarily disabling the garbage collector to see if that has any impact. Although it could be something else, just a wild guess.

Reference: https://stackoverflow.com/a/20495977

Yes John, I did some google yesteday and noticed that answer on stackoverflow too. But unforturnatly, disabling gc doesnt work for me. I guess it’s something else.