Object memory consumption in Python 3.11

Core Development
1

In Python 3.11, suppressing the creation of __dict__ greatly reduces memory consumption for the average case.

But for those classes with many attributes(>=30), instances in the 3.11 branch consume much more memory than in Python 3.10. It seems this is because dict objects allocate more spare memory in Python 3.11 than the 3.10. Perhaps some adjustments are required?

test script:

import tracemalloc

class C:
    def __init__(self):
        pass

def test_memusage(num_attrs):
    c = C()
    for i in range(num_attrs):
        setattr(c, f"xxx{i}", 200)
    del c

    tracemalloc.start()

    items = [C() for i in range(1000)]
    for c in items:
        for i in range(num_attrs):
            setattr(c, f"xxx{i}", 200)

    current, peak = tracemalloc.get_traced_memory()
    print(num_attrs,  current)
number of attrs Python 3.10.6 3.11 branch
1 160288 91944
2 160288 99736
3 160288 107536
4 160288 115344
5 160288 123160
6 200288 138808
7 200288 146640
8 200288 154480
9 200288 162328
10 200288 170184
11 288288 178048
12 288288 185920
13 288288 193800
14 288288 209576
15 288288 217472
16 288288 225376
17 288288 233288
18 288288 241208
19 288288 249136
20 288288 257072
21 288288 265016
22 456288 280920
23 456288 288880
24 456288 296848
25 456288 304824
26 456288 312808
27 456288 320800
28 456288 328800
29 456288 328800
30 456288 1648544
31 456288 1648544
32 456288 1648544
33 456288 1648544
34 456288 1648544
35 456288 1648544
36 456288 1648544
37 456288 1648544
38 456288 1648544
39 456288 1648544
40 456288 1648544
41 456288 1648544
42 456288 1648544
43 800288 1648544
44 800288 1648544
45 800288 1648544
46 800288 1648544
47 800288 1648544
48 800288 1648544
49 800288 1648544
50 800288 1648544
51 800288 1648544
52 800288 1648544
53 800288 1648544
54 800288 1648544
55 800288 1648544
56 800288 1648544
57 800288 1648544
58 800288 1648544
59 800288 1648544
60 800288 1648544
61 800288 1648544
62 800288 1648544
63 800288 1648544
64 800288 1648544
65 800288 1648598
66 800288 1648544
67 800288 1648544
68 800288 1648544
69 800288 1648652
70 800288 1648706
71 800288 1648706
72 800288 1648706
73 800288 1648760
74 800288 1648706
75 800288 1648706
76 800288 1648706
77 800288 1648544
78 800288 1648544
79 800288 1648544
80 800288 1648544
81 800288 1648706
82 800288 1648544
83 800288 1648706
84 800288 1648598
85 800288 1648652
86 1480288 3392706
87 1480288 3392706
88 1480288 3392706
89 1480288 3392706
90 1480288 3392760
91 1480288 3392760
92 1480288 3392760
93 1480288 3392652
94 1480288 3392814
95 1480288 3392706
96 1480288 3392706
97 1480288 3392760
98 1480288 3392814
99 1480288 3392814
100 1480288 3392814
2

In Python 3.10, objects having different insertion order couldn’t use key-sharing dict.

In Python 3.11, that limitation is fixed. Objects can use key-sharing dict even if insertion order is not same.
But it introduced maximam size of key sharing dict. It is 30 for now. It is why instances having 30+ attributes consumes much more memory.

See these PRs for detail.

3 Likes
3

I was not aware that a limit on key-sharing dictionaries had been introduced. Thank you for your clarification.