This is a very common phenomenon. Often, the release of memory doesn’t immediately free it up to the rest of the system (because that can only be done in fixed allocations, or because it’s inefficient to do so, or any other reason), but that memory IS available for reuse within that application. Try running this a second time to see if it uses more memory.
I’d be curious to see whether your results are comparable if you instead use a dictionary. Replace the calls to set()
with calls to dict.fromkeys()
and see what that does to your memory usage. In my testing, each dictionary came in at roughly half the size of the corresponding set.
Side note though: I don’t know what sort of figures you’re getting from sys.getsizeof(results)
, but to my knowledge, that is ONLY going to give you the size requirement for the list itself. So I would expect that figure to be basically the same for any list of the same size, and most likely will approximate to len(results) * word_size
where the word size would be 4 bytes for a 32-bit Python and 8 bytes for a 64-bit Python. It may be somewhat larger than that due to overallocation, though by a bounded amount. Your figure shown there of 17MB seems pretty plausible - that’s less than ten percent above the minimum.
Here’s my figures for a much simpler test, showing the effective costs of different Python data types. The usual caveats apply: this is my system and may differ from yours, the results will vary across Python versions, the results will vary on 32-bit and 64-bit builds, and your operating system and CPU architecture may play a part too. With that said, though, there does seem to be a notable difference here.
>>> sys.getsizeof(tuple(range(10000)))
80040
>>> sys.getsizeof(list(range(10000)))
80056
>>> sys.getsizeof(dict.fromkeys(range(10000)))
294992
>>> sys.getsizeof(set(range(10000)))
524504
>>> sys.getsizeof(tuple(range(23)))
224
>>> sys.getsizeof(list(range(23)))
248
>>> sys.getsizeof(dict.fromkeys(range(23)))
1168
>>> sys.getsizeof(set(range(23)))
2264
The overhead for a list is pretty much constant. In my testing of lists constructed directly from ranges (as in your some_function
example), the overhead is either 56 or 64 bytes, regardless of the length of the list; for a list constructed by repeated appends, the overhead also includes the slack space for future expansion, up to maybe 10%.
The overhead for a set, however, grows far more rapidly. There’s ever-growing overhead due to the internal slack space required for an efficient hash table. This varies considerably with the size of the set, but seems to be about 2KB for smallish sets, and settles in on about a 4:1 ratio as they get larger (that is to say, a set with N items takes up roughly as much memory as a list with 4*N items).
So I think what you’re seeing here is the inherent cost of sets: since they need random access by value, they can’t be as compact as lists, which need only look up by index. That fact, coupled with some quirks of how operating systems and processes manage their memory, means that you’re allocating more but then only returning it to the Python process without fully returning it to the OS. But…
… this part is curious. If you’re doing this repeatedly, and clearing out the old results before starting a new run, the same memory should be being reused. So maybe there’s more going on here. Can you post an example that actually crashes?