Itertools.tee delays finalizing items after all iterators consumed them

TL;DR: itertools.tee keeps an item referenced until its whole internal batch (57 items currently) is consumed, even after every branch has consumed the individual item. Therefore, __del__ and any resource release (e.g. GPU memory for CUDA tensors) happen later than expected. Two questions: (1) is a short doc note about this welcome, and (2) is there interest in changing this behavior, to free items as soon as all branches pass?


Note: This is a follow-up based on issue #138765 and PR #154369, which were closed. I’d like to reopen the discussion here rather than discussing on the closed items, I hope that’s alright. Happy to move to a different category if that’s more appropriate.

The issue was filed as a “memory leak”, which is not the case IMO. I believe this is about finalization timing. An item which is already consumed by all branches stays referenced until its whole batch is retired. So Py_DECREF and __del__ are delayed, in a non-trivial and surprising way. The original reporter’s issue was PyTorch CUDA tensors: the tensor’s GPU memory stays pinned long after every iterator has passed the item.

Two questions

1. Docs: Currently the docs state:

This itertool may require significant auxiliary storage (depending on how much temporary data needs to be stored). In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee().

This note is about storage size, it says nothing about delayed finalization of already-consumed items, and limits the note to the data that “needs to be stored”. I’d read this as the gap between the slowest and fastest iterator.

Would a short, implementation-independent sentence or change be welcome, e.g. something like: “Items already consumed by every iterator may not be released immediately, their finalizers can be delayed until internal buffers are recycled”? I understand the reluctance to over-specify (I filed PR #154369 pinning the 57 figure, which was rejected as an implementation detail).

2. Behavior: In the issue @storchaka describes a fix that releases items as soon as every branch has passed them, without adding memory or time cost.

My questions:

  • Is there interest in this behavioral change, or is the current design considered final?
  • @storchaka do you intend to implement the proposal yourself? If not, I’m happy to give it a shot.
  • If the design is final, is there some way to make this behavior more clear in the docs?

Thanks for your time and considerations!

cc @rhettinger


PS: A small example showing this in action:

from itertools import tee

class LoudLifecycle:
    def __init__(self, i):
        self.i = i
        print(f"INIT {self.i}")

    def __del__(self):
        print(f"DEL {self.i}")

def my_iter():
    for i in range(100):
        yield LoudLifecycle(i)

a, b = tee(my_iter())

for _ in zip(a, b):
    pass
4 Likes

I did some benchmarks some time ago.

LINKCELLS 2 iters 10 iters
1 (PY recipe) 18 54
1 6 10
9 2.3 6
25 1.8 6
57* 1.8 5
121 1.6 5

(milliseconds to consume tee on [1] * 100_000 via zip + deque(maxlen=0).extend)

By now I have my own tee and accept up to 4x worse performance. And I am kind of obsessed about the performance…

“I have an idea how to do this without increasing the memory or time cost.” - Serhiy

This would be great.


Also another issue on this: itertools.tee optimal de-reference · Issue #123056 · python/cpython · GitHub

1 Like

That doesn’t seem true. Quote from there:

When a new teeobject start referring to teedataobject, refcounts of all items are increased by 1. When a referrer stops referring to teedataobject, refcounts of all items are decreased by 1.

Changing refcounts of all items takes time. And if you then don’t actually iterate over all of them, then I think that’s additional time cost (compared to the current implementation). Consider for example the lookahead function shown in the tee documentation, which just briefly uses a new teeobject to peek for one item. That would get slower, wouldn’t it?

1 Like

The final word is for @rhettinger. If he is against it, then there is no point in wasting time on implementation and discussion.

It is easier now with AI, so I drafted a PR: gh-138765: Release items cached by itertools.tee once all iterators pass them by serhiy-storchaka · Pull Request #154880 · python/cpython · GitHub . Caveat: it crashes on free-threaded build.

2 Likes

Hi! I didn’t get to commenting on the issue yet, sorry.

I think we should document this as an implementation detail, without too many specifics and with a note that it may change at any time.
Promptly releasing resources is also a CPython implementation detail, but one that people do depend on, for better or worse.
Even if the implementation changes in 3.16, the note would be useful in older versions.

1 Like

This can not hurt indeed.

However, I think it is also worth to note that tee is used in many recipes - it can often be the best primitive for the task. e.g. more_itertools.unzip.

Adjusting tee itself would solve the issue at its root.
While there are quite a few places where this would need to be documented and new ones will likely be appearing over time.