Closable iterables

Typing
1

The close() method is not a part of iterable or iterator protocols, it is a part of the generator protocol. But some iterables and iterators have the close() method even if they do not implement the full generator protocol (because send() and throw() do not make sense for them). It allows to deterministically clean up resources. One of examples – os.scandir() which holds an open file descriptor. I am going to add also the close() method for iterparse() iterator which can hold an open file. For asynchronous iterators it is even more important, because they tends to create reference loops. You should always close an asynchronous iterators, not rely on the GC.

For now, when I used an iterable or iterator with close() method in type annotated code, I either ignored static type checking

it.close()  # type: ignore

or added a runtime check

if hasattr(it, close):
    it.close()  # type: ignore

and it is even more inconvenient if you use the closing() context manager.

I propose to add ClosableIterable and ClosableIterator (and corresponding asynchronous variants) for annotating such types.

6 Likes
2

You could very easily define your own protocols to handle these. typeshed and useful_types already define many protocols that aren’t part of collections.abc but are widely used in type annotations.

from collections.abc import Iterable, Iterator
from typing import Protocol, TypeVar

T_co = TypeVar("T_co", covariant=True)

class SupportsClose(Protocol):
    def close(self) -> object: ...

class ClosableIterable(Iterable[T_co], SupportsClose, Protocol[T_co]):
    pass

class ClosableIterator(Iterator[T_co], SupportsClose, Protocol[T_co]):
    pass

You could even mark them with @runtime_checkable if you want them to work in isinstance checks.

4 Likes
3

I’m a big fan of this. I think that having the most common protocols like this in Python’s standard library allows developers to type their code in a way that aligns with Python’s duck typing behavior which from what I’ve read on this forum seems to be a characteristic that many feel is a core aspect of Python. I think including these in the standard library encourages their use and improves developer understanding of the most common patterns in Python. While this can be defined as a protocol or a third-party package, doing so in each codebase for common patterns leads to more fragmentation and feels like we’re being asked to consistently reinvent the wheel.

I’m a +1 on this.

4

I agree [with David’s response], I don’t see the justification for adding this to standard library, and I don’t see why this combination would be any more common than other combinations of protocols.

5

I think this argument gets even stronger once/if intersections become a thing, because then you can actually easily compose more basic protocols to get what you want:

from collections.abc import Iterable, Iterator
from typing import Protocol

class SupportsClose(Protocol):
    def close(self) -> object: ...

def foo(x: Iterable[int] & SupportsClose) -> None: ...

Which in many cases will probably also produce something more readable than weird type names like CloseableSeekableReadableX, where the most relevant part of the type comes last, just because it’s grammatically nicer.

Or in other words: I would much rather see the addition of some more basic protocols, like the ones defined in typeshed or useful_types, so you can compose these more complex protocols yourself more easily, rather than add an increasing number of permutations of these protocols with increasingly strange names.

4 Likes