# Add \`flatten\_list\` to itertools

**URL:** <https://discuss.python.org/t/add-flatten-list-to-itertools/29181>\
**Category:** Ideas\
**Created:** [July 7, 2023, 2:01pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181 "2023-07-07T14:01:14Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![jacgoldsm](https://avatars.discourse-cdn.com/v4/letter/j/a183cd/32.png) [@jacgoldsm](https://discuss.python.org/u/jacgoldsm)\
**Post date:** [July 7, 2023, 2:01pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/1 "2023-07-07T14:01:14Z")

</div>

There are two “Pythonic” ways to flatten an iterable `l`:

1. `[item for sublist in l for item in sublist]`
2. `import itertools; itertools.chain.from_iterable(l)`

Both of those will remove one level of nesting from the list `l`, or if any of the elements of `l` is not iterable, it will raise an exception.

The issue with this is that if a list contains a mixture of leaves and iterables, you will have to wrap all of the leaves in a container to avoid an exception. Worse, if any of the elements of the iterable are themselves iterable but not intended to be flattened (e.g. strings), you will get silent data corruption.

Acknowledging that homogenous lists are the most common case, this is still a case that I find myself running into reasonably frequently:

```auto
import itertools
l = [[1],[2,3],"foo"]
# I want [1,2,3,"foo"]
print(list(itertools.chain.from_iterable(l)))
print([item for sublist in l for item in sublist])
# but in both cases I get [1,2,3,'f','o','o']

```

I propose a function `flatten_list` with the following semantics, in pseudocode.

```auto
 def flatten_list(l: list, *, max_depth: int=1) -> list:
        def flatten_list_recursive(l,depth):
                for elem in l:
                    if isinstance(elem, list) and depth:
                        flatten_list_recursive(elem,depth-1)
                    else:
                        out.append(elem)
        if not isinstance(l, list): raise TypeError
        out = []
        flatten_list_recursive(l,max_depth)
        return out

```

This function will continue to flatten lists until it hits the max depth, but will leave any other kind of iterable alone. This allows for flattening heterogenous data types that contain non-iterables, lists, and iterables that you don’t want to flatten, like strings, and it allows for arbitrary flattening, not just one level of recursion.

Does this seem like it could be a useful addition to the itertools module?

---

<div class="post-metadata">

**Author:** ![Rosuav](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rosuav/32/3429_2.png) [@Rosuav](https://discuss.python.org/u/Rosuav)\
**Post date:** [July 7, 2023, 2:08pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/2 "2023-07-07T14:08:39Z")

</div>

If it’s to be considered an itertool, it needs to accept arbitrary iterables (even infinite ones) and yield its results - a relatively small tweak to your algorithm, I expect.

What you may be interested in, though, is that it’s already in more-itertools:

[https://more-itertools.readthedocs.io/en/stable/api.html#more\_itertools.flatten](https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.flatten)

---

<div class="post-metadata">

**Author:** ![jacgoldsm](https://avatars.discourse-cdn.com/v4/letter/j/a183cd/32.png) [@jacgoldsm](https://discuss.python.org/u/jacgoldsm)\
**Post date:** [July 7, 2023, 2:12pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/3 "2023-07-07T14:12:38Z")

</div>

Got it, the only thing is that it feels wrong to me is to flatten things like strings and bytes the same way as lists, tuples, etc. Do you also have that intuition, and do you think there’s a way to distinguish between types that should be flattened and types that shouldn’t in a principled way?

---

<div class="post-metadata">

**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\
**Post date:** [July 7, 2023, 2:13pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/4 "2023-07-07T14:13:42Z")

</div>

The `collapse` function in `more_itertools` \[1\] will ignore strings and bytes. You can specify the thing you ~~want to flatten (e.g. only lists)~~ edit: actually I misread, you can specify the things you _don’t_ want to flatten.

* * *

1. Chris linked to `flatten` which only does one level

---

<div class="post-metadata">

**Author:** ![jacgoldsm](https://avatars.discourse-cdn.com/v4/letter/j/a183cd/32.png) [@jacgoldsm](https://discuss.python.org/u/jacgoldsm)\
**Post date:** [July 7, 2023, 2:16pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/5 "2023-07-07T14:16:09Z")

</div>

That’s exactly what I was looking for, thanks! It seems very useful to me and I didn’t know about it, IMO it could be a good addition to the standard library.

---

<div class="post-metadata">

**Author:** ![Rosuav](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rosuav/32/3429_2.png) [@Rosuav](https://discuss.python.org/u/Rosuav)\
**Post date:** [July 7, 2023, 2:16pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/6 "2023-07-07T14:16:40Z")

</div>

> [@jacgoldsm](#):
>
> Got it, the only thing is that it feels wrong to me is to flatten things like strings and bytes the same way as lists, tuples, etc.

That’s an arguable point, but there is one data type that definitely should be able to be flattened, and would be tricky to pin down: generators. In nearly every way, a generator (the result of calling a generator function) should behave identically to a list when passed to itertools functions. So all you could really do would be to pick out a couple of data types and exclude them, which is what collapse() does (btw, thanks for the correction there, I automatically reach for “flatten” as it’s what I expect that to be called).

---

<div class="post-metadata">

**Author:** ![pylang](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@pylang](https://discuss.python.org/u/pylang)\
**Post date:** [July 7, 2023, 9:56pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/8 "2023-07-07T21:56:39Z")

</div>

There is some “prior art” on this idea from older resources mentioned [here](https://stackoverflow.com/a/40857703/4531270) related to making your own `flatten()` function, one that handles nesting and mixed types.

If you were seeking something pre-implemented, there are some tools in `more-itertools` as mentioned above. Demonstrations found [here](https://stackoverflow.com/a/40938883/4531270).

As to whether it should be added to the standard library, the latter options have been around so long, I’m not sure. The demand is certainly there.

---

<div class="post-metadata">

**Author:** ![nobodyinperson](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nobodyinperson/32/24486_2.png) [@nobodyinperson](https://discuss.python.org/u/nobodyinperson)\
**Post date:** [December 11, 2024, 12:39pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/9 "2024-12-11T12:39:17Z")

</div>

My version of a recursive `flatten()` function:

```python
import itertools

def is_iterable(x):
    """
    Check if a given object is iterable but not a string.
    """
    if isinstance(x, str):
        return False
    try:
        iter(x)
    except TypeError:
        return False
    return True

def iter_(x):
    """
    Like :any:`iter` but yield uniterable objects instead of raising a
    :any:`TypeError`.
    """
    if isinstance(x, str):
        yield x
    try:
        yield from x
    except TypeError:
        yield x

def flatten(*sequences):
    """
    Flatten given sequence(s)
    """
    if len(sequences) > 1:
        for sequence in sequences:
            yield from flatten(sequence)
    else:
        sequence = next(iter_(sequences), tuple())
        if isinstance(sequence, str):
            yield sequence
        try:
            for element in sequence:
                if is_iterable(element):
                    yield from flatten(element)
                else:
                    yield element
        except TypeError:
            yield sequence

print(list(flatten([1,[2,3,[[(4,5),6]]],"a",["bla"]])))
# [1, 2, 3, 4, 5, 6, 'a', 'bla']

```

It is quite unfortunate that strings are iterables in Python, requiring extra handling, but this works.

---

<div class="post-metadata">

**Author:** ![Nineteendo](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nineteendo/32/19122_2.png) [@Nineteendo](https://discuss.python.org/u/Nineteendo)\
**Post date:** [December 11, 2024, 3:47pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/10 "2024-12-11T15:47:51Z")

</div>

Don’t forget bytes, bytearray and memoryview.

---

<div class="post-metadata">

**Author:** ![dg-pb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dg-pb/32/16248_2.png) [@dg-pb](https://discuss.python.org/u/dg-pb)\
**Post date:** [December 11, 2024, 6:51pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/11 "2024-12-11T18:51:42Z")

</div>

So it is a combination of these 3:  
a) [more\_itertools.always\_iterable](https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.always_iterable)  
b) [more\_itertools.flatten](https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.flatten)  
c) +nesting

> [@jacgoldsm](#):
>
> Does this seem like it could be a useful addition to the itertools module?

Probably not. What could be is to add `depth` and `base_type` arguments to `more_itertools.flatten`.

E.g.:

```python
import functools as ftl
import itertools as itl
import more_itertools as mitl

def flatten(iterable, depth=1, base_type=None):
    if base_type is not None:
        base_type = itl.repeat(base_type)
    while depth:
        if base_type is not None:
            iterable = map(mitl.always_iterable, iterable, base_type)
        iterable = itl.chain.from_iterable(iterable)
        depth -= 1
    return iterable

```

Although this doesn’t handle infinite depth, but I don’t think objects of unknown max depth are often encountered.

Also, performance is not too bad for the above. Although it does more what is needed (in contrast to @nobodyinperson), its performance is quite ok.

```python
l1 = [[1], [2,3], "foo"]
l2 = [1,[2,3,[[(4,5),6]]],"a",["bla"]]
l1 = l1 * 10_000
l2 = l2 * 10_000

BHOLE = collections.deque(maxlen=0).extend

lambdas = {
    'flatten_l1': lambda: BHOLE(flatten(l1, base_type=str)),
    'nobodyflatten_l1': lambda: BHOLE(flatten2(l1)),
    'flatten_l2': lambda: BHOLE(flatten(l2, 4, base_type=str)),
    'nobodyflatten_l2': lambda: BHOLE(flatten2(l2)),
}

"""
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ macOS-11.7.10-x86_64-i386-64bit | CPython: 3.12.4 ┃
┃ 5 repeats, 1 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ms 0 ┃
┃ ┏━━━━━━━━━━━━━━━━━━━┫
┃ flatten_l1 ┃ 6 ┃
┃ nobodyflatten_l1 ┃ 50 ┃
┃ flatten_l2 ┃ 115 ┃
┃ nobodyflatten_l2 ┃ 123 ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━┛
"""

```

---

<div class="post-metadata">

**Author:** ![dg-pb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dg-pb/32/16248_2.png) [@dg-pb](https://discuss.python.org/u/dg-pb)\
**Post date:** [December 11, 2024, 7:12pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/12 "2024-12-11T19:12:52Z")

</div>

To it seems that `more_itertools.flatten` would like some more functionality in it…

```python
def flatten(listOfLists):
    return chain.from_iterable(listOfLists)

```

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [December 11, 2024, 7:21pm UTC](https://discuss.python.org/t/add-flatten-list-to-itertools/29181/13 "2024-12-11T19:21:42Z")

</div>

@dg-pb Looks more like [`collapse`](https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.collapse) than like `flatten`.