# Nwise itertools

**URL:** <https://discuss.python.org/t/nwise-itertools/51718>\
**Category:** Ideas\
**Created:** [April 23, 2024, 3:44pm UTC](https://discuss.python.org/t/nwise-itertools/51718 "2024-04-23T15:44:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 3:44pm UTC](https://discuss.python.org/t/nwise-itertools/51718/1 "2024-04-23T15:44:45Z")

</div>

> <https://github.com/python/cpython/pull/118162>
>
> \<!--
> Thanks for your contribution!
> Please read this comment in its entirety. I…t's quite important.
> 
> \# Pull Request title
> 
> It should be in the following format:
> 
> \`\`\`
> gh-NNNNN: Summary of the changes made
> \`\`\`
> 
> Where: gh-NNNNN refers to the GitHub issue number.
> 
> Most PRs will require an issue number. Trivial changes, like fixing a typo, do not need an issue.
> 
> \# Backport Pull Request title
> 
> If this is a backport PR (PR made against branches other than \`main\`),
> please ensure that the PR title is in the following format:
> 
> \`\`\`
> \[X.Y\] \<title from the original PR\> (GH-NNNN)
> \`\`\`
> 
> Where: \[X.Y\] is the branch name, e.g. \[3.6\].
> 
> GH-NNNN refers to the PR number from \`main\`.
> 
> \--\>
> 
> 
> 
> \* Issue: gh-118179

I created this pulll request to add nwise to the itertools module. I saw that we had implemented pairwise, so I implemented nwise. I would have it marked as a draft. I would love some code review and some suggestions. However, I think this would be an excellent edition to the standard library.

---

<div class="post-metadata">

**Author:** ![ZeroIntensity](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/zerointensity/32/10639_2.png) [@ZeroIntensity](https://discuss.python.org/u/ZeroIntensity)\
**Post date:** [April 23, 2024, 4:36pm UTC](https://discuss.python.org/t/nwise-itertools/51718/2 "2024-04-23T16:36:29Z")

</div>

Not qualified to do a code review, but I would put the function name and return type on seperate lines, per PEP 7.

---

<div class="post-metadata">

**Author:** ![bschubert](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bschubert/32/26920_2.png) [@bschubert](https://discuss.python.org/u/bschubert)\
**Post date:** [April 23, 2024, 4:47pm UTC](https://discuss.python.org/t/nwise-itertools/51718/3 "2024-04-23T16:47:51Z")

</div>

I found at least one previous proposal for this: [[Python-Dev] Proposal for a new itertools function: iwindow](https://mail.python.org/pipermail/python-dev/2006-May/065276.html)

Additions to `itertools` (and in particular converting recipes to module functions) are a fairly common type of suggestion. It would help your proposal if you could find and address some of the concerns that have been expressed in the past. See for example the discussion thread that lead to `itertools.batched`:

> [@Add batching function to itertools module](https://discuss.python.org/t/add-batching-function-to-itertools-module/19357):
>
> Batching of lists, arrays, iterables, etc. is a pretty common operation, which is why I was surprised that this functionality has not already been implemented as part of the itertools module when I recently went looking for it! I feel like batching functionality is simple enough that it wouldn’t be too difficult to implement, and could be used across a wide variety of tasks, but complex enough that it would be nice to have a known solid reference implementation as part of the standard library ra…

Also:

> [@Why does itertools have recipes?](https://discuss.python.org/t/why-does-itertools-have-recipes/17622):
>
> I’m not really sure if this is the right place to ask this, but I’m kind of confused about why the itertools module has recipes like grouper and sliding\_window in the documentation, rather than in the module itself. I know that the itertools module itself is written in C for performance reasons, and the recipes are written in pure-Python but make use of code written in C (e.g. grouper calls zip\_longest) so they’re presumably “fast enough” without needing to be written in C, but I’m not asking “…

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 5:02pm UTC](https://discuss.python.org/t/nwise-itertools/51718/4 "2024-04-23T17:02:32Z")

</div>

You’re absolutely right. I will make those changes to formatting.

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 5:06pm UTC](https://discuss.python.org/t/nwise-itertools/51718/5 "2024-04-23T17:06:31Z")

</div>

I use it for time\_series stuff when I don’t have enough data to warrant numpy. I think it is a very common operation as well, so much so, that we have already allowed pairwise. I think nwise is just the generic case, and would be useful. I personally think it would be a great addition to the standard library.

```python
def nwise(lst, n):
    iters = tee(lst, n)
    for idx, itr in enumerate(iters):
        next(islice(itr, idx, idx), None)

    for group in zip(*iters):
        yield group

print(list(nwise([1, 2, 3, 4, 5], 3)))

```

This isn’t hard is just boilerplate when it would be nice to exist in the standard library.

Especially since we already have pairwise so why not just add the generic nwise

---

<div class="post-metadata">

**Author:** ![JamesParrott](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamesparrott/32/10534_2.png) [@JamesParrott](https://discuss.python.org/u/JamesParrott)\
**Post date:** [April 23, 2024, 5:16pm UTC](https://discuss.python.org/t/nwise-itertools/51718/6 "2024-04-23T17:16:17Z")

</div>

more\_itertools has had version of this (with bells and whistles) under a different name for the last 4 years:

[https://more-itertools.readthedocs.io/en/stable/api.html#more\_itertools.windowed](https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.windowed)

and has provided the recipe that the docs state is currently under consideration:

[https://more-itertools.readthedocs.io/en/stable/api.html#more\_itertools.sliding\_window](https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.sliding_window)

```python
def sliding_window(iterable, n):
    "Collect data into overlapping fixed-length chunks or blocks."
    # sliding_window('ABCDEFG', 4) → ABCD BCDE CDEF DEFG
    it = iter(iterable)
    window = collections.deque(islice(it, n-1), maxlen=n)
    for x in it:
        window.append(x)
        yield tuple(window)

```

> **[itertools — Functions creating iterators for efficient looping](https://docs.python.org/3/library/itertools.html#itertools-recipes)**
>
> This module implements a number of iterator building blocks inspired by constructs from APL, Haskell, and SML. Each has been recast in a form suitable for Python. The module standardizes a core set...

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [April 23, 2024, 5:58pm UTC](https://discuss.python.org/t/nwise-itertools/51718/7 "2024-04-23T17:58:36Z")

</div>

If you want less boilerplate, you could change your

```python
    for group in zip(*iters):
        yield group

```

to

```python
    yield from zip(*iters)

```

or

```python
    return zip(*iters)

```

---

<div class="post-metadata">

**Author:** ![elis.byberi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/elis.byberi/32/35937_2.png) [@elis.byberi](https://discuss.python.org/u/elis.byberi)\
**Post date:** [April 23, 2024, 6:02pm UTC](https://discuss.python.org/t/nwise-itertools/51718/8 "2024-04-23T18:02:53Z")

</div>

I couldn’t find a link to this discussion thread in the pull request or issue.

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 7:42pm UTC](https://discuss.python.org/t/nwise-itertools/51718/9 "2024-04-23T19:42:22Z")

</div>

I mean I agree however. Lowering boilerplate vs

```python
from itertools import nwise

```

and having to use generator incantations to achieve these things.  
I just personally think that since I wrote this is C if we write a nice convience function that is general like nwise, since we have pairwise. I think it makes good sense.

---

<div class="post-metadata">

**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\
**Post date:** [April 23, 2024, 7:51pm UTC](https://discuss.python.org/t/nwise-itertools/51718/10 "2024-04-23T19:51:06Z")

</div>

Generally, the answer for proposals to add something to `itertools` is something like “add it to `more_itertools` first, check if it’s used a lot and wait till it has a stable interface, then port it over if it isn’t a 3-line implementation or if the C implementation is significantly faster.” Seems like `more_itertools.windowed` is used quite a bit, I am not really sure where the cutoff is. But I would strongly prefer the more general interface of `windowed` instead of `nwise`.

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 8:24pm UTC](https://discuss.python.org/t/nwise-itertools/51718/11 "2024-04-23T20:24:24Z")

</div>

Windowed exists now. We have batched, so I would say I would tend to agree that of the more\_itertools pieces windowed would be used more. However, that’s functionally equivalent to batched. Which we just added. I think that sliding\_window in more\_itertools is most definitely used. I certainly think there is a use case for nwise as well, especially since we have pairwise, and I would personally love to have the general case. I also think that we would see speed ups from going from a python implementation in more\_itertools to a C implementation in itertools. I think we should think about what sort of keyword arguments we’d like to see maybe take some inspiration from more\_itertools. However, I am definitely for taking the more use functions like nwise → sliding\_window and batched → windowed and implementing them in itertools. I love reaching for the itertools library whenever I am iterating the first thing I think is. Can I use something from itertools? Is there something in there that will make my life easier? However, if it comes to installing a dependency, I will usually forego that, and implement it myself.

before batched was released I don’t know how many times I wrote

```python
def batched(it, n):
    itr = iter(it)
    while batch := tuple(islice(itr, n)):
          yield batch

```

I personally think that this is a 3 line implementation, but do I think that batched shouldn’t be added into itertools. I personally am very glad to see this make it’s way into itertools. I think nwise would be an excellent edition too. I would also like to benchmark it and see if we can get it to be as performant as possible.

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [April 23, 2024, 9:18pm UTC](https://discuss.python.org/t/nwise-itertools/51718/12 "2024-04-23T21:18:13Z")

</div>

> [@MegaIng](#):
>
> port it over if it isn’t a 3-line implementation or if the C implementation is significantly faster

The `pairwise` recipe was only three lines and often _faster_ than the C implementation, though.

---

<div class="post-metadata">

**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\
**Post date:** [April 23, 2024, 9:28pm UTC](https://discuss.python.org/t/nwise-itertools/51718/13 "2024-04-23T21:28:35Z")

</div>

- That means the C implementation needs to be fixed. By definition of CPython, it is always possible to write a C implementation that is at least as fast as the python implementation (by copying the exact sequence of API calls that would happen anyway for example)
- I guess the policy is less strict than I would have thought. @rhettinger added it with very little extra discussion from what I can tell.

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 9:34pm UTC](https://discuss.python.org/t/nwise-itertools/51718/14 "2024-04-23T21:34:58Z")

</div>

I am very curious as to how the pure python implementation of pairwise can be faster than the C version…

```c
static PyObject *
pairwise_next(pairwiseobject *po)
{
    PyObject *it = po->it;
    PyObject *old = po->old;
    PyObject *new, *result;

    if (it == NULL) {
        return NULL;
    }
    if (old == NULL) {
        old = (*Py_TYPE(it)->tp_iternext)(it);
        Py_XSETREF(po->old, old);
        if (old == NULL) {
            Py_CLEAR(po->it);
            return NULL;
        }
        it = po->it;
        if (it == NULL) {
            Py_CLEAR(po->old);
            return NULL;
        }
    }
    Py_INCREF(old);
    new = (*Py_TYPE(it)->tp_iternext)(it);
    if (new == NULL) {
        Py_CLEAR(po->it);
        Py_CLEAR(po->old);
        Py_DECREF(old);
        return NULL;
    }
    /* Future optimization: Reuse the result tuple as we do in enumerate() */
    result = PyTuple_Pack(2, old, new);
    Py_XSETREF(po->old, new);
    Py_DECREF(old);
    return result;
}

```

he is only loading two values and returning them… I would like to see some benchmarks because I don’t really believe python is faster than this implementation, and I still think it’s silly to say oh it only takes 3 lines to implement. that’s 3 lines of code duplicated all over the place. We write high level code with python to have these abstractions. I 100% am behind benchmarking and scrutinizing my implementation. In fact, I think it’s absolutely necessary. However, to say, it’s so easy to write why add a convience layer… is silly to me.

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 9:40pm UTC](https://discuss.python.org/t/nwise-itertools/51718/15 "2024-04-23T21:40:22Z")

</div>

I personally think

- batched → python3.12 great addition
- pairwise → python3.10 great addition

I can write batched in like 2 lines of code. I would love to test the performance vs itertools because I doubt it’s faster, and if it is we should 100% be scrutinizing that because there should be no way that’s possible.

however these additions only make code simpler

```python
from itertools import batched

```

I am 100% behind expanding itertools with sane iterating tools… ie… itertools

so… my opinion would be to discuss how to optimize nwise, so it’s the most efficient in C, so we can have a great sane iteration tool that is 100% faster than pure python…

---

<div class="post-metadata">

**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\
**Post date:** [April 23, 2024, 9:46pm UTC](https://discuss.python.org/t/nwise-itertools/51718/16 "2024-04-23T21:46:22Z")

</div>

> [@lsproule](#):
>
> I am 100% behind expanding itertools with sane iterating tools… ie… itertools

The reason why there is a general apprehension is decently exemplified by `pairwise`: It’s C implementation is apparently suboptimal, and therefore now needs to be fixed. This means extra work by multiple people, potentially a dozen man hours for a 3-line python function… Adding a function to the stdlib means practically indefinite support and effort, so it needs to justify it’s usefulness against that barrier. And C implementations also require extra implementation effort by all the other Python implementations which can’t just reuse the code.

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [April 23, 2024, 9:50pm UTC](https://discuss.python.org/t/nwise-itertools/51718/17 "2024-04-23T21:50:43Z")

</div>

> [@lsproule](#):
>
> I would like to see some benchmarks

Here’s one:

```python
 23.1 ± 0.5 μs consume(recipe(iterable))
 37.5 ± 0.5 μs consume(pairwise(iterable))

Python: 3.12.0 (main, Oct 7 2023, 10:42:35) [GCC 13.2.1 20230801]

```

That’s with:

```python
iterable = list(range(1000))
consume = deque(maxlen=0).extend

```

> **Benchmark script**
>
> ```python
> from timeit import repeat
> 
> setup = '''
> from itertools import pairwise, tee
> from collections import deque
> 
> def recipe(iterable):
> a, b = tee(iterable)
> next(b, None)
> return zip(a, b)
> 
> iterable = list(range(1000))
> consume = deque(maxlen=0).extend
> '''
> 
> codes = [
> 'consume(pairwise(iterable))',
> 'consume(recipe(iterable))',
> ]
> 
> from timeit import timeit
> from statistics import mean, stdev
> import sys
> import random
> 
> times = {code: [] for code in codes}
> def stats(code):
> ts = [t * 1e6 for t in sorted(times[code])[:5]]
> return f'{mean(ts):5.1f} ± {stdev(ts):3.1f} μs '
> for _ in range(25):
> random.shuffle(codes)
> for code in codes:
> number = 10**3
> t = timeit(code, setup, number=number) / number
> times[code].append(t)
> for code in sorted(codes, key=stats):
> print(stats(code), code)
> 
> print('\nPython:', sys.version)
> 
> ```
> 
> [Attempt This Online!](https://ato.pxeger.com/run?1=bVNLjtQwEF2w8ylqF7sVQppRIxSpr4DYNxHKp8JYJHawK8M0rTkJm9nAnj0rDsABOA1lJ-luPtkkqXr16tWz6_PX8Ui31jw-fpmoe_ry15O6c3YA0gNqAj2M1hE4HLEiITzSNMIekiQREaYJHVnb-xU5Vtp91B5TIMQZ09i-x4a0NWdUix8mFKLFjqkbPaIMRFXdoyoE8FOlUHMf5rhkYsLgPck6hVfWLBHHmpyBT3qUoUoJsVYwQa89SVeZdyi3eZ4rJRqWMQ0hF0XIobrv0exzlTEzmlaE2RjVomfMIbZIliK5TncRpZL0T8jf8wRAKcR_TJ3_5oyniliqbs4WDViZlOMt3okl5I9-_eSRWjsIETiCzlMQXMChhM46CD-gTXz7h2hzaOBlCCwOUxyPYANbfBGrKJR4ZsdWRt5DgJfqUOzK8trqLjkFdZK8KnbZtnuAH9_gFKXG2E2M_fzugW8JE78NxPMhPN8t7ecBMn87dV2PUZifz_OfAeaCePjTUKNj3dt8s7k5hynclGhm5GHXwjVNF_h-fil4tgQudZchs2oc-fAlKXHdf3EjykjhPR730cdlhtFpQ_LK2TQW8g2cM8kb8zouV5Gk4fCyO3Se10DNq7Zs3Lp5vwE)

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 10:00pm UTC](https://discuss.python.org/t/nwise-itertools/51718/18 "2024-04-23T22:00:45Z")

</div>

In this case, I don’t think that mean we should be hesitant about adding functionality. I do think that we need to revisit what is going on with pairwise, I will try seeing if I can do some benchmarks on the C code for pairwise, and see if we can implement it in terms of tee… That isn’t a bad idea actually. I mean since tee is written… I should be able to write them in terms of tee I want to see how this will affect performance. I 100% agree that this is unacceptable, however, at the end of the day… We can do better, and I think it’s worth it to do better.

---

<div class="post-metadata">

**Author:** ![Stefan2](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/stefan2/32/18492_2.png) [@Stefan2](https://discuss.python.org/u/Stefan2)\
**Post date:** [April 23, 2024, 10:00pm UTC](https://discuss.python.org/t/nwise-itertools/51718/19 "2024-04-23T22:00:54Z")

</div>

Related topic: [Why is `pairwise` slower than `zip`?](https://discuss.python.org/t/why-is-pairwise-slower-than-zip/34308)

---

<div class="post-metadata">

**Author:** ![lsproule](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lsproule/32/19420_2.png) [@lsproule](https://discuss.python.org/u/lsproule)\
**Post date:** [April 23, 2024, 10:04pm UTC](https://discuss.python.org/t/nwise-itertools/51718/20 "2024-04-23T22:04:51Z")

</div>

I mean tee is clearly faster, but we can use tee in C, and see how that affects performance… I will benchmark what I have, and see what I can do. I 100% agree with this take, if we are implementing in C there should be no excuse for it being slower.

[Next page](https://discuss.python.org/t/nwise-itertools/51718.md?page=2)