Itertools.endpoints - mark the first/last item in a sequence/iterable

I’d like to ask for some feedback for the following idea. There are some minor details left open, like the naming is probably not great, but most ideas are quickly rejected (thank goodness), so I’d like to test just the basic acceptance.

Proposal: Add itertools.endpoints(iterable) that will iterate the iterable (yielding values unchanged) and will provide two boolean properties .first/.last that are True if and only if the first/last item respectively was just yielded.

This proposal was inspired by Jinja2. I find it a useful feature there. Use-cases are functions similar to join, reduce, etc. where the position within the input sequence matters (begin, in the middle, end) because the algorithm needs some bootstrapping or finalization or just simply due to output formatting requirements.

Example of usage:

numbers = [10, 20, 30]
for n in (ep := endpoints(numbers)):
        if ep.first:
                total = n
        else:
                total += n
        if ep.last:
                print(f"{total=}")

Implementation of the .first is trivial. The .last is trivial for sequences with known length,
but in general a pre-fetch must be done to check for a StopIteration condition.

2 Likes

There is no such thing as “iterating the iterable unchanged”. Iterating an iterable is inherently permitted to change it. For example, when you iterate over a file, the file position changes. There is no general way to undo these kinds of changes, because the iterable’s internal state can be represented and updated in arbitrary ways according its own logic. For example, a generator progresses by executing more code, and has no memory of its previous state (and thus cannot revert to that state).

2 Likes

Thank you for the notice. It was a badly written sentence and I edited it.

Ah, I understand it now.

The need for a pre-fetch means the underlying iterator will in general be one position ahead of the wrapper, which users might need to be aware of.

You might also be interested in existing recipes for this sort of problem:

1 Like

I’m not sure how often this would be useful, but potentially there are users out there. One place for such a thing to prove its worth[1] is in the more-itertools package (if the maintainers agree with you). Looking at the docs, it seems similar to peekable but includes a flag for the first element.

I’ll also just say that your example usage is pretty unconvincing, because it’d be much more concise to not use this. A good example would take advantage of this in a way that’s not trivial to just put outside the loop.


  1. and maybe find a permanent home ↩︎

2 Likes

The example code you provided could be made much more simpler and readable if one simply uses the builtin sum function. What benefits could users have when using the endpoints function? Could you provide some code that either can be improved in any way by using the endpoints function or code that would be impossible to make without said endpoints function?

I also find that grouping the initializing, intermediate, and finalizing logic together in a for loop as demonstrated in the for loop not only causes me to waste time questioning why they were grouped together, but that it’s often unnecessary. This is because the beginning logic can be reimplemented and executed before the for loop and the finalizing logic can be placed after the for loop.

EDIT: Thanks to Neil Girdhar, I took a look at the mark_ends function in the more_itertools library and its usecases seem rather restricted to specific iterables such as files and requests. What is the off chance that, in general, an iterable will have both header and footer elements? I think the itertools modules is supposed to be for working with iterables in general, so I oppose the inclusion of this function in the itertools library for the sake of consistency and organization. So that leaves the question of if this function could be included in Python’s standard library, what module should it be in?

This is accomplished by mark_ends

4 Likes

I was focusing on showing the usage of a new feature and wanted to keep it as short and simple as possible.

Great! The function exists already!

I think we can close this discussion. I appreciate recevied comments, thanks to everybody.

3 Likes