Add built in flatmap function to functools

Python already provides built in functions for functional programming paradigms like map, filter, and accumulate (using functools reduce()). However, an equally useful function flatmap is not directly offered. Of course, there are ways for a python developer to implement this on their own, as shown in here and here. But I think offering this function as a built in operator would be very helpful for new developers and help developers make more concise, elegant code.

Specifically, flatmap would take the following arguments:

flatmap([function], [list or iterable])

It would output the result of applying the function to each item in the list or iterable and then flattening the entire list.

Ex:
flatmap(lambda x: [x, x, x], [1, 2, 3]) = [1, 1, 1, 2, 2, 2, 3, 3, 3]

2 Likes

Is this just flatten(map(func, values))?

First, we would need to decide what the semantics of flatten are. There is a reason why Python still does not have a flatten function after 30+ years.

Fairly certain sum(map(func, values), []) would do the trick

In [1]: sum(map(lambda x: [x, x, x], [1, 2, 3]), [])
Out[1]: [1, 1, 1, 2, 2, 2, 3, 3, 3]

edit: I see this as solution in OP 2nd link…ops

I disagree. flatten requires learning and memorising a new word, one which newcomers likely haven’t encountered yet. List comprehension with nested loops can be surmised from basic examples of Python’s syntax.

3 Likes
flatten = itertools.chain.from_iterable
7 Likes

Forgot about that one.

So…

flatmap = lambda func, iterable: itertools.chain.from_iterable(map(func, iterable))

Can we have this added to itertools Recipes for incubation to eventually include in itertools as a first-class function?

Good to know. Cannot say I like it…makes me wish for a __sum__ so you don’t have to think about these things. I know it’s a pitfall for users of numpy too, would be nice if sum could just delegate down to numpy.sum but that’s a different discussion

It is not clear on which type the __sum__ method would be looked up if the list is [a, b] where type(a) != type(b).

This is why we have operator.__iconcat__.

The problem is that a flatmap function is somewhat nieche. I have not had any need for a flat map function, and if I did I would just write my own. Unlike some communities we can and do write functions that are some what trivial, *Cough Cough* is-odd - npm *Cough cough*.

See https://github.com/python/cpython/blob/80b9b3a51757ebb1e3547afc349a229706eadfde/Python/bltinmodule.c#L2659

1 Like

So, … because it is “trivial”, and I can write it in one line, it need not be included??

10.12 Batteries Included :chains:

Python has a “batteries included” philosophy. This is best seen through the sophisticated and robust capabilities of its larger packages. […snip]

Let’s look at itertools for a second:

def chain(*iterables):
    return (element for it in iterables for element in it)

chain_from_iterable = lambda iterables: (element for it in iterables for element in it)

def compress(data, selectors):
    return (d for d, s in zip(data, selectors) if s)

def starmap(function, iterable):
    return (function(*args) for args in iterable)

Those are one-liners which were turned into precompile functions bundled in the itertools module. Why bother? We could easily write them ourselves.

The following are “one-liners” from the itertools recipes section:

def take(n, iterable):
    return list(islice(iterable, n))

def prepend(value, iterator):
    return chain([value], iterator)

def tabulate(function, start=0):
    return map(function, count(start))

def tail(n, iterable):
    return iter(collections.deque(iterable, maxlen=n))

def nth(iterable, n, default=None):
    return next(islice(iterable, n, None), default)

def quantify(iterable, pred=bool):
    return sum(map(pred, iterable))

def ncycles(iterable, n):
    return chain.from_iterable(repeat(tuple(iterable), n))

def sumprod(vec1, vec2):
    return sum(starmap(operator.mul, zip(vec1, vec2, strict=True)))

def sum_of_squares(it):
    return sumprod(*tee(it))

def transpose(it):
    return zip(*it, strict=True)

def matmul(m1, m2):
    return batched(starmap(sumprod, product(m1, transpose(m2))), len(m2[0]))

def flatten(list_of_lists):
    return chain.from_iterable(list_of_lists)

def triplewise(iterable):
    return ((a, b, c) for (a, _), (b, c) in pairwise(pairwise(iterable)))

def unique_justseen(iterable, key=None):
    return map(next, map(operator.itemgetter(1), groupby(iterable, key)))

def first_true(iterable, default=False, pred=None):
    return next(filter(pred, iterable), default)

All of these are simple enough we can write them ourselves, yet they’ve been added to the recipe area and into more-itertools. If “I have not had any need for a flat map function” is the discriminator here, how did tabulate get into the recipes? I’ve never had any need for it. So I think “niche” and “trivial” are not entry barriers here.

On the other hand (recently in the PSF blog):

Overall, there was agreement that the original motivations for a large, “batteries-included” standard library no longer held up to scrutiny. “In the good old days,” Ned Deily reminisced, “We said ‘batteries-included’ because we didn’t have a good story for third-party installation.” But in 2023, installing third-party packages from PyPI is much easier.

I’m guessing Raymond found it educational (from the recipes introduction):

The primary purpose of the itertools recipes is educational.

A secondary purpose of the recipes is to serve as an incubator.

The documentation also has a Functional Programming HOWTO, indicating Python supports it (functional programming, that is). Including flatmap could be considered educational. At the very least, if it were included in itertools recipes, or in the Function Progamming HOWTO, typing flatmap into the documentation search would cause it to pop up!