Why can't iterable unpacking be used in comprehension?

If you have

l = [[1,2,3],[4]]
[*b for b in d]

it gives

  File "<stdin>", line 1
    [*b for b in d]
     ^
SyntaxError: iterable unpacking cannot be used in comprehension

Is there a specific reason for the inability?

2 Likes

Notice!! I’m not an expert actually I only started a few days ago, but I did some research and this is what I found
that is only available in calls and when using assignments
I linked the reference below, I hope it helps

I don’t think that there is a specific reason, just the way the language has evolved. Comprehensions and iterable unpacking were added at different times.

See this thread here:

https://mail.python.org/archives/list/python-ideas@python.org/message/7G732VMDWCRMWM4PKRG6ZMUKH7SUC7SH/

where supporting that is unpacking in comprehensions is discussed.

At this point, I expect all it will take is:

  • someone to get a core developer to sponsor the idea;
  • and then write a PEP;
  • I expect that the Steering Council will probably accept it;
  • then do the work.

If nobody wants to do the work, or write the PEP, it won’t happen.

I saw this, the accepted answer just says you can’t use iterable unpacking here

I came here because I thought it should be added to the language. But, is it really a worthwhile feature to have?

I can only think of using it to flatten 2d lists/tuples.

I.e. something like [[1,2,3],[10,87]] into [1,2,3,10,87]

Note that you can have multiple levels of iteration in a comprehension

a = [[1, 2, 3], [4]]
[c for b in a for c in b]
# [1, 2, 3, 4]
1 Like

The following recursive function is designed to flatten a list of any depth. Admittedly, it is not the same as what you are trying to do, since it uses a test and a base case outside the list comprehension.

def flatten(a):
    return [c for b in a for c in flatten(b)] if hasattr(a, '__iter__') else [a]

print(flatten([[1, [2, [3, 4, 5], 6], 7], [8, 9, [10, [[[[11]]]], 12]], 13]))

output:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]

Unfortunately a general purpose flatten function is quite hard to get right.

Your version has at least five problems:

  • Your test for iterable objects is not correct.
  • Some iterable objects, like strings and bytes, may optionally need to be treated as if they were non-iterable.
  • As it stands, strings lead to RecursionError.
  • Recursive sequences also lead to RecursionError.
  • Nor are infinite iterables.

More detail follows, but now, She Who Must Be Obeyed is calling.

Actually, the function was an attempt at flattening only lists, so a specific test for that type is needed.

Here’s a version 2:

def flatten(a):
    return [c for b in a for c in flatten(b)] if isinstance(a, list) else [a]

There’s still a problem with this one, though. I don’t like that the following …

print(flatten(1))

… outputs …

[1]

EDIT:

Are we technically off-topic now, since the original post was specifically about iterable unpacking within a list comprehension?

The intent was to only flatten list objects. One of the reasons for this is that deciding what type the resulting objects should be when the original nested object is a mixture of iterable types would be problematic. There would also be the problem of how to handle dict objects. We would need to decide whether the values within embedded dict objects should be flattened. Also, in instances of any type that inherits from the list type should not be flattened. Maybe this would be a better test, assuming that a is the object to be flattened:

type(a) is type([])

Your revised flatten still fails on recursive lists, such as lists that contain themselves:

alist = [1, 2, 3]
alist.append(alist)

or lists which are mutually recursive (a contains b which contains a again).

alist = [1, 2, 3]
blist = [4, 5, 6, alist]
alist.append(blist)

Flattening lists is a deceptively tricky problem. You are not going to do it with a one-liner.

Yeah, it can be a challenge to deal with such things, including lists that have internal references to certain internals of themselves.

t = [[1, 2], [3, 4], [5, 6], [7, 8]]
def flatten(lists):
    flat = []
    for item in lists:
        if type(item) is type([]):
            flat += flatten(item)
        else:
            flat += [item]
    return flat

t[2].append(t[1])
t[1].append(t[2])
print(t)
print(flatten(t))

Output:

[[1, 2], [3, 4, [5, 6, [...]]], [5, 6, [3, 4, [...]]], [7, 8]]

RecursionError: maximum recursion depth exceeded

Failure on flattening that list is probably a feature, not a bug. But a different failure mode might be better.

1 Like

Those are good points, and the path ahead is left open as an exercise for the reader …

The wording of that error message could stand to be improved, as this restriction isn’t specific to comprehensions: you can’t write *b as a standalone expression in general, as the compiler doesn’t know what the type of the resulting sequence should be. Every time you use iterable unpacking, there has to be a surrounding construct that specifies what the iterable should be unpacked into.

Several other replies have covered ways of writing a flattening function that produces individual items from iterables in another iterable. If the goal is instead to unpack the comprehension into the list, then adding parentheses around the comprehension part will do the trick.

1 Like