List Comprehension as Compulsory Functional Style

Python Help
,
2

Ok, it was me, who recently pointed out that comprehensions should not be misused as an alternative way how to write a for loop. I will try to show my main arguments:

Readability

I would format the comprehension this way for readability:

[
    a_list.pop(len(a_list) - 1 - i)
    for i, ss in enumerate(a_list[::-1])
    if ss[-1].startswith('X1:')]

It is still a very complex piece of code: iteration in reverse, using iteration index, modifying the length of the original list, computing with this changing length :exclamation:, creating a new list… :exploding_head: Originally I thought that the code splits the list into two according to the condition but I realized that it fails to do so because the enumerate index i comes from the original length but you compute the pop index combining it with the actual (different) length of the list. You can easily make the index to go out of range.[1]

Here is much more readable implementation which works:

a_list_x1, a_list_nox1 = [], []
for item in a_list:
    if item[-1].startswith('X1:'):
        a_list_x1.append(item)
    else:
        a_list_nox1.append(item)

…and the same for lovers of dense code while retaining some readability:

a_list_x1, a_list_nox1 = [], []
for item in a_list:
    (a_list_nox1, a_list_x1)[item[-1].startswith('X1:')].append(item)

Conclusion for this use of comprehensions: readability suffers, similar dense code is a hotbed of mistakes.

Purpose

I think the philosophy of Python aims to minimize availability of multiple ways how can be the same thing accomplished. From the Zen of Python :slight_smile:

There should be one-- and preferably only one --obvious way to do it.

By the way it is the opposite of the Perl motto:

There’s more than one way to do it

Comprehensions and generator expressions were designed to be used for creating new containers (lists, sets, dictionaries) and iterators. See the documentation, PEP202, PEP0274.

If these constructs were explained to be used as another way of writing a for loop then I am sure that they would have never been accepted into Python.

Result of the expression is always list of certain number of None. E.g. [None, None, None] What is this for? It looks like the only result wanted from the expression is its side-effect. I think this is an obvious misuse of the list comprehensions. They were never intended to be used like that.

Note that the way the code works is a little bit complicated again. It iterates a_list from the end and for every match list.remove() searches the list from the beginning ! and removes the item according to the value. So it can remove a different item that the item matched!

More direct implementation using a for loop. Here we remove directly the item matched:

a_last_index = len(a_list) - 1
for reversed_index, item in enumerate(a_list[::-1]):
    if item[-1].startswith('X1:'):
        del a_list[a_last_index - reversed_index]

As we could se in both examples, modifying lists (in the sense of adding and removing items) in-place is pretty complicated. In Python we usually create an iterator:

(item for item in a_list if not item[-1].startswith('X1:'))

or a new list - for small lists, when we need to iterate it multiple times etc:

[item for item in a_list if not item[-1].startswith('X1:')]

Interesting references

https://mail.python.org/pipermail/python-list/2008-May/632671.html

  1. I hope I will soon publish my Jupyter notebooks with the tests. ↩︎