List Comprehension as Compulsory Functional Style

Python Help
,
17

By Leland Parker via Discussions on Python.org at 04Jul2022 15:53:

I appreciate everyone’s interest and consideration in addressing this
question. :+1: I’m just trying to understand the philosophical
foundation of using list comprehension only for functional style
procedures. […]

@cameron hit on the philosophical question with the point that list
comprehension creates a list in memory and therefore wastes resources.

For the record, I do not consider that a philosophical position but a
pragmatic position. I suppose that since programming is in the domain of
“getting things done” it ventures into the philosophy of programming,
but really I was speaking of pragmatic effects here.

Philosophically, to me a list comprehension (or its progressive variant
the generator expression) can be concise way of expressing “here’s a
bunch of values, all of which are derived from this expression here (the
leading expression in the comprehension)”. Thus making it clear that all
these things are instances of some generic situation or case.

[ fn(x)
  for x in source_of_data_here()
  if condition
]

Says to me:

  • all the values were computed the same way
  • they came from this source domain
  • intersected with the source domain of values satisfying condition

As such, the result has a clear semantic meaning.

Of course I also use list comprehensions for mundane practical reasons
too, like “I need a list” or “make a copy of these items” (often spelled
list(the_items) though).

As such, the list comprehension is a nice functional expression: that it
is implemented procedurally internally is not pertinent.

And that bring us to the issue of list comprehensions with side effects:
firstly they can be hard to read, requiring careful thought about the
order in which things happen and secondly the comprehension is such an
apt “functional” expression of some things that using it for operations
with side effects may be actively misleading.

Prior to the walrus operator (:= inline assignment) a comprehension
with side effects wouldn’t even have any overt assignments in it to give
the game away.

These are all reasons to my mind to pretty well never use a list
comprehension to modify data. I almost always read them as static
expressions producing functional results, and expect others would
usually do so as well.

At the very least such a thing requires a LOUD obvious leading comment.

This is an adverse and objective (not based on opinion) outcome. But
what happens to this potential consumption of memory if the
comprehension isn’t assigned to anything? I’m genuinely interested in
knowing so I can understand Python better.

The list gets constructed, consuming memory. Then its reference count
goes to zero and the list and memory are released. The heat death of the
universe advances further.

This is along the same lines. What are the inefficiencies?

Because building a list, particularly incrementally, has costs. Whenever
the list gets bigger, more storage is required. Usually the internals of
such things allocate storage in bursts i.e. over allocate memory for the
list to grow into. But that just mitigates things. When the buffer for
the list references fills, it becomes necessary to allocate a new chunk
of memory and copy the references into it.

[ Aside: there’s a length hint available for objects:
3. Data model — Python 3.12.1 documentation
which the internals can use to size an initial preallocation
for a list being built from an iterable.
Still just mitigation.
]

If you’re just iterating, none of that overhead needs to occur.

Readability doesn’t always prevent this shorthand for: because a simple list comprehension with side effects can be very transparent, just as a simple single-line if: can be more readable than when broken into two lines:
(else: break). (Off-topic example simply for illustration. No need to respond, especially since this is already the subject of a recent topic.)

I find this perfectly readable:

ints = [1,2,3,4,5,6,7,8,9]
[ints.remove(x) for x in ints[::-1] if x %2 == 1]  #removes the odd numbers

(Yes, it iterates backwards to ensure that the tail of the shrinking list doesn’t slip under the iteration. Yes, it’s probably “too clever” and yes, that was a fun puzzle to work out. :nerd_face: )

I was going to say exactly this re the backwards iteration.

The reader has to look at the ints[::-1], a well defined idiom which
is still rarely seen, and think why did the author choose this weird
form of the source values?

Versus a functional form:

ints = [1,2,3,4,5,6,7,8,9]
ints2 = [ x for x in ints if x % 2 == 0 ]  # keep the even values

which is far easier to read and would work with the source values
(ints) in any order. Because there are no side effects.

This is why functional forms are so nice: you don’t have to think hard
about order of operations and side effects (the ints.remove(x) manking
the iterator driving the comprehension).

I actually find your example an argument against comprehensions with
side effects.

Cheers,
Cameron Simpson cs@cskk.id.au

1 Like