What's new, but obscure in Python 3.9

Python Help
1

Hi, Just updated Anaconda to Python3.9 and re-read the what’s new, optimisations section which has this example code:

sums = [s for s in [0] for x in data for s in [s + x]]

I couldn’t work out what it did so ran it:

In [2]: data = [1, 2, 3]

In [3]: [s for s in [0] for x in data for s in [s + x]]
Out[3]: [1, 3, 6]

In [4]: data
Out[4]: [1, 2, 3]

In [5]:

It seems to create a running total but is obscure to me in how it works.

  • Could someone explain how it works for me please.
  • Is it of appropriate readability level given the likely audience of the what’s new page?

Thanks in advance, Paddy :slight_smile:

2

I know a lot more but it was unintuitive/obscure to me:


for s in [0]:
    for x in data:
        for s in [s + x]:
            print(s)
            
1
3
6

Is this an idiom? is it written about?

Thanks.

3

First, I have to admit this is not readable at all. It is a terrible example against the Zen of Python.

To get how it works, for x in [a] just turns a statement x = a to a for statement eligible in a list comprehension and they work the same. Wherever you can’t use such statements, you can use this little magic. So the list comprehension can be demystified as:

s = 0
for x in data:
    s = s + x
    print(s)
1 Like
4

Thanks @frostming. That tallies with what I have worked out.

I tried a more difficult example of returning sums and sums of squares and got the following:

In [21]: [(s, s2) for s, s2 in [(0, 0)] for x in data for s, s2 in [(s + x, s2 + x**2)]]
Out[21]: [(1, 1), (3, 5), (6, 14)]

I wonder is this the only way to do such running sums in a comprehension?

That’s two of us that think it’s obscure, and counting :slight_smile:

5

That’s an especially obfuscated example of how to create a temporary
variable inside a list comprehension, before the walrus operator.

If you want to do a running sum, you might do this:

data = [1, 2, 3, 4]
sums = []
total = 0
for value in data:
    total = total + value
    sums.append(total)

How can we write that as a list comprehension? Without using the walrus
operator. Here’s one attempt:

sums = [total for value in data]

but that doesn’t work because total has no value. We can initialise it
from the outside:

total = 0
sums = [total for value in data]

but we’re using two statements instead of a single comprehension, and
the total never gets updated into a running sum. How do we push the
initialisation into the comprehension? By turning it into a loop:

data = [1, 2, 3, 4]
sums = []
for total in [0]:
    for value in data:
       total = total + value
       sums.append(total)

That corresponds to this comprehension:

sums = [total for total in [0] for value in data]

Good news from the What’s New: that loop for total in [0] inside a
comprehension is now as efficient as a regular assignment total = 0.

Okay, so now we got our initial value inside the comprehension. But the
total is not being updated into a running sum, it’s always 0. How can we
do that, using only for-loops?

data = [1, 2, 3, 4]
sums = []
for total in [0]:
    for value in data:
       for total in [total + value]:
           sums.append(total)

which gives us this comprehension:

sums = [total for total in [0] for value in data for total in [total + value]]

Obfuscate the names:

# s = total (running sum)
# x = value
sums = [s for s in [0] for x in data for s in [s + x]]

and we’re done.

With the walrus operator, we can use the less obfuscated:

data = [1, 2, 3, 4]
total = 0
sums = [total:=(total + value) for value in data]

although it has the side-effect of updating the value of total.

1 Like
6

Thanks @steven.daprano I too took the time to write up my own, similar, explanation in my blog.
I need to add the walrus equivalent as it is not as ugly.

I guess the What’s new in 3.9 doc is stuck with the original?

7

What's New in 3.9 could be updated. Any proposed PR should be against the copy in main, which would be backported. If you submit one, ask the original author (contributed by …) to review. But keep in mind that the purpose of the example is to illustrate how one might benefit from the optimization.

8

On a slightly related side note, I do like itertools.accumulate
It’s not a comprehension, but it is very readable:

data = [1, 2, 3, 4]
list(itertools.accumulate(data))    # [1, 3, 6, 10]

Getting the running total of the data and the square of the data could be done with:

list(zip(itertools.accumulate(data), itertools.accumulate(x ** 2 for x in data)))

I think there’s probably something funky that could be done with func= on accumulate to get the pairs.

1 Like
9

Hint for Paddy3118 and all in this list :


The machines since they are digital, always
start to count with zero or nought like this :
0
When you want to write maps with braces like
tuples (for maps?)
write without gaps like this :
[[[[[0,1,2,3],[0,1,2,3],[0,1,2,3],[0,1,2,3]]
MS-DOS or free DOS (wingoofz) never start at zero, otherwise
they are bad copies of unix.
There might be chip error too in hardware.
You could check this bitwise with | or with or ||.
Do you have zero with ant egg or with MS-Dos like Zero ?
Do not torment your machines with overload of electricity …

`

10

Replace every comma with comma-space and you get the Python norm. I don’t understand your description of why it should change?