Chaining a single iterable does have its uses, but this isn’t one of them.
I agree here; this was the alternative that first occurred to me, too.
I think that’s a really important perspective, so I appreciate your input!
Yep, this was my initial thought as well. I certainly wasn’t planning on abandoning the core proposal because of this example, but I still figured that even the worst cases are worth discussing.
That may well be true. I don’t have a great sense of how to find an answer to that question, other than my quick poking through the standard library and the results of a quick search that turned up some stackoverflow posts: this one about the general idea of flattening a list of lists was viewed 4.6 million times (and there are others in the same vein with a smaller but still nontrivial number of views: 1, 2), which maybe suggests that the general problem is common, and a few more examples specifically discuss unpacking in comprehensions (1, 2, 3). There are, of course, many more search results that I’m not bringing up here. But I’m not sure I have the context to know whether that’s common or rare or somewhere in between in a global sense.
1 Like
I assume that the intention here was one of the following two forms for chaining multiple iterables together, both of which are equivalent to the original example of [*x for *x, y in list_of_lists]:
list(itertools.chain(*(x[:-1] for x in list_of_lists)))
list(itertools.chain.from_iterable(x[:-1] for x in list_of_lists))
And the fact that we’ve seen multiple instances in this thread of the itertools options being confused is further evidence that there’s room for improvement there.
2 Likes
indeed I meant this one
list(itertools.chain(*(x[:-1] for x in list_of_lists)))
sorry for the mistake
It’s not my intention to be negative; my goal is simply to discuss edge cases and help make them clearer for beginners. As mentioned earlier, this also serves as a good way to highlight and distinguish different usages properly.
Thanks for the demo. I was able to test many use cases, and the proposed syntax is definitely a lifesaver in several situations. That said, some cases can be a bit unintuitive, for example:
lists = [[1], {2:2}]
a = [*[x if isinstance(x, list) else list(x)] for x in lists]
print(a)
# current
nums = [1, "2"]
a = [x if isinstance(x, int) else int(x) for x in nums]
print(a)
On my first attempt, I tried something like [*x if isinstance(x, list) else *list(x) for x in data. I don’t think any of these use cases are deal-breakers in any way, but I believe they are worth mentioning in the PEP or documentation.
I encounter comprehensions so often, and they can get complex, so smoothing the learning curve for unpacking within them is valuable.
2 Likes
No worries! I certainly wasn’t reading your responses as negative. Thinking about unintuitive examples is very helpful, so I appreciate you pointing them out.
I’m glad you found it useful! I had fun putting it together, and I’m still kind of amazed that the whole Emscripten thing is even possible…
Good point. For me, both of the examples you give are cases where I would lean toward using some structure other than a comprehension, as I think there are clearer ways to phrase them.
But I definitely do think that the PEP and the docs could be clearer about * and ** only being allowed at the top-most level of that expression. I’ll try to improve the words in the next couple of days.
In addition to changes to the PEP and the docs, I also think it might be good to have better error messages for these kind of errors. For example, right now, the reference implementation says the following with your example:
File "<python-input-0>", line 1
[*x if isinstance(x, list) else *list(x) for x in data]
^^
SyntaxError: invalid syntax
which is not wrong, but maybe also not as informative as it could be. I expect there are a bunch of other similar places where we could provide more informative error messages.
I don’t think I saw anyone mention this
The x[:-1] version works if you have a sequence, but not for any arbitrary iterable, e.g. [*x for *x, _ in generator_of_generators] (Same general principle as having to use next(iter(x)) instead of x[0])
I recently added a feature at $work that was dealing with a lot of lazily constructed nested iterables and tree structures. Being able to use more unpacking expressions would have been nice to replace a lot of the uses of chain.from_iterable
A particular “real-world” application of that splitting and unpacking again behavior is in-order depth first traversal of a tree (Tree traversal - Wikipedia)
3 Likes
I thought about this while drafting my answer above. [1]
While true that slicing requires the parent iterator to produce Sequences, the statement [x for *x, y in generator_of_generators] necessarily materializes each child generator into a list. So the above statement is equivalent to [list(z)[:-1] for z in generator_of_generators] - you are still consuming the entirety of each child generator and storing the results in memory.
Regardless, this toy statement requires this child generators to be fully consumed, as there’s no other way to know which element is last. More generally, unpacking isn’t the right tool to use to consume generators efficiently, unless you expect to need the entirety of its response to be held in memory at once.
(Edit: minor editorial edits.)
Edit 2: @Stefan2 noted that the correct equivalent is list(z)[:-1]
Note that this point has no real bearing on this proposal, as it does not depend on new syntax. I don’t expect to reply further. ↩︎
6 Likes
Thanks again for all of the feedback, everyone! I made a pass to update the text of the draft PEP since I last posted. In case everything is hard to dig up from earlier in the thread, here are links to all the current info:
- Draft PEP: rendered, source repo
- Reference Implementation: Emscripten demo, source repo, diff against main repo
Still happy to continue the conversation if there are lingering questions / pieces of feedback!
4 Likes
This is a nice little feature that I’ve missed several times in the past. If you’re still looking for a sponsor, I’m happy to volunteer.
I just read the draft PEP and implementation and I think it’s in good shape already. Two areas where the PEP could be improved:
- There is a lot of discussion of exactly how this would be implemented in the bytecode. Bytecode is a CPython implementation detail that frequently changes, not a part of the language. As such, it shouldn’t be part of the “Specification” section of the PEP. If anything is particularly tricky it could be discussed under “Reference implementation”, but mostly this can be deferred to when you actually implement it in CPython.
- It could be useful to discuss whether other programming languages that have some sort of comprehension support this syntax.
18 Likes
Fantastic; thanks!!
Ah, OK, that makes sense. I’ll try to rephrase those parts of the PEP, and I’ll ping here again when I have an updated draft.
I can also try to look into this.
2 Likes
Alright, I made some updates to the draft PEP: rendered version, diff
The big change was replacing the portions referencing bytecode details with equivalent explanations in terms of actual Python code (or simply removing them if I thought they were adequately explained elsewhere). But I also fixed some typos and grammatical errors and changed some phrasing throughout. I also added links to a few more relevant StackOverflow posts, though I’m not sure they really add much.
I did a little bit of digging here, but I didn’t find any evidence of anyone supporting this kind of comprehension. I didn’t try to add any of this to the PEP yet, but I’ll include the results of some of my poking here so that we can discuss (with the caveat that I’m not an expert in any of these other languages so I’m not 100% sure there’s not a better way).
There are lots of other languages that support this kind of flattening with syntax similar to what we already have in Python, like comprehensions with double loops:
# python
[x for xs in [[1,2,3], [], [4,5]] for x in xs * 2]
-- haskell
[x | xs <- [[1,2,3], [], [4,5]], x <- xs ++ xs]
# julia
[x for xs in [[1,2,3], [], [4,5]] for x in [xs; xs]]
; clojure
(for [xs [[1 2 3] [] [4 5]] x (concat xs xs)] x)
or some with a combination of map and flatten:
# python
list(itertools.chain(*(xs*2 for xs in [[1,2,3], [], [4,5]])))
// Javascript
[[1,2,3], [], [4,5]].flatMap(xs => [...xs, ...xs])
-- haskell
concat (map (\x -> x ++ x) [[1,2,3], [], [4,5]])
But I haven’t yet found a language that has both unpacking and comprehensions and allows unpacking within comprehensions. For example, the following syntax is invalid in Julia:
julia> [xs... for xs in [[1,2,3], [], [4,5]]]
ERROR: syntax: "..." expression outside call around REPL[1]:1
Civet and coffeescript both accept that syntax, but they attach ... to the whole comprehension rather than just to xs:
🐱> [...xs for xs of [[1,2,3], [], [4,5]]]
...
[ [ 1, 2, 3 ], [], [ 4, 5 ] ]
coffee> [...xs for xs in [[1,2,3], [], [4,5]]]
[ [ 1, 2, 3 ], [], [ 4, 5 ] ]
I’m not sure of other languages that support both comprehensions and unpacking that we could use as additional points of comparison. If anyone has another example come to mind, please let me know!
5 Likes
FWIW, after a little more poking, it looks like Civet does have a form that kind of mirrors this mix of comprehension and unpacking:
🐱> for xs of [[1,2,3], [], [4,5]]
... ...(xs++xs)
[
1, 2, 3, 1, 2,
3, 4, 5, 4, 5
]
2 Likes
@Jelle (and others), I’m curious for your thoughts on where this discussion of other languages could fit into the PEP. I haven’t been able to find the right spot yet…
From a quick scan, it looks like PEPs that mention other languages tend to do so under their Rationale section, usually using the inclusion of their proposed feature in other mainstream languages as justification for its inclusion in Python. Unless we turn up more examples, though, it looks like unpacking within comprehensions is not supported in most languages that have both comprehensions and unpacking, so it doesn’t seem like it really fits under “Rationale,” at least not in that same way. The other place that jumps to mind is under “Concerns and Disadvantages,” but that heading seems too strongly-worded, i.e., I’m not sure that the lack of support in other languages is a “concern” or a “disadvantage,” per se. I suppose we could also add another section “Other Languages” or similar, but even then I’m not quite sure where it would best fit, or how much detail is worth including.
A brief appendix could work, similar to PEP 695 (PEP 695 – Type Parameter Syntax | peps.python.org). If there’s not much to say, it’s OK to not say much. Prior art in other languages is nice, but if Python is innovating here, that’s not a show-stopper for new syntax.
4 Likes
Ruby does have pretty much this exact thing called Enumerable.flat_map:
[[0, 1], [2, 3]].flat_map {|e| e + [100] } # [0, 1, 100, 2, 3, 100]
which, with this proposal, translates into:
[*e + [100] for e in [[0, 1], [2, 3]]]
One parenthese pair was omitted :
[*(e + [100]) for e in [[0, 1], [2, 3]]]
The parentheses are not needed here since the proposed grammar rule uses the existing star_named_expression rule, which does not require parentheses.
For example, [*[1] + [2]] currently evaluates to [1, 2].
Ok, I did not know that.
2 Likes
That’s really unfortunate because that’s very difficult to read.
7 Likes