I’m writing to introduce PEP 798, which proposes adding support for using unpacking syntax in comprehensions and generator expressions. We’ve already had some good discussion in a pre-PEP thread, but I look forward to continuing the conversation here 
. Any feedback is welcome!
50 Likes
I was in a situation just this week where I was wanting this feature. It makes sense and isn’t difficult to comprehend. Would definitely like to see this happen.
9 Likes
[*it for it in its] is still a lot of characters. I’m wondering, if
# These are analogous.
[t for t in things]
[*things]
Maybe it’d be reasonable to have
[*(*iterables)]
Similarly
{**(*dicts)}
Just a crazy idea.
Just sharing the demo URL here:
Also, the concerns I raised in the previous thread have all been addressed in the latest PEP version. Thank you!
4 Likes
I’d like to ask why the decision to not allow stared expressions for keys/values of mappings has been made.
{*k: v for k, v in items}
This syntax might be useful in the future, e.g. for Type Mappings. As we can see in the stubs, TypeVarTuples themselves are not hashable, but Unpack operators are (Unpacking returns a _UnpackAlias, a subclass of the hashable _GenericAlias). Now that would mean some code
from typing import TypeVarTuple
TV1 = TypeVarTuple("TV1")
TV2 = TypeVarTuple("TV2")
Sub = {
int: TV1,
float: TV2
}
Mapping = {*v: k for k, v in Sub} # a: b, c: d -> b: a, d: c
would not work, and raise the given errors. Now reversing key and value for some mapping could be useful, for some type Mapping situations, should they ever be added.
What would be the equivalent code for this? And why is it obvious that this is the equivalent code?
2 Likes
Something like
d = {}
for k, v in items:
d[Unpack[k]] = v # Remember: Unpack aliases are hashable
would be the closest fit I suppose.
Ok, but that then fails to be useful outside of the typing context - making a complete mismatch with the rest of the PEP. If something like that would be added, it should be in a different, typing-focused, PEP.
4 Likes
I suppose that would be the case, however it would be a good opportunity to introduce this now, right?
Probably not. I am not convinced that this is a useful feature, and I think it’s a off-topic in this discussion. Ideally you create your own discussion thread for this. The arguments for and against are pretty different to these discussions and e.g. Type Mappings Comprhensions like what you are describing don’t even exists yet, independent of new syntax. Ideally the typing features exists first, and then later syntax is added when the feature proved itself useful and widespread.
10 Likes
I think that this extension is out of scope for this PEP (which is specifically about extending the syntax for comprehensions rather than introducing another shorthand in their place).
I’m also a little unsure of how to interpret the suggested code in terms of more basic operations. [*things] has a clear, established meaning, but [*(*iterables)] doesn’t (it’s a syntax error currently); if it were allowed, though, this kind of syntax feels to me like it’s trying to say: unpack iterables into a generator/tuple, and then re-unpack that generator/tuple into a list. That is, if this were valid syntax, it looks like there are two unpackings there, and both of them on iterables (with nothing in the syntax there that explicitly suggests unpacking the elements inside of iterables).
I unfortunately think that adding this syntax would muddy the waters a little bit in terms of the current PEP, so I don’t think it should be included here; if we do want to consider that, it should be in a later proposal (though I’ll say I’m personally not in favor of extending things in that way).
10 Likes
How to Teach This says:
# equivalent to out = {**expr for x in it}
out = {}
for x in it:
out.update(expr)
That doesn’t seem true. For example:
it = [[(1, 2)]]
out = {}
for x in it:
out.update(x)
print(out)
print({**x for x in it})
The loop version works (prints {1: 2}), the comprehension version doesn’t (raises TypeError).
3 Likes
It’s closer to this:
out = {}
for x in it:
out.update({**x})
Although I do feel like this could be allowed by a future PEP.
1 Like
Yep, you’re right. Good catch. When I wrote that example I was only considering the case where expr evaluates to something that can be unpacked with **, which is the only usage I was intending to propose (that is, I think the TypeError is the right behavior). I can try to tighten up the wording there. Thanks!
Yep, that’s right. This equivalence is mentioned a little further down in the “How to Teach This” section, but I’m going to try to rephrase that as well; I don’t think that my attempt to combine the set and dict comprehension examples in that section was particularly clear (or even correct), so I’ll try to rephrase it.
2 Likes
its = [[1, 2], [2, 3]]
# This is given in the PEP
new_set = set()
for it in its:
new_set.update(it)
assert new_set == {1, 2, 3}
# These other ways also work and aren't shown in the PEP
new_set = set()
new_set.update(*its)
assert new_set == {1, 2, 3}
new_set = set().union(*its)
assert new_set == {1, 2, 3}
# Makes me wonder about the possibility of also allowing *args in other collections, i.e.
new_list = []
new_list.extend(*its)
new_dict = {}
new_dict.update(*dicts)
# Or constructors for all of them
new_list = list(*its)
new_set = set(*its)
new_dict = dict(*dicts)
That would mean that e.g. new_list.extend(1, 2) would also work. That could be useful but it’s not what this PEP is about.
In general, this PEP thread seems to attract a lot of ideas for related things that could also be changed in the language. Let’s keep this discussion focused only on the concrete proposal in PEP 798. If you have some other idea, open a thread in the “Ideas” category.
4 Likes
Why do list(*its) and [*its] not qualify as things that should be added to Rejected Alternative Proposals?
1 Like
Don’t you mean new_list.extend([1], [2])?
There’s still value in having them for some complicated use cases. e.g. [*f(x) for x in xs].
Also note that [*its] just returns a copy of its (assuming it’s a list).
My short answer to this is that, while these are both about unpacking, neither is about unpacking in comprehensions, which is what this PEP is about. So I think they’re out of scope for this proposal. The other ideas in the “Rejected Alternative Proposals” section are about the semantics of the change to the syntax that’s proposed in this PEP, whereas your suggestions are about changing the behavior of the built-in types.
I’m not necessarily saying I’m against the idea of adjusting list.extend or dict.update to take an arbitrary number of arguments (I would actually be in favor of that), nor am I necessarily taking a hard stance against adjusting the constructors in the same way (though do I think I’d be against that change personally). But I think those are both different enough from what’s being proposed here that they don’t need a mention in this PEP (and certainly not as rejected proposals); I think they’re worth discussing more, but in separate proposal.
2 Likes
In How to Teach This it is said:
Alternatively, we don’t need to think of the two ideas as separate; instead, with the new syntax, we can think of out = [...x... for x in it] as equivalent to the following code [1], regardless of whether or not ...x... uses *:
out = []
for x in it:
out.extend([...x...])
Similarly, we can think of out = {...x... for x in it} as equivalent to the following code, regardless of whether or not ...x... uses * or ** or ::
out = set()
for x in it:
out.update({...x...})
These examples are equivalent in them...
Where it is presented ...x... which I personally find really confusing, as it is not used anywhere else in the PEP and it is not explained.
Looking at the citation of the message from Guido I guess it implies “any transformation of x” as in “f(x).attr” or similar.
But I only got to guess, and I am not 100% sure this is the actual meaning of ...x....
Also I believe it can be confused by JavaScript’s unpacking.
If I am right and it implies “any transformation of x”, there are really great examples in the “Code Examples” section, so there could be added a message redirecting anyone reading that section instead.
I must say, I lack any context of the previous discussions so I don’t know if this was mentioned previously.
1 Like