Tuple comprehensions revisited, in relation to PEP 841 and async iterables

Ideas
1

PEP 841 proposes adding new display syntax for frozenset and frozenlist:

  • f{1, 2, 3} for a frozenset
  • f{"a": 1} for a frozendict

It also proposes comprehension syntax for both of these immutable collections, supporting both synchronous and asynchronous iterables:

  • f{item for item in items}/f{item async for item in ait}
  • f{k: v for item in items}/f{k: v async for item in ait}

If PEP 841 were to be accepted, there would be a pretty striking asymmetry in comprehension syntax for Python’s primitive container types:

dict: {k: v for k, v in items}
frozendict: f{k, v for k, v in items}
set: {item for item in items}
frozenset: f{item for item in items}
list: [item for item in items]
tuple: No such construct exists .

tuple(item for item in items) and tuple([item for item in items]) are the best alternatives. The generator expression-based approach to constructing a tuple has the benefit that, for large lazy iterators, a separate list does not need to be materialized. However, this approach fails when the iterable is an async iterable:

tuple(item async for item in ait) # Does not work, because tuple() does not accept AsyncGenerator

The original rationale for the exclusion of tuple comprehensions appears to be due to its immutability (reference). Serhiy Storchaka explicitly cites the same reasoning as the reason why frozenset comprehensions were excluded as well. If PEP 841 were to be accepted, this rationale would no longer hold, as frozenset comprehensions would be introduced to Python.

For these reasons, I think that the idea of tuple comprehensions deserves to be revisited specifically in relation to PEP 841. Besides the motivation for asynchronous iterables listed above, PEP 841 lists one of the motivations for the PEP is to “increase the use of immutable containers in CPython”, and a first-class representation of tuple comprehensions would help further that motivation.

In terms of syntax, the obvious representation is currently used by generator expressions, so I would propose the following:

  • (item for item in items,)

The trailing comma presents no backwards compatibility concerns, as this is not valid syntax in any Python version, and it mirrors the use of a trailing comma to distinguish a parenthesized expression ("foo") from a singleton tuple ("foo",).

An alternative is perhaps f[item for item in items], treating tuple as the frozen counterpart to list, but this notation may be confusing since tuples are normally denoted using parentheses, not brackets.

2 Likes
2

One key difference between a list and a tuple is that a list’s elements, while they have a sequence, are generally the same kind of thing; a tuple can often consist of distinctly different things with very different meanings. A tuple isn’t just a frozen list. Aside from the perceived incompleteness, is there any actual need for a tuple comprehension?

3

PEP 841 lists a motivation of for the new features relating to frozendict/frozensets to increase the use of immutable containers specifically preparing “for a concurrent future in which free-threading and subinterpreters become significantly more common”. Despite being commonly used to represent heterogeneous data, tuples are the best data structure in Python for an immutable, ordered, homogeneous sequence, too, and I would imagine any such concurrent application that needs such an immutable, ordered, homogeneous sequence would naturally turn to tuples.

Examples in the standard library of tuples being used for homogeneous data includes itertools: batched, permutations, combinations all yield tuples of elements from (typically homogeneous) input iterables.

3 Likes
4

often there is a need for a frozen list, for the sake of type checking or hashability, and in those situations I do indeed use tuples. I have never had a problem with the ‘workaround’ children = tuple(i for i in items). If there was a particular need to use an async iterable, I don’t think I’d have a problem with tuple([item async for item in ait]).

In terms of esthetics, I would have a very strong preference for grandchildren = f[*c.children for c in children] over grandchildren = (*c.children for c in children,). It’s just too hard to miss that ,, (...) isn’t as sharp as [...], and ( doesn’t announce what kind of object you’re dealing with, unlike f[.

Introducing a new type is perhaps too much change compared with ‘only’ introducing a new comprehension. Tuple does has one other downside compared with hypothetical frozen list: (1,2) + [3,4] raises, whereas frozenlist would be expected to satisfy f[1,2] + [3,4] == f[1,2,3,4].

5

For something THAT simple, you should be able to use children = tuple(items), but I presume there’s more to it than that.

2 Likes
6

There is however a clear need to distinguish the C-struct like use of tuples where a particular index has a distinct meaning, (e.g. tuples for coordinates). Maybe we should make a frozenlist distinct from a tuple in Python with different syntax and pythonic use cases. Method parameters accepting tuples but not lists might think of not accepting frozenlists either.

7

At first my intuition parses that as if it were

  • (item for item in (items,))

much like how return item, is equivalent to return (item,).

Though it may be growing on me so I suppose I’m -0 on this syntax for now.

I can’t support f[item for item in items] because f[ is valid syntax (in contrast to f{...} for a frozendict, which per PEP 841 is fine because f{ was not previously valid syntax ever).

Perhaps it would be better for any instance of tuple([...]) to be specifically checked for in source code and optimised, such that it does not create an intermediate list? This existing syntax would then be the optimal way to turn an async iterable (or anything else) into a tuple. I’m not sure at which step this optimisation would take place, though, and it’d have to somehow deal with the case where someone assigns the name tuple to have some other value.

8

This is reasonable, but do tuple type hints solve it? If I’m expecting a struct, I’ll pass an explicit argument list. If I’m expecting a frozenlist, I’ll use ellipses. Type checkers won’t allow “frozenlist → struct” conversions.

def normalize(vector: tuple[float, float]) -> tuple[float, float]:
    magnitude = vector[0]**2 + vector[1]**2
    return (vector[0] / magnitude, vector[1] / magnitude)

frozenlist: tuple[float, ...] = (1.0, 1.0)
normalize(frozenlist) # Type checker error
1 Like
9

I did overlook the possibility of f[item for item in items] being used as f.__getitem__(item for item in items), so it does present a backwards compatibility concern. However, I’m not personally aware of any instance of a generator expression being used as a valid index, and given that PEP 841 specifies that f{ is a single token with no intervening whitespace, it can be expected that this would function identically and the f-indexing operation could be spelled as f [item for item in items]. I will still admit that, however unlikely indexing into f (a variable name typically used for functions or files, which do not support indexing with a generator expression) with a generator expression may be in existing code, this is a backwards compatibility issue.

10

I don’t think it’s technically valid (you currently would need to write f[(item for item in items)] instead), though the confusion will be real. I think it’s best avoided just because it’ll LOOK like subscripting, even if technically the two notations can be differentiated. Confusingly-similar syntax trips people up and can lead to extremely confusing error messages (if you intended one syntax but it ended up looking more like the other).

11 
f = ...
f[item for item in ()]  # invalid syntax
f[(item for item in ())]  # generator expression index

You are correct, I tried this on Python 3.15 so f[... for ... in ...] is okay from a backwards compatibility standpoint.

12

Limiting f[...] to taking a generator expression would feel at odds with the rest of the collections with shorthand creation syntax that accept any iterable, including generators. I can create a list from an iterable by writing [iterable] but f[iterable] would not be allowed because it is ambiguous (frozenlist() or f.__getitem__(iterable)). So, while f[item for item in ()] is not valid syntax, using it as proposed would create a feature that does not align with the features it is being modeled after. It would likely trip up even experienced users who expect to pass any iterable to the collection shorthand syntax. Consider what happens when the user wants to add a condition to their generator expressions and it becomes unwieldy so they factor it out into a local variable and can’t use f[filtered_items]

13

Given that this proposal doesn’t introduce “frozenlist” as a new type, I don’t see this as an issue. If a (synchronous) iterable with the items already exists, it can be passed directly into tuple().

That said, I understand that there may be confusion from this proposal that a new “frozenlist” was in fact introduced. If that is a concern, I would suggest tuple[... for ... in ...] and t[... for ... in ...] as alternatives (in addition to (... for ... in ...,))

14

The concern is that f[...] would not function the same way as all other collection creation shorthand syntax ([], {}, etc) because what is acceptable inside the braces is restricted to avoid syntax ambiguities. tuple[... for...] has a similar syntax overloading issue with t: tuple[...] use for type annotations and t[...] just replaced ‘f’ with ‘t’ and doesn’t address the ambiguity as to whether t is being subscripted or ‘t[’ is creating a tuple.

15

Your point about tuple possibly being shadowed brings up an important advantage of new syntax over optimizing a builtin call. PEP 841 notes that one of the motivations of display syntax for frozenset and frozendict is that it provides the compiler a semantic guarantee: any instance of f{1, 2, 3} will produce a frozenset, whereas the compiler cannot be sure with an equivalent like frozenset({1, 2, 3}) that frozenset has not been shadowed.

This is the same issue that prevents an empty set() from being optimized. Because it is possible for set to be rebound, set() cannot be optimized as BUILD_SET 0. The bytecode generated on CPython 3.15 for set() is still:

LOAD_GLOBAL              1 (set + NULL)
CALL                     0

This is part of the motivation behind PEP 802 (which proposes {/} as display syntax for an empty set.)

I would apply the same logic to the idea of simply optimizing tuple([... (async) for ... in ...]). It is difficult for the compiler to optimize code that involves dynamic name lookup because it has no guarantee that the name has not been shadowed at runtime; at best, it can only provide a fast-path when tuple still resolves to the builtin tuple. Syntax, on the other hand, is capable of providing the compiler a much stronger semantic guarantee.