Customized iterable unpacking

Ideas
1

PEP-3132 (extended iterable unpacking) specifies the “starred” expression as (emphasis mine):

a subexpression that will be assigned a list of all items from the iterable being unpacked that are not assigned to any of the mandatory expressions, or an empty list if there are no such items.

I would like to propose a new __unpack__(before: int, after: int) → tuple magic method that allows an iterable class to customize how it is unpacked. It takes two arguments, the number of mandatory expressions before and after the starred expression and should return a tuple of before + 1 + after elements. With this, an extended iterable unpacking expressions such as

`a, b, *mid, c, d, e = iterable`

would become syntax sugar for
a, b, mid, c, d, e = iterable.__unpack__(2, 3)

Then for example a range.__unpack__ implementation would return mid as a range instead of a list. Likewise for Numpy arrays, Pandas series and other similar libraries.

1 Like
2

I’m not sure I understand what the value would be here. If I’m unpacking in Iterable that exists (i.e. list, tuple, etc) it’s repackaging existing objects. If it’s with one that doesn’t exists (something built on the fly like a range, or a generator) it still needs to iterate over that Iterable and create them.

Is there something I’m missing?

3

it still needs to iterate over that Iterable and create them.

Not necessarily. Here’s how range could be unpacked:

def unpack_range(self: range, before: int, after: int) -> tuple:
     result = list(self[:before])
     if after > 0:
         result.append(range(self[before], self[-after]))
         result += self[-after:]
     else:
         result.append(range(self[before], len(self)))
     return tuple(result)
4

I see a huge problem with anyone wanting to use this in combination with type hints (except for naming being similar (obviously) to typing).

How would one be able to hint the return type of __unpack__? I suppose tuple[T, .. ] would work, but being able to specify that we return a tuple[T*(before+after+1)] could be useful.

Also, how could this affect typing specific features relying on the syntax, mainly the unpacking of TypeVarTuples?

Still, after all these questions, it’s a likely +1 for me, being able to introduce such ‘context-aware’ special methods could help users to create more stuff they want implemented.

1 Like
5

But self is still a generator.

6

Yeah precise return type doesn’t seem possible, at least with today’s typing. E.g. for range best i can come up with is tuple[int | range, ...].

Ironically, the reason I came up with this idea in the first place was typing-related. I have a use case for a class that I’d like it to be iterable but not unpackable. If I could define __iter__ normally and set __unpack__ = None, a type checker would be able to flag unpacking without disallowing iteration.

EDIT: actually for this use case the opposite solution would be just as acceptable: if the type checker does not flag an unpacking of this iterable as error, I would need a way to ensure that it never never fails at runtime (without try/except). As of now there’s no way (afaict) to define an iterable for which both x, y = iterable and x, y, z = iterable succeed.

Both acceptable solutions (statically disallow unpacking or guarantee it never fails at runtime) would be feasible if there was an __unpack__ method.

7

Yeah, being able to do x, *y, z = non_iterable_but_unpackable would be useful imo, so yeah, cool idea.

8

It’s not a coincidence it’s one of the few (if not only) operations on a MagicMock that can fail:

In [2]: m  = MagicMock()

In [3]: x, *y, z = m
...
ValueError: not enough values to unpack (expected at least 2, got 0)
1 Like
9

It isn’t immediately clear to me what y is expected to be with this code. Should it be [], [MagicMock()] or even MagicMock()?

10

Good question, not sure tbh. Probably MagicMock() because that’s the result of m[1:-1] but empty list makes sense too.

The point is that it doesn’t work even without the starred expression that is less ambiguous; x, y = m results in the same error.