What are the subtyping rules for tuple[T, ...]?

guido · December 24, 2023, 12:21am

But struct.unpack() is only one of several examples – you’d need special checkers for SQL queries, Dataframes, and who knows what else. And remember that the majority of Python users don’t even use type annotations at all – we don’t want to shy them away with this kind of esoteric notation.

That said, we should just name this variant 2e so we can talk about it:

2e: In a generic function or class, if Ts is a variadic typevar (declared using e.g. def foo[*Ts]), tuple[*Ts] is a gradual type. As in 2a, 2c, and 2d, tuple[Any, ...] follows the same rules as tuple[T, ...] (i.e., a union, not a gradual type), and tuple is a synonym.

MegaIng · December 24, 2023, 3:59am

IMO tuple[*Ts] isn’t any more esoteric than tuple[...]. The former reads as “any number of Types”, but the latter I would have to look up if I didn’t see it in the context of thread first.

In contrast to the other options, 2e at least can be reasonably extended to fully type the call itself without having to rely on cast or annotations of the result, assuming the rules for unbound TypeVar-likes are changed a bit. (at least my understanding of what is currently allowed means that this is an edge case without clear documentation).

Requiring extra tools is ofcourse not ideal, but this option would allow those tools to operate in isolation without having to be a type checker plugin. Only people who really want those kinds of checks would benefit, but I don’t think most others would have any problems with this syntax (except having to learn it, which yes, isn’t irrelevant).

Semi-related side note: While trying to check the currently allowed behavior with regards to unbound TypeVarTuples, I saw that PEP 646 seems to have contradictory opinions on what tuple and tuple[Any, ...] should mean:

In the main body Behaviour when Type Parameters are not Specified says:

This implies that tuple[Any, ...] is a gradual type

However, latter on, in the rejected ideas section Unspecified Type Parameters: Tuple vs TypeVarTuple it states:

def takes_any_tuple(x: Tuple): ...
x: Tuple[int, str]
takes_any_tuple(x)  # Valid

def takes_specific_tuple(y: Tuple[int, str]): ...
y: Tuple
takes_specific_tuple(y)  # Error
The rules for Tuple were deliberately chosen such that the latter case is an error: it was thought to be more likely that the programmer has made a mistake than that the function expects a specific kind of Tuple but the specific kind of Tuple passed is unknown to the type checker

These two sections would imply that tuple != tuple[Any, ...], right? Doesn’t that contradict what the typing documentation has to say about this? Was this already a discussed topic when PEP 646 was written?

guido · December 24, 2023, 5:08am

Let’s continue this next year.

Daverball · December 24, 2023, 7:40am

I think the authors of the PEP just were confused about what the rules for tuple are, because it is not consistent across all type checkers, that’s the whole point of this topic. Maybe the type checker they’re most familiar with just happened to implement 2a.

You can convince yourself of this by pasting the example from the PEP into mypy-play: mypy Playground

This does not yield any errors, because mypy implements 2b, which is consistent with how PEP646 states variadic generics should behave. Which now also pushes me more towards 2b, unless we also implement 2c for variadic generics, i.e. also distinguish between Foo[...] and Foo[Any, ...]. They should all behave consistently, it does not make sense for tuple to be the odd one out, even if it is a builtin.

mikeshardmind · December 28, 2023, 9:18pm

Leaving this here in advance of the new year as I’ve had some things come up that will require my attention at the beginning of the new year…

I’m fine (in theory) with many of the options here. My litmus test for it is the behavior and typing of struct.unpack

The behavior I’d find most useful given what the type system currently supports in that context, is below.

def function_taking_an_int(x: int) -> None:
    ...

buffer: bytes
result: tuple[int] = struct.unpack("!Q", buffer)  # no error
x: int
x, = struct.unpack("!Q", buffer)  # no error
function_taking_an_int(struct.unpack("!Q", buffer))  # error
function_taking_an_int(*struct.unpack("!Q", buffer))  # no error
function_taking_an_int(*struct.unpack("!QQ", buffer))  # no type error (future extension possible to remedy this), yes runtime error
function_taking_an_int(*struct.unpack("!c", buffer))  # undetectable type error (future extension possible)

From that, the most useful versions for me here are treating tuple as somewhat symmetrical to callable in being a gradual type, or by allowing the use of typevartuples to explicitly state that and define that the type checker should use the annotation on the receiving end to resolve those in a case where you have a typevar tuple in a returned type, but not in the parameters.

It would be possible to define a function that took such string apis and returned a type that was only ever used in static analysis, but then you have code execution in static analysis which is rather undesirable. It would also be possible to create “new” typed apis that took a typevar tuple of types to parse into, but this would have multiple negative effects from performance to the fact that struct is intended to interact with sized types, so you break some symmetry. you also would lose the ability to use the struct module for length-prefixed data with formatted format strings. That is to say, this is one of those examples that shows that the type system can’t reasonably know everything without compromising on things people use right now in some way (all of the above are remediable at once except that of performance (and only due to the interpreted nature), which does matter in the case of struct)

mikeshardmind · December 30, 2023, 8:44pm

github.com/microsoft/pyright

Unknown type from struct.unpack

opened 09:24AM - 04 Mar 23 UTC

closed 02:39PM - 04 Mar 23 UTC

mikeshardmind

as designed

**Describe the bug** pyright does not understand struct.unpack/struct.unpack_fr…om It isn't feasible to handle this solely off of what's supported in the current typing ecosystem, but the [format strings in struct are rigidly defined ](https://docs.python.org/3/library/struct.html#format-characters)in such a way that literal format strings should allow for type inference. As such, I'm hoping that tooling can bridge the gap, at least in the case of literal format strings. **To Reproduce** Use `struct.unpack` / `struct.unpack_from` **Expected behavior** Type checking to understand the type returned by `struct.unpack` / `struct.unpack_from` **Screenshots or Code** ```python def _get_data_version(b: bytes, /) -> int: return struct.unpack("!b", b)[0] ``` In this particular case, `struct.unpack` may only return a single element tuple with that element being an integer that fits in a single byte or raise an exception. While pyright allows this, it doesn't correctly understand the type, only understanding it to be a tuple. The ideal scenario would also handle this case: ```py def _extract_length_prefixed_array(b: bytes, offset: int) -> tuple[int, ...]: (length, ) = struct.unpack_from("!b", b, offset) return struct.unpack_from("!b%dQ" % length, b, offset + 1) ``` As it is possible to tell from the format string that the only thing being formatted is a quantifier and that all elements in the resulting tuple must be an integer, but this might be a stretch for what can reasonably be supported. Note that while pyright allows what is above, it allows it in a way that removes any benefit to having this code be type-checked. If this were a more-involved function (this is only a minimal reproduction), this could have subtle type errors here. (shown by taking the solution used below and adding reveal_type use) ![image](https://user-images.githubusercontent.com/22035876/222890699-58af249b-9d9a-49f3-9021-187e89686026.png) **VS Code extension or command-line** Indirectly through pylance as a language server with strict type checking enabled, pylance version v2023.3.10 **Additional context** Currently, this is being solved in code as ```py def _get_data_version(b: bytes, /) -> int: (r,) = struct.unpack("!b", b) assert isinstance(r, int) return r ``` to have minimal impact while retaining the benefits of strict type checking, but the minimal impact relies on users running python with optimizations enabled, which is unfortunately (from various analytics collected previously) not a default option for most users. I'm currently mulling over options here as there are various places where struct handling should be as low-overhead as reasonably possible. The best case would involve direct support in pyright, with the second best case possibly ending up as a type ignore (unfortunately)

Actually, this was reported, by me, and you labeled it as “As designed” that pyright was treating tuple[Any, ...] that way, specifically with struct.unpack. The request itself was for pyright to understand struct.unpack better as a standard library module, but the underlying reason it was needed was that pyright would not just trust my annotations here.

mypy doesn’t have the issue pyright does here, and the cast is unnecessary if I did not support pyright users.

erictraut · January 16, 2024, 7:58am

Now that we’re back from the holidays, I’d like to see if we can come to a resolution on this topic.

To summarize, we were considering four options:

2a: tuple[Any, ...] follows the same rules as tuple[T, ...] . It implies a union of tuple[()] | tuple[Any] | tuple[Any, Any] | ... . tuple is a synonym for tuple[Any, ...] .

2b: Unlike the general case of tuple[T, ...] , tuple[Any, ...] is considered a gradual type. It is bidirectionally type compatible with any tuple regardless of length. tuple is a synonym for tuple[Any, ...] .

2c: We introduce a new form tuple[...] , a gradual type that is bidirectionally type compatible with any tuple of any length. The type tuple[Any, ...] is treated as described in option 2a. tuple is a synonym of tuple[...] .

2d: tuple[Any, ...] follows the same rules as tuple[T, ...]. The “bare” form of tuple (with no type arguments) is a gradual type that is bidirectionally type compatible with any tuple of any length.

2a is consistent with pyright’s current implementation.
2b is consistent with mypy’s current implementation.
2c is consistent with the use of ... in Callable. I’ll note that tuple[...] is evaluated without error by older versions of Python. This option provides partial backward compatibility for mypy users (those who are relying on the “bare” tuple to be a gradual type).
2c and 2d are very similar except that 2c provides an unambiguous way to spell the bidirectional form.

When I recently wrote the conformance tests for the typing spec, I ran across a use case described in PEP 646 that legitimately requires a tuple[Any, ...] that is bidirectionally compatible with all tuples of any length. That, along with some of the arguments above, shifts my view on option 2a, which doesn’t provide a solution for these use cases.

I remain strongly opposed to 2d for multiple reasons, so I’d prefer to take that off the table.

That leaves 2b and 2c, both of which I think are viable. I have a slight preference for 2c. The inconsistency of 2b bothers me, but I can understand the appeal of retaining full backward compatibility for mypy users. So I could get behind 2b if that’s the majority consensus.

Daverball · January 16, 2024, 8:05am

The only thing I’d add is that if we were to go for 2c, I think this should apply to all variadic generics, not just tuple, since PEP646 currently specifies that variadic generics should behave like 2b.

I think it would be bad to have inconsistent behavior between tuple and other variadic generics, so Foo/Foo[...] vs Foo[Any, ...] for Foo[*Ts] should behave the same way.

On second thought, maybe this singularity is fine, since the non-gradual version of arbitrary variadics Foo[Any, ...] is a lot less useful, since there’s currently no way to narrow it to a specific length without a cast. We can’t rely on len for narrowing in this case. But that would pull me more towards 2b than 2c, since the inconsistency with PEP646 still feels ugly.

beauxq · January 16, 2024, 4:22pm

I would like this to give me a type-checking error:

result: tuple[int, int, int, int] = struct.unpack("!5B", buffer)

I think that means 2a
(or 2c on the condition that unpack doesn’t use the new gradual type)
(or 2b on the condition that unpack is changed to something like tuple[object, ...])
whatever keeps my type-checking error for the above code.

I’m worried about making formal specifications that will forever force us into accepting false negative errors.

I find assert len(result) == 4 to be an acceptable way to tell the type checker the number of elements.

I’m interested in hearing more, because I haven’t seen anything in this thread that convinces me.

MegaIng · January 16, 2024, 4:40pm

With that you are in disagreement with most other people in this thread from what I can tell. AFAIK it should be possible to provide, for example, a mypy plugin that correctly produces the return type based on the literal struct string. (provide a general plugin system in the typing spec is also something I had been trying to imagine, but I really can’t think of a good system).

But in general, the point is that this should not give me a type error:

a, b, c, d = struct.unpack("!4B", buffer)

Both of these situations are less than ideal because python types system is nothing but a transparent layer on a very, very dynamic language.

guido · January 16, 2024, 4:41pm

I agree with Eric’s analysis.

I have a slight preference for 2b. I think we don’t need two similar-looking things that are nearly indistinguishable in behavior except in rare cases. Users will just be picking one at random or based on the wrong example.

beauxq · January 16, 2024, 5:03pm

Note that nothing in what I’m saying implies that this should give you a type error.

The point is that it should be up to the user how strict they want their type checking to be.
But making a formal specification that says “don’t report this type error” makes it more difficult for the users to get the type checking that they want.

rchen152 · January 17, 2024, 5:22am

My preference would be 2c > 2d > 2b > 2a.

2a leaves us with no way at all to express the gradual tuple form.
For 2b, tuple[Any, ...] meaning something completely different from tuple[<anything but Any>, ...] seems unnecessarily confusing for users. I also dislike the idea of having a special case in the spec for something as basic as tuple types, as we’d then have to take it into consideration for everything that builds on or interacts with them.
The only difference between 2c and 2d is that 2c also adds a tuple[...] form, and I weakly prefer having the option to spell out the gradual form explicitly.

The one thing I don’t like about 2c is that we would end up with three subtly different tuple-of-Any types: tuple[Any], tuple[Any, ...], and tuple[...]. But it still feels like the least bad outcome to me.

stroxler · January 17, 2024, 9:05pm

My preferences are the same, for the same reasons

erictraut · January 20, 2024, 7:25pm

Thanks everyone for the discussion and feedback.

It appears that there is no solution that is going to make everyone happy.

In an effort to break the deadlock, I’m going to propose that we move forward with option 2b. I think this is the best compromise at this time. It’s also the least disruptive to the typing community because it’s consistent with mypy’s current implementation.

Adoption of 2b doesn’t completely close the door on adopting 2c at some future time if we find that it’s necessary to do so.

erictraut · January 20, 2024, 7:30pm

I’ve written a draft update for the typing spec that incorporates rule 2b. Refer to this discussion thread for details.

Jelle · January 21, 2024, 10:34pm

Thanks. 2b is not a great option—it creates an unusual special case and the semantics are likely hard to understand for many—but it’s probably the least bad option. Let’s go with it for now.

Let me write down a few cases just to make the semantics clear.

from untyped_library import SomeType  # equivalent to Any
from typing import Any, Iterable

def f(untyped_iter: Iterable[Any], some_type_tuple: tuple[SomeType, ...], any_tuple: tuple[Any, ...], int_tuple: tuple[int, ...]):
    a: tuple[int, int] = tuple(untyped_iter)  # OK
    b: tuple[int, int] = some_type_tuple  # OK (but becomes an error if `untyped_library` becomes typed)
    c: tuple[int, int] = any_tuple  # OK
    d: tuple[int, int] = int_tuple  # type checker error