PEP 728: TypedDict with Typed Extra Items

PEPs
143

What is the status of this PEP?

144

I just released a release candidate of typing-extensions supporting the latest version of the PEP. I think that makes it about ready for submission to the Typing Council.

5 Likes
145

Should the PEP be updated to mention typing-extensions as a reference implementation?

146

Yes probably.

@PIG208 do you think you’re ready to submit the PEP to the Steering Council for approval? It might still make it in time for Python 3.14.

2 Likes
147

Probably with some minor edits. Yes. Will open a PR in the next few days.

6 Likes
148

I think PEP could benefit if it would explicitly specify what default values for new parameters are.
As I understand it’s closed=False and extra_items=typing.NoExtraItems.

After reading the PEP I had a question why not just make it work using just 1 parameter - extra_items. But after reading the discussion, clarity of closed=True, that would be used in the majority of cases, does makes sense. But I don’t really like that from the start there are two options to do the same thing - extra_items=Never and closed=True. And PEP doesn’t prescribe which way is preferable, so we’ll end up seeing both in the code.
I agree with what was mentioned above that maybe Never is too clever here and maybe it shouldn’t be supported and not have a meaning of closed=True, so the only way to make a dict closed is to actually do closed=True.

Also, since PEP is referring in it’s Motivation to the problem that currently for .items() /.values()/.keys() type checkers couldn’t infer more precise return types, so shouldn’t it include a section for recommended/expected behaviour from type checkers in that regard if PEP728 will be accepted? Or to add a note that it’s out of scope for this PEP and the exact behaviour should be decided later.

# pyright: enableExperimentalFeatures=true
from typing import TypedDict, assert_type, Literal

class Movie(TypedDict, closed=True):
    name: str
    year: int

def test(m: Movie):
    assert_type(list(m.items()), list[tuple[Literal["name", "year"], str | int]])
    assert_type(list(m.values()), list[str | int])
    assert_type(list(m.keys()), list[Literal["name", "year"]])

# Example from Motivation section.
class Movie(TypedDict, closed=True):
    name: str
    director: str

class Book(TypedDict, closed=True):
    name: str
    author: str

def fun(entry: Movie | Book) -> None:
    if "author" in entry:
        assert_type(entry, Book)
149

False is the default for closed, and extra_items is by default unset. Will make these a bit more explicit.

There is an example that mentions closed=True being preferred. But it would be better to mention it in prose. Will add that in the next revision.

class MovieNever(Movie, extra_items=Never):  # OK, but 'closed=True' is preferred
    pass

The two sections on Mapping[str, VT] and dict[str, VT] might answer your question about .items et al.

1 Like
150

Jelle recently merged some updates into the PEP 728 draft spec. Most of these updates make sense to me, but there is one clarification that surprised me. I’d like to get feedback on it. The latest draft says:

Extra keys included in TypedDict object construction should still be caught…

The “still” is referring to the fact that the typing spec indicates that when constructing a TypedDict object — which is typically done through a dict literal expression, the presence of extra items is flagged as an error.

class NoExtra(TypedDict):
    a: str

v1: NoExtra = {"a": "", "b": 42}  # Error

However, I’m not sure it makes sense to extend this rule to TypedDicts with extra_items specified.

class Extra(TypedDict, extra_items=int):
    a: str

v2: Extra = {"a": "", "b": 42}  # Error?

If someone has gone through the trouble of defining a TypedDict with extra_items, it seems to me that we should allow extra items to be supplied in a dict literal expression as long as the values are of a compatible type. What do others think?

On a related topic, the PEP is not clear about the behavior when a TypedDict with extra_items is used to annotate a **kwargs parameter. Should additional keyword arguments be allowed in this case? This came up in a recent pyright bug report.

def no_extra(**kwargs: Unpack[NoExtra]): ...
def extra(**kwargs: Unpack[Extra]): ...

no_extra(a="", b=42) # Error (according to typing spec)
extra(a="", b=42) # Error?

My sense is that the second call (to extra) should not result in a type error. Do others agree?

In any case, I think the answer to these two questions (about dict literals and keyword arguments) should be consistent. That is, we should either allow both or disallow both.

12 Likes
151

I agree with you on both counts.

152

The clarification was meant to apply only for the case “[when] neither extra_items nor closed=True is specified, closed=False is assumed.” which basically reiterates the existing behavior of TypedDict.

I agree with both points. Sent a quick update to fix them.

153

Thank you for the PEP! I’ve read it over and think it is generally very good. It specifies the feature clearly and motivates it well.

I do have a few comments. I’m happy to submit a PR to address all of these if they are uncontroversial.

Disallowing extra items specifically

This can also be resolved by defining a closed TypedDict type.

I’m confused by the “also” in this sentence. It seems to imply either that there is some previously-discussed alternative way to disallow extra items, or that there is some previous problem which is also resolved by the new closed parameter – but I don’t see either of those in the text.

Rationale

A type that allows extra items of type str on a TypedDict can be loosely described as the intersection between the TypedDict and Mapping[str, str].

I think it is more confusing than helpful to describe extra_items in this way, even “loosely”. If we have TypedDict("Foo", {"year": int}), the intersection of that type with Mapping[str, str] would require the key year to have a value which is both int and str, but this is not true of a TypedDict with extra_items.

Readers of the PEP (and the spec which will be derived from it) will either have thought carefully about type intersections, or they will not. For the former readers, this analogy suggests something that is incorrect. For the latter readers, referencing intersections is more likely to be intimidating/confusing than to aid understanding.

I think we should instead describe the semantics of extra_items directly, without any reference to intersections.

I would also remove the subsequent mention of “without introducing general intersection of types” – even if we had general type intersections, I don’t think they would serve as a usable replacement for extra_items.

(This point is correctly noted later in rejected items, but I think it should also lead us to avoid discussion of intersections entirely in the rationale.)

Supported and unsupported operations

Operations with arbitrary str keys (instead of string literals or other expressions with known string values) should generally be rejected.

This restriction does allow __getitem__ on a TypedDict with extra_items to be sound, but it’s more restrictive than necessary. If we have a TypedDict({'a': int}, extra_items=str), then __getitem__ with a str key does not have to be rejected, it could have type str | int.

I’m also curious what @erictraut thinks of including this restriction in the PEP, since my understanding is that pyright already allows this for regular TypedDict, for pragmatic reasons?

As I said earlier in this thread, I think the PEP currently still leaves too much to the imagination when it comes to the core, most common, behavior of a TypedDict – getting items from it and setting items in it. I think the intended signatures of __getitem__ and __setitem__ on a TypedDict with extra_items should be outlined clearly in the PEP. If we want to intentionally leave some flexibility for type checkers to define their own rules here (so some type checkers can opt for more soundness, and others can opt for more ease of use) I think that’s fine, but it should still be stated explicitly that there is implementation flexibility, and what the boundaries of that flexibility are. This reduces uncertainty for type checker authors, and for a future author of a spec conformance suite for this feature.

Assignability

Because 'year' is absent in Movie, extra_items is considered the corresponding key. 'year' being required violates this rule:

  • For each required key in A, the corresponding key is required in B.

I think something is not right in this example? The rule listed here has the context that we are checking whether B is assignable to A. But the code example above shows an attempt to assign MovieWithYear to Movie. So Movie is A – and year is not a required key in Movie. So the rule listed does not seem to apply to the assignment in the example.

Interaction with Mapping[str, VT]

I think it would be clarifying to show the .items() and .values() examples for MovieExtraInt instead of (or in addition to) MovieExtraStr. For the latter all types are str, so it is less clear that the known value types must be unioned with the extra-items type.

Interaction with dict[str, VT]

However, dict[str, VT] is not necessarily assignable to a TypedDict type

The word “necessarily” here suggests that in some cases it might be assignable, but I don’t think this is true? Can we just remove “necessarily” and say “is not assignable”?

Also, I think the code example here is not useful, because it shows a case where the type-checker knows the custom dict to be of type CustomDict. A more relevant example would be like this:

class CustomDict(dict[str, int]):
    pass

def f(might_not_be_a_builtin_dict: dict[str, int]):
    int_dict: IntDict = might_not_be_a_builtin_dict # Not OK

not_a_builtin_dict: CustomDict = {"num": 1}
f(not_a_builtin_dict)

5 Likes
154

Thanks for the detailed review! The proposed changes look good to me.

Below are some things I want to add:

I agree that we can expand more on what’s expected. I think the general idea is that operations allowed on the Mapping and dict that the TypedDict is assignable to should be sound and thus should be allowed.

Given the Mapping[str, VT] a TypedDict is assignable to, I think the signature should look like __getitem__(self ,key: str) -> VT.

Not all TypeDicts have VT' such that the TypedDict is assignable to a dict[str, VT'], but when one does, I think __setitem__(self, key: str, value: VT') should be allowed.

I agree that doesn’t seem right. I think this other rule applies here:

  • For each non-required key in A, if the item is not read-only in A, the corresponding key is not required in B.

Given that year is not-required and non-read-only in A (Movie), it shouldn’t be required in B (MovieWithYear) either.

2 Likes
155

Hi @PIG208 , may I ask about the status of this PEP? Is it being revised still or is the typing council reviewing it for a decision?
(And thanks for your work on it!)

156

The Typing Council is still making a decision.

1 Like
157

It looks like there’s an error in one of the examples under “The extra_items Class Parameter”:

class MovieBase(TypedDict, extra_items=int | None):
    name: str

class Movie(MovieBase):
    year: int

a: Movie = {"name": "Blade Runner", "year": None}  # Not OK. 'None' is incompatible with 'int'
b: Movie = {
    "name": "Blade Runner",
    "year": 1982,
    "other_extra_key": None,
}  # OK

I think MoviedBase should be defined like this, using ReadOnly, based on the discussion in section “Inheritance”, for Movie to be valid:

class MovieBase(TypedDict, extra_items=ReadOnly[int | None]):
    name: str
4 Likes
158

The Typing Council recommends acceptance of this PEP. Thanks for everyone’s work on getting this proposal ready!

18 Likes
159

Hi Jelle,

After several weeks of review, the Steering Council (SC) is pleased to approve PEP 728, in line with the recommendations from the Typing Council.
Thank you for your thorough responses and updates throughout the review process.

Donghee
on behalf of the Steering Council

Note: @barry plans to share additional ideas for typing documentation separately.

17 Likes
160

Thank you! Now we have to get this implemented. Here’s a TODO list of sorts; I’d welcome contributions from anyone interested.

7 Likes