Introduce Partial for TypedDict

[mikeshardmind (Michael H)]

The diffierence is important for how this will interact with other parts of the type system

class InnerStructure(TypedDict):
    always_here: ...
    ...

class SomeLibraryTypedDict(TypedDict):
    foo: InnerStructure
    ...

def library_func(**kwargs: Unpack[SomeLibraryTypedDict]):
    ...

# user code wrapping library function with application-specific defaults
def user_function(**kwargs: Unpack[SomeLibraryTypedDict.partial]):
    our_kwargs: SomeLibraryTypedDict = ...
    our_kwargs.update(**kwargs)
    library_func(**our_kwargs)

This allows for significantly more concise function signatures that are still type checked properly and done with idiomatic use of **kwargs without relying on duplicating library types.

I don’t think it’s appropriate to be recursively applied when looking at how this interacts with other type system features like unpack, paramspec kwargs where kwargs in a paramspec come from unpack, and so on.

in terms of design, If you own the types, redeclaring actually is an option, if you don’t, then composing this recursively makes it unusable with unpack.

As a bonus, we don’t encourage long .get() chains that bring us back to being not much better than dict[str, dict]

[USSX-Hares (Peter Zaitcev)]

@Daverball

@samuelcolvin

As I said before, I think the dataclasses case is very strong, I’m not saying it should be included in the PEP now, just that it’s a clear future extension.

I strongly disagree. dataclasses are structural types, so without a single constructor they are useless, because otherwise their type identity does not match.

For me, Partial[dataclass] should behave more like Protocol than dataclass - i.e., skip constructor at all. I even doubt if the should have an ability to be constructed at all (i.e., be purely virtual non-runtime type).

[Daverball (David Salvisberg)]

That’s one way to go about it, but then you are back at not being able to actually construct the type that satisfies that Protocol without either writing a dynamic constructor or manually writing a dataclass that matches the Protocol, which seems a lot less ergonomic than just having access to the new type.

And you also still have the discrepancy between NotRequired and Optional, which are completely different things.

[mikeshardmind (Michael H)]

@samuelcolvin

As I said before, I think the dataclasses case is very strong, I’m not saying it should be included in the PEP now, just that it’s a clear future extension.

The dataclasses case is non-existant. dataclasses participate in nominal subtyping, and such a construction has no possible well-defined behavior. The closest thing that would would be better served with protocols, but that’s putting the cart before the horse and saying that we want APIs that require people to do defensive getattr / hasattr instead of providing a better type.

[samuelcolvin]

For what it’s worth, I did a twitter poll on what syntax people would prefer:

[davidfstr (David Foster)]

The Syntax

FWIW, I like the syntax Partial[Movie] better than Movie.partial, where Movie is a TypedDict:

• It reads to me like a type constructor.
• The use of brackets ([]) signal that it is some kind of type-level operation.
• The syntax aligns with the use of Partial in TypeScript, so we’re staying consistent with the broader community of languages when it’s not too difficult to do so.

Seems that a user poll also agrees.

@Daverball

@samuelcolvin

But the real question is: “What’s the advantage of TypedDict.partial?” I really don’t see how it makes more sense than Partial[TypedDict].

I’ve said it before, but maybe in less obvious words: The advantage is you will get back a type constructor, that you can inspect and use as if you had written the partial version to begin with. I.e. you are putting no additional burden on runtime type introspection library authors to support Partial, it will just work.

It seems to me that a Partial object at runtime could be given a special method that returns the constructed TypedDict type, which a runtime type introspection library - such as the trycast library I maintain - could use without too much difficulty.

Applicability beyond TypedDict

I could see Partial[] be extended to Protocols in the future, since they - like TypedDicts - are structural types rather than nominal types.

I’m not as sure that Partial[] would extend naturally to cover dataclasses. I’d have to think about it more.

Regardless, I think Partial[] would be useful for TypedDict right now, and support for other inner types could be added later.

[Daverball (David Salvisberg)]

@davidfstr

• It reads to me like a type constructor.
• The use of brackets ([]) signal that it is some kind of type-level operation.

Does it actually though? This would be the only type constructor in Python that uses brackets. NewType uses a call syntax, TypedDict and NamedTuple are the same if you don’t use them in their class-form, same with type params.

I think people really need to consider this a bit more carefully, since Partial would be the first beast of its own kind, namely a type modifier. This is different from type qualifiers like Final, ClassVar, Required, ReadOnly, etc. which don’t change the type they wrap.

So the question remains: Is Partial as a type modifier useful enough on its own merits and the currently known potential use-cases, compared to a type constructor which would give you back a runtime type, that you can introspect with no needed additional knowledge about Partial^[1]. I don’t think the poll is actually asking the right question.

Currently I’m leaning towards “no”, hence the proposed solution of being conservative and adding a TypedDict-specific type constructor for now. If we end up encountering more compelling use-cases in types other than TypedDict we can still add Partial at that point. They’re not necessarily mutually exclusive. I like how Partial[Foo] looks like better as well, but aesthetics are secondary.

It’s also worth noting that I actually would prefer something like PartialFoo = PartialTypedDict("PartialFoo", Foo) over Foo.partial, in order to really drive home that you’re dealing with a type constructor, even if it means you need to give the partial dict its own name. It’s still a lot nicer than having to maintain two almost identical copies of the same typed dict.

---

1. This means all the runtime libraries that understand typed dictionaries today, will also understand partial typed dictionaries tomorrow. ↩︎

[Tinche (Tin Tvrtković)]

@Daverball

This would be the only type constructor in Python that uses brackets.

list[int]

[Daverball (David Salvisberg)]

“The only special form type constructor in Python that uses brackets.”

There, happy? We can devolve this discussion into straw man arguments about the semantics of what is or isn’t a type constructor in the truest sense of the word, but that doesn’t really seem particularly helpful and I think you and I both know that there’s an important distinction between how <SpecialForm>[type] and <type>[T] are read, since the operand and the operator are reversed.

[Jelle (Jelle Zijlstra)]

This is an interesting idea, but I wonder if it’s worth generalizing a bit more. With PEP 705, we’ll also have the ReadOnly qualifier on TypedDicts, and it’s natural to want a read-only version of an existing TypedDict. What if we could instead write:

• Map[NotRequired, SomeTypedDict]
• Map[ReadOnly, SomeTypedDict]

(And also Map[Required, SomeTypedDict], though I don’t know when you’d want that.)

Map has of course been proposed in other contexts too, mostly around TypeVarTuple. As long as the different uses are conceptually similar, I don’t think that should block us from using it in multiple places in the type system.

There’s also desire in this thread for supporting something like Partial on dataclasses, which might mean something like “all fields may be None”. That idea creates bigger issues, because dataclasses are nominal and not structural types. Still, if we find a coherent way to add this concept to the type system, it could also be expressed using Map[], though I’m struggling to find a good spelling:

• Map[Optional, SomeDataclass] (I’d like to get rid of Optional, and doesn’t generalize to unions with other values)
• Map[None.__or__, SomeDataclass] (yuck)
• Map[lambda T: T | None, SomeDataclass] (yuck)
• Map[InputType | None, SomeDataclass], where InputType is a new primitive in typing (generalizes well and doesn’t look ugly, but introduces new complexity)

Still, the conceptual problems with structural transformations on a nominal type mean that this shouldn’t be in the initial PEP proposing the concept for TypedDicts.

[mikeshardmind (Michael H)]

I don’t particularly think it generalizes well, and that we may be better off looking at tools like Pick and Omit (typescript) along with the work on intersections to allow mixing and matching of structural types. It could be reasonable that Pick specifically could pick only the signature of one or more parts of either a nominal or structural type to create a structural type with that “shape” requirement.

[samuelcolvin]

Thanks so much @Jelle for your feedback.

I for one would be happy with Map[NotRequired, SomeTypedDict], especially if you could you create an alias:

T = NewType('T', type[DataClass])
Partial = Map[NotRequired, T]

?

“Map” seems like a slightly confusing name given it’s widespread use, but I can’t think of anything better.

[lawsonjl (Joe)]

@Jelle

• Map[Optional, SomeDataclass] (I’d like to get rid of Optional, and doesn’t generalize to unions with other values)
• Map[None.__or__, SomeDataclass] (yuck)
• Map[lambda T: T | None, SomeDataclass] (yuck)
• Map[InputType | None, SomeDataclass], where InputType is a new primitive in typing (generalizes well and doesn’t look ugly, but introduces new complexity)

IMO, this is just begging to be a generator expression:

Map[T | None for T in SomeDataclass]

[Zomatree]

Using a generator makes it a lot harder and sometimes impossible for this to work at runtime, its not worth it even though it does make reading it easier.

[Daverball (David Salvisberg)]

That’s where I sometimes wish Python hadn’t overloaded subscript for generics and other type forms. If we went with a new syntax instead, then we wouldn’t have as many of the problems as we’re facing today and could come up with a grammar specific to type expressions that’s as expressive as possible without having to weigh it against ergonomic runtime introspection, because we can create the most convenient object structure from the AST. PEP 695 has demonstrated that syntax extensions^[1] can have a major impact on ergonomics^[2].

---

1. where they’re still possible, given the limitations of the square bracket notation ↩︎

2. and to some extent expressiveness ↩︎

[Viicos (Victorien)]

I went through all this thread to make sure this wasn’t proposed before (maybe it was in another thread):

Edit: not possible (see below). Hiding post but keeping for archival purposes.

(Not)Required is used to mark a non-required key as required (or a required key as non-required). Similarly, and to avoid having to introduce a Partial special form, could (Not)Required be expended to typed dicts as well?

class Structure(TypedDict):
    x: int
    y: int

class OptionalStructure(TypedDict, total=False):
    x: int
    y: int

NotRequired[Structure] <=> OptionalStructure
Required[OptionalStructure]  <=> Structure

Benefits:

• Avoids having to define a new Partial special form
• Allows for a typed dict with total=False to become total=True

Drawbacks:

• Less obvious than Partial. Seeing NotRequired[Structure] in a type annotation is (really) confusing, a bit like it did with Optional where people thought it meant <type> | None = None
• Might not be extendable to other types (e.g. as proposed in this thread, for Protocol – NotRequired[SomeProto] doesn’t really make much sense).
• Implementation limitations?

Drawback 1 and 2 kind of kills this alternative, but benefit 1 would be really nice to have (unless there’s already a way to do it that I’m missing?).

[Jelle (Jelle Zijlstra)]

This doesn’t work because a TypedDict can be nested in another one.

class TD1(TypedDict):
    a: int

class TD2(TypedDict):
    b: NotRequired[TD1]  # does this mean that key "b" is not required, or that TD1 is partial?

[flying-sheep (Philipp A.)]

Would be great to have this. Is there any path forward here or did people just lose interest over bikeshedding it?

[Jelle (Jelle Zijlstra)]

The way forward is for someone to write a PEP with a concrete, well-defined proposal.

[flying-sheep (Philipp A.)]

I like the generator expression. Map would work without something like it, but that would require an intermediate generic type:

class MyTd(TypedDict, total=True):
    x: int

type HasFoo[T] = Annotated[T, Foo]

type MyFooTd = Map[HasFoo, MyTd]

Is is possible to have multiple arities for a generic type? Then we could have the two-parameter Map now and maybe a generator-syntax one-parameter invocation of Map later