Introduce Partial for TypedDict

I’d be an “absolutely not” on this. As others have stated, dataclasses participate in nominal subtyping. Additionally, presuming None to be a reasonable default may make sense to someone only considering json web apis, but None is a terrible default in a lot of other domains. Making it a default, especially in cases where without Partial None wasn’t valid for that field would break LSP and subtyping more than they already are currently. (and people can just set proper defaults in dataclasses themselves, allowing unpacking to fill only some things…)

Partial as proposed likely wouldn’t compose properly with other things being worked on for structural subtyping, such as intersections (whereas looking at typescript’s Pick and Omit, each of those would compose properly, but don’t do what you want.) To me that’s a sign that this shouldn’t be a special typing form and should be possible via typed dict itself. And it is, via total=False, but there are ergonomic issues there with keeping multiple definitions in sync or definitions you might not control.

It was previously mentioned that total=False in a subclass only applies to keys directly in that declaration (not applying to parent class’s keys), can that be changed? This gives an easy way to keep this contained to the construct where it makes sense and doesn’t present challenges with how this would compose with other things.

if not due to breaking concerns, can we add something to TypedDict so that SomeTypedDict.partial is a valid annotation with the desired meaning?

Having TypedDict behave more like Protocol would make them 100% more useful.

You should have to opt-in to more strict “no extra keys” checking, which is all that exists at the moment.

In the current situation most of the useful cases for TypedDict can’t even be expressed, most of the time seeing one is almost like a bad code smell i.e. “why are you even using a dict then”.

You may be interested in PEP 728 which solves exactly that: PEP 728: TypedDict with Typed Extra Items

@samuelcolvin, could you create a PEP for that this? Or should we discuss more first?

That could be a nice to have also, though mostly I am looking for unknown/untyped extra keys.

I like Eric Traut’s suggestion there that you would opt-in to current “closed” TypedDict behaviour by:

class Point1(TypedDict, closed=True):
    x: float
    y: float

(which I very much think is how it should have been in the first place)

3 Likes

You can achieve that with PEP 728 as well, if your extra keys are ReadOnly[object][1] they are arbitrary, alternatively you could mark them as Any, but that would be less type safe.


Also note that TypedDict is currently not actually fully closed. While there are errors when you include extra keys, they are still only considered as compatible with Mapping[str, object] since they may contain extra keys with arbitrary types at runtime. So they’re in a weird state where they are closed as type recipients, but open as type providers[2]. Incidentally inheritance wouldn’t work if they weren’t open as type providers.


  1. Provided PEP 705 gets accepted, which is likely considering it has been recommended for acceptance by the typing council ↩︎

  2. with the exception of Unpack, where they’re also considered closed ↩︎

I would recommend seeing if you can find a potential sponsor for a PEP in this direction first. Currently I see no voices from typing oriented core developers (or members of the typing council) in this discussion. Community interest helps, but it’s not enough on its own.

1 Like

Here’s the main takeaways I see in this thread:

  1. Applying this beyond TypedDict is unpopular, but something for only TypedDict is OK.
  2. Adding a special form Partial[...] that only accepts TypedDict is weirdly specific, but TypedDict.partial seems better.
  3. There is currently no complete workaround/solution for this use case. As explained here, total=False in a subclass isn’t always enough.

In a nutshell, there seems to be some support for TypedDict.partial in this thread, no strong objections, and a lot of desire from the broader community as seen in various links.

I think .partial looks very nice as a type constructor, e.g. MyDict.partial(foo=1). MyDict.fromkeys(['foo']) works (at runtime at least) so it’s not unprecedented.

The runtime overhead should be negligible. _TypedDictMeta has __call__ = dict which is the primary ingredient of how it works at runtime. Adding partial = dict just below makes MyDict.partial(...) work. Putting tp_dict.partial = tp_dict inside _TypedDictMeta.__new__ also works, and makes class SubDict(MyDict.partial) call _TypedDictMeta.__new__ which does something sane. More code would actually be needed to set correct values of __optional_keys__ and so on but the point is that the only additional overhead of .partial (after the subclass has been constructed once) should literally be the attribute lookup, not the calling.

2 Likes

The setting of TypedDict.partial = dict wouldn’t work on non-static type checkers and documentation genetators, so this should be a special form whose __call__ and some other methods refer to dict.

Agreed on everything else.

I think the proper thing to do would be to make MyDict.partial an instance of _TypedDictMeta, just like MyDict itself but separate. It could be constructed at the same time as MyDict, or lazily if (1) slowing down class construction is considered a problem and (2) using cached_property or an equivalent technique is possible.

1 Like

Thanks @alexmojaki for summarising.

Personally I’m strongly in favour of Partial[...] (although I’d be fine with adding TypedDict.partial too)

Reasons for Partial[...]:

  • Although Partial would be somewhat specific initially, there are many precedents for single-use typing objects - Required, NotRequird, InitVar,KW_ONLY
  • there is definitely the possibility of expanding its use in future
  • how would use you use TypedDict.partial as an annotation? def foobar(x: MyTypedDict.partial)? That looks very odd IMHO.
  • there’s a strong analogy to be drawn (and like it or not, people will draw it) betwene TypedDict partial and typescript Partial - surely it makes sense to match that general construction, unless there’s a good reason not to

@samuelcolvin, could you create a PEP for that this? Or should we discuss more first?

I would be keen to write a PEP, however we need:

  • to agree on Partial[...] - I don’t think it would make sense for me to write a PEP I don’t agree with
  • to find a core/typing dev to sponsor it
  • we (pydantic) are a bit busy at the moment, so I won’t be able to do anything for a few weeks
1 Like

All of your examples of single-use typing objects affect the type that contains the annotation and not the other way around. In addition KW_ONLY is a sentinel type, so it shouldn’t really be used as an argument for an additional type modifier. Also both InitVar and KW_ONLY are part of dataclasses, since dataclasses already require special-casing in order to be understood by type checkers, a couple of convenience types to end up with a more ergonomic API seems justified. Also you don’t need to be able to deal with either of those annotations at runtime if you inspect the dataclass directly, so you can treat them entirely as implementation details, which is not the case with an actual type modifier.

In addition Required and NotRequired have strong potential future use-cases in protocols. Partial may be useful there as well, once NotRequired within a Protocol is allowed, but it’s less obvious to me due to the previously stated caveats.

Then please name possible future use-cases to strengthen your case. The one you’ve already named doesn’t seem like a good idea. Protocol is the only other structural type we have, for nominal types you need a type constructor, otherwise you can’t ever satisfy it. Should the standard library really put the burden of constructing that type on the end user? What if you need more than one constructor for the same type? The type system would have to consider them the same type, but at runtime they’d be different.

What about P.args and P.kwargs, do those also look odd to you? You could create a TypeAlias if you don’t like how it looks, which you may wanna do anyways, if you want to encourage people to use the partial type at runtime.

TypeScript only has structural types, it has no nominal types, so I don’t think the analogy is as strong as you think it is.

Then please name possible future use-cases to strengthen your case.

As I said before, I think the dataclasses case is very strong, I’m not saying it should be included in the PEP now, just that it’s a clear future extension.

Should the standard library really put the burden of constructing that type on the end user?

No, on libraries like Pydantic, which already do lots of complex runtime stuff with types at runtime.

Some people are already doing this themselves, I think it makes sense to add the type and semantic meaning to the standard library, rather than let numerous conceptual definitions proliferate.

What about P.args and P.kwargs , do those also look odd to you?

Yes, a bit. I think,

class MyClass:
    thing: Partial[MyThing]

will be much clearer than

class MyClass:
    thing: MyThing.partial

to most developers.

TypeScript only has structural types, it has no nominal types, so I don’t think the analogy is as strong as you think it is.

Which is interesting to those deeply involved in typing and typing theory and means nothing to 99.9% of Python developers.


But the real question is: “What’s the advantage of TypedDict.partial?” I really don’t see how it makes more sense than Partial[TypedDict].

1 Like

I’ve said it before, but maybe in less obvious words: The advantage is you will get back a type constructor, that you can inspect and use as if you had written the partial version to begin with. I.e. you are putting no additional burden on runtime type introspection library authors to support Partial, it will just work.

Every special form we add, adds complexity to runtime analysis, so unless there’s a strong advantage to using a special form over a type constructor, I don’t think we should use a special form.

I strongly disagree. dataclasses are structural types, so without a single constructor they are useless, because otherwise their type identity does not match.

Yes, and those people return a new type from Partial, i.e. Partial is a type constructor, not a special form. That’s the only way this can work with nominal types.

Whether or not it means something to most Python developers is irrelevant. It needs to actually be possible to implement in a way that is internally consistent and coherent.

What if Partial[] just called a classmethod __partial_factory__ (or something, I don’t care about the name), and simply raised an exception if that method missing?

Then __partial_factory__ simply replaces partial on a TypedDict, and third party libraries can implement their own __partial_factory__ if they want to.

Implementing __partial_factory__ for a dataclass seems entirely feasible - the logic for creating the dataclass __init__ function is already pretty magic. To be clear adding this wouldn’t be required for the initial PEP, but it’s possible.

That’s me - the creator and lead maintainer of by far the most widely used runtime type library. I’m fine with the burden, and I’m sure others would be too. This seems like a pretty trival thing to implement in the grand scheme of things.

Perhaps this sentiment is why we disagree so strongly - I think “meaning something” to most Python developers is second most important after being possible. Having spent the last 6 years maintaining Pydantic, I’m totally confident this is possible.

1 Like

Sure, that’s a possible way to implement this and I would be more on board with this than making Partial a special form that returns a typing._GenericAlias, although there would be no type checker support for any partial types beyond TypedDict and possibly some other types in the standard library, so I fail to see the point, unless you can make the type checker understand what a partial version of that type looks like and I don’t know how you would do that in a generalized way.

Although there isn’t really any precedent in the typing module to use __class_getitem__ for anything other than returning some introspectable special form. There is however precedent for type constructors, such as NewType. So I would be more on board with this if it was Partial("PartialFoo", Foo).

Absolutely, but it is weird to have Partial for TypedDict and a dataclass mean completely different things, i.e. NotRequired in the former and Optional in the latter.

Yes, I know, but you always have to make the cost benefit calculation for the whole ecosystem, not just one library. I have seen many complaints from people trying to write their own introspection code about there already being too many special forms they have to deal with and special cases to consider. So we should be careful about adding more, even if it seems trivial compared to others at first glance.

I think it’s very important that things work the way most people would expect, I fully agree with you there. But using a fully structural type system as justification for a feature in a gradual type system with both nominal and structural types is not going to fly. You will need to be a bit more rigorous if you want to convince people that it’s actually possible to accomplish your goals in a way that can both be understood by type checkers and lead to sane runtime behavior.

One thing I would bring up, it’s too consider how this work for nested declarations. If a typed of dictionary contains the typed dictionary should this partial apply recursively? If not, how would I achieve that behavior?

No, it shouldn’t be recursively applied. If it’s recursively applied, you only have the option for the entire structure to be partial. There are plenty of real-world APIs where you optionally have a key to another optional mapping, but when you have it, it’s known to have a specific shape.

If you need partial on an interior structure, you should also specify it in the interior structure.

IMHO, if I need to re-declare it for nested, then this gives me little benefit. If we’re saying that just redeclare it explicitly the nested structures, then why we’re not saying to redeclare it explicitly as NotRequired the root class to (as a new separate class, which I’ll already need to do for the nested ones)?

1 Like