Introduce Partial for TypedDict

[lawsonjl (Joe)]

@flying-sheep

Is is possible to have multiple arities for a generic type?

My understanding is that in the absence of any defined parameter type defaults, if GenericType[T1, T2] is a valid form, then GenericType[T1] always means GenericType[T1, Any] (unless GenericType is variadic).

So to support both forms, it’d need to be a special case, or you’d need a very clever default type for the second type parameter. There’s already some precedent for special cases^[1] so it’s not unheard of, but of course special cases should probably be kept to a minimum.

Really the main thing the proposals in this thread are missing (including my drive-by generator-expression one) is they don’t really define the iteration over the type-members of the source type, or the aggregation of the mapped type-members back into the target type. This, IMO, is what leaves all these proposals feeling a little too “magical” or “limited”.

Hopefully an example helps demonstrate what I meant by that nonsense I just said:

class Something(TypedDict):
    ... # who cares what's in here, doesn't matter

type Attributes[T] = ...  # "iteration over the type-members of the source type"
type IntoTypedDict[T] = ... # "aggregation of the mapped type-members back into the target type"

# Something, but all attributes are Optional now
type OptionalSomething = IntoTypedDict[Map[Optional, Attributes[Something]]]

# This loosely corresponds to something like:
result = dict(map(func, iter(source)))

# Which suggests a more "complete" generator-expression syntax than the one I previously proposed:
type OptionalSomething = IntoTypedDict[(name, T | None) for name, T in Attributes[Something]]

Making those “aggregate” and “iterate” parts explicit has a few benefits:

1. It, subjectively, feels less magical.
2. It’s more clear how to generalize this beyond TypedDict for use in dataclasses, Protocols, or other constructions - just define the appropriate “type aggregators” and/or “type iterators”.
3. It may help solve the runtime concern @Zomatree expressed with using generator-expressions, by making the “iterate” piece actually iterable at runtime, and making the “aggregate” piece actually
   construct a type instance at runtime - while still remaining statically analyzable.^[2]

I wish I had the time or the requisite background knowledge to even attempt to draft up a PEP for this, because I really think the generator-expression approach (with “type iterators” and “type aggregators”) is the right path forward here. After all, it’s pretty much what TypeScript is doing with its mapped types, except here it’s using one of Python’s most iconic syntax features to make it work.^[3] I realize “TypeScript does it that way” isn’t enough on its own to declare an approach the right answer, but it’s certainly strong evidence that it’s a powerful, composable approach. Certainly more powerful/composable than a Map[T, U] that only works for TypedDict and only works with transforms that look like Transform[T], anyway.

---

1. e.g. tuple[str, ...] doesn’t mean “a tuple of a string and an instance of ...” ↩︎

2. I’m not sure you even need to go that far - the AST of Aggregate[Transform[T] for T in Iterate[S]] may have everything you need for runtime type inspection? ↩︎

3. You could also get conditional types for free, with the generator-expression’s if clause, Never, and an appropriately defined “type aggregator”. ↩︎

[Viicos (Victorien)]

Together with @samuelcolvin and Sydney Runkle, we’d like to try pushing this forward. Here are a couple thoughts I had:

• The idea would be to introduce a partial equivalent to the one defined in TypeScript. However, TypeScript also defines a Required<Type> counterpart. I think it would be beneficial to have both implemented in Python ^[1].

• If the partial implementation is implemented using a new Partial[...] form and we want to support a required implementation as well, this causes issues because Required already exists. So we could either:

  • Use a different naming (Total[...] and NonTotal[...] for instance). This would allow the following:

  class SomeTD(TypedDict, total=False):
      f1: Required[OtherTD]
      f2: Total[OtherTD]
  

  • Use attributes: SomeTD.partial / SomeTD.required as suggested in this thread. The benefit is that we don’t need to introduce new members to the typing module, the downside is that it can’t be used for anything else than typed dictionaries (leaving structural types aside). Protocols were mentioned, for instance.

• It would also maybe make sense to have an equivalent to TypeScript’s Omit and Pick. If we were to use the same form with brackets, how should this be spelled? Pick[TD, 'k1', 'k2'] / Pick[TD, ('k1', 'k2')]? What about as a attribute?

• Regarding support for structural types – mainly dataclass-like types – I agree we shouldn’t mix the two. One other reason not to do so imo is that it can be ambiguous (using attrs):

  from attrs import NOTHING, define
  
  @define
  class A:
      a: int
      b: int | None
  
  pA = Partial[A]
  # `a` is 'int | None = None', or 'int | NOTHING = NOTHING'?
  # `b` is 'int | None = None', or 'int | None | NOTHING = NOTHING'?
  # Let's go with `None` for now..
  
  rA = Required[A]  # name TBD
  # `a` is 'int', or 'int | None'?
  # `b` is 'int', or 'int | None'?
  

  I believe we could make use of Annotated (cross ref this comment) or tweak the @dataclass_transform() spec ^[2] to do so. Quick API sketch:

   from typing import PartialSpec
  
  partial_spec = PartialSpec(
      default=None,  # Or MISSING, etc.
  )
  
  @dataclass_transform(...)
  class BaseModel:
      @classmethod
       def as_partial(cls) -> Annotated[type[Self], partial_spec]: ...
  

---

1. We could argue that only having a partial implementation is enough, as SomeTD and Partial[SomeTD] gives you both variants, but you could also have SomeTD defined with some fields as required and some not required. ↩︎

2. Converters are an example of something not supported by stdlib dataclasses but supported by dataclass transforms. ↩︎

[erictraut]

I see the value of Partial, Required, Pick and Omit, but implementing each of these as custom special forms in the type system doesn’t strike me as the best approach.

In TypeScript, these are not special-cased operators. Rather, they are built on a small number of more general type operators that allow other useful transforms (Readonly, Record, Exclude, Extract, etc.) to be built. These operators give TypeScript a “super power” that Python’s type system currently lacks.

For context, here are the definitions for Partial, Required, Pick, and Omit in TypeScript:

// Make all properties in T optional
type Partial<T> = {
    [P in keyof T]?: T[P];
};

// Make all properties in T required
type Required<T> = {
    [P in keyof T]-?: T[P];

// From T, pick a set of properties whose keys are in the union K
type Pick<T, K extends keyof T> = {
    [P in K]: T[P];
};

// Construct a type with the properties of T except for those in type K.
type Omit<T, K extends keyof any> = Pick<T, Exclude<keyof T, K>>;

Before converging on a concrete proposal for how to implement these in the Python type system, I encourage you to consider more generalized solution like the one found in TypeScript. I started to explore this idea here but didn’t fully flesh it out.

We may decide in the end that hard-coding the behavior of a handful of these transforms is the only viable solution, but that feels to me like a missed opportunity.

[alwaysmpe (Marc Edwards)]

I wanted to add my 2 cents because I would find better type mapping super useful and have messed around with the type system quite a bit. Currently I don’t use TypedDict because in the typeshed typing interface it’s a subclass of Mapping[str, object] and basically isn’t compatible with typed dictionaries so you either go all or nothing. I more frequently use the standard mapping type. I instead type it using Literal string keys giving comparable behaviour with a more concise mapped type for use outside of the mapping. It’s important to note the difference between Literal and LiteralString, one is less constraining than the other.

Being able to do something like the below would be really nice:

from typing import Any, reveal_type, overload, 

type KVT = tuple[LiteralString, Any]

class LitMapping[KvT: KVT]:
    def __getitem__[Kt: LiteralString, Vt, OtherT: KVT](
        self: LitMapping[tuple[Kt, Vt] | OtherT],
        key: Kt
    ) -> Vt:
        ...

    # a different way to annotate
    # the above with different results
    def alt_get_item[Kt: LiteralString, Vt](
        self: LitMapping[tuple[Kt, Vt]],
        key: Kt
    ) -> Vt:
        ...

type IntKvPair = tuple[Literal["int_arg"], int]
type StrKvPair = tuple[Literal["str_arg"], str]

type IntStrMapping = LitMapping[IntKvPair | StrKvPair]

def get_mapped_vals(mapping: IntStrMapping):
    int_val = mapping["int_arg"]
    reveal_type(int_val) # revealed type: int - correct
    str_val = mapping["str_arg"] # type error: "str_arg" not permitted?
    reveal_type(str_val) # revealed type: int?
    str_val = mapping.alt_get_item("str_arg")
    reveal_type(str_val) # revealed type: int | str - not perfect but not bad.

def remove_key[OtherT: KVT](
    mapping: LitMapping[tuple[Literal["removed_key"], Any] | OtherT]
) -> LitMapping[OtherT]:
    ...

def add_key[ExistingT: KVT](
    mapping: LitMapping[ExistingT]
) -> LitMapping[ExistingT | tuple[Literal["added_key"], Any]]:
    ...

def merge_mappings[ExistingLHS: KVT, ExistingRHS: KVT](
    lhs_mapping: LitMapping[ExistingLHS],
    rhs_mapping: LitMapping[ExistingRHS]
) -> LitMapping[ExistingRHS | ExistingRHS]:
    ...

And from a typing point of view… A lot of that works today in pyright. Pyright doesn’t correctly and consistently unpack the typing union so depending on how the getitem function is annotated gives slightly different results. But it’s not far off supporting a typed mapping already. For a single value type, the alt_get_item version works as is with correct key literals.

This could then be extended to permit required/optional mapping arguments:

class OptMapping[ReqTv: KVT, OptKv: KVT]:
    @overload
    def __getitem__[Kt: LiteralString, Vt, OtherReqT: KVT, OptionalT: KVT](
        self: OptMapping[tuple[Kt, Vt] | OtherReqT, OptionalT], 
        key: Kt
    ) -> Vt:
        ...

    @overload
    def __getitem__[ReqT: KVT, Kt: LiteralString, Vt, OtherOptionalT: KVT](self: OptionalMapping[ReqT, tuple[Kt, Vt] | OtherOptionalT], key: Kt) -> Vt:
        ...

    def __getitem__(self, key: Any) -> Any:
        ...

This gives a type error in the overload as the LiteralString is less constraining than Literal. As such the overload is considered indistinguishable from the first as a result.

Iterating through the items within the mapping is already typed by the union.

It’s currently quite verbose and some of the typing is redundant so could be simplified.

In the type system as I understand it, the following are equivalent:

type OrTuple = tuple[Literal["a"], int] | tuple[Literal["a"], str]
type OrValue = tuple[Literal["a"], int | str]

So a union of tuples can work like a mapping.

I’ve got a heavy C++/computer vision real time data background so that probably massively colors my preferences. I’m happy with templating, but I’m aware the above might look off putting to others.

[alwaysmpe (Marc Edwards)]

Bit more thought and comparison with typescripts functions. Assuming it were possible to constrain a type parameter to a Literal type, then with a mapping class like the below and typing to match, the desired functionality could be done as described at the end

from typing import Any, assert_type, Literal

type KVT = tuple[Literal[str], Any]

class Mapping[KvT: KVT]:
    def __getitem__[Kt: Literal[str], Vt, OtherT: KVT](
        self: Mapping[tuple[Kt, Vt] | OtherT],
        key: Kt
    ) -> Vt:
        ...
type IntKey = Literal["a", "b", "c"]
type IntKV = tuple[IntKey, int]
type StrKey = Literal["d", "e", "f"]
type StrKV = tuple[StrKey, str]
type StrIntKey = Literal["g", "h", "i"]
type StrIntKV = tuple[StrIntKey, str | int]

type MyMapping = Mapping[IntKV | StrKV | StrIntKV]

def inspect_mapping(mapping: MyMapping):
    int_a_val = mapping["a"]
    assert_type(int_a_val, int)
    str_d_val = mapping["d"]
    assert_type(str_d_val, str)
    intstr_g_val = mapping["g"]
    assert_type(str_d_val, str | int)

There has been some discussion about how to implement for example typescript’s Partial. This could be typed using typing.NotRequired applied to keys. The below wouldn’t type check but hopefully makes sense.

type Partial[M: Mapping[tuple[TKey, TVal]] =  Mapping[tuple[NotRequired[TKey], TVal]]
type MyPartialMapping = Partial[MyMapping]

and Pick

type Pick[M: Mapping[tuple[TPicked, TPVal] | TRest], TPicked: Literal[str]] = Mapping[tuple[TPicked, TPVal]]
type MyIntOnlyMapping = Pick[MyMapping, IntKey]
assert_type(MyIntOnlyMapping , Mapping[tuple[Literal["a", "b", "c"], int]])

Omit would be similar

type Omit[M: Mapping[tuple[TOmit, Any] | TKept], TOmit: Literal[str]] = Mapping[TKept]
type MyOmitNonIntMapping = Omit[MyMapping, StrKey | StrIntKey]
assert_type(MyOmitNonIntMapping, MyIntOnlyMapping)

Required is similar to Partial:

type Required[M: Mapping[tuple[TKey, TVal]] =  Mapping[tuple[Required[TKey], TVal]]
# Assumes the below
assert_type(Required[NotRequired[Literal["a"]]], Required[Literal["a"])

So far I’ve talked about string keys, but as tuple is a mapping the below would be an example of specifying tuple’s mapping interface

type MyTuple = tuple[int, int, str, int | str]
type MyTupleMapping = Mapping[tuple[Literal[0,1], int] | tuple[Literal[2], str] | tuple[Literal[3, int | str]]

And for one last extreme aside, a 2 by 2 array (numpy?)

type TwoByTwo = Mapping[tuple[Literal[0,1], Literal[0,1], int]]

Last update: With a little coersion pyright can as is use the above mapping type to infer value types from mapped literal keys.

class GetterFn[Tk]:
    @staticmethod
    def get[Tv, To](key: Tk, mapping: Mapping[tuple[Tk, Tv ] | To]) -> Tv:
        ...

def use_get_fn(mapping: IntStrMapping):
    int_val = GetterFn[IntKey].get("int_arg", mapping)
    reveal_type(int_val) # inferred type: int - correct
    str_val = GetterFn[StrKey].get("str_arg", mapping)
    reveal_type(str_val) # inferred type: str - correct

So correct mapping types could be achieved if pyright was less eagerly merging tuple types. If a TypedDict can be transformed to this mapped type representation and key/value types can be manipulated as above then the rest follows using existing typing.NotRequired and typing.Required.

Practically speaking what you’d need to manipulate these is Map, Unpack, GetN, and Zip. For comparison, if you had a tuple of tuples of int/str and wanted to pass the first value in each tuple to the str constructor, those are the operations you’d need. GetN to provide a type index operator. Union of types behaves like a set (actually a bidirectional mapping depending on which key you constrain the tuple type by), typevartuple behaves like a sequence. They’re both iterable, order only has meaning in one case. I think with all of those you could also specify a lot of panda’s type interface if it was given a basic column name to column type mapping. The distinction between a mapping and ordered mapping would become important for projects like pandas though. Aim to manipulate the Literal[key] type rather than the Mapping type.

An alternative and simpler approach would be to allow a type statement to be templated, similar to c++. Essentially the syntax I gave above.

Taking this to another extreme, having a way to convert classes to this typed mapping and back would mean it’s possible for dataclass_transform operators to type the changes they make to the class. ORMs? How about typed database queries?

[sakgoyal (Saksham Goyal)]

VSCode already kind of has support for partials. in Pylance, the update function for TypedDicts has a Partial[T] type but it is not exposed to the user because its implemented in JS.
So it’s definately possible to do this. I just wish it was part of python itself. its an extremely useful feature to have