PEP 764: Inlined typed dictionaries

PEPs
101

Working on one of my projects today, I encountered this TypedDict:

class PrivateBinAPIJSON(TypedDict):
    paste: str
    attachment: NotRequired[str | list[str]]
    attachment_name: NotRequired[str | list[str]]

This is actually wrong. the PrivateBin API accepts and returns 3 possible combinations:

  1. A paste with no attachments
  2. A paste with a single attachment
  3. A paste with multiple attachements

This can be correctly typed by PEP 764:

type PrivateBinAPIJSON = (
    TypedDict[{"paste": str}]
    | TypedDict[{"paste": str, "attachment": str, "attachment_name": str}]
    | TypedDict[{"paste": str, "attachment": list[str], "attachment_name": list[str]}]
)

I can achieve this today with 3 TypedDicts but that’s simply too verbose too bother with.

4 Likes
102

Couldn’t you pack attachment and attachment_name into a tuple of two elements (str, str), then do something like this?

type AttachmentData = tuple[str, str]
# Or another TypedDict for AttachmentData

class PrivateBinAPIJSON(TypedDict):
    paste: str
    attachment_data: tuple[AttachmentData, ...]

It would also stop usage of the class with say 3 attachments but 5 attachment-names. Maybe even a dictionary for name to attachment would work (unless names can be repeated).

I think for this case, and probably many others, a bit of redesigning can be helpful.

1 Like
103

I don’t think it’s fair to assume that people have the ability to redesign external JSON APIs (which seems like what’s being used here) just for the sake of simplifying typing them in Python.

5 Likes
104

I don’t control the PrivateBin API. It only accepts and returns one of those 3 shapes.

2 Likes
105

I assume @jobe’s idea was not to redesign the PrivateBin API but to choose your own more ergonomic data structure(s) and convert to/from PrivateBin’s structure at the boundaries where you send/receive that data.

I would probably model an API like that with a NewType(dict[str, Any]) and then make my own dataclass, e.g.,

PrivateBinAPIJSON = NewType(dict[str, Any])

@dataclass
class PrivateBinData:
    paste: str
    attachments: dict[str, str]

def ingest_privatebin(pbdict: PrivateBinAPIJSON) -> PrivateBinData: ...
def prepare_privatebin(pbdata: PrivateBinData) -> PrivateBinAPIJSON: ...

While using the type union would allow the type checker to warn you that you missed one of the cases, a small restructure means that you only need to care about it in exactly two places.

1 Like
106

It is already exclusively used at the boundaries. Doesn’t mean I don’t want type safety there, though.

Something like:

def create(
    paste: str,
    attachments: Attachment | Iterable[Attachment] | None,
) -> Paste:
    data = PrivateBinAPIJSON(paste=paste)

    match attachments:
        case Attachment():
            data["attachment"] = attachments.content
            data["attachment_name"] = attachments.name

        case Iterable():
            data["attachment"] = [a.content for a in attachments]
            data["attachment_name"] = [a.name for a in attachments]

    response = httpx.post(
        url,
        json=data,
    )

    return Paste.from_response(response)

Regardless, I don’t really like where this conversation is going. The current type system makes a real-world API like this impractical to type, while this PEP makes it trivial. The fact that you can tuck the untyped code away somewhere else or isolate it at the boundary is orthogonal to that because it doesn’t address the underlying limitation.

4 Likes
107

I can achieve this today with 3 TypedDicts but that’s simply too verbose too bother with.

Nerd-sniped into doing it with 2 :wink:

class Paste(TypedDict):
    paste: str


class PasteAttachment[A: (str, list[str])](Paste):
    attachment: A
    attachment_name: A


type PrivateBinAPIJSON[A: (str, list[str])] = Paste | PasteAttachment[A]
3 Likes
108

Feel we’re getting off topic with the PasteBin stuff, but I would find this general PEP super useful, even with the consideration of alternatives raised by @Viicos

PEP 9999 – Inline type expressions and inline typed dictionaries | peps.python.org seems less clean and friendly,

PEP 827: Type Manipulation seems to be quite controversial in format/grammar.

Would really like to see this progress, as I think its entirely do-able, and would be massively beneficial.

109

Can you explain with an example possibly where this would be a problem?

110 
def format_name_of_type(t: TypeHint) -> str:
    module = getattr(t, '__module__', None)
    name = getattr(t, '__name__', None)
    args = getattr(t, '__args__', None)
    if module is not None and name is not None and args is None:
        return f'{modle}.{name}'
    # ... Handle other edge cases

The above could wouldn’t fail or fall into the edge cases if __name__ == '' - it would produce __main__. which is just confusing. Either don’t set it at all, or set it to a non-empty string. An empty string is an obviously terrible idea.

2 Likes