PEP 705 – TypedMapping

PEPs
1

Hi all,

I’d like to put PEP 705 forward for discussion. It is proposing a read-only mapping protocol, similar to TypedDict but without the mutator methods. This permits correctly type hinting read-only parameters, which currently are very hard to make fully general.

See also the prior discussion on typing-sig.

5 Likes
2

I haven’t used Final much so excuse my ignorance, but beyond the fact that this implements the Mapping protocol, how is this proposal different from annotating a TypedDict with Final or @final?

3

Marking a TypedDict class final prevents it being subclassed. That is currently interpreted by typecheckers as meaning all keys not specified in the type definition must be absent. It has no effect on whether instances are mutable or not.

1 Like
4

The @final decorator means that a class cannot be subclassed. The Final modifier means that a variable cannot be reassigned. Both of those are orthogonal to the problems that TypedMapping solves, which are about mutability of the object itself.

2 Likes
5

Thanks for this, I think it will be helpful! A couple minor questions:

In section “Multiple inheritance and TypedDict”, the phrasing “mutator methods” confused me a bit since I think of that as referring to just setitem etc. independent of a particular key. If a mutating operation is done with something that is not statically known to be a key in the TypedDict, I assume that should cause a checker error? e.g. assert somestr in movie; movie[somestr] = None

Re:

but is not itself a protocol unless it inherits directly from TypedMapping or Protocol

Isn’t a class currently not a protocol unless it inherits directly from Protocol? Why would we add TypedMapping to that rule?

6

the phrasing “mutator methods” confused me

That’s a good point. I think of them as separate methods (which they would be in e.g. Java) but in Python they’re overloads on a single method. Would it be clearer if it was phrased this way?

  • adds mutator method overloads only for fields it explicitly (re)declares

The code sample would change to

movie["year"] = 1985  # Fine; setitem mutator overload for "year" added in definition
movie["name"] = "Terminator"  # Type check error; "name" not in setitem overloads

Isn’t a class currently not a protocol unless it inherits directly from Protocol?

Currently, yes. With the addition of TypedMapping, however, it becomes necessary to distinguish between a “protocol” lower-case-p and Protocol upper-case-P. A TypedMapping type is a protocol type, lower-case-p, because it acts, once created, exactly like any other protocol†. Protocol is now just one of two ways to create a protocol type. A simple motivating example:

class One(Protocol):
  def clear() -> None: ...

class Two(TypedMapping, total=False):
  key: int

class Three(One, Two, Protocol):
  pass

class Four(One, Two, TypedMapping):
  pass

Classes Three and Four are structurally identical: they can be used completely interchangeably. It makes sense to me therefore to just call them both protocols, lower-case-p.

† Except that TypedDict will allow mixing in a TypedMapping protocol, but will not currently allow mixing in a Protocol protocol even if it logically could. This is a matter of pragmatism rather than necessity, though.

7

Another possibility you could maybe discuss in “Rejected Alternatives” is something like:

from typing import ReadOnly, TypedDict

class Movie(TypedDict):
    name: ReadOnly[str]
    year: ReadOnly[int | None]

where any field marked as ReadOnly[] may not be mutated.

I think this would give you the same features? The main downside I see is the verbose syntax; especially if you combine it with NotRequired:

from typing import NotRequired, ReadOnly, TypedDict

class Movie(TypedDict):
    name: ReadOnly[str]
    year: ReadOnly[NotRequired[int | None]]
8

The PEP looks really good a couple of comments/questions

https://peps.python.org/pep-0705/#specification

  • The | operator is not supported.

Maybe this is just my brain on typing all the time but I think it would be a good idea to clarify that it’s on instances of TypedMapping and not the types themselves.

I think mentioning multiple inheritance with a nominal mapping subclass might be a good idea. I’m not entirely sure what your opinions are on this, it’s something that I would like to see but I don’t feel super strongly about it.

1 Like
9

Hi @alicederyn I’m glad you’re working on TypedMapping! This has been an idea for a while.

Some feedback from reading the PEP:

Abstract

  • It might be nice to say up-front that:
    • TypedMapping is like a TypedDict but with read-only keys.
      • (I think this message is very easy to understand.)
    • It implements only the read-only methods from Mapping but not the write methods from dict.
      • (Mentioning Mapping gives a hint/explanation about why this new type is called a TypedMapping.)

  • As TypedDict is a mutable type, it is difficult to correctly annotate methods which accept read-only parameters in a way that doesn’t prevent valid inputs.

    • A more direct phrasing might be: “TypedMapping can be used to annotate methods which accept read-only dictionary-like parameters, unlike TypedDict which also permits write operations.”
      • (I think a phrasing that emphasizes what TypedMapping is rather than what it is not is easier to understand.)

Specification

    • The runtime type of a TypedMapping object is not constrained to be a dict.
    • Probably worth being explicit that it is constrained to be a Mapping.
    • May want to add/clarify: …although it can be a dict.
      • I speculate dict literals will be a common way to create instances of TypedMapping, so it’s nice to mention clearly that its possible to do.

  • Multiple inheritance and TypedDict

    • I feel a bit uneasy about permitting a class to inherit from both a TypedMapping and a TypedDict (or even a vanilla dict) at the same time. It makes a partially mutable type, muddying the distinction between a fully immutable and a fully mutable type. Maybe this feature could be omitted from the PEP?
      • Are there compelling use cases for defining a dict with some fields as mutable and others as immutable? (I can’t think of any easily.)
      • If there are such cases, would it make more sense to propose a syntax to mark individual fields of a TypedDict as read-only instead?
        • Perhaps something like the Required[] or NotRequired[] syntax could be used, maybe using Final[] to mark read-only values?

Backwards Compatibility

  • This PEP changes the rules for how TypedDict behaves (allowing subclasses to inherit from TypedMapping protocols […]

    • FYI: If we remove the ability to inherit from both TypedDict and TypedMapping at the same time, as suggested above, this backward compatibility consideration goes away, leaving no notable backward compatibility considerations remaining.

Reference Implementation

  • No reference implementation exists yet.

    • I’d recommend getting an implementation drafted/reviewed for at least one typechecker (ex: mypy, pyright) to discover any remaining edge cases in this PEP’s specification that would be worth mentioning, before submitting the PEP for Steering Council approval.

Fin

Thanks again for taking the time to bring TypedMapping to life!

4 Likes
10

Thanks for writing this PEP. Overall, it’s looking good to me.

There’s one area that I feel strongly needs to be changed. I don’t think it will work to support multiple inheritance between TypedDict and TypedMapping. Conceptually and practically, hybrid mutable/immutable classes are problematic. I recommend replacing the multiple inheritance section with the statement that says any attempt to create a class that inherits from both TypedDict and TypedMapping will result in a runtime exception.

Today, a TypedDict that inherits from any type other than another TypedDict generates an exception. Likewise, a TypedMapping that inherits from any type other than another TypedMapping should generate an exception. The one exception to this rule is Generic, which is now permitted as a base class for a TypedDict as of Python 3.10. The same exception should obviously be made for TypedMapping. I recommend mentioning this explicitly in the PEP so it’s not overlooked by the implementor(s).

The current draft doesn’t say anything about the behavioral intersection with PEP 692. I presume that TypedMapping can also be used with **kwargs and Unpack, but the **kwargs value will be treated as immutable in this case.

I’m happy to implement the draft PEP in pyright to uncover any remaining edge cases. I’ll try to get to that within the next couple of days.

2 Likes
11

Small correction: the ability to create generic TypedDicts was added in Python 3.11, not 3.10.

2 Likes
12

These types are necessary to correctly type hint methods that read some keys and write others, without artificially limiting what types the type checker permits. For instance:

def set_hash_field(message):
    message["hash"] = hasher(
        message["key1"], message["key2"]
    )

The types of key1 and key2 can be any type hasher accepts, including subtypes, but that doesn’t mean that any type hasher accepts could be written to key1/key2, so no mutator method declarations make sense.

(Aside: I prefer the term “read-only” rather than “immutable” here, as I think it is less misleading. In protocols, you can’t ever declare something “immutable”, you can only declare it read-only in the context of your structurally typed method. It could always be mutated by concurrent code, aliases, methods, callbacks…)

I’m not sure why this would be conceptually problematic, given the language already supports Protocols with a mix of mutable and read-only properties, for instance. Perhaps you could give some details of what problems you anticipate encountering?

13

I don’t have a strong opinion. This would let users reuse an existing TypedMapping type, but it doesn’t AFAIK let them express anything they couldn’t already express. Is it worth the additional implementation burden? I defer to your expertise here :slight_smile:

14

I started looking into implementing TypedMapping in typing-extensions and ran into a few issues, overlapping with what Eric posted above.

I would like the PEP to include an explicit list covering all the allowed ways TypedMapping can interact with inheritance: what classes can be base classes of a TypedMapping, and what classes can inherit from a TypedMapping?

For example, is this legal?

class Base(TypedMapping):
    a: int

class Child(Base):
    @property
    def prop(self) -> str: ...

If so, what does that mean? Does it create a protocol?

The PEP does say explicitly that it is possible to double-inherit from a TypedMapping and a TypedDict. As Eric already said, this is not allowed by the runtime implementation of TypedDict. That’s not an insurmountable issue (we can change TypedDict for 3.13+ in CPython and provide a backport in typing-extensions), but it will make it harder to use the feature for now.

I would also prefer to remove support for mixed TypedDicts/TypedMappings. They add considerable complexity and I don’t see them as particularly useful. I don’t think the use case in your previous post is particularly common or convincing.

The PEP says that “A TypedMapping type defines a protocol with the same methods as Mapping”, which is a little problematic because Mapping is not in fact a protocol.

1 Like
16

That’s a good idea, I can crib the language from PEP 544.

It works like Protocol. From the PEP:

A class definition defines a TypedMapping protocol if and only if TypedMapping appears directly in its class bases.

Child doesn’t have TypedMapping in its class bases, so it is a regular class (which must implement mapping methods).

I will add a section explicitly about non-multiple inheritance, with example, to make this clearer.

Unfortunately this mixing is important for the code we are trying to type-hint. Our original proposal didn’t even include a non-mixed TypedMapping type, that was a later change suggested on the typing mailing list to make it more general. It was just a proposal to add read-only keys to TypedDict.

It is very common when writing a lot of dict manipulation code to have common snippets like this that you want to factor out. Messages often have a mix of common elements and differing elements, and methods often modify data structures in-place.

17

If your goal is to add an immutable TypedMapping counterpart to a TypedDict, then I don’t think it should support hybrid mutable/immutable objects. This will be conceptually confusing for users, won’t compose well with existing and new typing features, and create a bunch of challenging edge cases for type checkers. As just one example, it’s not clear whether the update method should be present on an object that is a hybrid of a TypedDict and TypedMapping. If it is, what argument type should it accept?

We should also learn a lesson from the original TypedDict design, which introduced the notion of “required” and “not required” fields but allowed them to be combined only through a cumbersome inheritance mechanism where some layers of the class hierarchy were marked total and others were not. This problem was addressed by PEP 655, which introduced the more flexible Required and NotRequired special forms. Even if we were all convinced that hybrid mutable/immutable objects weren’t a problem, this design would create a new problem similar to the one that needed to be fixed by PEP 655.

If your goal is to add support for read-only keys to TypedDict (and I think that’s a reasonable goal), then I’d like to propose an alternative approach. (It sounds like this was the direction you were exploring with your earlier iterations of the idea.) Rather than add a new TypedMapping type, we could introduce a new special form called ReadOnly. It would be similar to Required and NotRequired and would be applied to type declarations for keys within a TypedDict. This ReadOnly form would need to compose with Required and NotRequired, but I think that’s easily doable.

class MyHybridDict(TypedDict):
    a: ReadOnly[str]
    b: ReadOnly[Required[int]]
    c: str

A new read_only keyword parameter could also be added to TypedDict (similar to total) that marks all fields as read-only even if they are not annotated as ReadOnly.

@final
class MyMapping(TypedDict, total=False, read_only=True):
    a: str
    b: int

If all of the keys in a TypedDict are marked read-only and the class is marked @final, then the type would be considered compatible with Mapping[str, object]. If any keys are not read-only or the class is not marked @final, then it would not be compatible with Mapping[str, object].

Here are some benefits I see to this alternative proposal:

  1. If the goal is to introduce the ability to introduce read-only keys for TypedDict, then it more directly models this intent. That makes it conceptually simpler for users to understand.
  2. It provides a standardized way to “spell” the type of a field that is read-only. This is important for error messages and language server features (hover text, etc.).
  3. The rules for inheritance are conceptually simpler. We simply need to define what happens when a subclass overrides a key within a base class and the read-only attribute disagrees. (It’s fine to override a writable key as read-only but not the other way around.)
  4. It composes better with other features that have been discussed in this part of the type system including inlined TypedDict definitions (e.g. dict[{"a": int, "b": ReadOnly[str]}]) and the ability to specify that a TypedDict supports an arbitrary number of additional keys that match a specific type. It’s less clear to me how these new features would work with the currently-proposed design.
  5. I could also see the ReadOnly special form being useful in other typing contexts in the future — for example, for protocol attributes and dataclass fields.

In summary, I think this PEP should choose one of two different routes:

  1. Introduce a TypedMapping type that is an immutable counterpart of TypedDict. Most Python users have a good understanding of the relationship between Mapping and dict, and it’s reasonable to assume that the same relationship exists between TypedMapping and TypedDict.
  2. Introduce something like a ReadOnly special form and allow it to be applied to keys within a TypedDict.
4 Likes
18

I’m happy to add this to the PEP in addition to the new type. I’m reluctant to discard prior community feedback that TypedMapping is strongly desired, however.

I think there are three parts to the proposal:

  1. A standalone TypedMapping
  2. Addition of read-only keys to TypedDict
  3. Ability for TypedDicts to extend TypedMappings

Are there any complications to introducing both 1 and 2 in the same PEP? Because I believe 3 is then simply a way to specify some read-only keys in a TypedDict using an existing TypedMapping.

That seems to violate the Liskov substitution principle. I believe it should be the other way around?

I don’t believe it should be, as I’m not aware of a way to prevent the readonly keys being updated. That said, I’m not sure how you can validly specify update at all on a TypedDict, without accidentally letting users violate type safety.

class A(TypedDict):
  foo: int

class B(TypedDict, A):
  bar: str

class C(TypedDict, A):
  bar: int

b: B = { "foo": 1, "bar": "baz" }
c: C = { "foo": 2, "bar": 3 }
a: A = b
a.update(c)  # Is this allowed? b is no longer a B.

Have I missed something?

(edit note: coalesced two posts as the site told me this was preferred)

Another motivating example for read-only keys in TypedDicts:

def set_some_fields(x):
    a, b, c = do_some_request()
    x["a"] = a
    x["submessage"]["b"] = b
    x["submessage"]["c"] = c

There’s no way using just TypedDict as it stands to correctly type hint this method; nested TypedDicts are effectively not structurally typed. With TypedMapping and/or ReadOnly, you can declare the “submessage” key read-only.

20

I went back to the original typing-sig thread to check and it seems this email by @Gobot1234 is the one bringing up the idea of a new type called TypedMapping instead of addding readonly=True to TypedDict. I didn’t see much justification for the change in the email. Maybe @Gobot1234 can elaborate?

21

My idea to make it a concrete class was based off of multiple inheritance being supported with other mapping subclasses like MultiDict or bidict because I thought it would be nice to be able to preserve typing info about them if you were working with structured data

22

I’m interested to hear from other members of the community, but personally I do not think the PEP should include multiple concepts, especially if there are significant overlaps between them. I also don’t see a need for both TypedMapping and read-only TypedDict keys. I’d prefer to pick one solution, not two overlapping ones.

That seems to violate the Liskov substitution principle. I believe it should be the other way around?

Yep, good catch. I stated it backward.

[On the topic of the update method…] I’m not sure how you can validly specify update at all on a TypedDict, without accidentally letting users violate type safety.

The way update is currently defined for TypedDict, your code sample produces a type violation. You can verify this in mypy or pyright. Sounds like update should not be available if any keys in a TypedDict are read-only.

@Gobot1234, I don’t see how it would work to allow subclassing of both TypedMapping and other dictionary-like classes like MultiDict or bidict (or even dict, for that matter). Do you buy my argument that a ReadOnly special form for the existing TypedDict has many advantages if the goal is to permit read-only fields in a TypedDict? Any other thoughts on the ReadOnly proposal?