PEP 814: Add frozendict built-in type

[vstinner (Victor Stinner)]

@ElevenPhonons

You define the interface as implementing collections.abc.Mapping then in a completely different section you define the interface relative to dict.

frozendict implements the Mapping ABC and has almost the same interface as dict (differences with dict are listed in the dedicated section). dict is a superset of Mapping. The interface is defined by the Specification section.

@ElevenPhonons

I get why you’re adding |= but I wish this wasn’t in there

frozendict has no __ior__() (a |= b) method, but implements __or__() (a | b). Python implements a |= b for frozendict. It’s the same as:

class MyClass:
    def __init__(self, value):
        self.value = value
    def __or__(self, other):
        return MyClass(self.value | other)

a = MyClass(1)
a |= 2
print(a.value)  # output: 3

@ElevenPhonons

To avoid giving up on type safeness, it it possible for the typing spec to raise an type error for cases like x: frozendict[str, list[int]](a=[1,2]); hash(x)

frozendict[str, list[int]] is a valid frozendict. Why would it raise an exception? Or do you expect a linter to emit a warning/error on hash(x)?

@ElevenPhonons

I understand the motivation to constrain the scope, but if possible, it would be useful to get the typed abstraction (typing.TypedFrozenDict or however this will work ) defined in this PEP.

It can be easily added after PEP 814 implementation. IMO the PEP doesn’t have to cover the full scope of all possible frozendict usages. typing.TypedDict was already discussed earlier.

[ElevenPhonons]

@vstinner

frozendict has no __ior__() (a |= b) method, but implements __or__() (a | b). Python implements a |= b for frozendict. It’s the same as:

I get understand the motivation to be consistent with the language and frozenset and friends. The language has evolved and we don’t really use += on a str or list anymore, and people are often confused or surprised when they see += on a tuple. The |= seems like it’s in the same boat.

@vstinner

frozendict[str, list[int]] is a valid frozendict. Why would it raise an exception? Or do you expect a linter to emit a warning/error on hash(x)

I’m talking about in the context of mypy and other static analysis tools.

[ElevenPhonons]

@Jelle

This might create a bit of a chicken-and-egg problem though; people may be hesitant to start using it if they can’t use it with TypedDict.

I agree. There’s a lot of typing leaning Python users now.

[steve.dower]

@ElevenPhonons

The language has evolved and we don’t really use += on a str or list anymore, and people are often confused or surprised when they see += on a tuple. The |= seems like it’s in the same boat.

The language still provides for executing |= by using __or__ and an assignment (__ior__ is just the ability to optimise, it isn’t the only way the operation exists). If you’re proposing to remove this from the language, this is definitely not the right PEP or thread.

Victor’s point is that |= works naturally because of the language. The PEP doesn’t have to indicate this one way or the other - those who know how augmented assignments work will understand, those who don’t will learn something, and those who will evaluate the PEP are (hopefully) drawn from the first group.

[MegaIng (Cornelius Krupp)]

@ElevenPhonons

The language has evolved and we don’t really use += on a str or list anymore

But we use it on int. And the exact same mechanisms that make it work on int make it work on str, tuple and |= with frozendict. ^[1]

If you genuinely think this is a footgun, make a new thread. But be prepare for that to be a massive uphill battle.

In general, I would suggest you make sure you understand how and why the language works before making sweeping claims, especially when using language like “we don’t” implying you speak for the community when you may not be doing that.

---

1. Note that list is not part of this of …list because it actually has a difference between a += b and a = a + b ↩︎

[ElevenPhonons]

@MegaIng

In general, I would suggest you make sure you understand how and why the language works before making sweeping claims, especially when using language like “we don’t” implying you speak for the community when you may not be doing that.

My apologies. My intent wasn’t to derail the discussion.

[davidfstr (David Foster)]

@ElevenPhonons

it would be useful to get the typed abstraction (typing.TypedFrozenDict or however this will work ) defined in this PEP. There’s enough people in Python that are using type hints and annotations that the typing aspect shouldn’t be thought of as second class citizen.

Speaking as a person who has implemented TypedDict features in type checkers: There’s enough specialized knowledge§ to get type system changes specified (and implemented in type checkers) that I’d advocate keeping any TypedFrozenDict out of the scope of the initial PEP.

§ There’s different specialized knowledge to get a built in type like frozendict landed in CPython

[vstinner (Victor Stinner)]

We submitted PEP 814 to the Steering Council! Thanks everyone for this constructive discussion! We tried to address all feedback in PEP 814. For example, a Type annotation section has been added.

[ekohilas (Evan Kohilas)]

I’m excited for this and have some clarifying questions for the PEP.

The PEP mentions:

Using an immutable mapping as a function parameter’s default value avoids the problem of mutable default values.

Could an example be given to make it more clear how and when frozendict should be used to obtain these benefits?

If I understand correctly, previously one could erroneously write:

def f(key, value, d={}):
    d[key] = value
    return d

f("a", 1) # {"a": 1}
f("b", 2) # {"a": 1, "b": 2}

And thus would instead have to write:

def f(key, value, d=None):
    if d is None:
       d = {}
    d[key] = value
    return d

f("a", 1) # {"a": 1}
f("b", 2) # {"b": 2}

As I understand, with frozendicts, the idea would be:

def f(key, value, d=frozendict()):
    d[key] = value # error
    return d

But this would fail for trying to update an immutable.

So the expected case is instead:

def f(key, value, d=frozendict()):
    d |= {key: value} # O(n)
    return d

f("a", 1) # {"a": 1} (frozendict)
f("b", 2) # {"b": 2} (frozendict)

With the caveat being that this and all future updates are O(n)?

And thus to avoid this, would the following need to be written instead?

def f(key, value, d=frozendict()):
    d = dict(d) # O(n)
    d[key] = value # not O(n)
    return d

f("a", 1) # {"a": 1} (dict)
f("b", 2) # {"b": 2} (dict)

Am I understanding the use case correctly to not have a way to replace the if d is None syntax without an O(n) slowdown?

[dr_carlos (Carlos)]

@ekohilas

Am I understanding the use case correctly to not have a way to replace the if d is None syntax without an O(n) slowdown?

For now, it seems so.

This is the opposite issue - turning a frozendict → dict, but it’s a similar case.

@vstinner

I propose to defer the idea of adding a method to convert a mutable dict to an immutable frozendict: see draft PEP change. It can be added later, but it doesn’t have to be added right now. I prefer to reduce the scope of the PEP.

@vstinner

By the way, there is no way in Python to convert a list to a tuple with O(1) complexity. Same issue to convert a set to a frozenset. Only bytearray recently got a new specific take_bytes() method just for that. If someone wants a method to convert a dict to a frozendict, maybe an unified method is needed (see PEP 351 – The freeze protocol )

So at some point, there may be a way - but this PEP is really focused on getting the feature into the language first

[oscarbenjamin]

@ekohilas

Am I understanding the use case correctly to not have a way to replace the if d is None syntax without an O(n) slowdown?

Yes but your example is not typical:

@ekohilas

def f(key, value, d=None):
    if d is None:
       d = {}
    d[key] = value
    return d

What is this function? What name would you give to it rather than f that would describe what it does? Something like set_key_in_dict_or_make_new_dict?

The function mutates a dict that was passed in but the argument to be mutated is optional and if you don’t pass it then it creates a different dict and returns that? Why does the function both mutate its argument and return the mutated result?

The more common case here is that you don’t really want to mutate the argument whether passed in or using the default value. Although you don’t intend to mutate it you don’t like risking the possibility that some unthinking code added in future might mutate it or that the function might return a reference that some other code mutates.

Seeing that the default value is immutable immediately reassures that obvious mistakes that could mutate it will not happen without needing to trace all of the rest of the code to verify that nothing mutates it or leaks a reference to it.

[brettcannon]

@ekohilas

As I understand, with frozendicts, the idea would be:

def f(key, value, d=frozendict()):
    d[key] = value # error
    return d

But this would fail for trying to update an immutable.

Correct.

@ekohilas

So the expected case is instead:

def f(key, value, d=frozendict()):
    d |= {key: value} # O(n)
    return d

f("a", 1) # {"a": 1} (frozendict)
f("b", 2) # {"b": 2} (frozendict)

With the caveat being that this and all future updates are O(n)?

The use case from the PEP is the if d is None one, but when the code already doesn’t mutate d. It’s more of a safety mechanism so you don’t accidently mutate d (which is typically a bug that’s hard to notice if you use a dict as the default value).

[ekohilas (Evan Kohilas)]

Yes but your example is not typical …

What is this function? …

Apologies for not specifying, the example was purely hypothetical to match what is typically seen for explaining how mutable arguments can go wrong.

The questions raised regardless are quite valid!

you don’t really want to mutate the argument whether passed in or using the default value.

Agree, hence it may be helpful to provide examples of how best to use frozendict for mutable defaults.

[dg-pb]

@ekohilas

If I understand correctly

Kind of, but with one nuance - it is not frozendict’s job to do what dict does.

Object is either mutable or not and somehow you expect the object to be in a quantum state of being both at the same time.

---

The use case for it in defaults is the same as the one for tuple.

def foo(these_are_ok=('a', 'b')):
    # I need to extend once
    these_are_ok += ('c',)
    # Or if many times
    these_are_ok = list(these_are_ok)
    these_are_ok.append
    these_are_ok.append
    these_are_ok.append

The point here is to have a default which can not be modified.

If you need to extend it, you find the solution that serves your case best.

In short, you can draw examples and best practices from tuple default cases.

---

Now this is regarding convenience.
But as you have noted complexities, I suppose you are worried about performance as well.

On this I can not comment much as I haven’t investigated it.
All I know is that it hasn’t become an issue for tuple despite long practice of such use.

Same might not apply to frozendict.
But if does need addressing, it is unlikely to be worth breaking any semantics for it.

@ekohilas

Agree, hence it may be helpful to provide examples of how best to use frozendict for mutable defaults.

I think your input with real life use cases where you are concerned with performance would be very helpful in this regard.

As far as I have seen, there are ideas for optimizations, but I think all input is valuable to determine the order of importance.

[parbake]

From the PEP:

Keys must be hashable and therefore immutable, but values can be mutable.

I think someone mentioned this above, but it’s worth repeating: hashable doesn’t imply immutable.

Using immutable values creates a hashable frozendict.

I don’t think immutable implies hashable either. Perhaps these two sentences should be re-written to avoid referring to mutability at all, something like:

Keys must be hashable, but values need not be. Using hashable values creates a hashable frozendict.