Add a dict method to convert to a frozendict?

Core Development
21

Would it help if the method was framed as “clear and return the content as a frozendict” rather than “convert to frozendict and clear the source”. That way, clearing is obviously the significant action, with getting a frozen copy of the dict being secondary (even though it’s likely why the method is being used).

Maybe name the method clear_and_freeze[1]? We could even add a keyword argument freeze_content=False to the existing clear() method, but IMO that’s probably a step too far (boolean args that change the return type are obnoxious).

  1. I don’t love this name, but it’s the best I can think of. Better names are welcomed :slightly_smiling_face: ↩︎

1 Like
22

Maybe drain_and_freeze? I like that “drain” implies that there is a flow of data.

3 Likes
23

Looking at these use-cases, particularly number 1, another API which seems natural to me would be a with statement builder. E.g.:

with frozendict.builder() as frzn:
  # Build up the dict, mutate it
  frzn["foo"] = "bar" 

# Exiting takes contents and freezes the dict
assert isinstance(frzn, frozendict)

If there is a concern about Python users misusing a take_frozendict()-style method, this would be an alternative which makes clear that the mutable context is limited and that the mutable dict inside the with block shouldn’t be exported outside of that block.

The obvious downside to me is just that re-using a with target after the with block is a bit unnatural. And (though I am unsure) it may be complex to completely change the object that the with target refers to within an exit function.

2 Likes
24

+1, esp. on the list → tuple part. It could take us closer to efficiently comprehending into an immutable data structure.

1 Like
25

Not really, I’d prefer it was framed as with FrozenDictBuilder that lets you add items to it and then get a frozendict at the end, and it doesn’t “behave” like a dict in the meantime, so there’s no temptation to pass it around like one.

But ultimately, I’d rather not be solving these problems by creating new types that enforce constraints when documentation/design will do.

If frozendict has some incredibly more efficient storage mechanism than a regular dict, then great, but obviously there will be some cost in constructing it, and so an “O(1) conversion from a fundamentally different data structure” is kinda silly. And if it’s the same implementation as a dict (such that “O(1) conversion” makes sense) but has code looking over your shoulder every time you use it, then I’d rather just ignore it and stick to dict (as I said during the earlier discussions about the type itself).

From an API perspective, I expect the function (provided by the library) to document its expectations/intentions for the values (provided by the user), and then it’s up to the user to choose an appropriate type. If the function then needs to convert it to do its task, that’s up to the function to implement it efficiently, provided it doesn’t breach its contract (such as not clearing the user’s dict to “freeze” it).

Providing a function - especially a method, since methods are way more attractive to our OO friends - is a massive temptation to do something that probably never actually needs to be done. So a context-managed type to do in-place construction is better from my POV, because it is obviously meant to be used solely by the user (that is, the creator of the value).

I’m still yet to see a use for frozendict in a public API that is in any way better than “dict and we promise we won’t modify it”. (Similarly for Cody’s bytearray/bytes examples - returning a bytearray and allocating a new one is just as efficient as moving the memory from the first into the second and allocating new memory, and all you get is a weak/expensive signal to the user that they aren’t expected to write into it, as if being returned from .read() wasn’t clear enough that it’s already got the value you want.)

1 Like
26

That’s a very fair point. I was assuming there was a pre-existing justification for using a frozendict, and measurements demonstrating that the extra copy needed to create a frozendict from a dict was important enough to want to avoid. But on re-reading, it looks like @vstinner is leaning heavily on the logic used for zero-copy bytearray/bytes conversion, and even if you accept that was a reasonable conversion to add, the use cases for bytes seem very different than those for frozendict/dict.

So I guess I’m -0 on the idea, on the basis of “why bother?”, and the difficulty in coming up with a good name for the operation moves me towards -1 (because if we can’t decide what to call it, that suggests we don’t have a good feeling for what it’s useful for).

2 Likes
27

I think where it holds to the most value is where you start with values that shouldn’t be modified, and have apis that may not explicitly promise this, but currently don’t modify it. For providing a concurrency safe api, the promise itself + documenting taking any mapping-like (perhaps collections.abc.Mapping) is plenty.

On the consumer side, passing in a frozendict prevents unpleasant surprises if any of those apis change in the future, whether intentionally or accidentally. you get a clear failure immediately rather than unexpected behavior that isn’t an exception.

28

Right, but this is defensive programming - negative programming, or hopeless/helpless programming :wink: Whatever word you like, it should come with a negative connotation.

Python should be a place for hopeful programming, where we don’t have to assume that one day our dependencies are going to screw us over.

There’s plenty of space to argue about how best to do that, but given the choice between “status quo” and “handing dependencies a tool that clears dicts on use”, I think the second is pretty clearly increasing the risk of needing something like frozendict to catch the bad behaviour. And I don’t particularly want to see either option be adopted by the language (style guides, type-checkers and linters are a different matter, since they’re independently opt-in on both sides of the equation, whereas a stdlib type feature is unilaterally available and nobody can opt-out).

1 Like
29

Hmm. I don’t think I see it that way, but I get the perspective. A builder pattern[1] avoids the most negative outcome there, and I’m not opposed to a builder pattern, but I see either a function or a method here as just more convenient, more closely matching the c api, and something that python should document as “only use this for construction, dont use this on dicts you got from elsewhere”, placing the burden in the right place on deciding when to create one this way.

Another option is not providing this api on the python side at all, and optimizing the frozendict({...}) case.

  1. specifically a builder pattern that doesn’t allow using the builder like a dict, and only provides the frozendict once the context is exited ↩︎

30

I’ve got clear use cases for frozen dicts, based on an “ownership model” that ensures you don’t have to think about cache corruption because it won’t happen.
I don’t think it deserves a negative connotation. It’s more worry-free programming than anything else.

The core semi-conflicting axioms are these:

  • you don’t have ownership over objects passed into a function as arguments - you’re merely borrowing them
  • you give ownership of any objects that you return from a function
    (there’s also some rules to do with classes)

This is a ‘problem’ that can be illustrated by a very simple function:

def f(x): return x

Here f does not have ownership over x, but f gives away ownership over x.

The dogmatic approach would be to define instead

def f(x): return deepcopy(x)

but that’s doing extra compute, and often without any benefit, because the place that’s passing in the arguments is usually already ‘owner’ of the object it’s passing in, or maybe it’s the case that the thing being passed into the function is a newly created objects, or sometimes there’s some kind of guarantee that the receiver of x won’t change it.

So I have to think about information flows, and I don’t like it when I’m forced to think. Especially about information flows, because those are non-local by their very nature.

All of this thought can be avoided if x is of a frozen type.

There is no critical need, because 80% of the time we can resolve this with frozen dataclasses.

All that said, I’d vastly prefer a better frozendict constructor to a dict → frozendict method.

I think all the places I’ve needed/wanted frozen dicts so far, it’s been possible to write the dict using a single dict expression, and I think a dict expression that starts with f{, frozen{ or frozendict{ looks significantly better that a dict expression that ends with }.take_frozendict(), especially if the dict expression takes several lines.

1 Like
31

That’s only true if it’s also true for the unfrozen type.

What you’re demonstrating here is that the “fix” actually makes things worse, because it makes you think you’re safe when you may not be, and so you’re actually more likely to get caught out by more obscure [mis]use.

If you want to transfer ownership of the return value to the caller, then make sure it’s a transparent object (typically stdlib defined). If you don’t, make it opaque (a custom type that all they can do is pass to another one of your functions, or well-defined interfaces that you explicitly support). Living in the middle is how you get in trouble.

33

I’m very confused what you’re saying. I wrote a very long post earlier, but that’s probably not the way to go. Could you be a bit more explicit about how you think one would be likely to be caught out?

34

This is correct (unless specifically documented otherwise), and not conflicting. Your illustration is actually fine, because returning an argument back to the owner isn’t “giving away” ownership, and so you wouldn’t deepcopy an argument (but you might deepcopy your own internal data structures to ensure you retain ownership of them).

The place where it becomes risky is where you try to approach this situation and decide to “efficiently” convert your internal data structure into a frozen type, and now you’ve wiped your internal copy of the data.

Yes, it’s possible to do the conversion at the “right” time, but now you’re baking the frozen-ness of your data structure into the core of your application logic, which gives you a lot of places to get it wrong. Or you just do the deepcopy (or equivalent) when you’re writing a boundary function that returns internal data to outside and give ownership to the caller (or you wrap it in your own type and only advertise the read-only interfaces, and then your internal data never actually changes ownership).

36

The point is that it can just juggle some pointers instead of copying something that is going to be discarded anyway.