Collected thoughts on static typing and immutability
This is an attempt to collect answers to a lot of different points made above in one coherent place. Hope this makes sense!
Regarding static type checking — this is a very interesting question, and several of us have been working on (or are working on) type systems that include deep immutability in a different context. I will update the PEP with a discussion about typing under the Deferred Ideas section. As that suggests, we don’t think that typing is on the critical path for this work (although we’d love to explore this more in follow-up work). Below is an attempt to respond to multiple posts above regarding both typing of immutability and its relation to when things can become immutable and what that means for type-based reasoning.
Safety and when things can become immutable
if any mutable object can become immutable at any time, reasoning about what is safe is going to be real tricky, typing or not
This is a good point, and to some extent part of the very problem we are trying to solve. Types — as was pointed out elsewhere in the discussion — do not capture everything. For example, with Python’s reflective powers it is possible to remove a method from a class or change an object’s type in ways that will break type safety. But more along the lines we were thinking when we started on this PEP — types capture neither thread-safety nor whether objects are shared across threads (etc.), so reasoning about whether a call to a method on a mutable object is safe or not is already very tricky! At least with immutability, mistakes will lead to exceptions — not silent errors, so while it is not perfect that an object can become immutable, we feel it is a step in the right direction.
We envision that an exception thrown by an attempt to mutate an immutable object will show the place in the code where the object was frozen, as part of the exception.. This should help tracking down inconsistency bugs like this one.
Can freeze(x) make a deep copy, or would that cost too much?
Freezing by copy avoids the problem of mutable objects possibly becoming immutable by a non-local operation. However, because freezing objects also freeze types, there are some challenges with this approach. Consider the following:
>>> class Foo: pass
>>>
>>> f = Foo()
>>> ff = freeze_by_copy(f)
>>>
>>> f.__class__ == ff.__class__
false
...
>>> fff = freeze_by_copy(f)
>>>
>>> ff.__class__ == fff.__class__
???
First, if we are to follow the principle that freezing never turns an existing immutable object mutable, then we have to copy the Foo class before making it immutable above. Maybe this is natural since Foo and ImmutableFoo (or however we might represent them) are different types.
The second time we freeze f, what is the class of the resulting copy fff? We could keep track of the frozen copy of the Foo class, but the … in the example may have altered the Foo type, so there is no guarantee that ff.__class__ and fff.__class__ are the same. Thus, having types such as Foo and ImmutableFoo does not suffice since the shape of the type is fixed at the time of freezing. This is of course a hard thing to capture in type systems, and maybe that’s fine? I mean, there is no guarantee in Python that a type will not be changed at run-time. A pragmatic solution could be to allow certain objects to be ”frozen in-place”, such as type objects for example. Just like the PEP proposes a type that prevents freezing, we could have a type that opts in to supporting freezing in place.
Freezing by copy would open the door for optimising things. For example, we can move all frozen objects and lay them out nicely — or more compactly perhaps — in memory. For example, all immutable objects in a cycle will have the same life-time so we could make a single allocation for them, rather than individual allocations for the individual objects.
Also, freeze by copy might have unexpected effects for users of id(). The thread about hashing pointed out that the default hash implementation of Python uses the object id. This would mean that hash(x) would return something different from hash(freeze(x)). This might be the correct solution, but could also be confusing to users.
Another approach to freeze(x) would be one that ensures that all references — except x — to the objects being frozen come from within the object graph being frozen. If we detect that freezing would make an externally accessible object immutable, we could raise an exception (or possibly try to solve the problem by making a copy, but this can become tricky depending on the shape of the object graph). With this approach, we are never able to freeze a type if it has multiple incoming references (for example, more than one instance). We would have to make type immutable (which is probably good) to be able to freeze types at all.
One possibility is to support multiple styles of freezing and let the programmer decide. If copying is mostly put in to serve static typing, it seems wrong (IMO) to push this on all programmers that do not use static typing. So maybe there is space for both a version of freezing that is efficient but hard on the static type system and a version which is more easily integrated with static typing?
In summary (albeit a bit vague):
| Kinds of freezing |
Performance |
Static Typing |
Notes |
| by copy |
worst |
better |
Copies of types? |
| only isolated object graph |
worse |
better |
How to handle types? By copy? Freeze in-place? Fail? |
| in-place (like proposed in the PEP) |
better |
worse |
Solves types problem |
Immutability vs. Read-Only
The point about tracking ”mutability state” of variables vs. objects is essentially immutability vs. read-only — unless we add some extra uniqueness tracking (or similar) to ensure that variables that point to the same object always share the same immutability state. This is possible in e.g. Rust because of how Rust places very strict limits on an object graph, but not in Python because Python’s object graphs are full of cycles and pointer aliasing.
Read-only references are a lot less powerful than immutable objects and most importantly not thread-safe. If we weakened immutability to shallow or read-only, we could no longer safely share immutable objects across threads. A read-only type is therefore not strong enough to capture what we need for this PEP.
Arguably, if operations on a type T can fail because the T object has been made immutable, that is similar to type systems where T can always be ”null”.
Challenges of typing immutability
There are several challenges when adding immutability to a type system for an object-oriented programming language. First, self typing becomes “more important” — some methods require that self is mutable, some require that self is immutable (e.g. to be thread-safe), and some methods can operate on either self type. We would need a way of expressing this in the type system. Furthermore, deep immutability requires some form of ”view-point adaption”, meaning that when x is immutable, x.f is also immutable, regardless of the declared type of f. Neither is (we believe) supported yet in Python’s type system. These challenges for typing are orthogonal to the design considerations above such as whether freezing happens in-place or by copy, on isolated object graphs, and the handling of types.
In conclusion
We believe that freezing objects in-place (like in the PEP) is a good starting point for adding immutability to Python (but it need not be the end point). Freezing in-place does not make type annotations fundamentally less safe than currently – admittedly it does add another foot gun in term of the ability to freeze objects in-place. However, at the same time, it makes sharing objects between threads safe, in particular avoiding problems which are harder to debug than the potential problems due to in-place freezing.
Down the line, we would be interested in looking into extending Python’s type system with support for immutability and as part of that it may make sense to also look at adding new variants of freeze, e.g. by copy. Maybe by that time we will have made some types and other objects immutable by default (e.g. type and all integers) which might make it easier to add versions of freeze which are more amenable to static typing.
; but I think “this looks like a dict, why is .update() raising an exception?” category of user confusion needs addressing, and a “How to teach this” section will be important. There is a whole “do we now have 2x types?” mental area to cover…
) which is probably going to be very quickly. The proposed technique of “we’ll just know that the arguments have been frozen” doesn’t work well in large codebases.