PEP 795: Add deep immutability to Python

PEPs
122

Tracing back up the object hierarchy in this way will inevitably hit a wall where the hierarchy meets the guts of the interpreter implementation. Even if we don’t hit sys, we still have to deal with object and type. I’ll refer to these objects and modules as “freezing endpoints”.

Such objects and modules are part of internal machinery and will need to be designed to allow cross-interpreter isolation or sharing whilst remaining mutable by the runtime (not necessarily mutable by user code though). sys and os are just examples of freezing endpoints, others may include inspect, weakref, types, collections, collections.abc, typing, importlib, etc. as well as their internal implementations in both C and Python.

In short: large parts of the standard library will need to be rewritten to support freezing.

Is this architecturally possible and feasible given what these objects and modules do without breaking existing functionality? Who will do this work? How much time and effort will it take? How will we communicate and coordinate this with other parties (e.g. extension developers and other implementations such as Cython) – will we end up in a situation where some class is freezable in CPython but not other implementations? Are the benefits worth the costs?

(These are engineering questions which the nogil people had pondered for long but this PEP unfortunately does not currently address in the form of answers along the lines of: “yes free-threading is possible and technically feasible though previous efforts were abortive; the group would do a lot of work adding locks around access and testing them; GIL/nogil is CPython-specific and no change to the Python language; existing code that appears thread-safe only due to GIL is already broken; there will be benchmarked memory and single-threaded performance acceptable loss and multithreaded gains…”)

Given the efforts and costs, the merits of this proposal needs to be judged together with the overall proposition.

123

Not just large parts of the standard library, but also large parts of the library ecosystem. For example, numpy, requests, pytorch, django, …

And in any case, none of this answers my question, what does it mean to freeze a module. Because it’s not even clear to me yet that it’s possible to make all modules support freezing. So so if some key module can’t support freezing (regardless of whether it’s in the stdlib or not - sys was as I said merely an example) the whole process breaks down as soon as that module appears anywhere in the object tree.

Also, part of the point of my question about what it means to freeze a module was that if freezing a module transitively freezes all of its imports, then a module that doesn’t support freezing stops all of its dependents from supporting freezing as well. And breaking that deadlock will be a massive amount of work.

Personally, I’d rather see us working on ways of improving data sharing between interpreters that doesn’t require a massive modification effort that affects all of the 3rd party library ecosystem before it produces benefits…

Absolutely. And right now, there’s no real sign that these questions about costs and engineering effort are being taken seriously :slightly_frowning_face:

5 Likes
124

I understand it as “locking down a module object’s state sufficiently such that it allows cross-interpreter sharing or isolation”. Locking down the globals in module scope in pure Python is one thing. Many modules listed above are written in C (or have a C accelerator) and I’m not sure “true freezing” is even possible for them, though they may possibly work with cross-interpreter sharing or isolation.

In my opinion this proposal should recognize the fact that some modules might never be made freezable, and work around that to still derive some benefits, albeit with less power of a completely frozen object hierarchy. This will be much better in terms of feasibility than the currently proposed all-or-nothing approach. In particular this will allow existing classes and modules to be progressively rewritten to allow freezing over time.

1 Like
125 
from immutable import freeze, isfrozen

class Foo:
    pass

f = Foo()
g = Foo()
h = Foo()

f.f = g
g.f = h
h.f = g # cycles are OK!
del g # Remove local ref to g, so g's RC = 1
del h # Remove local reg to h, so h's RC = 1

g.x = "African Swallow" # OK # <--- g was deleted so this is a NameError.
freeze(f) # Makes, f, g and h immutable
g.x = "European Swallow" # Throws an exception "g is immutable"
isfrozen(h) # returns True
h = None # Cycle detector will eventually find and collect the cycle

I think the del g and del h need to be moved to right before the h = None where h = None is changed to f = None as @alwaysmpe pointed out.

126

I don’t think one needs to go any further than these last four messages back and forth @matthewyu0311 and @pf_moore to see very clearly the idea as it stands is stillborn.

Even if by any chance the PEP is approved in the current way, and even if there is an implementation, no one will want to use that - such a fundamental departure of the nature of the language.

One could even create a program that would make use of freezing - but it would not be possible to combine it with anything else, including most 3rd party libraries (to use any of those, it would have to bridge to a normal, non-module freezing counterpart using some sort of RPC which will serialize objects, making the freezing ability moot).

(update editing: And the current concession of the authors, of making freezing opt-in, helps far less than it should: either fundamental parts of stdlib are ‘opted out of freezing’, leaving the freezing usefulness very reduced, or are ‘opted in’ to it, leaving to the super-extension of freezing problems that is appointed above)

Now, as myself and others advanced e a couple times both in this thread and on the previous “ideas” one: taking a couple steps back, and rethinking on either: either (1) using a “frozen proxy” object, applied lazily as objects are retrieved from a frozen hierarchy, or (2) limit freezing to a few ‘quasi-literal’ classes (or better: instances of those classes), including already existing imutable objects:c strings, numbers, date-times, and then dictionaries lists and other containers, and opt in classes: but just the instances of those classes. (If a class itself wants to be freezable, like it seems to be such a normal idea by the authors, its metaclass would have to opt-in freezing).

And I reiterate that the similar behavior to (2) used in pickle: in which instances are serializable, but classes are just retrieved in the current context upon de-serialization, that this approach is not only quite feasible, as it works seamlessly in almost all cases.

3 Likes
127

The key difference between pickle (and its copy / deepcopy analogues ) and deep freezing is that pickle can be made shallow. Through __reduce__ and related methods the author is allowed full control of how the object instances of their classes should be pickled and unpickled. It is up to the class author to recreate the non-pickleable constituent objects sensibly during unpickle. You don’t need the everything in the entire application to be pickle-able as well in order for your class instance to be pickle-able.

Just as not everything is pickle-able, neither will everything be freezable. But this hasn’t stopped us from implementing pickle, nor should this stop us from making freeze work.

I think freezing should progress in the same way by means of a composition of shallow immutability. We need a way in the language to mark an object (or class of objects) as immutable, and for the runtime to detect what kind of frozen hierarchy has been made and determine what optimization it can apply. By limiting the area of responsibility to within a class or module, we can ease the transition much easier, and allow more objects to benefit from immutability. This will require upfront language support (proxy objects, read-only descriptors, etc.) but will be much less work than rewriting many existing stdlib modules at once.