A proposal (and implementation) to add assignment and LOAD overloading

This message originally appeared on the python ideas mailing list. It was requested that the discussion be moved onto this platform. The message is reproduced below:

I am proposing that two new magic methods be added to python that will control assignment and loading of class
instances. This means that if an instance is bound to a variable name, any attempts to rebind that name will
result in a call to the setself (name negotiable) of the instance already bound to that name. Likewise
when a class instance bound to a name is loaded by the interpreter, if present, the getself method of that
instance will be called and its result will be returned instead. I have been internally calling these cloaking
variables as they “cloak” the underlying instance, parallelling the idea of shadowing. Feel free to suggest
better names.

On first read, that may be surprising, but it extends a behavior pattern that already exists for things like
properties (and generically descriptors) to object instances themselves. Similar caveats and behaviors will
apply here as well.

A working implementation built against python 3.7 can be found here:
https://github.com/natelust/cpython/tree/cloakingVars. This is not pull ready quality code, but the diffs may
be interesting to read.

An example for what is possible for this new behavior are instance level properties as seen in the demo at the
end of this message.

These changes have minimal impact on the runtime of existing code, and require no modifications to existing
syntax other than the use of the names setself and getself.

A more detailed write-up with more examples can be found at
https://github.com/natelust/CloakingVarWriteup/blob/master/writeup.md, with the example executable demo in the same repo as examples.py

The demos include:

  • Variables which keep track of their assignment history, with ability to rollback (possibly useful with try
    except blocks)
  • Variables which write out their value to disk when assigned to
  • An implementation of context variables using only this new framework (does not implement tokens, but could
    be added)
  • const variables that can be used to protect module level ‘constants’
  • Instance properties (reproduced below) that allow dynamically adding properties
  • An implementation of templated expression, to defer the addition of many arrays to a single for loop,
    saving possibly expensive python iterations.

I am sure the community can come up with many more interesting ideas.

The mailing list had only one example, but several are reproduced here:

# These are "builtin" functions for working with cloaked variables                                                                
# They may be implemented in c at some point to avoid the inspect
# module, but this is pragmatic for now

def getcloaked(name):
    Retrieves the object underlying a cloaked variable
    var: str
        Variable name to look up
    import inspect
    outer = inspect.stack()
    if len(outer) == 0:
        ns = outer[0]
        ns = outer[1]
    return ns.frame.f_locals[name]

def setcloaked(name, value):
    Reassigns the variable associated with "name" to a new value,
    by passing any __setself__ defined on the cloaked variable
    var: str
        Variable name to look up
    import inspect
    outer = inspect.stack()
    if len(outer) == 0:
        ns = outer[0]
        ns = outer[1]
    ns.frame.f_locals[name] = value

def cloaksset(var, deep=1):
    Returns true if variable cloaks assignment
    var: str
        Variable name to look up
    import inspect
    outer = inspect.stack()
    if len(outer) == 0:
        ns = outer[0]
        ns = outer[deep]
    return hasattr(ns.frame.f_locals[var], '__setself__')

def cloaksget(var, deep=1):
    Returns true if the variable cloaks the LOAD operation
    var: str
        Variable name to look up
    import inspect
    outer = inspect.stack()
    if len(outer) == 0:
        ns = outer[0]
        ns = outer[deep]
    return hasattr(ns.frame.f_locals[var], '__getself__')

def iscloaked(var):
    Returns True if the variable cloaks LOAD or assignment
    var: str
        Variable name to look up
    return cloaksset(var, deep=2) or cloaksget(var, deep=2)

class HistoricVar:
    def __init__(self, start):
        self.var = start
        self.history = []

    def __repr__(self):
        return "This is a HistoricVar"

    def __getself__(self):
        return self.var

    def __setself__(self, value):
        self.var = value

    def rollback_n(self, n):
        if n > len(self.history):
            raise ValueError("Can't roll back before history started")
        for i in range(n-1):

        self.var = self.history.pop()

    def get_history(self):
        return list(self.history)

print("Demoing a variable with history:")
g = HistoricVar(2)
g = 12
g = "hello world"
g = [1, 2, 3]
print(f"The current value of g is {g}")
his = getcloaked('g').get_history()
print(f"The history of g is {his}")

# Roll the variable state back
print("Rolling back the history on g")
print(f"The current value of g is {g}")
his = locals()['g'].get_history()
print(f"The history of g is {his}")

# An Example of a variable that is writes its contents to disk
# when assigned to

class FileBackedVar:
    def __init__(self, filename, starting):
        self.file = open(filename, 'wb')
        self.fileOpen = True
        self.value = starting

    def __getself__(self):
        return self.value

    def __setself__(self, value):
        import pickle  # noqa: F811
        self.value = value
        if self.fileOpen:
            pickle.dump(value, self.file)

    def close_file(self):
        if self.fileOpen:
            self.FileOpen = False

print("Demoing a variable that syncs to disk")
print("Creating a new file backed variable, with value 'hello world'")
fileVar = FileBackedVar('exampleFileVar', "hello world")
print("Reassigning the value to 'Brave new world'")
fileVar = "Brave new world"
print("Close the backing file")

with open('exampleFileVar', 'rb') as f:
    import pickle
    print("Load back in the saved var")
    value = pickle.load(f)
    print(f"The file var stored the value {value}")


# An implementation of Context Variables                                                                                          

class Context:
    declaredContextVars = {}

    def __init__(self):
        self.context_dict = {}

    def run(self, goer):
        for val in self.declaredContextVars.values():

class ContextVar:
    def __init__(self, varname, default):
        Context.declaredContextVars[varname] = self
        self.default = default
        self.varname = varname

    def __getself__(self):
            retval = self.ctx.get(self.varname, self.default)
            return retval
        except Exception:
            return self.default

    def __setself__(self, value):
            self.ctx[self.varname] = value
        except Exception:
            raise TypeError("Can't set Context variable outside context")

    def setcontext(self, ctx):
        self.ctx = ctx

context1 = Context()
context2 = Context()
convar = ContextVar('convar', "hello world")

def set_context():
    global convar
    convar = 1

def get_context():

print("Demoing an implementation of context variables:")
print("Setting the context variable in context 1")
print("Printing the context variable in context 1")
print("Printing the context variable in context 2,"
      " it has the default value")

# Constants                                                                                                                       

class Constant:
    def __init__(self, wrapped):
        self.wrapped = wrapped

    def __getself__(self):
        return self.wrapped

    def __setself__(self, value):
        raise TypeError("Constant variables can't be reassigned")

CRITICAL_NUMBER = Constant(100)

print("Demoing cost variables:")
print("The declared constant is:")
print("The type of the declared constant is (i.e. the cloaking type):")
print("The real type is:")

print("Attempting to reassign throws an error:")
except TypeError as e:

# Instance properties                                                                                                             

class InstanceProperty:
    def __init__(self, wrapped, getter, setter=None):
        self.wrapped = wrapped
        self.getter = getter
        self.setter = setter

    def __getself__(self):
        return self.getter(self.wrapped)

    def __setself__(self, value):
        if self.setter:
            return self.setter(self.wrapped, value)

class MachineState:
    def __init__(self):
        self._fields = {}

    def add_input(self, name, start):
        def getter(slf):
            return slf._fields[name]

        def setter(slf, value):
            the state of a machine part can only be above zero or below
            if value < 0:
                value = 0
            if value > 100:
                value = 100
            slf._fields[name] = value
        setter(self, start)
        inst_prop = InstanceProperty(self, getter, setter)  # noqa: F841
        # Need to directly assign the instance property, or decloak it.
        setattr(self, name, getcloaked('inst_prop'))

machine = MachineState()

for letter, start in zip(['a', 'b', 'c'], [-1, 0, 1]):
    machine.add_input(letter, start)

print("Demoing instance properties:")
print("This instance property only allows values between 0 and 100")
print("Instantiated with a: -1, b: 0, c: 1")
print(f"machine.a is {machine.a}")
print(f"machine.b is {machine.b}")
print(f"machine.c is {machine.c}")

# Assign a value that is too high
print("Assing a value of 200 to attribute c")
machine.c = 200

print(f"machine.c is {machine.c}")

# Template expressions                                                                                                            

class SimpleArrayExecutor:
    def __init__(self, nodes):
        if len(nodes) < 1:
            raise ValueError("There must be at least one node at"
                             " initialization")
        self.nodes = nodes
        self.length = len(nodes[0].values)
        self.cached = None

    def __getself__(self):
        if self.cached is not None:
            return self.cached
        print("Doing all the additions")
        addedValues = [0]*self.length
        for i in range(self.length):
            for node in self.nodes:
                addedValues[i] += node.values[i]

        self.cached = SimpleArray(addedValues)
        return self.cached

    def __add__(self, other):
        if not isinstance(other, SimpleArray) and not \
                isinstance(other, SimpleArrayExecutor):
            raise TypeError("Can only add SimpleArrays, or Simple"
                            " ArrayExecutors")
        if isinstance(other, SimpleArray):
            if len(other.values) != self.length:
                raise ValueError("Can only add Arrays of the same length")

        if isinstance(other, SimpleArrayExecutor):
            self.nodes += other.nodes
        return self

class SimpleArray:
    def __init__(self, iterable):
        self.values = list(iterable)

    def __add__(self, other):
        if not isinstance(other, SimpleArray) and not\
                isinstance(other, SimpleArrayExecutor):
            raise TypeError("Can only add SimpleArray to SimpleArray")
        if isinstance(other, SimpleArrayExecutor):
            return other + self
        if len(self.values) != len(other.values):
            raise ValueError("Can only add arrays of the same length")
        return SimpleArrayExecutor([self, other])

print("Creating 6 'long' arrays with 201 element (last element is 200)")
arr1 = SimpleArray(range(201))
arr2 = SimpleArray(range(201))
arr3 = SimpleArray(range(201))
arr4 = SimpleArray(range(201))
arr5 = SimpleArray(range(201))
arr6 = SimpleArray(range(201))

print("Add them all together with a single loop over the values")
print("Will only print 'Doing all the additions once")
arr7 = arr1 + arr2 + arr3 + arr4 + arr5 + arr6

print(f'The final array element of the combined array is {arr7.values[-1]}')


Demoing a variable with history:

The current value of g is [1, 2, 3]
The history of g is [2, 12, 'hello world']
Rolling back the history on g
The current value of g is 12
The history of g is [2]

Demoing a variable that syncs to disk

Creating a new file backed variable, with value 'hello world'
Reassigning the value to 'Brave new world'
Close the backing file
Load back in the saved var
The file var stored the value Brave new world

Demoing an implementation of context variables:

Setting the context variable in context 1
Printing the context variable in context 1
Printing the context variable in context 2, it has the default value
hello world

Demoing cost variables:
The declared constant is:
The type of the declared constant is (i.e. the cloaking type):
<class 'int'>
The real type is:
<class '__main__.Constant'>
Attempting to reassign throws an error:
Constant variables can't be reassigned

Demoing instance properties:

This instance property only allows values between 0 and 100
Instantiated with a: -1, b: 0, c: 1
machine.a is 0
machine.b is 0
machine.c is 1
Assing a value of 200 to attribute c
machine.c is 100
Creating 6 'long' arrays with 201 element (last element is 200)
Add them all together with a single loop over the values
Will only print 'Doing all the additions once
Doing all the additions
The final array element of the combined array is 1200

Hi Nate, thanks for trying to improve Python. Your message seems to assume that this is a done deal except for some improvements to the implementation (and testing etc.), but I think you are underestimating how fundamentally this proposal violates existing wisdom about how things work in Python.

Also, your description appears to be lacking a precise specification. From your implementation I see that you are intercepting two types of operations: getitem/setitem on dictionaries, and get/set of so-called “fast locals”. Here, it would seem that dictionaries already have everything you need to implement what you want (just use a dict subclass that overloads __getitem__ and __setitem__) and the code for “fast locals” looks like it has considerable trouble for an important edge case.

The process for accepting a change like this requires a PEP to be written, and since you’re not a core dev, such a PEP requires a core dev as a “sponsor”. (This hurdle is intentionally designed to prevent proposals that will never fly from wasting too many resources of the community debating the PEP.) Has a single core dev expressed any interest in supporting your proposal? (To find out who is or isn’t a core dev, you can check the user database on bugs.python.org.)

My personal opinion is that this is not a good idea – variables in Python are very different beasts than variables in C++, and you’ll just run into more and more unfortunate edge cases as you keep trying to make the CPython test suite pass. Keep trying though! You clearly have a desire to improve Python and you know your way around the code. Your next idea may be worth gold (or at least, a burnished reputation :-).


Hi Guido,
Thank you for your response. I did take some time to read up on the Pep procedure which talked about discussing ideas on on the python mailing list before starting to go down the formal PEP process. I posted some messages there, and was asked to put up a topic here, so I created one. If I in any way misrepresented this as anything other that a preliminary proposal for discussion I apologize, it was not my intention. I was just trying to do much of the leg worth to bring the best proposal for consideration and discussion as I could. The changes I posted currently pass all the tests included with python make, except for the test that checks the size of type, as I am intentionally changing it. This is not unexpected however, as I tried to make it so that existing code works exactly the same, and that there was a 1:1 api available. I am sure there are additional tests out there that I did not test against however.

This all actual started on the mailing list, as I watched messaged debating overloading assignment. This got me wondering what it would take to do so, and what the implications would be, this lead to the idea of the LOAD overloading. I figured it would be a good exercise to learn the python c api if nothing else. I previously only dabbled in the interpreter working on an old idea sent to the mailing list to implement a continue class keyword. Once I had the basic implementation in the interpreter I wanted to see what could be done with this, which lead to the examples I posted above.

Some of the functionality of those examples can be emulated with methods on objects, but some simply cant, as the type class is not allowed to hold a custom dict type from what I can tell. If I am not wrong about that, then looking up a variable, or setting one, cant really have side effects like in my HistoryVariable example. This could of course be emulated with a method call to a class, but then that class could not be used in a large number of existing libraries as they are not coded to use that assignment method. They would simply rebind the name, and the point of the history tracking variable would be lost. However, your comment makes me optimistic that at least I may be wrong about using custom dicts for class level namespaces with the type metaclass.

As I said on some mail thread that is now probably buried, I am actually not fully convinced myself that this is the best course for python. I was hoping with a mostly working prototype that there would be some good ideas, or simply more to learn. The fact that somewhat similar behavior exists with properties made me decide it was worth trying out. It is possible to write pathological properties that seem to violate assignment rules, but when used wisely they are very useful. I was hoping that might be the case for this proposal.

I do like many of the things that could be enabled with this functionality (such as a debugging variable that calls a registered callback when it is changed), but as you say there are undoubtedly many edge cases. Ultimately I do feel like I learned things, and am grateful for the feedback I have gotten so far. I appreciate everyone who has taken the time to read over what I have written.

It seems the request on the mailing list to create this topic here may have been premature. I am happy to close out this thread (or have a moderator do it) and go back to the mailing list for any future discussions on this topic.

I know this should be closed, but I felt like I owed you an apology Guido. I should have acted slower and listened to what you had to say with more attention, you do after all know much more about what you are talking about. I was just so sure I had the c-api test passing, but I must not have been triggering it in some of my tests. You are indeed right, that the c-api test is not currently passing.

Anyway I am sorry for not slowing down and listening, and will take this as a lesson to remember in the future.
-best wishes

1 Like