How Do You Store Objects in Shared Memory (3.8)?

Hi,
I’m trying to figure out how to use python 3.8’s shared memory to pass objects between processes.
Haven’t been able to crack it yet. Anyone got a code snippet for this?

thanks!!

I’ll reply again with examples later when I have a keyboard, but I first wanted to mention that if you’re passing data between processes, you might want to check out multiprocessing pipes. Shared values and arrays are more for distributed compute.

A simple example of sharing an array between processes:

import multiprocessing


def target1(local_arr):
    local_arr[2] = 2


def target2(local_arr):
    local_arr[1] = 4


if __name__ == "__main__":
    arr = multiprocessing.Array("B", 3)
    print(list(arr))
    p1 = multiprocessing.Process(target=target1, args=(arr,))
    p2 = multiprocessing.Process(target=target2, args=(arr,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print(list(arr))
# output
[0, 0, 0]
[0, 4, 2]

And a simple example of sending objects between processes (note: objects sent must be picklable):

import multiprocessing


class A:
    def __init__(self, b):
        self.b = b


def target1(conn):
    a = A(42)
    conn.send(a)


def target2(conn):
    a = conn.recv()
    print(a.b)


if __name__ == "__main__":
    conn1, conn2 = multiprocessing.Pipe()
    p1 = multiprocessing.Process(target=target1, args=(conn1,))
    p2 = multiprocessing.Process(target=target2, args=(conn2,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
# output
42
1 Like

Hi. Thanks for the code snippet. What I was hoping for was a way to do this using python 3.8’s new shared memory capability. Perhaps you are unaware of it? With shared memory you don’t pass or pickle objects between processes. They reside in a single space in memory and can be accessed in place by multiple processes. No pickling (which is slow).

https://docs.python.org/3/library/multiprocessing.shared_memory.html

I can do it for byte data using the sharedmemory buffer, but for objects it would require pickling which I don’t want to do. Looks like maybe it’s not possible.

Ah no, I didn’t know about this new functionality. It seems you’re right, in that it doesn’t provide methods to share arbitrary objects (although the shareable list can be quite beneficial). Either you would have to pickle, or write a C extension to create Python objects allocated in the shared memory address space.

@dgovoni Did you find any workaround?