Allow Manual Control Over resource_tracker in shared_memory

Ideas
1

Context and Problem Statement

I am working on a system where a main module is responsible for creating and deleting shared memory segments, while sub-modules only connect to these segments and write data. However, when a sub-module stops, resource_tracker incorrectly deletes the shared memory segment, even though it should persist.

This is a critical issue because resource_tracker assumes it has full control over shared memory cleanup, while in complex architectures like mine, only the main module should be responsible for managing these resources.

Proposed Solution

I propose adding a mechanism to control whether resource_tracker tracks a shared memory segment. There are several possible approaches:

  1. Add a flag (e.g., track=False) when creating a SharedMemory object:

    shm = SharedMemory(name="my_shared_segment", create=True, track=False)

    This would allow developers to specify that they do not want resource_tracker to track this object.

  2. Allow replacing resource_tracker with a custom implementation, so developers can define their own resource management logic.

  3. Provide an API to disable resource_tracker entirely for advanced users who manage shared memory explicitly:

    from multiprocessing import resource_tracker
resource_tracker.disable()

Why This is Needed

  • In systems with complex shared memory architectures, resource_tracker does not have enough context to decide which segments should be deleted.
  • Allowing manual control would prevent unintended resource deletion, reducing potential data loss or race conditions in multi-process applications.
  • Manually patching resource_tracker or modifying the Python standard library is not an ideal solution.

Current Workaround

For now, I have resorted to commenting out resource_tracker code in the standard library, which I acknowledge is not a proper solution.

Would the Python maintainers consider adding such an option in multiprocessing.shared_memory?

Thank you!

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

2 Likes
2

This makes sense.

I think you might have more interactions if you provided a minimal reproducible example of the problem: a small script (or two) which will create some shared memory, access it in a subprocess and in which we can see things crashing down due to premature finalization of the shared memory.
Because otherwise everyone who might be interested in contributing here would have to recreate this scenario by themselves.

3

**"I need some time to provide a minimal reproducible script. From my observations, this issue does not occur on macOS, only on Linux.

I will create a master and a host process. After the host process terminates, an internal mechanism triggers that deletes the shared memory segment. As a result, when the host process starts again, it can no longer find the segment.

master.py:

import asyncio
from multiprocessing.shared_memory import SharedMemory

async def master():
    shm = SharedMemory(name="my_shared_segment", create=True, size=1024)
    print(f"[MASTER] Created shared memory: {shm.name}")

    try:
        while True:
            await asyncio.sleep(1)
    finally:
        print("[MASTER] Cleaning up shared memory...")
        shm.unlink()

asyncio.run(master())

host.py:

import asyncio
from multiprocessing.shared_memory import SharedMemory


async def host():
	try:
		shm = SharedMemory(name="my_shared_segment")
		print(f"[HOST] Connected to shared memory: {shm.name}")
	except FileNotFoundError:
		print("[HOST] Shared memory segment not found! Exiting...")
		return

	await asyncio.sleep(2)  
	print("[HOST] Exiting now...")


asyncio.run(host())

server: 5.10.0-21-arm64 #1 SMP Debian 5.10.162-1 (2023-01-21) aarch64 GNU/Linux
python: Python 3.12.5

1 Like
4

I can see the issue now (python 3.13t on fedora 41, amd64) - the fact you find it works on MacOS may suggest it is a bug instead of a needed feature.

5

**"It works on macOS because the check specifically looks for POSIX systems before it decides to remove segments that were registered at creation.

I’ve reviewed the entire code, and what’s even more interesting is that the function itself operates through a LOCK, which can cause blocking issues when working in async. Honestly, it’s strange why it was designed this way. :man_shrugging:

But it is what it is! I’m waiting for further feedback. I also posted this on GitHub, and they recommended that I bring the discussion here."**

1 Like
6

It was certainly designed before async - which just made into the mainstream language in version 3.5 - no, it is a new thing from Python 3.8, but…but…

hey - I opened the docs right now to check the version these were added to the language - did you check the track parameter that was added in 3.13? multiprocessing.shared_memory — Shared memory for direct access across processes — Python 3.13.2 documentation

Adding the “track=False” argument to the "host’ snippet above make things work for me -
It is probably just the same thing you are asking for - actually, it even has teh same name and semantics!!!

Oh god, everyone of us thinking that @guido 's time machinne have been lost, and it is there, being actively used to further Python innovation all the time!

7

"Oh, this is just brilliant! :joy: I also checked Python 3.13, and honestly, I was impressed—the parameter is exactly as I described it!

Looks like Python evolves by the principle of “just wait a little, and it will appear on its own”. :laughing:

And yes, I must confess… I used ChatGPT when writing my response! Seems like we’re missing something, while AI already knows all the future Python features. :robot::hourglass_flowing_sand:

Either way, glad to see the issue is already solved!"

1 Like
8

Even with the Python 3.13 changes it would be great to be able to disable resource tracking altogether. From the docs its not clear whether setting track=False on all shared memory allocations will prevent the resource tracker from starting. I need to use shared memory in an application deployed with uwsgi, and spawning/forking the resource tracker was resulting in lots of error messages and failed requests. It seems like its a rather large assumption that because I am using shared memory, it’s ok to spawn/fork an additional Python process. There are many applications of shared memory that don’t use the rest of multiprocessing (ie sharing memory across different programming languages), and spawning a process can break them.

This works to disable the resource tracker in Python 3.12:

from multiprocessing import resource_tracker

def _f_null(*args, **kwargs):
    return None

# Monkey patch multiprocessing to disable resource_tracker when running with uwsgi
resource_tracker.register = _f_null
resource_tracker.unregister = _f_null
resource_tracker.ensure_running = _f_null
resource_tracker.getfd = _f_null