Multiprocessing without the unnecessary imports?

Python Help
1

Use case
Deep within the codebase, I want to load a file, split it into parts, process it in parallel and return some dicts to the parent.

Problem
All imports from the module where I submit the tasks are being re-loaded, some of which have a large memory footprint/are slow to load.

Python supports different start methods in the multiprocessing library:

  • fork is fast but can lead to deadlocks.
  • forkserver and spawn are safer, but all imports will still be re-run (once for forkserver if set_forkserver_preload is used , N times for spawn).

This makes me think, I don’t really need any of the context from the parent process in my subprocess. I don’t want that baggage. My subprocesses just need to import the processing function once, and then process the content each time data is received.

Is there a lightweight way of doing that? Is this be what PEP 734 would have been for?

1 Like
2

Let me cite the docs

The parent process starts a fresh Python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.

Sounds like you need to carefully inspect your package structure, payload and data to understand why unwanted “context” is imported.

1 Like
3

Here is a minimal example:

# main.py
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor, as_completed

from . import slow_import  # noqa
from .helper import hello

if __name__ == '__main__':
    mp.set_start_method('spawn')
    with ProcessPoolExecutor(max_workers=3) as pool:
        futures = [pool.submit(hello, i) for i in range(3)]

    for future in as_completed(futures):
        print(future.result())


#helper.py
def hello(value: int) -> int:
    return value**2

# slow_import.py
import time

time.sleep(1)
print("I slept")

Running main produces:

I slept
I slept
I slept
0
4
1

Meaning each subprocess incurred the cost of slow_import.

Putting slow_import under the __main__ guard is not an option, as in reality the multiprocessing code is not in main.py but in a different module.

4

Looks like the main module is needed for some reason. I was able to not import slow_import by putting an early return into this one:

I simplified your example though towards multiprocessing.Process. It seems like you may also hack through it like this

import multiprocessing as mp

import slow_import  # noqa
from helper import hello

if __name__ == '__main__':
    __file__ = None
    mp.set_start_method('spawn')
    p = mp.Process(target=hello, args=(42,))
    p.start()
    p.join()
5

You could turn the slow import into a lazy import using importlib.util.LazyLoader.

There’s an example in the documentation, although I think you have to modify the call to find_spec and pass in package=__package__ to use relative imports.

Also, keep in mind (apologies if I’m pointing out the obvious) that you only have to pay this cost once per worker, not for each submitted task. In your example you’re creating 3 workers and submitting 3 tasks and being subjected to 3 slow imports, but if you were to keep the pool around and submit 1000 tasks, the slow import would still only occur the first 3 times.

1 Like
6

I dont think this plays well with type annotations (one will get into the if TYPE_CHEKING: world, etc).

My point is that this feels like a gap in the python standard library. I should not need to do convoluted things to avoid reimporting all the modules when wanting to interact with a subprocess.

As @pulkin shows with this “early return” code pointer, what I think should be provided is more granular control when starting a process. But I am still wondering whether I am missing something obvious that allows one to do that.

1 Like
7

well, to put it simple your main problem is the design of the app :wink:

8

Joking aside, why shouldn’t I be able spawn a simple clean process without refactoring a messy app? One shouldn’t need to design for every possible requirement upfront.

1 Like
9

I think it is a valid concern: documentation says that it is tight on inheriting parent resources while it seems to import the main module unnecessarily.

Important disclaimer: this is valid for a specific example on my platform and in my python.

1 Like
10

well, simple clean question: why is the slow import in the executor module ? :upside_down_face:

2 Likes
11

I don’t know what you are referring to. See the example. Other modules are needed for any business reason.

12

well, modules are the basic organizational entities in Python (namespace etc) and (usually) hold common functinonality together (they are definitely not just a collection of code) :melting_face:

13

You sound condescending so im going to ignore you.

2 Likes
14

multiprocressing cannot know what (global) state your application needs setup for the functions you are using to work correctly - so it executes the main module[1] in the hopes that this sets up all needed state in the new process.

It is now on you, the user of multiprocressing to minimize the work being done. This means moving unnecessary, slow imports in your main file into the if __name__ == '__main__' block - this is not something the python stdlib can do for you.

  1. Somewhat. See the method you linked for a bit of the slightly confusing details ↩︎

1 Like
15

IMO the non configurable behavior to run parent’s main in worker process (_fixup_main_from_name _fixup_main_from_path in multiprocessing/spawn.py) is more or less unfriendly. If it was not the only possible option (I can’t tell), I would say it’s assuming too much.

I think we can all agree it’s impossible to automatically import what are strictly needed.

But not importing enough can be worked around by a user. Importing too many cannot.

1 Like
16

Yes it can? I genuinely don’t know why you think this is not possible, so I can’t give you suggestions.

17

I may not control all the mains that ultimately trigger my multiprocessing function. I know however, what imports I need to make my function run.

18

Ok, but then it isn’t your program and the overhead is not your problem. You just need to document “this library uses multiprocessing, make sure your main is compatible with that”.

1 Like
19

I don’t think that is the right answer, it can lead to frustrations and extra work. It seems better to allow the developer to opt-in to more control.

1 Like
20

Can we have an example from you to work around OP’s case?

Unless of course it requires a rewrite of parent’s entrypoint, because not everyone gets to control the entrypoint. There are cases like library authors, plugin authors, framework users etc.

1 Like