When using spawn as the multiprocessing start method, is there an easy way to persist the environment variables (as a snapshot) from os.environ into the new processes?
We have an old case where fork was used and this all worked out, but now with spawn we lose the environment variables changed in the parent python process when the child starts up. I’m sure I could save them and pass them as a function parameter or something but that seems like something that someone may have already solved for.
Something else is probably wrong. The environment variables should pass on their own. Example:
from multiprocessing import Pool, get_start_method
import os
def print_test():
print(str(os.environ.get('TEST')))
if __name__ == '__main__':
print(get_start_method())
os.environ['TEST'] = 'hello world'
with Pool() as p:
p.apply_async(print_test).get()
os.environ['TEST'] = 'hello again'
with Pool() as p:
p.apply_async(print_test).get()
What was the problem at the end? I am running into something similar.
In my case, spawned subprocesses are about to load a c++ module but this doesn’t see the environment variables. The parent process and the subprocesses do see them.
I don’t fully remember. I think it had to do with the Pool being started before the environment variable was set. (The env var has to be set before the spawn/fork happens).