Hi team, I’ve have a task with below steps where i need to execute a function let’s say generate_wat_files() for each specific file(should end with some pattern) on some set of given path directories.
The generate_wat_files(0 is time consuming where it opens some other file and write some content there which is time consuming and i’ve been using multiprocessing pool with map, however i want reduce it much more and trying to use imap with some customoized chunksize depending the no.of files in a specific path.
However I’m seeing some unexpected results with imap approach. Is there some diference btw the execution of map vs imap and some modifications need to be done with imap approach.
Also the function called by each process will just append some content to some files, so return values are not needed in my case.