I have a function that is using param_grid and ParameterSampler to make a combinations and list of parameters (i.e, sampler contains 5 arguments and the length is 10. Then, using these parameters I am generating dataset and passing another function. Finally, getting result row and appending in rows.
def analyse() -> DataFrame:
param_grid = {......}
sampler = list(ParameterSampler(param_grid, n_iter=param_iters))
rows = []
for params in tqdm(sampler, total=len(sampler), desc=f"Computing '{method}'"):
dataset = generate_dataset(**params)
row = calculate(dataset, regressor=regressor, reg_name=reg_name, method=method, k=k, repetitions=n_reps)
rows.append(row)
df: DataFrame = pd.concat(rows, axis=0, ignore_index=True)
return df
The code is taking too time. So, I am trying to use multiprocessing based on the this post
def parallel_process(params, regressor,reg_name, method, bounds,k,n_reps)
params["bounds"] = bounds
dataset = generate_dataset(**params)
row = calculate(dataset, regressor=regressor, reg_name=reg_name, method=method, k=k, repetitions=n_reps)
row.insert(0, "sigma", params["sigma"])
return row
The main function
def analyse() -> DataFrame:
param_grid = {......}
sampler = list(ParameterSampler(param_grid, n_iter=param_iters))
rows = []
num_cores = multiprocessing.cpu_count()
foo_ = partial(parallel_process, bounds, regressor, reg_name, method, k, n_reps)
output = Parallel(n_jobs=num_cores)(delayed(foo_)(i) for i in sampler)
But I am getting this error TypeError: 'tuple' object does not support item assignment
My guess, I am getting the error because the sampler is a list and contains tuple information.
Could you tell me, how can I do parallelize and append the result as my previous code?