Parallel processing

Parallel Processing with joblib

Grid Search Parallelization

from joblib import Parallel, delayed
import itertools

def evaluate_params(min_samples, epsilon, shape_weight, citsci_grouped, expert_grouped, all_images):
    # ... evaluate one hyperparameter combination
    return f1_avg, delta_avg, min_samples, epsilon, shape_weight

param_grid = list(itertools.product(
    range(3, 10),           # min_samples
    range(4, 25, 2),        # epsilon
    [round(0.9 + i*0.1, 1) for i in range(11)]  # shape_weight
))

results = Parallel(n_jobs=-1)(
    delayed(evaluate_params)(ms, eps, sw, citsci_grouped, expert_grouped, all_images)
    for ms, eps, sw in param_grid
)

Key Points

• n_jobs=-1 uses all available cores
• delayed() wraps the function for lazy evaluation
• Each call should be independent (no shared mutable state)
• Pass pre-grouped DataFrames to avoid redundant groupby in each worker