Bayesian Optimization
A strategy for optimizing expensive-to-evaluate functions by building a probabilistic model of the objective and using it to select the most promising points to evaluate.
How It Works
Bayesian optimization maintains a surrogate model (typically a Gaussian Process) that estimates the objective function. An acquisition function balances exploration (trying uncertain regions) and exploitation (focusing on promising regions) to choose the next evaluation point.
Primary Use Case
Hyperparameter tuning is the most common application. When each evaluation (training a model) takes hours or days, Bayesian optimization finds good hyperparameters in far fewer trials than grid or random search.
Tools
Optuna, Weights & Biases Sweeps, Ray Tune, and BoTorch are popular implementations.