Hyperparameter
Configuration settings that control the training process and model architecture, set before training begins (unlike model parameters/weights that are learned during training).
Common Hyperparameters
Learning rate: How big each weight update step is. Batch size: Examples per training step. Number of layers/hidden units: Model capacity. Dropout rate: Regularization strength. Number of epochs: Training duration. Weight decay: L2 regularization.
Tuning Methods
Grid search (try all combinations), random search (sample randomly), Bayesian optimization (model-guided search), and population-based training. Automated tools like Optuna, Ray Tune, and Weights & Biases make tuning systematic.
Impact
Poor hyperparameters can make even the best architecture fail. The learning rate alone can mean the difference between a model that converges beautifully and one that diverges completely. Hyperparameter tuning is often the difference between a mediocre and a great model.