Ensemble Method
A technique that combines predictions from multiple models to achieve better performance than any single model, leveraging the 'wisdom of crowds' principle.
Key Approaches
Bagging (Random Forest): Train multiple models on bootstrap samples, average predictions. Boosting (XGBoost, LightGBM): Sequentially train models that correct previous models' errors. Stacking: Train a meta-model on the predictions of base models.
Why Ensembles Work
Different models make different errors. By combining them, individual errors cancel out while shared correct predictions reinforce each other. Ensembles reduce variance and can reduce bias.
In Practice
Ensembles dominate Kaggle competitions and tabular data tasks. For LLMs, a form of ensembling is 'majority voting' across multiple sampled responses (self-consistency). Production systems often ensemble multiple specialized models.