Self-Supervised Learning
A training paradigm where models learn from unlabeled data by creating their own supervisory signal from the data itself, such as predicting masked tokens or image patches.
Methods
Masked language modeling: Predict hidden words (BERT). Next-token prediction: Predict the next word (GPT). Contrastive learning: Learn to distinguish similar/dissimilar pairs (SimCLR, CLIP). Masked image modeling: Predict hidden image patches (MAE).
Impact
Self-supervised learning is the foundation of modern AI. It enables learning from the vast amounts of unlabeled data available on the internet, producing representations that transfer to countless downstream tasks.