Curriculum Learning
A training strategy that presents examples to the model in a meaningful order -- typically from easy to hard -- mimicking how humans learn.
The Idea
Just as students learn arithmetic before calculus, models may benefit from seeing simple examples first and gradually encountering harder ones. This can lead to faster convergence and better final performance.
Implementation
Define a difficulty metric (sentence length, label noise, complexity), sort or weight training examples accordingly, and gradually increase difficulty during training.
In LLM Training
Some LLM training recipes start with cleaner, simpler data and introduce harder, noisier data later. The exact curriculum is one of the most impactful (and secretive) aspects of training frontier models.