Long Context
The ability of language models to process very long inputs (100K+ tokens), enabling analysis of entire books, codebases, or lengthy conversation histories in a single interaction.
Techniques
RoPE scaling (extend position encodings), Flash Attention (efficient memory usage), sparse attention (attend to subsets), retrieval-augmented approaches, and architectural innovations like Mamba's linear-time processing.
Practical Challenges
The 'lost in the middle' problem (models attend poorly to information in the middle of long contexts). Cost increases linearly or quadratically with length. Evaluation of long-context capabilities is still developing (RULER, needle-in-a-haystack tests).