ELECTRA
A pre-training method that uses replaced token detection instead of masked language modeling.
Overview
ELECTRA is a pre-training approach introduced by Google in 2020 that replaces BERT's masked language modeling with a more sample-efficient objective called replaced token detection. A small generator network produces plausible replacements for some tokens, and the main discriminator network learns to identify which tokens were replaced.
Key Details
Because the discriminator trains on all input tokens (not just the 15% that are masked), ELECTRA learns from every example more efficiently. A small ELECTRA model can match the performance of a much larger BERT model, making it an excellent choice for resource-constrained settings.