LLaMA (Large Language Model Meta AI)

What LLaMA Is

LLaMA is a series of decoder-only transformer language models released by Meta AI starting in February 2023. Like GPT-4, Claude, or Gemini, LLaMA models are trained on trillions of tokens of text and learn to predict the next token in a sequence. What makes the family notable is not its architecture, which is broadly conventional, but the licensing and distribution model: Meta releases the trained weights, allowing researchers, hobbyists, and companies to download, fine-tune, and run the models on their own hardware.

This stands in contrast to closed-weight competitors like GPT-4 and Claude, which are accessible only through paid APIs. With LLaMA, the underlying parameters of the model can be inspected, modified, and deployed offline. The practical effect has been to make state-of-the-art language modeling available to anyone with sufficient GPU memory, fundamentally changing what individuals and small teams can build.

Versions and Capabilities

LLaMA 1 (February 2023) shipped in four sizes: 7B, 13B, 33B, and 65B parameters. Trained on approximately 1.4 trillion tokens of public text, it was originally released under a research-only license to academic institutions. Within days, the weights leaked publicly via BitTorrent, and the research community began building on top of them. The leak became a watershed moment for open AI: small teams produced instruction-tuned variants (Alpaca, Vicuna, WizardLM) within weeks.

LLaMA 2 (July 2023) arrived in 7B, 13B, and 70B sizes, trained on 2 trillion tokens, and was the first major frontier-class model released under a license that permitted commercial use. Meta also published LLaMA 2 Chat, an aligned, RLHF-trained variant ready for assistant use cases. The 70B model was competitive with GPT-3.5 on many benchmarks and triggered an explosion of commercial products built on open weights.

LLaMA 3 (April 2024) launched at 8B and 70B, with a 405B-parameter version following in July 2024. Trained on more than 15 trillion tokens with a substantially larger vocabulary (128k tokens vs. 32k in LLaMA 2), it closed much of the remaining gap with proprietary frontier models. LLaMA 3.1 added a 128k token context window; LLaMA 3.2 introduced multimodal vision variants and small models tuned for on-device inference.

Architecture

LLaMA models are decoder-only transformers with a few practical refinements over the original GPT design. They use RMSNorm (instead of LayerNorm) for slightly faster training and inference, SwiGLU activations in the feed-forward blocks (instead of ReLU or GELU) for better quality at the same parameter count, and rotary positional embeddings (RoPE) rather than learned absolute positions, which generalize better to longer sequences.

From LLaMA 2 onward, the larger models also use grouped-query attention (GQA), which shares key/value projections across multiple query heads. This significantly reduces the memory footprint of the KV cache during inference, making long-context generation cheaper. Aside from these refinements, the architecture is intentionally close to vanilla transformer designs — the value of the LLaMA series lies in scale, training data, and careful engineering, not exotic architecture.

Why LLaMA Mattered

Before LLaMA, the open ecosystem had small models (GPT-2, GPT-Neo, BLOOM) that were several capability tiers behind the frontier. LLaMA closed that gap. Researchers could now study, probe, and improve a near-frontier model directly rather than treating it as a black box behind an API. Companies could deploy LLM features without sending user data to a third party, which mattered for healthcare, finance, defense, and other privacy-sensitive industries.

The fine-tuning explosion that followed was equally significant. Methods like LoRA and QLoRA made it cheap to specialize a 7B or 13B model on a single consumer GPU, leading to thousands of community variants for code, medicine, law, role-play, and dozens of non-English languages. Vicuna, Mistral's mixtures, Code Llama, Llama Guard, and many others all trace some lineage to the LLaMA release.

LLaMA also reshaped the policy debate. Open frontier models forced regulators, safety researchers, and competitors to take a position on whether widely-released weights are net beneficial or net harmful. That discussion is ongoing and is now central to AI governance.

Limitations and Considerations

LLaMA is "open-weight" but not strictly "open-source" in the traditional sense. The training data is not released, the training code is only partially documented, and the license — while permissive for most uses — restricts deployment by companies with very large user bases (over 700 million monthly active users, as of LLaMA 2's terms). Some communities argue this falls short of the Open Source Initiative's definition of open source.

The base models are pre-training checkpoints: they predict text but do not naturally follow instructions, refuse harmful requests, or hold a coherent persona. To use LLaMA as a chat assistant, you typically use the Chat or Instruct variants, or fine-tune the base model yourself. Even aligned variants exhibit the standard frontier-model failure modes: hallucination, jailbreak susceptibility, and sometimes inconsistent reasoning on adversarial inputs.

Running larger LLaMA models requires substantial hardware. The 70B model needs roughly 140 GB of memory in FP16, and the 405B model needs over 800 GB. Quantization to INT8 or INT4 makes consumer-GPU inference viable for 7B and 13B models, but the largest variants still require multi-GPU servers.

Key Takeaways

LLaMA is the most influential open-weight LLM family released to date. It compressed the gap between proprietary and open models from years to months, democratized access to near-frontier capabilities, and seeded an ecosystem of fine-tuned descendants that now powers a meaningful share of production AI workloads. For practitioners, "LLaMA" is increasingly shorthand for "the open default" — the model you reach for when you cannot or do not want to depend on a hosted API.

References

1. Touvron et al., 2023. "LLaMA: Open and Efficient Foundation Language Models." arXiv:2302.13971
2. Touvron et al., 2023. "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv:2307.09288
3. Meta AI, 2024. "Introducing Meta Llama 3."
4. Meta AI, 2024. "Introducing Llama 3.1: Our most capable models to date."