Meta-Llama-3-70B

Meta-Llama-3-70B is the 70-billion-parameter base (pretrained) model in Meta's Llama 3 family, released on April 18, 2024. It is an auto-regressive transformer with Grouped-Query Attention (GQA), trained on over 15 trillion tokens of publicly available data with a knowledge cutoff of December 2023. Unlike the instruction-tuned variants, this checkpoint is intended as a foundation for fine-tuning and adaptation across a wide range of natural language generation tasks.

On standard benchmarks, the base 70B model scores 79.5 on MMLU (5-shot), 81.3 on BIG-Bench Hard (3-shot, CoT), 93.0 on ARC-Challenge (25-shot), and 79.7 on DROP (3-shot, F1). It also achieves 63.0 on AGIEval English and 52.78 on MMLU-Pro. These results made Llama 3 70B one of the strongest openly available base models at launch, significantly outperforming Llama 2 70B across knowledge, reasoning, and reading comprehension tasks.

The model supports an 8,000-token context window and is distributed under the Meta Llama 3 Community License, allowing commercial use for organizations with fewer than 700 million monthly active users. Weights are available on Hugging Face and can be deployed via transformers, vLLM, TGI, or the original llama3 codebase.