Llama-3.1-405B-FP8

Llama 3.1 405B FP8 is Meta's official FP8-quantized variant of the Llama 3.1 405B Instruct model, released in July 2024 as part of the Llama 3.1 family. It applies 8-bit floating-point quantization to major linear layers (covering roughly 75% of inference FLOPs), cutting weight memory roughly in half versus BF16 while preserving near-identical accuracy on standard benchmarks. Independent evaluations report ~99.99% score recovery versus the full-precision instruct model, making it practical to run the world's largest open-weights LLM on a single 8-GPU node.

The underlying Llama 3.1 405B model is a 405-billion-parameter autoregressive transformer with grouped-query attention (GQA), pretrained on over 15 trillion tokens and instruction-tuned with SFT and RLHF for multilingual dialogue. It supports a 128K-token context window and eight languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai), with strong performance across knowledge, reasoning, math, and coding tasks. On Meta's internal benchmarks the instruct model reaches 87.3% on MMLU, 96.8% on GSM-8K (CoT), 89.0% on HumanEval, and 73.8% on MATH (CoT).

Released under the Llama 3.1 Community License, the FP8 checkpoint is intended for commercial and research deployment where single-node inference efficiency matters. It is compatible with mainstream serving stacks including vLLM, TensorRT-LLM, and Hugging Face Transformers, and is recommended for assistant-style chat, tool use, long-context applications, synthetic data generation, and distillation workflows.