Llama-3.2-3B

Llama-3.2-3B is Meta's 3.21-billion-parameter pretrained text-only language model from the Llama 3.2 family, released in September 2024. It is designed as a lightweight foundation model for edge and mobile deployment, supporting a 128K-token context window and eight languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai). The model uses an optimized Transformer architecture with Grouped-Query Attention (GQA) and shared embeddings, trained on up to 9 trillion tokens with a knowledge cutoff of December 2023.

During pretraining, Llama-3.2-3B incorporates knowledge distillation from the larger Llama 3.1 8B and 70B models, where logits from these teacher models serve as token-level targets to recover performance after pruning. This approach enables the compact 3B model to punch above its weight class on standard benchmarks while remaining efficient enough for on-device inference. Quantized variants (SpinQuant and QLoRA) further reduce memory footprint to roughly 2.4–2.5 GB, enabling deployment on smartphones and other constrained hardware.

As a base pretrained model, Llama-3.2-3B is intended for fine-tuning and adaptation across a wide range of natural language generation tasks. It achieves competitive results among models in its size class on MMLU, ARC-Challenge, and reading comprehension benchmarks, and serves as the foundation for the instruction-tuned Llama-3.2-3B-Instruct variant optimized for dialogue, summarization, and tool-use applications.