Qwen2.5-3B-Instruct

Qwen2.5-3B-Instruct is a 3.09-billion-parameter instruction-tuned causal language model from Alibaba's Qwen team, released in September 2024 as part of the Qwen2.5 family. It uses a decoder-only Transformer architecture with grouped-query attention (16Q/2KV heads), 36 layers, RoPE positional encoding, SwiGLU feed-forward networks, and RMSNorm. The model supports a native context window of 32,768 tokens with generation up to 8,192 tokens, multilingual coverage of 29+ languages, and structured output including JSON.

On official benchmarks from the Qwen2.5 technical report and blog, Qwen2.5-3B-Instruct achieves MMLU-redux 64.4, MMLU-Pro 43.7, GSM8K 86.7, HumanEval pass@1 74.4%, MATH 65.9, and MBPP 72.7. Despite having fewer non-embedding parameters (2.77B) than competitors like Phi-3.5-mini (3.6B) and MiniCPM3-4B (4.0B), it matches or exceeds them on mathematics and coding tasks while remaining competitive on language understanding. On the HuggingFace Open LLM Leaderboard (Sep 2024), it scored BBH 46.55 and IFEval strict-prompt 60.07.

The model is widely regarded as one of the strongest sub-4B open-weight instruction models for edge deployment, fine-tuning, and local inference. It is governed by the Qwen Research License (non-commercial research use; commercial use requires separate licensing from Alibaba), and weights are available on HuggingFace and ModelScope with broad ecosystem support including vLLM, SGLang, llama.cpp, and Ollama.