Llama-3.1-Swallow-8B-Instruct-v0.5

Llama-3.1-Swallow-8B-Instruct-v0.5 is an open-source instruction-tuned large language model developed by the Tokyo Institute of Technology Swallow team (tokyotech-llm). It is built on Meta's Llama 3.1 8B architecture through continual pre-training on approximately 200 billion tokens from the Swallow Corpus (Japanese web data, Wikipedia, math, and code), followed by supervised fine-tuning using the Gemma-3-LMSYS-Chat-1M-Synth dataset. The v0.5 release specifically enhances conversational capabilities by imitating gemma-3-27b-it behavior, representing a major upgrade over v0.3.

On official Swallow benchmarks (evaluated via lm-evaluation-harness v0.4.2), the model achieves MMLU 66.3% (5-shot), GSM8K 71.7%, and HumanEval 55.4% (pass@1). Its standout strength is Japanese language performance: it ranks state-of-the-art among open-source models with ≤8B parameters on Japanese MT-Bench (JMTAvg 0.719, judged by gpt-4o-2024-08-06), outperforming Qwen2.5-7B-Instruct (0.665) and its predecessor v0.3 (0.705) by 1.5 points. It also leads on Japanese tasks including JMMLU (58.1%), JHumanEval (49.6%), and MGSM (60.4%).

The model supports bilingual Japanese-English chat and text generation with a 128K token context window inherited from Llama 3.1. It is released under the Meta Llama 3.1 Community License and Gemma Terms of Use, making it suitable for research and commercial applications requiring strong Japanese conversational AI at the 8B parameter scale. English performance remains competitive but trails top models like Qwen2.5-7B-Instruct on MMLU and HumanEval.