opt-125m

OPT-125m is the smallest model in Meta AI's Open Pre-trained Transformer (OPT) family, introduced in May 2022 alongside models scaling up to 175B parameters. It is a decoder-only Transformer with 125 million parameters, 12 layers, 12 attention heads, and a 768-dimensional embedding space, trained on 180B tokens of predominantly English text using causal language modeling with GPT-2 BPE tokenization and ReLU activations. Released fully open-source under Meta's responsible AI license, it was designed to enable reproducible LLM research and provide an accessible baseline comparable in architecture to GPT-3.

On the Hugging Face Open LLM Leaderboard, OPT-125m achieves an average score of 29.17, with MMLU at 26.01% (5-shot), HellaSwag at 31.44%, TruthfulQA at 42.87%, and Winogrande at 51.62%. Its GSM8K score is 0.23% (5-shot), reflecting very limited mathematical reasoning. Research literature notes that models of this scale are generally unsuitable for MMLU evaluation in meaningful comparison to larger models, with zero-shot MMLU around 23%. HumanEval pass@1 is effectively 0% for the base model, as it lacks code-specific pretraining.

Despite weak benchmark performance by modern standards, OPT-125m remains widely used as a lightweight baseline for NLP research, quantization experiments, fine-tuning studies, and as a draft model in speculative decoding pipelines (e.g., paired with OPT-13B). It supports a 2048-token context window and integrates natively with Hugging Face Transformers, making it a common choice for educational purposes and rapid prototyping on resource-constrained hardware.