gpt2

GPT-2 (Generative Pre-trained Transformer 2) is a decoder-only Transformer language model developed by OpenAI, introduced in the paper "Language Models are Unsupervised Multitask Learners" (Radford et al., 2019). It was pretrained on WebText, a corpus of roughly 40 GB of text from 8 million web pages linked from Reddit, using causal (autoregressive) next-token prediction. The family includes four sizes: 124M, 355M, 774M, and 1.5B parameters, with the largest variant (GPT-2 XL) released under the MIT license on November 5, 2019 after a staged rollout over misuse concerns.

GPT-2 was groundbreaking for demonstrating strong zero-shot task transfer without task-specific fine-tuning. On its original evaluation suite it set state-of-the-art zero-shot results on LAMBADA (63.24% accuracy), Winograd Schema Challenge (70.70%), and several language modeling benchmarks. However, on modern standards such as MMLU (~22.4% for GPT-2 XL, 2-shot), GSM8K (<3%, near random), and HumanEval (0%), it performs far below contemporary models. Hugging Face Open LLM Leaderboard v2 evaluations of the base gpt2 model (Nov 2024) report MMLU-Pro at 11.5%, BBH at 30.0%, and MATH at 0.3%.

Architecturally, GPT-2 uses a 12-to-48-layer Transformer decoder with 1024-token context, 50,257 BPE vocabulary, pre-norm LayerNorm, and modified initialization. It established the scaling-law paradigm that larger models yield log-linear gains across tasks, directly paving the way for GPT-3 and the modern LLM era. Today it remains widely used as an educational baseline, lightweight text-generation demo, and historical reference rather than a production-capability model.