starcoderbase

StarCoderBase is a 15.5-billion-parameter code large language model developed by the BigCode collaborative research project. Trained on 1 trillion tokens from The Stack v1.2—a corpus of permissively licensed source code spanning 80+ programming languages, along with Git commits, GitHub issues, and Jupyter notebooks—the model is designed as a foundation for code generation and completion rather than conversational instruction following.

The model uses a GPT-2-style decoder-only Transformer architecture with multi-query attention for efficient large-batch inference and Fill-in-the-Middle (FIM) training for code infilling. It supports an 8,192-token context window and was trained in bfloat16 on 512 Tesla A100 GPUs over 24 days. On standard code benchmarks, StarCoderBase achieves 30.4% pass@1 on HumanEval and 49.0% on MBPP, outperforming prior open-access code models and matching or exceeding OpenAI's closed code-cushman-001.

Released in May 2023 under the BigCode OpenRAIL-M license, StarCoderBase serves as the multilingual pretraining base for StarCoder (fine-tuned on an additional 35B Python tokens). The release includes model weights, training code, evaluation harnesses, PII redaction tools, and a code attribution search index to support responsible open deployment of code AI systems.