AI Agent Hub
Back to models
japanese-gpt2-small logo

japanese-gpt2-small

Open Source rinna Released 2021-08-25
-- 0.11B params 1K context Open Source

About this model

japanese-gpt2-small

rinna-icon

This repository provides a small-sized Japanese GPT-2 model. The model was trained using code from Github repository rinnakk/japanese-pretrained-models by rinna Co., Ltd.

How to use the model

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt2-small", use_fast=False)
tokenizer.do_lower_case = True  # due to some bug of tokenizer config loading

model = AutoModelForCausalLM.from_pretrained("rinna/japanese-gpt2-small")

Model architecture

A 12-layer, 768-hidden-size transformer-based language model.

Training

The model was trained on Japanese CC-100 and Japanese Wikipedia to optimize a traditional language modelling objective on 8\*V100 GPUs for around 15 days. It reaches around 21 perplexity on a chosen validation set from CC-100.

Tokenization

The model uses a sentencepiece-based tokenizer, the vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script.

Release date

August 25, 2021

How to cite

@misc{rinna-japanese-gpt2-small,
    title = {rinna/japanese-gpt2-small},
    author = {Zhao, Tianyu and Sawada, Kei},
    url = {https://huggingface.co/rinna/japanese-gpt2-small}
}

@inproceedings{sawada2024release,
    title = {Release of Pre-Trained Models for the {J}apanese Language},
    author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
    booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
    month = {5},
    year = {2024},
    pages = {13898--13905},
    url = {https://aclanthology.org/2024.lrec-main.1213},
    note = {\url{https://arxiv.org/abs/2404.01657}}
}

Licenese

The MIT license

Technical Specs

  • Parameters: 0.11B
  • Architecture: GPT-2
  • Context Window: 1,024 tokens
  • Input Modalities: text

Hardware Requirements

  • VRAM: 2.0 GB
  • Compute: Inference runs on CPU or any entry-level GPU with ~2 GB VRAM; originally trained on 8× NVIDIA V100 GPUs for approximately 15 days.