AI Agent Hub
Back to models
deepseek-moe-16b-base logo

deepseek-moe-16b-base

Open Source deepseek-ai Released 2024-01-11
-- 16.4B params 4.1K context Open Source

About this model

DeepSeek Chat

[🏠Homepage] | [🤖 Chat with DeepSeek LLM] | [Discord] | [Wechat(微信)]

Paper Link👁️

1. Introduction to DeepSeekMoE

See the Introduction for more details.

2. How to Use

Here give some examples of how to use our model.

Text Completion

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

model_name = "deepseek-ai/deepseek-moe-16b-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
model.generation_config = GenerationConfig.from_pretrained(model_name)
model.generation_config.pad_token_id = model.generation_config.eos_token_id

text = "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_new_tokens=100)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

3. License

This code repository is licensed under the MIT License. The use of DeepSeekMoE models is subject to the Model License. DeepSeekMoE supports commercial use.

See the LICENSE-MODEL for more details.

4. Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.

Benchmark Scores

ARC
53.24
DROP
32.9
MATH
4.3
MBPP
39.2
MMLU
46.31
GSM8K
18.8
C-Eval
40.6
HellaSwag
79.77
HumanEval
26.8
TruthfulQA
36.08

Technical Specs

  • Parameters: 16.4B
  • Architecture: Mixture-of-Experts (MoE)
  • Context Window: 4,096 tokens
  • Input Modalities: text

Hardware Requirements

  • VRAM: 40.0 GB
  • Compute: Single GPU with ~40GB VRAM for BF16 inference without quantization (e.g., NVIDIA A100 40GB); activates ~2.8B of 16.4B parameters per token