AI Agent Hub
Back to models
deepseek-moe-16b-chat logo

deepseek-moe-16b-chat

Open Source deepseek-ai Released 2024-01-11
-- 16.4B params 4.1K context Open Source

About this model

DeepSeek Chat

[🏠Homepage] | [🤖 Chat with DeepSeek LLM] | [Discord] | [Wechat(微信)]

Paper Link👁️

1. Introduction to DeepSeekMoE

See the Introduction for more details.

2. How to Use

Here give some examples of how to use our model.

Chat Completion

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

model_name = "deepseek-ai/deepseek-moe-16b-chat"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
model.generation_config = GenerationConfig.from_pretrained(model_name)
model.generation_config.pad_token_id = model.generation_config.eos_token_id

messages = [
    {"role": "user", "content": "Who are you?"}
]
input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_tensor.to(model.device), max_new_tokens=100)

result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)

Avoiding the use of the provided function apply_chat_template, you can also interact with our model following the sample template. Note that messages should be replaced by your input.

User: {messages[0]['content']}

Assistant: {messages[1]['content']}<｜end▁of▁sentence｜>User: {messages[2]['content']}

Assistant:

Note: By default (add_special_tokens=True), our tokenizer automatically adds a bos_token (<｜begin▁of▁sentence｜>) before the input text. Additionally, since the system prompt is not compatible with this version of our models, we DO NOT RECOMMEND including the system prompt in your input.

3. License

This code repository is licensed under the MIT License. The use of DeepSeekMoE models is subject to the Model License. DeepSeekMoE supports commercial use.

See the LICENSE-MODEL for more details.

4. Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.

Benchmark Scores

ARC
50.0
BBH
42.2
DROP
33.8
MATH
15.2
MBPP
46.2
MMLU
47.2
GSM8K
62.2
C-Eval
40.0
HellaSwag
72.2
HumanEval
45.7

Technical Specs

  • Parameters: 16.4B
  • Architecture: MoE
  • Context Window: 4,096 tokens
  • Input Modalities: text

Hardware Requirements

  • VRAM: 40.0 GB
  • Compute: Single NVIDIA GPU with 40GB+ VRAM (e.g. A100 40GB); BF16 inference without quantization