Running Local Models in DeepSeek Harness: vmarcelo49/dsh-local-models

Introduction¶

DeepSeek Harness (DSH) uses a plugin architecture, and its extension capabilities are implemented through plugins. Directly invoking local models from the Web interface usually requires manually managing the llama-server process or using external tools. The vmarcelo49/dsh-local-models plugin provides a solution by adding a “Local Models” tab directly to the DSH Web interface, allowing users to select .gguf files, adjust inference parameters, and launch models via llama-server (llama.cpp).

The plugin is maintained by Vmarcelo49 and licensed under the MIT license. It does not fork llama.cpp; instead, it uses the upstream binary directly and offers an installation method with zero build steps.

Core Features¶

Model Selector¶

The interface includes an embedded file browser that shows only directories and .gguf files. The backend parses the GGUF header information to display the architecture, quantization type, number of layers, context length, and MoE detection results.

Startup Options¶

Provides fine-grained control over model loading parameters:
* Context slider: Adjusted in 8k increments, capped by the model’s trained context length.
* KV Cache quantization: Select quantization types for K and V separately. The interface displays bytes per element.
* MTP and thinking: Fixed MTP draft depth (0-7) and thinking level (off/low/medium/xhigh), with an option to preserve thinking content.
* Vision and MoE: Supports an optional vision projector (mmproj) and MoE expert placement settings.

Real-Time VRAM Estimation¶

The system detects the GPU (NVIDIA via nvidia-smi, AMD via sysfs) and compares weights, KV cache, computation graph overhead, and total GPU VRAM. The interface displays three recommended values: “fit,” “safe margin,” and “maximum context.”

Profiles and Routing Mode¶

• Profiles: Save named startup configurations and reload them with one click.
• Routing mode: Serve models through an OpenAI-compatible endpoint. Models are loaded on demand, with one resident model by default. After the router starts, the plugin automatically registers the model as an LLM provider route in dsh.

Terminal Overlay¶

Provides real-time log viewing, allowing you to check the running status and output of llama-server directly within the tab.

Installation and Activation¶

Installation Steps¶

This plugin manages dependencies with pnpm; ensure pnpm is available in your system PATH.

# 从 npm 注册表安装
dsh plugin --profile web add dsh-local-models

# 或直接从 Git 克隆安装（无构建步骤）
dsh plugin --profile web add github:Vmarcelo49/dsh-local-models

After installation, the plugin is written to the dsh.profile.bundles array. Because Node-side route injection is involved, you must restart the DSH Web process after installing, refresh the browser, and enable the Local Models tab in settings.

Prerequisites¶

• DSH must have the web profile enabled.
• The llama-server binary (upstream llama.cpp) must be installed on the system.

Typical Usage¶

1. Load a model: In the Local Models tab, select a .gguf file using the file browser.
2. Adjust parameters: Based on the VRAM estimate, adjust the context length, KV quantization types, MTP depth, and thinking level. Save the configuration, then click Load.
3. Register and use: After the model loads successfully, click the “Register in dsh” button. The plugin registers the running service as an llm-pi-ai provider route, after which you can find and invoke it in DSH’s general model selector.
4. Configure routing: To switch between multiple models on a single endpoint, save configurations as Profiles, then click “Start router (from profiles)”.
5. Autostart: Check “Start the router automatically when dsh starts” in the Router card. The plugin automatically loads the router and registers the model when DSH starts, without manual intervention.

Configuration¶

The plugin runs with the following environment variables (default values can be modified in lib/client.js):

Notes¶

• No version gating: The plugin declares no peer dependencies, so it is not rejected during installation due to DSH version mismatches.
• When changes take effect: Node-side configuration or route changes require restarting DSH; client-side UI changes only require refreshing the page.
• Resource management: Routing mode supports idle eviction; by default, models are unloaded after 30 minutes without requests to free VRAM and prevent resource exhaustion.