Introduction

In DeepSeek Harness (DSH) development and usage scenarios, agents often need to access locally accumulated documents (such as Markdown, PDF, and Office files). These documents are scattered across the workspace and contain substantial Chinese content. Traditional keyword-based search makes it difficult to meet semantic understanding needs, while common remote embedding APIs are network-dependent and relatively costly.

As a DSH Bundle, dsh-doc-index aims to solve this problem. It converts local files into a retrievable semantic index, provides hybrid retrieval capabilities (BM25 + embedding vectors), and supports incremental updates with precise line number citations.

Positioning

  • Name: dsh-doc-index
  • Maintainer: JohnXu22786
  • Category: Memory
  • Core value: Turns an ordinary workspace directory into a local knowledge base with semantic search capabilities. It indexes files such as Markdown, plain text, PDF, DOCX, PPTX, and XLSX, supports Chinese tokenization, and can return snippets with precise line numbers and relevance scores.

Installation and Activation

Before installation, ensure the environment meets the following requirements:

  • Node.js version >= 22.5 (using built-in node:sqlite).
  • DeepSeek Harness is installed.

Install the plugin from the command line:

dsh plugin add <path-to-this-package>

After installation, the plugin inserts a doc-index line into cordis.patch.yml, indexes the current workspace to $DSH_HOME/doc-index/index.db by default, and automatically starts file watching for incremental updates.

Core Capabilities

The plugin provides hybrid retrieval capabilities, combining exact matching and semantic matching, and merges results using Reciprocal Rank Fusion (RRF).

  1. Hybrid retrieval:

    • BM25 retrieval: Implemented with SQLite FTS5 and a built-in CJK n-gram tokenizer, supporting Chinese tokenization without additional dependencies.
    • Semantic retrieval: Supports local embeddings, using the zero-dependency ngram algorithm (deterministic feature hashing) by default, and can be switched to transformers to load a local ONNX model.

  2. Incremental updates:

    • File watching (update: watch) is enabled by default, automatically reindexing after file changes.
    • Provides the CLI tool doc_scan, supporting a specified path or forced refresh.

  3. Result citations:

    • Search results include the document path, line number (line), snippet (snippet), and relevance score (score).

Embedding and Retrieval Configuration

The plugin controls the underlying implementation of semantic retrieval through embedding.provider:

  • ngram (default): Zero-dependency, based on character n-gram feature hashing. Suitable for offline environments, requires no model download, and has good Chinese support.
  • transformers (optional): Loads an ONNX model via @huggingface/transformers (default is multilingual MiniLM), providing richer semantic understanding.
  • none: Enables BM25 lexical retrieval only.

Search result fusion uses the Reciprocal Rank Fusion (RRF) strategy, with fusion weighting adjusted through search.rrfK.

Tools and CLI

The plugin exposes CLI tools and a DSH toolset for models or developers to invoke.

Command-line tools

# 扫描并建立索引
docindex scan

# 搜索，top 5 结果
docindex query "flux pipeline timeout" --top 5

# 重建索引
docindex reindex --full

# 查看统计信息
docindex stats

DSH tools and services

In a DSH context, the model can use the following tools, and other plugins can call the ctx.docIndex service:

  • doc_query: Performs search, with parameters including query, topK, mode (auto/lexical/semantic), and highlight.
  • doc_scan: Incremental indexing, with optional path scope and force refresh.
  • doc_reindex: Rebuilds the index, with the full parameter clearing the database before rebuilding.
  • doc_stats: Views index statistics (document count, paragraph count, size, etc.).

Notes

  • Experimental warning: Because Node.js node:sqlite is used, you may see ExperimentalWarning messages at startup. This is normal and does not affect functionality.
  • Performance limits: The plugin has built-in capacity limits, including maxDocs (default 20000), maxSegments (default 300000), and maxFileBytes (default 5MB); files or directories exceeding the limits are automatically skipped or excluded.
  • Exclusion rules: Default support includes .gitignore-like exclusion patterns, which can be extended through the excludes configuration.
  • Ecosystem note: This plugin is part of the DSH ecosystem and is injected through cordis.patch.yml; the installation path must point to a local package or npm package.