Introduction

The philosophy of DeepSeek Harness (DSH) is “everything is a plugin.” The community provides various tools through independent sites (such as skillhub.cn). When building local agents, recognizing handwritten content (such as notes and drafts) is a common requirement. Traditional solutions often rely on cloud APIs, which pose privacy risks and may require network access. This article introduces a local OCR plugin that can convert images of handwritten Chinese and mathematical formulas into Markdown text and LaTeX formulas without internet connectivity.

What Is This

kl3jd/handwritten-ocr is a local OCR plugin developed by the maintainer KL3jd. Its core feature is whole-page handwritten note recognition: it extracts Chinese plain text from images and converts mathematical formulas into LaTeX format. The plugin runs entirely locally, so data does not leave the local machine. It can be invoked as a plugin in DSH or other LLM tool frameworks.

Core Features

  • Whole-page recognition: Supports converting a full handwritten note page into Markdown, including Chinese plain text and LaTeX formulas.
  • Multi-backend support: Supports GPU (DirectML, recommended), CPU, and NPU inference modes.
  • Multiple interfaces: Provides a FastAPI HTTP service and a CLI command-line tool for convenient integration.
  • DSH tool integration: Includes a built-in DSH tool, allowing the LLM to directly call the ocr_image tool to recognize local images.

Installation

The plugin provides a one-click installation script for Windows, which automatically completes virtual environment setup, dependency installation, model download, and DSH preset configuration.

powershell -ExecutionPolicy Bypass -File install.ps1

The installation script automatically performs the following steps:
1. Create a Python virtual environment.
2. Install dependency packages (preferring the Tsinghua mirror, falling back to the official source if it fails).
3. Download the OCR model (using the hf-mirror domestic mirror).
4. Install the DSH preset file.

If the model already exists locally, you can add the -SkipModels parameter to the installation command to skip the download.

Usage

After installation, the plugin can be invoked via CLI, HTTP service, or DSH tools.

CLI Invocation

Use the command-line tool to recognize images directly. Because a new process is started each time, the model is reloaded, which results in slower speed.

python ocr_cli.py 笔记照片.jpg --mode gpu

HTTP Service

It is recommended to start a persistent HTTP service. The model only needs to be loaded once, and subsequent requests respond very quickly.

# 启动服务（默认端口 8765）
python ocr_server.py

After the service starts, you can access http://127.0.0.1:8765/ in a browser to view the management panel, monitor service status, PID, and engine loading status, and modify configuration parameters (with hot reload support).

DSH Tool Invocation

The installation script automatically copies the preset file to DSH’s preset-ocr-assistant directory. After refreshing, the LLM can use the ocr_image tool.

  • First call: When DSH calls the tool, it automatically starts the OCR service (about 25 seconds to load the model).
  • Subsequent calls: Recognition typically returns results in about 2~3 seconds.
  • Environment variables: If the system cannot find Python, you can specify the path by setting the OCR_PY or OCR_CLI environment variables.

Settings

The plugin’s core configuration is centralized in the settings.json file, allowing behavior to be adjusted without modifying code.

The main configuration items include:
* mode: Inference mode (gpu / npu / cpu / auto).
* formula_backend: Formula backend (unimernet / mfr / auto).
* formula_iou, mfd_conf: Used to control formula detection thresholds and merging logic.
* merge_x_iou, merge_y_gap: Used to merge split formula fragments.

In addition, the default configuration path can be overridden using the environment variable OCR_SETTINGS.

Known Limitations

  • NPU mode issue: The AMD RyzenAI-SW#389 issue blocks NPU mode (DPU dispatch error under Windows + RAI 1.8.0). It is recommended to use GPU or CPU mode on AMD hardware.
  • Matrix output: Complete bmatrix output for handwritten matrices is still being polished. Currently, it may be split into multiple formula boxes.
  • Privacy: Inference runs entirely locally, and images and data do not leave the local machine.

Summary

This plugin uses a local solution to solve the problem of digitizing handwritten content. It is suitable for scenarios that require processing sensitive data locally or using offline solutions. With DSH’s plugin-based ecosystem, it can be easily integrated into agent workflows.

  • Plugin directory: https://www.skillhub.cn/plugins/KL3jd/handwritten-ocr
  • GitHub repository: https://github.com/KL3jd/handwritten-ocr