DeepSeek Harness (DSH) uses a plugin-based architecture, and lak321/dsh-screen-agent adds desktop and web automation capabilities to DSH. This plugin implements local screen recognition, UI automation (mouse/keyboard control), and drawing operations through a Windows Host route and PowerShell scripts, without requiring third-party dependency configuration.

Core Features

  • Screen recognition: Captures full-screen and multi-monitor screenshots, outputting PNG and base64 formats. Integrates Windows built-in OCR, supporting local text recognition and pixel coordinate extraction.
  • UI automation: Provides mouse click/drag, window activation, keyboard typing, and key control.
  • Drawing capability: Directly operates the Windows Paint tool to draw shapes.

Installation and Enablement

The legacy manual method of copying host/ and injecting into cordis.patch.yml is deprecated. Please use bundle-layer stack installation.

  1. Run the installation command (replace <commit> with the latest commit ID):
    dsh plugin --profile web add "github:lak321/dsh-screen-agent#<commit>&path:/"
  1. Restart DSH to apply the configuration:
    npx -y @deepseek-ai/dsh web

This plugin is compatible with DSH Profile: `web`.

Usage

Call the endpoints through HTTP requests, with JSON-formatted parameters.

  • Screenshot:
    POST /api/screen
    { "op": "capture" }

Screenshot results are saved to `~/.dsh/screenshots/`.
  • OCR recognition:
    POST /api/screen
    { "op": "ocr", "path": "...", "lang": "zh-CN" }
  • Mouse click:
    POST /api/screen
    { "op": "click", "x": 100, "y": 200 }
  • Keyboard typing:
    POST /api/screen
    { "op": "type", "text": "你好" }
  • Chat usage:
    In a DSH agent conversation, issue instructions directly, such as “Take a screenshot” or “Type xxx in the input box.” The Agent will automatically execute them through scripts or routes.

Technical Implementation and Notes

  • Platform limitation: Windows-only. Uses PowerShell, user32 / WinRT OCR, with zero third-party dependencies.
  • DPI scaling: SetProcessDPIAware() must be called when the process starts; otherwise, mouse coordinate calculation may be offset under DPI scaling.
  • Win11 Paint (WinUI 3) notes: The newer Paint has interaction differences. When using the Paint feature, note the following:
    • Color palette hang: Clicking any color swatch may cause the process to become unresponsive (WM_CLOSE ineffective); the process can only be terminated through Task Manager. It is recommended to use the eraser or default colors.
    • First drag not drawing: After switching tools, the first drag may only activate without drawing; perform the operation twice or draw a discard line first.
    • Small ellipse drawing: The small ellipse tool is difficult to draw (e.g., rx135 ry18); it is recommended to draw multiple segments with the pencil.
    • Text tool: Pressing Esc to cancel loses entered text; font size must be set by typing a number with the keyboard and pressing Enter.

License

MIT