Preface¶

By default, agents in DeepSeek Harness (DSH) can only process text. When faced with a local video file, the common approach is to manually extract frames, transcribe, and then feed the results into the conversation—a process that is cumbersome and often misaligned with specific timestamps.

Below, we introduce the community plugin dsh-video-lens (maintained by dundunhan). It registers three tools in the DSH Profile, uses ffprobe/ffmpeg for metadata extraction and frame grabbing, and then interfaces with an OpenAI-compatible vision model and optional ASR (Automatic Speech Recognition) to synthesize visual, transcription, and timeline data into a structured evidence JSON, enabling subsequent reasoning by pure text models.

What is This¶

dsh-video-lens is a video understanding plugin for DeepSeek Harness. The current version is v0.3.1, licensed under MIT, with source code available on GitHub (approximately 28 stars). Its directory page is: SkillHub.

It does not include built-in decoders; all media processing is delegated to the system’s ffmpeg/ffprobe. Visual and speech recognition are invoked via configurable baseUrl, model, and environment variable keys, avoiding lock-in to a single provider.

Core Features¶

The plugin exposes three tools, which function as follows:

The processing flow can be summarized as:

video file ──► video_probe ──► ffprobe ──► metadata JSON
           └─► video_analyze ──► scdet scene detection ──► shot boundaries
                                 ├─► representative frame sampling per shot (with limits)
                                 ├─► audio extraction ──► ASR transcription (optional)
                                 └─► OpenAI-compatible vision API ──► evidence JSON

Key implementation details (from the README):

• Scene detection relies on the scdet filter from ffmpeg ≥ 6.0; if no cuts are detected, it falls back to uniform mid-point sampling.
• ASR is an optional capability: if asrApiKeyEnv is not configured or ASR fails, visual analysis still completes, and transcript is null.
• video_ask expands the window around matched transcription segments by askPaddingSec (default 2 seconds) before re-extracting frames.

Installation and Enabling¶

Requirements: Node.js ≥ 20; ffmpeg ≥ 6.0 (recommended) and ffprobe in the PATH (e.g., via brew install ffmpeg or apt install ffmpeg).

Install dependencies in the DSH Profile directory (the level containing package.json):

pnpm add dsh-video-lens

Register the bundle in the Profile’s package.json. The complete example from the README is:

{
  "dependencies": {
    "dsh-video-lens": "^0.3"
  },
  "dsh": {
    "profile": {
      "bundles": [
        "@deepseek-ai/dsh-base",
        "@deepseek-ai/dsh-web-app",
        "dsh-video-lens"
      ]
    }
  }
}

Configure API keys and restart the Profile:

export VIDEO_LENS_API_KEY=sk-...        # Vision model
export VIDEO_LENS_ASR_KEY=sk-...        # Optional, for ASR

For development and debugging, you can also clone the repository and mount it via local linking. Steps are detailed in the GitHub README.

Configuration¶

Common DSH configuration keys (default values are based on the v0.3.1 README):

The full list is available in the Configuration section of the repository README.

Typical Usage¶

Simply describe the local video path to the agent, for example:

What’s in /tmp/demo.mp4?

The agent will typically first call video_probe, then video_analyze. The returned evidence JSON includes metadata, a list of shots, sampled frames, optional transcription, and an analysis field generated by the vision model (overall summary, timeline, on-screen text, etc.). The structure is roughly as follows:

{
  "metadata": { "container": "mov,mp4,m4a,3gp,3g2,mj2", "durationSec": 268.4 },
  "shots": [{ "timeSec": 12.3, "score": 45.2 }],
  "framesSampled": [{ "timestampSec": 5.5, "jpegBytes": 12345 }],
  "transcript": {
    "text": "…",
    "segments": [{ "start": 0.0, "end": 2.4, "text": "…" }],
    "language": "zh"
  },
  "visionModel": "Qwen/Qwen3-VL-8B-Instruct",
  "analysis": {
    "overall_summary": "…",
    "timeline": [{ "timestamp_sec": 5.5, "description": "…" }],
    "on_screen_text": "…",
    "visual_style": "…",
    "notable_moments": "…"
  }
}

To ask about a specific time point or dialogue segment, use video_ask. The plugin parses the time or keywords and re-extracts frames within that window to respond.

Use Cases and Considerations¶

Suitable for: Developers who need to understand local videos directly within DSH conversations—such as for review notes, meeting summary generation, tutorial segment locating, or time-stamped Q&A—and are willing to provide their own OpenAI-compatible vision/ASR endpoints.

Compatibility (as per README): Tested with @deepseek-ai/dsh-base + @deepseek-ai/dsh-web-app bundles; macOS / Linux have test records, Windows is untested.

Permissions and Security: DSH plugins run as trusted code within the host process. The community directory and DeepSeek / High-Flyer have no official affiliation; please review the source code and SECURITY.md before installation. This plugin reads local paths passed by the agent (via ffprobe/ffmpeg), executes ffmpeg/ffprobe from the PATH (using argv arrays only, not spawning a shell), and sends frame data to the configured visionBaseUrl. If ASR is enabled, it also sends audio to asrBaseUrl. Keys are only sent to your configured endpoints; the payload size for a single analysis is constrained by maxFrames and frameMaxWidth.

To uninstall, remove dsh-video-lens from dsh.profile.bundles, then run pnpm remove dsh-video-lens and reinstall the Profile.

Summary¶

dsh-video-lens chains “probing → scene-aware frame extraction → optional transcription → visual understanding” into a toolset callable by DSH, allowing pure text agents to perform structured analysis and time-anchored Q&A based on local videos. For more information, see the SkillHub directory page and the GitHub repository.