Introduction¶

The philosophy of DeepSeek Harness (DSH) is “everything is a plugin,” but the base model on the chassis is purely text-based: there is no vision, and no keyboard or mouse. If you ask the model to “open Notepad, write a line of text, and save,” it cannot see if Notepad is open or click the menu. In the past, building such desktop automation usually required constructing a pipeline yourself: screenshots, visual recognition, and then writing glue code to drive the mouse and keyboard. dsh-computer-use encapsulates this pipeline into a DSH plugin, allowing the model to run the entire closed loop by calling just a few tools.

What is it¶

dsh-computer-use is maintained by JasonWei04 and is licensed under MIT. One-sentence positioning: Screenshot/Screen recording → Xiaomi MIMO visual analysis → Real mouse/keyboard/window/program control → Autonomous loop until task completion.

It operates on a real Windows desktop, not a simulator: mouse movement, clicks, drags, scrolling, keyboard input, launching programs, and activating windows all happen on the real system. Visual capabilities come from the Xiaomi MIMO visual model (OpenAI-compatible endpoint), responsible for “seeing” screen content and element coordinates, giving the text-only model a pair of eyes.

Workflow¶

The workflow is fixed in three steps:

1. Observe: computer_screenshot saves a screenshot as PNG, or computer_record extracts frames from screen recording (JPEG);
2. Analyze: The screenshot or recording frames are handed to MIMO, returning a screen description and coordinates;
3. Execute: computer_action performs a single real-world operation; or hand it to computer_use to run an autonomous loop of “Observe → Visual Decision → Execute → Observe again” until the task is complete.

The underlying Win32 calls are handled by a C# helper: the source code is embedded in index.js, automatically written to the workspace on first use, and compiled using the system’s built-in csc.exe without needing to install any runtime. Temporary screenshots, recording frames, and visual request files are automatically cleaned up after analysis or the loop ends, leaving no garbage files.

Tools Overview¶

There is a simple principle for selection: use computer_screenshot for static scenes (checking layout, confirming interface); use computer_record for dynamic processes (e.g., scrolling down in WeChat to find Moments), capturing multiple frames from one recording to see clearly and avoid repeated screenshots. All three visual tools support the model parameter to specify a model per call (e.g., mimo-v2.5, mimo-v2.5-pro).

Installation and Activation¶

Prerequisites¶

• DSH must be the web version running on Windows; the system must be Windows 10/11 (the C# helper relies on the system’s built-in .NET Framework csc.exe and Win32 API);
• You must first install @linxin666/dsh-web-ui-all (the dsh-web-ui full suite): this plugin is developed based on the framework capabilities of defineTool, installSettingsSection, systemPrompt.section, mountOnce single-instance guard, etc.; the full suite must be installed, otherwise the plugin will fail to load;
• MIMO_API_KEY is optional but strongly recommended (for visual analysis).

Installation Steps¶

Installation is manual; there is no one-click command in the marketplace; it consists of three steps.

1. Copy the entire dsh-computer-use/ folder to the profile’s plugins/ directory:

C:\Users\<your_username>\.dsh\profiles\web\plugins\dsh-computer-use\

2. Open the profile’s patch file C:\Users\<your_username>\.dsh\profiles\web\cordis.patch.yml and append to the end of the plugins: list:

- insert:
    - id: computer-use
      name: './plugins/dsh-computer-use/index.js'

3. Restart DSH. Static plugins are loaded at startup and require no approval.

Verification: Send the model “Check computer use plugin status”; you should receive an assistant ready message.

Uninstallation is just as simple: delete the three lines of computer-use insert in cordis.patch.yml, delete the plugins/dsh-computer-use/ folder, and restart DSH.

Configuring MIMO Vision¶

Set the Windows environment variable (user-level is sufficient):

MIMO_API_KEY = sk-your_key_here

The key is read only from the environment variable and is never written to the plugin files; the plugin automatically re-detects every 30 seconds. Without a key, the screenshot and recording tools are still usable, but visual decision-making for analyze and computer_use is unavailable. After setting the environment variable, wait 30 seconds for automatic re-detection, or restart DSH.

After installation, you can adjust configuration items such as baseURL, model, apiKeyEnv, maxOutputTokens, timeoutMs, helperExe, enabled, and announceToAgent in the dsh-computer-use section of the DSH settings interface.

Typical Usage¶

Open Notepad and Save¶

Taking “open Notepad, type Hello, DSH! and save” as an example, the four-step call is:

computer_action(action="run", command="notepad")
computer_screenshot(analyze=true)              # Screenshot to confirm Notepad is open
computer_action(action="type", text="Hello, DSH!")
computer_action(action="keys", keys="^{s}")    # Ctrl+S to save

Send Message in WeChat¶

Sending a message to the current chat can be done in one step:

computer_action(action="sendchat", text="Meeting at 8 PM tonight")

sendchat is only for WeChat (Weixin/WeChat process), with the input box position estimated based on 62% of the WeChat window width and 55px from the bottom edge.

Let computer_use Complete It Autonomously¶

There is no need to choreograph the entire task step-by-step; just describe it directly:

computer_use(task="Open PyCharm, create test.py in the opened folder with content print('hi')")

The observe parameter controls the observation method, supporting auto, screenshot, and record; the default is auto—use screenshots for static interfaces and switch to recording when observing dynamic processes:

computer_use(task="Scroll down in WeChat to find a Moment from three days ago and screenshot it", observe="auto")

Use Recording for Dynamic Processes¶

For processes like scrolling, sliding, animations, and loading, it’s better to record a segment rather than take repeated screenshots:

computer_record(seconds=4, fps=2)

Record for 4 seconds and extract 8 frames, handing them to MIMO all at once to see the whole process.

Specify Visual Model Per Call¶

computer_screenshot(analyze=true, model="mimo-v2.5-pro")
computer_use(task="...", model="mimo-v2.5")

All three tools accept the model parameter, allowing you to switch models step-by-step within the same session based on task difficulty.

Suitable Scenarios and Notes¶

Who is it for: Users running the DSH web version on Windows who want the model to directly complete GUI tasks—launching programs, clicking and typing in interfaces, scrolling to find things, sending messages in WeChat, or handing over the entire workflow to computer_use for autonomous execution.

A few notes:

• Platform and dependencies: DSH must be the web version running on Windows, and the system must be Windows 10/11; the @linxin666/dsh-web-ui-all full suite must be installed first, otherwise the plugin will not load.
• Coordinates are estimates: Coordinates provided by the visual model may not be accurate; for complex interfaces, it is recommended to perform small steps and gradually approach the target; please confirm manually for high-risk operations such as deletion, submission, and payment.
• Real desktop: The plugin operates on a real desktop and consumes real resources; please confirm your intent before letting the model execute.
• Scope of sendchat: Only supports the WeChat process, with the input box position estimated based on fixed proportions.
• Permissions and review: The plugin runs with the permissions of the current DSH process and has real control over the desktop; please read the plugin source code before installation to ensure the license (MIT) is compatible with your environment before enabling it.

Conclusion¶

dsh-computer-use packs the entire pipeline of “screenshot/recording → MIMO visual analysis → real keyboard/mouse operations → autonomous loop” into a set of DSH tools, enabling the text-only model to truly see and move the Windows desktop. If you can accept its boundary conditions (Windows 10/11, manual installation, coordinate estimation), it is worth a try.

• GitHub: https://github.com/JasonWei04/dsh-computer-use
• Community Plugin Directory: https://www.skillhub.cn/plugins/JasonWei04/dsh-computer-use (Community independent site, no official affiliation with DeepSeek or HF)