Preface

The philosophy of DSH is “everything is a plugin”. In Windows desktop development, AI agents often need to know which window, UI element, or screen area the user is currently viewing. Manually describing the interface is costly, and letting the model directly guess UI states is not stable enough. Little Mouse Pointer addresses this problem: the user first selects the target with the mouse, the plugin then generates context via UI Automation, screenshots, and local OCR, and after confirmation, provides it to OpenAI Codex, DeepSeek Harness, or other compatible clients via MCP.

What is it

Little Mouse Pointer is a Windows desktop UI context picker for AI agents, maintained by Fish121380, and is MIT licensed. The repository address is:

https://github.com/Fish121380/auto-mouse

Its plugin directory is:

plugins/little-mouse-pointer

Simply put, it turns “mouse selection of a window, UI element, or screen area” into a confirmable MCP context for DSH or compatible clients to call.

Core Features

Below are the confirmed capabilities.

  • Windows desktop UI context picker.
  • Select window, UI element, or screen area with mouse.
  • Obtain context via UI Automation, screenshots, and local OCR.
  • After user confirmation, provide to OpenAI Codex, DeepSeek Harness, or other compatible clients via MCP.
  • Provide MCP tools:
  • ui_context_pick
  • ui_context_current
  • ui_context_clear
  • Provide installers conforming to Harness dsh.bundle specification.
  • Automatically locate the bundled Windows x64 exe and register @deepseek-ai/dsh-mcp-client.

Installation and Enablement

Harness Bundle Installation

If it is a DeepSeek Harness scenario, first execute the following command to install a fixed-tag bundle, then dump the configuration to check:

dsh plugin --profile demo add github:Fish121380/auto-mouse#v0.1.1
dsh --profile demo --dump-config

After the above steps, the bundle will locate the Windows x64 exe it carries and register @deepseek-ai/dsh-mcp-client. It only applies to Windows x64.

Codex Installation

For the Codex scenario, replace OWNER/REPOSITORY with the actual GitHub repository according to the README instructions:

codex plugin marketplace add OWNER/REPOSITORY --ref main
codex plugin add little-mouse-pointer@auto-mouse

If you only need to download the plugin directory, you can use the GitHub release zip, or install the plugins/little-mouse-pointer layer locally. Do not mistake the repository root directory as the direct plugin root directory.

Manual MCP stdio Integration

If manually integrating MCP stdio, you must start the executable directly:

plugins/little-mouse-pointer/app/MousePointer.Windows.exe

Do not use run.ps1 as an MCP stdio bridge. run.ps1 is only used for manually launching the GUI and self-test.

Typical Usage

The usage flow can be understood in the following steps:

  1. Enable the MCP plugin in Codex, DeepSeek Harness, or other compatible clients.
  2. Select window, UI element, or screen area with mouse.
  3. Plugin generates context via UI Automation, screenshots, and local OCR.
  4. After user confirmation, the client reads the context via MCP tools.

Corresponding MCP tools are:

ui_context_pick
ui_context_current
ui_context_clear

If the port needs to be adjusted, you can override it in the profile’s cordis.patch.yml, for example:

- replace:
    - id: little-mouse-pointer
      config:
        port: 49154

The default port is 49153. endpoint and MOUSE_POINTER_HTTP_PORT must be consistent.

Use Cases and Considerations

Suitable for scenarios in a Windows x64 environment where AI agents need to supplement “the user’s currently selected desktop UI context”. Note the following points before use:

  • Only for Windows x64.
  • No need to install .NET, the release package is self-contained.
  • Keep all files in plugins/little-mouse-pointer/app.
  • Close old Little Mouse Pointer instances before running Codex/Harness.
  • Default port is 49153, endpoint and MOUSE_POINTER_HTTP_PORT must be consistent.
  • Do not treat run.ps1 as an MCP stdio bridge.
  • The plugin runs with the current dsh process permissions, reading UI Automation, screenshots, and local OCR results; it is recommended to check the source code, configuration, and MIT license before installation.
  • DeepSeek Harness manual Cordis configuration can refer to:
plugins/little-mouse-pointer/docs/deepseek-harness.md

Links

  • GitHub repository:
https://github.com/Fish121380/auto-mouse
  • Plugin directory:
plugins/little-mouse-pointer
  • Beginner’s guide:
plugins/little-mouse-pointer/docs/getting-started.md
  • DeepSeek Harness configuration:
plugins/little-mouse-pointer/docs/deepseek-harness.md