Foreword

When building agents on Windows, developers often face inconsistencies between desktop application control and browser control. Existing solutions often lack a unified recovery protocol and result definitions. win-pilot, as an agent-agnostic plugin, provides a unified computer tool for Windows computer-use scenarios, covering desktop applications and isolated Chromium sessions.

What Is This

win-pilot is an Agent Plugin package designed to provide a consistent operating interface for Windows agents. It supports MCP (Model Context Protocol), CLI, Skills, and native host adapters, ensuring consistent behavior across plugins, CLIs, and native integrations. This plugin is maintained by JeremyWangCY and released under the MIT License.

Core Features

The plugin provides the following core capabilities:

  • Agent Plugin package: Bundles the MCP server and operating system skills in .codex-plugin/plugin.json.
  • Native Windows control: Provides native control through UI Automation, Win32 input/window management, and window-scoped screenshots.
  • Background-first actions: Supports background capture and input paths, avoiding unnecessary window focus stealing.
  • Isolated browser control: Launches Edge sessions and controls targets by binding through CDP (Chrome DevTools Protocol).
  • Recoverable protocol: Uses stable ok, outcome, error_code fields and post-action observations to support handling unknown outcomes.
  • Universal fallbacks: Provides local stdio MCP for tool-aware agents, and JSON CLI plus Skills for shell-capable agents.
  • Schema portability: The tool schemas have been validated for consumers in the OpenAI and Gemini families.
  • DSH provider lifecycle: Requires DSH 0.2.0-rc.2+ to support Config forms, and occupies the dedicated computerUse provider slot.

Installation and Enablement

Before installing, make sure the system requirements are met: Windows 10/11 x64, Node.js 22.12 or later, and Windows PowerShell 5.1.

The installation process is as follows:

  1. Install the latest version globally using npm:
npm install -g win-pilot@latest
  1. After installation, run the diagnostic command to verify the environment:
win-pilot doctor --probe

Installing the plugin does not bypass host approvals, sandbox rules, or task authorization.

Typical Usage

For shell agents on non-plugin hosts, you can perform operations using the JSON CLI interface.

  1. List system applications:
win-pilot request --payload '{"action":"list_apps"}' --
  1. Get the state of a specified window, including text content:
win-pilot request --payload '{"action":"get_window_state","hwnd":12345,"include_text":true}' --
  1. Execute a click action, locating the target by snapshot ID and element index:
win-pilot request --payload '{"action":"click","hwnd":12345,"element_index":7,"snapshot_id":"<snapshot>"}' --

Applicable Scenarios and Considerations

  • Permissions and privacy: The plugin runs with the permissions of the current Windows user. It does not include telemetry and does not upload screenshots. Browser navigation only initiates ordinary network requests to the requested sites.
  • Sandbox limitations: When used in sandboxed environments, configure win-pilot mcp in a persistent plugin or MCP supervisor, and do not detach the process from temporary commands.
  • DSH configuration: After installing the plugin, the DSH configuration file must be restarted separately for changes to take effect.

Summary

win-pilot provides a standardized Windows computer-use protocol, resolving the fragmentation between desktop and browser control for agents. Through a unified MCP interface and a recoverable protocol, it lowers the barrier for developers migrating across different agent hosts.

Plugin Directory
GitHub Repository