Introduction

Implementing desktop application control in DeepSeek Harness (DSH) usually requires writing scripts for specific applications or relying on fragile UI automation testing frameworks. The dsh-mac-cua plugin uses the native macOS Accessibility API to provide DSH with a general-purpose Computer Use solution. Instead of relying on predefined UI mappings, it uses a persistent JavaScript REPL to let agents directly read and manipulate desktop elements as code.

Plugin Overview

dsh-mac-cua is a DSH plugin maintained by Saunato. Its core goal is to enable agents to operate macOS applications like a human. It converts the complex system accessibility tree into readable text and provides a sky API for agents to call.

Installation and Configuration

After installing the plugin, you need to run the setup script to write the necessary absolute path configuration and grant system permissions.

  1. Install the plugin
    Use the following command to install the plugin:
    dsh plugin --profile web add dsh-cua
  1. Configure paths
    After the plugin is installed, run the setup script to ensure the paths are correctly written to the configuration file:
    dsh-cua-setup --write
    dsh-cua-setup

If you installed it manually or need to override the configuration, the `dsh-cua-setup` command can display or remove configuration lines.

  1. Grant permissions
    The plugin depends on system permissions, which must be enabled manually in “System Settings”:

    • Accessibility: Required. Path: System Settings → Privacy & Security → Accessibility → add and enable DSH Desktop.
    • Screen Recording: Optional. Only required when using screenshot features. Path: System Settings → Privacy & Security → Screen Recording → add and enable DSH Desktop.

    After enabling the permissions, restart DSH Desktop.

Core Features

The plugin provides a fixed and powerful toolset. All operations are implemented through the sky object.

  • Read state: Use sky.get_app_state to read the accessibility tree text for any application. After the first read, subsequent reads return incremental changes (diffs), reducing token cost. Screenshots can also be returned simultaneously.
  • Execute actions: Supports clicking an element or coordinates, typing text, pasting, key combinations, scrolling, dragging, selecting text, and invoking accessibility actions.
  • Image feedback: Screenshots that are read can be returned to the agent context as images.

Usage Examples

Agents write JavaScript code using the nodeRepl tool to interact with the sky API.

1. Read application state

var state = await sky.get_app_state({ app: "Finder" });
nodeRepl.write(state.text);

2. Operate an element and read the result
Click using the index obtained from the previous step, then read the state again:

await sky.click({ app: "Finder", element_index: 42 });
nodeRepl.write((await sky.get_app_state({ app: "Finder" })).text);

3. Read screenshots
Stream the screenshot data back into the conversation:

const fs = await import("node:fs/promises");
const { fileURLToPath } = await import("node:url");
var state = await sky.get_app_state({ app: "Finder" });
if (state.screenshot) {
  await nodeRepl.emitImage({
    bytes: await fs.readFile(fileURLToPath(state.screenshot.url)),
    mimeType: "image/png",
  });
}

API Reference

The following are the main methods of the sky object:

  • sky.get_app_state({ app, disableDiff?, screenshot? }): Retrieves application state and returns { text, screenshot }.
  • sky.list_apps(): Lists all accessible applications.
  • sky.get_screenshot(): Captures the current screen.
  • sky.click({ app, element_index?, x?, y?, mouse_button?, click_count? }): Clicks an element or coordinates.
  • sky.set_value({ app, element_index, value }): Sets a field value.
  • sky.select_text({ app, element_index, text, prefix?, suffix?, selection_type? }): Selects text.
  • sky.perform_secondary_action({ app, element_index, action }): Performs an accessibility action.
  • sky.type_text({ app, text }): Types text.
  • sky.press_key({ app, key }): Presses a key.
  • sky.paste({ app, text, format }): Pastes text and restores your previous clipboard content afterward.
  • sky.scroll({ app, element_index?, x?, y?, direction, pages? }): Scrolls the page.
  • sky.drag({ app, from_x, from_y, to_x, to_y }): Drags an object.

The app parameter supports display names (such as “Finder”), Bundle IDs (such as “com.apple.finder”), or full paths.

Notes

  1. Environment requirements: Requires macOS 14.4 or later (Apple Silicon or Intel), and Node.js 20 or later.
  2. Index validity: element_index is only valid for the element tree at the point when the snapshot was generated. Each get_app_state call creates a new index mapping; stale indexes are rejected and prompt you to read the state again.
  3. Input behavior: type_text waits for keyboard focus and reads the field’s echoed value to ensure the input is correct. Including \n in the text triggers the Return key.
  4. Clipboard restoration: The paste operation restores your previous clipboard content after pasting, avoiding disruption to your workflow.
  5. Code review: Although the plugin is under the MIT license, we recommend reviewing the source code before using it in production to ensure it meets your security and compliance requirements.

Summary

dsh-mac-cua provides DSH agents with a native macOS control solution based on the Accessibility API. Through the JavaScript REPL and the sky API, developers can build agents capable of handling complex desktop interactions, making it suitable for scenarios such as office automation and GUI automation testing.

  • Plugin directory: https://www.skillhub.cn/plugins/Saunato/dsh-mac-cua
  • Source repository: https://github.com/Saunato/dsh-mac-cua