Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license
This commit is contained in:
@@ -0,0 +1,620 @@
|
||||
---
|
||||
title: Computer Use
|
||||
sidebar_position: 16
|
||||
---
|
||||
|
||||
# Computer Use
|
||||
|
||||
Hermes Agent can drive your desktop — clicking, typing, scrolling,
|
||||
dragging — in the **background** on **macOS, Windows, and Linux**. Your
|
||||
cursor doesn't move, keyboard focus doesn't change, and your virtual
|
||||
desktops / Spaces don't switch on you. You and the agent co-work on the
|
||||
same machine.
|
||||
|
||||
Unlike most computer-use integrations, this works with **any tool-capable
|
||||
model** — Claude, GPT, Gemini, or an open model on a local
|
||||
OpenAI-compatible endpoint. There's no Anthropic-native schema to worry
|
||||
about.
|
||||
|
||||
## How it works
|
||||
|
||||
The built-in `computer_use` toolset is the recommended Hermes integration. It
|
||||
speaks MCP over stdio to
|
||||
[`cua-driver`](https://github.com/trycua/cua), an open-source background
|
||||
computer-use driver. Each platform uses the appropriate accessibility +
|
||||
input stack under the hood:
|
||||
|
||||
| Platform | Accessibility tree | Input dispatch |
|
||||
|---|---|---|
|
||||
| macOS | AX (private SkyLight SPIs) | `SLPSPostEventRecordTo` — pid-scoped, no cursor warp |
|
||||
| Windows | UIAutomation | `SendInput` + `PostMessage` — no focus steal |
|
||||
| Linux | AT-SPI (X11 + Wayland) | XTest (X11) / virtual-keyboard (Wayland) |
|
||||
|
||||
The result is the same on every platform: the agent can read the
|
||||
accessibility tree of any visible window AND post synthesized events
|
||||
without bringing it to front, switching virtual desktops, or moving the
|
||||
real OS cursor.
|
||||
|
||||
For the underlying contract — *why* background mode matters, the
|
||||
no-foreground invariant, click-dispatch internals — see
|
||||
**[cua.ai/docs/explanation/the-no-foreground-contract](https://cua.ai/docs/explanation/the-no-foreground-contract)**.
|
||||
|
||||
## Enabling
|
||||
|
||||
**Fresh installs already have the driver.** The Hermes installer
|
||||
(`install.sh` / `install.ps1`) pre-installs `cua-driver` (best-effort;
|
||||
pass `--skip-computer-use` / `-SkipComputerUse` to opt out), so enabling
|
||||
Computer Use is just a config flip:
|
||||
|
||||
- **`hermes tools`** → pick `🖱️ Computer Use` — installs the driver
|
||||
automatically if it's still missing.
|
||||
- **Dashboard / desktop app** → toggle the Computer Use toolset — if the
|
||||
driver is missing, the toggle kicks off the install in the background
|
||||
automatically (watch progress in the toolset panel).
|
||||
|
||||
**Manual fallback (older installs, skipped installer step):**
|
||||
|
||||
```
|
||||
hermes computer-use install
|
||||
```
|
||||
|
||||
This fetches and runs the upstream cua-driver installer — `install.sh`
|
||||
on macOS/Linux, `install.ps1` on Windows. Use `hermes computer-use
|
||||
status` to verify the install.
|
||||
|
||||
Already have cua-driver? Hermes reuses it when it supports the 0.20 runtime
|
||||
contract. During setup, toolset enablement, `hermes update`, and the first
|
||||
`computer_use` call of a session, Hermes checks the local version and
|
||||
manifest. It repairs an old or incomplete standard installation through
|
||||
the upstream installer (at most once per session at runtime). A binary
|
||||
selected with `HERMES_CUA_DRIVER_CMD` stays
|
||||
under your control, so Hermes reports the incompatibility and leaves it
|
||||
unchanged.
|
||||
|
||||
If you install Cua Driver first, `cua-driver skills install` installs Cua's
|
||||
skill pack under `~/.cua-driver/skills/cua-driver`. Hermes autodetection is a
|
||||
planned cua-driver follow-up, so currently point Hermes at that directory or
|
||||
symlink it into your skill space. You can also register raw Cua MCP tools as a
|
||||
custom MCP server, but that is an alternative for users who need the low-level
|
||||
interface. The built-in toolset provides Hermes actions, configuration,
|
||||
approvals, and diagnostics.
|
||||
|
||||
After installing, regardless of which path you took, grant the
|
||||
platform-appropriate prereqs:
|
||||
|
||||
| Platform | Prereqs |
|
||||
|---|---|
|
||||
| **macOS** | System Settings → Privacy & Security → **Accessibility** + **Screen Recording**. Grant the identity named by `hermes computer-use doctor`. Standard mode uses CuaDriver.app; bounded and unrestricted modes use the Hermes host identity. |
|
||||
| **Windows** | None at install time. If you're driving over SSH (not RDP / console), you need the autostart pattern — see [cua.ai/docs/how-to-guides/driver/windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh) for the Session 0 ↔ Session 1+ proxy. |
|
||||
| **Linux** | A reachable display server: `DISPLAY` set for X11, or `XDG_SESSION_TYPE=wayland`. Wayland sessions need an XWayland bridge for capture. AT-SPI must be on (default on GNOME/KDE/Xfce). |
|
||||
|
||||
Then start a session with the toolset enabled:
|
||||
|
||||
```
|
||||
hermes -t computer_use chat
|
||||
```
|
||||
|
||||
or add `computer_use` to your enabled toolsets in `~/.hermes/config.yaml`.
|
||||
|
||||
## Permission modes and logged-in browser profiles
|
||||
|
||||
Hermes maps its existing approval UX onto cua-driver's immutable runtime
|
||||
modes. Permission mode and capability manifest approval are launch settings.
|
||||
They cannot change after the runtime starts:
|
||||
|
||||
| Hermes session | cua-driver mode | Human intervention |
|
||||
|---|---|---|
|
||||
| Manual or smart approvals (default) | `standard` | Normal Hermes approvals; Cua stops at its protected boundary |
|
||||
| `computer_use.permission_mode: bounded` + reviewed manifest | private `bounded` daemon | You review and approve the capability manifest once, at launch |
|
||||
| `--yolo`, `/yolo`, or `approvals.mode: off` | private `unrestricted` daemon | One explicit Hermes risk acceptance; no runtime Cua prompts |
|
||||
|
||||
Browser work — including pages in a signed-in profile — goes through the
|
||||
`browser` toolset (`browser_exec`), not `computer_use`. The former
|
||||
`computer_use.grant_existing_profile` opt-in was removed along with the typed
|
||||
browser route; a leftover key in config.yaml is ignored.
|
||||
|
||||
### Bounded mode for repeatable automation
|
||||
|
||||
For recurring browser automation (cron jobs, scheduled research against an
|
||||
authenticated app), `bounded` mode uses a capability manifest you review once:
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
computer_use:
|
||||
permission_mode: bounded
|
||||
capability_manifest: ~/.hermes/cua-manifest.yaml
|
||||
```
|
||||
|
||||
The manifest names the apps, browser profile kinds, allowed origins, and
|
||||
typed tools the session may use (see the
|
||||
[cua-driver permission modes reference](https://cua.ai/docs/reference/cua-driver/permission-modes)
|
||||
for the format). Hermes launches a private runtime with
|
||||
`--capability-manifest ... --approve-capability-manifest`; anything outside
|
||||
the manifest fails closed inside cua-driver. A missing or unreadable manifest
|
||||
fails loudly at session start rather than silently downgrading. Session YOLO
|
||||
still overrides bounded for that one session.
|
||||
|
||||
On macOS, private-session daemons launch through the installed
|
||||
`CuaDriver.app` bundle (so permission grants attribute to the driver's own
|
||||
identity instead of resetting with every Hermes build), and Hermes verifies
|
||||
the bundle's code signature — exact `com.trycua.driver` identifier and the
|
||||
official signing team — before launching it. If you build cua-driver from
|
||||
source (unsigned), opt in explicitly:
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
computer_use:
|
||||
allow_unsigned_driver: true # local driver development only
|
||||
```
|
||||
|
||||
Each MCP transport owns a private lifecycle session inside its runtime. A
|
||||
public session name is only a label for cursor identity and session-scoped
|
||||
state. It does not select, share, or keep a runtime alive. Turning `/yolo` off,
|
||||
resetting or closing the Hermes session, cancellation cleanup, or process exit
|
||||
closes that transport session. Hermes also stops private runtimes that it
|
||||
launched for bounded or unrestricted access. One Hermes
|
||||
conversation cannot change another runtime's mode or grants. Bounded and
|
||||
unrestricted modes use a private
|
||||
embedded service under the Hermes host identity.
|
||||
|
||||
`smart` approval remains `standard`: an LLM classification cannot stand in for
|
||||
a reviewed manifest.
|
||||
|
||||
<div class="alert alert--warning">
|
||||
|
||||
YOLO/unrestricted mode does not protect against prompt injection or unintended
|
||||
input. Use it only in a disposable VM or with accounts and data whose full
|
||||
compromise you accept.
|
||||
|
||||
</div>
|
||||
|
||||
## `hermes computer-use doctor` — your first triage stop
|
||||
|
||||
`hermes computer-use doctor` runs cua-driver's structured
|
||||
`health_report` MCP tool and prints a per-check matrix. It's the single
|
||||
fastest way to find out *why* an action isn't working.
|
||||
|
||||
```
|
||||
$ hermes computer-use doctor
|
||||
⚠️ cua-driver VERSION on darwin: degraded
|
||||
✅ binary_version: cua-driver VERSION
|
||||
✅ platform_supported: macOS 26.4.1 (arm64)
|
||||
✅ session_active: MCP session is active.
|
||||
❌ bundle_identity: Process has no CFBundleIdentifier.
|
||||
→ Run the binary inside CuaDriver.app so TCC grants attribute correctly.
|
||||
✅ tcc_accessibility: Accessibility is granted.
|
||||
✅ tcc_screen_recording: Screen Recording is granted.
|
||||
✅ ax_capability: AX is trusted and reachable.
|
||||
✅ screen_capture_capability: ScreenCaptureKit reachable; 1 display(s) shareable.
|
||||
```
|
||||
|
||||
- **Exit code 0** when overall is `ok` — everything's wired up.
|
||||
- **Exit code 1** when `degraded` or `failed` — at least one check failed; the hint on each failure tells you what to fix.
|
||||
- **Exit code 2** when the cua-driver binary itself isn't reachable.
|
||||
|
||||
Useful flags:
|
||||
|
||||
- `--include CHECK` — run only the listed checks (repeat for multiple)
|
||||
- `--skip CHECK` — skip a check (wins over `--include`)
|
||||
- `--json` — emit the raw structured payload, same shape as the
|
||||
`tools/call health_report` MCP response
|
||||
|
||||
The check matrix is platform-aware: `bundle_identity` / `tcc_*` are
|
||||
`skip` on Windows + Linux because those concepts don't apply.
|
||||
`ax_capability` checks AX on macOS, UIA on Windows, AT-SPI on Linux —
|
||||
each with the right diagnostic hint when it can't reach.
|
||||
|
||||
## The agent cursor and sessions
|
||||
|
||||
When the agent acts, you'll see a **tinted overlay cursor** glide
|
||||
across the screen to where each click / type / scroll lands. The real
|
||||
OS cursor never moves. The overlay shows where the agent is acting. Each
|
||||
Hermes run declares a public cua-driver **session name** (something like
|
||||
`hermes-3a7b9c14d2e8`). The name labels cursor identity and related state, so
|
||||
concurrent runs and subagents get distinct cursors. The MCP transport owns the
|
||||
private lifecycle session inside the runtime; the public name does not.
|
||||
|
||||
The overlay cursor is cosmetic — captures, clicks, and typing all work
|
||||
without it. Hermes disables it automatically where it is a known failure
|
||||
mode: macOS (idle CPU burn), headless Linux / WSL2 / containers, and
|
||||
**Linux X11 desktops** (the overlay is a fullscreen always-on-top window
|
||||
that can get stuck over every workspace after an unclean session end,
|
||||
wedging desktop input). Linux Wayland and Windows keep the overlay. Set
|
||||
`computer_use.no_overlay: false` in `config.yaml` to force the cursor on
|
||||
(or `true` to force it off) on any platform.
|
||||
|
||||
Tune the cursor with `cua-driver`'s CLI flags or the runtime
|
||||
`set_agent_cursor_style` MCP tool — see
|
||||
[cua.ai/docs/how-to-guides/driver/personalize-cursor](https://cua.ai/docs/how-to-guides/driver/personalize-cursor)
|
||||
for the full menu (built-in `arrow` vs `teardrop` silhouette, custom
|
||||
SVG / PNG / ICO via `--cursor-icon`, runtime gradient colors, bloom
|
||||
halo).
|
||||
|
||||
## Going deeper — the cua-driver skill pack
|
||||
|
||||
Hermes keeps its wrapper skill (`skills/autonomous-ai-agents/computer-use/SKILL.md`)
|
||||
focused on the Hermes-side `computer_use` workflow and action vocabulary. For
|
||||
platform details, recording semantics, browser page interaction, and other
|
||||
deep Cua behavior, install the skill pack that the cua-driver team ships and
|
||||
maintains directly:
|
||||
|
||||
```
|
||||
cua-driver skills install
|
||||
```
|
||||
|
||||
The command installs the pack under `~/.cua-driver/skills/cua-driver`. Hermes
|
||||
autodetection is a planned cua-driver follow-up, so currently point Hermes at
|
||||
that directory or symlink it into your skill space. The wrapper remains the
|
||||
workflow layer and points to Cua's installed skill for driver behavior. The
|
||||
pack contains:
|
||||
|
||||
| File | Topic |
|
||||
|---|---|
|
||||
| `SKILL.md` | The cross-platform core (snapshot invariant, no-foreground contract, click dispatch, AX-tree mechanics) |
|
||||
| `MACOS.md` | macOS specifics: no-foreground contract, AXMenuBar navigation, SkyLight click dispatch, Apple Events JS bridge |
|
||||
| `WINDOWS.md` | Windows specifics: UIA tree, UWP / `ApplicationFrameHost` hosting, Session 0 isolation, autostart pattern |
|
||||
| `LINUX.md` | Linux specifics: AT-SPI tree, X11 / Wayland, terminal-emulator detection |
|
||||
| `RECORDING.md` | Trajectory + video recording semantics |
|
||||
| `WEB_APPS.md` | Browser-page interaction tips |
|
||||
| `TESTS.md` | Replay-by-trajectory workflow |
|
||||
|
||||
These are **platform deep dives, not duplicates of the Hermes skill** —
|
||||
when an agent reports "on Windows, my click landed on the wrong
|
||||
element," it reads `WINDOWS.md` for the UIA / UWP context that
|
||||
explains why and what to do differently.
|
||||
|
||||
`cua-driver skills status` shows what's installed and which agent
|
||||
harnesses it's linked into. Today the autodetect list covers Claude
|
||||
Code, Codex, OpenCode, OpenClaw, and Antigravity; **Hermes
|
||||
autodetection is planned as a follow-up in `trycua/cua`** — until
|
||||
then, run `cua-driver skills install` once and point your harness at
|
||||
the resulting `~/.cua-driver/skills/cua-driver` directory (or symlink
|
||||
it into your usual skill space).
|
||||
|
||||
## Quick example
|
||||
|
||||
User prompt: *"Find my latest email from Stripe and summarise what they want me to do."*
|
||||
|
||||
The agent's plan (this is the same shape on macOS / Windows / Linux —
|
||||
the model substitutes the platform's idiomatic shortcut and app name):
|
||||
|
||||
1. `computer_use(action="capture", mode="som", app="Mail")` — gets a
|
||||
screenshot of the email app with every sidebar item, toolbar button,
|
||||
and message row numbered.
|
||||
2. `computer_use(action="click", element=14)` — clicks the search field.
|
||||
3. `computer_use(action="type", text="from:stripe")`
|
||||
4. `computer_use(action="key", keys="return", capture_after=True)` —
|
||||
submit and get the new screenshot.
|
||||
5. Click the top result, read the body, summarise.
|
||||
|
||||
During all of this, your cursor stays wherever you left it and the email
|
||||
app never comes to front.
|
||||
|
||||
## Receiving the actual screenshot
|
||||
|
||||
Screenshots taken during computer control are normally internal — they exist
|
||||
so the model can see the screen, and the agent replies in text. But every
|
||||
image capture also saves a bounded, shareable copy under Hermes' image cache
|
||||
and reports its path, so on attachment-capable surfaces (Telegram, Discord,
|
||||
Desktop, and other gateway platforms) you can simply ask:
|
||||
|
||||
> *"Send me a screenshot of my screen."*
|
||||
|
||||
and the agent delivers the real image as a native attachment, not just a
|
||||
description. On the CLI there is no attachment channel, so the agent gives
|
||||
you the saved file's path instead.
|
||||
|
||||
Only the 20 most recent capture files are kept, and screenshots are never
|
||||
sent automatically — only when you ask for one.
|
||||
|
||||
### Whole screen vs. desktop surface
|
||||
|
||||
"Screenshot my screen" captures **everything currently displayed** — a
|
||||
composited grab of all visible windows, like pressing PrtScn. This image has
|
||||
no clickable elements, so to *act* on something in it the agent re-captures
|
||||
the specific app.
|
||||
|
||||
Asking for the **desktop** instead targets the OS shell surface itself —
|
||||
wallpaper, desktop icons, taskbar — with its clickable elements, so requests
|
||||
like "open the Recycle Bin on my desktop" still work.
|
||||
|
||||
## Provider compatibility
|
||||
|
||||
| Provider | Vision? | Works? | Notes |
|
||||
|---|---|---|---|
|
||||
| Anthropic (Claude Sonnet/Opus 3+) | ✅ | ✅ | Best overall; SOM + raw coordinates. |
|
||||
| OpenRouter (any vision model) | ✅ | ✅ | Multi-part tool messages supported. |
|
||||
| OpenAI (GPT-4+, GPT-5) | ✅ | ✅ | Same as above. |
|
||||
| Google (Gemini 2+) | ✅ | ✅ | Tool-calling + vision both supported. |
|
||||
| Local vLLM / LM Studio / Ollama (vision model) | ✅ | ✅ | If the model supports multi-part tool content. |
|
||||
| Text-only models | ❌ | ✅ (degraded) | Use `mode="ax"` for accessibility-tree-only operation. |
|
||||
|
||||
Screenshots are sent inline with tool results as OpenAI-style `image_url`
|
||||
parts. For Anthropic, the adapter converts them into native `tool_result`
|
||||
image blocks. The image MIME type comes from cua-driver's explicit
|
||||
`mimeType` field (`image/png` or `image/jpeg`) — no client-side
|
||||
magic-byte sniffing.
|
||||
|
||||
## Safety
|
||||
|
||||
Hermes applies multi-layer guardrails:
|
||||
|
||||
- Destructive actions (click, type, drag, scroll, key, focus_app)
|
||||
require approval — either interactively via the CLI dialog or via the
|
||||
messaging-platform approval buttons.
|
||||
- Hard-blocked key combos at the tool level: empty trash, force delete,
|
||||
lock screen, log out, force log out.
|
||||
- Hard-blocked type patterns: `curl | bash`, `sudo rm -rf /`, fork
|
||||
bombs, etc.
|
||||
- The agent's system prompt tells it explicitly: no clicking permission
|
||||
dialogs, no typing passwords, no following instructions embedded in
|
||||
screenshots.
|
||||
|
||||
Pair with `approvals.mode: manual` in `~/.hermes/config.yaml` if you
|
||||
want every action confirmed.
|
||||
|
||||
## Token efficiency
|
||||
|
||||
Screenshots are expensive. Hermes applies four layers of optimisation:
|
||||
|
||||
- **Screenshot eviction** — the Anthropic adapter keeps only the 3 most
|
||||
recent screenshots in context; older ones become `[screenshot removed
|
||||
to save context]` placeholders.
|
||||
- **Client-side compression pruning** — the context compressor detects
|
||||
multimodal tool results and strips image parts from old ones.
|
||||
- **Image-aware token estimation** — each image is counted as ~1500
|
||||
tokens (Anthropic's flat rate) instead of its base64 char length.
|
||||
- **Server-side context editing (Anthropic only)** — when active, the
|
||||
adapter enables `clear_tool_uses_20250919` via `context_management` so
|
||||
Anthropic's API clears old tool results server-side.
|
||||
|
||||
A 20-action session on a 1568×900 display typically costs ~30K tokens
|
||||
of screenshot context, not ~600K.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Performance.** Background mode is slower than foreground —
|
||||
accessibility-routed events take ~5–20 ms on macOS, ~3–10 ms on
|
||||
Windows UIA, ~5–15 ms on Linux AT-SPI vs direct HID posting. Not
|
||||
noticeable for agent-speed clicking; noticeable if you try to record
|
||||
a speed-run.
|
||||
- **No keyboard password entry.** `type` has hard-block patterns on
|
||||
command-shell payloads; for passwords, use the system's autofill
|
||||
(macOS Keychain / Windows Credential Manager / GNOME Keyring /
|
||||
KWallet).
|
||||
- **Some apps don't expose an accessibility tree.** Modern UWP apps on
|
||||
Windows, Electron < 28 on Linux, and a few macOS apps with custom
|
||||
drawing (Logic, Final Cut, some games) have sparse or empty AX trees.
|
||||
Fall back to pixel coordinates if the tree is empty — or skip the
|
||||
task entirely.
|
||||
- **Windows: elevated (admin) windows can't be driven from a normal
|
||||
agent.** Windows UIPI (User Interface Privilege Isolation) enforces
|
||||
integrity-level boundaries: a Medium-integrity process (the default
|
||||
Hermes agent) cannot enumerate the UIA tree of, or inject mouse input
|
||||
into, a window owned by a High-integrity (Administrator) process.
|
||||
Symptom: `capture(mode='som')` returns 0 elements and `click(...)`
|
||||
reports success while doing nothing, even though the screenshot
|
||||
renders fine (GDI capture sits below the integrity check). Keyboard
|
||||
events partially bypass UIPI, so Tab / Enter can still navigate an
|
||||
elevated dialog. This is an OS constraint, not a cua-driver bug — it
|
||||
affects every Windows automation stack. To drive elevated windows,
|
||||
run the Hermes agent itself at High integrity (launch from an
|
||||
elevated terminal); otherwise target non-elevated windows.
|
||||
- **Platform-specific deployment gotchas:**
|
||||
- **macOS** uses private SkyLight SPIs. Apple can change them in any
|
||||
OS update. Hermes warns when the installed cua-driver is older than
|
||||
the version it was tested against.
|
||||
- **Windows** SSH sessions run in **Session 0**, which has no
|
||||
interactive desktop. Drive Hermes from inside the RDP / console
|
||||
session, or set up cua-driver's autostart Scheduled Task —
|
||||
[windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh)
|
||||
has the recipe.
|
||||
- **Linux** requires a reachable display server. Headless servers
|
||||
need Xvfb (`Xvfb :99 -screen 0 1920x1080x24`) before
|
||||
`computer_use` can capture or inject events. Pure Wayland sessions
|
||||
need an XWayland bridge for screen capture (cua-driver's Wayland
|
||||
inject path handles input independently).
|
||||
|
||||
For cross-platform GUI automation without the desktop overhead (and
|
||||
without TCC / Session 0 / X11 setup), the `browser` toolset uses a
|
||||
real headless Chromium and is the right answer for web-only tasks.
|
||||
|
||||
## Configuration
|
||||
|
||||
Permission mode and manifest (see
|
||||
[Permission modes](#permission-modes-and-logged-in-browser-profiles) above):
|
||||
|
||||
```yaml
|
||||
computer_use:
|
||||
permission_mode: standard # standard (default) | bounded
|
||||
capability_manifest: "" # capability manifest path, required for bounded
|
||||
```
|
||||
|
||||
On Linux, native Wayland support remains an explicit opt-in. Hermes passes the
|
||||
opt-in to every cua-driver process, including gateway sessions, only when that
|
||||
process also has `WAYLAND_DISPLAY`:
|
||||
|
||||
```yaml
|
||||
computer_use:
|
||||
native_wayland: true
|
||||
```
|
||||
|
||||
Restart a running gateway after changing this setting.
|
||||
|
||||
Override the driver binary path (tests / CI / local builds):
|
||||
|
||||
```
|
||||
HERMES_CUA_DRIVER_CMD=/path/to/your/cua-driver
|
||||
```
|
||||
|
||||
Swap the backend entirely (for testing):
|
||||
|
||||
```
|
||||
HERMES_COMPUTER_USE_BACKEND=noop # records calls, no side effects
|
||||
```
|
||||
|
||||
### Telemetry
|
||||
|
||||
cua-driver ships with anonymous usage telemetry (PostHog) enabled by default
|
||||
upstream. **Hermes disables it for you** — on every cua-driver invocation
|
||||
(the MCP backend, `status`, `doctor`, and install) Hermes sets
|
||||
`CUA_DRIVER_RS_TELEMETRY_ENABLED=0` in the driver's environment.
|
||||
|
||||
To opt back in (let cua-driver use its own default and send telemetry), set
|
||||
this in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
computer_use:
|
||||
cua_telemetry: true # default: false (telemetry off)
|
||||
```
|
||||
|
||||
When it's on, `hermes computer-use doctor` reports `telemetry: enabled`;
|
||||
when off (the default), it reports `telemetry: disabled via
|
||||
CUA_DRIVER_RS_TELEMETRY_ENABLED`.
|
||||
|
||||
## Testing against a local cua-driver build
|
||||
|
||||
When you're developing cua-driver itself — or want to test an
|
||||
unreleased fix — point Hermes at a binary you built from source instead
|
||||
of the published release. Hermes resolves the driver with
|
||||
`shutil.which("cua-driver")` and **does not enforce
|
||||
`HERMES_CUA_DRIVER_VERSION`**, so a local build (reported as
|
||||
`0.0.0-local-*`) is accepted as-is. Two approaches:
|
||||
|
||||
### Option A — `install-local` (build + put it on PATH)
|
||||
|
||||
From your `trycua/cua` checkout, run the upstream local installer. It
|
||||
builds the Rust backend in release mode and drops `cua-driver` into the
|
||||
same install layout the production installer uses, adding its bin dir
|
||||
to your PATH:
|
||||
|
||||
```powershell
|
||||
# Windows (PowerShell), from the cua repo root
|
||||
./libs/cua-driver/scripts/install-local.ps1 -NoAutoStart
|
||||
```
|
||||
|
||||
```bash
|
||||
# macOS / Linux, from the cua repo root (defaults to a debug build without --release)
|
||||
./libs/cua-driver/scripts/install-local.sh --release
|
||||
```
|
||||
|
||||
- Windows stages the build under `%USERPROFILE%\.cua-driver\packages\…`
|
||||
and junctions
|
||||
`%LOCALAPPDATA%\Programs\Cua\cua-driver\bin` (added to your User
|
||||
PATH) to it. macOS/Linux symlinks `cua-driver` into `~/.local/bin`
|
||||
(override with `--bin-dir <path>`).
|
||||
- `-NoAutoStart` skips registering the `cua-driver-serve` logon daemon
|
||||
— you don't need it for Hermes testing (see notes).
|
||||
|
||||
Then open a fresh shell (so the PATH change is visible) and confirm:
|
||||
|
||||
```
|
||||
cua-driver --version # local builds report 0.0.0-local-release
|
||||
# Windows: (Get-Command cua-driver).Source
|
||||
# macOS/Linux: which cua-driver
|
||||
```
|
||||
|
||||
### Option B — point Hermes straight at the built binary (fastest loop)
|
||||
|
||||
Skip the install ceremony entirely: `cargo build` and set
|
||||
`HERMES_CUA_DRIVER_CMD` to the resulting binary. Best for rapid
|
||||
edit/build/test.
|
||||
|
||||
```bash
|
||||
cargo build -p cua-driver # add --release for a release build; run from libs/cua-driver/rust
|
||||
```
|
||||
|
||||
```
|
||||
# Windows (.env)
|
||||
HERMES_CUA_DRIVER_CMD=C:\path\to\cua\libs\cua-driver\rust\target\debug\cua-driver.exe
|
||||
# macOS / Linux (.env)
|
||||
HERMES_CUA_DRIVER_CMD=/path/to/cua/libs/cua-driver/rust/target/debug/cua-driver
|
||||
```
|
||||
|
||||
### Confirm Hermes is using your build
|
||||
|
||||
- `hermes computer-use status` prints the resolved binary path and
|
||||
version.
|
||||
- `hermes computer-use doctor` confirms the binary is reachable and
|
||||
exercises the full MCP path end-to-end.
|
||||
- In a session, `computer_use(action="capture")` exercises the spawned
|
||||
`cua-driver mcp` child process.
|
||||
|
||||
### Notes & gotchas
|
||||
|
||||
- **Hermes spawns a `cua-driver mcp` stdio proxy.** In a normal session the
|
||||
proxy connects to (and may start) the standard machine daemon. In explicit
|
||||
Hermes YOLO, Hermes instead owns a private `cua-driver serve --embedded`
|
||||
child and points the proxy at its private socket or named pipe. The Windows
|
||||
autostart/UIAccess pattern still matters for interactive Session 1+ input
|
||||
from SSH — see the Limitations section.
|
||||
- **Locked binary on Windows.** A running `cua-driver-serve` daemon can
|
||||
hold `cua-driver.exe` and block an overwrite on rebuild.
|
||||
`install-local.ps1` renames the locked binary out of the way
|
||||
automatically; if you `cargo build` manually (Option B), stop it
|
||||
first with `cua-driver autostart disable` (or `schtasks /End /TN
|
||||
cua-driver-serve`).
|
||||
- **Rebuild loop.** After editing cua-driver source, re-run
|
||||
`install-local` (rebuilds, restages, flips the `current` junction)
|
||||
for Option A, or just re-`cargo build` for Option B — no Hermes
|
||||
change needed either way.
|
||||
- **Local builds skip the version check.** Hermes warns when the
|
||||
installed cua-driver is older than its per-OS tested baseline, but
|
||||
exempts `0.0.0-local-*` dev builds — so your local build never
|
||||
triggers that warning.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**First action when anything's off: run `hermes computer-use doctor`.**
|
||||
The structured per-check matrix tells you (and any agent helping you
|
||||
debug) exactly what's wrong.
|
||||
|
||||
Specific failure modes the doctor doesn't catch:
|
||||
|
||||
**`computer_use backend unavailable: cua-driver is not installed`** —
|
||||
Run `hermes computer-use install` to fetch the cua-driver binary, or
|
||||
run `hermes tools` and enable the Computer Use toolset.
|
||||
|
||||
**Clicks seem to have no effect** — Capture and verify. A modal you
|
||||
didn't see may be blocking input. Dismiss it with `escape` or the close
|
||||
button.
|
||||
|
||||
**Element indices are stale** — SOM indices are only valid until the
|
||||
next `capture`. Re-capture after any state-changing action. The
|
||||
wrapper carries opaque `element_token`s for stale detection — you'll
|
||||
see an explicit error rather than a wrong click.
|
||||
|
||||
**"blocked pattern in type text"** — The text you tried to `type`
|
||||
matches the dangerous-shell-pattern list. Break the command up or
|
||||
reconsider.
|
||||
|
||||
**Empty captures on Linux** — `DISPLAY` not set, or you're on pure
|
||||
Wayland without an XWayland bridge. `hermes computer-use doctor` will
|
||||
flag this as `ax_capability: fail` with a `Set DISPLAY (X11)…` hint.
|
||||
|
||||
**Empty captures on Windows over SSH** — You're in Session 0 (the
|
||||
services session). Drive from RDP / console directly, or set up the
|
||||
autostart pattern — see
|
||||
[cua.ai/docs/how-to-guides/driver/windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh).
|
||||
|
||||
## See also
|
||||
|
||||
- **Hermes-side skill** — `skills/autonomous-ai-agents/computer-use/SKILL.md` — teaches the
|
||||
Hermes `computer_use` action vocabulary; this is what the agent loads.
|
||||
- **cua-driver skill pack** — for platform-specific deep dives
|
||||
(macOS no-foreground contract, Windows UIA + Session 0, Linux AT-SPI
|
||||
+ X11/Wayland, recording, browser pages), run
|
||||
`cua-driver skills install` and read `MACOS.md` / `WINDOWS.md` /
|
||||
`LINUX.md` / `RECORDING.md` / `WEB_APPS.md`. Hermes autodetection is a
|
||||
planned follow-up; currently point Hermes at the installed pack directory
|
||||
or symlink it into your skill space.
|
||||
- **cua.ai/docs** — the cua-driver project's documentation:
|
||||
- [What is computer use?](https://cua.ai/docs/explanation/what-is-computer-use) — concept intro
|
||||
- [The no-foreground contract](https://cua.ai/docs/explanation/the-no-foreground-contract) — *why* background mode matters
|
||||
- [Install reference](https://cua.ai/docs/how-to-guides/driver/install) — cross-platform install details
|
||||
- [Personalize the agent cursor](https://cua.ai/docs/how-to-guides/driver/personalize-cursor) — built-in shapes, custom assets, runtime overrides
|
||||
- [Drive Windows over SSH](https://cua.ai/docs/how-to-guides/driver/windows-ssh) — the Session 0 → Session 1+ autostart pattern
|
||||
- [Keep cua-driver running](https://cua.ai/docs/how-to-guides/driver/keep-running) — autostart / daemon lifecycle
|
||||
- [Connect your agent](https://cua.ai/docs/how-to-guides/driver/connect-your-agent) — register cua-driver with various harnesses (Hermes among them)
|
||||
- [cua-driver source (trycua/cua)](https://github.com/trycua/cua)
|
||||
- [Browser automation](./browser.md) for cross-platform web tasks where you don't need to drive native apps.
|
||||
Reference in New Issue
Block a user