yilsem/aiturk-hermes-ide
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 6 Wiki Activity

Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license

Browse Source
This commit is contained in:
yilsem
2026-09-05 13:26:46 +03:00
commit 03634b1ca3
11340 changed files with 3442369 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
website/docs/user-guide/features/_category_.json
+8
View File
@@ -0,0 +1,8 @@
{
"label": "Features",
"position": 4,
"link": {
"type": "generated-index",
"description": "Explore the powerful features of Hermes Agent."
}
}
website/docs/user-guide/features/acp.md
+398
View File
@@ -0,0 +1,398 @@
---
sidebar_position: 11
title: "ACP Host Integration"
description: "Use Hermes Agent inside ACP-compatible editors and collaboration platforms"
---
# ACP Host Integration
Hermes Agent can run as an ACP server, letting ACP-compatible hosts talk to
Hermes over stdio. Editors can render:
- chat messages
- tool activity
- file diffs
- terminal commands
- approval prompts
- streamed thinking / response chunks
Other hosts can use the same protocol to route collaboration events into
Hermes. ACP is a good fit when you want Hermes to keep its existing identity,
provider setup, memory, skills, and tools while another application owns the
conversation transport.
## What Hermes exposes in ACP mode
Hermes runs with a curated `hermes-acp` toolset designed for editor workflows. It includes:
- file tools: `read_file`, `write_file`, `patch`, `search_files`
- terminal tools: `terminal`, `process`
- web/browser tools
- memory, todo, session search
- skills
- execute_code and delegate_task
- vision
It intentionally excludes things that do not fit typical editor UX, such as messaging delivery and cronjob management.
## Installation
Install Hermes normally, then add the ACP extra from the install checkout:
```bash
cd ~/.hermes/hermes-agent && uv pip install -e '.[acp]'
```
This installs the `agent-client-protocol` dependency and enables:
- `hermes acp`
- `hermes-acp`
- `python -m acp_adapter`
## Launching the ACP server
Any of the following starts Hermes in ACP mode:
```bash
hermes acp
```
```bash
hermes-acp
```
```bash
python -m acp_adapter
```
Hermes logs to stderr so stdout remains reserved for ACP JSON-RPC traffic.
For non-interactive checks:
```bash
hermes acp --version
hermes acp --check
```
### Browser tools (optional)
Browser tools (`browser_navigate`, `browser_click`, etc.) depend on the
`agent-browser` npm package and Chromium, which aren't part of the Python
wheel. Install them with:
```bash
hermes acp --setup-browser # interactive (prompts before ~400 MB download)
hermes acp --setup-browser --yes # accept the download non-interactively
```
This is the standalone command. The terminal-auth flow (`hermes acp --setup`) also offers the browser bootstrap as a follow-up question after model selection, so most users never need to run `--setup-browser` directly.
What it does:
- Installs Node.js 26 into `~/.hermes/node/` if missing
- `npm install -g agent-browser @askjo/camofox-browser` into that prefix (no sudo needed — `npm`'s `--prefix` points at the user-writable Hermes-managed Node)
- Installs Playwright Chromium, or uses a detected system Chrome/Chromium when available
The bootstrap is idempotent — re-running it is fast and skips work that's already done.
## Host setup
### Buzz channels (relay bridge)
[Buzz](https://github.com/block/buzz) is a Nostr-based collaboration platform
for people and agents. Its `buzz-acp` harness connects Buzz channels to any ACP
agent over stdio:
```text
Buzz relay <-- WebSocket --> buzz-acp <-- ACP over stdio --> Hermes Agent
```
This is a transport integration, not a second Hermes installation. The
subprocess launched by `buzz-acp` uses the same Hermes configuration,
credentials, memory, skills, and state as `hermes` on that host.
(This is distinct from [Buzz Desktop's managed runtime](#buzz-desktop), which
spawns Hermes locally as a preset harness. The relay bridge is for joining Buzz
*channels* as an agent identity, typically on a server.)
Prerequisites:
- Complete the ACP installation and `hermes acp --check` above.
- Build `buzz-acp` and the `buzz` CLI from the
[Buzz repository](https://github.com/block/buzz)
(`cargo build --release -p buzz-acp`).
- Mint a dedicated Nostr keypair for Hermes (`buzz-admin generate-key`) and
register it as a relay member (`buzz-admin add-member`). Every agent needs
its own identity — do not reuse a human keypair.
- Add that identity to the intended Buzz channels.
Start a bridge with:
```bash
export BUZZ_RELAY_URL="wss://community.example.com"
export BUZZ_PRIVATE_KEY="..."
export BUZZ_API_TOKEN="..."
export BUZZ_ACP_AGENT_COMMAND="hermes"
export BUZZ_ACP_AGENT_ARGS="acp"
buzz-acp
```
`BUZZ_API_TOKEN` is needed only when the relay enforces token authentication.
Do not commit or paste the private key or API token.
For a persistent server deployment, run `buzz-acp` under a service manager as
the same operating-system user that owns the intended Hermes home. Setup,
key generation, channel discovery, and per-agent options are documented in the
[buzz-acp README](https://github.com/block/buzz/tree/main/crates/buzz-acp).
The bridge discovers every Buzz channel where the Hermes identity is a member
and automatically subscribes when it is added to another channel. Buzz channel
membership therefore remains the access boundary; Hermes does not need a
separate channel list in its own configuration.
To expose Hermes ACP activity in the owner's Buzz Desktop, add:
```bash
export BUZZ_ACP_RELAY_OBSERVER="true"
```
This publishes encrypted kind `24200` observer frames addressed to the agent's
owner (Buzz's NIP-AO). Desktop renders the live lifecycle, tool, response, and
usage stream in the agent's **Activity log**. The relay treats these frames as
ephemeral, so Desktop must be online before the turn starts; its local observer
archive is the durable owner-side history.
Headless bridges answer ACP permission requests themselves because no editor
is present to show approval dialogs — see
[Keep Buzz agents owner-only](#keep-buzz-agents-owner-only). Treat the bridge
as privileged automation: use a dedicated operating-system account, restrict
which Buzz users can prompt the agent (`buzz-acp` supports an owner-only
respond gate via `BUZZ_ACP_AGENT_OWNER`), and grant membership only in channels
where Hermes is expected to work.
### VS Code
Install the [ACP Client](https://marketplace.visualstudio.com/items?itemName=formulahendry.acp-client) extension.
To connect:
1. Open the ACP Client panel from the Activity Bar.
2. Select **Hermes Agent** from the built-in agent list.
3. Connect and start chatting.
If you want to define Hermes manually, add it through VS Code settings under `acp.agents`:
```json
{
"acp.agents": {
"Hermes Agent": {
"command": "hermes",
"args": ["acp"]
}
}
}
```
### Zed
Configure Hermes as a custom agent server in Zed settings:
1. Open the Agent Panel.
2. Add a custom agent server with the following configuration:
```json
{
"agent_servers": {
"hermes-agent": {
"type": "custom",
"command": "hermes",
"args": ["acp"]
}
}
}
```
3. Start a new Hermes external-agent thread.
Prerequisites:
- Configure Hermes provider credentials first with `hermes model`, or set them in `~/.hermes/.env` / `~/.hermes/config.yaml`.
### JetBrains
Use an ACP-compatible plugin and point it at `hermes acp` or `hermes-acp`.
### Buzz Desktop
[Buzz](https://github.com/block/buzz) ships Hermes Agent as a preset runtime.
With Hermes installed the normal way, Buzz discovers it automatically —
open **Settings → Runtimes** and Hermes appears under your runtimes.
If discovery fails (older installs), make sure the ACP launcher resolves on a
login-shell PATH:
```bash
command -v hermes-acp || command -v hermes
```
Recent installs write both `hermes` and `hermes-acp` launchers into
`~/.local/bin`; running `hermes update` adds the `hermes-acp` launcher to
older installs. As a manual fallback, configure Buzz's agent command as
`hermes` with args `["acp"]`.
#### Model picker
Buzz Desktop (v0.5.1+) renders Hermes' full model menu in the agent's runtime
settings. The list comes from Hermes itself over ACP: it shows every model
from providers you have authenticated in Hermes (the same inventory behind
`hermes model` and the `/model` command), so a model missing from the menu
means its provider has no credentials configured on the Hermes side.
Entry IDs take the form `provider:model` (e.g. `openrouter:z-ai/glm-5.1`), or
`custom:<name>:<model>` for custom OpenAI-compatible endpoints defined in
`config.yaml`. Picking a model applies to that agent's session; it does not
change your Hermes-wide default — use `hermes model` for that.
#### Keep Buzz agents owner-only
Buzz creates every agent with **Who can talk to this agent** set to `Owner only`.
Leave it there when the runtime is Hermes.
Two behaviors combine on this path. The `hermes-acp` toolset includes `terminal`
and `execute_code`, and Buzz's ACP bridge answers Hermes' permission requests
itself with `allow_once` rather than surfacing them. A Hermes agent in Buzz
therefore runs shell commands on the host without prompting. I asked one to run
`rm -rf` against a scratch directory and it deleted it, no prompt anywhere.
Selecting `Anyone` hands that same shell access to every author who can reach
the channel. Buzz does not warn when you pick it.
Neither of the obvious mitigations works today:
- `approvals.mode: manual` does make Hermes raise the permission request, but
Buzz auto-approves it and the command still runs.
- `platform_toolsets.acp` does not narrow the ACP toolset, so it cannot be used
to drop `terminal`.
`!shutdown` from the owner stops the agent in any mode, and Buzz ignores that
command from everyone else.
## Configuration and credentials
ACP mode uses the same Hermes configuration as the CLI:
- `~/.hermes/.env`
- `~/.hermes/config.yaml`
- `~/.hermes/skills/`
- `~/.hermes/state.db`
Provider resolution uses Hermes' normal runtime resolver, so ACP inherits the currently configured provider and credentials. Hermes also advertises a terminal auth method (`--setup`) for first-run ACP clients; this opens Hermes' interactive model/provider setup.
## Host integration
These variables are set by an **ACP host process** (an editor or another agent
harness) on the Hermes subprocess it spawns. They are not user configuration —
do not set them by hand in `.env` or `config.yaml`.
| Variable | Value | Effect |
|----------|-------|--------|
| `HERMES_ACP_SKIP_CONFIGURED_MCP` | `1` | Skip starting the **globally configured** MCP servers from `config.yaml` before the ACP JSON-RPC loop begins. |
Hermes normally starts every MCP server configured in `config.yaml` before it
enters the ACP JSON-RPC loop. A host that owns MCP itself — passing the
session's servers explicitly through `session/new` — does not need that global
startup, and an unrelated slow or interactive MCP server would otherwise delay
`initialize`. Setting the marker to exactly `1` lets such a host skip it.
Only the global `config.yaml` discovery is skipped. **MCP servers supplied by
the ACP session through `session/new` are still registered**, so a host loses
no capability it asked for. Any other value (unset, empty, `0`, `false`) keeps
the default behavior, so an unrelated truthy-looking string cannot silently
disable MCP.
## Session behavior
ACP sessions are tracked by the ACP adapter's in-memory session manager while the server is running.
Each session stores:
- session ID
- working directory
- selected model
- current conversation history
- cancel event
The underlying `AIAgent` still uses Hermes' normal persistence/logging paths, but ACP `list/load/resume/fork` are scoped to the currently running ACP server process.
## Working directory behavior
ACP sessions bind the editor's cwd to the Hermes task ID so file and terminal tools run relative to the editor workspace, not the server process cwd.
## Approvals
Dangerous terminal commands can be routed back to the editor as approval prompts. ACP approval options are simpler than the CLI flow:
- allow once
- allow always
- deny
Whether you actually see a prompt is up to the host. A host is free to answer the
request programmatically instead of showing it to you, in which case these
options exist on the wire but never reach a human. Buzz Desktop does this, so
treat that path as unattended execution regardless of your `approvals` setting.
On timeout or error, the approval bridge denies the request.
### Session-scoped edit auto-approval
ACP exposes a third tier between *allow once* and *allow always*: **Allow for session**. Picking it from the editor's permission prompt records the approval inside the current ACP session only — every subsequent matching command in that session goes through without prompting, but a new ACP session (or restarting the editor) resets the slate and re-prompts the first time.
| Option | Editor label | Scope | Persisted across restarts |
|---|---|---|---|
| `allow_once` | Allow once | This one tool call | No |
| `allow_session` | Allow for session | All matching calls in this ACP session | No — cleared when the session ends |
| `allow_always` | Allow always | All future sessions | Yes (written to the Hermes permanent allowlist) |
| `deny` | Deny | This one tool call | No |
`allow_session` is the right default for an editor workflow where you trust an agent for the duration of a task but don't want to grant a long-lived allowlist entry. The safety trade-off is straightforward: the broader the scope, the less the editor will interrupt you, and the more damage a misbehaving agent (or prompt injection) can do before you notice. Start with `allow_once` for unfamiliar commands; promote to `allow_session` once you've seen the agent run the same pattern correctly a few times; reserve `allow_always` for truly idempotent commands you trust forever (e.g. `git status`).
The ACP bridge maps these options onto Hermes' internal approval semantics — `allow_always` writes a permanent allowlist entry the same way the CLI does, while `allow_session` only affects the in-process approval cache for the current ACP session.
## Troubleshooting
### ACP agent does not appear in the editor
Check:
- For manual/local development, verify the host command points to `hermes acp`.
- Hermes is installed and on your PATH.
- The ACP extra is installed (`cd ~/.hermes/hermes-agent && uv pip install -e '.[acp]'`).
### ACP starts but immediately errors
Try these checks:
```bash
hermes acp --version
hermes acp --check
hermes doctor
hermes status
```
### Missing credentials
ACP mode uses Hermes' existing provider setup. Configure credentials with:
```bash
hermes model
```
or by editing `~/.hermes/.env`. The terminal auth flow (`hermes acp --setup`) can also trigger the interactive provider/model setup.
## See also
- [Buzz ACP harness](https://github.com/block/buzz/tree/main/crates/buzz-acp)
- [ACP Internals](../../developer-guide/acp-internals.md)
- [Provider Runtime Resolution](../../developer-guide/provider-runtime.md)
- [Tools Runtime](../../developer-guide/tools-runtime.md)
website/docs/user-guide/features/api-server.md
+771
View File
@@ -0,0 +1,771 @@
---
sidebar_position: 14
title: "API Server"
description: "Expose hermes-agent as an OpenAI-compatible API for any frontend"
---
# API Server
The API server exposes hermes-agent as an OpenAI-compatible HTTP endpoint. Any frontend that speaks the OpenAI format — Open WebUI, LobeChat, LibreChat, NextChat, ChatBox, and hundreds more — can connect to hermes-agent and use it as a backend.
Your agent handles requests with its full toolset (terminal, file operations, web search, memory, skills) and returns the final response. When streaming, tool progress indicators appear inline so frontends can show what the agent is doing.
:::tip One backend covers models + tools
Hermes itself needs a configured provider and tool backends for the API server to be useful. A [Nous Portal](/user-guide/features/tool-gateway) subscription handles both — 300+ models plus web/image/TTS/browser via the Tool Gateway. Run `hermes setup --portal` once before starting the API server and frontends like Open WebUI or LobeChat get a fully tool-equipped backend.
:::
## Quick Start
### 1. Enable the API server
Add to `~/.hermes/.env`:
```bash
API_SERVER_ENABLED=true
API_SERVER_KEY=change-me-local-dev
# Optional: only if a browser must call Hermes directly
# API_SERVER_CORS_ORIGINS=http://localhost:3000
```
### 2. Start the gateway
```bash
hermes gateway
```
You'll see:
```
[API Server] API server listening on http://127.0.0.1:8642
```
### 3. Connect a frontend
Point any OpenAI-compatible client at `http://localhost:8642/v1`:
```bash
# Test with curl
curl http://localhost:8642/v1/chat/completions \
-H "Authorization: Bearer change-me-local-dev" \
-H "Content-Type: application/json" \
-d '{"model": "hermes-agent", "messages": [{"role": "user", "content": "Hello!"}]}'
```
Or connect Open WebUI, LobeChat, or any other frontend — see the [Open WebUI integration guide](/user-guide/messaging/open-webui) for step-by-step instructions.
## Endpoints
### POST /v1/chat/completions
Standard OpenAI Chat Completions format. Stateless — the full conversation is included in each request via the `messages` array.
**Request:**
```json
{
"model": "hermes-agent",
"messages": [
{"role": "system", "content": "You are a Python expert."},
{"role": "user", "content": "Write a fibonacci function"}
],
"stream": false
}
```
**Response:**
```json
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"created": 1710000000,
"model": "hermes-agent",
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": "Here's a fibonacci function..."},
"finish_reason": "stop"
}],
"usage": {"prompt_tokens": 50, "completion_tokens": 200, "total_tokens": 250}
}
```
**Inline image input:** user messages may send `content` as an array of `text` and `image_url` parts. Both remote `http(s)` URLs and `data:image/...` URLs are supported:
```json
{
"model": "hermes-agent",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What is in this image?"},
{"type": "image_url", "image_url": {"url": "https://example.com/cat.png", "detail": "high"}}
]
}
]
}
```
Uploaded files (`file` / `input_file` / `file_id`) and non-image `data:` URLs return `400 unsupported_content_type`.
**Streaming** (`"stream": true`): Returns Server-Sent Events (SSE) with token-by-token response chunks. For **Chat Completions**, the stream uses standard `chat.completion.chunk` events plus Hermes' custom `hermes.tool.progress` event for tool-start UX. For **Responses**, the stream uses OpenAI Responses event types such as `response.created`, `response.output_text.delta`, `response.output_item.added`, `response.output_item.done`, and `response.completed`.
**Tool progress in streams**:
- **Chat Completions**: Hermes emits `event: hermes.tool.progress` for tool-start visibility without polluting persisted assistant text.
- **Responses**: Hermes emits spec-native `function_call` and `function_call_output` output items during the SSE stream, so clients can render structured tool UI in real time.
### POST /v1/responses
OpenAI Responses API format. Supports server-side conversation state via `previous_response_id` — the server stores full conversation history (including tool calls and results) so multi-turn context is preserved without the client managing it.
**Request:**
```json
{
"model": "hermes-agent",
"input": "What files are in my project?",
"instructions": "You are a helpful coding assistant.",
"store": true
}
```
**Response:**
```json
{
"id": "resp_abc123",
"object": "response",
"status": "completed",
"model": "hermes-agent",
"output": [
{"type": "function_call", "status": "completed", "name": "terminal", "arguments": "{\"command\": \"ls\"}", "call_id": "call_1"},
{"type": "function_call_output", "status": "completed", "call_id": "call_1", "output": "README.md src/ tests/"},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "Your project has..."}]}
],
"usage": {"input_tokens": 50, "output_tokens": 200, "total_tokens": 250}
}
```
Tool calls in the `output` array were already executed server-side by the Hermes agent — they are replayed with `"status": "completed"` for structured tool UI, never as pending calls for the client to execute.
**Inline image input:** `input[].content` can contain `input_text` and `input_image` parts. Both remote URLs and `data:image/...` URLs are supported:
```json
{
"model": "hermes-agent",
"input": [
{
"role": "user",
"content": [
{"type": "input_text", "text": "Describe this screenshot."},
{"type": "input_image", "image_url": "data:image/png;base64,iVBORw0K..."}
]
}
]
}
```
Uploaded files (`input_file` / `file_id`) and non-image `data:` URLs return `400 unsupported_content_type`.
#### Multi-turn with previous_response_id
Chain responses to maintain full context (including tool calls) across turns:
```json
{
"input": "Now show me the README",
"previous_response_id": "resp_abc123"
}
```
The server reconstructs the full conversation from the stored response chain — all previous tool calls and results are preserved. Chained requests also share the same session, so multi-turn conversations appear as a single entry in the dashboard and session history.
#### Named conversations
Use the `conversation` parameter instead of tracking response IDs:
```json
{"input": "Hello", "conversation": "my-project"}
{"input": "What's in src/?", "conversation": "my-project"}
{"input": "Run the tests", "conversation": "my-project"}
```
The server automatically chains to the latest response in that conversation. Like the `/title` command for gateway sessions.
### GET /v1/responses/\{id\}
Retrieve a previously stored response by ID.
### DELETE /v1/responses/\{id\}
Delete a stored response.
### GET /v1/models
Lists the agent as an available model. The advertised model name defaults to the [profile](/user-guide/profiles) name (or `hermes-agent` for the default profile). Required by most frontends for model discovery.
`/v1/models` is intentionally the cheap OpenAI-compat surface. It does **not**
enumerate every authenticated provider/model combination Hermes can route to,
and it does not do pricing or capability enrichment.
### GET /api/model/options
Hermes-aware clients can request the same curated provider/model inventory used
by the dashboard and TUI. This route uses the API server's normal bearer
authentication and returns provider rows, model capability hints, and pricing
metadata that do not belong in the OpenAI-compatible `/v1/models` response:
```bash
curl \
-H "Authorization: Bearer $API_SERVER_KEY" \
"http://127.0.0.1:8642/api/model/options"
```
That payload is the same substrate the dashboard Models page and the TUI
`model.options` RPC use. It returns authenticated providers, curated model
lists, per-model pricing, and model capability hints.
Normal opens are intentionally conservative for custom providers: Hermes probes
only the **currently selected** custom endpoint so a stale or offline saved
endpoint does not block the picker. An explicit refresh flips to full probing
and busts the provider model cache:
```bash
curl \
-H "Authorization: Bearer $API_SERVER_KEY" \
"http://127.0.0.1:8642/api/model/options?refresh=1"
```
Use `/v1/models` when an OpenAI-compatible client only needs a model name to
send back in chat/responses requests. Use `/api/model/options` when an
authenticated UI needs the richer Hermes-specific picker metadata.
### GET /v1/capabilities
Returns a machine-readable description of the API server's stable surface for external UIs, orchestrators, and plugin bridges.
```json
{
"object": "hermes.api_server.capabilities",
"platform": "hermes-agent",
"model": "hermes-agent",
"auth": {"type": "bearer", "required": true},
"features": {
"chat_completions": true,
"responses_api": true,
"run_submission": true,
"run_status": true,
"run_events_sse": true,
"run_stop": true
}
}
```
Use this endpoint when integrating dashboards, browser UIs, or control planes so they can discover whether the running Hermes version supports runs, streaming, cancellation, and session continuity without depending on private Python internals.
## Browser-extension control
Hermes can route browser tools through an authenticated extension that controls
the browser session associated with the current Hermes session. The feature is
disabled by default; set `browser.extension_control.enabled` to `true` to opt in:
```yaml
browser:
extension_control:
enabled: true
```
The local API path also requires the API server bearer key. A controller may
register only for an existing server session. Hermes derives the controller
principal from authenticated server state; a client-supplied `principal_id` is
ignored.
Discover the live contract through `GET /v1/capabilities`. The
`browser_extension_control` object reports whether the feature is enabled, the
protocol version, transport names, and the exact capability allowlist:
```text
controller.noop
browser_back
browser_click
browser_navigate
browser_press
browser_screenshot
browser_scroll
browser_snapshot
browser_tab_activate
browser_tabs
browser_type
```
Requested capabilities outside that list are filtered out. Raw CDP, arbitrary
script evaluation, console access, uploads, image extraction, and vision are not
part of the controller protocol.
When a request has no bound controller identity, or when the feature is disabled,
Hermes preserves the existing browser backend. Once the gateway binds a
controller principal and transport family to the request, that extension lane
is authoritative: missing, ambiguous, disconnected, or incapable controllers
fail closed instead of silently switching to a different local/cloud browser.
After an exact controller is selected, its result or error is authoritative and
Hermes never retries the same action through another backend.
### Local API registration
1. Send an authenticated `POST /v1/browser-control/register` with
`protocol_version`, `session_id`, `controller_id`, `browser_profile_id`, and
the requested `capabilities`.
2. Hermes returns a single-use ticket with a 30-second TTL and the filtered,
server-bound controller scope.
3. Open `GET /v1/browser-control/ws` with both WebSocket subprotocols:
`hermes-browser-control-v1` and
`hermes-browser-control-ticket.<ticket>`.
The ticket is never accepted in the query string. Unknown, expired, reused, or
malformed tickets fail before WebSocket upgrade.
### Controller frames
Hermes sends `browser.controller.command` frames containing `command_id`,
`action`, immutable `arguments`, browser/controller ids, and the originating
`tool_call_id`. The controller replies with `browser.controller.result`, the
same `command_id`, an exact boolean `ok`, and either `result` or `error`.
Cancellation and timeout emit `browser.controller.cancel`; late results are
ignored.
An unexpected socket loss marks the controller offline and preserves work
already in flight until each command's original deadline. A reconnect with the
same principal, profile, session, controller id, browser profile, and transport
identity refreshes the transport without admitting new work before any deferred
cancels are flushed. Negotiated capabilities may change on that reconnect; they
are not an identity field. A different controller id or browser profile in the
same authenticated session lane is a hard replacement: old pending work is
cancelled before the successor becomes routable. Send
`browser.controller.detach` on the authenticated controller transport for an
intentional hard detach — that immediately cancels pending work. Merely closing
the socket is treated as a recoverable disconnect.
The authenticated dashboard transport exposes the same registration, result,
heartbeat, capability, and ownership semantics over its Gateway RPC/event
channel. In both transports, selection requires one unambiguous exact match on
principal, profile, session, controller, browser profile, transport family, and
capability. Once selected, a controller failure is authoritative and is never
retried through a different browser backend.
## Per-request model selection
Authenticated clients can override Hermes' default model selection per request
by sending:
- `model` — the target model id for this turn
- `provider` — the Hermes provider slug to resolve credentials/runtime for this turn
- `model_options` — request-scoped reasoning / service-tier controls
The same request fields are accepted on:
- `POST /v1/chat/completions`
- `POST /v1/responses`
- `POST /v1/runs`
- `POST /api/sessions/{session_id}/chat`
- `POST /api/sessions/{session_id}/chat/stream`
Precedence is deterministic:
1. Session `/model` override, if that session already has one
2. A static `gateway.platforms.api_server.model_routes` mapping selected when
the request's `model` is a configured route alias
3. Direct request `model` / `provider` when no route alias matches
4. Global gateway config / environment defaults
`model_options` stays request-scoped regardless of which model/provider wins.
If a request sends a `provider` that conflicts with a configured `model_routes`
alias, Hermes rejects the request with `400` instead of silently remixing route
credentials with another provider.
**Bare `model` values on the OpenAI-compatible endpoints are opt-in.** Generic
OpenAI clients routinely hardcode model names (`gpt-4o`, ...), and existing
deployments rely on those falling back to the gateway default. On
`POST /v1/chat/completions` and `POST /v1/responses`, a `model` value sent
WITHOUT a `provider` is therefore ignored unless you enable:
```yaml
gateway:
platforms:
api_server:
direct_model_requests: true
```
Requests that include an explicit `provider` — and the Hermes-native
`/v1/runs` and session-chat endpoints — always honor the requested model
regardless of this flag.
Example:
```json
{
"model": "MiniMax-M3",
"provider": "minimax",
"model_options": {
"reasoning_effort": "high",
"service_tier": "priority"
},
"messages": [
{"role": "user", "content": "Summarize the repo status."}
]
}
```
### GET /health
Health check. Returns `{"status": "ok"}`. Also available at **GET /v1/health** for OpenAI-compatible clients that expect the `/v1/` prefix.
### GET /health/detailed
Authenticated readiness check for monitoring and control planes. It reports
bounded status for the active profile's config, state database, configured
model, disk space, gateway/platform state, active API runs, pending process
completions, and active delegations. The response exposes status and counts,
not config values, credentials, paths, commands, queue payloads, or raw errors.
The public `/health` route remains a cheap liveness probe and does not run
readiness checks. A degraded readiness result still uses HTTP 200; inspect the
top-level `status` and `readiness.checks` fields.
## Runs API (streaming-friendly alternative)
In addition to `/v1/chat/completions` and `/v1/responses`, the server exposes a **runs** API for long-form sessions where the client wants to subscribe to progress events instead of managing streaming themselves.
### POST /v1/runs
Create a new agent run. Returns a `run_id` that can be used to subscribe to progress events.
```json
{
"run_id": "run_abc123",
"status": "started"
}
```
Runs accept a simple `input` string and optional `session_id`, `instructions`, `conversation_history`, or `previous_response_id`. When `session_id` is provided, Hermes surfaces it in the run status so external UIs can correlate runs with their own conversation IDs.
For safely retryable creation, send an `Idempotency-Key` header (1–255 visible ASCII characters). Hermes durably reserves the key before starting work. An identical retry returns the original `run_id` with HTTP 202 and `Idempotency-Replayed: true`, including after a gateway restart and after the run has completed, failed, or been cancelled. Reusing the same key with a different JSON payload returns HTTP 409 with code `idempotency_key_conflict`. Keys are isolated by authenticated API profile/credential and retained for 24 hours after their last status update; clients should use unique, unguessable keys and must not reuse them for unrelated operations. Requests without the header retain the legacy behavior and always create a new run.
When `session_id` identifies an existing Hermes session and no explicit
`conversation_history` or `previous_response_id` is supplied, the run loads
that session's active transcript. Session turn leases serialize concurrent
writers and refresh the transcript after a contended wait.
### GET /v1/runs/\{run_id\}
Poll the current run state. This is useful for dashboards that need status without holding an SSE connection open, or for UIs that reconnect after navigation.
```json
{
"object": "hermes.run",
"run_id": "run_abc123",
"status": "completed",
"session_id": "space-session",
"model": "hermes-agent",
"output": "Done.",
"usage": {"input_tokens": 50, "output_tokens": 200, "total_tokens": 250}
}
```
Statuses are retained briefly after terminal states (`completed`, `failed`, or `cancelled`) for polling and UI reconciliation.
### GET /v1/runs/\{run_id\}/events
Server-Sent Events stream of the run's tool-call progress, token deltas, and lifecycle events. Designed for dashboards and thick clients that want to attach/detach without losing state.
When the agent delegates work to background subagents, the stream also carries
`subagent.start` and `subagent.complete` lifecycle events, so clients can
observe delegation outcomes — including timeouts and failures — instead of the
run going silent while a child works. The `subagent.complete` payload carries
the child's status, summary, duration, token/cost figures, a
`child_session_id` for correlation, and the `delegation_id` of the batch it
belongs to (so concurrent or nested fan-outs stay distinguishable); free-text fields pass forced secret
redaction before leaving the process. Per-tool child events
(`subagent.tool`, progress ticks) are intentionally **not** forwarded — they
are high-volume UI noise; use the per-child live transcript files for
play-by-play.
Unconsumed event buffers expire after five minutes so a detached client cannot
grow memory indefinitely. This expires transport state only: a run that is
still executing remains visible to status polling, approval, stop control, and
concurrency accounting until its executor work actually exits. A connected SSE
subscriber continues draining normally.
### POST /v1/runs/\{run_id\}/stop
Interrupt a running agent turn. The endpoint returns immediately with `{"status": "stopping"}` while Hermes asks the active agent to stop at the next safe interruption point.
The run stays tracked as `stopping` until the executor-backed work exits, then
settles as `cancelled`; requesting stop never hides a worker that is still
running.
### POST /v1/runs/\{run_id\}/approval
Resolve a pending approval for a run that is waiting on a human decision (for example, a tool call gated behind an approval policy). The body carries the approval decision; the run resumes once the decision is recorded. This endpoint is advertised in `/v1/capabilities` as the `run_approval` feature so external UIs can detect support before surfacing an approval prompt.
## Jobs API (background scheduled work)
The server exposes a lightweight jobs CRUD surface for managing scheduled / background agent runs from a remote client. All endpoints are gated behind the same bearer auth.
### GET /api/jobs
List all scheduled jobs.
### POST /api/jobs
Create a new scheduled job. Body accepts the same shape as `hermes cron` — prompt, schedule, skills, provider override, delivery target.
### GET /api/jobs/\{job_id\}
Fetch a single job's definition and last-run state.
### PATCH /api/jobs/\{job_id\}
Update fields on an existing job (prompt, schedule, etc.). Partial updates are merged.
### DELETE /api/jobs/\{job_id\}
Remove a job. Also cancels any in-flight run.
### POST /api/jobs/\{job_id\}/pause
Pause a job without deleting it. Next-scheduled-run timestamps are suspended until resumed.
### POST /api/jobs/\{job_id\}/resume
Resume a previously paused job.
### POST /api/jobs/\{job_id\}/run
Trigger the job to run immediately, out of schedule.
## Sessions API (session control over REST)
External UIs can manage Hermes sessions over REST without standing up the dashboard. All endpoints are gated by `API_SERVER_KEY` and live under `/api/sessions/*`.
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/sessions` | List sessions (paginated — `limit`, `offset`, `source`, `include_children`) |
| `POST` | `/api/sessions` | Create an empty session |
| `GET` | `/api/sessions/{id}` | Read session metadata |
| `PATCH` | `/api/sessions/{id}` | Update title or `end_reason` |
| `DELETE` | `/api/sessions/{id}` | Delete a session |
| `GET` | `/api/sessions/{id}/messages` | Message history for a session |
| `POST` | `/api/sessions/{id}/fork` | Branch the session via `SessionDB` lineage (matches CLI `/branch` semantics) |
| `POST` | `/api/sessions/{id}/chat` | Run one synchronous agent turn |
| `POST` | `/api/sessions/{id}/chat/stream` | SSE wrapper over a single turn — emits `assistant.delta`, `tool.started`, `tool.completed`, `run.completed` events |
`/v1/capabilities` advertises the full surface via `session_*` feature flags and `endpoints.session_*` entries so external UIs can detect support and fall back safely. Inline images are supported in `chat` and `chat/stream` payloads (multimodal-aware path).
```bash
# fork a session and run one turn
curl -X POST http://localhost:8642/api/sessions/$ID/fork \
-H "Authorization: Bearer $API_SERVER_KEY" \
-d '{"title": "explore alt path"}'
# stream a turn over SSE
curl -N -X POST http://localhost:8642/api/sessions/$ID/chat/stream \
-H "Authorization: Bearer $API_SERVER_KEY" \
-d '{"input": "what files changed in the last hour?"}'
```
## Skills and toolsets discovery
`GET /v1/skills` and `GET /v1/toolsets` let external clients enumerate the agent's capabilities deterministically over REST instead of asking the model. Both are read-only and gated by `API_SERVER_KEY`.
```bash
curl http://localhost:8642/v1/skills \
-H "Authorization: Bearer $API_SERVER_KEY"
# → [{"name": "github-pr-workflow", "description": "...", "category": "..."}, ...]
curl http://localhost:8642/v1/toolsets \
-H "Authorization: Bearer $API_SERVER_KEY"
# → [{"name": "core", "label": "...", "description": "...", "enabled": true,
# "configured": true, "tools": ["read_file", "write_file", ...]}, ...]
```
`/v1/skills` returns the same metadata the skills hub uses internally. `/v1/toolsets` returns toolsets resolved for the `api_server` platform with the concrete `tools` list each one expands to. Both are advertised under `endpoints.*` in `/v1/capabilities`.
## Long-term memory scoping (`X-Hermes-Session-Key`)
Multi-user frontends like Open WebUI need a stable per-channel identifier for long-term memory (Honcho, etc.) that is **independent** of the transcript-scoped `X-Hermes-Session-Id` (which rotates on `/new`). Pass `X-Hermes-Session-Key` on `/v1/chat/completions`, `/v1/responses`, or `/v1/runs` and Hermes threads it through to `AIAgent(gateway_session_key=...)`, where the Honcho memory provider uses it to derive a stable scope.
```http
POST /v1/chat/completions HTTP/1.1
Authorization: Bearer ***
X-Hermes-Session-Id: transcript-alpha
X-Hermes-Session-Key: agent:main:webui:dm:user-42
```
Rules: max 256 chars, control characters (`\r`, `\n`, `\x00`) are rejected, and the value is echoed back on responses (JSON + SSE). `/v1/capabilities` advertises support via `"session_key_header": "X-Hermes-Session-Key"`. Without the key, Honcho's `per-session` strategy produces a different scope per `session_id` — exactly the behavior Hermes had before.
## System Prompt Handling
When a frontend sends a `system` message (Chat Completions) or `instructions` field (Responses API), hermes-agent **layers it on top** of its core system prompt. Your agent keeps all its tools, memory, and skills — the frontend's system prompt adds extra instructions.
This means you can customize behavior per-frontend without losing capabilities:
- Open WebUI system prompt: "You are a Python expert. Always include type hints."
- The agent still has terminal, file tools, web search, memory, etc.
## Authentication
Bearer token auth via the `Authorization` header:
```
Authorization: Bearer ***
```
Configure the key via `API_SERVER_KEY` env var. If you need a browser to call Hermes directly, also set `API_SERVER_CORS_ORIGINS` to an explicit allowlist.
### Multi-profile routing (`/p/<profile>/…`)
When [multi-profile gateway routing](/user-guide/multi-profile-gateways) is
enabled (`gateway.multiplex_profiles`), the shared listener serves every
profile through a `/p/<profile>/` URL prefix — and **authentication is bound
to the routed profile**:
- Requests to `/p/<profile>/v1/...` must present that profile's own
`API_SERVER_KEY` (from `~/.hermes/profiles/<profile>/.env`). The default
listener's key is rejected on named-profile prefixes.
- Unprefixed routes and `/p/default/...` keep using the default profile's key.
- A named profile with no `API_SERVER_KEY` of its own fails closed — its
prefix is unreachable until you set one.
- Runs are per-profile scoped: `/v1/runs/{run_id}` and its `events`, `stop`,
`steer`, and `approval` routes only answer for the profile that created
the run (including runs started via `/api/sessions/{id}/chat/stream`);
another profile's run id returns `404`, never `403`.
:::warning Breaking change (July 2026)
Before this fix, a valid default-profile key was accepted on any
`/p/<profile>/` prefix. If you relied on one shared key across profile
prefixes, set a distinct `API_SERVER_KEY` in each profile's `.env` — reused
default keys on named prefixes now return `401`.
:::
:::warning Security
The API server gives full access to hermes-agent's toolset, **including terminal commands**. `API_SERVER_KEY` is **required for every deployment**, including the default loopback bind on `127.0.0.1`. Keep `API_SERVER_CORS_ORIGINS` narrow to control browser access when you explicitly allow browser callers.
:::
## Configuration
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `API_SERVER_ENABLED` | `false` | Enable the API server |
| `API_SERVER_PORT` | `8642` | HTTP server port |
| `API_SERVER_HOST` | `127.0.0.1` | Bind address (localhost only by default) |
| `API_SERVER_KEY` | _(required)_ | Bearer token for auth |
| `API_SERVER_CORS_ORIGINS` | _(none)_ | Comma-separated allowed browser origins |
| `API_SERVER_MODEL_NAME` | _(profile name)_ | Model name on `/v1/models`. Defaults to profile name, or `hermes-agent` for default profile. |
### config.yaml
The same settings can live in `~/.hermes/config.yaml` under a nested `gateway.api_server:` section:
```yaml
gateway:
api_server:
enabled: true
port: 8642
host: 127.0.0.1
key: your-secret-key
cors_origins: http://localhost:3000
model_name: my-hermes
max_concurrent_runs: 10 # concurrent-run cap; 0 disables the limit
```
`port`, `key`, `host`, `cors_origins`, and `model_name` are automatically bridged into the platform's `extra` settings, so they behave exactly like their `API_SERVER_*` environment-variable counterparts. Environment variables take precedence over `config.yaml` values. The block is also accepted under `gateway.platforms.api_server:` or a top-level `platforms.api_server:` section.
### Concurrent-run cap
The API server limits how many agent runs may execute at once across the OpenAI-compatible and Runs endpoints. The cap is read from `gateway.api_server.max_concurrent_runs` (default **10**; `0` disables the limit, negative values clamp to 0). When the cap is reached, new run-starting requests are rejected with **HTTP 429** `Too many concurrent runs (max N)` — clients should back off and retry.
## Security Headers
All responses include security headers:
- `X-Content-Type-Options: nosniff` — prevents MIME type sniffing
- `Referrer-Policy: no-referrer` — prevents referrer leakage
## CORS
The API server does **not** enable browser CORS by default.
For direct browser access, set an explicit allowlist:
```bash
API_SERVER_CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
```
When CORS is enabled:
- **Preflight responses** include `Access-Control-Max-Age: 600` (10 minute cache)
- **SSE streaming responses** include CORS headers so browser EventSource clients work correctly
- **`Idempotency-Key`** is an allowed request header — clients can send it for deduplication (responses are cached by key for 5 minutes)
Most documented frontends such as Open WebUI connect server-to-server and do not need CORS at all.
## Compatible Frontends
Any frontend that supports the OpenAI API format works. Tested/documented integrations:
| Frontend | Stars | Connection |
|----------|-------|------------|
| [Open WebUI](/user-guide/messaging/open-webui) | 126k | Full guide available |
| LobeChat | 73k | Custom provider endpoint |
| LibreChat | 34k | Custom endpoint in librechat.yaml |
| AnythingLLM | 56k | Generic OpenAI provider |
| NextChat | 87k | BASE_URL env var |
| ChatBox | 39k | API Host setting |
| Jan | 26k | Remote model config |
| HF Chat-UI | 8k | OPENAI_BASE_URL |
| big-AGI | 7k | Custom endpoint |
| OpenAI Python SDK | — | `OpenAI(base_url="http://localhost:8642/v1")` |
| curl | — | Direct HTTP requests |
## Multi-User Setup with Profiles
To give multiple users their own isolated Hermes instance (separate config, memory, skills), use [profiles](/user-guide/profiles):
```bash
# Create a profile per user
hermes profile create alice
hermes profile create bob
# Configure each profile's API server on a different port. API_SERVER_* are env
# vars (not config.yaml keys), so write them to each profile's .env:
cat >> ~/.hermes/profiles/alice/.env <<EOF
API_SERVER_ENABLED=true
API_SERVER_PORT=8643
API_SERVER_KEY=alice-secret
EOF
cat >> ~/.hermes/profiles/bob/.env <<EOF
API_SERVER_ENABLED=true
API_SERVER_PORT=8644
API_SERVER_KEY=bob-secret
EOF
# Start each profile's gateway
hermes -p alice gateway &
hermes -p bob gateway &
```
Each profile's API server automatically advertises the profile name as the model ID:
- `http://localhost:8643/v1/models` → model `alice`
- `http://localhost:8644/v1/models` → model `bob`
In Open WebUI, add each as a separate connection. The model dropdown shows `alice` and `bob` as distinct models, each backed by a fully isolated Hermes instance. See the [Open WebUI guide](/user-guide/messaging/open-webui#multi-user-setup-with-profiles) for details.
## Limitations
- **Response storage** — stored responses (for `previous_response_id`) are persisted in SQLite and survive gateway restarts. Max 100 stored responses (LRU eviction).
- **No file upload** — inline images are supported on both `/v1/chat/completions` and `/v1/responses`, but uploaded files (`file`, `input_file`, `file_id`) and non-image document inputs are not supported through the API.
- **Simple OpenAI clients still see an alias** — `/v1/models` advertises the
stable Hermes alias (`hermes-agent` or the active profile name). Richer
clients can send explicit `provider` / `model_options` overrides on requests.
## Proxy Mode
The API server also serves as the backend for **gateway proxy mode**. When another Hermes gateway instance is configured with `GATEWAY_PROXY_URL` pointing at this API server, it forwards all messages here instead of running its own agent. This enables split deployments — for example, a Docker container handling Matrix E2EE that relays to a host-side agent.
See [Matrix Proxy Mode](/user-guide/messaging/matrix#proxy-mode-e2ee-on-macos) for the full setup guide.
website/docs/user-guide/features/batch-processing.md
+230
View File
@@ -0,0 +1,230 @@
---
sidebar_position: 12
title: "Batch Processing"
description: "Generate agent trajectories at scale — parallel processing, checkpointing, and toolset distributions"
---
# Batch Processing
Batch processing lets you run the Hermes agent across hundreds or thousands of prompts in parallel, generating structured trajectory data. This is primarily used for **training data generation** — producing ShareGPT-format trajectories with tool usage statistics that can be used for fine-tuning or evaluation.
## Overview
The batch runner (`batch_runner.py`) processes a JSONL dataset of prompts, running each through a full agent session with tool access. Each prompt gets its own isolated environment. The output is structured trajectory data with full conversation history, tool call statistics, and reasoning coverage metrics.
## Quick Start
```bash
# Basic batch run
python batch_runner.py \
--dataset_file=data/prompts.jsonl \
--batch_size=10 \
--run_name=my_first_run \
--model=anthropic/claude-sonnet-4.6 \
--num_workers=4
# Resume an interrupted run
python batch_runner.py \
--dataset_file=data/prompts.jsonl \
--batch_size=10 \
--run_name=my_first_run \
--resume
# List available toolset distributions
python batch_runner.py --list_distributions
```
:::tip Predictable cost at scale
Batch runs spin up many concurrent agent sessions, each making model calls and tool calls. A [Nous Portal](/user-guide/features/tool-gateway) subscription bundles model access plus web search, image gen, TTS, and cloud browsers under one bill — useful when you want stable cost-per-trajectory without juggling rate limits across five vendor accounts. Set up with `hermes setup --portal`, then point `--model` at a Nous model.
:::
## Dataset Format
The input dataset is a JSONL file (one JSON object per line). Each entry must have a `prompt` field:
```jsonl
{"prompt": "Write a Python function that finds the longest palindromic substring"}
{"prompt": "Create a REST API endpoint for user authentication using Flask"}
{"prompt": "Debug this error: TypeError: cannot unpack non-iterable NoneType object"}
```
Entries can optionally include:
- `image` or `docker_image`: A container image to use for this prompt's sandbox (works with Docker, Modal, and Singularity backends)
- `cwd`: Working directory override for the task's terminal session
## Configuration Options
| Parameter | Default | Description |
|-----------|---------|-------------|
| `--dataset_file` | (required) | Path to JSONL dataset |
| `--batch_size` | (required) | Prompts per batch |
| `--run_name` | (required) | Name for this run (used for output dir and checkpointing) |
| `--distribution` | `"default"` | Toolset distribution to sample from |
| `--model` | `claude-sonnet-4.6` | Model to use |
| `--base_url` | `https://openrouter.ai/api/v1` | API base URL |
| `--api_key` | (env var) | API key for model |
| `--max_turns` | `10` | Maximum tool-calling iterations per prompt |
| `--num_workers` | `4` | Parallel worker processes |
| `--resume` | `false` | Resume from checkpoint |
| `--verbose` | `false` | Enable verbose logging |
| `--max_samples` | all | Only process first N samples from dataset |
| `--max_tokens` | model default | Maximum tokens per model response |
### Provider Routing (OpenRouter)
| Parameter | Description |
|-----------|-------------|
| `--providers_allowed` | Comma-separated providers to allow (e.g., `"anthropic,openai"`) |
| `--providers_ignored` | Comma-separated providers to ignore (e.g., `"together,deepinfra"`) |
| `--providers_order` | Comma-separated preferred provider order |
| `--provider_sort` | Sort by `"price"`, `"throughput"`, or `"latency"` |
### Reasoning Control
| Parameter | Description |
|-----------|-------------|
| `--reasoning_effort` | Reasoning effort: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`, `ultra` |
| `--reasoning_disabled` | Completely disable reasoning/thinking tokens |
### Advanced Options
| Parameter | Description |
|-----------|-------------|
| `--ephemeral_system_prompt` | System prompt used during execution but NOT saved to trajectories |
| `--log_prefix_chars` | Characters to show in log previews (default: 100) |
| `--prefill_messages_file` | Path to JSON file with prefill messages for few-shot priming |
## Toolset Distributions
Each prompt gets a randomly sampled set of toolsets from a **distribution**. This ensures training data covers diverse tool combinations. Use `--list_distributions` to see all available distributions.
In the current implementation, distributions assign a probability to **each individual toolset**. The sampler flips each toolset independently, then guarantees that at least one toolset is enabled. This is different from a hand-authored table of prebuilt combinations.
## Output Format
All output goes to `data/<run_name>/`:
```text
data/my_run/
├── trajectories.jsonl # Combined final output (all batches merged)
├── batch_0.jsonl # Individual batch results
├── batch_1.jsonl
├── ...
├── checkpoint.json # Resume checkpoint
└── statistics.json # Aggregate tool usage stats
```
### Trajectory Format
Each line in `trajectories.jsonl` is a JSON object:
```json
{
"prompt_index": 42,
"conversations": [
{"from": "human", "value": "Write a function..."},
{"from": "gpt", "value": "I'll create that function...",
"tool_calls": [...]},
{"from": "tool", "value": "..."},
{"from": "gpt", "value": "Here's the completed function..."}
],
"metadata": {
"batch_num": 2,
"timestamp": "2026-01-15T10:30:00",
"model": "anthropic/claude-sonnet-4.6"
},
"completed": true,
"partial": false,
"api_calls": 3,
"toolsets_used": ["terminal", "file"],
"tool_stats": {
"terminal": {"count": 2, "success": 2, "failure": 0},
"read_file": {"count": 1, "success": 1, "failure": 0}
},
"tool_error_counts": {
"terminal": 0,
"read_file": 0
}
}
```
The `conversations` field uses a ShareGPT-like format with `from` and `value` fields. Tool stats are normalized to include all possible tools with zero defaults, ensuring consistent schema across entries for HuggingFace datasets compatibility.
## Checkpointing
The batch runner has robust checkpointing for fault tolerance:
- **Checkpoint file:** Saved after each batch completes, tracking which prompt indices are done
- **Content-based resume:** On `--resume`, the runner scans existing batch files and matches completed prompts by their actual text content (not just indices), enabling recovery even if the dataset order changes
- **Failed prompts:** Only successfully completed prompts are marked as done — failed prompts will be retried on resume
- **Batch merging:** On completion, all batch files (including from previous runs) are merged into a single `trajectories.jsonl`
### How Resume Works
1. Scan all `batch_*.jsonl` files for completed prompts (by content matching)
2. Filter the dataset to exclude already-completed prompts
3. Re-batch the remaining prompts
4. Process only the remaining prompts
5. Merge all batch files (old + new) into final output
## Quality Filtering
The batch runner applies automatic quality filtering:
- **No-reasoning filter:** Samples where zero assistant turns contain reasoning (no `<REASONING_SCRATCHPAD>` or native thinking tokens) are discarded
- **Corrupted entry filter:** Entries with hallucinated tool names (not in the valid tool list) are filtered out during the final merge
- **Reasoning statistics:** Tracks percentage of turns with/without reasoning across the entire run
## Statistics
After completion, the runner prints comprehensive statistics:
- **Tool usage:** Call counts, success/failure rates per tool
- **Reasoning coverage:** Percentage of assistant turns with reasoning
- **Samples discarded:** Count of samples filtered for lacking reasoning
- **Duration:** Total processing time
Statistics are also saved to `statistics.json` for programmatic analysis.
## Use Cases
### Training Data Generation
Generate diverse tool-use trajectories for fine-tuning:
```bash
python batch_runner.py \
--dataset_file=data/coding_prompts.jsonl \
--batch_size=20 \
--run_name=coding_v1 \
--model=anthropic/claude-sonnet-4.6 \
--num_workers=8 \
--distribution=default \
--max_turns=15
```
### Model Evaluation
Evaluate how well a model uses tools across standardized prompts:
```bash
python batch_runner.py \
--dataset_file=data/eval_suite.jsonl \
--batch_size=10 \
--run_name=eval_gpt4 \
--model=openai/gpt-4o \
--num_workers=4 \
--max_turns=10
```
### Per-Prompt Container Images
For benchmarks requiring specific environments, each prompt can specify its own container image:
```jsonl
{"prompt": "Install numpy and compute eigenvalues of a 3x3 matrix", "image": "python:3.11-slim"}
{"prompt": "Compile this Rust program and run it", "image": "rust:1.75"}
{"prompt": "Set up a Node.js Express server", "image": "node:20-alpine", "cwd": "/app"}
```
The batch runner verifies Docker images are accessible before running each prompt.
website/docs/user-guide/features/browser.md
+859
View File
@@ -0,0 +1,859 @@
---
title: Browser Automation
description: Control browsers with multiple providers, local Chromium-family browsers via CDP, or cloud browsers for web interaction, form filling, scraping, and more.
sidebar_label: Browser
sidebar_position: 5
---
# Browser Automation
Hermes Agent includes a full browser automation toolset with multiple backend options:
- **Browser Use cloud mode** via [Browser Use](https://browser-use.com) for managed Chromium with stealth, residential proxies, CAPTCHA solving, and reusable browser profiles
- **Browserbase cloud mode** via [Browserbase](https://browserbase.com) as an alternative cloud browser provider with anti-bot tooling
- **Browser Use mode** via the [Browser Use CLI 3.0](https://github.com/browser-use/browser-use), the default browser driver for local Chrome and Browser Use cloud browsers
- **Firecrawl cloud mode** via [Firecrawl](https://firecrawl.dev) for cloud browsers with built-in scraping
- **Camofox local mode** via [Camofox](https://github.com/jo-inc/camofox-browser) for local anti-detection browsing (Firefox-based fingerprint spoofing)
- **Lightpanda local engine** via [Lightpanda](https://lightpanda.io) — a headless browser built from scratch in Zig for machines; instant start up, 16x lower memory and 9x faster than Chrome. Works in Browser Use mode (Hermes spawns it, no Chromium or Node needed) and with the built-in tools (automatic Chrome fallback for actions it doesn't support yet)
- **Local Chromium-family CDP** — connect browser tools to your own Chrome, Brave, Chromium, or Edge instance using `/browser connect`
- **Local browser mode** via the `agent-browser` CLI and a local Chromium installation
In all modes, the agent can navigate websites, interact with page elements, fill forms, and extract information.
## Overview
Pages are represented as **accessibility trees** (text-based snapshots), making them ideal for LLM agents. Interactive elements get ref IDs (like `@e1`, `@e2`) that the agent uses for clicking and typing.
Key capabilities:
- **Multi-provider cloud execution** — Browser Use, Browserbase, or Firecrawl — no local browser needed
- **Local Chromium-family integration** — attach to your running Chrome, Brave, Chromium, or Edge browser via CDP for hands-on browsing
- **Cloud anti-bot support** — Browser Use Cloud includes stealth, residential proxies, and CAPTCHA solving
- **Persistent cloud profiles** — Browser Use Cloud can reuse cookies, localStorage, and saved passwords across sessions
- **Session isolation** — each task gets its own browser session
- **Automatic cleanup** — inactive sessions are closed after a timeout
- **Vision analysis** — screenshot + AI analysis for visual understanding
## Setup
:::tip Nous Subscribers
If you have a paid [Nous Portal](https://portal.nousresearch.com) subscription, you can use browser automation through the **[Tool Gateway](tool-gateway.md)** without any separate API keys. New installs can run `hermes setup --portal` to log in and turn on every gateway tool at once; existing installs can pick **Nous Subscription** as the browser provider via `hermes model` or `hermes tools`.
:::
### Browser Use cloud mode
To use Browser Use as your cloud browser provider, add:
```bash
# Add to ~/.hermes/.env
BROWSER_USE_API_KEY=***
```
Get your API key at [browser-use.com](https://browser-use.com).
Browser Use Cloud runs managed Chromium with [stealth](https://docs.browser-use.com/cloud/browser/stealth) and [residential proxies](https://docs.browser-use.com/cloud/browser/proxies) enabled by default, includes CAPTCHA solving, and supports [persistent profiles](https://docs.browser-use.com/cloud/guides/authentication) for cookies, localStorage, and saved passwords.
### Browserbase cloud mode
To use Browserbase-managed cloud browsers, add:
```bash
# Add to ~/.hermes/.env
BROWSERBASE_API_KEY=***
BROWSERBASE_PROJECT_ID=your-project-id-here
```
Get your credentials at [browserbase.com](https://browserbase.com).
:::note Selecting the provider
The `.env` keys above supply **credentials only**. The active cloud browser is chosen by the `browser.cloud_provider` selection written by `hermes tools` → Browser Automation (`browserbase`, `browser-use`, `camofox`, or `nous` for the Nous Subscription). Once a selection exists, adding or removing a key does not switch providers — and a selected provider with a missing key errors with guidance to run `hermes tools` instead of silently rerouting. Never-configured setups still autodetect from available credentials.
:::
### Browser Use mode (default)
Browser Use mode uses the [Browser Use CLI 3.0](https://github.com/browser-use/browser-use) instead of the built-in browser tools. The agent writes and executes Python in the browser to click, type, drag, scrape, and interact with webpages.
**This is the default browser mode**: when `browser.backend` is unset and the `browser-use` CLI is runnable (installed, or available through `uvx`), the agent gets the single `browser_exec` tool. If the CLI can't run, Hermes falls back to the built-in browser tools automatically.
The mode is a **driver** that composes with your configured browser backend: it drives your local Chrome, a Nous-subscription cloud browser, Browserbase, Firecrawl, or Browser Use cloud browsers — whichever browser source is selected in `hermes tools` → Browser Automation. The one exception is Camofox, which has no CDP endpoint for the harness to attach to; Camofox setups automatically keep the built-in browser tools.
**Concurrent sessions:** `browser_exec` accepts a `session=<name>` argument that isolates browser work per name on every backend. Each name gets its own harness daemon (its own IPC socket, log, and state), and on cloud backends its own browser — so parallel subagents or simultaneous chats no longer clobber a single shared connection. Omitting `session` uses the shared default daemon, which is fine for one-at-a-time browsing.
To opt out and force the built-in browser tools, use `/browser use off`, or:
```yaml
# Add to ~/.hermes/config.yaml
browser:
backend: "off"
```
(`backend: "browser-use"` remains valid to force the mode explicitly.)
Browser Use's own cloud browsers need `browser-use auth login` or `BROWSER_USE_API_KEY`; other browser sources use their existing credentials unchanged.
:::note
Because Browser Use mode executes model-written Python on your machine, the
`browser_exec` tool is only offered to sessions that also have terminal
access. Platforms configured without the terminal toolset (e.g. a locked-down
messaging surface) keep the default browser tools instead.
:::
### Firecrawl cloud mode
To use Firecrawl as your cloud browser provider, add:
```bash
# Add to ~/.hermes/.env
FIRECRAWL_API_KEY=fc-***
```
Get your API key at [firecrawl.dev](https://firecrawl.dev). Then select Firecrawl as your browser provider:
```bash
hermes setup tools
# → Browser Automation → Firecrawl
```
Optional settings:
```bash
# Self-hosted Firecrawl instance (default: https://api.firecrawl.dev)
FIRECRAWL_API_URL=http://localhost:3002
# Session TTL in seconds (default: 300)
FIRECRAWL_BROWSER_TTL=600
```
### Hybrid routing: cloud for public URLs, local for LAN/localhost
When a cloud provider is configured, Hermes auto-spawns a **local Chromium sidecar**
for URLs that resolve to a private/loopback/LAN address (`localhost`, `127.0.0.1`,
`192.168.x.x`, `10.x.x.x`, `172.16-31.x.x`, `*.local`, `*.lan`, `*.internal`,
IPv6 loopback `::1`, link-local `169.254.x.x`). Public URLs continue to use the
cloud provider in the same conversation.
This solves the common "I'm developing locally but using Browserbase" workflow —
the agent can screenshot your dashboard at `http://localhost:3000` AND scrape
`https://github.com` without you switching providers or disabling the SSRF guard.
The cloud provider never sees the private URL.
The feature is **on by default**. To disable it (all URLs go to the configured
cloud provider, as before):
```yaml
# ~/.hermes/config.yaml
browser:
cloud_provider: browserbase
auto_local_for_private_urls: false
```
With auto-routing disabled, private URLs are rejected with
`"Blocked: URL targets a private or internal address"` unless you also set
`browser.allow_private_urls: true` (which lets the cloud provider attempt them —
usually won't work since Browserbase etc. can't reach your LAN).
Requirements: the local sidecar uses the same `agent-browser` CLI as pure local
mode, so you need it installed (`hermes setup tools → Browser Automation`
auto-installs it). Post-navigation redirects from a public URL onto a private
address are still blocked (you can't use a redirect-to-internal trick to reach
your LAN through the public path).
### Real profile browsing (use your own logins)
By default, local browsing runs in a clean, throwaway profile — the agent is
logged into nothing. Turn on **real profile browsing** to let the agent browse
as *you*, with your existing logins and cookies:
```yaml
# ~/.hermes/config.yaml
browser:
use_real_profile: true
```
When enabled, Hermes copies your default browser's **active** profile — the one
you actually browse (`Local State → profile.last_used`), with its cookies, saved
logins, and preferences — into a managed snapshot under
`~/.hermes/browser-profile/<browser>/`, then launches your **real browser
binary** on that snapshot and attaches its browsing engine to it. Launching the
real binary (instead of a bundled Chromium with mock-keychain switches) is what
keeps OS-encrypted cookies decryptable — on macOS, Chrome cookies are encrypted
through the Keychain, and a mock-keychain launch would silently drop every one
of them, opening signed out. Your live browser profile is **never opened
directly**: the
snapshot is a separate directory, so it doesn't fight your running browser for
the profile lock and it sidesteps Chrome 136+'s block on remote-debugging the
default profile directory. The auth files (cookies/logins/preferences) are
re-synced from your real profile whenever a fresh session is launched, so logins
you do in your own browser show up in the agent's session. Only the active
profile is copied — other Chrome profiles are never snapshotted.
The snapshot browser runs **headless** — it drives your profile in the
background with no visible window and never steals focus, so you can keep
working while the agent tweets, fills forms, or scrapes on your behalf.
(Headless here uses Chrome's *new* headless mode, which reads your normal
cookie store, so your logins still load.) If you'd rather watch it work, the
same [headed-mode](#headed-mode-visible-browser-window) toggle applies —
`browser.headed: true` (or `AGENT_BROWSER_HEADED=1`) opens a visible window for
real-profile browsing too. On a display-less host (servers, CI) it always runs
headless regardless.
If your browser has several profiles (say a work profile and a personal one)
and you don't want "whichever profile you touched last" deciding the agent's
identity, pin the snapshot source explicitly:
```yaml
# ~/.hermes/config.yaml
browser:
use_real_profile: true
real_profile_pin: "Profile 2" # directory name under the browser's user-data dir
```
A pin naming a profile directory that doesn't exist fails closed with a
fixable message — it never silently falls back to the last-used profile.
When you turn the toggle back off, Hermes deletes the snapshot store
(`~/.hermes/browser-profile/`) on the next browser use, so the copied
credentials don't linger after you revoke consent.
:::note Windows: the browser must be fully closed
On Windows a running Chrome/Edge/Brave holds its cookie and login databases with
an exclusive (deny-all) lock, so Hermes cannot copy them while the browser is
open — it fails fast with a "fully quit the browser and retry" message rather
than hang or produce a signed-out session. Real-profile browsing on Windows
therefore requires the browser **fully quit**, including any background/tray
instance (Chrome's "continue running background apps when closed" keeps a
`chrome.exe` alive after you close the window). macOS and Linux can copy the
profile while the browser is running.
Set `browser.real_profile_autoclose: true` to let Hermes **offer to close the
browser for you** when it's holding the profile. Even with this on, Hermes never
closes it automatically — when the profile is locked it always stops and the
agent asks you first; only on your approval does it run `hermes browser
close-profile` (terminates the browser process tree bound to that profile,
losing unsaved tabs), then retries. If the profile is still locked after that
(e.g. a background/tray instance relaunched), Hermes stays blocked and tells you
to fully quit the browser — it won't loop or kill again on its own.
:::
- **Supported browsers:** Chrome, Edge, Brave, Brave Origin, Chromium (whichever is your OS
default). A non-Chromium default (e.g. Firefox) fails closed with a clear
message rather than guessing.
- **Works on any backend.** On a local backend it's automatic once the toggle
is on. Under a **cloud** browser backend, the agent can still open a
real-profile local session on demand via the `browser_exec` tool's `local`
argument (the tool only exposes that argument when this toggle is on) — the
cloud backend keeps serving everything else.
- **Security framing:** this is a consent-gated convenience, not an isolation
boundary. A page the agent visits runs with your real logins, so only enable
it when you want the agent acting as you. Off by default.
- **Desktop:** toggle it in **Capabilities → Tools → Browser → Use My Real
Browser Profile** (the switch sits above the backend options), or in
Settings → Config under the `browser` section.
### Camofox local mode
[Camofox](https://github.com/jo-inc/camofox-browser) is a self-hosted Node.js server wrapping Camoufox (a Firefox fork with C++ fingerprint spoofing). It provides local anti-detection browsing without cloud dependencies.
```bash
# Clone the Camofox browser server first
git clone https://github.com/jo-inc/camofox-browser
cd camofox-browser
# Build and start with Docker using the default container settings
# (auto-detects arch: aarch64 on M1/M2, x86_64 on Intel)
make up
# Stop and remove the default container
make down
# Force a clean rebuild (for example, after upgrading VERSION/RELEASE)
make reset
# Just download binaries without building
make fetch
# Override arch or version explicitly
make up ARCH=x86_64
make up VERSION=135.0.1 RELEASE=beta.24
```
`make up` starts the default container immediately. If you want custom runtime settings such as a larger Node heap, VNC, or a persistent profile directory, build the image first and then run it yourself:
```bash
# Build the image without starting the default container
make build
# Start with persistence, VNC live view, and a larger Node heap
mkdir -p ~/.camofox-docker
docker run -d \
--name camofox-browser \
--restart unless-stopped \
-p 9377:9377 \
-p 6080:6080 \
-p 5901:5900 \
-e CAMOFOX_PORT=9377 \
-e ENABLE_VNC=1 \
-e VNC_BIND=0.0.0.0 \
-e VNC_RESOLUTION=1920x1080 \
-e MAX_OLD_SPACE_SIZE=2048 \
-v ~/.camofox-docker:/root/.camofox \
camofox-browser:135.0.1-aarch64
```
With VNC enabled, the browser runs in headed mode and can be watched live in your browser at `http://localhost:6080` (noVNC). You can also connect a native VNC client to `localhost:5901`.
If you already ran `make up`, stop and remove that default container before starting the custom one:
```bash
make down
# then run the custom docker run command above
```
Then set in `~/.hermes/.env`:
```bash
CAMOFOX_URL=http://localhost:9377
```
If Camofox is running in Docker and you want it to open web apps served from the host machine, enable loopback rewriting. `CAMOFOX_URL` should still point at the host-published control API, but page URLs such as `http://127.0.0.1:3000` must be opened from inside the container as `http://host.docker.internal:3000`:
```yaml
# ~/.hermes/config.yaml
browser:
camofox:
rewrite_loopback_urls: true
loopback_host_alias: host.docker.internal # default; use a LAN IP if needed
```
Equivalent env vars:
```bash
CAMOFOX_REWRITE_LOOPBACK_URLS=true
CAMOFOX_LOOPBACK_HOST_ALIAS=host.docker.internal
```
The rewrite only applies to page navigation URLs with loopback hosts (`localhost`, `127.0.0.1`, `::1`). It does not change `CAMOFOX_URL`. Leave it disabled for non-Docker Camofox installs, where the browser already runs on the host and loopback URLs are correct.
Or configure via `hermes tools` → Browser Automation → Camofox.
Camofox is selected like any other browser backend: pick **Camofox** in `hermes tools` → Browser Automation, which writes `browser.cloud_provider: camofox` to `config.yaml`. `CAMOFOX_URL` is only the server address — setting it no longer selects the backend by itself once a browser selection exists (never-configured setups still autodetect it).
#### Persistent browser sessions
By default, each Camofox session gets a random identity — cookies and logins don't survive across agent restarts. To enable persistent browser sessions, add the following to `~/.hermes/config.yaml`:
```yaml
browser:
camofox:
managed_persistence: true
```
Then fully restart Hermes so the new config is picked up.
:::warning Nested path matters
Hermes reads `browser.camofox.managed_persistence`, **not** a top-level `managed_persistence`. A common mistake is writing:
```yaml
# ❌ Wrong — Hermes ignores this
managed_persistence: true
```
If the flag is placed at the wrong path, Hermes silently falls back to a random ephemeral `userId` and your login state will be lost on every session.
:::
##### What Hermes does
- Sends a deterministic profile-scoped `userId` to Camofox so the server can reuse the same Firefox profile across sessions.
- Skips server-side context destruction on cleanup, so cookies and logins survive between agent tasks.
- Scopes the `userId` to the active Hermes profile, so different Hermes profiles get different browser profiles (profile isolation).
##### What Hermes does not do
- It does not force persistence on the Camofox server. Hermes only sends a stable `userId`; the server must honor it by mapping that `userId` to a persistent Firefox profile directory.
- If your Camofox server build treats every request as ephemeral (e.g. always calls `browser.newContext()` without loading a stored profile), Hermes cannot make those sessions persist. Make sure you are running a Camofox build that implements userId-based profile persistence.
##### Verify it's working
1. Start Hermes and your Camofox server.
2. Open Google (or any login site) in a browser task and sign in manually.
3. End the browser task normally.
4. Start a new browser task.
5. Open the same site again — you should still be signed in.
If step 5 logs you out, the Camofox server isn't honoring the stable `userId`. Double-check your config path, confirm you fully restarted Hermes after editing `config.yaml`, and verify your Camofox server version supports persistent per-user profiles.
##### Where state lives
Hermes derives the stable `userId` from the profile-scoped directory `~/.hermes/browser_auth/camofox/` (or the equivalent under `$HERMES_HOME` for non-default profiles). The actual browser profile data lives on the Camofox server side, keyed by that `userId`. To fully reset a persistent profile, clear it on the Camofox server and remove the corresponding Hermes profile's state directory.
#### Externally managed Camofox sessions
When another app drives the visible Camofox browser (a desktop assistant, a custom integration, another agent), configure Hermes to operate inside that same identity instead of spawning its own isolated profile.
Three knobs control the behavior:
| Setting | Env var | Effect |
|---------|---------|--------|
| `browser.camofox.user_id` | `CAMOFOX_USER_ID` | Camofox `userId` Hermes uses when creating tabs. Setting this opts the session into "externally managed" mode. |
| `browser.camofox.session_key` | `CAMOFOX_SESSION_KEY` | `sessionKey` (a.k.a. `listItemId`) sent on tab creation. Used to match an existing tab during adoption. Defaults to a per-task value if unset. |
| `browser.camofox.adopt_existing_tab` | `CAMOFOX_ADOPT_EXISTING_TAB` | When true, Hermes calls `GET /tabs?userId=<user_id>` on first use and reuses an existing tab before creating a new one. |
Env vars take precedence over `config.yaml`. Either form works:
```yaml
browser:
camofox:
user_id: shared-camofox
session_key: visible-tab
adopt_existing_tab: true
```
```bash
CAMOFOX_USER_ID=shared-camofox
CAMOFOX_SESSION_KEY=visible-tab
CAMOFOX_ADOPT_EXISTING_TAB=true
```
**What changes when `user_id` is set:**
- Hermes skips destructive cleanup at task end (same as `managed_persistence: true`). The other app's tab/cookies/profile survive.
- Hermes does **not** call `DELETE /sessions/<user_id>` — that endpoint wipes all user data, so it would nuke the external app's session if it fired.
**How tab adoption works (when `adopt_existing_tab: true`):**
1. On the first browser tool call after a process start, Hermes issues `GET /tabs?userId=<user_id>` (5-second timeout).
2. If any tab in the response has `listItemId == session_key`, Hermes adopts the most recently created one in that group.
3. Otherwise, Hermes adopts the most recently created tab for the user (any `listItemId`).
4. If no tabs exist or the request fails, Hermes falls back to creating a new tab on the next operation.
Adoption only fires until `tab_id` is populated for the session. If the external app closes the adopted tab mid-run, the next browser tool call will surface a Camofox error — Hermes does not re-poll for a fresh tab on every call.
**Picking `session_key`:** if you want Hermes to reliably attach to a *specific* existing tab, set `session_key` to the `listItemId` the external app used when creating it. If you leave `session_key` unset and only set `user_id`, Hermes generates a per-task `session_key` (`task_<id>`) — Hermes will share cookies and the profile with the external app, but will open its own tab alongside instead of reusing one.
**Concurrency note:** the external app and Hermes can drive the same Camofox `userId` simultaneously, but Camofox does not coordinate per-tab focus between clients. Coordinate ownership at the application layer (e.g. the external app pauses while Hermes runs).
#### VNC live view
When Camofox runs in headed mode (with a visible browser window), it exposes a VNC port in its health check response. Hermes automatically discovers this and includes the VNC URL in navigation responses, so the agent can share a link for you to watch the browser live.
### Lightpanda local engine
[Lightpanda](https://lightpanda.io) is an open-source headless browser written from scratch. It starts instantly, runs 9x faster and uses 16x less memory than Chrome, which matters for agents that live on small VMs for long stretches.
Lightpanda is a **local engine** (a browser source, like "Local Browser"), not a cloud provider. Install the binary and put it on your `PATH` (see the [Lightpanda installation guide](https://lightpanda.io/docs/run-locally/installation/one-liner)), then pick **Lightpanda** in `hermes tools` → Browser Automation, or set:
```yaml
# Add to ~/.hermes/config.yaml
browser:
cloud_provider: local
engine: lightpanda
```
Or via environment variable:
```bash
AGENT_BROWSER_ENGINE=lightpanda
```
The engine works with both browser drivers:
- **Browser Use mode (the default).** Hermes launches `lightpanda serve --host 127.0.0.1 --port <free>` itself — one process per `browser_exec` session name (or per task) — and points the Browser Use CLI at it. No Chromium, Playwright or Node.js is needed. The process is reaped after `browser.inactivity_timeout`, on exit, and by the orphan sweep if Hermes crashes. All of these processes share one on-disk HTTP cache at `$HERMES_HOME/cache/browser-use/lightpanda/http-cache`, so repeat visits skip re-downloading assets. Hermes passes the cache flag only when the installed Lightpanda supports it (0.3.x+); older binaries simply run without a cache. To clear it, stop your Lightpanda sessions first, then delete that directory. Lightpanda has no graphical renderer, so `capture_screenshot()` is unavailable and the tool description tells the model to work text-first; it also holds one page per session, so the model is told to call `new_tab()` once and `goto_url()` afterwards (tracked upstream in [lightpanda-io/browser#1962](https://github.com/lightpanda-io/browser/issues/1962)).
- **Built-in browser tools** (`/browser use off`). Hermes drives Lightpanda through `agent-browser --engine lightpanda` over CDP, the same way it drives local Chrome, with **automatic Chrome fallback**: Lightpanda handles the actions it supports (navigate, snapshot, click, type, scroll, back, press, eval) and Hermes transparently retries on Chrome for anything it doesn't. Screenshots and `browser_vision` are routed straight to Chrome.
**When the engine is ignored.** `browser.engine` is the lowest-precedence browser setting: a cloud provider (including the Nous subscription browser — and on never-configured setups, any `BROWSERBASE_API_KEY` / `BROWSER_USE_API_KEY` in `~/.hermes/.env` auto-selects one), Camofox, a `browser.cdp_url` / `/browser connect` override, or `browser.use_real_profile` all take precedence. Picking Lightpanda in `hermes tools` writes `cloud_provider: local` for you; `/browser status` and `hermes doctor` report when the engine is configured but shadowed, and by what.
### Local Chromium-family browser via CDP (`/browser connect`)
Instead of a cloud provider, you can attach Hermes browser tools to your own running Chrome, Brave, Chromium, or Edge instance via the Chrome DevTools Protocol (CDP). This is useful when you want to see what the agent is doing in real-time, interact with pages that require your own cookies/sessions, or avoid cloud browser costs.
:::note
`/browser connect` is an **interactive-CLI slash command** — it is not dispatched by the gateway. If you try to run it inside a WebUI, Telegram, Discord, or other gateway chat, the message will be sent to the agent as plain text and the command will not execute. Start Hermes from the terminal (`hermes` or `hermes chat`) and issue `/browser connect` there.
:::
In the CLI, use:
```
/browser connect # Auto-launch/connect to a local Chromium-family browser at http://127.0.0.1:9222
/browser connect ws://host:port # Connect to a specific CDP endpoint
/browser status # Check current connection
/browser disconnect # Detach and return to cloud/local mode
```
If a browser isn't already running with remote debugging, Hermes will attempt to auto-launch a supported Chromium-family browser with `--remote-debugging-port=9222`. Detection includes Brave, Brave Origin/Nightly, Google Chrome, Chromium, and Microsoft Edge, with common Linux install paths and binary names such as `brave-origin`, `brave-origin-nightly`, `/opt/brave.com/brave-origin/brave-origin`, `/opt/brave.com/brave-origin-nightly/brave-origin`, `/opt/brave-bin/brave`, and `/snap/bin/brave`.
:::tip
To start a Chromium-family browser manually with CDP enabled, use a dedicated user-data-dir so the debug port actually comes up even if the browser is already running with your normal profile:
```bash
# Linux — Brave
brave-browser \
--remote-debugging-port=9222 \
--user-data-dir=$HOME/.hermes/chrome-debug \
--no-first-run \
--no-default-browser-check &
# Linux — Google Chrome
google-chrome \
--remote-debugging-port=9222 \
--user-data-dir=$HOME/.hermes/chrome-debug \
--no-first-run \
--no-default-browser-check &
# macOS — Brave
"/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" \
--remote-debugging-port=9222 \
--user-data-dir="$HOME/.hermes/chrome-debug" \
--no-first-run \
--no-default-browser-check &
# macOS — Google Chrome
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
--remote-debugging-port=9222 \
--user-data-dir="$HOME/.hermes/chrome-debug" \
--no-first-run \
--no-default-browser-check &
```
Then launch the Hermes CLI and run `/browser connect`.
**Why `--user-data-dir`?** Without it, launching a Chromium-family browser while a regular instance is already running typically opens a new window on the existing process — and that existing process was not started with `--remote-debugging-port`, so port 9222 never opens. A dedicated user-data-dir forces a fresh browser process where the debug port actually listens. `--no-first-run --no-default-browser-check` skips the first-launch wizard for the fresh profile.
**Chrome 136+ makes the dedicated profile mandatory.** As a security hardening change, Chrome 136 and later silently refuse to open the remote debugging port when `--remote-debugging-port` is combined with the *default* user-data-dir — even from a cold start with no other Chrome running. The browser launches normally but nothing ever listens on 9222, so `/browser connect` (and any manual `curl http://127.0.0.1:9222/json/version`) fails with connection refused. There is no error message. The fix is exactly the commands above: always pass a `--user-data-dir` pointing somewhere other than your default profile directory (e.g. `$HOME/.hermes/chrome-debug`). This applies to Chrome, Chromium, Edge, and Brave builds that have picked up the change.
:::
When connected via CDP, all browser tools (`browser_navigate`, `browser_click`, etc.) operate on your live browser instance instead of spinning up a cloud session.
### WSL2 + Windows Chrome: prefer MCP over `/browser connect`
If Hermes runs inside WSL2 but the Chrome window you want to control runs on the Windows host, `/browser connect` is often not the best path.
Why:
- `/browser connect` expects Hermes itself to reach a usable CDP endpoint
- modern Chrome live-debugging sessions often expose a host-local endpoint that is not directly reachable from WSL the same way a classic `9222` port is
- even when Windows Chrome is debuggable, the cleanest integration is often to let a Windows-side browser MCP server attach to Chrome and let Hermes talk to that MCP server
For that setup, prefer `chrome-devtools-mcp` through Hermes MCP support.
See the MCP guide for the practical setup:
- [Use MCP with Hermes](../../guides/use-mcp-with-hermes.md#wsl2-bridge-hermes-in-wsl-to-windows-chrome)
### Local browser mode
If you do **not** set any cloud credentials and don't use `/browser connect`, Hermes can still use the browser tools through a local Chromium install driven by `agent-browser`.
### Optional Environment Variables
```bash
# Residential proxies for better CAPTCHA solving (default: "true")
BROWSERBASE_PROXIES=true
# Advanced stealth with custom Chromium — requires Scale Plan (default: "false")
BROWSERBASE_ADVANCED_STEALTH=false
# Session reconnection after disconnects — requires paid plan (default: "true")
BROWSERBASE_KEEP_ALIVE=true
# Custom session timeout in seconds (max 21600 = 6 hours) (default: project default)
# Examples: 600 (10min), 1800 (30min), 21600 (6h max)
BROWSERBASE_SESSION_TIMEOUT=1800
# Inactivity timeout before auto-cleanup in seconds (default: 120)
BROWSER_INACTIVITY_TIMEOUT=120
# Local browser engine. Equivalent to browser.engine in config.yaml. In
# Browser Use mode (default) "lightpanda" makes Hermes spawn `lightpanda serve`;
# with the built-in tools it is passed to agent-browser as --engine.
# auto — Chrome (default)
# lightpanda — Lightpanda
# chrome — force Chrome explicitly
AGENT_BROWSER_ENGINE=auto
# Extra Chromium launch flags (comma- or newline-separated). Hermes auto-injects
# `--no-sandbox,--disable-dev-shm-usage` when it detects root or AppArmor-restricted
# unprivileged user namespaces (Ubuntu 23.10+, DGX Spark, many container images),
# so most users don't need to set this. Set it manually only if you need a flag
# Hermes doesn't add automatically; setting it disables the auto-injection.
AGENT_BROWSER_ARGS=--no-sandbox
```
### Install agent-browser CLI
You don't need to install anything — `agent-browser` resolves automatically via
`npx agent-browser` on first browser-tool use. To avoid the one-time npx fetch,
you can install it globally ahead of time (optional):
```bash
npm install -g agent-browser
```
:::info
The `browser` toolset must be included in your config's `toolsets` list or enabled via `hermes config set toolsets '["hermes-cli", "browser"]'`.
:::
## Available Tools
### `browser_navigate`
Navigate to a URL. Must be called before any other browser tool. Initializes the Browserbase session.
```
Navigate to https://github.com/NousResearch
```
:::tip
For simple information retrieval, prefer `web_search` or `web_extract` — they are faster and cheaper. Use browser tools when you need to **interact** with a page (click buttons, fill forms, handle dynamic content).
:::
### `browser_snapshot`
Get a text-based snapshot of the current page's accessibility tree. Returns interactive elements with ref IDs like `@e1`, `@e2` for use with `browser_click` and `browser_type`.
- **`full=false`** (default): Compact view showing only interactive elements
- **`full=true`**: Complete page content
Snapshots larger than `browser.snapshot_threshold` (default 15,000 characters — the same per-page budget as `web_extract`) are automatically truncated at line boundaries; no LLM summarization is involved. When that happens, the complete snapshot is saved to `~/.hermes/cache/web/` and the tool output includes the file path plus a ready-to-use `read_file` call, so the agent can page through the full accessibility tree — including element refs beyond the cut — without re-snapshotting.
Increase the threshold for long pages where more source content should reach the agent inline:
```yaml
# ~/.hermes/config.yaml
browser:
snapshot_threshold: 30000
```
You can also run `hermes config set browser.snapshot_threshold 30000`. The setting applies to both explicit `browser_snapshot` calls and the automatic snapshot returned after navigation, including the Camofox backend (minimum 1000). Restart the current Hermes session after changing it so the browser config cache reloads.
### `browser_click`
Click an element identified by its ref ID from the snapshot.
```
Click @e5 to press the "Sign In" button
```
### `browser_type`
Type text into an input field. Clears the field first, then types the new text.
```
Type "hermes agent" into the search field @e3
```
### `browser_scroll`
Scroll the page up or down to reveal more content.
```
Scroll down to see more results
```
### `browser_press`
Press a keyboard key. Useful for submitting forms or navigation.
```
Press Enter to submit the form
```
Supported keys: `Enter`, `Tab`, `Escape`, `ArrowDown`, `ArrowUp`, and more.
### `browser_back`
Navigate back to the previous page in browser history.
### `browser_get_images`
List all images on the current page with their URLs and alt text. Useful for finding images to analyze.
### `browser_vision`
Take a screenshot and analyze it with vision AI. Use this when text snapshots don't capture important visual information — especially useful for CAPTCHAs, complex layouts, or visual verification challenges.
The screenshot is saved persistently and the file path is returned alongside the AI analysis. On messaging platforms (Telegram, Discord, Slack, WhatsApp), you can ask the agent to share the screenshot — it will be sent as a native photo attachment via the `MEDIA:` mechanism.
```
What does the chart on this page show?
```
Screenshots are stored in `~/.hermes/cache/screenshots/` and automatically cleaned up after 24 hours.
### `browser_console`
Get browser console output (log/warn/error messages) and uncaught JavaScript exceptions from the current page. Essential for detecting silent JS errors that don't appear in the accessibility tree.
```
Check the browser console for any JavaScript errors
```
Use `clear=True` to clear the console after reading, so subsequent calls only show new messages.
`browser_console` also evaluates JavaScript when called with an `expression` argument — same shape as DevTools console, the result comes back parsed (JSON-serialized objects become dicts; primitive values stay primitive).
```
browser_console(expression="document.querySelector('h1').textContent")
browser_console(expression="JSON.stringify(performance.timing)")
```
When a CDP supervisor is active for the current session (typical for any session that's run `browser_navigate` against a CDP-capable backend), evaluation runs over the supervisor's persistent WebSocket — no subprocess startup cost. Falls through to the standard agent-browser CLI path otherwise. Behaviour is identical either way; only latency changes.
Evaluation is unrestricted by default — the agent can use `fetch`, read storage, query form values, and run any DOM extraction. Requests targeting private/internal addresses are still blocked on non-local backends (the SSRF guard is independent of this setting). If you browse hostile pages with a logged-in profile and want a strict denylist over sensitive JS primitives (cookies, storage, clipboard, network calls, form values), opt in with `browser.restrict_evaluate: true` in `config.yaml`. Note the denylist matches primitive *names*, so it also blocks legitimate expressions that merely contain words like `fetch` or `cookie`.
### `browser_cdp`
Raw Chrome DevTools Protocol passthrough — the escape hatch for browser operations not covered by the other tools. Use for native dialog handling, iframe-scoped evaluation, cookie/network control, or any CDP verb the agent needs.
**Only available when a CDP endpoint is reachable at session start** — meaning `/browser connect` has attached to a running Chrome, Brave, Chromium, or Edge browser, or `browser.cdp_url` is set in `config.yaml`. The default local agent-browser mode, Camofox, and cloud providers (Browserbase, Browser Use, Firecrawl) do not currently expose CDP to this tool — cloud providers have per-session CDP URLs but live-session routing is a follow-up.
**CDP method reference:** https://chromedevtools.github.io/devtools-protocol/ — the agent can `web_extract` a specific method's page to look up parameters and return shape.
Common patterns:
```
# List tabs (browser-level, no target_id)
browser_cdp(method="Target.getTargets")
# Handle a native JS dialog on a tab
browser_cdp(method="Page.handleJavaScriptDialog",
params={"accept": true, "promptText": ""},
target_id="<tabId>")
# Evaluate JS in a specific tab
browser_cdp(method="Runtime.evaluate",
params={"expression": "document.title", "returnByValue": true},
target_id="<tabId>")
# Get all cookies
browser_cdp(method="Network.getAllCookies")
```
Browser-level methods (`Target.*`, `Browser.*`, `Storage.*`) omit `target_id`. Page-level methods (`Page.*`, `Runtime.*`, `DOM.*`, `Emulation.*`) require a `target_id` from `Target.getTargets`. Each stateless call is independent — sessions do not persist between calls.
**Cross-origin iframes:** pass `frame_id` (from `browser_snapshot.frame_tree.children[]` where `is_oopif=true`) to route the CDP call through the supervisor's live session for that iframe. This is how `Runtime.evaluate` inside a cross-origin iframe works on Browserbase, where stateless CDP connections would hit signed-URL expiry. Example:
```
browser_cdp(
method="Runtime.evaluate",
params={"expression": "document.title", "returnByValue": True},
frame_id="<frame_id from browser_snapshot>",
)
```
Same-origin iframes don't need `frame_id` — use `document.querySelector('iframe').contentDocument` from a top-level `Runtime.evaluate` instead.
### `browser_dialog`
Responds to a native JS dialog (`alert` / `confirm` / `prompt` / `beforeunload`). Before this tool existed, dialogs would silently block the page's JavaScript thread and subsequent `browser_*` calls would hang or throw; now the agent sees pending dialogs in `browser_snapshot` output and responds explicitly.
**Workflow:**
1. Call `browser_snapshot`. If a dialog is blocking the page, it shows up as `pending_dialogs: [{"id": "d-1", "type": "alert", "message": "..."}]`.
2. Call `browser_dialog(action="accept")` or `browser_dialog(action="dismiss")`. For `prompt()` dialogs, pass `prompt_text="..."` to supply the response.
3. Re-snapshot — `pending_dialogs` is empty; the page's JS thread has resumed.
**Detection happens automatically** via a persistent CDP supervisor — one WebSocket per task that subscribes to Page/Runtime/Target events. The supervisor also populates a `frame_tree` field in the snapshot so the agent can see the iframe structure of the current page, including cross-origin (OOPIF) iframes.
**Availability matrix:**
| Backend | Detection via `pending_dialogs` | Response (`browser_dialog` tool) |
|---|---|---|
| Local Chrome via `/browser connect` or `browser.cdp_url` | ✓ | ✓ full workflow |
| Browserbase | ✓ | ✓ full workflow (via injected XHR bridge) |
| Camofox / default local agent-browser | ✗ | ✗ (no CDP endpoint) |
**How it works on Browserbase.** Browserbase's CDP proxy auto-dismisses real native dialogs server-side within ~10ms, so we can't use `Page.handleJavaScriptDialog`. The supervisor injects a small script via `Page.addScriptToEvaluateOnNewDocument` that overrides `window.alert`/`confirm`/`prompt` with a synchronous XHR. We intercept those XHRs via `Fetch.enable` — the page's JS thread stays blocked on the XHR until we call `Fetch.fulfillRequest` with the agent's response. `prompt()` return values round-trip back into page JS unchanged.
**Dialog policy** is configured in `config.yaml` under `browser.dialog_policy`:
| Policy | Behavior |
|--------|----------|
| `must_respond` (default) | Capture, surface in snapshot, wait for explicit `browser_dialog()` call. Safety auto-dismiss after `browser.dialog_timeout_s` (default 300s) so a buggy agent can't stall forever. |
| `auto_dismiss` | Capture, dismiss immediately. Agent still sees the dialog in `browser_state` history but doesn't have to act. |
| `auto_accept` | Capture, accept immediately. Useful when navigating pages with aggressive `beforeunload` prompts. |
**Frame tree** inside `browser_snapshot.frame_tree` is capped to 30 frames and OOPIF depth 2 to keep payloads bounded on ad-heavy pages. A `truncated: true` flag surfaces when limits were hit; agents needing the full tree can use `browser_cdp` with `Page.getFrameTree`.
## Practical Examples
### Filling Out a Web Form
```
User: Sign up for an account on example.com with my email john@example.com
Agent workflow:
1. browser_navigate("https://example.com/signup")
2. browser_snapshot() → sees form fields with refs
3. browser_type(ref="@e3", text="john@example.com")
4. browser_type(ref="@e5", text="SecurePass123")
5. browser_click(ref="@e8") → clicks "Create Account"
6. browser_snapshot() → confirms success
```
### Researching Dynamic Content
```
User: What are the top trending repos on GitHub right now?
Agent workflow:
1. browser_navigate("https://github.com/trending")
2. browser_snapshot(full=true) → reads trending repo list
3. Returns formatted results
```
## Session Recording
Automatically record browser sessions as WebM video files:
```yaml
browser:
record_sessions: true # default: false
```
When enabled, recording starts automatically on the first `browser_navigate` and saves to `~/.hermes/browser_recordings/` when the session closes. Works in both local and cloud (Browserbase) modes. Recordings older than 72 hours are automatically cleaned up.
## Headed Mode (Visible Browser Window)
By default, the local browser runs headless. Enable headed mode to get a visible Chromium window you can watch and interact with:
```yaml
browser:
headed: true # default: false
```
Or via environment variable: `AGENT_BROWSER_HEADED=1`.
Headed mode does two things:
1. **Launches Chromium with a visible window** (passes `--headed` to agent-browser in local mode).
2. **Keeps the window open between turns.** Normally the browser session is cleaned up after every agent reply; in headed mode the per-turn cleanup is skipped so you can watch the agent work, intervene manually (sign-in challenges, CAPTCHAs), and keep login state warm across the conversation.
Idle sessions are still reaped after `browser.inactivity_timeout` (default 120s of no browser activity), and all sessions are closed on shutdown. Headed mode only affects the local browser — cloud sessions (Browserbase) are unaffected.
## Stealth Features
Browserbase provides automatic stealth capabilities:
| Feature | Default | Notes |
|---------|---------|-------|
| Basic Stealth | Always on | Random fingerprints, viewport randomization, CAPTCHA solving |
| Residential Proxies | On | Routes through residential IPs for better access |
| Advanced Stealth | Off | Custom Chromium build, requires Scale Plan |
| Keep Alive | On | Session reconnection after network hiccups |
:::note
If paid features aren't available on your plan, Hermes automatically falls back — first disabling `keepAlive`, then proxies — so browsing still works on free plans.
:::
## Session Management
- Each task gets an isolated browser session via Browserbase
- Sessions are automatically cleaned up after inactivity (default: 2 minutes)
- A background thread checks every 30 seconds for stale sessions
- Emergency cleanup runs on process exit to prevent orphaned sessions
- Sessions are released via the Browserbase API (`REQUEST_RELEASE` status)
## Limitations
- **Text-based interaction** — relies on accessibility tree, not pixel coordinates
- **Snapshot size** — large pages are truncated at `browser.snapshot_threshold` (default 15,000 characters, matching `web_extract`; no LLM summarization); the complete snapshot is saved to `~/.hermes/cache/web/` and the output points at it for `read_file` paging
- **Session timeout** — cloud sessions expire based on your provider's plan settings
- **Cost** — cloud sessions consume provider credits; sessions are automatically cleaned up when the conversation ends or after inactivity. Use `/browser connect` for free local browsing.
- **No file downloads** — cannot download files from the browser
website/docs/user-guide/features/built-in-plugins.md
+329
View File
@@ -0,0 +1,329 @@
---
sidebar_position: 12
sidebar_label: "Built-in Plugins"
title: "Built-in Plugins"
description: "Plugins shipped with Hermes Agent that run automatically via lifecycle hooks — disk-cleanup and friends"
---
# Built-in Plugins
Hermes ships a small set of plugins bundled with the repository. They live under `<repo>/plugins/<name>/` and load automatically alongside user-installed plugins in `~/.hermes/plugins/`. They use the same plugin surface as third-party plugins — hooks, tools, slash commands — just maintained in-tree.
See the [Plugins](/user-guide/features/plugins) page for the general plugin system, and [Build a Hermes Plugin](/developer-guide/plugins) to write your own.
## How discovery works
The `PluginManager` scans four sources, in order:
1. **Bundled** — `<repo>/plugins/<name>/` (what this page documents)
2. **User** — `~/.hermes/plugins/<name>/`
3. **Project** — `./.hermes/plugins/<name>/` (requires `HERMES_ENABLE_PROJECT_PLUGINS=1`)
4. **Pip entry points** — `hermes_agent.plugins`
On name collision, later sources win — a user plugin named `disk-cleanup` would replace the bundled one.
`plugins/memory/` and `plugins/context_engine/` are deliberately excluded from bundled scanning. Those directories use their own discovery paths because memory providers and context engines are single-select providers configured through `hermes memory setup` / `context.engine` in config.
## Bundled plugins are opt-in
Bundled plugins ship disabled. Discovery finds them (they appear in `hermes plugins list` and the interactive `hermes plugins` UI), but none load until you explicitly enable them:
```bash
hermes plugins enable disk-cleanup
```
Or via `~/.hermes/config.yaml`:
```yaml
plugins:
enabled:
- disk-cleanup
```
This is the same mechanism user-installed plugins use. Bundled plugins are never auto-enabled — not on fresh install, not for existing users upgrading to a newer Hermes. You always opt in explicitly.
To turn a bundled plugin off again:
```bash
hermes plugins disable disk-cleanup
# or: remove it from plugins.enabled in config.yaml
```
## Currently shipped
The repo ships these bundled plugins under `plugins/`. All are opt-in — enable them via `hermes plugins enable <name>`.
| Plugin | Kind | Purpose |
|---|---|---|
| `disk-cleanup` | hooks + slash command | Auto-track ephemeral files and clean them on session end |
| `security-guidance` | hooks | Pattern-match dangerous code on `write_file`/`patch` and append a security warning (or block) — 25 rules (Apache-2.0 fork of Anthropic's `claude-plugins-official` patterns) |
| `observability/langfuse` | hooks | Trace turns / LLM calls / tools to [Langfuse](https://langfuse.com) |
| `teams_pipeline` | standalone | Microsoft Teams meeting pipeline — Graph-backed, transcript-first meeting summaries |
| `spotify` | backend (7 tools) | Native Spotify playback, queue, search, playlists, albums, library |
| `google_meet` | standalone | Join Meet calls, live-caption transcription, optional realtime duplex audio |
| `image_gen/openai` | image backend | OpenAI `gpt-image-2` image generation backend (alternative to FAL) |
| `image_gen/openai-codex` | image backend | OpenAI image generation via Codex OAuth |
| `image_gen/xai` | image backend | xAI `grok-2-image` backend |
| `hermes-achievements` | dashboard tab | Steam-style collectible badges generated from your real Hermes session history |
| `kanban/dashboard` | dashboard tab | Kanban board UI for the multi-agent dispatcher — tasks, comments, fan-out, board switching. See [Kanban Multi-Agent](./kanban.md). |
Memory providers (`plugins/memory/*`) and context engines (`plugins/context_engine/*`) are listed separately on [Memory Providers](./memory-providers.md) — they're managed through `hermes memory` and `hermes plugins` respectively. The full per-plugin detail for the two long-running hooks-based plugins follows.
### disk-cleanup
Auto-tracks and removes ephemeral files created during sessions — test scripts, temp outputs, cron logs, stale chrome profiles — without requiring the agent to remember to call a tool.
**How it works:**
| Hook | Behaviour |
|---|---|
| `post_tool_call` | When `write_file` / `terminal` / `patch` creates a file matching `test_*`, `tmp_*`, or `*.test.*` inside `HERMES_HOME` or `/tmp/hermes-*`, track it silently as `test` / `temp` / `cron-output`. |
| `on_session_end` | If any test files were auto-tracked during the turn, run the safe `quick` cleanup and log a one-line summary. Stays silent otherwise. |
**Deletion rules:**
| Category | Threshold | Confirmation |
|---|---|---|
| `test` | every session end | Never |
| `temp` | >7 days since tracked | Never |
| `cron-output` | >14 days since tracked | Never |
| empty dirs under HERMES_HOME | always | Never |
| `research` | >30 days, beyond 10 newest | Always (deep only) |
| `chrome-profile` | >14 days since tracked | Always (deep only) |
| files >500 MB | never auto | Always (deep only) |
**Slash command** — `/disk-cleanup` available in both CLI and gateway sessions:
```
/disk-cleanup status # breakdown + top-10 largest
/disk-cleanup dry-run # preview without deleting
/disk-cleanup quick # run safe cleanup now
/disk-cleanup deep # quick + list items needing confirmation
/disk-cleanup track <path> <category> # manual tracking
/disk-cleanup forget <path> # stop tracking (does not delete)
```
**State** — everything lives at `$HERMES_HOME/disk-cleanup/`:
| File | Contents |
|---|---|
| `tracked.json` | Tracked paths with category, size, and timestamp |
| `tracked.json.bak` | Atomic-write backup of the above |
| `cleanup.log` | Append-only audit trail of every track / skip / reject / delete |
**Safety** — cleanup only ever touches paths under `HERMES_HOME` or `/tmp/hermes-*`. Windows mounts (`/mnt/c/...`) are rejected. Well-known top-level state dirs (`logs/`, `memories/`, `sessions/`, `cron/`, `cache/`, `skills/`, `plugins/`, `disk-cleanup/` itself) are never removed even when empty — a fresh install does not get gutted on first session end.
**Enabling:** `hermes plugins enable disk-cleanup` (or check the box in `hermes plugins`).
**Disabling again:** `hermes plugins disable disk-cleanup`.
### security-guidance
Fast pattern-matched security warnings on file writes. When the agent's `write_file` / `patch` / `skill_manage` calls carry content matching a known-dangerous code pattern — `pickle.load`, `yaml.load` without `SafeLoader`, `eval(`, `os.system`, `subprocess(..., shell=True)`, JS `child_process.exec`, React `dangerouslySetInnerHTML`, raw `.innerHTML =` / `.outerHTML =` / `document.write`, Node `crypto.createCipher`, AES ECB mode, TLS verification disabled, XXE-prone `xml.etree` / `minidom` parsers, `<script src="//..." >` without SRI, `torch.load` without `weights_only=True`, GitHub Actions `${{ github.event.* }}` injection — the plugin appends a `⚠️ Security guidance` block to the tool's result.
The file is still written. The model reads the warning in the next turn's tool message and can either fix the code or document why the construct is safe in this context. Pattern matching has a non-trivial false-positive rate, which is why warn (not block) is the default.
**Coverage:** 25 rules total, covering unsafe deserialization, command injection, XSS sinks, crypto footguns, XXE, supply-chain (SRI), and CI/CD workflow injection. The pattern data is a verbatim Apache-2.0 fork of [Anthropic's `claude-plugins-official`](https://github.com/anthropics/claude-plugins-official/tree/main/plugins/security-guidance/hooks) — see the plugin's `LICENSE` and `NOTICE` files for attribution.
**Modes:**
| Env var | Effect |
|---|---|
| (unset) | **warn mode** (default) — file is written, warning appended to result |
| `SECURITY_GUIDANCE_BLOCK=1` | **block mode** — write refused, warning returned as the block reason |
| `SECURITY_GUIDANCE_DISABLE=1` | kill switch — plugin loads but does nothing |
**Enabling:** `hermes plugins enable security-guidance` (or check the box in `hermes plugins`).
**Disabling again:** `hermes plugins disable security-guidance`.
**What it does not do (yet):** the upstream Anthropic plugin has two more layers — an LLM diff review on each agent turn that touched files, and an agentic commit-time review that traces data flow across files. Neither is ported. The agent can already run those reviews on demand via `delegate_task`.
### observability/langfuse
Traces Hermes turns, LLM calls, and tool invocations to [Langfuse](https://langfuse.com) — an open-source LLM observability platform. One span per turn, one generation per API call, one tool observation per tool call. Usage totals, per-type token counts, and cost estimates come out of Hermes' canonical `agent.usage_pricing` numbers, so the Langfuse dashboard sees the same breakdown (input / output / `cache_read_input_tokens` / `cache_creation_input_tokens` / `reasoning_tokens`) that appears in `hermes logs`.
The plugin is fail-open: no SDK installed, no credentials, or a transient Langfuse error — all turn into a silent no-op in the hook. The agent loop is never impacted.
**Setup (interactive — recommended):**
```bash
hermes tools # → Langfuse Observability → Cloud or Self-Hosted
```
The wizard collects your keys, `pip install`s the `langfuse` SDK, and adds `observability/langfuse` to `plugins.enabled` for you. Restart Hermes and the next turn ships a trace.
**Setup (manual):**
```bash
pip install langfuse
hermes plugins enable observability/langfuse
```
Then put the credentials in `~/.hermes/.env`:
```bash
HERMES_LANGFUSE_PUBLIC_KEY=pk-lf-...
HERMES_LANGFUSE_SECRET_KEY=sk-lf-...
HERMES_LANGFUSE_BASE_URL=https://cloud.langfuse.com # or your self-hosted URL
```
**How it works:**
| Hook | Behaviour |
|---|---|
| `pre_api_request` / `pre_llm_call` | Open (or reuse) a per-turn root span "Hermes turn". Start a `generation` child observation for this API call with serialized recent messages as input. |
| `post_api_request` / `post_llm_call` | Close the generation, attach `usage_details`, `cost_details`, `finish_reason`, assistant output + tool calls. If no tool calls and non-empty content, close the turn. |
| `pre_tool_call` | Start a `tool` child observation with sanitized `args`. |
| `post_tool_call` | Close the tool observation with sanitized `result`. `read_file` payloads get summarized (head + tail + omitted-line count) so a huge file read stays under `HERMES_LANGFUSE_MAX_CHARS`. |
Session grouping keys off the Hermes session ID (or task ID for sub-agents) via `langfuse.propagate_attributes`, so everything in a single `hermes chat` session lives under one Langfuse session.
**Verify:**
```bash
hermes plugins list # observability/langfuse should show "enabled"
hermes chat -q "hello" # check the Langfuse UI for a "Hermes turn" trace
```
**Optional tuning** (in `.env`):
| Variable | Default | Purpose |
|---|---|---|
| `HERMES_LANGFUSE_ENV` | — | Environment tag on traces (`production`, `staging`, …) |
| `HERMES_LANGFUSE_RELEASE` | — | Release/version tag |
| `HERMES_LANGFUSE_SAMPLE_RATE` | `1.0` | Sampling rate passed to the SDK (0.0–1.0) |
| `HERMES_LANGFUSE_MAX_CHARS` | `12000` | Per-field truncation for message content / tool args / tool results |
| `HERMES_LANGFUSE_DEBUG` | `false` | Verbose plugin logging to `agent.log` |
Hermes-prefixed and standard SDK env vars (`LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_BASE_URL`) are both accepted — Hermes-prefixed wins when both are set.
**Performance:** the Langfuse client is cached after the first hook call. If credentials or SDK are missing, that decision is also cached — subsequent hooks fast-return without re-checking env vars or reloading config.
**Disabling:** `hermes plugins disable observability/langfuse`. The plugin module is still discovered, but no module code runs until you re-enable.
### NeMo Relay native integration (migration note)
NeMo Relay is no longer a bundled Hermes plugin. Do not run `hermes plugins enable observability/nemo_relay`; Hermes core now owns the Relay session, turn, LLM, and tool lifecycles.
To opt into Relay middleware or exporters, create a standard Relay `plugins.toml`, then set `HERMES_NEMO_RELAY_PLUGINS_TOML` to that file before starting Hermes. The policy is process-wide for every profile hosted by that Hermes process. See the [NeMo Relay observability configuration](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/about) for ATOF, ATIF, and OpenTelemetry options.
The old `HERMES_NEMO_RELAY_ATOF_*` and `HERMES_NEMO_RELAY_ATIF_*` settings no longer activate exporters. `hermes doctor` reports these stale settings when no replacement `plugins.toml` is selected.
#### Session-span segmentation (continuous sessions)
Relay exports a span when its scope closes. A continuous gateway session can keep its session span open for days even though each turn span exports normally. Optional segmentation rotates only the session scope at a turn boundary:
```yaml
gateway:
telemetry:
session_segments:
on_compaction: false # rotate after context compaction
max_turns: 0 # 0 = unlimited; N = turns per segment
```
| Key | Default | Behavior |
|---|---:|---|
| `on_compaction` | `false` | Rotate after compaction completes, at the next turn boundary. |
| `max_turns` | `0` | Rotate after every N completed turns; `0` disables the cap. |
Both defaults preserve one session scope for the full session. Rotated spans retain the same `session_id` and add `hermes.session.segment` plus `hermes.session.segment_reason` (`compaction` or `max_turns`).
### google_meet
Lets the agent **join, transcribe, and participate in Google Meet calls** — take notes on a meeting, summarize the back-and-forth after, follow up on specific points, and (optionally) speak replies back into the call via TTS.
**What it adds:**
- A headless virtual participant that joins a Meet URL using browser automation
- Live transcription of the meeting audio via the configured STT provider
- A `meet_join` / `meet_status` / `meet_transcript` / `meet_leave` / `meet_say` toolset the agent invokes to join calls, poll the live transcript, and act on what it heard
- Post-meeting artifacts (transcript, status) saved under `~/.hermes/workspace/meetings/<meeting_id>/`
**Setup:**
```bash
hermes plugins enable google_meet
hermes meet setup # preflight: playwright, chromium, auth file
hermes meet auth # opens a browser to sign into Google and saves session state —
# needs a Google account with Meet access. Host approval may be
# required if the meeting enforces "only invited participants can join".
```
Usage from chat:
> "Join meet.google.com/abc-defg-hij and take notes. After the call, send me a summary with action items."
The agent kicks off the meeting join, streams the transcription back into its context as the call proceeds, and produces a structured summary when the meeting ends (or when you tell it to stop).
**When to use it:** recurring standups where you want a bot to transcribe + summarize for async attendees; deposition-style interviews where you want structured notes; any case where you'd otherwise need Fireflies / Otter / Grain. When you'd rather not have an AI listening in — don't enable it.
**Disabling:** `hermes plugins disable google_meet`. Any saved transcripts stay in `~/.hermes/workspace/meetings/` until you remove them.
### hermes-achievements
Adds a **Steam-style achievements tab to the dashboard** — 60+ collectible, tiered badges generated from your real Hermes session history. Tool-chain feats, debugging patterns, vibe-coding streaks, skill/memory usage, model/provider variety, lifestyle quirks (weekend and night sessions). Originally authored by [@PCinkusz](https://github.com/PCinkusz) as an external plugin; brought in-tree so it stays in lockstep with Hermes feature changes.
**How it works:**
- Scans your entire `~/.hermes/state.db` session history on the dashboard backend
- Per-session stats are cached by `(started_at, last_active)` fingerprint, so only new or changed sessions re-analyze on subsequent scans
- First-ever scan runs in a background thread — the dashboard never blocks waiting for it, even on databases with thousands of sessions
- Unlock state is persisted to `$HERMES_HOME/plugins/hermes-achievements/state.json`
**Tier progression:** Copper → Silver → Gold → Diamond → Olympian. Each card exposes a "What counts" section listing the exact metric being tracked.
**Achievement states:**
| State | Meaning |
|---|---|
| Unlocked | At least one tier achieved |
| Discovered | Known achievement, progress visible, not yet earned |
| Secret | Hidden until Hermes detects the first related signal in your history |
**API** — routes mount under `/api/plugins/hermes-achievements/`:
| Endpoint | Purpose |
|---|---|
| `GET /achievements` | Full catalog with per-badge unlock state (returns a pending placeholder while the first cold scan is running) |
| `GET /scan-status` | State of the background scanner: `idle` / `running` / `failed`, last duration, run count |
| `GET /recent-unlocks` | Twenty most recently unlocked badges, newest first |
| `GET /sessions/{id}/badges` | Badges earned primarily in one specific session |
| `POST /rescan` | Manual synchronous rescan (blocks; use when the user clicks the rescan button) |
| `POST /reset-state` | Clear unlock history and cached snapshot |
**State files** — live under `$HERMES_HOME/plugins/hermes-achievements/`:
| File | Contents |
|---|---|
| `state.json` | Unlock history: which badges you've earned and when. Stable across Hermes updates. |
| `scan_snapshot.json` | Last completed scan payload (served immediately on dashboard load) |
| `scan_checkpoint.json` | Per-session stats cache keyed by fingerprint (makes warm rescans fast) |
**Performance notes:**
- Cold scan on ~8,000 sessions takes a few minutes. It runs in a background thread on first dashboard request; the UI sees a pending placeholder and polls `/scan-status`.
- **Incremental results during a cold scan** — the scanner publishes a partial snapshot every ~250 sessions so each dashboard refresh shows more badges unlocked as the scan progresses. No minute-long stare at zeros.
- Warm rescan reuses per-session stats for every session whose `started_at` + `last_active` fingerprint matches the checkpoint — completes in seconds even on large histories.
- The in-memory snapshot TTL is 120s; stale requests serve the old snapshot immediately and kick a background refresh. You never wait on a spinner just because TTL expired.
**Enabling:** Nothing to enable — `hermes-achievements` is a dashboard-only plugin (no lifecycle hooks, no model-visible tools). It auto-registers as a tab in `hermes dashboard` on first launch. The `plugins.enabled` config only gates lifecycle/tool plugins; dashboard plugins are discovered purely via their `dashboard/manifest.json`.
**Opting out:** Delete or rename `plugins/hermes-achievements/dashboard/manifest.json`, or override it with a user plugin of the same name in `~/.hermes/plugins/hermes-achievements/` that ships no dashboard. The plugin's state files under `$HERMES_HOME/plugins/hermes-achievements/` survive — reinstalling preserves your unlock history.
## Adding a bundled plugin
Bundled plugins are written exactly like any other Hermes plugin — see [Build a Hermes Plugin](/developer-guide/plugins). The only differences are:
- Directory lives at `<repo>/plugins/<name>/` instead of `~/.hermes/plugins/<name>/`
- Manifest source is reported as `bundled` in `hermes plugins list`
- User plugins with the same name override the bundled version
A plugin is a good candidate for bundling when:
- It has no optional dependencies (or they're already `pip install .[all]` deps)
- The behaviour benefits most users and is opt-out rather than opt-in
- The logic ties into lifecycle hooks that the agent would otherwise have to remember to invoke
- It complements a core capability without expanding the model-visible tool surface
Counter-examples — things that should stay as user-installable plugins, not bundled: third-party integrations with API keys, niche workflows, large dependency trees, anything that would meaningfully change agent behaviour by default.
website/docs/user-guide/features/code-execution.md
+296
View File
@@ -0,0 +1,296 @@
---
sidebar_position: 8
title: "Code Execution"
description: "Programmatic Python execution with RPC tool access — collapse multi-step workflows into a single turn"
---
# Code Execution (Programmatic Tool Calling)
The `execute_code` tool lets the agent write Python scripts that call Hermes tools programmatically, collapsing multi-step workflows into a single LLM turn. The script runs in a child process on the agent host, communicating with Hermes over a Unix domain socket RPC.
## How It Works
1. The agent writes a Python script using `from hermes_tools import ...`
2. Hermes generates a `hermes_tools.py` stub module with RPC functions
3. Hermes opens a Unix domain socket and starts an RPC listener thread
4. The script runs in a child process — tool calls travel over the socket back to Hermes
5. Only the script's `print()` output is returned to the LLM; intermediate tool results never enter the context window
```python
# The agent can write scripts like:
from hermes_tools import web_search, web_extract
results = web_search("Python 3.13 features", limit=5)
for r in results["data"]["web"]:
content = web_extract([r["url"]])
# ... filter and process ...
print(summary)
```
**Available tools inside scripts:** `web_search`, `web_extract`, `read_file`, `write_file`, `search_files`, `patch`, `terminal` (foreground only).
## When the Agent Uses This
The agent uses `execute_code` when there are:
- **3+ tool calls** with processing logic between them
- Bulk data filtering or conditional branching
- Loops over results
The key benefit: intermediate tool results never enter the context window — only the final `print()` output comes back, dramatically reducing token usage.
## Practical Examples
### Data Processing Pipeline
```python
from hermes_tools import search_files, read_file
import json
# Find all config files and extract database settings
matches = search_files("database", path=".", file_glob="*.yaml", limit=20)
configs = []
for match in matches.get("matches", []):
content = read_file(match["path"])
configs.append({"file": match["path"], "preview": content["content"][:200]})
print(json.dumps(configs, indent=2))
```
### Multi-Step Web Research
```python
from hermes_tools import web_search, web_extract
import json
# Search, extract, and summarize in one turn
results = web_search("Rust async runtime comparison 2025", limit=5)
summaries = []
for r in results["data"]["web"]:
page = web_extract([r["url"]])
for p in page.get("results", []):
if p.get("content"):
summaries.append({
"title": r["title"],
"url": r["url"],
"excerpt": p["content"][:500]
})
print(json.dumps(summaries, indent=2))
```
### Bulk File Refactoring
```python
from hermes_tools import search_files, read_file, patch
# Find all Python files using deprecated API and fix them
matches = search_files("old_api_call", path="src/", file_glob="*.py")
fixed = 0
for match in matches.get("matches", []):
result = patch(
path=match["path"],
old_string="old_api_call(",
new_string="new_api_call(",
replace_all=True
)
if "error" not in str(result):
fixed += 1
print(f"Fixed {fixed} files out of {len(matches.get('matches', []))} matches")
```
### Build and Test Pipeline
```python
from hermes_tools import terminal, read_file
import json
# Run tests, parse results, and report
result = terminal("cd /project && python -m pytest --tb=short -q 2>&1", timeout=120)
output = result.get("output", "")
# Parse test output
passed = output.count(" passed")
failed = output.count(" failed")
errors = output.count(" error")
report = {
"passed": passed,
"failed": failed,
"errors": errors,
"exit_code": result.get("exit_code", -1),
"summary": output[-500:] if len(output) > 500 else output
}
print(json.dumps(report, indent=2))
```
## Execution Mode
`execute_code` has two execution modes controlled by `code_execution.mode` in `~/.hermes/config.yaml`:
| Mode | Working directory | Python interpreter |
|------|-------------------|--------------------|
| **`project`** (default) | The session's working directory (same as `terminal()`) | Active `VIRTUAL_ENV` / `CONDA_PREFIX` python, falling back to Hermes's own python |
| `strict` | A temp staging directory isolated from the user's project | `sys.executable` (Hermes's own python) |
**When to leave it on `project`:** you want `import pandas`, `from my_project import foo`, or relative paths like `open(".env")` to work the same way they do in `terminal()`. This is almost always what you want.
**When to flip to `strict`:** you need maximum reproducibility — you want the same interpreter every session regardless of which venv the user activated, and you want scripts quarantined from the project tree (no risk of accidentally reading project files through a relative path).
```yaml
# ~/.hermes/config.yaml
code_execution:
mode: project # or "strict"
```
Fallback behavior in `project` mode: if `VIRTUAL_ENV` / `CONDA_PREFIX` is unset, broken, or points at a Python older than 3.8, the resolver falls back cleanly to `sys.executable` — it never leaves the agent without a working interpreter.
Security-critical invariants are identical across both modes:
- environment scrubbing (API keys, tokens, credentials stripped)
- tool whitelist (scripts cannot call `execute_code` recursively, `delegate_task`, or MCP tools)
- resource limits (timeout, stdout cap, tool-call cap)
Switching mode changes where scripts run and which interpreter runs them, not what credentials they can see or which tools they can call.
## Resource Limits
| Resource | Limit | Notes |
|----------|-------|-------|
| **Timeout** | 5 minutes (300s) | Script is killed with SIGTERM, then SIGKILL after 5s grace |
| **Stdout** | 50 KB | Output truncated with `[output truncated at 50KB]` notice |
| **Stderr** | 10 KB | Included in output on non-zero exit for debugging |
| **Tool calls** | 50 per execution | Error returned when limit reached |
All limits are configurable via `config.yaml`:
```yaml
# In ~/.hermes/config.yaml
code_execution:
mode: project # project (default) | strict
timeout: 300 # Max seconds per script (default: 300)
max_tool_calls: 50 # Max tool calls per execution (default: 50)
```
## How Tool Calls Work Inside Scripts
When your script calls a function like `web_search("query")`:
1. The call is serialized to JSON and sent over a Unix domain socket to the parent process
2. The parent dispatches through the standard `handle_function_call` handler
3. The result is sent back over the socket
4. The function returns the parsed result
This means tool calls inside scripts behave identically to normal tool calls — same rate limits, same error handling, same capabilities. The only restriction is that `terminal()` is foreground-only (no `background` or `pty` parameters).
## Error Handling
When a script fails, the agent receives structured error information:
- **Non-zero exit code**: stderr is included in the output so the agent sees the full traceback
- **Timeout**: Script is killed and the agent sees `"Script timed out after 300s and was killed."`
- **Interruption**: If the user sends a new message during execution, the script is terminated and the agent sees `[execution interrupted — user sent a new message]`
- **Tool call limit**: When the 50-call limit is hit, subsequent tool calls return an error message
The response always includes `status` (success/error/timeout/interrupted), `output`, `tool_calls_made`, and `duration_seconds`.
## Security
:::danger Security Model
The child process runs with a **minimal environment**. API keys, tokens, and credentials are stripped by default. The script accesses tools exclusively via the RPC channel — it cannot read secrets from environment variables unless explicitly allowed.
:::
Environment variables containing `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, `CREDENTIAL`, `PASSWD`, or `AUTH` in their names are excluded. Only safe system variables (`PATH`, `HOME`, `LANG`, `SHELL`, `PYTHONPATH`, `VIRTUAL_ENV`, etc.) are passed through.
### Skill Environment Variable Passthrough
When a skill declares `required_environment_variables` in its frontmatter, those variables are **automatically passed through** to both `execute_code` and `terminal` child processes after the skill is loaded. This lets skills use their declared API keys without weakening the security posture for arbitrary code.
For non-skill use cases, you can explicitly allowlist variables in `config.yaml`:
```yaml
terminal:
env_passthrough:
- MY_CUSTOM_KEY
- ANOTHER_TOKEN
```
See the [Security guide](/user-guide/security#environment-variable-passthrough) for full details.
### `HERMES_*` variables in the child
The child process receives only a small, fixed set of operational `HERMES_*`
variables by exact name:
- `HERMES_HOME`
- `HERMES_PROFILE`
- `HERMES_CONFIG`
- `HERMES_ENV`
(plus `HERMES_RPC_DIR` / `HERMES_RPC_SOCKET` / `TZ` / `HOME`, which Hermes
injects explicitly so the RPC channel works).
:::note Behavior change
Earlier versions passed **any** variable whose name began with `HERMES_`
through to the child. That broad prefix was removed for security hardening: it
could leak `HERMES_*`-named configuration that doesn't match a secret substring
(for example `HERMES_BASE_URL`, `HERMES_KANBAN_DB`, or a `HERMES_*_WEBHOOK`
endpoint) into arbitrary sandboxed code.
If an `execute_code` script — or a repo/plugin module it imports at import time
— relied on a `HERMES_*` variable outside the four operational names above, it
will now find that variable **unset** in the child. The drop is intentional,
not a bug.
:::
**Workaround — opt the variable back in explicitly.** Both routes pass the
variable through `execute_code` *and* `terminal` children, and neither weakens
the secret-stripping guarantee (Hermes-managed provider credentials can never
be re-allowed this way):
1. **Per-machine, in `config.yaml`** — add the exact variable name to the
passthrough allowlist:
```yaml
terminal:
env_passthrough:
- HERMES_KANBAN_DB
- HERMES_BASE_URL
```
2. **Per-skill, in the skill's frontmatter** — declare it so it is registered
automatically whenever that skill is loaded:
```yaml
required_environment_variables:
- HERMES_KANBAN_DB
```
**Diagnosing it.** When the child drops one or more non-allowlisted `HERMES_*`
variables, Hermes emits a one-line `debug` log naming them and pointing at the
`env_passthrough` escape hatch. Run with debug logging (`hermes logs --level
DEBUG`, or check `~/.hermes/logs/agent.log`) and look for
`execute_code: dropped N non-allowlisted HERMES_* var(s)` if a script behaves
as though a `HERMES_*` variable is missing.
Hermes always writes the script and the auto-generated `hermes_tools.py` RPC stub into a temp staging directory that is cleaned up after execution. In `strict` mode the script also *runs* there; in `project` mode it runs in the session's working directory (the staging directory stays on `PYTHONPATH` so imports still resolve). The child process runs in its own process group so it can be cleanly killed on timeout or interruption.
## execute_code vs terminal
| Use Case | execute_code | terminal |
|----------|-------------|----------|
| Multi-step workflows with tool calls between | ✅ | ❌ |
| Simple shell command | ❌ | ✅ |
| Filtering/processing large tool outputs | ✅ | ❌ |
| Running a build or test suite | ❌ | ✅ |
| Looping over search results | ✅ | ❌ |
| Interactive/background processes | ❌ | ✅ |
| Needs API keys in environment | ⚠️ Only via [passthrough](/user-guide/security#environment-variable-passthrough) | ✅ (most pass through) |
**Rule of thumb:** Use `execute_code` when you need to call Hermes tools programmatically with logic between calls. Use `terminal` for running shell commands, builds, and processes.
## Platform Support
Code execution is available on **Linux, macOS, and Windows**. On Linux and macOS the RPC channel uses a Unix domain socket; on Windows, where `AF_UNIX` is unreliable, Hermes automatically falls back to a loopback TCP socket for the sandbox RPC transport. Remote terminal backends (Docker/SSH/Modal/etc.) use a file-based RPC transport instead and additionally require Python 3 inside the backend.
website/docs/user-guide/features/codex-app-server-runtime.md
+460
View File
@@ -0,0 +1,460 @@
---
title: Codex App-Server Runtime (optional)
sidebar_label: Codex App-Server Runtime
---
# Codex App-Server Runtime
Hermes can optionally hand `openai/*` and `openai-codex/*` turns to the [Codex CLI app-server](https://github.com/openai/codex) instead of running its own tool loop. When enabled, terminal commands, file edits, sandboxing, and MCP tool calls all execute inside Codex's runtime — Hermes becomes the shell around it (sessions DB, slash commands, gateway, memory and skill review).
This is **opt-in only**. Default Hermes behavior is unchanged unless you flip the flag. Hermes never auto-routes you onto this runtime.
:::tip
Not using OpenAI Codex? `hermes setup --portal` configures a non-Codex backend with Claude/Gemini/etc. in one step. See [Nous Portal](/integrations/nous-portal).
:::
## Why
- Run OpenAI agent turns against your **ChatGPT subscription** (no API key required) using the same auth flow Codex CLI uses.
- Use **Codex's own toolset and sandbox** — `shell` for terminal/read/write/search, `apply_patch` for structured edits, `update_plan` for planning, all running inside seatbelt/landlock sandboxing.
- **Native Codex plugins** — Linear, GitHub, Gmail, Calendar, Canva, etc. — installed via `codex plugin` are auto-migrated and active in your Hermes session.
- **Hermes' richer tools come along** — web_search, web_extract, browser automation, vision, image generation, skills, and TTS work via an MCP callback. Codex calls back into Hermes for tools it doesn't have built in.
- **Memory and skill nudges keep working** — Codex's events are projected into Hermes' message shape so the self-improvement loop sees a normal-looking transcript.
## What tools the model actually has
This is the part most users want to know up front. When this runtime is on, the model running your turn has three independent sources of tools:
### 1. Codex's built-in toolset (always on)
These ship with `codex app-server` itself — no Hermes involvement, no MCP, no plugins. All five are available the moment the runtime starts:
- **`shell`** — runs arbitrary shell commands inside the sandbox. This is how the model reads files (`cat`, `head`, `tail`), writes them (`echo > foo`, heredocs), searches them (`find`, `rg`, `grep`), navigates directories (`ls`, `cd`), runs builds, manages processes, and anything else you'd do in bash.
- **`apply_patch`** — applies a structured multi-file diff in Codex's patch format. The model uses this for non-trivial code edits (adding a function, refactoring across files); shell heredocs are still available for one-off writes.
- **`update_plan`** — codex's internal todo / plan tracker. Equivalent of Hermes' `todo` tool, but managed entirely inside codex's runtime.
- **`view_image`** — load a local image file into the conversation so the model can see it.
- **`web_search`** — codex has its own built-in web search when configured. Hermes also exposes `web_search` (Firecrawl-backed) via the callback below; the model picks whichever it prefers.
So **anything you'd do via terminal — read/write/search/find/run — codex does natively**. The sandbox profile (`:workspace` by default when you enable the runtime) controls what's writable.
### 2. Native Codex plugins (auto-migrated from your `codex plugin` install)
When you enable the runtime, Hermes queries codex's `plugin/list` RPC and writes a `[plugins."<name>@openai-curated"]` entry for every plugin you have installed. The plugins themselves are managed by codex and authorized once via codex's own UI.
Examples (the ones the OpenClaw thread highlighted as "YouTube-video-worthy"):
- **Linear** — find/update issues
- **GitHub** — search code, view PRs, comment
- **Gmail** — read/send mail
- **Google Calendar** — create/find events
- **Outlook calendar/email** — same shape via the Microsoft connector
- **Canva** — design generation
- ...whatever else you've installed via `codex plugin marketplace add openai-curated` + `codex plugin install ...`
What's NOT migrated:
- Plugins you haven't installed yet — install them in Codex first.
- ChatGPT app marketplace entries (`app/list`) — these are already enabled inside codex by virtue of your account auth.
### 3. Hermes tool callback (MCP server, registered in `~/.codex/config.toml`)
Hermes registers itself as an MCP server so codex can call back for tools codex doesn't ship with. Available via the callback:
- **`web_search`** / **`web_extract`** — Firecrawl-backed; tends to be cleaner than scraping for structured content.
- **`browser_navigate` / `browser_click` / `browser_type` / `browser_press` / `browser_snapshot` / `browser_scroll` / `browser_back` / `browser_get_images` / `browser_console` / `browser_vision`** — full browser automation via Camofox or Browserbase.
- **`vision_analyze`** — call a separate vision model to inspect an image (different from codex's `view_image` which loads it into the conversation).
- **`image_generate`** — image generation through Hermes' image_gen plugin chain.
- **`skill_view` / `skills_list`** — read from Hermes' skill library.
- **`text_to_speech`** — TTS through Hermes' configured provider.
When the model wants one of these, codex spawns the `hermes_tools_mcp_server` subprocess via stdio MCP, the call is dispatched through `model_tools.handle_function_call()` (same code path as Hermes' default runtime), and the result is returned to codex like any other MCP response.
### What's NOT available on this runtime
These four Hermes tools require the running AIAgent context (mid-loop state) to dispatch, and a stateless MCP callback can't drive them. Switch back to the default runtime (`/codex-runtime auto`) when you need any of them:
- **`delegate_task`** — spawn subagents
- **`memory`** — Hermes' persistent memory store
- **`session_search`** — cross-session search
- **`todo`** — Hermes' todo store (codex's `update_plan` is the in-runtime equivalent)
## Workflow features (`/goal`, kanban, cron)
### `/goal` (the Ralph loop)
**Works on this runtime.** Goals persist in `state_meta` keyed by session id, the continuation prompt feeds back as a normal user message through `run_conversation()`, and codex executes the next turn natively. The goal judge runs via the auxiliary client (configured via `auxiliary.goal_judge` in config.yaml), independent of which runtime is active. The judge's "blocked, needs user input" verdict is a clean escape if codex stalls on approvals.
**One thing to be aware of:** each continuation prompt is a fresh codex turn, which means codex re-evaluates command approval policy from scratch. If you're doing a long-running goal with lots of writes, expect more approval prompts than you'd see on a single in-session task. Set `default_permissions = ":workspace"` (which Hermes does automatically when you enable the runtime) so simple workspace writes don't require prompting.
### Kanban (multi-agent worktree dispatch)
**Works on this runtime, with one subtle dependency.** The kanban dispatcher spawns each worker as a separate `hermes chat -q` subprocess that reads the user's config — which means if `model.openai_runtime: codex_app_server` is set globally, workers also come up on the codex runtime.
What works inside a codex-runtime worker:
- Codex's full toolset (shell, apply_patch, update_plan, view_image, web_search) — the worker does its actual task work natively
- The migrated codex plugins — Linear, GitHub, etc.
- The Hermes tool callback for browser_*, vision, image_gen, skills, TTS
What also works because the MCP callback exposes them:
- **`kanban_complete` / `kanban_block` / `kanban_comment` / `kanban_heartbeat`** — the worker handoff tools. These read `HERMES_KANBAN_TASK` from env (set by the dispatcher), gate access correctly, and write to the per-board SQLite DB pinned by `HERMES_KANBAN_DB`. Without these in the callback, a worker on this runtime could do its task but couldn't report back, hanging until the dispatcher's timeout.
- **`kanban_show` / `kanban_list`** — read-only board queries for the worker to check its own context.
- **`kanban_create` / `kanban_unblock` / `kanban_link`** — orchestrator-only operations. Available for orchestrator agents running on the codex runtime that need to dispatch new tasks.
The kanban tools are gated by `HERMES_KANBAN_TASK` env var the dispatcher sets — that var is propagated to the codex subprocess (codex inherits env) and from there to the spawned `hermes-tools` MCP server subprocess. So the tools see the right task id and gate correctly. For Codex app-server workers, Hermes also passes narrow app-server sandbox overrides when `HERMES_KANBAN_TASK` is present: keep `workspace-write` sandboxing, add the **board DB directory plus every Kanban path the dispatcher pinned** as extra writable roots (`HERMES_KANBAN_WORKSPACES_ROOT`, `HERMES_KANBAN_WORKSPACE`, legacy `HERMES_KANBAN_ROOT` — deduplicated, DB-dir first), and keep network disabled by default. This avoids the brittle `:danger-no-sandbox` workaround while letting `kanban_complete` / `kanban_block` update the board DB **and** letting workers write reports/artifacts under workspace mounts that live outside the DB directory (e.g. `/media/.../kanban-workspaces/...` on a separate drive — [issue #27941](https://github.com/NousResearch/hermes-agent/issues/27941)).
### Cron jobs
**Not specifically tested.** Cron jobs run via `cronjob` → `AIAgent.run_conversation`, the same code path as the CLI. If the cron job's config has `openai_runtime: codex_app_server` it'll run on codex. The same tool-availability rules apply — codex built-ins + plugins + MCP callback work, agent-loop tools (delegate_task, memory, session_search, todo) don't. If your cron job relies on those, scope the cron to a profile that uses the default runtime.
## Trade-offs
| | Hermes default runtime | Codex app-server (opt-in) |
|---|---|---|
| `delegate_task` subagents | yes | not available — needs agent loop context |
| `memory`, `session_search`, `todo` | yes | not available — needs agent loop context |
| `web_search`, `web_extract` | yes | yes (via MCP callback) |
| Browser automation (Camofox/Browserbase) | yes | yes (via MCP callback) |
| `vision_analyze`, `image_generate` | yes | yes (via MCP callback) |
| `skill_view`, `skills_list` | yes | yes (via MCP callback) |
| `text_to_speech` | yes | yes (via MCP callback) |
| Codex `shell` (terminal/read/write/search/find/run) | — | yes (Codex built-in) |
| Codex `apply_patch` (structured multi-file edits) | — | yes (Codex built-in) |
| Codex `update_plan` (in-runtime todo) | — | yes (Codex built-in) |
| Codex `view_image` (load image into conversation) | — | yes (Codex built-in) |
| Codex sandbox (seatbelt/landlock, profiles) | — | yes (Codex built-in) |
| ChatGPT subscription auth | — | yes (via `openai-codex` provider) |
| Native Codex plugins (Linear, GitHub, etc.) | — | yes (auto-migrated) |
| User MCP servers | yes | yes (auto-migrated to codex) |
| Memory + skill review (background) | yes | yes (via item projection) |
| Multi-turn conversations | yes | yes |
| `/goal` (Ralph loop) | yes | yes |
| Kanban worker dispatch | yes | yes (via callback) |
| Kanban orchestrator tools | yes | yes (via callback) |
| All gateway platforms | yes | yes |
| Non-OpenAI providers | yes | n/a — OpenAI/Codex-scoped |
### Live display
Even though the agent loop runs inside the Codex subprocess, the runtime
bridges Codex's event stream into the same display path the default runtime
uses:
- Live assistant deltas, reasoning (including summary deltas), and stable-ID
tool start/completion events surface in the TUI, desktop, and messaging
gateways as the turn runs. The completion-only history projector remains
separate, so a resumed session hydrates the same tool cards shown during
the turn.
- Gateway commentary stays visible when token streaming is disabled, and
live tool events are forwarded even for notifications drained ahead of an
approval request. Commentary honors `display.show_commentary`.
## Prerequisites
1. **Codex CLI installed:**
```bash
npm i -g @openai/codex
codex --version # 0.130.0 or newer
```
2. **Codex OAuth login.** The codex subprocess reads `~/.codex/auth.json`. Two ways to populate it:
```bash
codex login # writes tokens to ~/.codex/auth.json
```
Hermes' own `hermes auth add openai-codex` writes to `~/.hermes/auth.json` — that's a separate session. **Run `codex login` separately** if you haven't.
3. **(Optional) Install the Codex plugins you want.** When you enable the runtime, Hermes auto-migrates whichever curated plugins you've already installed via Codex CLI:
```bash
codex plugin marketplace add openai-curated
# then via codex's TUI, install Linear / GitHub / Gmail / etc.
```
Hermes will discover them and write `[plugins."<name>@openai-curated"]` entries to `~/.codex/config.toml` automatically.
## Enabling
In a Hermes session:
```
/codex-runtime codex_app_server
```
That command:
- Verifies the `codex` CLI is installed (blocks with an install hint if not).
- Persists `model.openai_runtime: codex_app_server` to your config.yaml.
- Migrates user MCP servers from `~/.hermes/config.yaml` to `~/.codex/config.toml`.
- **Discovers and migrates installed native Codex plugins** (Linear, GitHub, Gmail, Calendar, Canva, etc.) by querying Codex's `plugin/list` RPC.
- **Registers Hermes' own tools as an MCP server** so the codex subprocess can call back for tools codex doesn't ship with.
- **Writes `default_permissions = ":workspace"`** so the sandbox allows writes within the workspace without prompting for every operation.
- Tells you what was migrated. Takes effect on the **next** session — the current cached agent keeps the prior runtime so prompt caches stay valid.
Synonyms: `/codex-runtime on`, `/codex-runtime off`, `/codex-runtime auto`.
To check current state without changing anything:
```
/codex-runtime
```
You can also set it manually in `~/.hermes/config.yaml`:
```yaml
model:
openai_runtime: codex_app_server # default is "auto" (= Hermes runtime)
```
## Self-improvement loop (memory + skill nudges)
Hermes' background self-improvement fires on counter thresholds:
- Every 10 user prompts → a forked review agent looks at the conversation and decides whether anything should be saved to memory.
- Every 10 tool iterations within a single turn → same idea but for skills (`skill_manage` writes).
**Both keep working on the codex runtime.** The codex path projects each completed `commandExecution` / `fileChange` / `mcpToolCall` / `dynamicToolCall` item into a synthetic `assistant tool_call` + `tool` result message, so by the time the review runs it sees the same shape it sees on the default Hermes runtime.
How the wiring stays equivalent:
| | Default runtime | Codex runtime |
|---|---|---|
| `_turns_since_memory` increments | per user prompt, in run_conversation pre-loop | same code path, before the early-return |
| `_iters_since_skill` increments | per tool iteration in the chat-completions loop | by `turn.tool_iterations` after the codex turn returns |
| Memory trigger (`_turns_since_memory >= _memory_nudge_interval`) | computed in pre-loop, fires after response | computed in pre-loop, passed through to codex helper |
| Skill trigger (`_iters_since_skill >= _skill_nudge_interval`) | computed after the loop | computed after the codex turn |
| `_spawn_background_review(messages_snapshot=..., review_memory=..., review_skills=...)` | called when either trigger fires | called identically when either trigger fires |
One detail: the review fork itself needs to call Hermes' agent-loop tools (`memory`, `skill_manage`), which require Hermes' own dispatch. So when the parent agent is on `codex_app_server`, the review fork is **downgraded to `codex_responses`** — same OAuth credentials, same `openai-codex` provider, but talks to OpenAI's Responses API directly so Hermes owns the loop and the agent-loop tools work. This is invisible to the user.
Net effect: enable the codex runtime and your memory + skill nudges keep firing exactly as they would otherwise.
## How approvals work
Codex requests approval before executing commands or applying patches. These get translated into Hermes' standard "Dangerous Command" prompt:
```
╭───────────────────────────────────────╮
│ Dangerous Command │
│ │
│ /bin/bash -lc 'echo hello > foo.txt' │
│ │
│ ❯ 1. Allow once │
│ 2. Allow for this session │
│ 3. Deny │
│ │
│ Codex requests exec in /your/cwd │
╰───────────────────────────────────────╯
```
- **Allow once** → approve this single command.
- **Allow for this session** → Codex won't re-prompt for similar commands.
- **Deny** → command is rejected; Codex continues in read-only mode.
For `apply_patch` (file edit) approvals, Hermes shows a summary of what changed (`1 add, 1 update: /tmp/new.py, /tmp/old.py`) when codex provides the data via the corresponding `fileChange` item.
## Permission profiles
Codex has three built-in permission profiles:
- `:read-only` — no writes; every shell command requires approval
- `:workspace` — writes within the current workspace allowed without prompts (Hermes' default when you enable the runtime)
- `:danger-no-sandbox` — no sandbox at all (don't use this unless you understand it)
You can override the default in `~/.codex/config.toml` outside Hermes' managed block:
```toml
default_permissions = ":read-only"
```
(Hermes will preserve your override on re-migration as long as it lives outside the `# managed by hermes-agent` markers.)
## Auxiliary tasks and ChatGPT subscription token cost
When this runtime is on with the `openai-codex` provider, **auxiliary tasks (title generation, context compression, vision auto-detect, the background self-improvement review fork) also flow through your ChatGPT subscription by default**, because Hermes' auxiliary client uses the main provider/model when no per-task override is set.
This isn't specific to `codex_app_server` — it's true for the existing `codex_responses` path too — but it's more visible here because you're explicitly opting in for the subscription billing.
To route specific aux tasks to a cheaper / different model, set explicit overrides in `~/.hermes/config.yaml`:
```yaml
auxiliary:
title_generation:
provider: openrouter
model: google/gemini-3-flash-preview
compression:
provider: openrouter
model: google/gemini-3-flash-preview
vision:
provider: openrouter
model: google/gemini-3-flash-preview
goal_judge:
provider: openrouter
model: google/gemini-3-flash-preview
```
The self-improvement review fork inherits the main runtime via `_current_main_runtime()` and Hermes downgrades it from `codex_app_server` to `codex_responses` automatically (so the fork can actually call `memory` and `skill_manage` — Hermes' own agent-loop tools). That fork still uses your subscription auth unless you've routed aux tasks elsewhere.
## Editing `~/.codex/config.toml` safely
Hermes wraps everything it manages between two marker comments:
```toml
# managed by hermes-agent — `hermes codex-runtime migrate` regenerates this section
default_permissions = ":workspace"
[mcp_servers.filesystem]
...
[plugins."github@openai-curated"]
...
# end hermes-agent managed section
```
Anything **outside** that block is yours. Re-running migration (via `/codex-runtime codex_app_server` or whenever you toggle the runtime on) replaces the managed block in place but preserves user content above and below it verbatim. This means you can:
- Add your own MCP servers Hermes doesn't know about
- Override `default_permissions` to `:read-only` if you prefer to be prompted
- Configure codex-only options (model, providers, otel, etc.)
- Add user-defined permission profiles in `[permissions.<name>]` tables
Anything you add **inside** the managed block will get clobbered on the next migration. If you need a tweak that requires editing the managed block, file an issue and we'll add the knob.
## Multi-profile / multi-tenant setups
By default, Hermes points the codex subprocess at `~/.codex/` regardless of which Hermes profile is active. This means `hermes -p work` and `hermes -p personal` share the same Codex auth, plugins, and config. For most users this is the right behavior — it matches what running `codex` CLI directly would do.
If you want per-profile Codex isolation (separate auth, separate installed plugins, separate config), set `CODEX_HOME` explicitly per profile. The cleanest way is to point at a directory under your `HERMES_HOME`:
```bash
# Inside the work profile, you might wrap hermes:
CODEX_HOME=~/.hermes/profiles/work/codex hermes chat
```
You'll need to re-run `codex login` once with that `CODEX_HOME` set so the OAuth tokens land in the profile-scoped location. After that, `hermes -p work` will operate on isolated Codex state.
We don't auto-scope this because moving an existing user's `~/.codex/` would silently invalidate their Codex CLI auth — anyone who already ran `codex login` would have to re-authenticate. Opt-in feels safer than surprising users.
## HOME environment variable passthrough
Hermes does NOT rewrite `HOME` when spawning the codex app-server subprocess (we use `os.environ.copy()` and only overlay `CODEX_HOME` and `RUST_LOG`). This means:
- Commands codex runs via its `shell` tool see the real user `HOME` and find `~/.gitconfig`, `~/.gh/`, `~/.aws/`, `~/.npmrc`, etc. correctly.
- Codex's internal state stays isolated through `CODEX_HOME` (which points at `~/.codex/` by default).
This matches the boundary OpenClaw arrived at after some early experimentation: isolate Codex's state, leave the user's home alone. (Cf. openclaw/openclaw#81562.)
## MCP server migration
Hermes' `mcp_servers` config is auto-translated to the TOML format Codex expects. The migration runs every time you enable the runtime and is idempotent — re-runs replace the managed section but preserve any user-edited Codex config.
What translates:
| Hermes (`config.yaml`) | Codex (`config.toml`) |
|---|---|
| `command` + `args` + `env` | stdio transport |
| `url` + `headers` | streamable_http transport |
| `timeout` | `tool_timeout_sec` |
| `connect_timeout` | `startup_timeout_sec` |
| `enabled: false` | `enabled = false` |
What's not migrated:
- Hermes-specific keys like `sampling` (Codex's MCP client has no equivalent — these are dropped with a per-server warning).
## Native Codex plugin migration
Plugins installed via `codex plugin` (Linear, GitHub, Gmail, Calendar, Canva, etc.) are discovered through Codex's `plugin/list` RPC. For each plugin where `installed: true`, Hermes writes a `[plugins."<name>@openai-curated"]` block enabling it in your Hermes session.
This means: when your friend says "I have Calendar and GitHub set up in my Codex CLI" and they enable Hermes' codex runtime, Hermes activates those automatically. No re-configuration needed.
What's NOT migrated:
- Plugins you haven't installed yet — install them in Codex first.
- Plugins where codex reports `availability != AVAILABLE` (broken install, expired OAuth, removed from marketplace, etc.). These are skipped to avoid writing config that would fail at activation time.
- ChatGPT app marketplace entries (the per-account `app/list` results — these are already enabled inside codex by virtue of your account auth).
- Plugin OAuth — you authorize each plugin once in Codex itself; Hermes doesn't touch credentials.
## Hermes tool callback (the new MCP server)
Codex's built-in toolset covers shell/file ops/patches but doesn't have web search, browser automation, vision, image generation, etc. To keep those usable in a codex turn, Hermes registers itself as an MCP server in `~/.codex/config.toml`:
```toml
[mcp_servers.hermes-tools]
command = "/path/to/python"
args = ["-m", "agent.transports.hermes_tools_mcp_server"]
env = { HERMES_HOME = "/your/.hermes", PYTHONPATH = "...", HERMES_QUIET = "1" }
startup_timeout_sec = 30.0
tool_timeout_sec = 600.0
```
When the model calls `web_search` (or another exposed Hermes tool), codex spawns the `hermes_tools_mcp_server` subprocess via stdio, the request is dispatched through `model_tools.handle_function_call()`, and the result is projected back to codex like any other MCP response.
**Tools available via the callback:** `web_search`, `web_extract`, `browser_navigate`, `browser_click`, `browser_type`, `browser_press`, `browser_snapshot`, `browser_scroll`, `browser_back`, `browser_get_images`, `browser_console`, `browser_vision`, `vision_analyze`, `image_generate`, `skill_view`, `skills_list`, `text_to_speech`.
**Tools NOT available:** `delegate_task`, `memory`, `session_search`, `todo`. These need the running AIAgent context to dispatch (mid-loop state) and a stateless MCP callback can't drive them. Use the default Hermes runtime (`/codex-runtime auto`) when you need these.
## Disabling
Switch back at any time:
```
/codex-runtime auto
```
Effective on the next session. The Codex managed block stays in `~/.codex/config.toml` so you can re-enable later without losing config — or remove it manually if you prefer.
## Limitations
This runtime is **opt-in beta**. Working as of Hermes Agent 2026.5 + Codex CLI 0.130.0:
- Multi-turn conversations
- `commandExecution` and `fileChange` (apply_patch) approvals via Hermes UI
- MCP tool calls (verified against `@modelcontextprotocol/server-filesystem` and the new `hermes-tools` callback)
- Native Codex plugin migration (verified against Linear / GitHub / Calendar inventory)
- Deny/cancel paths
- Toggle on/off cycle
- Memory and skill nudge counters (verified live via integration tests)
- Hermes web_search through codex (verified live: "OpenAI Codex CLI – Getting Started" returned end-to-end)
Known limitations:
- **Hermes auth and codex auth are separate sessions.** You need both `codex login` AND `hermes auth add openai-codex` for the cleanest UX (the runtime uses codex's session for the LLM call). This is a deliberate design choice in Hermes' `_import_codex_cli_tokens` — Hermes won't share OAuth state with codex CLI to avoid clobbering each other on token refresh.
- **`delegate_task`, `memory`, `session_search`, `todo` are unavailable on this runtime.** They need the running AIAgent context which a stateless MCP callback can't provide. Use `/codex-runtime auto` when you need these.
- **No inline patch preview in approval prompts when codex doesn't track the changeset.** Codex's `fileChange` approval params don't always carry the changeset. Hermes caches the data from the corresponding `item/started` notification when possible, but if approval arrives before the item has streamed, the prompt falls back to whatever `reason` codex provides.
- **Sub-second cancellation isn't guaranteed.** Mid-stream interrupts (Ctrl+C while codex is responding) are sent via `turn/interrupt`, but if codex has already flushed the final message, you get the response anyway.
If you find a bug, [open an issue](https://github.com/NousResearch/hermes-agent/issues) with the output of `hermes logs --since 5m`. Mention `codex-runtime` in the title so it's easy to triage.
## Architecture
```
┌─── Hermes shell (CLI / TUI / gateway) ───┐
│ sessions DB · slash commands · memory │
│ & skill review · cron · session pickers │
└──┬──────────────────────────────────────┬┘
│ user_message final │
▼ text + │
┌──────────────────────────────────┐ projected │
│ AIAgent.run_conversation() │ messages │
│ if api_mode == codex_app_server │ │
│ → CodexAppServerSession │ │
│ else: chat_completions / codex_responses (default)
└────┬─────────────────────────────┘ │
│ JSON-RPC over stdio │
▼ │
┌──────────────────────────────────┐ │
│ codex app-server (subprocess) │──────────────┘
│ thread/start, turn/start │
│ item/* notifications │
│ shell + apply_patch + update_plan│
│ view_image + sandbox │
│ ┌─────────────────────────┐ │
│ │ MCP client │ │
│ │ ├─ user MCP servers │ │
│ │ ├─ native plugins │ │
│ │ │ (linear, github, │ │
│ │ │ gmail, calendar, │ │
│ │ │ canva, ...) │ │
│ │ └─ hermes-tools ───────┼─────────────────┐
│ │ (callback to │ │ │
│ │ Hermes' richer │ │ │
│ │ tools) │ │ │
│ └─────────────────────────┘ │ │
└──────────────────────────────────┘ │
│
▼
┌──────────────────────────────────────────────────────────┐
│ hermes_tools_mcp_server.py (subprocess on demand) │
│ web_search, web_extract, browser_*, vision_analyze, │
│ image_generate, skill_view, skills_list, text_to_speech│
└──────────────────────────────────────────────────────────┘
```
For implementation details, see [PR #24182](https://github.com/NousResearch/hermes-agent/pull/24182) and the [Codex app-server protocol README](https://github.com/openai/codex/blob/main/codex-rs/app-server/README.md).
website/docs/user-guide/features/computer-use.md
+620
View File
@@ -0,0 +1,620 @@
---
title: Computer Use
sidebar_position: 16
---
# Computer Use
Hermes Agent can drive your desktop — clicking, typing, scrolling,
dragging — in the **background** on **macOS, Windows, and Linux**. Your
cursor doesn't move, keyboard focus doesn't change, and your virtual
desktops / Spaces don't switch on you. You and the agent co-work on the
same machine.
Unlike most computer-use integrations, this works with **any tool-capable
model** — Claude, GPT, Gemini, or an open model on a local
OpenAI-compatible endpoint. There's no Anthropic-native schema to worry
about.
## How it works
The built-in `computer_use` toolset is the recommended Hermes integration. It
speaks MCP over stdio to
[`cua-driver`](https://github.com/trycua/cua), an open-source background
computer-use driver. Each platform uses the appropriate accessibility +
input stack under the hood:
| Platform | Accessibility tree | Input dispatch |
|---|---|---|
| macOS | AX (private SkyLight SPIs) | `SLPSPostEventRecordTo` — pid-scoped, no cursor warp |
| Windows | UIAutomation | `SendInput` + `PostMessage` — no focus steal |
| Linux | AT-SPI (X11 + Wayland) | XTest (X11) / virtual-keyboard (Wayland) |
The result is the same on every platform: the agent can read the
accessibility tree of any visible window AND post synthesized events
without bringing it to front, switching virtual desktops, or moving the
real OS cursor.
For the underlying contract — *why* background mode matters, the
no-foreground invariant, click-dispatch internals — see
**[cua.ai/docs/explanation/the-no-foreground-contract](https://cua.ai/docs/explanation/the-no-foreground-contract)**.
## Enabling
**Fresh installs already have the driver.** The Hermes installer
(`install.sh` / `install.ps1`) pre-installs `cua-driver` (best-effort;
pass `--skip-computer-use` / `-SkipComputerUse` to opt out), so enabling
Computer Use is just a config flip:
- **`hermes tools`** → pick `🖱️ Computer Use` — installs the driver
automatically if it's still missing.
- **Dashboard / desktop app** → toggle the Computer Use toolset — if the
driver is missing, the toggle kicks off the install in the background
automatically (watch progress in the toolset panel).
**Manual fallback (older installs, skipped installer step):**
```
hermes computer-use install
```
This fetches and runs the upstream cua-driver installer — `install.sh`
on macOS/Linux, `install.ps1` on Windows. Use `hermes computer-use
status` to verify the install.
Already have cua-driver? Hermes reuses it when it supports the 0.20 runtime
contract. During setup, toolset enablement, `hermes update`, and the first
`computer_use` call of a session, Hermes checks the local version and
manifest. It repairs an old or incomplete standard installation through
the upstream installer (at most once per session at runtime). A binary
selected with `HERMES_CUA_DRIVER_CMD` stays
under your control, so Hermes reports the incompatibility and leaves it
unchanged.
If you install Cua Driver first, `cua-driver skills install` installs Cua's
skill pack under `~/.cua-driver/skills/cua-driver`. Hermes autodetection is a
planned cua-driver follow-up, so currently point Hermes at that directory or
symlink it into your skill space. You can also register raw Cua MCP tools as a
custom MCP server, but that is an alternative for users who need the low-level
interface. The built-in toolset provides Hermes actions, configuration,
approvals, and diagnostics.
After installing, regardless of which path you took, grant the
platform-appropriate prereqs:
| Platform | Prereqs |
|---|---|
| **macOS** | System Settings → Privacy & Security → **Accessibility** + **Screen Recording**. Grant the identity named by `hermes computer-use doctor`. Standard mode uses CuaDriver.app; bounded and unrestricted modes use the Hermes host identity. |
| **Windows** | None at install time. If you're driving over SSH (not RDP / console), you need the autostart pattern — see [cua.ai/docs/how-to-guides/driver/windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh) for the Session 0 ↔ Session 1+ proxy. |
| **Linux** | A reachable display server: `DISPLAY` set for X11, or `XDG_SESSION_TYPE=wayland`. Wayland sessions need an XWayland bridge for capture. AT-SPI must be on (default on GNOME/KDE/Xfce). |
Then start a session with the toolset enabled:
```
hermes -t computer_use chat
```
or add `computer_use` to your enabled toolsets in `~/.hermes/config.yaml`.
## Permission modes and logged-in browser profiles
Hermes maps its existing approval UX onto cua-driver's immutable runtime
modes. Permission mode and capability manifest approval are launch settings.
They cannot change after the runtime starts:
| Hermes session | cua-driver mode | Human intervention |
|---|---|---|
| Manual or smart approvals (default) | `standard` | Normal Hermes approvals; Cua stops at its protected boundary |
| `computer_use.permission_mode: bounded` + reviewed manifest | private `bounded` daemon | You review and approve the capability manifest once, at launch |
| `--yolo`, `/yolo`, or `approvals.mode: off` | private `unrestricted` daemon | One explicit Hermes risk acceptance; no runtime Cua prompts |
Browser work — including pages in a signed-in profile — goes through the
`browser` toolset (`browser_exec`), not `computer_use`. The former
`computer_use.grant_existing_profile` opt-in was removed along with the typed
browser route; a leftover key in config.yaml is ignored.
### Bounded mode for repeatable automation
For recurring browser automation (cron jobs, scheduled research against an
authenticated app), `bounded` mode uses a capability manifest you review once:
```yaml
# config.yaml
computer_use:
permission_mode: bounded
capability_manifest: ~/.hermes/cua-manifest.yaml
```
The manifest names the apps, browser profile kinds, allowed origins, and
typed tools the session may use (see the
[cua-driver permission modes reference](https://cua.ai/docs/reference/cua-driver/permission-modes)
for the format). Hermes launches a private runtime with
`--capability-manifest ... --approve-capability-manifest`; anything outside
the manifest fails closed inside cua-driver. A missing or unreadable manifest
fails loudly at session start rather than silently downgrading. Session YOLO
still overrides bounded for that one session.
On macOS, private-session daemons launch through the installed
`CuaDriver.app` bundle (so permission grants attribute to the driver's own
identity instead of resetting with every Hermes build), and Hermes verifies
the bundle's code signature — exact `com.trycua.driver` identifier and the
official signing team — before launching it. If you build cua-driver from
source (unsigned), opt in explicitly:
```yaml
# config.yaml
computer_use:
allow_unsigned_driver: true # local driver development only
```
Each MCP transport owns a private lifecycle session inside its runtime. A
public session name is only a label for cursor identity and session-scoped
state. It does not select, share, or keep a runtime alive. Turning `/yolo` off,
resetting or closing the Hermes session, cancellation cleanup, or process exit
closes that transport session. Hermes also stops private runtimes that it
launched for bounded or unrestricted access. One Hermes
conversation cannot change another runtime's mode or grants. Bounded and
unrestricted modes use a private
embedded service under the Hermes host identity.
`smart` approval remains `standard`: an LLM classification cannot stand in for
a reviewed manifest.
<div class="alert alert--warning">
YOLO/unrestricted mode does not protect against prompt injection or unintended
input. Use it only in a disposable VM or with accounts and data whose full
compromise you accept.
</div>
## `hermes computer-use doctor` — your first triage stop
`hermes computer-use doctor` runs cua-driver's structured
`health_report` MCP tool and prints a per-check matrix. It's the single
fastest way to find out *why* an action isn't working.
```
$ hermes computer-use doctor
⚠️ cua-driver VERSION on darwin: degraded
✅ binary_version: cua-driver VERSION
✅ platform_supported: macOS 26.4.1 (arm64)
✅ session_active: MCP session is active.
❌ bundle_identity: Process has no CFBundleIdentifier.
→ Run the binary inside CuaDriver.app so TCC grants attribute correctly.
✅ tcc_accessibility: Accessibility is granted.
✅ tcc_screen_recording: Screen Recording is granted.
✅ ax_capability: AX is trusted and reachable.
✅ screen_capture_capability: ScreenCaptureKit reachable; 1 display(s) shareable.
```
- **Exit code 0** when overall is `ok` — everything's wired up.
- **Exit code 1** when `degraded` or `failed` — at least one check failed; the hint on each failure tells you what to fix.
- **Exit code 2** when the cua-driver binary itself isn't reachable.
Useful flags:
- `--include CHECK` — run only the listed checks (repeat for multiple)
- `--skip CHECK` — skip a check (wins over `--include`)
- `--json` — emit the raw structured payload, same shape as the
`tools/call health_report` MCP response
The check matrix is platform-aware: `bundle_identity` / `tcc_*` are
`skip` on Windows + Linux because those concepts don't apply.
`ax_capability` checks AX on macOS, UIA on Windows, AT-SPI on Linux —
each with the right diagnostic hint when it can't reach.
## The agent cursor and sessions
When the agent acts, you'll see a **tinted overlay cursor** glide
across the screen to where each click / type / scroll lands. The real
OS cursor never moves. The overlay shows where the agent is acting. Each
Hermes run declares a public cua-driver **session name** (something like
`hermes-3a7b9c14d2e8`). The name labels cursor identity and related state, so
concurrent runs and subagents get distinct cursors. The MCP transport owns the
private lifecycle session inside the runtime; the public name does not.
The overlay cursor is cosmetic — captures, clicks, and typing all work
without it. Hermes disables it automatically where it is a known failure
mode: macOS (idle CPU burn), headless Linux / WSL2 / containers, and
**Linux X11 desktops** (the overlay is a fullscreen always-on-top window
that can get stuck over every workspace after an unclean session end,
wedging desktop input). Linux Wayland and Windows keep the overlay. Set
`computer_use.no_overlay: false` in `config.yaml` to force the cursor on
(or `true` to force it off) on any platform.
Tune the cursor with `cua-driver`'s CLI flags or the runtime
`set_agent_cursor_style` MCP tool — see
[cua.ai/docs/how-to-guides/driver/personalize-cursor](https://cua.ai/docs/how-to-guides/driver/personalize-cursor)
for the full menu (built-in `arrow` vs `teardrop` silhouette, custom
SVG / PNG / ICO via `--cursor-icon`, runtime gradient colors, bloom
halo).
## Going deeper — the cua-driver skill pack
Hermes keeps its wrapper skill (`skills/autonomous-ai-agents/computer-use/SKILL.md`)
focused on the Hermes-side `computer_use` workflow and action vocabulary. For
platform details, recording semantics, browser page interaction, and other
deep Cua behavior, install the skill pack that the cua-driver team ships and
maintains directly:
```
cua-driver skills install
```
The command installs the pack under `~/.cua-driver/skills/cua-driver`. Hermes
autodetection is a planned cua-driver follow-up, so currently point Hermes at
that directory or symlink it into your skill space. The wrapper remains the
workflow layer and points to Cua's installed skill for driver behavior. The
pack contains:
| File | Topic |
|---|---|
| `SKILL.md` | The cross-platform core (snapshot invariant, no-foreground contract, click dispatch, AX-tree mechanics) |
| `MACOS.md` | macOS specifics: no-foreground contract, AXMenuBar navigation, SkyLight click dispatch, Apple Events JS bridge |
| `WINDOWS.md` | Windows specifics: UIA tree, UWP / `ApplicationFrameHost` hosting, Session 0 isolation, autostart pattern |
| `LINUX.md` | Linux specifics: AT-SPI tree, X11 / Wayland, terminal-emulator detection |
| `RECORDING.md` | Trajectory + video recording semantics |
| `WEB_APPS.md` | Browser-page interaction tips |
| `TESTS.md` | Replay-by-trajectory workflow |
These are **platform deep dives, not duplicates of the Hermes skill** —
when an agent reports "on Windows, my click landed on the wrong
element," it reads `WINDOWS.md` for the UIA / UWP context that
explains why and what to do differently.
`cua-driver skills status` shows what's installed and which agent
harnesses it's linked into. Today the autodetect list covers Claude
Code, Codex, OpenCode, OpenClaw, and Antigravity; **Hermes
autodetection is planned as a follow-up in `trycua/cua`** — until
then, run `cua-driver skills install` once and point your harness at
the resulting `~/.cua-driver/skills/cua-driver` directory (or symlink
it into your usual skill space).
## Quick example
User prompt: *"Find my latest email from Stripe and summarise what they want me to do."*
The agent's plan (this is the same shape on macOS / Windows / Linux —
the model substitutes the platform's idiomatic shortcut and app name):
1. `computer_use(action="capture", mode="som", app="Mail")` — gets a
screenshot of the email app with every sidebar item, toolbar button,
and message row numbered.
2. `computer_use(action="click", element=14)` — clicks the search field.
3. `computer_use(action="type", text="from:stripe")`
4. `computer_use(action="key", keys="return", capture_after=True)` —
submit and get the new screenshot.
5. Click the top result, read the body, summarise.
During all of this, your cursor stays wherever you left it and the email
app never comes to front.
## Receiving the actual screenshot
Screenshots taken during computer control are normally internal — they exist
so the model can see the screen, and the agent replies in text. But every
image capture also saves a bounded, shareable copy under Hermes' image cache
and reports its path, so on attachment-capable surfaces (Telegram, Discord,
Desktop, and other gateway platforms) you can simply ask:
> *"Send me a screenshot of my screen."*
and the agent delivers the real image as a native attachment, not just a
description. On the CLI there is no attachment channel, so the agent gives
you the saved file's path instead.
Only the 20 most recent capture files are kept, and screenshots are never
sent automatically — only when you ask for one.
### Whole screen vs. desktop surface
"Screenshot my screen" captures **everything currently displayed** — a
composited grab of all visible windows, like pressing PrtScn. This image has
no clickable elements, so to *act* on something in it the agent re-captures
the specific app.
Asking for the **desktop** instead targets the OS shell surface itself —
wallpaper, desktop icons, taskbar — with its clickable elements, so requests
like "open the Recycle Bin on my desktop" still work.
## Provider compatibility
| Provider | Vision? | Works? | Notes |
|---|---|---|---|
| Anthropic (Claude Sonnet/Opus 3+) | ✅ | ✅ | Best overall; SOM + raw coordinates. |
| OpenRouter (any vision model) | ✅ | ✅ | Multi-part tool messages supported. |
| OpenAI (GPT-4+, GPT-5) | ✅ | ✅ | Same as above. |
| Google (Gemini 2+) | ✅ | ✅ | Tool-calling + vision both supported. |
| Local vLLM / LM Studio / Ollama (vision model) | ✅ | ✅ | If the model supports multi-part tool content. |
| Text-only models | ❌ | ✅ (degraded) | Use `mode="ax"` for accessibility-tree-only operation. |
Screenshots are sent inline with tool results as OpenAI-style `image_url`
parts. For Anthropic, the adapter converts them into native `tool_result`
image blocks. The image MIME type comes from cua-driver's explicit
`mimeType` field (`image/png` or `image/jpeg`) — no client-side
magic-byte sniffing.
## Safety
Hermes applies multi-layer guardrails:
- Destructive actions (click, type, drag, scroll, key, focus_app)
require approval — either interactively via the CLI dialog or via the
messaging-platform approval buttons.
- Hard-blocked key combos at the tool level: empty trash, force delete,
lock screen, log out, force log out.
- Hard-blocked type patterns: `curl | bash`, `sudo rm -rf /`, fork
bombs, etc.
- The agent's system prompt tells it explicitly: no clicking permission
dialogs, no typing passwords, no following instructions embedded in
screenshots.
Pair with `approvals.mode: manual` in `~/.hermes/config.yaml` if you
want every action confirmed.
## Token efficiency
Screenshots are expensive. Hermes applies four layers of optimisation:
- **Screenshot eviction** — the Anthropic adapter keeps only the 3 most
recent screenshots in context; older ones become `[screenshot removed
to save context]` placeholders.
- **Client-side compression pruning** — the context compressor detects
multimodal tool results and strips image parts from old ones.
- **Image-aware token estimation** — each image is counted as ~1500
tokens (Anthropic's flat rate) instead of its base64 char length.
- **Server-side context editing (Anthropic only)** — when active, the
adapter enables `clear_tool_uses_20250919` via `context_management` so
Anthropic's API clears old tool results server-side.
A 20-action session on a 1568×900 display typically costs ~30K tokens
of screenshot context, not ~600K.
## Limitations
- **Performance.** Background mode is slower than foreground —
accessibility-routed events take ~5–20 ms on macOS, ~3–10 ms on
Windows UIA, ~5–15 ms on Linux AT-SPI vs direct HID posting. Not
noticeable for agent-speed clicking; noticeable if you try to record
a speed-run.
- **No keyboard password entry.** `type` has hard-block patterns on
command-shell payloads; for passwords, use the system's autofill
(macOS Keychain / Windows Credential Manager / GNOME Keyring /
KWallet).
- **Some apps don't expose an accessibility tree.** Modern UWP apps on
Windows, Electron < 28 on Linux, and a few macOS apps with custom
drawing (Logic, Final Cut, some games) have sparse or empty AX trees.
Fall back to pixel coordinates if the tree is empty — or skip the
task entirely.
- **Windows: elevated (admin) windows can't be driven from a normal
agent.** Windows UIPI (User Interface Privilege Isolation) enforces
integrity-level boundaries: a Medium-integrity process (the default
Hermes agent) cannot enumerate the UIA tree of, or inject mouse input
into, a window owned by a High-integrity (Administrator) process.
Symptom: `capture(mode='som')` returns 0 elements and `click(...)`
reports success while doing nothing, even though the screenshot
renders fine (GDI capture sits below the integrity check). Keyboard
events partially bypass UIPI, so Tab / Enter can still navigate an
elevated dialog. This is an OS constraint, not a cua-driver bug — it
affects every Windows automation stack. To drive elevated windows,
run the Hermes agent itself at High integrity (launch from an
elevated terminal); otherwise target non-elevated windows.
- **Platform-specific deployment gotchas:**
- **macOS** uses private SkyLight SPIs. Apple can change them in any
OS update. Hermes warns when the installed cua-driver is older than
the version it was tested against.
- **Windows** SSH sessions run in **Session 0**, which has no
interactive desktop. Drive Hermes from inside the RDP / console
session, or set up cua-driver's autostart Scheduled Task —
[windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh)
has the recipe.
- **Linux** requires a reachable display server. Headless servers
need Xvfb (`Xvfb :99 -screen 0 1920x1080x24`) before
`computer_use` can capture or inject events. Pure Wayland sessions
need an XWayland bridge for screen capture (cua-driver's Wayland
inject path handles input independently).
For cross-platform GUI automation without the desktop overhead (and
without TCC / Session 0 / X11 setup), the `browser` toolset uses a
real headless Chromium and is the right answer for web-only tasks.
## Configuration
Permission mode and manifest (see
[Permission modes](#permission-modes-and-logged-in-browser-profiles) above):
```yaml
computer_use:
permission_mode: standard # standard (default) | bounded
capability_manifest: "" # capability manifest path, required for bounded
```
On Linux, native Wayland support remains an explicit opt-in. Hermes passes the
opt-in to every cua-driver process, including gateway sessions, only when that
process also has `WAYLAND_DISPLAY`:
```yaml
computer_use:
native_wayland: true
```
Restart a running gateway after changing this setting.
Override the driver binary path (tests / CI / local builds):
```
HERMES_CUA_DRIVER_CMD=/path/to/your/cua-driver
```
Swap the backend entirely (for testing):
```
HERMES_COMPUTER_USE_BACKEND=noop # records calls, no side effects
```
### Telemetry
cua-driver ships with anonymous usage telemetry (PostHog) enabled by default
upstream. **Hermes disables it for you** — on every cua-driver invocation
(the MCP backend, `status`, `doctor`, and install) Hermes sets
`CUA_DRIVER_RS_TELEMETRY_ENABLED=0` in the driver's environment.
To opt back in (let cua-driver use its own default and send telemetry), set
this in `config.yaml`:
```yaml
computer_use:
cua_telemetry: true # default: false (telemetry off)
```
When it's on, `hermes computer-use doctor` reports `telemetry: enabled`;
when off (the default), it reports `telemetry: disabled via
CUA_DRIVER_RS_TELEMETRY_ENABLED`.
## Testing against a local cua-driver build
When you're developing cua-driver itself — or want to test an
unreleased fix — point Hermes at a binary you built from source instead
of the published release. Hermes resolves the driver with
`shutil.which("cua-driver")` and **does not enforce
`HERMES_CUA_DRIVER_VERSION`**, so a local build (reported as
`0.0.0-local-*`) is accepted as-is. Two approaches:
### Option A — `install-local` (build + put it on PATH)
From your `trycua/cua` checkout, run the upstream local installer. It
builds the Rust backend in release mode and drops `cua-driver` into the
same install layout the production installer uses, adding its bin dir
to your PATH:
```powershell
# Windows (PowerShell), from the cua repo root
./libs/cua-driver/scripts/install-local.ps1 -NoAutoStart
```
```bash
# macOS / Linux, from the cua repo root (defaults to a debug build without --release)
./libs/cua-driver/scripts/install-local.sh --release
```
- Windows stages the build under `%USERPROFILE%\.cua-driver\packages\…`
and junctions
`%LOCALAPPDATA%\Programs\Cua\cua-driver\bin` (added to your User
PATH) to it. macOS/Linux symlinks `cua-driver` into `~/.local/bin`
(override with `--bin-dir <path>`).
- `-NoAutoStart` skips registering the `cua-driver-serve` logon daemon
— you don't need it for Hermes testing (see notes).
Then open a fresh shell (so the PATH change is visible) and confirm:
```
cua-driver --version # local builds report 0.0.0-local-release
# Windows: (Get-Command cua-driver).Source
# macOS/Linux: which cua-driver
```
### Option B — point Hermes straight at the built binary (fastest loop)
Skip the install ceremony entirely: `cargo build` and set
`HERMES_CUA_DRIVER_CMD` to the resulting binary. Best for rapid
edit/build/test.
```bash
cargo build -p cua-driver # add --release for a release build; run from libs/cua-driver/rust
```
```
# Windows (.env)
HERMES_CUA_DRIVER_CMD=C:\path\to\cua\libs\cua-driver\rust\target\debug\cua-driver.exe
# macOS / Linux (.env)
HERMES_CUA_DRIVER_CMD=/path/to/cua/libs/cua-driver/rust/target/debug/cua-driver
```
### Confirm Hermes is using your build
- `hermes computer-use status` prints the resolved binary path and
version.
- `hermes computer-use doctor` confirms the binary is reachable and
exercises the full MCP path end-to-end.
- In a session, `computer_use(action="capture")` exercises the spawned
`cua-driver mcp` child process.
### Notes & gotchas
- **Hermes spawns a `cua-driver mcp` stdio proxy.** In a normal session the
proxy connects to (and may start) the standard machine daemon. In explicit
Hermes YOLO, Hermes instead owns a private `cua-driver serve --embedded`
child and points the proxy at its private socket or named pipe. The Windows
autostart/UIAccess pattern still matters for interactive Session 1+ input
from SSH — see the Limitations section.
- **Locked binary on Windows.** A running `cua-driver-serve` daemon can
hold `cua-driver.exe` and block an overwrite on rebuild.
`install-local.ps1` renames the locked binary out of the way
automatically; if you `cargo build` manually (Option B), stop it
first with `cua-driver autostart disable` (or `schtasks /End /TN
cua-driver-serve`).
- **Rebuild loop.** After editing cua-driver source, re-run
`install-local` (rebuilds, restages, flips the `current` junction)
for Option A, or just re-`cargo build` for Option B — no Hermes
change needed either way.
- **Local builds skip the version check.** Hermes warns when the
installed cua-driver is older than its per-OS tested baseline, but
exempts `0.0.0-local-*` dev builds — so your local build never
triggers that warning.
## Troubleshooting
**First action when anything's off: run `hermes computer-use doctor`.**
The structured per-check matrix tells you (and any agent helping you
debug) exactly what's wrong.
Specific failure modes the doctor doesn't catch:
**`computer_use backend unavailable: cua-driver is not installed`** —
Run `hermes computer-use install` to fetch the cua-driver binary, or
run `hermes tools` and enable the Computer Use toolset.
**Clicks seem to have no effect** — Capture and verify. A modal you
didn't see may be blocking input. Dismiss it with `escape` or the close
button.
**Element indices are stale** — SOM indices are only valid until the
next `capture`. Re-capture after any state-changing action. The
wrapper carries opaque `element_token`s for stale detection — you'll
see an explicit error rather than a wrong click.
**"blocked pattern in type text"** — The text you tried to `type`
matches the dangerous-shell-pattern list. Break the command up or
reconsider.
**Empty captures on Linux** — `DISPLAY` not set, or you're on pure
Wayland without an XWayland bridge. `hermes computer-use doctor` will
flag this as `ax_capability: fail` with a `Set DISPLAY (X11)…` hint.
**Empty captures on Windows over SSH** — You're in Session 0 (the
services session). Drive from RDP / console directly, or set up the
autostart pattern — see
[cua.ai/docs/how-to-guides/driver/windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh).
## See also
- **Hermes-side skill** — `skills/autonomous-ai-agents/computer-use/SKILL.md` — teaches the
Hermes `computer_use` action vocabulary; this is what the agent loads.
- **cua-driver skill pack** — for platform-specific deep dives
(macOS no-foreground contract, Windows UIA + Session 0, Linux AT-SPI
+ X11/Wayland, recording, browser pages), run
`cua-driver skills install` and read `MACOS.md` / `WINDOWS.md` /
`LINUX.md` / `RECORDING.md` / `WEB_APPS.md`. Hermes autodetection is a
planned follow-up; currently point Hermes at the installed pack directory
or symlink it into your skill space.
- **cua.ai/docs** — the cua-driver project's documentation:
- [What is computer use?](https://cua.ai/docs/explanation/what-is-computer-use) — concept intro
- [The no-foreground contract](https://cua.ai/docs/explanation/the-no-foreground-contract) — *why* background mode matters
- [Install reference](https://cua.ai/docs/how-to-guides/driver/install) — cross-platform install details
- [Personalize the agent cursor](https://cua.ai/docs/how-to-guides/driver/personalize-cursor) — built-in shapes, custom assets, runtime overrides
- [Drive Windows over SSH](https://cua.ai/docs/how-to-guides/driver/windows-ssh) — the Session 0 → Session 1+ autostart pattern
- [Keep cua-driver running](https://cua.ai/docs/how-to-guides/driver/keep-running) — autostart / daemon lifecycle
- [Connect your agent](https://cua.ai/docs/how-to-guides/driver/connect-your-agent) — register cua-driver with various harnesses (Hermes among them)
- [cua-driver source (trycua/cua)](https://github.com/trycua/cua)
- [Browser automation](./browser.md) for cross-platform web tasks where you don't need to drive native apps.
website/docs/user-guide/features/context-files.md
+237
View File
@@ -0,0 +1,237 @@
---
sidebar_position: 8
title: "Context Files"
description: "Project context files — .hermes.md, AGENTS.md, CLAUDE.md, global SOUL.md, and .cursorrules — automatically injected into every conversation"
---
# Context Files
Hermes Agent automatically discovers and loads context files that shape how it behaves. Some are project-local and discovered from your working directory. `SOUL.md` is now global to the Hermes instance and is loaded from `HERMES_HOME` only.
## Supported Context Files
| File | Purpose | Discovery |
|------|---------|-----------|
| **.hermes.md** / **HERMES.md** | Project instructions (highest priority) | Walks to git root |
| **AGENTS.override.md** | Personal, per-directory override of AGENTS.md (typically gitignored) | CWD at startup + subdirectories progressively |
| **AGENTS.md** | Project instructions, conventions, architecture | CWD at startup + subdirectories progressively |
| **CLAUDE.md** | Claude Code context files (also detected) | CWD at startup + subdirectories progressively |
| **SOUL.md** | Global personality and tone customization for this Hermes instance | `HERMES_HOME/SOUL.md` only |
| **.cursorrules** | Cursor IDE coding conventions | CWD only |
| **.cursor/rules/*.mdc** | Cursor IDE rule modules | CWD only |
:::info Priority system
Only **one** project context type is loaded per session (first match wins): `.hermes.md` → `AGENTS.override.md` → `AGENTS.md` → `CLAUDE.md` → `.cursorrules`. **SOUL.md** is always loaded independently as the agent identity (slot #1).
If an `AGENTS.override.md` exists next to an `AGENTS.md`, the override is loaded **instead of** the committed file — keep a personal (usually gitignored) `AGENTS.override.md` when you want different instructions than the ones checked into the repo, without editing the tracked `AGENTS.md`.
:::
## AGENTS.md
`AGENTS.md` is the primary project context file. It tells the agent how your project is structured, what conventions to follow, and any special instructions.
### Directory Chain (git root → working directory)
When your working directory sits inside a git repository, Hermes loads a **merged chain** of `AGENTS.md` files at session start: the git-root `AGENTS.md` first, then the `AGENTS.md` in every intermediate directory down to your working directory. Deeper files appear later in the prompt, so more specific guidance takes precedence. Each file gets its own provenance header (e.g. `## ../../AGENTS.md`), and identical copies along the chain are deduplicated.
```
monorepo/ (git root, cwd = packages/webapp/)
├── AGENTS.md ← Loaded first (repo-wide conventions)
└── packages/
├── AGENTS.md ← Loaded second
└── webapp/
└── AGENTS.md ← Loaded last (most specific, takes precedence)
```
Outside a git repository, only the working directory itself is checked — parents are never consulted, so an `AGENTS.md` planted in `/tmp` or `$HOME` can't leak into unrelated sessions.
### Progressive Subdirectory Discovery
At session start, Hermes loads the `AGENTS.md` from your working directory into the system prompt. As the agent navigates into subdirectories during the session (via `read_file`, `terminal`, `search_files`, etc.), it **progressively discovers** context files in those directories and injects them into the conversation at the moment they become relevant.
```
my-project/
├── AGENTS.md ← Loaded at startup (system prompt)
├── frontend/
│ └── AGENTS.md ← Discovered when agent reads frontend/ files
├── backend/
│ └── AGENTS.md ← Discovered when agent reads backend/ files
└── shared/
└── AGENTS.md ← Discovered when agent reads shared/ files
```
This approach has two advantages over loading everything at startup:
- **No system prompt bloat** — subdirectory hints only appear when needed
- **Prompt cache preservation** — the system prompt stays stable across turns
Each subdirectory is checked at most once per session. The discovery also walks up parent directories, so reading `backend/src/main.py` will discover `backend/AGENTS.md` even if `backend/src/` has no context file of its own.
:::info
Subdirectory context files go through the same [security scan](#security-prompt-injection-protection) as startup context files. Malicious files are blocked.
:::
### Example AGENTS.md
```markdown
# Project Context
This is a Next.js 14 web application with a Python FastAPI backend.
## Architecture
- Frontend: Next.js 14 with App Router in `/frontend`
- Backend: FastAPI in `/backend`, uses SQLAlchemy ORM
- Database: PostgreSQL 16
- Deployment: Docker Compose on a Hetzner VPS
## Conventions
- Use TypeScript strict mode for all frontend code
- Python code follows PEP 8, use type hints everywhere
- All API endpoints return JSON with `{data, error, meta}` shape
- Tests go in `__tests__/` directories (frontend) or `tests/` (backend)
## Important Notes
- Never modify migration files directly — use Alembic commands
- The `.env.local` file has real API keys, don't commit it
- Frontend port is 3000, backend is 8000, DB is 5432
```
## SOUL.md
`SOUL.md` controls the agent's personality, tone, and communication style. See the [Personality](/user-guide/features/personality) page for full details.
**Location:**
- `~/.hermes/SOUL.md`
- or `$HERMES_HOME/SOUL.md` if you run Hermes with a custom home directory
Important details:
- Hermes seeds a default `SOUL.md` automatically if one does not exist yet
- Hermes loads `SOUL.md` only from `HERMES_HOME`
- Hermes does not probe the working directory for `SOUL.md`
- If the file is empty, nothing from `SOUL.md` is added to the prompt
- If the file has content, the content is injected verbatim after scanning and truncation
## .cursorrules
Hermes is compatible with Cursor IDE's `.cursorrules` file and `.cursor/rules/*.mdc` rule modules. If these files exist in your project root and no higher-priority context file (`.hermes.md`, `AGENTS.md`, or `CLAUDE.md`) is found, they're loaded as the project context.
This means your existing Cursor conventions automatically apply when using Hermes.
## How Context Files Are Loaded
### At startup (system prompt)
Context files are loaded by `build_context_files_prompt()` in `agent/prompt_builder.py`:
1. **Scan working directory** — checks for `.hermes.md` → `AGENTS.md` → `CLAUDE.md` → `.cursorrules` (first match wins)
2. **Content is read** — each file is read as UTF-8 text
3. **Security scan** — content is checked for prompt injection patterns
4. **Truncation** — files exceeding the character cap are head/tail truncated (70% head, 20% tail, with a marker in the middle). The cap is an explicit `context_file_max_chars` from config.yaml when set; otherwise it scales dynamically with the model's context window (floor 20,000 chars, ceiling 500,000)
5. **Assembly** — all sections are combined under a `# Project Context` header
6. **Injection** — the assembled content is added to the system prompt
### During the session (progressive discovery)
`SubdirectoryHintTracker` in `agent/subdirectory_hints.py` watches tool call arguments for file paths:
1. **Path extraction** — after each tool call, file paths are extracted from arguments (`path`, `workdir`, shell commands)
2. **Ancestor walk** — the directory and up to 5 parent directories are checked (stopping at already-visited directories)
3. **Hint loading** — if an `AGENTS.md`, `CLAUDE.md`, or `.cursorrules` is found, it's loaded (first match per directory)
4. **Security scan** — same prompt injection scan as startup files
5. **Truncation** — capped at 8,000 characters per file
6. **Injection** — appended to the tool result, so the model sees it in context naturally
The final prompt section looks roughly like:
```text
# Project Context
The following project context files have been loaded and should be followed:
## AGENTS.md
[Your AGENTS.md content here]
## .cursorrules
[Your .cursorrules content here]
[Your SOUL.md content here]
```
Notice that SOUL content is inserted directly, without extra wrapper text.
## Security: Prompt Injection Protection
All context files are scanned for potential prompt injection before being included. The scanner checks for:
- **Instruction override attempts**: "ignore previous instructions", "disregard your rules"
- **Deception patterns**: "do not tell the user"
- **System prompt overrides**: "system prompt override"
- **Hidden HTML comments**: `<!-- ignore instructions -->`
- **Hidden div elements**: `<div style="display:none">`
- **Credential exfiltration**: `curl ... $API_KEY`
- **Secret file access**: `cat .env`, `cat credentials`
- **Invisible characters**: zero-width spaces, bidirectional overrides, word joiners
If any threat pattern is detected, the file is blocked:
```
[BLOCKED: AGENTS.md contained potential prompt injection (prompt_injection). Content not loaded.]
```
:::warning
This scanner protects against common injection patterns, but it's not a substitute for reviewing context files in shared repositories. Always validate AGENTS.md content in projects you didn't author.
:::
## Size Limits
| Limit | Value |
|-------|-------|
| Max chars per file | `context_file_max_chars` when set; otherwise dynamic (scales with model context window, floor 20,000, ceiling 500,000) |
| Read timeout per file | `context_file_read_timeout` (default 5 seconds); a file that takes longer to read — e.g. on iCloud Drive, OneDrive or NFS — is skipped with a warning |
| Head truncation ratio | 70% |
| Tail truncation ratio | 20% |
| Truncation marker | 10% (shows char counts and suggests using file tools) |
When a file exceeds the configured limit, the truncation message reads:
```
[...truncated AGENTS.md: kept 14000+4000 of 25000 chars. Use file tools to read the full file.]
```
## Tips for Effective Context Files
:::tip Best practices for AGENTS.md
1. **Keep it concise** — stay under your configured `context_file_max_chars`; the agent reads it every turn
2. **Structure with headers** — use `##` sections for architecture, conventions, important notes
3. **Include concrete examples** — show preferred code patterns, API shapes, naming conventions
4. **Mention what NOT to do** — "never modify migration files directly"
5. **List key paths and ports** — the agent uses these for terminal commands
6. **Update as the project evolves** — stale context is worse than no context
:::
### Per-Subdirectory Context
For monorepos, put subdirectory-specific instructions in nested AGENTS.md files:
```markdown
<!-- frontend/AGENTS.md -->
# Frontend Context
- Use `pnpm` not `npm` for package management
- Components go in `src/components/`, pages in `src/app/`
- Use Tailwind CSS, never inline styles
- Run tests with `pnpm test`
```
```markdown
<!-- backend/AGENTS.md -->
# Backend Context
- Use `poetry` for dependency management
- Run the dev server with `poetry run uvicorn main:app --reload`
- All endpoints need OpenAPI docstrings
- Database models are in `models/`, schemas in `schemas/`
```
website/docs/user-guide/features/context-references.md
+142
View File
@@ -0,0 +1,142 @@
---
sidebar_position: 9
sidebar_label: "Context References"
title: "Context References"
description: "Inline @-syntax for attaching files, folders, git diffs, and URLs directly into your messages"
---
# Context References
Type `@` followed by a reference to inject content directly into your message. Hermes expands the reference inline and appends the content under an `--- Attached Context ---` section.
## Supported References
| Syntax | Description |
|--------|-------------|
| `@file:path/to/file.py` | Inject file contents |
| `@file:path/to/file.py:10-25` | Inject specific line range (1-indexed, inclusive) |
| `@folder:path/to/dir` | Inject directory tree listing with file metadata |
| `@diff` | Inject `git diff` (unstaged working tree changes) |
| `@staged` | Inject `git diff --staged` (staged changes) |
| `@git:5` | Inject last N commits with patches (max 10) |
| `@url:https://example.com` | Fetch and inject web page content |
## Usage Examples
```text
Review @file:src/main.py and suggest improvements
What changed? @diff
Compare @file:old_config.yaml and @file:new_config.yaml
What's in @folder:src/components?
Summarize this article @url:https://arxiv.org/abs/2301.00001
```
Multiple references work in a single message:
```text
Check @file:main.py, and also @file:test.py.
```
Trailing punctuation (`,`, `.`, `;`, `!`, `?`) is automatically stripped from reference values.
## CLI Tab Completion
In the interactive CLI, typing `@` triggers autocomplete:
- `@` shows all reference types (`@diff`, `@staged`, `@file:`, `@folder:`, `@git:`, `@url:`)
- `@file:` and `@folder:` trigger filesystem path completion with file size metadata
- Bare `@` followed by partial text shows matching files and folders from the current directory
## Line Ranges
The `@file:` reference supports line ranges for precise content injection:
```text
@file:src/main.py:42 # Single line 42
@file:src/main.py:10-25 # Lines 10 through 25 (inclusive)
```
Lines are 1-indexed. Invalid ranges are silently ignored (full file is returned).
## Size Limits
Context references are bounded to prevent overwhelming the model's context window:
| Threshold | Value | Behavior |
|-----------|-------|----------|
| Soft limit | 25% of context length | Warning appended, expansion proceeds |
| Hard limit | 50% of context length | Expansion refused, original message returned unchanged |
| Folder entries | 200 files max | Excess entries replaced with `- ...` |
| Git commits | 10 max | `@git:N` clamped to range [1, 10] |
## Security
### Sensitive Path Blocking
These paths are always blocked from `@file:` references to prevent credential exposure:
- SSH keys and config: `~/.ssh/id_rsa`, `~/.ssh/id_ed25519`, `~/.ssh/authorized_keys`, `~/.ssh/config`
- Shell profiles: `~/.bashrc`, `~/.zshrc`, `~/.profile`, `~/.bash_profile`, `~/.zprofile`
- Credential files: `~/.netrc`, `~/.pgpass`, `~/.npmrc`, `~/.pypirc`
- Hermes env: `$HERMES_HOME/.env`
These directories are fully blocked (any file inside):
- `~/.ssh/`, `~/.aws/`, `~/.gnupg/`, `~/.kube/`, `$HERMES_HOME/skills/.hub/`
### Path Traversal Protection
All paths are resolved relative to the working directory. References that resolve outside the allowed workspace root are rejected.
### Binary File Detection
Binary files are detected via MIME type and null-byte scanning. Known text extensions (`.py`, `.md`, `.json`, `.yaml`, `.toml`, `.js`, `.ts`, etc.) bypass MIME-based detection. Binary files are rejected with a warning.
## Platform Availability
Context references are primarily a **CLI feature**. They work in the interactive CLI where `@` triggers tab completion and references are expanded before the message is sent to the agent.
In **messaging platforms** (Telegram, Discord, etc.), the `@` syntax is not expanded by the gateway — messages are passed through as-is. The agent itself can still reference files via the `read_file`, `search_files`, and `web_extract` tools.
## Interaction with Context Compression
When conversation context is compressed, the expanded reference content is included in the compression summary. This means:
- Large file contents injected via `@file:` contribute to context usage
- If the conversation is later compressed, the file content is summarized (not preserved verbatim)
- For very large files, consider using line ranges (`@file:main.py:100-200`) to inject only relevant sections
## Common Patterns
```text
# Code review workflow
Review @diff and check for security issues
# Debug with context
This test is failing. Here's the test @file:tests/test_auth.py
and the implementation @file:src/auth.py:50-80
# Project exploration
What does this project do? @folder:src @file:README.md
# Research
Compare the approaches in @url:https://arxiv.org/abs/2301.00001
and @url:https://arxiv.org/abs/2301.00002
```
## Error Handling
Invalid references produce inline warnings rather than failures:
| Condition | Behavior |
|-----------|----------|
| File not found | Warning: "file not found" |
| Binary file | Warning: "binary files are not supported" |
| Folder not found | Warning: "folder not found" |
| Git command fails | Warning with git stderr |
| URL returns no content | Warning: "no content extracted" |
| Sensitive path | Warning: "path is a sensitive credential file" |
| Path outside workspace | Warning: "path is outside the allowed workspace" |
website/docs/user-guide/features/credential-pools.md
+267
View File
@@ -0,0 +1,267 @@
---
title: Credential Pools
description: Pool multiple API keys or OAuth tokens per provider for automatic rotation and rate limit recovery.
sidebar_label: Credential Pools
sidebar_position: 9
---
# Credential Pools
Credential pools let you register multiple API keys or OAuth tokens for the same provider. When one key hits a rate limit or billing quota, Hermes automatically rotates to the next healthy key — keeping your session alive without switching providers.
This is different from [fallback providers](./fallback-providers.md), which switch to a *different* provider entirely. Credential pools are same-provider rotation; fallback providers are cross-provider failover. Pools are tried first — if all pool keys are exhausted, *then* the fallback provider activates.
:::warning Key rotation resets the prompt cache
Provider-side prompt caches (Anthropic, OpenAI, OpenRouter) are scoped to the account/API key that made the request. When the pool rotates to a different key mid-session, the new key has no cached prefix for your conversation — the next request re-reads the full history at undiscounted input price, and rotating back later is another full re-read unless the earlier key's cache TTL is still alive. Rotation keeps your session running, which is the point, but on long conversations each rotation costs one full-price pass over the context.
:::
:::tip
Credential pools are mainly for API-key providers (OpenRouter, Anthropic). A single [Nous Portal](/integrations/nous-portal) OAuth covers 300+ models, so most users don't need a pool when on Portal.
:::
## How It Works
```
Your request
→ Pick key from pool (round_robin / least_used / fill_first / random)
→ Send to provider
→ 429 rate limit?
→ Plan/usage limit reached (e.g. ChatGPT/Codex "usage limit reached")?
→ Rotate to next pool key immediately (no retry — the cap won't clear on retry)
→ Generic / transient 429?
→ Retry same key once (transient blip)
→ Second 429 → rotate to next pool key
→ All keys exhausted → fallback_model (different provider)
→ 402 billing error?
→ Immediately rotate to next pool key (1h cooldown)
→ 401 auth expired?
→ Try refreshing the token (OAuth)
→ Refresh failed → rotate to next pool key
→ Success → continue normally
```
## Quick Start
If you already have an API key set in `.env`, Hermes auto-discovers it as a 1-key pool. To benefit from pooling, add more keys:
```bash
# Add a second OpenRouter key
hermes auth add openrouter --api-key sk-or-v1-your-second-key
# Add a second Anthropic key
hermes auth add anthropic --type api-key --api-key sk-ant-api03-your-second-key
# Add an Anthropic OAuth credential (requires Claude Max plan + extra usage credits)
hermes auth add anthropic --type oauth
# Opens browser for OAuth login
```
Check your pools:
```bash
hermes auth list
```
Output:
```
openrouter (2 credentials):
#1 OPENROUTER_API_KEY api_key env:OPENROUTER_API_KEY ←
#2 backup-key api_key manual
anthropic (3 credentials):
#1 hermes_pkce oauth hermes_pkce ←
#2 claude_code oauth claude_code
#3 ANTHROPIC_API_KEY api_key env:ANTHROPIC_API_KEY
```
The `←` marks the currently selected credential.
## Interactive Management
Run `hermes auth` with no subcommand for an interactive wizard:
```bash
hermes auth
```
This shows your full pool status and offers a menu:
```
What would you like to do?
1. Add a credential
2. Remove a credential
3. Reset cooldowns for a provider
4. Set rotation strategy for a provider
5. Exit
```
For providers that support both API keys and OAuth (Anthropic, Nous, Codex), the add flow asks which type:
```
anthropic supports both API keys and OAuth login.
1. API key (paste a key from the provider dashboard)
2. OAuth login (authenticate via browser)
Type [1/2]:
```
## CLI Commands
| Command | Description |
|---------|-------------|
| `hermes auth` | Interactive pool management wizard |
| `hermes auth list` | Show all pools and credentials |
| `hermes auth list <provider>` | Show a specific provider's pool |
| `hermes auth add <provider>` | Add a credential (prompts for type and key) |
| `hermes auth add <provider> --type api-key --api-key <key>` | Add an API key non-interactively |
| `hermes auth add <provider> --type oauth` | Add an OAuth credential via browser login |
| `hermes auth remove <provider> <index>` | Remove credential by 1-based index |
| `hermes auth reset <provider>` | Clear all cooldowns/exhaustion status |
## Rotation Strategies
Configure via `hermes auth` → "Set rotation strategy" or in `config.yaml`:
```yaml
credential_pool_strategies:
openrouter: round_robin
anthropic: least_used
```
| Strategy | Behavior |
|----------|----------|
| `fill_first` (default) | Use the first healthy key until it's exhausted, then move to the next |
| `round_robin` | Cycle through keys evenly, rotating after each selection |
| `least_used` | Always pick the key with the lowest request count |
| `random` | Random selection among healthy keys |
## Error Recovery
The pool handles different errors differently:
| Error | Behavior | Cooldown |
|-------|----------|----------|
| **429 Rate Limit** | Retry same key once (transient). Second consecutive 429 rotates to next key | 1 hour |
| **402 Billing/Quota** | Immediately rotate to next key | 1 hour |
| **401 Auth Expired** | Try refreshing the OAuth token first. Rotate only if refresh fails | 5 minutes |
| **All keys exhausted** | Fall through to `fallback_model` if configured | — |
Provider-supplied `reset_at` timestamps override these default cooldowns.
The `has_retried_429` flag resets on every successful API call, so a single transient 429 doesn't trigger rotation.
## Custom Endpoint Pools
Custom OpenAI-compatible endpoints (Together.ai, RunPod, local servers) get their own pools, keyed by the endpoint name from the `providers:` dict in config.yaml (or the legacy `custom_providers` list, which is auto-migrated).
When you set up a custom endpoint via `hermes model`, it auto-generates a name like "Together.ai" or "Local (localhost:8080)". This name becomes the pool key.
```bash
# After setting up a custom endpoint via hermes model:
hermes auth list
# Shows:
# Together.ai (1 credential):
# #1 config key api_key config:Together.ai ←
# Add a second key for the same endpoint:
hermes auth add Together.ai --api-key sk-together-second-key
```
Custom endpoint pools are stored in `auth.json` under `credential_pool` with a `custom:` prefix:
```json
{
"credential_pool": {
"openrouter": [...],
"custom:together.ai": [...]
}
}
```
## Auto-Discovery
Hermes automatically discovers credentials from multiple sources and seeds the pool on startup:
| Source | Example | Auto-seeded? |
|--------|---------|-------------|
| Environment variables | `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY` | Yes |
| OAuth tokens (auth.json) | Codex device code, Nous device code | Yes |
| Claude Code credentials | `~/.claude/.credentials.json` | Yes (Anthropic) |
| Hermes PKCE OAuth | `~/.hermes/auth.json` | Yes (Anthropic) |
| Custom endpoint config | `model.api_key` in config.yaml | Yes (custom endpoints) |
| Manual entries | Added via `hermes auth add` | Persisted in auth.json |
Auto-seeded entries are updated on each pool load — if you remove an env var, its pool entry is automatically pruned. Manual entries (added via `hermes auth add`) are never auto-pruned.
Borrowed runtime secrets (for example env vars, Bitwarden/Vault/keyring/systemd references, and custom config values) are reference-only at the `auth.json` boundary. Hermes can use the resolved value in memory for the current run, but it persists only metadata such as the source ref, label, status, request counters, and a non-reversible fingerprint. Manual entries and Hermes-owned OAuth/device-code state keep the durable tokens they need to refresh.
## Delegation & Subagent Sharing
When the agent spawns subagents via `delegate_task`, the parent's credential pool is automatically shared with children:
- **Same provider** — the child receives the parent's full pool, enabling key rotation on rate limits
- **Different provider** — the child loads that provider's own pool (if configured)
- **No pool configured** — the child falls back to the inherited single API key
This means subagents benefit from the same rate-limit resilience as the parent, with no extra configuration needed. Per-task credential leasing ensures children don't conflict with each other when rotating keys concurrently.
## Thread Safety
The credential pool uses a threading lock for all state mutations (`select()`, `mark_exhausted_and_rotate()`, `try_refresh_current()`, `mark_used()`). This ensures safe concurrent access when the gateway handles multiple chat sessions simultaneously.
Across processes (many subagents, a gateway plus a CLI, cron jobs), OAuth refreshes are serialized through a file lock on `auth.json`. When one shared OAuth grant expires under many concurrent processes, exactly one process performs the refresh; the others detect that the on-disk token no longer matches the one that failed and adopt it instead of rotating the single-use refresh token again. A process that loses the lock race keeps its entry healthy and retries — lock contention is never recorded as a credential failure.
## Architecture
For the full data flow diagram, see [`docs/credential-pool-flow.excalidraw`](https://excalidraw.com/#json=2Ycqhqpi6f12E_3ITyiwh,c7u9jSt5BwrmiVzHGbm87g) in the repository.
The credential pool integrates at the provider resolution layer:
1. **`agent/credential_pool.py`** — Pool manager: storage, selection, rotation, cooldowns
2. **`hermes_cli/auth_commands.py`** — CLI commands and interactive wizard
3. **`hermes_cli/runtime_provider.py`** — Pool-aware credential resolution
4. **`run_agent.py`** — Error recovery: 429/402/401 → pool rotation → fallback
## Storage
Pool state is stored in `~/.hermes/auth.json` under the `credential_pool` key:
```json
{
"version": 1,
"credential_pool": {
"openrouter": [
{
"id": "abc123",
"label": "OPENROUTER_API_KEY",
"auth_type": "api_key",
"priority": 0,
"source": "env:OPENROUTER_API_KEY",
"secret_source": "bitwarden",
"secret_fingerprint": "sha256:12ab34cd56ef7890",
"last_status": "ok",
"request_count": 142
}
],
"anthropic": [
{
"id": "manual1",
"label": "personal-api-key",
"auth_type": "api_key",
"priority": 0,
"source": "manual",
"access_token": "sk-ant-api03-..."
}
]
}
}
```
The OpenRouter entry above was borrowed from an external source, so the raw key is not stored in `auth.json`. The manual Anthropic entry was intentionally added to Hermes' credential store, so its token remains persistable.
Strategies are stored in `config.yaml` (not `auth.json`):
```yaml
credential_pool_strategies:
openrouter: round_robin
anthropic: least_used
```
website/docs/user-guide/features/cron.md
+1126
View File
File diff suppressed because it is too large Load Diff
website/docs/user-guide/features/curator.md
+401
View File
@@ -0,0 +1,401 @@
---
sidebar_position: 3
title: "Curator"
description: "Background maintenance for agent-created skills — usage tracking, staleness, archival, and LLM-driven review"
---
# Curator
The curator is a background maintenance pass for **agent-created skills**. It tracks how often each skill is viewed, used, and patched, moves long-unused skills through `active → stale → archived` states, and periodically spawns a short auxiliary-model review that proposes consolidations or patches drift.
It exists so that skills created via the [self-improvement loop](/user-guide/features/skills#agent-managed-skills-skill_manage-tool) don't pile up forever. Every time the agent solves a novel problem and saves a skill, that skill lands in `~/.hermes/skills/`. Without maintenance, you end up with dozens of narrow near-duplicates that pollute the catalog and waste tokens.
By default (`prune_builtins: true`) the curator can archive **unused bundled built-in skills** (shipped with the repo) after `archive_after_days` of non-use, alongside the agent-created skills it primarily manages. Hub-installed skills (from [agentskills.io](https://agentskills.io)) are always off-limits. Set `curator.prune_builtins: false` to restore the old agent-created-only behavior, where bundled skills are never touched. The curator also **never auto-deletes** — the worst outcome is archival into `~/.hermes/skills/.archive/`, which is recoverable.
Tracks [issue #7816](https://github.com/NousResearch/hermes-agent/issues/7816).
## How it runs
The curator is triggered by an inactivity check, not a cron daemon. On CLI session start, and on a recurring tick inside the gateway's cron-ticker thread, Hermes checks whether:
1. Enough time has passed since the last curator run (`interval_hours`, default **7 days**), and
2. The agent has been idle long enough (`min_idle_hours`, default **2 hours**).
If both are true, it spawns a background fork of `AIAgent` — the same pattern used by the memory/skill self-improvement nudges. The fork runs in its own prompt cache and never touches the active conversation.
:::info First-run behavior
On a brand-new install (or the first time a pre-curator install ticks after `hermes update`), the curator **does not run immediately**. The first observation seeds `last_run_at` to "now" and defers the first real pass by one full `interval_hours`. This gives you a full interval to review your skill library, pin anything important, or opt out entirely before the curator ever touches it.
If you want to see what the curator *would* do before it runs for real, run `hermes curator run --dry-run` — it produces the same review report without mutating the library.
:::
A run has two phases:
1. **Automatic transitions** (deterministic, no LLM). Skills unused for `stale_after_days` (30) become `stale`; skills unused for `archive_after_days` (90) are moved to `~/.hermes/skills/.archive/`. This is the always-on pruning behavior — it runs whenever the curator is enabled, with no aux-model cost.
- **Pinned skills** and **skills referenced by any cron job** (including paused/disabled jobs) are skipped entirely — treated like pin for auto-transitions so a slow or paused schedule cannot archive a skill out from under a job. Consolidation also rewrites cron skill references when it merges umbrellas.
- **Never-used skills** (`use_count == 0`) get a grace floor: they are not archived until they are at least `stale_after_days` old. Zero uses is absence of evidence, not proof the skill is disposable.
2. **LLM consolidation** (single aux-model pass with a high iteration ceiling — a full curation sweep typically takes 50–100 API calls) — **OFF by default**. When `curator.consolidate: true`, the forked agent surveys the agent-created skills, can read any of them with `skill_view`, and decides per-skill whether to keep, patch (via `skill_manage`), consolidate overlapping ones into class-level umbrellas, or archive via the terminal tool. Consolidation treats a skill as a full package: if a skill has `references/`, `templates/`, `scripts/`, `assets/`, or relative links to those paths, the curator must either keep it standalone, re-home the needed support files and rewrite paths, or archive the entire package unchanged — not flatten only `SKILL.md` into another skill's `references/` file.
:::info Consolidation is opt-in
By default the curator only **prunes** — the deterministic inactivity pass marks skills stale and archives long-unused ones. The opinionated LLM **consolidation** pass (umbrella-building, merging overlapping skills) is off by default because it costs aux-model tokens on every run and makes broad structural changes to your library. Turn it on with `curator.consolidate: true`, or run it once on demand with `hermes curator run --consolidate`.
:::
Pinned skills are off-limits to both the curator's auto-transitions and the agent's own `skill_manage` tool. See [Pinning a skill](#pinning-a-skill) below.
## Configuration
All settings live in `config.yaml` under `curator:` (not `.env` — this isn't a secret). Defaults:
```yaml
curator:
enabled: true
interval_hours: 168 # 7 days
min_idle_hours: 2
stale_after_days: 30
archive_after_days: 90
consolidate: false # LLM umbrella-building pass — opt-in (prune-only by default)
prune_builtins: true # archive unused bundled built-in skills too (hub skills always exempt)
```
To disable entirely, set `curator.enabled: false`. To keep the always-on pruning but opt into LLM consolidation, set `curator.consolidate: true`.
### Running the review on a cheaper aux model
The curator's LLM review pass is a regular auxiliary task slot — `auxiliary.curator` — alongside Vision, Compression, Session Search, etc. "Auto" means "use my main chat model"; override the slot to pin a specific provider + model for the review pass instead.
**Easiest — `hermes model`:**
```bash
hermes model # → "Auxiliary models — side-task routing"
# → pick "Curator" → pick provider → pick model
```
The same picker is available in the web dashboard under the **Models** tab.
**Direct config.yaml (equivalent):**
```yaml
auxiliary:
curator:
provider: openrouter
model: google/gemini-3-flash-preview
timeout: 600 # generous — reviews can take several minutes
```
Leaving `provider: auto` (the default) routes the review pass through whatever your main chat model is, matching the behavior of every other auxiliary task.
:::note Legacy config
Earlier releases used a one-off `curator.auxiliary.{provider,model}` block. That path still works but emits a deprecation log line — please migrate to `auxiliary.curator` above so the curator shares the same plumbing (`hermes model`, dashboard Models tab, `base_url`, `api_key`, `timeout`, `extra_body`) as every other aux task.
:::
## CLI
```bash
hermes curator status # last run, counts, pinned list, LRU top 5
hermes curator run # trigger a run now (blocks until done). Prune-only unless curator.consolidate: true
hermes curator run --consolidate # force the LLM consolidation pass on for this run, overriding the config default
hermes curator run --background # fire-and-forget: start the run in a background thread
hermes curator run --dry-run # preview only — report without any mutations
hermes curator backup # take a manual snapshot of ~/.hermes/skills/
hermes curator rollback # restore from the newest snapshot
hermes curator rollback --list # list available snapshots
hermes curator rollback --id <ts> # restore a specific snapshot
hermes curator rollback -y # skip the confirmation prompt
hermes curator pause # stop runs until resumed
hermes curator resume
hermes curator pin <skill> # never auto-transition this skill
hermes curator unpin <skill>
hermes curator adopt <skill> # hand an unmanaged skill to the curator
hermes curator adopt --all-unmanaged # hand over every unmanaged skill
hermes curator list-unmanaged # itemize skills with no provenance marker
hermes curator restore <skill> # move an archived skill back to active
hermes curator list-archived # list skills currently in ~/.hermes/skills/.archive/
hermes curator archive <skill> # manually archive a single skill now
hermes curator prune [--days N] # bulk-archive agent-created skills idle >= N days (default 90)
hermes curator ledger # list the per-mutation audit ledger (all actors)
hermes curator ledger --skill <name> --limit 50 # filter/paginate ledger entries
hermes curator rollback <entry-id> # undo a single mutation from the ledger
hermes curator purge [--days N] [--dry-run] # delete archived skills older than the TTL (explicit only)
```
## Backups and rollback
Before every real curator pass, Hermes takes a tar.gz snapshot of `~/.hermes/skills/` at `~/.hermes/skills/.curator_backups/<utc-iso>/skills.tar.gz`. If a pass archives or consolidates something you didn't want touched, you can undo the whole run with one command:
```bash
hermes curator rollback # restore newest snapshot (with confirmation)
hermes curator rollback -y # skip the prompt
hermes curator rollback --list # see all snapshots with reason + size
```
The rollback itself is reversible: before replacing the skills tree, Hermes takes another snapshot tagged `pre-rollback to <target-id>`, so a mistaken rollback can be undone by rolling forward to that one with `--id`.
You can also take manual snapshots at any time with `hermes curator backup --reason "before-refactor"`. The `--reason` string lands in the snapshot's `manifest.json` and is shown in `--list`.
Snapshots are pruned to `curator.backup.keep` (default 5) to keep disk usage bounded:
```yaml
curator:
backup:
enabled: true
keep: 5
```
Set `curator.backup.enabled: false` to disable automatic snapshotting. The manual `hermes curator backup` command still works when backups are disabled only if you set `enabled: true` first — the flag gates both paths symmetrically so there's no way to accidentally skip the pre-run snapshot on mutating runs.
`hermes curator status` also lists the five least-recently-used skills — a quick way to see what's likely to become stale next.
The same subcommands are available as the `/curator` slash command inside a running session (CLI or gateway platforms).
## Audit ledger and single-edit rollback
Whole-run snapshots answer "undo everything the last curator pass did" — but sometimes you want to know *who changed what* and undo exactly one mutation. Every skill mutation — curator auto-transitions, agent `skill_manage` calls, and your own CLI archive/restore/purge — appends one entry to the append-only JSONL ledger at `~/.hermes/skills/.curator_ledger.jsonl`:
- **actor** — `curator` (background review fork / auto-transitions), `agent` (foreground agent tool calls), or `user` (CLI commands)
- **action** — `create`, `edit`, `patch`, `delete`, `write_file`, `remove_file`, `archive`, `restore`, `purge`, `rollback`
- **evidence** — delete intent (`absorbed_into` for consolidations, empty for prunes, and whether the recoverable-archive path handled it), triggering session id when available
- **before/after** — per-file `{path, sha256}` manifests. File contents are stored content-addressed (deduped by hash) under `~/.hermes/.curator_backups/blobs/`, so a hundred entries touching the same unchanged file cost one blob.
```bash
hermes curator ledger # newest 20 entries
hermes curator ledger --skill my-skill --limit 50
hermes curator rollback <entry-id> # restore that one mutation's before-state
```
Single-entry rollback restores exactly the files that mutation touched (and removes files it created) from the blob store — nothing else in the skills tree moves. Like whole-tree rollback, it takes a safety ledger entry of the current state first and **fails closed**: if the safety capture can't be written, nothing is changed. Because foreground deletes are ledgered too, `hermes curator rollback <entry-id>` can resurrect a hard-deleted skill.
The ledger is telemetry, never a gate — if writing an entry fails, the mutation still goes through. Disable it with:
```yaml
skills:
ledger: false
```
## Archive TTL purge
Archived skills are kept forever by default. If you want `~/.hermes/skills/.archive/` bounded, set a TTL and purge explicitly — purging never runs automatically, and every purged skill is captured into the ledger (with blobs) first, so even a purge leaves an auditable, recoverable trail:
```yaml
curator:
archive_ttl_days: 180 # 0 (default) = never purge
```
```bash
hermes curator purge --dry-run # preview what would be deleted
hermes curator purge # delete archives older than the TTL (with confirmation)
hermes curator purge --days 90 # one-off TTL override
```
## What "agent-created" means
The curator only manages skills explicitly marked as **agent-created** in
`~/.hermes/skills/.usage.json`. A skill qualifies when ALL of the following
are true:
1. Its name is **not** in `~/.hermes/skills/.bundled_manifest` (bundled skills shipped with the repo).
2. Its name is **not** in `~/.hermes/skills/.hub/lock.json` (hub-installed skills).
3. Its `.usage.json` entry has `"created_by": "agent"` or `"agent_created": true`.
Currently, only the **background self-improvement review fork** sets this marker
— when it creates a new umbrella skill during its periodic review pass (~every 10
agent turns). The background fork runs with a write origin of `"background_review"`
(via `tools/skill_provenance.py`), which is the only path that triggers the
`mark_agent_created()` call in `skill_manage`.
Skills the foreground agent creates via `skill_manage(action="create")` during a
conversation are **not** marked as agent-created — they are considered
user-directed and the curator intentionally leaves them alone.
:::warning Your hand-written skills are NOT curated
If you manually created a `SKILL.md` or pointed Hermes at an external skill
directory, that skill will have a `.usage.json` entry with `created_by: null`
(or the field absent). The curator will not touch it. The same applies to
skills the foreground agent created at your request.
**To see which skills the curator actually manages**, run `hermes curator status`.
If the agent-created count is 0, no skills are currently in the curator's
jurisdiction — the LLM review pass is skipped and the report will show
`Model: (not resolved) via (not resolved)` with `Duration: 0s`.
:::
### Adopting unmanaged skills
`hermes curator status` reports an **unmanaged** count alongside the managed
one:
```
curator-managed skills: 43 total (agent-created=43 bundled=0)
active 41
stale 2
archived 0
unmanaged (no provenance marker): 112 total
pre-dates marker 34
foreground-created 78
never auto-staled or archived — `hermes curator adopt <name>` hands one over
```
Those 112 are curation-*eligible* but permanently invisible to the lifecycle,
for one of two reasons:
- **pre-dates marker** — the record was written before `created_by` existed, so
it carries no provenance signal at all. Authorship is genuinely unknowable
from the record.
- **foreground-created** — a foreground `skill_manage(create)` left the marker
unset by design, since skills you ask for belong to you.
A large library can therefore look fully curated while most of it is
untouchable. `adopt` closes that gap by **declaration**:
```bash
hermes curator list-unmanaged # itemize them, with reasons
hermes curator adopt <name> [<name> ...] # hand specific skills over
hermes curator adopt --all-unmanaged --dry-run # preview the full list
hermes curator adopt --all-unmanaged # hand over everything (prompts)
hermes curator adopt --all-unmanaged --yes # skip the prompt
```
Adoption writes the same `created_by: agent` marker the background review fork
writes. It does **not** reset the inactivity clock — an adopted skill keeps its
existing `last_activity_at`, so handing over a library you already stopped
using does not buy it a fresh 90-day window. Expect adopted long-idle skills to
go `stale` (or `archived`) on the next pass; that is the point.
Adoption is also what unblocks autonomous *improvement*. The background review
fork refuses to patch a skill that isn't curator-managed, so if it notices one
of your skills is outdated it will say so and recommend adoption rather than
edit it. Foreground (user-directed) edits are never affected — you and the
agent can always edit your own skills on request.
:::note `created_by` is a policy flag, not a provenance claim
The stored field is named `created_by`, but it is consumed as "may autonomous
curation touch this?" — not "who wrote this file". Those are different
questions, and for records predating the marker the authorship answer is simply
unrecoverable. The name is kept because it is already on disk in every
`.usage.json`; read it as policy. `hermes curator adopt` changes the policy, and
says nothing about who authored the file.
:::
:::note Provenance is declared, never inferred
Adoption is deliberately manual. Telemetry cannot establish authorship: a skill
with thousands of patches proves the agent **maintains** it, not that the agent
**wrote** it — Hermes edits user-authored skills on your behalf constantly. An
automatic "looks agent-made, adopt it" heuristic would eventually archive
something you hand-wrote. `adopt` refuses bundled, hub-installed, external, and
protected built-in skills, which have an owner other than you.
:::
Skills that ARE agent-created follow the full lifecycle:
- `active` → (30d unused) `stale` → (90d unused) `archived`
- Pinned skills bypass all auto-transitions
- Archives are recoverable via `hermes curator restore <name>`
If you want to protect a specific skill from ever being touched — for example a
hand-authored skill you rely on — use `hermes curator pin <name>`. See the next
section.
## Pinning a skill
Pinning protects a skill from deletion — both the curator's automated archive passes and the agent's `skill_manage(action="delete")` tool call. Once a skill is pinned:
- The **curator** skips it during auto-transitions (`active → stale → archived`), and its LLM review pass is instructed to leave it alone.
- The **agent's `skill_manage` tool** refuses `delete` on it, pointing the user at `hermes curator unpin <name>`. Patches and edits still go through, so the agent can improve a pinned skill's content as pitfalls come up without a pin/unpin/re-pin dance.
Pin and unpin with:
```bash
hermes curator pin <skill>
hermes curator unpin <skill>
```
The flag is stored as `"pinned": true` on the skill's entry in `~/.hermes/skills/.usage.json`, so it survives across sessions.
Skills named in any cron job's `skills:` list are protected the same way for **auto-transitions** (the curator never stales/archives them while the reference remains), even when the job is paused or disabled. Prefer an explicit pin when you also want `skill_manage delete` blocked.
Only **agent-created** skills can be pinned — `hermes curator pin` refuses on bundled and hub-installed skills with an explanatory message if you try. Hub-installed skills are never subject to curator mutation. Bundled built-in skills are only touched when `curator.prune_builtins: true` (the default), and even then only archived after `archive_after_days` of non-use — never patched, consolidated, or deleted. Set `curator.prune_builtins: false` to exempt bundled skills entirely.
A small set of **protected built-ins** can be hardcoded as never-archivable and never-consolidatable, regardless of `curator.prune_builtins`, pin state, or LLM judgment. These back load-bearing UX, so silently archiving one would turn its slash command into an "Unknown command" error with no signal to you. (The set is currently empty — `plan`, its original member, graduated to a built-in `/plan` command with no skill on disk.) Protected built-ins are filtered out of the curator's candidate list entirely, so the consolidation pass never sees them.
If you want a stronger guarantee than "no deletion" — for instance, freezing a skill's content entirely while the agent still reads it — edit `~/.hermes/skills/<name>/SKILL.md` directly with your editor. The pin guards tool-driven deletion, not your own filesystem access.
## Usage telemetry
The curator maintains a sidecar at `~/.hermes/skills/.usage.json` with one entry per skill:
```json
{
"my-skill": {
"use_count": 12,
"view_count": 34,
"last_used_at": "2026-04-24T18:12:03Z",
"last_viewed_at": "2026-04-23T09:44:17Z",
"patch_count": 3,
"last_patched_at": "2026-04-20T22:01:55Z",
"created_at": "2026-03-01T14:20:00Z",
"state": "active",
"pinned": false,
"archived_at": null
}
}
```
Counters increment when:
- `view_count`: the agent calls `skill_view` on the skill.
- `use_count`: the skill is loaded into a conversation's prompt.
- `patch_count`: `skill_manage patch/edit/write_file/remove_file` runs on the skill.
Bundled and hub-installed skills are explicitly excluded from telemetry writes.
## Per-run reports
Every curator run writes a timestamped directory under `~/.hermes/logs/curator/`:
```
~/.hermes/logs/curator/
└── 20260429-111512/
├── run.json # machine-readable: full fidelity, stats, LLM output
└── REPORT.md # human-readable summary
```
`REPORT.md` is a quick way to see what a given run did — which skills transitioned, what the LLM reviewer said, which skills it patched. Good for auditing without having to grep `agent.log`.
:::note No candidates? Report shows `(not resolved)`
When the curator has **no agent-created skills** to review, the LLM review pass
is skipped entirely. The report header will show
`Model: (not resolved) via (not resolved)` with `Duration: 0s` — this does **not**
indicate a configuration error or model resolution failure. It simply means there
were no candidates, so no model was ever invoked. The auto-transition phase still
runs and reports its counts normally.
:::
### Rename map in the summary
If a run consolidated multiple skills under an umbrella (or merged near-duplicates), the user-visible summary printed at the end of the run includes an explicit rename map showing every `old-name → new-name` pair the curator applied. This is in addition to per-skill transition lines, so when a wave of renames lands you can spot them at a glance without diffing the JSON report. The hint also surfaces under `hermes curator pin` so you can pin the umbrella name immediately if you want to lock the new label in.
## Restoring an archived skill
If the curator archived something you still want:
```bash
hermes curator restore <skill-name>
```
This moves the skill back from `~/.hermes/skills/.archive/` to the active tree and resets its state to `active`. The restore refuses if a bundled or hub-installed skill has since been installed under the same name (would shadow upstream).
## Disabling per environment
The curator is on by default. To turn it off:
- **For one profile only:** edit `~/.hermes/config.yaml` (or the active profile's config) and set `curator.enabled: false`.
- **For just one run:** `hermes curator pause` — the pause persists across sessions; use `resume` to re-enable.
The curator also refuses to run if `min_idle_hours` hasn't elapsed, so on an active dev machine it naturally only runs during quiet stretches.
## See also
- [Skills System](/user-guide/features/skills) — how skills work in general and the self-improvement loop that creates them
- [Memory](/user-guide/features/memory) — a parallel background review that maintains long-term memory
- [Bundled Skills Catalog](/reference/skills-catalog)
- [Issue #7816](https://github.com/NousResearch/hermes-agent/issues/7816) — original proposal and design discussion
website/docs/user-guide/features/delegation.md
+544
View File
@@ -0,0 +1,544 @@
---
sidebar_position: 7
title: "Subagent Delegation"
description: "Spawn isolated child agents for parallel workstreams with delegate_task"
---
# Subagent Delegation
The `delegate_task` tool spawns child AIAgent instances with isolated context, inherited tool access, and their own terminal sessions. Each child gets a fresh conversation and works independently — only its final summary enters the parent's context.
Top-level model calls run in the background automatically. Hermes returns a handle immediately so the conversation can continue, then posts the result back as a new message. An orchestrator subagent waits for its own workers so it can synthesize their results before returning.
## Single Task
```python
delegate_task(
goal="Debug why tests fail",
context="Error: assertion in test_foo.py line 42"
)
```
## Parallel Batch
Up to 3 concurrent subagents by default (configurable, no hard ceiling):
```python
delegate_task(tasks=[
{"goal": "Research topic A", "context": "Focus on recent primary sources"},
{"goal": "Research topic B", "context": "Compare the leading explanations"},
{"goal": "Fix the build", "context": "Project root: /home/user/project"}
])
```
## How Subagent Context Works
:::warning Critical: Subagents Know Nothing
Subagents start with a **completely fresh conversation**. They have zero knowledge of the parent's conversation history, prior tool calls, or anything discussed before delegation. The subagent's only context comes from the `goal` and `context` fields the parent agent populates when it calls `delegate_task`.
:::
One exception: when the parent has a resolved workspace directory, every subagent's system prompt embeds that workspace's **project context files** (`.hermes.md` > AGENTS.md chain > CLAUDE.md > `.cursorrules` — the same discovery, priority, and size caps as the main agent's system prompt; SOUL.md is excluded). Subagents working in a repo operate under the repo's own conventions without having to rediscover them.
This means the parent agent must pass **everything** the subagent needs in the call:
```python
# BAD - subagent has no idea what "the error" is
delegate_task(goal="Fix the error")
# GOOD - subagent has all context it needs
delegate_task(
goal="Fix the TypeError in api/handlers.py",
context="""The file api/handlers.py has a TypeError on line 47:
'NoneType' object has no attribute 'get'.
The function process_request() receives a dict from parse_body(),
but parse_body() returns None when Content-Type is missing.
The project is at /home/user/myproject and uses Python 3.11."""
)
```
The subagent receives a focused system prompt built from your goal and context, instructing it to complete the task and provide a structured summary of what it did, what it found, any files modified, and any issues encountered.
## Practical Examples
### Parallel Research
Research multiple topics simultaneously and collect summaries:
```python
delegate_task(tasks=[
{
"goal": "Research the current state of WebAssembly in 2025",
"context": "Focus on: browser support, non-browser runtimes, language support"
},
{
"goal": "Research the current state of RISC-V adoption in 2025",
"context": "Focus on: server chips, embedded systems, software ecosystem"
},
{
"goal": "Research quantum computing progress in 2025",
"context": "Focus on: error correction breakthroughs, practical applications, key players"
}
])
```
### Code Review + Fix
Delegate a review-and-fix workflow to a fresh context:
```python
delegate_task(
goal="Review the authentication module for security issues and fix any found",
context="""Project at /home/user/webapp.
Auth module files: src/auth/login.py, src/auth/jwt.py, src/auth/middleware.py.
The project uses Flask, PyJWT, and bcrypt.
Focus on: SQL injection, JWT validation, password handling, session management.
Fix any issues found and run the test suite (pytest tests/auth/)."""
)
```
### Multi-File Refactoring
Delegate a large refactoring task that would flood the parent's context:
```python
delegate_task(
goal="Refactor all Python files in src/ to replace print() with proper logging",
context="""Project at /home/user/myproject.
Use the 'logging' module with logger = logging.getLogger(__name__).
Replace print() calls with appropriate log levels:
- print(f"Error: ...") -> logger.error(...)
- print(f"Warning: ...") -> logger.warning(...)
- print(f"Debug: ...") -> logger.debug(...)
- Other prints -> logger.info(...)
Don't change print() in test files or CLI output.
Run pytest after to verify nothing broke."""
)
```
## Batch Mode Details
When a top-level agent provides a `tasks` array, Hermes returns one background handle, runs the subagents in parallel, and posts one consolidated result after every child finishes. An orchestrator subagent waits for its batch in the current turn so it can synthesize the results.
- **Maximum concurrency:** 3 tasks by default (configurable via `delegation.max_concurrent_children` or the `DELEGATION_MAX_CONCURRENT_CHILDREN` env var; floor of 1, no hard ceiling). Batches larger than the limit return a tool error rather than being silently truncated.
- **Thread pool:** Uses `ThreadPoolExecutor` with the configured concurrency limit as max workers
- **Progress display:** In CLI mode, a tree-view shows tool calls from each subagent in real-time with per-task completion lines. In gateway mode, progress is batched and relayed to the parent's progress callback
- **Result ordering:** Results are sorted by task index to match input order regardless of completion order
- **Cancellation:** Follow-up messages do not cancel a top-level background batch. `/stop` or closing/resetting the owning session cancels its active children. Synchronous orchestrator children still follow their parent's interrupt state
Synchronous single-task delegation from an orchestrator runs directly without thread pool overhead.
### Durable background completions
When a background delegation finishes, Hermes stores its completion event in
the active profile's `state.db` before publishing it to the normal fresh-turn
queue. If Hermes restarts after completion but before delivery, the pending
event is restored and routed through the same ownership checks. Competing
consumers use a durable claim, so only the consumer that successfully accepts
the synthetic turn acknowledges delivery; failed attempts release the claim for
retry.
This does not resume child execution after a crash. A delegation whose owner
process disappears while it is still running is recorded as `unknown`, because
Hermes cannot prove whether its external side effects happened. Pending and
delivered records are bounded and profile-local.
### Child background-process notifications
Background processes a subagent starts (e.g. `npm ci` with
`notify_on_complete`) technically route their completion and watch-pattern
notifications to the **parent** conversation, because anything that outlives
the child needs a durable consumer. By default those notifications are
**suppressed** in the parent chat — the child's consolidated delegation result
is the deliverable, and mid-conversation "process finished" walls from a
child's internal builds are noise. Suppressed events are logged at debug level
with the process session ID and subagent task ID, so they remain diagnosable.
The delegation result itself is never suppressed. To restore delivery of the
child process notifications (each carries a "Started by subagent …"
attribution line):
```yaml
delegation:
surface_child_process_notifications: true # default: false
```
## Model Override
You can configure a different model for subagents via `config.yaml` — useful for delegating simple tasks to cheaper/faster models:
```yaml
# In ~/.hermes/config.yaml
delegation:
model: "google/gemini-flash-2.0" # Cheaper model for subagents
provider: "openrouter" # Optional: route subagents to a different provider
```
If omitted, subagents use the same model as the parent.
### Cost strategy: frontier planner, inexpensive workers
Decomposing a problem into well-specified subtasks takes frontier-level judgment; executing a subtask that already comes with a clear goal, full context, and an output contract usually doesn't. Meanwhile the children are where the tokens go — a parallel batch of subagents typically burns the large majority of a run's total tokens, so the worker model is where the cost actually lives. Pinning `delegation.model` to an inexpensive model while your main session stays on a frontier model keeps the planning quality where it matters and cuts spend where the volume is:
```yaml
# ~/.hermes/config.yaml
model:
default: "your-frontier-model" # parent (planner) stays on the frontier model
delegation:
model: "your-inexpensive-model" # all delegate_task children run on this
provider: "openrouter" # optional: route children to a different provider
```
Resolution order: `delegation.base_url` (direct endpoint) takes precedence, then `delegation.provider` (full credential bundle resolved via the runtime provider system), and when neither is set children inherit the parent's provider and credentials; `delegation.model` applies in all cases, and when it is empty children inherit the parent's model. Setting `delegation.provider` alongside `delegation.base_url` keeps the explicit endpoint but carries that provider's request overrides and max output tokens into the child. An explicit `delegation.request_overrides` dict is honored on every branch and merges over those runtime-derived values (see [Configuration](#configuration) below).
Note that the pin is global: `delegate_task` has no per-task model parameter, so every child in a batch runs on the configured delegation model. For quality-sensitive subtasks that need a stronger model, either leave `delegation.model` unset for that session or hand the task to the [kanban board](kanban.md#per-task-model-override), which does support a per-task model override.
## The `/review` Command
`/review` spawns an independent, full-privilege background subagent whose only job is to review the work your conversation just produced — a PR, a diff, code, documentation, a design. It works on every surface: CLI, TUI, the Desktop app, and every gateway messaging platform.
```
/review # review whatever the last 10 messages presented
/review focus on security # add extra instructions for the reviewer
```
What happens:
1. The last 10 user/assistant messages are snapshotted as the reviewer's starting evidence (tool output and system messages are excluded).
2. A reviewer subagent is dispatched on the same background delegation rail as `delegate_task` — it gets the full normal subagent toolset (terminal, web, files, browser...), so it actually opens the PR, reads the diff, and runs code rather than judging from the excerpt.
3. The reviewer inherits the primary agent's working context: any skills the primary agent had loaded (launch-preloaded or via `skill_view` during the session) are named in its briefing with an instruction to load them and judge the work against their conventions. Like every subagent, its system prompt also embeds the workspace's project context files (AGENTS.md / CLAUDE.md / .cursorrules) as binding conventions.
4. When it finishes, its full review re-enters the same session as a normal background-subagent completion — your primary agent sees it and can act on it (fix the findings, push follow-ups, reply to you).
The canonical flow: your main agent opens a PR, you type `/review`, and a second pair of eyes investigates it while you keep working; the review lands back in the chat addressed to the agent that created the PR.
### Review model
By default the reviewer runs on your main model. To pin a dedicated review model, set `auxiliary.review` in `config.yaml`:
```yaml
auxiliary:
review:
provider: openrouter # or nous, anthropic, a direct base_url, ...
model: anthropic/claude-opus-4.6 # a strong reviewer model
```
Credentials resolve exactly like a `delegation.provider` pin (full runtime-provider bundle: base_url, api key, api_mode). `provider: auto` with an empty `model` means "inherit the main agent's model" — the default.
`/review` is deliberately separate from `/refine`: `/refine` reviews the conversation to update memory and skills, `/review` reviews the *work product* the conversation created.
## Inherited Tool Access
`delegate_task` does not accept a model-facing `toolsets` parameter. Each subagent inherits the parent's enabled toolsets so the model cannot grant a child capabilities that the parent does not have. Configure the parent's tools before starting the conversation if delegated work needs additional capabilities.
Certain tools are blocked for subagents even when the parent has them:
- `delegate_task` — blocked for leaf subagents (the default). Retained for `role="orchestrator"` children, bounded by `max_spawn_depth` — see [Depth Limit and Nested Orchestration](#depth-limit-and-nested-orchestration) below.
- `clarify` — subagents cannot interact with the user
- `memory` — no writes to shared persistent memory
- `send_message` — no cross-platform side effects
- `cronjob` — no scheduling more work in the parent's name
Both roles retain `execute_code` (programmatic tool calling) so children can batch mechanical work.
## Max Iterations
Each subagent has an iteration limit (default: 50) that controls how many tool-calling turns it can take:
```python
delegate_task(
goal="Quick file check",
context="Check if /etc/nginx/nginx.conf exists and print its first 10 lines",
max_iterations=10 # Simple task, don't need many turns
)
```
## Child Timeout
By default there is **no wall-clock timeout** on subagents. Children fail only from what they're actually doing — API errors, tool errors, or hitting their iteration budget — never from a delegation-level stopwatch. Earlier releases shipped a hard cap (300s, later 600s), which kept killing legitimately busy children mid-task: deep code reviews, large research fan-outs, and slow reasoning models routinely need more than 10 minutes while making steady progress the whole time.
Genuinely stuck children are still detected: the heartbeat staleness monitor stops refreshing the parent's activity when a child makes no progress (no API calls, no tool starts, and no activity-timestamp ticks), letting the gateway inactivity timeout fire on a truly wedged worker. An in-flight model wait still counts as progress — subagents refresh the activity clock while waiting on the provider, so a slow local / long-prefill completion is not treated as stalled.
If you want a hard cap anyway (e.g. cost control on unattended cron-driven delegation), opt in per-install:
```yaml
delegation:
child_timeout_seconds: 0 # default: 0 = no timeout
# child_timeout_seconds: 1800 # opt-in hard cap (floor 30s)
```
A positive value enforces a hard wall-clock limit on each child; `0` or a negative value disables it.
When a configured cap fires, the child's result carries structured timeout
metadata alongside the error message so parents and hooks can distinguish a
stopwatch kill from other failures without parsing text: `timeout_seconds`
(the configured cap), `timed_out_after_seconds` (actual wall clock), and
`timeout_phase` (`before_first_llm_call` when the child never reached its
first request, `after_llm_calls` otherwise). All three are `null` on
non-timeout errors.
## Failure Visibility
A subagent that fails — non-retryable provider error (404/400), timeout, crash, or no usable output — is never silent:
- **CLI**: the delegation tree prints a one-line reason: `⚠️ Subagent failed — "your goal": HTTP 404: model not found (after 12s)`. Batch runs append the reason to the per-task `✗` completion line.
- **Gateway platforms** (Telegram, Discord, Slack, ...): the same clean line is delivered as a standalone chat notice, **even when `tool_progress` is off** for that platform.
- **Parent agent**: the tool result entry carries `status: "failed"` plus the full `error` text, so the model can react (retry, re-route, report).
Error text is reduced to the single most informative line (the exception message, not a traceback wall) and capped in length.
:::tip Diagnostic dump on zero-call timeout
With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout-<session>-<timestamp>.log` containing the subagent's config snapshot, credential-resolution trace, any early error messages, and stack traces for **all** live threads (not just the child's own) — a child parked waiting on a nested helper thread is indistinguishable from a slow provider without the full picture.
:::
## Stall Detection for Background Subagents
Background delegations (`delegate_task(background=true)`) are watched by a
**progress-based stall monitor** — on by default, zero config. Unlike a
wall-clock timeout, it never touches a child that is making progress, no
matter how long it runs.
The monitor samples each detached child's progress signals — API-call count,
current tool, and last-activity timestamp (which ticks on **every streamed
token**, tool transition, and API-call boundary, so a child mid-stream on a
long response always counts as alive):
1. **Progressing children are never touched.** Any advancing signal resets
the clock.
2. A child whose progress is completely frozen past the stale threshold
(450s idle, 1200s while inside a tool — legitimately slow terminal
commands and web fetches get the higher ceiling) is **interrupted** and
given a 120s grace window. A child that unwinds in time delivers its
partial results through the normal completion path.
3. A child that never returns is force-finalized with a terminal `stalled`
completion event, so the owning session hears an outcome instead of
going silent, and the async slot frees for new work.
The `stalled` event carries structured metadata mirroring the sync-path
timeout fields: `stalled_after_quiet_seconds`, `stall_threshold_seconds`,
`stall_phase` (`idle` / `in_tool`), and `stall_grace_seconds`.
This closed a long-standing failure mode where a wedged background child
left its session looking dead until a process restart. The underlying wedge
(children hanging at their first API call after multi-day gateway uptime)
was also fixed at the root: delegated children now run their OpenAI-wire
API requests inline on their own conversation thread instead of a nested
worker thread — the layer where the wedge lived. The stall monitor remains
as the safety net for anything else.
## Monitoring Running Subagents (`/agents`)
The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegate_task` fan-out into a first-class audit surface:
- Live tree view of running and recently-finished subagents, grouped by parent
- Per-branch cost, token, and file-touched rollups
- Kill and pause controls — cancel a specific subagent mid-flight without interrupting its siblings
- Post-hoc review: step through each subagent's turn-by-turn history even after they've returned to the parent
The classic CLI just prints `/agents` as a text summary; the TUI is where the overlay shines. See [TUI — Slash commands](/user-guide/tui#slash-commands).
On the classic CLI and every gateway platform (Telegram, Discord, Slack, ...),
`/agents` also lists **background delegations with live per-child activity**,
sampled directly from each running child:
```
Background delegations: 1 running
- deleg_ab12cd34 · running · research the delegation stall monitor
- child 1: 4 api calls · in web_search · active 12s ago
- child 2: 7 api calls · between turns · active 3s ago
```
A delegation the stall monitor has flagged shows as
`stalling · no progress 450s — interrupting`, and long-quiet-but-healthy
children show their quiet time so you can tell "slow" from "stuck" at a
glance.
## Steering a Running Subagent
Interrupting a child throws away its in-flight work; often you just want to redirect it.
### From the parent agent (model-facing)
The parent agent orchestrates its own running children with the same `delegate_task` tool it spawned them with — no separate control tool:
```json
{"action": "list"}
{"action": "steer", "subagent_id": "sa-0-1a2b3c4d", "message": "focus on pricing instead"}
{"action": "stop", "subagent_id": "sa-0-1a2b3c4d"}
```
- **`list`** returns the conversation's live children: `subagent_id`, goal, status, `running_seconds`, `accepting_steer`, and the live transcript path. Ids also come back in the spawn dispatch response as `subagent_ids`.
- **`steer`** queues a course correction into a running child without stopping it (delivery semantics below).
- **`stop`** ends a child early at its next iteration boundary; the partial result still re-enters the conversation as a normal completion message.
Control actions run synchronously in-turn (never backgrounded), are scoped to the caller's own spawn tree — a conversation can never see or control another session's children — and never consume the per-turn subagent spawn cap, so `stop` keeps working even after the cap is hit.
### From the TUI / gateway (session-facing)
`steer_subagent(subagent_id, text)` in `tools/delegate_tool.py` is the redirection-side mirror of `interrupt_subagent()`: it queues text into a live child through the same mechanism as [`/steer`](/reference/slash-commands) — the text is appended to the child's last tool result at its next iteration boundary, the in-flight tool call is never cut, and the child sees it as an out-of-band user message. Programmatic hosts reach it through the session-scoped `subagent.steer` gateway RPC, which sits beside `subagent.interrupt`:
```json
{"method": "subagent.steer", "params": {"session_id": "owning-ui-session", "subagent_id": "sa-0-1a2b3c4d", "text": "focus on pricing instead"}}
```
Subagent ids come from `delegation.status` (or `list_active_subagents()`) — the same place `subagent.interrupt` gets them. The gateway accepts steering only from the exact live UI/gateway session that spawned the child. A missing, foreign, ambiguous, or stale/recycled session identity is rejected; knowing a global subagent id is not authority. Direct in-process callers retain the unscoped helper contract deliberately.
**Queued is not delivered, but it is never synthetic success.** A `"queued"` response means the text was accepted before the child's completion boundary, not necessarily that the child has seen it. Acceptance and completion are synchronized: either the child can still consume the text, or its exact text is drained into the result as `pending_steer`. Calls after closure return `"rejected"`. If a child accepted the steer but had already produced its final answer, the completion entry the parent receives retains it as `missed_steer`, with a note appended to the summary:
```
[steer did not land — the subagent finished before it could be delivered: focus on pricing instead]
```
So the parent (or the operator driving it) can tell a steered child from one that finished on the old instructions, and re-issue the guidance as a follow-up instead of trusting that it landed.
## Live Transcripts
Every `delegate_task` dispatch also creates one **append-only, human-readable log per task** so you (or the parent agent) can watch a subagent work in real time instead of waiting for the consolidated summary:
```
<hermes_home>/cache/delegation/live/<delegation_id>/task-<n>.log
```
The dispatch response includes the paths as `live_transcripts`, and the files are pre-created at dispatch time, so this works immediately:
```bash
tail -f ~/.hermes/cache/delegation/live/deleg_ab12cd34/task-0.log
```
Each line is timestamped and shows the child's assistant text, thinking snippets, tool calls (`-> tool_name({args})`), tool results, and a final status marker. A `manifest.json` in the same directory describes the batch (goals, task count, per-task status). The logs persist after completion — they double as the full-fidelity operational record alongside the summary — and directories older than 7 days are pruned automatically on new dispatches. Because they live under `cache/delegation`, they are also readable from remote terminal backends (Docker/Modal/SSH).
## Depth Limit and Nested Orchestration
By default, delegation is **flat**: a parent (depth 0) spawns children (depth 1), and those children cannot delegate further. This prevents runaway recursive delegation.
For multi-stage workflows (research → synthesis, or parallel orchestration over sub-problems), a parent can spawn **orchestrator** children that *can* delegate their own workers:
```python
delegate_task(
goal="Survey three code review approaches and recommend one",
role="orchestrator", # Allows this child to spawn its own workers
context="...",
)
```
- `role="leaf"` (default): child cannot delegate further — identical to the flat-delegation behavior.
- `role="orchestrator"`: child retains the `delegation` toolset. Gated by `delegation.max_spawn_depth` (default **1** = flat, so `role="orchestrator"` is a no-op at defaults). Raise `max_spawn_depth` to 2 to allow orchestrator children to spawn leaf grandchildren; 3+ for deeper trees. There is no upper ceiling — cost is the practical limit.
- `delegation.orchestrator_enabled: false`: global kill switch that forces every child to `leaf` regardless of the `role` parameter.
**Cost warning:** With `max_spawn_depth: 3` and `max_concurrent_children: 3`, the tree can reach 3×3×3 = 27 concurrent leaf agents. Each extra level multiplies spend — raise `max_spawn_depth` intentionally.
## Lifetime and Durability
:::warning Background completion durability is not durable execution
Top-level model-facing `delegate_task` calls run in the background automatically where the session supports later delivery. Hermes returns a handle immediately, and the result re-enters the conversation after the child or batch finishes. Orchestrator subagents wait for their workers in the current turn because they must synthesize those results before returning. Stateless request/response endpoints fall back to synchronous execution when they cannot deliver a detached result later.
- Normal follow-up messages do not cancel background children. `/stop` cancels running background delegations, and closing or resetting the owning session discards its active children.
- Explicit session close/reset interrupts that session's background children. Closing a TUI viewer of a gateway-owned session does not kill the gateway's work.
- A Hermes process restart does **not** resume a running child. Its attempt becomes `unknown` because Hermes cannot prove which side effects happened.
- A child that completed before restart but whose result was not delivered is restored and routed back through the owning session's normal checks.
- Cancelled children return a structured result (`status="interrupted"`, `exit_reason="interrupted"`), but because the parent was interrupted too, that result often never makes it into a user-visible reply.
For **durable execution** that must survive session closure or process restart, use:
- `cronjob` (action=`create`) — schedules a separate agent run; immune to parent-turn interrupts.
- `terminal(background=True, notify_on_complete=True)` — long-running shell commands that keep running while the agent does other things.
:::
## Key Properties
- Each subagent gets its **own terminal session** (separate from the parent)
- Subagents inherit the parent's enabled toolsets; the model cannot select or widen them per call
- **Nested delegation is opt-in** — only `role="orchestrator"` children can delegate further, and only when `max_spawn_depth` is raised from its default of 1 (flat). Disable globally with `orchestrator_enabled: false`.
- Leaf subagents **cannot** call: `delegate_task`, `clarify`, `memory`, `send_message`, `cronjob`. Orchestrator subagents retain `delegate_task` but keep the other blocks. Both roles retain `execute_code` (programmatic tool calling) so children can batch mechanical work instead of burning reasoning iterations.
- **Cancellation follows ownership** — `/stop` or closing/resetting the owning session cancels its background children; synchronous descendants under orchestrators follow their parent's interrupt state
- Only the final summary enters the parent's context, keeping token usage efficient
- Subagents inherit the parent's **API key, provider configuration, and credential pool** (enabling key rotation on rate limits)
## Worktree Isolation
By default, subagents share the parent's working directory — fine for research
and read-heavy work, but parallel children editing the same repo can collide.
Set `delegation.worktree_isolation: true` to give each child its own git
worktree, branched from the repo's current `HEAD` (inspired by Muse Code's
`--subagent-worktree-isolation`):
```yaml
delegation:
worktree_isolation: true # default: false
```
With isolation on:
- Each child starts its terminal in `<repo>/.worktrees/subagent-<id>` on its
own branch `hermes-subagent/subagent-<id>`, and its goal message tells it to
work and commit there.
- The parent's checkout stays untouched; children can't clobber each other's
edits.
- When a child finishes, its result entry gains a `worktree` field reporting
`path`, `branch`, `commits` (ahead of the base), and `dirty`. The parent
reviews or merges each branch (`git log <branch>`, `git merge <branch>`).
- A worktree left with **no commits and a clean tree is pruned automatically**
(`pruned: true`); anything holding work is kept.
- Pruning requires proof. If a git inspection probe fails — or finalization
itself errors — the worktree and branch are kept and the entry carries
`inspection_failed: true` plus a `note` — `commits`/`dirty` are then
defaults, not measurements, so inspect the worktree rather than assuming
the child produced nothing.
Scope: opt-in, git-only, and local-terminal-backend-only. In a non-git
directory, on docker/ssh/modal backends, or if worktree creation fails, the
setting degrades silently to today's shared-workspace behavior — never an
error.
## Delegation vs execute_code
| Factor | delegate_task | execute_code |
|--------|--------------|-------------|
| **Reasoning** | Full LLM reasoning loop | Just Python code execution |
| **Context** | Fresh isolated conversation | No conversation, just script |
| **Tool access** | All non-blocked tools with reasoning | 7 tools via RPC, no reasoning |
| **Parallelism** | 3 concurrent subagents by default (configurable) | Single script |
| **Best for** | Complex tasks needing judgment | Mechanical multi-step pipelines |
| **Token cost** | Higher (full LLM loop) | Lower (only stdout returned) |
| **User interaction** | None (subagents can't clarify) | None |
**Rule of thumb:** Use `delegate_task` when the subtask requires reasoning, judgment, or multi-step problem solving. Use `execute_code` when you need mechanical data processing or scripted workflows.
## Configuration
```yaml
# In ~/.hermes/config.yaml
delegation:
max_iterations: 50 # Max turns per child (default: 50)
# max_concurrent_children: 3 # Parallel children per batch (default: 3)
# worktree_isolation: false # Give each child its own git worktree (see Worktree Isolation above)
# max_spawn_depth: 1 # Tree depth (floor 1, no ceiling, default 1 = flat). Raise to 2 to allow orchestrator children to spawn leaves; 3+ for deeper trees.
# orchestrator_enabled: true # Disable to force all children to leaf role.
model: "google/gemini-3-flash-preview" # Optional provider/model override
provider: "openrouter" # Optional built-in provider
api_mode: anthropic_messages # optional; auto-detected from base_url for anthropic_messages endpoints
# Or use a direct custom endpoint instead of provider:
delegation:
model: "qwen2.5-coder"
base_url: "http://localhost:1234/v1"
api_key: "local-key"
# api_mode: "anthropic_messages" # Optional. Wire protocol override for base_url ("chat_completions", "codex_responses", or "anthropic_messages"). Empty = auto-detect from URL (e.g. /anthropic suffix). Set explicitly for endpoints the heuristic can't classify (Azure AI Foundry, MiniMax, Zhipu GLM, LiteLLM proxies, …).
# Send per-child request settings on every subagent API call — e.g. OpenRouter
# routing hints when delegating straight to openrouter.ai via base_url:
delegation:
model: "deepseek/deepseek-v4-flash-0731"
base_url: "https://openrouter.ai/api/v1"
api_key: "sk-or-..."
request_overrides:
extra_body:
provider:
sort: throughput # children route to the fastest OpenRouter provider
```
When `base_url` points at an Anthropic-compatible endpoint — for example a path ending in `/anthropic`, an Azure Foundry Claude route, or a MiniMax `/anthropic` proxy — `api_mode` is auto-detected as `anthropic_messages` so the subagent uses the right wire format without you setting anything. Set `api_mode` explicitly when the auto-detection guess is wrong (rare).
`delegation.request_overrides` works on **all three** resolution branches — direct `base_url`, named `provider`, and pure inherit — so it always takes effect. Top-level keys are API kwargs (e.g. `service_tier`); an `extra_body` sub-dict is merged into the request's `extra_body`. Explicit values merge **over** runtime- or parent-derived overrides: explicit top-level keys win, and `extra_body` is deep-merged one level, so a provider's own request personality (e.g. `thinking: {type: disabled}`) survives unless your key redefines it. See [Configuration → Delegation](../configuration.md#delegation) for details.
:::tip
The agent handles delegation automatically based on the task complexity. You don't need to explicitly ask it to delegate — it will do so when it makes sense.
:::
website/docs/user-guide/features/deliverable-mode.md
+134
View File
@@ -0,0 +1,134 @@
---
title: Deliverable Mode (Artifacts in Chat)
sidebar_label: Deliverable Mode
description: How the agent ships generated charts, PDFs, spreadsheets, and other files as native attachments in messaging platforms.
---
# Deliverable Mode
When Hermes Agent runs inside a messaging gateway (Slack, Discord, Telegram,
WhatsApp, Signal, etc.), it can deliver generated files directly into the
chat — not as paths the user has to copy, but as native attachments.
A chart shows up as an inline image. A PDF report shows up as a file
download. A spreadsheet uploads as `.xlsx`. The agent does not need to
write a `MEDIA:` tag or do anything special — it just generates the file
and mentions its absolute path in the response. The gateway picks the path
out of the text, removes it from the visible message, and uploads the
file natively.
## How it works
Three pieces fit together:
1. **The agent has tools that produce files.** `execute_code` for charts via
matplotlib, the `docx` skill for Word documents, the `xlsx` skill for
spreadsheets, the `pdf` and `latex-pdf-report` skills for PDFs, the
`powerpoint` skill for decks, `image_generate` for images,
`text_to_speech` for audio, and so on.
2. **The gateway scans agent responses for file paths.** Any absolute path
(`/tmp/...`) or home-relative path (`~/...`) ending in a supported
extension gets extracted. Paths inside code blocks and inline code are
ignored so code samples are never mutilated.
3. **The gateway dispatches by file type.** Images embed inline where the
platform supports it; videos embed inline; audio routes to voice/audio
attachments; everything else uploads as a file attachment.
## Supported file extensions
| Category | Extensions | Delivery |
|---|---|---|
| Images | `.png .jpg .jpeg .gif .webp .bmp .tiff .svg` | Inline embed |
| Video | `.mp4 .mov .avi .mkv .webm .3gp` | Inline embed (where supported) |
| Audio | `.mp3 .m2a .wav .ogg .opus .m4a .flac` | Voice / audio attachment |
| Documents | `.pdf .docx .doc .odt .rtf .txt .md .epub` | File upload |
| Data | `.xlsx .xls .ods .csv .tsv .json .xml .yaml .yml` | File upload |
| Geospatial | `.kmz .kml .geojson .gpx` | File upload |
| Presentations | `.pptx .ppt .odp .key` | File upload |
| Archives | `.zip .tar .gz .tgz .bz2 .xz .7z .rar .apk .ipa` | File upload |
| Web | `.html .htm` | File upload |
`.py`, `.log`, and other source-file extensions are intentionally excluded so
the agent doesn't auto-ship arbitrary source files; if you want to send code
to the user, use a code block.
## Encouraging the agent to produce artifacts
The agent doesn't reach for artifacts by default — it has to know to.
Two ways to nudge it:
**Per-session:** ask explicitly ("send me the comparison as a chart",
"return the data as a CSV") or write your own custom-instructions /
personality entry that biases toward artifact-style replies on
messaging platforms.
**Project-level:** add the bias to `AGENTS.md` / `CLAUDE.md` /
`.cursorrules` in a project the agent works from, to your global
persona in `~/.hermes/SOUL.md`, or as a named preset under
`agent.personalities` in `~/.hermes/config.yaml` (switchable per session
via `/personality`).
The mechanic the agent has to use is simple: render the file to an
absolute path (e.g. `/tmp/q3-revenue.png`) and mention that path as
plain text in the reply. The gateway does the rest. Paths inside
fenced code blocks or backticks are ignored so code samples are never
mutilated.
## Kanban: artifacts ride completion notifications
If you use Hermes' kanban multi-agent workflow, workers can attach
deliverable files to their `kanban_complete` call:
```python
kanban_complete(
summary="rendered Q3 revenue chart and report",
artifacts=[
"/tmp/q3-revenue.png",
"/tmp/q3-report.pdf",
],
)
```
When the gateway notifier delivers the "task completed" message to whoever
subscribed to the task in Slack/Telegram/etc., it also uploads each artifact
as a native attachment to that chat. The human gets the deliverable and the
summary in one place.
Files that don't exist on disk when the notifier runs are silently skipped.
## Connecting more services with MCP
Beyond the artifact-delivery pipeline, the agent can reach into other
services via MCP (Model Context Protocol). The MCP ecosystem ships
community servers for most popular tools — install whichever you need:
| Service | What it unlocks |
|---|---|
| **Notion** | Read/write Notion pages, databases, query workspace |
| **GitHub** | Issues, PRs, comments, repo search beyond the gh CLI |
| **Linear** | Tickets, projects, cycles |
| **Slack** | Workspace-wide search, read other channels |
| **Gmail** | Inbox triage, send mail, label management |
| **Salesforce** | Leads, opportunities, account data |
| **Snowflake / BigQuery** | SQL against data warehouses |
| **Google Drive** | File search, contents, share management |
Install MCP servers via `~/.hermes/config.yaml` under the `mcp_servers`
section. See [MCP integration](./mcp.md) for the full setup guide.
## Comparison to Perplexity Computer in Slack
Perplexity Computer's Slack integration is built around the same idea:
the agent generates a deliverable (chart, PDF, slide deck) and posts it
back into the thread as a native attachment. Hermes Agent's deliverable
mode provides the same user-facing pattern locally:
- Generation happens in the user's own venv / sandbox (no remote tenant).
- Files land in the chat via the same Slack `files.uploadV2` API.
- Connector breadth comes via MCP rather than a curated catalog of 400
hosted integrations — install the ones you actually use.
OAuth tokens stay on the user's machine in `auth.json` / `.env`. No hosted
token storage. No multi-tenant microVM. Same end result.
website/docs/user-guide/features/document-extraction.md
+58
View File
@@ -0,0 +1,58 @@
---
sidebar_position: 3
title: "Document Extraction"
description: "How read_file converts PDFs, Office documents, and notebooks to text — and what to do when a PDF is scanned images"
---
# Document Extraction
The `read_file` tool automatically converts common document formats to readable text, so the agent can inspect a PDF or spreadsheet the same way it reads source code.
## Supported formats
| Format | Extensions | Converter | Availability |
|--------|-----------|-----------|--------------|
| Jupyter notebooks | `.ipynb` | Built-in (stdlib) | Always |
| Word documents | `.docx` | Built-in (stdlib) | Always |
| Excel workbooks | `.xlsx` | Built-in (stdlib) | Always |
| PDF | `.pdf` | Optional `anydoc` converter | Auto-installed on first use* |
| Legacy Office | `.doc`, `.ppt`, `.xls`, `.pptx`, and variants | Optional `anydoc` converter | Auto-installed on first use* |
| OpenDocument | `.odt`, `.ods`, `.odp` | Optional `anydoc` converter | Auto-installed on first use* |
| Rich text / eBooks | `.rtf`, `.epub` | Optional `anydoc` converter | Auto-installed on first use* |
\* The optional converter is the `firecrawl-anydoc` package, installed lazily where installs are permitted (`security.allow_lazy_installs` in `config.yaml`). Without it, the three stdlib formats still work; other formats fall back to the binary-file guard.
Conversion output is Markdown, paginated through `read_file`'s normal `offset`/`limit` window. Documents over 50 MB are refused to keep tool turns bounded.
Extraction works with remote terminal backends (Docker, Modal, SSH): the file's bytes are transferred across the backend boundary and converted host-side, so a document inside a sandbox reads the same as a local one.
## Scanned PDFs: the coverage warning
PDF conversion reads the **text layer only**. Pages that are scanned images — common in legal documents, resale packages, signed contracts, faxes — contain no text layer and silently convert to nothing. The telltale signature is section headers with empty bodies.
When a meaningful share of pages yields no text (over 20% of the document, or 10+ pages absolute), `read_file` prepends a warning to the extraction. Each unreadable gap is labeled with the last text extracted before it — usually a section divider — so the agent can target only the gaps it actually needs instead of OCRing the whole document:
```
[EXTRACTION COVERAGE WARNING: 198 of 311 pages in this PDF yielded no
text. ... Unreadable gaps, each labeled with the last text extracted
before it:
pages 42-77 (36 pages) — after "Antigua Maintenance Corp Bylaws" (p41)
pages 92-213 (122 pages) — after "... Covenants, Codes and Regulations" (p91)
page 224 (1 page) — after "... Insurance Declaration Pages" (p223)
Decide which gaps you actually need — do NOT OCR or render everything. ...]
```
The warning lists the exact page ranges and the recovery paths:
1. **A few pages — render + vision.** Convert the pages to images and read them with the vision tool:
```bash
pdftoppm -jpeg -r 150 -f 92 -l 94 document.pdf /tmp/page
```
Then inspect each image with `vision_analyze`. Zero extra dependencies (poppler is required for the detection itself).
2. **Many pages — OCR.** The `ocr-and-documents` skill covers bulk OCR with marker-pdf (90+ languages, handles equations and tables; ~3-5 GB install).
Detection uses poppler's `pdftotext` for per-page text counts. If poppler is not installed, extraction still works — the coverage check is silently skipped.
:::tip
The agent handles the warning on its own — it will offer to render or OCR the missing pages. If you're reading extractions yourself, treat "header with an empty body" as a scanned section, not a missing one.
:::
website/docs/user-guide/features/extending-the-dashboard.md
+927
View File
@@ -0,0 +1,927 @@
---
sidebar_position: 17
title: "Extending the Dashboard"
description: "Build themes and plugins for the Hermes web dashboard — palettes, typography, layouts, custom tabs, shell slots, page-scoped slots, and backend API routes"
---
# Extending the Dashboard
The Hermes web dashboard (`hermes dashboard`) is built to be reskinned and extended without forking the codebase. Three layers are exposed:
1. **Themes** — YAML files that repaint the dashboard's palette, typography, layout, and per-component chrome. Drop a file in `~/.hermes/dashboard-themes/`; it appears in the theme switcher.
2. **UI plugins** — a directory with `manifest.json` + a JavaScript bundle that registers a tab, replaces a built-in page, augments one via page-scoped slots, or injects components into named shell slots.
3. **Backend plugins** — a Python file inside that plugin directory that exposes a FastAPI `router`; routes are mounted under `/api/plugins/<name>/` and called from the plugin's UI.
All three are **drop-in at runtime**: no repo clone, no `npm run build`, no patching the dashboard source. This page is the canonical reference for all three.
If you just want to use the dashboard, see [Web Dashboard](./web-dashboard). If you want to reskin the terminal CLI (not the web dashboard), see [Skins & Themes](./skins) — the CLI skin system is unrelated to dashboard themes.
:::note Not the desktop app
This page covers the **web dashboard** (`hermes dashboard`) plugin system — `window.__HERMES_PLUGIN_SDK__`, a `manifest.json`, and a pre-built JS bundle. The **native desktop app** (`hermes desktop`) has its own, unrelated SDK — `@hermes/plugin-sdk`, a single ESM file, no build step — documented at [Desktop Plugin SDK](/developer-guide/desktop-plugin-sdk). Only the backend `plugin_api.py` namespace (`/api/plugins/<name>`) is shared between them.
:::
:::note How the pieces compose
Themes and plugins are independent but synergistic. A theme can stand alone (just a YAML file). A plugin can stand alone (just a tab). Together they let you build a complete visual reskin with custom HUDs — the example `strike-freedom-cockpit` demo (lives in the `hermes-example-plugins` companion repo — see [Combined theme + plugin demo](#combined-theme--plugin-demo) for install steps) does exactly that.
:::
---
## Table of contents
- [Themes](#themes)
- [Quick start — your first theme](#quick-start--your-first-theme)
- [Palette, typography, layout](#palette-typography-layout)
- [Layout variants](#layout-variants)
- [Theme assets (images as CSS vars)](#theme-assets-images-as-css-vars)
- [Component chrome overrides](#component-chrome-overrides)
- [Color overrides](#color-overrides)
- [Raw `customCSS`](#raw-customcss)
- [Built-in themes](#built-in-themes)
- [Full theme YAML reference](#full-theme-yaml-reference)
- [Plugins](#plugins)
- [Quick start — your first plugin](#quick-start--your-first-plugin)
- [Directory layout](#directory-layout)
- [Manifest reference](#manifest-reference)
- [The Plugin SDK](#the-plugin-sdk)
- [Shell slots](#shell-slots)
- [Replacing built-in pages (`tab.override`)](#replacing-built-in-pages-taboverride)
- [Augmenting built-in pages (page-scoped slots)](#augmenting-built-in-pages-page-scoped-slots)
- [Slot-only plugins (`tab.hidden`)](#slot-only-plugins-tabhidden)
- [Backend API routes](#backend-api-routes)
- [Custom CSS per plugin](#custom-css-per-plugin)
- [Plugin discovery & reload](#plugin-discovery--reload)
- [Combined theme + plugin demo](#combined-theme--plugin-demo)
- [API reference](#api-reference)
- [Troubleshooting](#troubleshooting)
---
## Themes
Themes are YAML files stored in `~/.hermes/dashboard-themes/`. The file name doesn't matter (the theme's `name:` field is what the system uses), but convention is `<name>.yaml`. Every field is optional — missing keys fall back to the built-in `default` theme, so a theme can be as small as one color.
### Quick start — your first theme
```bash
mkdir -p ~/.hermes/dashboard-themes
```
```yaml
# ~/.hermes/dashboard-themes/neon.yaml
name: neon
label: Neon
description: Pure magenta on black
palette:
background: "#000000"
midground: "#ff00ff"
```
Refresh the dashboard. Click the palette icon in the header and pick **Neon**. The background goes black, text and accents go magenta, and every derived color (card, border, muted, ring, etc.) is recomputed from that 2-color triplet via `color-mix()` in CSS.
That's the whole onboarding: one file, two colors. Everything below is optional refinement.
### Palette, typography, layout
These three blocks are the heart of a theme. Each is independent — override one, leave the others.
#### Palette (3-layer)
The palette is a triplet of color layers plus a warm-glow vignette color and a noise-grain multiplier. The dashboard's design-system cascade derives every shadcn-compatible token (card, popover, muted, border, primary, destructive, ring, etc.) from this triplet via CSS `color-mix()`. Overriding three colors cascades into the whole UI.
| Key | Description |
|-----|-------------|
| `palette.background` | Deepest canvas color — typically near-black. Drives the page background and card fill. |
| `palette.midground` | Primary text and accent. Most UI chrome reads this (foreground text, button outlines, focus rings). |
| `palette.foreground` | Top-layer highlight. The default theme sets this to white at alpha 0 (invisible); themes that want a bright accent on top can raise its alpha. |
| `palette.warmGlow` | `rgba(...)` string used as the vignette color by `<Backdrop />`. |
| `palette.noiseOpacity` | 0–1.2 multiplier on the grain overlay. Lower = softer, higher = grittier. |
Each layer accepts either `{hex: "#RRGGBB", alpha: 0.0–1.0}` or a bare hex string (alpha defaults to 1.0).
```yaml
palette:
background:
hex: "#05091a"
alpha: 1.0
midground: "#d8f0ff" # bare hex, alpha = 1.0
foreground:
hex: "#ffffff"
alpha: 0 # invisible top layer
warmGlow: "rgba(255, 199, 55, 0.24)"
noiseOpacity: 0.7
```
#### Typography
| Key | Type | Description |
|-----|------|-------------|
| `fontSans` | string | CSS font-family stack for body copy (applied to `html`, `body`). |
| `fontMono` | string | CSS font-family stack for code blocks, `<code>`, `.font-mono` utilities. |
| `fontDisplay` | string | Optional heading/display stack. Falls back to `fontSans`. |
| `fontUrl` | string | Optional external stylesheet URL. Injected as `<link rel="stylesheet">` in `<head>` on theme switch. Same URL is never injected twice. Works with Google Fonts, Bunny Fonts, self-hosted `@font-face` sheets — anything linkable. |
| `baseSize` | string | Root font size — controls the rem scale. E.g. `"14px"`, `"16px"`. |
| `lineHeight` | string | Default line-height. E.g. `"1.5"`, `"1.65"`. |
| `letterSpacing` | string | Default letter-spacing. E.g. `"0"`, `"0.01em"`, `"-0.01em"`. |
```yaml
typography:
fontSans: '"Orbitron", "Eurostile", "Impact", sans-serif'
fontMono: '"Share Tech Mono", ui-monospace, monospace'
fontDisplay: '"Orbitron", "Eurostile", sans-serif'
fontUrl: "https://fonts.googleapis.com/css2?family=Orbitron:wght@400;500;600;700&family=Share+Tech+Mono&display=swap"
baseSize: "14px"
lineHeight: "1.5"
letterSpacing: "0.04em"
```
##### Changing the font from the UI (no YAML)
The theme picker in the dashboard header has a **Font** section below the
theme list. Pick any font there and it overrides the body font of whatever
theme is active — the choice is independent of the theme and persists across
theme switches (stored in `config.yaml` under `dashboard.font`). Choose
**Theme default** to clear the override and fall back to the active theme's
own `fontSans`.
The picker offers a curated catalog (system stacks plus a set of Google-Fonts
families across sans / serif / mono). It deliberately does **not** accept a
free-text font URL — the font's stylesheet is injected as a `<link>`, so the
catalog keeps the injected origins fixed. For a fully custom face, set
`fontSans` + `fontUrl` in a theme YAML as shown above. The theme's `fontMono`
(code blocks, terminal) is always left untouched by the UI override.
#### Layout
| Key | Values | Description |
|-----|--------|-------------|
| `radius` | any CSS length (`"0"`, `"0.25rem"`, `"0.5rem"`, `"1rem"`, ...) | Corner-radius token. Maps to `--radius` and cascades into `--radius-sm/md/lg/xl` — every rounded element shifts together. |
| `density` | `compact` \| `comfortable` \| `spacious` | Spacing multiplier applied as the `--spacing-mul` CSS var. `compact = 0.85×`, `comfortable = 1.0×` (default), `spacious = 1.2×`. Scales Tailwind's base spacing, so padding, gap, and space-between utilities all shift proportionally. |
```yaml
layout:
radius: "0"
density: compact
```
### Layout variants
`layoutVariant` picks the overall shell layout. Defaults to `"standard"` when absent.
| Variant | Behaviour |
|---------|-----------|
| `standard` | Single column, 1600px max-width (default). |
| `cockpit` | Left sidebar rail (260px) + main content. Populated by plugins via the `sidebar` slot — see [Shell slots](#shell-slots). Without a plugin the rail shows a placeholder. |
| `tiled` | Drops the max-width clamp so pages can use the full viewport width. |
```yaml
layoutVariant: cockpit
```
The current variant is exposed as `document.documentElement.dataset.layoutVariant`, so raw CSS in `customCSS` can target it via `:root[data-layout-variant="cockpit"] ...`.
### Theme assets (images as CSS vars)
Ship artwork URLs with a theme. Each named slot becomes a CSS var (`--theme-asset-<name>`) that the built-in shell and any plugin can read. The `bg` slot is automatically wired into the backdrop; other slots are plugin-facing.
```yaml
assets:
bg: "https://example.com/hero-bg.jpg" # auto-wired into <Backdrop />
hero: "/my-images/strike-freedom.png" # for plugin sidebars
crest: "/my-images/crest.svg" # for header-left plugins
logo: "/my-images/logo.png"
sidebar: "/my-images/rail.png"
header: "/my-images/header-art.png"
custom:
scanLines: "/my-images/scanlines.png" # → --theme-asset-custom-scanLines
```
Values accept:
- Bare URLs — wrapped in `url(...)` automatically.
- Pre-wrapped `url(...)`, `linear-gradient(...)`, `radial-gradient(...)` expressions — used as-is.
- `"none"` — explicit opt-out.
Every asset is also emitted as `--theme-asset-<name>-raw` (the unwrapped URL), in case a plugin needs to pass it to `<img src>` instead of `background-image`.
Plugins read these with plain CSS or JS:
```javascript
// In a plugin slot
const hero = getComputedStyle(document.documentElement)
.getPropertyValue("--theme-asset-hero").trim();
```
### Component chrome overrides
`componentStyles` restyles individual shell components without writing CSS selectors. Each bucket's entries become CSS vars (`--component-<bucket>-<kebab-property>`) that the shell's shared components read. So `card:` overrides apply to every `<Card>`, `header:` to the app bar, etc.
```yaml
componentStyles:
card:
clipPath: "polygon(12px 0, 100% 0, 100% calc(100% - 12px), calc(100% - 12px) 100%, 0 100%, 0 12px)"
background: "linear-gradient(180deg, rgba(10, 22, 52, 0.85), rgba(5, 9, 26, 0.92))"
boxShadow: "inset 0 0 0 1px rgba(64, 200, 255, 0.28)"
header:
background: "linear-gradient(180deg, rgba(16, 32, 72, 0.95), rgba(5, 9, 26, 0.9))"
tab:
clipPath: "polygon(6px 0, 100% 0, calc(100% - 6px) 100%, 0 100%)"
sidebar: {}
backdrop: {}
footer: {}
progress: {}
badge: {}
page: {}
```
Supported buckets: `card`, `header`, `footer`, `sidebar`, `tab`, `progress`, `badge`, `backdrop`, `page`.
Property names use camelCase (`clipPath`) and are emitted as kebab (`clip-path`). Values are plain CSS strings — anything CSS accepts (`clip-path`, `border-image`, `background`, `box-shadow`, `animation`, ...).
### Color overrides
Most themes won't need this — the 3-layer palette derives every shadcn token. Use `colorOverrides` when you want a specific accent the derivation won't produce (a softer destructive red for a pastel theme, a specific success green for a brand).
```yaml
colorOverrides:
primary: "#ffce3a"
primaryForeground: "#05091a"
accent: "#3fd3ff"
ring: "#3fd3ff"
destructive: "#ff3a5e"
border: "rgba(64, 200, 255, 0.28)"
```
Supported keys: `card`, `cardForeground`, `popover`, `popoverForeground`, `primary`, `primaryForeground`, `secondary`, `secondaryForeground`, `muted`, `mutedForeground`, `accent`, `accentForeground`, `destructive`, `destructiveForeground`, `success`, `warning`, `border`, `input`, `ring`.
Each key maps 1:1 to the `--color-<kebab>` CSS var (e.g. `primaryForeground` → `--color-primary-foreground`). Any key set here wins over the palette cascade for the active theme only — switching to another theme clears the overrides.
### Raw `customCSS`
For selector-level chrome that `componentStyles` can't express — pseudo-elements, animations, media queries, theme-scoped overrides — drop raw CSS into `customCSS`:
```yaml
customCSS: |
/* Scanline overlay — only visible when cockpit variant is active. */
:root[data-layout-variant="cockpit"] body::before {
content: "";
position: fixed;
inset: 0;
pointer-events: none;
z-index: 100;
background: repeating-linear-gradient(to bottom,
transparent 0px, transparent 2px,
rgba(64, 200, 255, 0.035) 3px, rgba(64, 200, 255, 0.035) 4px);
mix-blend-mode: screen;
}
```
The CSS is injected as a single scoped `<style data-hermes-theme-css>` tag on theme apply and cleaned up on theme switch. **Capped at 32 KiB per theme.**
### Built-in themes
Each built-in ships its own palette, typography, and layout — switching produces visible changes beyond color alone.
| Theme | Palette | Typography | Layout |
|-------|---------|------------|--------|
| **Hermes Teal** (`default`) | Dark teal + cream | System stack, 15px | 0.5rem radius, comfortable |
| **Hermes Teal (Large)** (`default-large`) | Same as default | System stack, 18px, line-height 1.65 | 0.5rem radius, spacious |
| **Midnight** (`midnight`) | Deep blue-violet | Inter + JetBrains Mono, 14px | 0.75rem radius, comfortable |
| **Ember** (`ember`) | Warm crimson + bronze | Spectral (serif) + IBM Plex Mono, 15px | 0.25rem radius, comfortable |
| **Mono** (`mono`) | Grayscale | IBM Plex Sans + IBM Plex Mono, 13px | 0 radius, compact |
| **Cyberpunk** (`cyberpunk`) | Neon green on black | Share Tech Mono everywhere, 14px | 0 radius, compact |
| **Rosé** (`rose`) | Pink + ivory | Fraunces (serif) + DM Mono, 16px | 1rem radius, spacious |
Themes that reference Google Fonts (all except Hermes Teal) load the stylesheet on demand — the first time you switch to them a `<link>` tag is injected into `<head>`.
### Full theme YAML reference
Every knob in one file — copy and trim what you don't need:
```yaml
# ~/.hermes/dashboard-themes/ocean.yaml
name: ocean
label: Ocean Deep
description: Deep sea blues with coral accents
# 3-layer palette (accepts {hex, alpha} or bare hex)
palette:
background:
hex: "#0a1628"
alpha: 1.0
midground:
hex: "#a8d0ff"
alpha: 1.0
foreground:
hex: "#ffffff"
alpha: 0.0
warmGlow: "rgba(255, 107, 107, 0.35)"
noiseOpacity: 0.7
typography:
fontSans: "Poppins, system-ui, sans-serif"
fontMono: "Fira Code, ui-monospace, monospace"
fontDisplay: "Poppins, system-ui, sans-serif" # optional
fontUrl: "https://fonts.googleapis.com/css2?family=Poppins:wght@400;500;600&family=Fira+Code:wght@400;500&display=swap"
baseSize: "15px"
lineHeight: "1.6"
letterSpacing: "-0.003em"
layout:
radius: "0.75rem"
density: comfortable
layoutVariant: standard # standard | cockpit | tiled
assets:
bg: "https://example.com/ocean-bg.jpg"
hero: "/my-images/kraken.png"
crest: "/my-images/anchor.svg"
logo: "/my-images/logo.png"
custom:
pattern: "/my-images/waves.svg"
componentStyles:
card:
boxShadow: "inset 0 0 0 1px rgba(168, 208, 255, 0.18)"
header:
background: "linear-gradient(180deg, rgba(10, 22, 40, 0.95), rgba(5, 9, 26, 0.9))"
colorOverrides:
destructive: "#ff6b6b"
ring: "#ff6b6b"
customCSS: |
/* Any additional selector-level tweaks */
```
Refresh the dashboard after creating the file. Switch themes live from the header bar — click the palette icon. Selection persists to `config.yaml` under `dashboard.theme` and is restored on reload.
---
## Plugins
A dashboard plugin is a directory with a `manifest.json`, a pre-built JS bundle, and optionally a CSS file and a Python file with FastAPI routes. Plugins live next to other Hermes plugins in `~/.hermes/plugins/<name>/` — the dashboard extension is a `dashboard/` subfolder inside that plugin directory, so one plugin can extend both the CLI/gateway and the dashboard from a single install.
Plugins don't bundle React or UI components. They use the **Plugin SDK** exposed on `window.__HERMES_PLUGIN_SDK__`. This keeps plugin bundles tiny (typically a few KB) and avoids version conflicts.
### Quick start — your first plugin
Create the directory structure:
```bash
mkdir -p ~/.hermes/plugins/my-plugin/dashboard/dist
```
Write the manifest:
```json
// ~/.hermes/plugins/my-plugin/dashboard/manifest.json
{
"name": "my-plugin",
"label": "My Plugin",
"icon": "Sparkles",
"version": "1.0.0",
"tab": {
"path": "/my-plugin",
"position": "after:skills"
},
"entry": "dist/index.js"
}
```
Write the JS bundle (a plain IIFE — no build step needed):
```javascript
// ~/.hermes/plugins/my-plugin/dashboard/dist/index.js
(function () {
"use strict";
const SDK = window.__HERMES_PLUGIN_SDK__;
const { React } = SDK;
const { Card, CardHeader, CardTitle, CardContent } = SDK.components;
function MyPage() {
return React.createElement(Card, null,
React.createElement(CardHeader, null,
React.createElement(CardTitle, null, "My Plugin"),
),
React.createElement(CardContent, null,
React.createElement("p", { className: "text-sm text-muted-foreground" },
"Hello from my custom dashboard tab.",
),
),
);
}
window.__HERMES_PLUGINS__.register("my-plugin", MyPage);
})();
```
Refresh the dashboard — your tab appears in the nav bar, after **Skills**.
:::tip Skip React.createElement
If you prefer JSX, use any bundler (esbuild, Vite, rollup) with React as an external and IIFE output. The only hard requirement is that the final file is a single JS file loadable via `<script>`. React is never bundled; it comes from `SDK.React`.
:::
### Directory layout
```
~/.hermes/plugins/my-plugin/
├── plugin.yaml # optional — existing CLI/gateway plugin manifest
├── __init__.py # optional — existing CLI/gateway hooks
└── dashboard/ # dashboard extension
├── manifest.json # required — tab config, icon, entry point
├── dist/
│ ├── index.js # required — pre-built JS bundle (IIFE)
│ └── style.css # optional — custom CSS
└── plugin_api.py # optional — backend API routes (FastAPI)
```
A single plugin directory can carry three orthogonal extensions:
- `plugin.yaml` + `__init__.py` — CLI/gateway plugin ([see plugins page](./plugins)).
- `dashboard/manifest.json` + `dashboard/dist/index.js` — dashboard UI plugin.
- `dashboard/plugin_api.py` — dashboard backend routes.
None of them are required; include only the layers you need.
### Manifest reference
```json
{
"name": "my-plugin",
"label": "My Plugin",
"description": "What this plugin does",
"icon": "Sparkles",
"version": "1.0.0",
"tab": {
"path": "/my-plugin",
"position": "after:skills",
"override": "/",
"hidden": false
},
"slots": ["sidebar", "header-left"],
"entry": "dist/index.js",
"css": "dist/style.css",
"api": "plugin_api.py"
}
```
| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Unique plugin identifier. Lowercase, hyphens ok. Used in URLs and registration. |
| `label` | Yes | Display name shown in the nav tab. |
| `description` | No | Short description (shown in dashboard admin surfaces). |
| `icon` | No | Lucide icon name. Defaults to `Puzzle`. Unknown names fall back to `Puzzle`. |
| `version` | No | Semver string. Defaults to `0.0.0`. |
| `tab.path` | Yes | URL path for the tab (e.g. `/my-plugin`). |
| `tab.position` | No | Where to insert the tab. `"end"` (default), `"after:<path>"`, or `"before:<path>"` — value after the colon is the **path segment** of the target tab (no leading slash). Examples: `"after:skills"`, `"before:config"`. |
| `tab.override` | No | Set to a built-in route path (`"/"`, `"/sessions"`, `"/config"`, ...) to **replace** that page instead of adding a new tab. See [Replacing built-in pages](#replacing-built-in-pages-taboverride). |
| `tab.hidden` | No | When true, register the component and any slots without adding a tab to the nav. Used by slot-only plugins. See [Slot-only plugins](#slot-only-plugins-tabhidden). |
| `slots` | No | Named shell slots this plugin populates. **Documentation aid only** — actual registration happens from the JS bundle via `registerSlot()`. Listing slots here makes discovery surfaces more informative. |
| `entry` | Yes | Path to the JS bundle relative to `dashboard/`. Defaults to `dist/index.js`. |
| `css` | No | Path to a CSS file to inject as a `<link>` tag. |
| `api` | No | Path to a Python file with FastAPI routes. Mounted at `/api/plugins/<name>/`. |
#### Available icons
Plugins use Lucide icon names. The dashboard maps these by name — unknown names silently fall back to `Puzzle`.
Currently mapped: `Activity`, `BarChart3`, `Clock`, `Code`, `Database`, `Eye`, `FileText`, `Globe`, `Heart`, `KeyRound`, `MessageSquare`, `Package`, `Puzzle`, `Settings`, `Shield`, `Sparkles`, `Star`, `Terminal`, `Wrench`, `Zap`.
Need a different icon? Open a PR to `web/src/App.tsx`'s `ICON_MAP` — pure additive change.
### The Plugin SDK
Everything a plugin needs is on `window.__HERMES_PLUGIN_SDK__`. Plugins should never import React directly.
```javascript
const SDK = window.__HERMES_PLUGIN_SDK__;
// React + hooks
SDK.React // the React instance
SDK.hooks.useState
SDK.hooks.useEffect
SDK.hooks.useCallback
SDK.hooks.useMemo
SDK.hooks.useRef
SDK.hooks.useContext
SDK.hooks.createContext
// UI components (shadcn/ui primitives)
SDK.components.Card
SDK.components.CardHeader
SDK.components.CardTitle
SDK.components.CardContent
SDK.components.Badge
SDK.components.Button
SDK.components.Input
SDK.components.Label
SDK.components.Select
SDK.components.SelectOption
SDK.components.Separator
SDK.components.Tabs
SDK.components.TabsList
SDK.components.TabsTrigger
SDK.components.PluginSlot // render a named slot (useful for nested plugin UIs)
// Hermes API client + raw fetcher
SDK.api // typed client — getStatus, getSessions, getConfig, ...
SDK.fetchJSON // raw fetch for custom endpoints (plugin-registered routes)
// Utilities
SDK.utils.cn // Tailwind class merger (clsx + twMerge)
SDK.utils.timeAgo // "5m ago" from unix timestamp
SDK.utils.isoTimeAgo // "5m ago" from ISO string
// Hooks
SDK.useI18n // i18n hook for multi-language plugins
```
#### Calling your plugin's backend
```javascript
SDK.fetchJSON("/api/plugins/my-plugin/data")
.then((data) => console.log(data))
.catch((err) => console.error("API call failed:", err));
```
`fetchJSON` injects the session auth token, surfaces errors as thrown exceptions, and parses JSON automatically.
#### Calling built-in Hermes endpoints
```javascript
// Agent status
SDK.api.getStatus().then((s) => console.log("Version:", s.version));
// Recent sessions
SDK.api.getSessions(10).then((resp) => console.log(resp.sessions.length));
```
See [Web Dashboard → REST API](./web-dashboard#rest-api) for the full list.
### Shell slots
Slots let a plugin inject components into named locations of the app shell — the cockpit sidebar, the header, the footer, an overlay layer — without claiming a whole tab. Multiple plugins can populate the same slot; they render stacked in registration order.
Register from inside the plugin bundle:
```javascript
window.__HERMES_PLUGINS__.registerSlot("my-plugin", "sidebar", MySidebar);
window.__HERMES_PLUGINS__.registerSlot("my-plugin", "header-left", MyCrest);
```
#### Slot catalogue
**Shell-wide slots** (render anywhere in the app chrome):
| Slot | Location |
|------|----------|
| `backdrop` | Inside the `<Backdrop />` layer stack, above the noise layer. |
| `header-left` | Before the Hermes brand in the top bar. |
| `header-right` | Before the theme/language switchers in the top bar. |
| `header-banner` | Full-width strip below the nav. |
| `sidebar` | Cockpit sidebar rail — **only rendered when `layoutVariant === "cockpit"`**. |
| `pre-main` | Above the route outlet (inside `<main>`). |
| `post-main` | Below the route outlet (inside `<main>`). |
| `footer-left` | Footer cell content (replaces default). |
| `footer-right` | Footer cell content (replaces default). |
| `overlay` | Fixed-position layer above everything else. Useful for chrome (scanlines, vignettes) `customCSS` can't achieve alone. |
**Page-scoped slots** (render only on the named built-in page — use these to inject widgets, cards, or toolbars into an existing page without overriding the whole route):
| Slot | Where it renders |
|------|------------------|
| `sessions:top` / `sessions:bottom` | Top / bottom of the `/sessions` page. |
| `analytics:top` / `analytics:bottom` | Top / bottom of the `/analytics` page. |
| `logs:top` / `logs:bottom` | Top (above filter toolbar) / bottom (below log viewer) of `/logs`. |
| `cron:top` / `cron:bottom` | Top / bottom of the `/cron` page. |
| `skills:top` / `skills:bottom` | Top / bottom of the `/skills` page. |
| `config:top` / `config:bottom` | Top / bottom of the `/config` page. |
| `env:top` / `env:bottom` | Top / bottom of the `/env` (Keys) page. |
| `docs:top` / `docs:bottom` | Top (above the iframe) / bottom of `/docs`. |
| `chat:top` / `chat:bottom` | Top / bottom of `/chat` (only active when embedded chat is enabled). |
Example — add a banner card to the top of the Sessions page:
```javascript
function PinnedSessionsBanner() {
return React.createElement(Card, null,
React.createElement(CardContent, { className: "py-2 text-xs" },
"Pinned note injected by my-plugin"),
);
}
window.__HERMES_PLUGINS__.registerSlot("my-plugin", "sessions:top", PinnedSessionsBanner);
```
Combine page-scoped slots with `tab.hidden: true` if your plugin only augments existing pages and doesn't need a sidebar tab of its own.
The shell only renders `<PluginSlot name="..." />` for the slots above. Additional names are accepted by the registry for nested plugin UIs — a plugin can expose its own slots via `SDK.components.PluginSlot`.
#### Re-registration and HMR
If the same `(plugin, slot)` pair is registered twice, the later call replaces the earlier one — this matches how React HMR expects plugin re-mounts to behave.
### Replacing built-in pages (`tab.override`)
Setting `tab.override` to a built-in route path makes the plugin's component replace that page instead of adding a new tab. Useful when a theme wants a custom home page (`/`) but wants to keep the rest of the dashboard intact.
```json
{
"name": "my-home",
"label": "Home",
"tab": {
"path": "/my-home",
"override": "/",
"position": "end"
},
"entry": "dist/index.js"
}
```
With `override` set:
- The original page component at `/` is removed from the router.
- Your plugin renders at `/` instead.
- No nav tab is added for `tab.path` (the override is the point).
Only one plugin can override a given path. If two plugins claim the same override, the first wins and the second is ignored with a dev-mode warning.
If you only need to add a card or toolbar to an existing page without taking it over, use [page-scoped slots](#augmenting-built-in-pages-page-scoped-slots) instead.
### Augmenting built-in pages (page-scoped slots)
Full replacement via `tab.override` is heavy — your plugin now owns the entire page, including any future updates we ship to it. Most of the time you just want to add a banner, card, or toolbar to an existing page. That's what **page-scoped slots** are for.
Every built-in page exposes `<page>:top` and `<page>:bottom` slots rendered at the top and bottom of its content area. Your plugin populates one by calling `registerSlot()` — the built-in page keeps working normally, and your component renders alongside it.
Available slots: `sessions:*`, `analytics:*`, `logs:*`, `cron:*`, `skills:*`, `config:*`, `env:*`, `docs:*`, `chat:*` (each with `:top` and `:bottom`). See the full catalogue in [Shell slots → Slot catalogue](#slot-catalogue).
Minimal example — pin a banner to the top of the Sessions page:
```json
// ~/.hermes/plugins/session-notes/dashboard/manifest.json
{
"name": "session-notes",
"label": "Session Notes",
"tab": { "path": "/session-notes", "hidden": true },
"slots": ["sessions:top"],
"entry": "dist/index.js"
}
```
```javascript
// ~/.hermes/plugins/session-notes/dashboard/dist/index.js
(function () {
const SDK = window.__HERMES_PLUGIN_SDK__;
const { React } = SDK;
const { Card, CardContent } = SDK.components;
function Banner() {
return React.createElement(Card, null,
React.createElement(CardContent, { className: "py-2 text-xs" },
"Remember to label important sessions before archiving."),
);
}
// Placeholder for the hidden tab.
window.__HERMES_PLUGINS__.register("session-notes", function () { return null; });
// The real work.
window.__HERMES_PLUGINS__.registerSlot("session-notes", "sessions:top", Banner);
})();
```
Key points:
- `tab.hidden: true` keeps the plugin out of the sidebar — it has no standalone page.
- The `slots` manifest field is documentation only. The actual binding happens in the JS bundle via `registerSlot()`.
- Multiple plugins can claim the same page-scoped slot. They render stacked in registration order.
- Zero footprint when no plugin registers: the built-in page renders exactly as before.
A reference plugin (`example-dashboard` in [`hermes-example-plugins`](https://github.com/NousResearch/hermes-example-plugins/tree/main/example-dashboard)) ships a live demo that injects a banner into `sessions:top` — install it to see the pattern end-to-end.
### Slot-only plugins (`tab.hidden`)
When `tab.hidden: true`, the plugin registers its component (for direct URL visits) and any slots, but never adds a tab to the navigation. Used by plugins that only exist to inject into slots — a header crest, a sidebar HUD, an overlay.
```json
{
"name": "header-crest",
"label": "Header Crest",
"tab": {
"path": "/header-crest",
"position": "end",
"hidden": true
},
"slots": ["header-left"],
"entry": "dist/index.js"
}
```
The bundle still calls `register()` with a placeholder component (good practice in case someone hits the URL directly) and then `registerSlot()` to do the real work.
### Backend API routes
Plugins can register FastAPI routes by setting `api` in the manifest. Create the file and export a `router`:
```python
# ~/.hermes/plugins/my-plugin/dashboard/plugin_api.py
from fastapi import APIRouter
router = APIRouter()
@router.get("/data")
async def get_data():
return {"items": ["one", "two", "three"]}
@router.post("/action")
async def do_action(body: dict):
return {"ok": True, "received": body}
```
Routes are mounted under `/api/plugins/<name>/`, so the above becomes:
- `GET /api/plugins/my-plugin/data`
- `POST /api/plugins/my-plugin/action`
Plugin API routes sit behind the dashboard's normal auth gate — unauthenticated requests get a `401` before the plugin route runs, and requests to a disabled plugin's routes are rejected at request time. Still, **don't expose the dashboard on a public interface with `--host 0.0.0.0` if you run untrusted plugins** — an authenticated session can reach their routes too.
#### Accessing Hermes internals
Backend routes run inside the dashboard process, so they can import from the hermes-agent codebase directly:
```python
from fastapi import APIRouter
from hermes_state import SessionDB
from hermes_cli.config import load_config
router = APIRouter()
@router.get("/session-count")
async def session_count():
db = SessionDB()
try:
count = len(db.list_sessions(limit=9999))
return {"count": count}
finally:
db.close()
@router.get("/config-snapshot")
async def config_snapshot():
cfg = load_config()
return {"model": cfg.get("model", {})}
```
### Custom CSS per plugin
If your plugin needs styles beyond Tailwind classes and inline `style=`, add a CSS file and reference it in the manifest:
```json
{
"css": "dist/style.css"
}
```
The file is injected as a `<link>` tag on plugin load. Use specific class names to avoid conflicts with the dashboard's styles, and reference the dashboard's CSS vars to stay theme-aware:
```css
/* dist/style.css */
.my-plugin-chart {
border: 1px solid var(--color-border);
background: var(--color-card);
color: var(--color-card-foreground);
padding: 1rem;
}
.my-plugin-chart:hover {
border-color: var(--color-ring);
}
```
The dashboard exposes every shadcn token as `--color-*` plus theme extras (`--theme-asset-*`, `--component-<bucket>-*`, `--radius`, `--spacing-mul`). Reference those and your plugin automatically reskins with the active theme.
### Plugin discovery & reload
The dashboard scans three directories for `dashboard/manifest.json`:
| Priority | Directory | Source label |
|----------|-----------|--------------|
| 1 (wins on conflict) | `~/.hermes/plugins/<name>/dashboard/` | `user` |
| 2 | `<repo>/plugins/memory/<name>/dashboard/` | `bundled` |
| 2 | `<repo>/plugins/<name>/dashboard/` | `bundled` |
| 3 | `./.hermes/plugins/<name>/dashboard/` | `project` — only when `HERMES_ENABLE_PROJECT_PLUGINS` is set |
Discovery results are cached per dashboard process. After adding a new plugin, either:
```bash
# Force a rescan without restart
curl http://127.0.0.1:9119/api/dashboard/plugins/rescan
```
…or restart `hermes dashboard`.
#### Plugin load lifecycle
1. Dashboard loads. `main.tsx` exposes the SDK on `window.__HERMES_PLUGIN_SDK__` and the registry on `window.__HERMES_PLUGINS__`.
2. `App.tsx` calls `usePlugins()` → fetches `GET /api/dashboard/plugins`.
3. For each manifest: CSS `<link>` is injected (if declared), then a `<script>` tag loads the JS bundle.
4. The plugin's IIFE runs and calls `window.__HERMES_PLUGINS__.register(name, Component)` — and optionally `.registerSlot(name, slot, Component)` for each slot.
5. The dashboard resolves the registered component against the manifest, adds the tab to navigation (unless `hidden`), and mounts the component as a route.
Plugins have up to **2 seconds** after their script loads to call `register()`. After that the dashboard stops waiting and finishes initial render. If a plugin later registers, it still appears — the nav is reactive.
If a plugin's script fails to load (404, syntax error, exception during IIFE), the dashboard logs a warning to the browser console and continues without it.
---
## Combined theme + plugin demo
The [`strike-freedom-cockpit`](https://github.com/NousResearch/hermes-example-plugins/tree/main/strike-freedom-cockpit) plugin (companion repo `hermes-example-plugins`) is a complete reskin demo. It pairs a theme YAML with a slot-only plugin to produce a cockpit-style HUD without forking the dashboard.
**What it demonstrates:**
- A full theme using palette, typography, `fontUrl`, `layoutVariant: cockpit`, `assets`, `componentStyles` (notched card corners, gradient backgrounds), `colorOverrides`, and `customCSS` (scanline overlay).
- A slot-only plugin (`tab.hidden: true`) that registers into three slots:
- `sidebar` — an MS-STATUS panel with live telemetry bars driven by `SDK.api.getStatus()`.
- `header-left` — a faction crest that reads `--theme-asset-crest` from the active theme.
- `footer-right` — a custom tagline replacing the default org line.
- The plugin reads theme-supplied artwork via CSS vars, so swapping themes changes the hero/crest without plugin code changes.
**Install:**
```bash
git clone https://github.com/NousResearch/hermes-example-plugins.git
# Theme
cp hermes-example-plugins/strike-freedom-cockpit/theme/strike-freedom.yaml \
~/.hermes/dashboard-themes/
# Plugin
cp -r hermes-example-plugins/strike-freedom-cockpit ~/.hermes/plugins/
```
Open the dashboard, pick **Strike Freedom** from the theme switcher. The cockpit sidebar appears, the crest shows in the header, the tagline replaces the footer. Switch back to **Hermes Teal** and the plugin remains installed but invisible (the `sidebar` slot only renders under the `cockpit` layout variant).
Read the plugin source (`strike-freedom-cockpit/dashboard/dist/index.js` in the companion repo) to see how it reads CSS vars, guards against older dashboards without slot support, and registers three slots from one bundle.
---
## API reference
### Theme endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/dashboard/themes` | GET | List available themes + active name. Built-ins return `{name, label, description}`; user themes also include a `definition` field with the full normalised theme object. |
| `/api/dashboard/theme` | PUT | Set active theme. Body: `{"name": "midnight"}`. Persists to `config.yaml` under `dashboard.theme`. |
### Plugin endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/dashboard/plugins` | GET | List discovered plugins (with manifests, minus internal fields). |
| `/api/dashboard/plugins/rescan` | GET | Force re-scan the plugin directories without restarting. |
| `/dashboard-plugins/<name>/<path>` | GET | Serve static assets from a plugin's `dashboard/` directory. Path traversal is blocked. |
| `/api/plugins/<name>/*` | * | Plugin-registered backend routes. |
### SDK on `window`
| Global | Type | Provider |
|--------|------|----------|
| `window.__HERMES_PLUGIN_SDK__` | object | `registry.ts` — React, hooks, UI components, API client, utils. |
| `window.__HERMES_PLUGINS__.register(name, Component)` | function | Register a plugin's main component. |
| `window.__HERMES_PLUGINS__.registerSlot(name, slot, Component)` | function | Register into a named shell slot. |
---
## Troubleshooting
**My theme doesn't appear in the picker.**
Check that the file is in `~/.hermes/dashboard-themes/` and ends in `.yaml` or `.yml`. Refresh the page. Run `curl http://127.0.0.1:9119/api/dashboard/themes` — your theme should be in the response. If the YAML has a parse error, the dashboard logs to `errors.log` under `~/.hermes/logs/`.
**My plugin's tab doesn't show up.**
1. Check the manifest is at `~/.hermes/plugins/<name>/dashboard/manifest.json` (note the `dashboard/` subdirectory).
2. `curl http://127.0.0.1:9119/api/dashboard/plugins/rescan` to force re-discovery.
3. Open browser dev tools → Network — confirm `manifest.json`, `index.js`, and any CSS loaded without 404s.
4. Open browser dev tools → Console — look for errors during the IIFE or `window.__HERMES_PLUGINS__ is undefined` (indicates the SDK didn't initialize, usually a React render crash earlier).
5. Verify your bundle calls `window.__HERMES_PLUGINS__.register(...)` with the **same name** as `manifest.json:name`.
**Slot-registered components don't render.**
The `sidebar` slot only renders when the active theme has `layoutVariant: cockpit`. Other slots always render. If you're registering into a slot with no hits, add `console.log` inside `registerSlot` to confirm the plugin bundle ran at all.
**Plugin backend routes return 404.**
1. Confirm the manifest has `"api": "plugin_api.py"` pointing to an existing file inside `dashboard/`.
2. Restart `hermes dashboard` — plugin API routes are mounted once at startup, **not** on rescan.
3. Check that `plugin_api.py` exports a module-level `router = APIRouter()`. Other export names are not picked up.
4. Tail `~/.hermes/logs/errors.log` for `Failed to load plugin <name> API routes` — import errors are logged there.
**Theme change drops my color overrides.**
`colorOverrides` are scoped to the active theme and cleared on theme switch — that's by design. If you want overrides that persist, put them in your theme's YAML, not in the live switcher.
**Theme customCSS gets truncated.**
The `customCSS` block is capped at 32 KiB per theme. Split large stylesheets across multiple themes, or switch to a plugin that injects a full stylesheet via its `css` field (no size cap).
**I want to ship a plugin on PyPI.**
Dashboard plugins are installed by directory layout, not by pip entry point. The cleanest distribution path today is a git repo the user clones into `~/.hermes/plugins/`. A pip-based installer for dashboard plugins is not currently wired up.
website/docs/user-guide/features/fallback-providers.md
+435
View File
@@ -0,0 +1,435 @@
---
title: Fallback Providers
description: Configure automatic failover to backup LLM providers when your primary model is unavailable.
sidebar_label: Fallback Providers
sidebar_position: 8
---
# Fallback Providers
Hermes Agent has three layers of resilience that keep your sessions running when providers hit issues:
1. **[Credential pools](./credential-pools.md)** — rotate across multiple API keys for the *same* provider (tried first)
2. **Primary model fallback** — automatically switches to a *different* provider:model when your main model fails
3. **Auxiliary task fallback** — independent provider resolution for side tasks like vision and compression
Credential pools handle same-provider rotation (e.g., multiple OpenRouter keys). This page covers cross-provider fallback. Both are optional and work independently.
## Primary Model Fallback
When your main LLM provider encounters errors — rate limits, server overload, auth failures, connection drops — Hermes can automatically switch to a backup provider:model pair mid-session without losing your conversation.
### Configuration
The easiest path is the interactive manager:
```bash
hermes fallback
```
`hermes fallback` reuses the provider picker from `hermes model` — same provider list, same credential prompts, same validation. Use the subcommands `add`, `list` (alias `ls`), `remove` (alias `rm`), and `clear` to manage the chain. Changes persist under the top-level `fallback_providers:` list in `config.yaml`.
If you'd rather edit the YAML directly, add a top-level `fallback_providers` list to `~/.hermes/config.yaml`:
```yaml
fallback_providers:
- provider: openrouter
model: anthropic/claude-sonnet-4
```
Each entry requires both `provider` and `model`. Entries missing either field are ignored.
:::note `fallback_model` vs `fallback_providers`
`fallback_providers` (plural, list) is the current config shape and supports multiple fallbacks tried in order. `fallback_model` (singular) is the legacy single-fallback key — Hermes still honors it for back-compat, but `hermes fallback` writes the current `fallback_providers` key and migrates legacy config on write. When both are set, `fallback_providers` takes priority.
:::
### Supported Providers
| Provider | Value | Requirements |
|----------|-------|-------------|
| AI Gateway | `ai-gateway` | `AI_GATEWAY_API_KEY` |
| OpenRouter | `openrouter` | `OPENROUTER_API_KEY` |
| Nous Portal | `nous` | `hermes setup --portal` (fresh) or `hermes auth add nous` (OAuth) |
| OpenAI Codex | `openai-codex` | `hermes model` → **ChatGPT or Codex Subscription** (ChatGPT OAuth) |
| GitHub Copilot | `copilot` | `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` |
| GitHub Copilot ACP | `copilot-acp` | External process (editor integration) |
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` or Claude Code credentials |
| z.ai / GLM | `zai` | `GLM_API_KEY` |
| Kimi / Moonshot | `kimi-coding` | `KIMI_API_KEY` |
| MiniMax | `minimax` | `MINIMAX_API_KEY` |
| MiniMax (China) | `minimax-cn` | `MINIMAX_CN_API_KEY` |
| DeepSeek | `deepseek` | `DEEPSEEK_API_KEY` |
| NVIDIA NIM | `nvidia` | `NVIDIA_API_KEY` (optional: `NVIDIA_BASE_URL`) |
| GMI Cloud | `gmi` | `GMI_API_KEY` (optional: `GMI_BASE_URL`) |
| Upstage Solar | `upstage` (alias `solar`) | `UPSTAGE_API_KEY` (optional: `UPSTAGE_BASE_URL`) |
| StepFun | `stepfun` | `STEPFUN_API_KEY` (optional: `STEPFUN_BASE_URL`) |
| Ollama Cloud | `ollama-cloud` | `OLLAMA_API_KEY` |
| Google AI Studio | `gemini` | `GOOGLE_API_KEY` (alias: `GEMINI_API_KEY`) |
| xAI (Grok) | `xai` (alias `grok`) | `XAI_API_KEY` (optional: `XAI_BASE_URL`) |
| xAI Grok OAuth (SuperGrok) | `xai-oauth` (alias `grok-oauth`) | `hermes model` → xAI Grok OAuth (browser login; SuperGrok subscription) |
| AWS Bedrock | `bedrock` | Standard boto3 auth (`AWS_REGION` + `AWS_PROFILE` or `AWS_ACCESS_KEY_ID`) |
| Qwen Portal (OAuth) | `qwen-oauth` | `hermes model` (Qwen Portal OAuth; optional: `HERMES_QWEN_BASE_URL`) |
| MiniMax (OAuth) | `minimax-oauth` | `hermes model` (MiniMax portal OAuth) |
| OpenCode Zen | `opencode-zen` | `OPENCODE_ZEN_API_KEY` |
| CommandCode | `commandcode` (alias `commandcode-chat`; Claude via `commandcode-anthropic`) | `COMMANDCODE_API_KEY` |
| OpenCode Go | `opencode-go` | `OPENCODE_GO_API_KEY` |
| OpenCode Free | `opencode-free` | — (keyless, no credential) |
| Kilo Code | `kilocode` | `KILOCODE_API_KEY` |
| Ramp Router | `router` | `RAMP_ROUTER_API_KEY` |
| Xiaomi MiMo | `xiaomi` | `XIAOMI_API_KEY` |
| Arcee AI | `arcee` | `ARCEEAI_API_KEY` |
| GMI Cloud | `gmi` | `GMI_API_KEY` |
| Nebius Token Factory | `nebius-token-factory` | `NEBIUS_API_KEY` |
| Alibaba / DashScope | `alibaba` | `DASHSCOPE_API_KEY` |
| Alibaba Coding Plan | `alibaba-coding-plan` | `ALIBABA_CODING_PLAN_API_KEY` (falls back to `DASHSCOPE_API_KEY`) |
| Kimi / Moonshot (China) | `kimi-coding-cn` | `KIMI_CN_API_KEY` |
| StepFun | `stepfun` | `STEPFUN_API_KEY` |
| Tencent TokenHub | `tencent-tokenhub` | `TOKENHUB_API_KEY` |
| Tencent TokenPlan | `tencent-tokenplan` | `TOKENPLAN_API_KEY` |
| Microsoft Foundry | `azure-foundry` | `AZURE_FOUNDRY_API_KEY` + `AZURE_FOUNDRY_BASE_URL` |
| LM Studio (local) | `lmstudio` | `LM_API_KEY` (or none for local) + `LM_BASE_URL` |
| Hugging Face | `huggingface` | `HF_TOKEN` |
| Custom endpoint | `custom` | `base_url` + `key_env` (see below) |
### Custom Endpoint Fallback
For a custom OpenAI-compatible endpoint, add `base_url` and optionally `key_env`:
```yaml
fallback_providers:
- provider: custom
model: my-local-model
base_url: http://localhost:8000/v1
key_env: MY_LOCAL_KEY # env var name containing the API key
```
### When Fallback Triggers
The fallback activates automatically when the primary model fails with:
- **Rate limits** (HTTP 429) — after exhausting retry attempts
- **Server errors** (HTTP 500, 502, 503) — after exhausting retry attempts
- **Auth failures** (HTTP 401, 403) — immediately (no point retrying)
- **Not found** (HTTP 404) — immediately
- **Invalid responses** — when the API returns malformed or empty responses repeatedly
When triggered, Hermes:
1. Resolves credentials for the fallback provider
2. Builds a new API client
3. Swaps the model, provider, and client in-place
4. Resets the retry counter and continues the conversation
The switch is seamless — your conversation history, tool calls, and context are preserved. The agent continues from exactly where it left off, just using a different model.
:::warning Fallback resets the prompt cache
Prompt caches are keyed to the model (and on most providers, the account) serving the request. When fallback fires, the new provider:model has no cached prefix for your conversation, so the next request re-reads the entire history at full input-token price instead of the ~75–90% discounted cached rate. The same applies when the turn ends and the primary is restored — that first request back on the primary is a full re-read too (unless the primary's cache TTL hasn't expired). This is unavoidable — it's the cost of staying alive through an outage — but it's why a long session that bounces between providers can cost noticeably more than one that stays put.
:::
:::info Per-Turn, Not Per-Session
Fallback is **turn-scoped**: each new user message starts with the primary model restored. If the primary fails mid-turn, fallback activates for that turn only. On the next message, Hermes tries the primary again. Within a single turn, fallback activates at most once — if the fallback also fails, normal error handling takes over (retries, then error message). This prevents cascading failover loops within a turn while giving the primary model a fresh chance every turn.
The per-turn retry is **reset-aware**: when the primary's credentials report a rate-limit reset time that hasn't elapsed yet (subscription windows like Claude Pro/Max's 5-hour blocks or Codex weekly limits report these as hours or days), Hermes skips the doomed retry and stays on the fallback until the reset passes — avoiding two pointless provider switches (and two prompt-cache invalidations) per turn. The moment the reset time elapses, the next turn goes back to the primary automatically. Transient 429s without a reset time keep the existing behavior: a short cooldown, then retry every turn.
:::
### Examples
**OpenRouter as fallback for Anthropic native:**
```yaml
model:
provider: anthropic
default: claude-sonnet-4-6
fallback_providers:
- provider: openrouter
model: anthropic/claude-sonnet-4
```
**Nous Portal as fallback for OpenRouter:**
```yaml
model:
provider: openrouter
default: anthropic/claude-opus-4
fallback_providers:
- provider: nous
model: nous-hermes-3
```
**Local model as fallback for cloud:**
```yaml
fallback_providers:
- provider: custom
model: llama-3.1-70b
base_url: http://localhost:8000/v1
key_env: LOCAL_API_KEY
```
**Codex OAuth as fallback:**
```yaml
fallback_providers:
- provider: openai-codex
model: gpt-5.3-codex
```
### Where Fallback Works
| Context | Fallback Supported |
|---------|-------------------|
| CLI sessions | ✔ |
| Messaging gateway (Telegram, Discord, etc.) | ✔ |
| Subagent delegation | ✔ (subagents inherit the parent fallback chain) |
| Cron jobs | ✔ (cron agents inherit configured fallback providers) |
| Auxiliary tasks on `provider: auto` | ✔ (try per-task fallback, then the main fallback chain before built-in aux discovery) |
:::tip
There are no environment variables for the primary fallback chain — configure it exclusively through `config.yaml` or `hermes fallback`. This is intentional: fallback configuration is a deliberate choice, not something a stale shell export should override.
:::
---
## Auxiliary Task Fallback
Hermes uses separate lightweight models for side tasks. Each task has its own provider resolution chain that acts as a built-in fallback system.
### Tasks with Independent Provider Resolution
| Task | What It Does | Config Key |
|------|-------------|-----------|
| Vision | Image analysis, browser screenshots | `auxiliary.vision` |
| Compression | Context compression summaries | `auxiliary.compression` |
| Skills Hub | Skill search and discovery | `auxiliary.skills_hub` |
| MCP | MCP helper operations | `auxiliary.mcp` |
| Approval | Smart command-approval classification | `auxiliary.approval` |
| Title Generation | Session title summaries | `auxiliary.title_generation` |
| Review | `/review` reviewer subagent (full agent, not a single LLM call) | `auxiliary.review` |
| Triage Specifier | `hermes kanban specify` / dashboard ✨ button — fleshes out a one-liner triage task into a real spec | `auxiliary.triage_specifier` |
### Auto-Detection Chain
When a task's provider is set to `"auto"` (the default), Hermes first tries the main provider + main model for that auxiliary task. If that route is unavailable or later fails with a capacity-style error, Hermes now honors user-configured fallback policy before using the built-in discovery chain:
```text
Main provider + main model → auxiliary.<task>.fallback_chain →
fallback_providers / fallback_model → built-in auxiliary discovery chain
```
The task-specific chain is most precise and wins when present. The top-level `fallback_providers` chain is the same policy the main agent uses, so free-only or same-provider fallback rules apply to auxiliary tasks on `auto` as well.
**Built-in text discovery chain (compression, web extract, title generation, etc.):**
```text
OpenRouter → Nous Portal → Custom endpoint → Codex OAuth →
API-key providers (z.ai, Kimi, MiniMax, Xiaomi MiMo, Hugging Face, Anthropic) → give up
```
**Built-in vision discovery chain:**
```text
Main provider (if vision-capable) → OpenRouter → Nous Portal →
Codex OAuth → Anthropic → Custom endpoint → give up
```
Those built-in chains are a convenience fallback for users who have not declared a task-specific or main fallback policy.
### Configuring Auxiliary Providers
Each task can be configured independently in `config.yaml`:
```yaml
auxiliary:
vision:
provider: "auto" # auto | openrouter | nous | codex | main | anthropic
model: "" # e.g. "openai/gpt-4o"
base_url: "" # direct endpoint (takes precedence over provider)
api_key: "" # API key for base_url
compression:
provider: "auto"
model: ""
fallback_chain: # optional, task-specific fallback policy
- provider: openrouter
model: inclusionai/ring-2.6-1t:free
skills_hub:
provider: "auto"
model: ""
mcp:
provider: "auto"
model: ""
```
Every task above follows the same **provider / model / base_url** pattern. Each task can also declare its own `fallback_chain`; if omitted, `provider: auto` uses the top-level `fallback_providers` chain before Hermes' built-in auxiliary discovery chain.
Context compression is configured under `auxiliary.compression`:
```yaml
auxiliary:
compression:
provider: main # Same provider options as other auxiliary tasks
model: google/gemini-3-flash-preview
base_url: null # Custom OpenAI-compatible endpoint
```
And the primary fallback chain uses:
```yaml
fallback_providers:
- provider: openrouter
model: anthropic/claude-sonnet-4
# base_url: http://localhost:8000/v1 # Optional custom endpoint
```
All three — auxiliary, compression, fallback — work the same way: set `provider` to pick who handles the request, `model` to pick which model, and `base_url` to point at a custom endpoint (overrides provider).
### Provider Options for Auxiliary Tasks
These options apply to `auxiliary:`, `compression:`, and `fallback_providers:` entries only — `"main"` is **not** a valid value for your top-level `model.provider`. For custom endpoints, use `provider: custom` in your `model:` section (see [AI Providers](/integrations/providers)).
| Provider | Description | Requirements |
|----------|-------------|-------------|
| `"auto"` | Try providers in order until one works (default) | At least one provider configured |
| `"openrouter"` | Force OpenRouter | `OPENROUTER_API_KEY` |
| `"nous"` | Force Nous Portal | `hermes auth` |
| `"codex"` | Force Codex OAuth | `hermes model` → ChatGPT or Codex Subscription |
| `"main"` | Use whatever provider the main agent uses (auxiliary tasks only) | Active main provider configured |
| `"anthropic"` | Force Anthropic native | `ANTHROPIC_API_KEY` or Claude Code credentials |
### Direct Endpoint Override
For any auxiliary task, setting `base_url` bypasses provider resolution entirely and sends requests directly to that endpoint:
```yaml
auxiliary:
vision:
base_url: "http://localhost:1234/v1"
api_key: "local-key"
model: "qwen2.5-vl"
```
`base_url` takes precedence over `provider`. Hermes uses the configured `api_key` for authentication, falling back to `OPENAI_API_KEY` if not set. It does **not** reuse `OPENROUTER_API_KEY` for custom endpoints.
---
## Auxiliary Capacity-Error Fallback
When you set an explicit auxiliary provider (e.g. `auxiliary.vision.provider: glm`), Hermes treats that as your preferred choice — but if the provider literally cannot serve the request because of a **capacity error** (HTTP 402 payment required, HTTP 429 daily-quota exhaustion, connection failure), Hermes falls back through a layered chain instead of failing silently:
1. **Primary aux provider** — the one you configured (tried first, always)
2. **`auxiliary.<task>.fallback_chain`** — your per-task override list, if you wrote one
3. **Main agent provider + model** — last-resort safety net (always tried, even if you didn't write a chain)
4. **Warn + re-raise** — if every layer fails, Hermes logs `Auxiliary <task>: ... all fallbacks exhausted` at WARNING level and re-raises the original error
Transient HTTP 429 rate limits (`Retry-After: ...`) are treated as request constraints, not capacity problems — they respect your explicit provider choice and do **not** trigger the fallback ladder. Only daily/monthly quota exhaustion, payment errors, and connection failures bypass the explicit-provider gate.
For users on `provider: auto` (no explicit aux provider), the existing auto-detection chain runs in place of steps 2–3. Its first step is already the main agent model, so `auto` users get the same outcome with zero config.
### Optional: per-task fallback chain
If you want a different fallback ordering than "main agent model first", configure `fallback_chain` explicitly. Each entry needs at least `provider`; `model`, `base_url`, and `api_key` are optional.
```yaml
auxiliary:
vision:
provider: glm
model: glm-4v-flash
fallback_chain:
- provider: openrouter
model: google/gemini-3-flash-preview
- provider: nous
model: anthropic/claude-sonnet-4
compression:
provider: openrouter
fallback_chain:
- provider: openai
model: gpt-4o-mini
timeout: 240 # optional — this candidate's own deadline (seconds)
```
You do **not** need to configure `fallback_chain` to get fallback — the main-agent safety net runs regardless. Use it only when you specifically want a different order than the default.
Each `fallback_chain` entry may also declare its own `timeout` (seconds). Without it, a fallback candidate inherits the task-level timeout — which may be tuned for the primary provider. Declaring a per-entry `timeout` lets a slower-but-reliable fallback (e.g. a large-context summarizer) get the budget it actually needs instead of dying on the primary's clock.
### Provider quota errors that trigger fallback
Hermes recognizes these as capacity-equivalent to 402 credit exhaustion (not transient rate limits):
- Bedrock / LiteLLM: `Too many tokens per day`, `daily limit`, `tokens per day`
- Vertex AI / GCP: `quota exceeded`, `resource exhausted`, `RESOURCE_EXHAUSTED`
- Generic: `daily quota`, `quota_exceeded`
If your provider returns a different phrase for daily-quota exhaustion and Hermes doesn't trigger fallback, that's a bug — open an issue with the exact error string.
---
## Context Compression Fallback
Context compression uses the `auxiliary.compression` config block to control which model and provider handles summarization:
```yaml
auxiliary:
compression:
provider: "auto" # auto | openrouter | nous | main
model: "google/gemini-3-flash-preview"
```
:::info Legacy migration
Older configs with `compression.summary_model` / `compression.summary_provider` / `compression.summary_base_url` are automatically migrated to `auxiliary.compression.*` on first load (config version 17).
:::
If no provider is available for compression, Hermes drops middle conversation turns without generating a summary rather than failing the session.
---
## Delegation Provider Override
Subagents spawned by `delegate_task` inherit the parent agent's primary fallback chain. You can still route subagents to a different primary provider:model pair for cost optimization:
```yaml
delegation:
provider: "openrouter" # override provider for all subagents
model: "google/gemini-3-flash-preview" # override model
# base_url: "http://localhost:1234/v1" # or use a direct endpoint
# api_key: "local-key"
```
See [Subagent Delegation](/user-guide/features/delegation) for full configuration details.
---
## Cron Job Providers
Cron jobs inherit your configured `fallback_providers` chain (or legacy `fallback_model`) when they create an agent. To use a different primary provider for a cron job, configure `provider` and `model` overrides on the cron job itself:
```python
cronjob(
action="create",
schedule="every 2h",
prompt="Check server status",
provider="openrouter",
model="google/gemini-3-flash-preview"
)
```
See [Scheduled Tasks (Cron)](/user-guide/features/cron) for full configuration details.
---
## Summary
| Feature | Fallback Mechanism | Config Location |
|---------|-------------------|----------------|
| Main agent model | `fallback_providers` in config.yaml — per-turn failover on errors (primary restored each turn) | `fallback_providers:` (top-level list) |
| Auxiliary tasks (any) — auto users | Full auto-detection chain (main agent model first, then provider chain) on capacity errors | `auxiliary.<task>.provider: auto` |
| Auxiliary tasks (any) — explicit provider | `fallback_chain` (if set) → main agent model → warn + raise, on capacity errors only | `auxiliary.<task>.fallback_chain` |
| Vision | Layered (see above) + internal OpenRouter retry | `auxiliary.vision` |
| Context compression | Layered (see above); degrades to no-summary if all layers unavailable | `auxiliary.compression` |
| Skills hub | Layered (see above) | `auxiliary.skills_hub` |
| MCP helpers | Layered (see above) | `auxiliary.mcp` |
| Approval classification | Layered (see above) | `auxiliary.approval` |
| Title generation | Layered (see above) | `auxiliary.title_generation` |
| Triage specifier | Layered (see above) | `auxiliary.triage_specifier` |
| Delegation | Inherits the parent's `fallback_providers` chain; optional provider/model override | `delegation.provider` / `delegation.model` |
| Cron jobs | Inherit the configured `fallback_providers` chain; optional per-job provider override | Per-job `provider` / `model` |
website/docs/user-guide/features/goals.md
+289
View File
@@ -0,0 +1,289 @@
---
sidebar_position: 16
title: "Persistent Goals"
description: "Set a standing goal and let Hermes keep working across turns until it's done. Our take on the Ralph loop."
---
# Persistent Goals (`/goal`)
`/goal` gives Hermes a standing objective that survives across turns. After every turn a lightweight judge model checks whether the goal is satisfied by the assistant's last response. If not, Hermes automatically feeds a continuation prompt back into the same session and keeps working — until the goal is achieved, you pause or clear it, or the turn budget runs out.
It's our take on the **Ralph loop**, directly inspired by [Codex CLI 0.128.0's `/goal`](https://github.com/openai/codex) by Eric Traut (OpenAI). The core idea — keep a goal alive across turns and don't stop until it's achieved — is theirs. The implementation here is independent and adapted to Hermes' architecture.
## When to use it
Use `/goal` for tasks where you want Hermes to iterate on its own without you re-prompting every turn:
- "Fix every lint error in `src/` and verify `ruff check` passes"
- "Port feature X from repo Y, including tests, and get CI green"
- "Investigate why session IDs sometimes drift on mid-run compression and write up a report"
- "Build a small CLI to rename files by their EXIF dates, then test it against the photos/ folder"
Tasks where the agent does one turn and stops don't need `/goal`. Tasks where *you'd otherwise have to say "keep going" three times* are where this shines.
## Goals vs Kanban: which one do I want?
`/goal` and [Kanban](./kanban) both keep Hermes working without you re-prompting, so it's tempting to assume one flows into the other. It doesn't — the boundary is sharp:
- **`/goal` is single-session.** The loop feeds continuation prompts back into *this* conversation until the judge says done. Setting a goal never creates a kanban card, never assigns work to another profile, and never fans out. There is no handoff to the board, implicit or otherwise.
- **Kanban is a board of many tasks.** Each card is dispatched to its own worker process with its own session. Cards, dependencies, assignees, and handoffs live on the board — not in `/goal`.
- **The overlap is deliberate, and small.** A kanban card created with `--goal` runs the same Ralph-style continuation engine as `/goal` — but *inside that one card's worker session*. It borrows the engine, not the board. See [Goal-mode cards](./kanban#goal-mode-cards---goal).
| You want | Reach for |
|---|---|
| Keep iterating on one task in this chat until it's done | `/goal <text>` |
| Many independent tasks, with dependencies, handoffs, or multiple profiles | [Kanban](./kanban) — `hermes kanban create …` |
| One card on the board that should keep iterating until its acceptance criteria are met | A kanban card with `--goal` |
:::note
If you want work on the board, put it there yourself (`hermes kanban create …`) — `/goal` won't do it for you. The reverse is also true: pausing, resuming, or clearing a goal in this chat never creates, claims, or moves a kanban card.
:::
## Quick start
```
/goal Fix every failing test in tests/hermes_cli/ and make sure scripts/run_tests.sh passes for that directory
```
What you'll see:
1. **Goal accepted** — `⊙ Goal set (20-turn budget): <your goal>`
2. **Turn 1 runs** — Hermes starts working as if you'd sent the goal as a normal message.
3. **Judge runs** — after the turn, the judge model decides `done`, `continue`, or `blocked`.
4. **Loop fires if needed** — if `continue`, you'll see `↻ Continuing toward goal (1/20): <judge's reason>` and Hermes takes the next step automatically.
5. **Terminates** — eventually you see either `✓ Goal achieved: <reason>` or `⏸ Goal paused — N/20 turns used`.
## Commands
| Command | What it does |
|---|---|
| `/goal <text>` | Set (or replace) the standing goal. Kicks off the first turn immediately so you don't need to send a separate message. |
| `/goal draft <text>` | Draft a structured completion contract from a plain-language objective, then set it. See [Completion contracts](#completion-contracts). |
| `/goal show` | Print the active goal's completion contract. |
| `/goal` or `/goal status` | Show the current goal, its status, and turns used. |
| `/goal pause` | Stop the auto-continuation loop without clearing the goal. |
| `/goal resume` | Resume the loop (resets the turn counter back to zero). |
| `/goal clear` | Drop the goal entirely. |
| `/goal wait <pid> [reason]` | Park the loop on a background process — it stops re-poking the agent every turn while the process runs, and auto-resumes when it exits. |
| `/goal unwait` | Drop the wait barrier and resume the loop immediately. |
| `/goal gate add <command>` | Add a **quality gate**: a shell command that must pass before the goal can be judged done. See [Quality gates](#quality-gates). |
| `/goal gate` or `/goal gate list` | List the goal's gates and their pass/fail state. |
| `/goal gate remove <N>` | Remove the Nth gate (1-based). |
| `/goal gate clear` | Remove all gates. |
Works identically on the CLI and every gateway platform (Telegram, Discord, Slack, Matrix, Signal, WhatsApp, SMS, iMessage, Webhook, API server, and the web dashboard).
## Completion contracts
A bare `/goal <text>` works fine, but a *vague* goal makes for vague judging — the judge can only check what you told it to want. Codex's `/goal` guidance makes the same point: a durable objective works best when it names **what done means, how to prove it, what not to break, what's in scope, and when to stop**. Hermes adapts this as an optional **completion contract** layered on top of the existing goal loop.
A contract has five fields, all optional:
| Field | Meaning |
|---|---|
| `outcome` | The single end state that must be true when done. |
| `verification` | The specific test / command / artifact that *proves* the outcome. |
| `constraints` | What must not change or regress. |
| `boundaries` | Which files, dirs, tools, or systems are in scope. |
| `stop_when` | The condition under which Hermes should stop and ask for input. |
When a contract is set, both prompts change: the **continuation prompt** tells the agent to target the verification surface and respect the constraints, and the **judge prompt** decides `done` *only when the verification criterion is met with concrete evidence* (a command result, file excerpt, test output) — not a loose "looks done" claim. This directly tightens the most common `/goal` failure mode (premature completion or endless over-continuation on an underspecified objective).
### Two ways to set a contract
**1. Let Hermes draft it** (recommended — adapted from Codex's "let the agent draft the goal" tip):
```
/goal draft Migrate the auth service from session cookies to JWT
```
Hermes expands your one-liner into a full contract via the `goal_judge` auxiliary model, sets it, and shows you the result so you can review or tighten any field. If the aux model is unavailable, it falls back to a plain free-form goal — drafting never blocks setting a goal.
**2. Write it inline** with `field: value` lines:
```
/goal Migrate auth to JWT
verify: pytest tests/auth passes
constraints: keep the /login response shape unchanged
boundaries: only touch services/auth and its tests
stop when: a DB schema migration is required
```
The first non-field line(s) are the goal headline; recognized field prefixes (`verify:`, `verified by:`, `constraints:`, `preserve:`, `boundaries:`, `scope:`, `stop when:`, `blocked:`, …) populate the contract. A plain goal with an incidental colon (`Fix bug: the parser drops commas`) is **not** mangled — only known field prefixes are pulled out.
Use `/goal show` to review the active contract. Contracts persist in `SessionDB.state_meta` alongside the goal, so they survive `/resume`. Old goals from before this feature load unchanged (no contract). Contracts and `/subgoal` criteria compose: subgoals fold into the contract as extra criteria the judge must also satisfy.
## Adding criteria mid-goal: `/subgoal`
While a goal is active you can append extra acceptance criteria with `/subgoal <text>` without resetting the loop. Each call adds one numbered item to the goal's subgoal list; the **continuation prompt** the agent sees on the next turn includes the original goal plus an "Additional criteria the user added mid-loop" block, and the **judge prompt** is rewritten so the verdict must consider every subgoal — the goal isn't marked done until the original objective **and** every subgoal are met.
| Command | What it does |
|---|---|
| `/subgoal <text>` | Append a new criterion to the active goal. Requires an active `/goal`. |
| `/subgoal` (no args) | Show the current numbered subgoal list. |
| `/subgoal remove <N>` | Remove the Nth subgoal (1-based). |
| `/subgoal clear` | Drop every subgoal but keep the original goal intact. |
Subgoals are persisted alongside the goal in `SessionDB.state_meta`, so they survive `/resume`. Setting a new `/goal <text>` replaces the goal and clears the subgoal list; `/goal clear` does the same.
Use this when you start a loop ("fix the failing tests") and notice partway through that you also want it to "and add a regression test for the bug you just patched" — `/subgoal add a regression test` tightens the success criteria without breaking the running loop.
## Quality gates
A completion contract makes the judge stricter, but the judge is still an LLM reading prose. A **quality gate** is stronger: a deterministic shell command that must exit 0 before the goal can complete at all. Inspired by Prime-Agent's bounded autonomous mode (`--autonomous-gate`).
```
/goal Fix the flaky session tests
/goal gate add scripts/run_tests.sh tests/hermes_cli/test_goals.py
```
How it works, each turn:
1. **Gates run before the judge.** If any gate fails, the judge is *not called* — a red gate is deterministic evidence the goal isn't done. The gate's exit code and output tail (last ~3 KB) become the continuation prompt, so the agent iterates against the actual failure instead of a vibe.
2. **All gates pass → normal judging.** The LLM judge then decides done/blocked/continue/wait exactly as before.
3. **Unchanged workspace → no re-run.** If a gate failed and nothing changed in the workspace since (tracked via a git fingerprint of HEAD + working-tree status), the gate is not re-run — the recorded failure is replayed and the attempt count advances. A stuck agent can't burn wall-clock re-running an identical red suite. Outside a git repo, gates simply always re-run.
4. **Retries are bounded.** Each gate defaults to 3 retries and a 5-minute timeout. When a gate exhausts its retries the goal auto-pauses (like the turn budget) with a message telling you to fix it manually, remove the gate, or `/goal resume`.
Gates persist with the goal in `SessionDB.state_meta` (they survive `/resume` and context compression), and gate management (`/goal gate …`) is safe mid-run on the gateway — gates only run at turn boundary.
Gates and contracts compose: use a contract to shape *what the agent aims for*, and gates to make *"done" mechanically checkable*. When both are set, gates run first.
## Parking on a background process: automatic, with a manual override
Some goals are gated on something that takes minutes and runs on its own — CI on a pushed PR, a long build, a test matrix, a deploy, a rate-limit cooldown. Without help, the goal loop would re-poke the agent every turn into "is it done yet?" busy-work while it waits.
**This is handled automatically.** Every turn, the judge is shown the agent's live background processes (the `terminal(background=true)` registry — pid, session id, command, uptime, recent output, and any `watch_patterns` / `notify_on_complete` trigger) alongside the goal and the agent's response. When the agent's progress is genuinely gated on one of them, the judge returns a **`wait`** verdict instead of `continue`, and the loop **parks**: the next turns are skipped (no judge call, no continuation, no turn consumed) until the wait is satisfied — then it resumes normally with the result in hand. The judge can also park on a **time** basis (`wait_for_seconds`) for backoff/cooldown waits. `/goal status` shows `⏳ Goal (parked …)` while parked.
The judge picks the right kind of wait from the process's own signal:
- **`wait_on_session <id>`** — releases when the process's *own trigger* fires: it exits, **or** (if it was started with `watch_patterns`) its pattern matches. This is the one for a long-lived watcher / server / poller that signals **mid-run** (e.g. a build process that prints `BUILD SUCCESSFUL` and keeps running, or a `notify_on_complete` watcher) and may never exit on its own.
- **`wait_on_pid <pid>`** — releases on process exit only.
- **`wait_for_seconds <n>`** — releases after a fixed delay.
You don't type anything for this — it's the judge's decision, made from the process context the loop hands it. The manual commands exist as an override:
| Command | What it does |
|---|---|
| `/goal wait <pid> [reason]` | Manually park the loop until the process with that PID exits. |
| `/goal unwait` | Clear any wait barrier (judge- or manually-set) and resume immediately. |
The barrier (pid- or time-based) is persisted with the goal in `SessionDB.state_meta`, so it survives `/resume`. `/goal pause`, `/goal resume`, and `/goal clear` all drop it. If the PID is already dead when the barrier is set (or dies while parked), or the time deadline passes, the barrier clears on the next check — a stale barrier can never wedge the loop.
Typical flow: the agent pushes a PR, starts a CI watcher with `terminal(background=true, notify_on_complete=true)`, and reports "watching CI." The judge sees the watcher process still running, returns `wait` on its pid, and the loop goes quiet — then picks back up the instant CI finishes and judges the goal against the actual result.
## Behavior details
### The judge
After every turn, Hermes calls an auxiliary model with:
- The standing goal text
- The agent's most recent final response (last ~4 KB of text)
- A system prompt telling the judge to reply with strict one-line JSON: `{"verdict": "done" | "blocked" | "continue" | "wait", "reason": "<one-sentence rationale>"}` (wait verdicts add `wait_on_session` / `wait_on_pid` / `wait_for_seconds`; the legacy `{"done": <bool>, "reason": "..."}` shape is still accepted)
The judge is deliberately conservative: it marks a goal `done` only when the response **explicitly** confirms the goal is complete, when the final deliverable is clearly produced. A goal the agent explains is **unachievable** (impossible, out of scope, needs user input) gets a `blocked` verdict instead — never `done`: the goal **pauses** with the judge's reason (`🚫 Goal judged unachievable — paused`), so you can re-scope it with `/goal <text>` or override with `/goal resume` rather than burning budget or having an impossible task waved through as complete.
### Fail-open semantics
If the judge errors (network blip, malformed response, unavailable aux client), Hermes treats the verdict as `continue` — a broken judge never wedges progress. The **turn budget** is the real backstop.
### Turn budget
Default is 20 continuation turns (`goals.max_turns` in `config.yaml`). When the budget is hit, Hermes auto-pauses and tells you exactly how to proceed:
```
⏸ Goal paused — 20/20 turns used. Use /goal resume to keep going, or /goal clear to stop.
```
`/goal resume` resets the counter to zero, so you can keep going in measured chunks.
### User messages always preempt
Any real message you send while a goal is active takes priority over the continuation loop. On the CLI your message lands in `_pending_input` ahead of the queued continuation; on the gateway it goes through the adapter FIFO the same way. The judge runs again after your turn — so if your message happens to complete the goal, the judge will catch it and stop.
### Mid-run safety (gateway)
While an agent is already running, `/goal status`, `/goal pause`, `/goal clear`, `/goal wait`, and `/goal unwait` are safe to run — they only touch control-plane state and don't interrupt the current turn. Setting a **new** goal mid-run (`/goal <new text>`) is rejected with a message telling you to `/stop` first, so the old continuation can't race the new one.
### Persistence
Goal state lives in `SessionDB.state_meta` keyed by `goal:<session_id>`. That means `/resume` picks up right where you left off — set a goal, close your laptop, come back tomorrow, `/resume`, and the goal is still standing exactly as you left it (active, paused, or done).
### Prompt cache
The continuation prompt is a plain user-role message appended to history. It does **not** mutate the system prompt, swap toolsets, or touch the conversation in any way that invalidates Hermes' prompt cache. Running a 20-turn goal costs the same cache-wise as 20 turns of normal conversation.
## Configuration
Add to `~/.hermes/config.yaml`:
```yaml
goals:
# Max continuation turns before Hermes auto-pauses and asks you to
# /goal resume. Default 20. Lower this if you want tighter loops;
# raise it for long-running refactors.
max_turns: 20
```
### Choosing the judge model
The judge uses the `goal_judge` auxiliary task. By default it resolves to your main model (see [Auxiliary Models](/user-guide/configuration#auxiliary-models)). If you want to route the judge to a cheap fast model to keep costs down, add an override:
```yaml
auxiliary:
goal_judge:
provider: openrouter
model: google/gemini-3-flash-preview
```
The judge call is small (~200 output tokens) and runs once per turn, so a cheap fast model is usually the right call.
## Example walkthrough
```
You: /goal Create four files /tmp/note_{1..4}.txt, one per turn, each containing its number as text
⊙ Goal set (20-turn budget): Create four files /tmp/note_{1..4}.txt, one per turn, each containing its number as text
Hermes: Creating /tmp/note_1.txt now.
💻 echo "1" > /tmp/note_1.txt (0.1s)
I've created /tmp/note_1.txt with the content "1". I'll continue with the remaining files on the next turn as you specified.
↻ Continuing toward goal (1/20): Only 1 of 4 files has been created; 3 files remain.
Hermes: [Continuing toward your standing goal]
💻 echo "2" > /tmp/note_2.txt (0.1s)
Created /tmp/note_2.txt. Two more to go.
↻ Continuing toward goal (2/20): 2 of 4 files created; 2 remain.
Hermes: [Continuing toward your standing goal]
💻 echo "3" > /tmp/note_3.txt (0.1s)
Created /tmp/note_3.txt.
↻ Continuing toward goal (3/20): 3 of 4 files created; 1 remains.
Hermes: [Continuing toward your standing goal]
💻 echo "4" > /tmp/note_4.txt (0.1s)
All four files have been created: /tmp/note_1.txt through /tmp/note_4.txt, each containing its number.
✓ Goal achieved: All four files were created with the specified content, completing the goal.
You: _
```
Four turns, one `/goal` invocation, zero "keep going" prompts from you.
## When the judge gets it wrong
No judge is perfect. Two failure modes to watch for:
**False negative — judge says continue when the goal is actually done.** The turn budget catches this. You'll see `⏸ Goal paused` and can `/goal clear` or just send a new message.
**False positive — judge says done when work remains.** You'll see `✓ Goal achieved` but you know better. Send a follow-up message to continue, or re-set the goal more precisely: `/goal <more specific text>`. The judge's system prompt is deliberately conservative to make false positives rarer than false negatives.
If you find a judge verdict unconvincing, the reason text in the `↻ Continuing toward goal` or `✓ Goal achieved` line tells you exactly what the judge saw. That's usually enough to diagnose whether the goal text was ambiguous or the model's response was.
## Attribution
`/goal` is Hermes' take on the **Ralph loop** pattern. The user-facing design — keep a goal alive across turns, don't stop until it's achieved, with create/pause/resume/clear controls — was popularised and shipped in [Codex CLI 0.128.0](https://github.com/openai/codex) by Eric Traut on OpenAI's Codex team. Our implementation is independent (central `CommandDef` registry, `SessionDB.state_meta` persistence, auxiliary-client judge, adapter-FIFO continuation on the gateway side) but the idea is theirs. Credit where credit's due.
website/docs/user-guide/features/heartbeat.md
+65
View File
@@ -0,0 +1,65 @@
---
sidebar_position: 17
title: "Session Heartbeats"
description: "A recurring prompt that re-enters your current session whenever it's idle — /heartbeat every 10m Check the deployment."
---
# Session Heartbeats (`/heartbeat`)
`/heartbeat` gives the **current session** one recurring instruction. Whenever the session is idle and the interval has elapsed, the prompt fires as a normal user turn — same conversation, same context, same prompt cache.
```
/heartbeat every 10m Check the deployment and report meaningful changes
```
Inspired by Prime-Agent's `/heartbeat`. The Hermes adaptation keeps the strict message-flow invariants: the heartbeat is injected only between turns (never mid-run), as a plain user-role message.
## Heartbeat vs cron: which one do I want?
They look similar but serve different jobs:
| | `/heartbeat` | [`hermes cron`](./cron) |
|---|---|---|
| Runs in | **This conversation** — full context, memory of the discussion | A fresh isolated session per tick |
| Survives process restart | State survives (SessionDB); firing resumes next time the session is driven | Yes — fully durable scheduler |
| How many | One per session | Unlimited jobs |
| Best for | "Keep an eye on X *in this thread* while we work" | Standing jobs, reports, watchdogs, deliveries |
Rule of thumb: if the recurring prompt needs the conversation's context, use `/heartbeat`. If it's a self-contained job, use cron.
## Commands
| Command | What it does |
|---|---|
| `/heartbeat every <interval> <prompt>` | Set (or replace) the session's heartbeat. Intervals: `90s`, `10m`, `2h`, `1d` (minimum 60s). |
| `/heartbeat` or `/heartbeat status` | Show the heartbeat, its interval, and time to next fire. |
| `/heartbeat pause` | Stop firing without clearing. |
| `/heartbeat resume` | Resume (re-anchors the timer — no instant stale fire). |
| `/heartbeat clear` | Remove the heartbeat. |
`/hb` is an alias. Works on the CLI and gateway platforms (on Slack, use `/hermes heartbeat …`).
## Behavior details
- **Idle-only.** A heartbeat never interrupts a running turn. If the agent is busy when the tick comes due, it fires at the next idle poll.
- **Missed ticks coalesce.** If the session was busy (or the process wasn't running) through several intervals, you get **one** heartbeat turn, not a backlog. The timer re-anchors on every fire.
- **User messages win.** A queued user message always takes priority; the heartbeat waits for the input queue to drain.
- **Cache-safe.** The injected prompt is an ordinary user message. No system-prompt mutation, no toolset change.
- **Persistence.** State lives in `SessionDB.state_meta` keyed by `heartbeat:<session_id>` — it survives `/resume` and rides across context-compression session rotations. Firing requires the owning process (CLI session or gateway) to be running; for schedules that must survive anything, use cron.
- **Don't-invent-work guard.** The injected prompt tells the agent to reply briefly and stop when nothing meaningful changed, so an idle heartbeat doesn't generate busywork.
## Example
```
You: /heartbeat every 15m Check whether the CI run for PR #1234 finished; summarize the result when it does
♥ Heartbeat set (every 15m): Check whether the CI run for PR #1234 finished; ...
[15 minutes of you working on other things in the same session]
Hermes: [Heartbeat — recurring instruction, fires every 15m]
💻 gh pr checks 1234 (1.2s)
CI is still running (14/37 checks complete). Nothing to report yet.
```
When the answer stops changing, `/heartbeat clear` it — or let it keep watch.
website/docs/user-guide/features/honcho.md
+265
View File
@@ -0,0 +1,265 @@
---
sidebar_position: 99
title: "Honcho Memory"
description: "AI-native persistent memory via Honcho — dialectic reasoning, multi-agent user modeling, and deep personalization"
---
# Honcho Memory
[Honcho](https://github.com/plastic-labs/honcho) is an AI-native memory backend that adds dialectic reasoning and deep user modeling on top of Hermes's built-in memory system. Instead of simple key-value storage, Honcho maintains a running model of who the user is — their preferences, communication style, goals, and patterns — by reasoning about conversations after they happen.
:::info Honcho is a Memory Provider Plugin
Honcho is integrated into the [Memory Providers](./memory-providers.md) system. All features below are available through the unified memory provider interface.
:::
## What Honcho Adds
| Capability | Built-in Memory | Honcho |
|-----------|----------------|--------|
| Cross-session persistence | ✔ File-based MEMORY.md/USER.md | ✔ Server-side with API |
| User profile | ✔ Manual agent curation | ✔ Automatic dialectic reasoning |
| Session summary | — | ✔ Session-scoped context injection |
| Multi-agent isolation | — | ✔ Per-peer profile separation |
| Observation modes | — | ✔ Unified or directional observation |
| Conclusions (derived insights) | — | ✔ Server-side reasoning about patterns |
| Search across history | ✔ FTS5 session search | ✔ Semantic search over conclusions |
**Dialectic reasoning**: After each conversation turn (gated by `dialecticCadence`), Honcho analyzes the exchange and derives insights about the user's preferences, habits, and goals. These accumulate over time, giving the agent a deepening understanding that goes beyond what the user explicitly stated. The dialectic supports multi-pass depth (1–3 passes) with automatic cold/warm prompt selection — cold start queries focus on general user facts while warm queries prioritize session-scoped context.
**Session-scoped context**: Base context now includes the session summary alongside the user representation and peer card. This gives the agent awareness of what has already been discussed in the current session, reducing repetition and enabling continuity.
**Multi-agent profiles**: When multiple Hermes instances talk to the same user (e.g., a coding assistant and a personal assistant), Honcho maintains separate "peer" profiles. Each peer sees only its own observations and conclusions, preventing cross-contamination of context.
## Setup
```bash
hermes memory setup # select "honcho" from the provider list
```
Or configure manually:
```yaml
# ~/.hermes/config.yaml
memory:
provider: honcho
```
```bash
echo 'HONCHO_API_KEY=***' >> ~/.hermes/.env
```
Get an API key at [honcho.dev](https://honcho.dev).
## Architecture
### Two-Layer Context Injection
Every turn (in `hybrid` or `context` mode), Honcho assembles two layers of context injected into the system prompt:
1. **Base context** — session summary, user representation, user peer card, AI self-representation, and AI identity card. Refreshed on `contextCadence`. This is the "who is this user" layer.
2. **Dialectic supplement** — LLM-synthesized reasoning about the user's current state and needs. Refreshed on `dialecticCadence`. This is the "what matters right now" layer.
Both layers are concatenated and truncated to the `contextTokens` budget (if set).
### Cold/Warm Prompt Selection
The dialectic automatically selects between two prompt strategies:
- **Cold start** (no base context yet): General query — "Who is this person? What are their preferences, goals, and working style?"
- **Warm session** (base context exists): Session-scoped query — "Given what's been discussed in this session so far, what context about this user is most relevant?"
This happens automatically based on whether base context has been populated.
### Three Orthogonal Config Knobs
Cost and depth are controlled by three independent knobs:
| Knob | Controls | Default |
|------|----------|---------|
| `contextCadence` | Turns between `context()` API calls (base layer refresh) | `1` |
| `dialecticCadence` | Turns between `peer.chat()` LLM calls (dialectic layer refresh) | `2` (recommended 1–5) |
| `dialecticDepth` | Number of `.chat()` passes per dialectic invocation (1–3) | `1` |
These are orthogonal — you can have frequent context refreshes with infrequent dialectic, or deep multi-pass dialectic at low frequency. Example: `contextCadence: 1, dialecticCadence: 5, dialecticDepth: 2` refreshes base context every turn, runs dialectic every 5 turns, and each dialectic run makes 2 passes.
### Dialectic Depth (Multi-Pass)
When `dialecticDepth` > 1, each dialectic invocation runs multiple `.chat()` passes:
- **Pass 0**: Cold or warm prompt (see above)
- **Pass 1**: Self-audit — identifies gaps in the initial assessment and synthesizes evidence from recent sessions
- **Pass 2**: Reconciliation — checks for contradictions between prior passes and produces a final synthesis
Each pass uses a proportional reasoning level (lighter early passes, base level for the main pass). Override per-pass levels with `dialecticDepthLevels` — e.g., `["minimal", "medium", "high"]` for a depth-3 run.
Passes bail out early if the prior pass returned strong signal (long, structured output), so depth 3 doesn't always mean 3 LLM calls.
### Session-Start Prewarm
On session init, Honcho fires a dialectic call in the background at the full configured `dialecticDepth` and hands the result directly to turn 1's context assembly. A single-pass prewarm on a cold peer often returns thin output — multi-pass depth runs the audit/reconcile cycle before the user ever speaks. If prewarm hasn't landed by turn 1, turn 1 falls back to a synchronous call with a bounded timeout.
### Query-Adaptive Reasoning Level
The auto-injected dialectic scales `dialecticReasoningLevel` by query length: +1 level at ≥120 chars, +2 at ≥400, clamped at `reasoningLevelCap` (default `"high"`). Disable with `reasoningHeuristic: false` to pin every auto call to `dialecticReasoningLevel`. Available levels: `minimal`, `low`, `medium`, `high`, `max`.
## Configuration Options
Honcho is configured in `~/.honcho/config.json` (global) or `$HERMES_HOME/honcho.json` (profile-local). The setup wizard handles this for you.
### Self-Hosted Honcho with Authentication
When pointing Hermes at a self-hosted Honcho server, `hermes honcho setup` (and `hermes memory setup`) ask for a **local JWT / bearer token** after the base URL. Paste a JWT signed with the server's `AUTH_JWT_SECRET` (the Honcho compose env var) to enable authenticated access; leave it blank for servers running with `AUTH_USE_AUTH=false`. The local token is stored under the host block (`hosts.<host>.apiKey` in `honcho.json`), separate from any cloud `apiKey`, so you can flip the `Cloud or local?` prompt back to `cloud` later without losing either credential.
### Full Config Reference
| Key | Default | Description |
|-----|---------|-------------|
| `contextTokens` | `null` (uncapped) | Token budget for auto-injected context per turn. Set to an integer (e.g. 1200) to cap. Truncates at word boundaries |
| `contextCadence` | `1` | Minimum turns between `context()` API calls (base layer refresh) |
| `dialecticCadence` | `2` | Minimum turns between `peer.chat()` LLM calls (dialectic layer). Recommended 1–5. In `tools` mode, irrelevant — model calls explicitly |
| `dialecticDepth` | `1` | Number of `.chat()` passes per dialectic invocation. Clamped to 1–3 |
| `dialecticDepthLevels` | `null` | Optional array of reasoning levels per pass, e.g. `["minimal", "low", "medium"]`. Overrides proportional defaults |
| `dialecticReasoningLevel` | `'low'` | Base reasoning level: `minimal`, `low`, `medium`, `high`, `max` |
| `dialecticDynamic` | `true` | When `true`, model can override reasoning level per-call via tool param |
| `dialecticMaxChars` | `600` | Max chars of dialectic result injected into system prompt |
| `recallMode` | `'hybrid'` | `hybrid` (auto-inject + tools), `context` (inject only), `tools` (tools only) |
| `writeFrequency` | `'async'` | When to flush messages: `async` (background thread), `turn` (sync), `session` (batch on end), or integer N |
| `saveMessages` | `true` | Whether to persist messages to Honcho API |
| `observationMode` | `'directional'` | `directional` (all on) or `unified` (shared pool). Override with `observation` object for granular control |
| `messageMaxChars` | `25000` | Max chars per message sent via `add_messages()`. Chunked if exceeded |
| `dialecticMaxInputChars` | `10000` | Max chars for dialectic query input to `peer.chat()` |
| `sessionStrategy` | `'per-directory'` | `per-directory`, `per-repo`, `per-session`, or `global` |
| `pinUserPeer` | `false` | Gateway only. When `true`, every platform user collapses to `peerName` |
| `userPeerAliases` | `{}` | Gateway only. Map of runtime IDs to peers (`{"7654321": "alice"}`). Many-to-one |
| `runtimePeerPrefix` | `""` | Gateway only. Namespaces unknown runtime IDs (`telegram_7654321`) when no alias matches |
**Session strategy** controls how Honcho sessions map to your work:
- `per-session` — each `hermes` run gets a fresh session. Clean starts, memory via tools. Recommended for new users.
- `per-directory` — one Honcho session per working directory. Context accumulates across runs.
- `per-repo` — one session per git repository.
- `global` — single session across all directories.
**Recall mode** controls how memory flows into conversations:
- `hybrid` — context auto-injected into system prompt AND tools available (model decides when to query).
- `context` — auto-injection only, tools hidden.
- `tools` — tools only, no auto-injection. Agent must explicitly call `honcho_reasoning`, `honcho_search`, etc.
**Settings per recall mode:**
| Setting | `hybrid` | `context` | `tools` |
|---------|----------|-----------|---------|
| `writeFrequency` | flushes messages | flushes messages | flushes messages |
| `contextCadence` | gates base context refresh | gates base context refresh | irrelevant — no injection |
| `dialecticCadence` | gates auto LLM calls | gates auto LLM calls | irrelevant — model calls explicitly |
| `dialecticDepth` | multi-pass per invocation | multi-pass per invocation | irrelevant — model calls explicitly |
| `contextTokens` | caps injection | caps injection | irrelevant — no injection |
| `dialecticDynamic` | gates model override | N/A (no tools) | gates model override |
In `tools` mode, the model is fully in control — it calls `honcho_reasoning` when it wants, at whatever `reasoning_level` it picks. Cadence and budget settings only apply to modes with auto-injection (`hybrid` and `context`).
## Gateway Identity Mapping
These settings only matter when you run the [Hermes gateway](../../developer-guide/gateway-internals.md) — the one entrypoint where users arrive with platform-native runtime IDs (Telegram UID, Discord snowflake, Slack user). CLI, TUI, and desktop sessions have no runtime ID and always resolve to `peerName`, so off-gateway these keys do nothing.
The setup wizard detects whether a gateway platform is connected and skips this step entirely if not. When it runs, it asks one question — *who talks to this gateway?* — and derives the keys:
| Answer | Result |
|--------|--------|
| **just me** | `pinUserPeer: true` — every non-agent gateway user collapses to your peer. Pin overrides all aliases, so pick this only when no user-side identity needs its own peer. If separate agents reach the gateway and each needs a distinct peer, do **not** pin — leave `pinUserPeer: false` and map them via `userPeerAliases` (the `[e]` editor) instead |
| **me + other people** (pooled) | `pinUserPeer: false` + `userPeerAliases` mapping your runtime IDs to `peerName` — you stay on your shared history, others get their own peers |
| **only other people** | `pinUserPeer: false`, optional `runtimePeerPrefix` — each user gets their own peer |
Pick `[e]` at the prompt to set the three keys directly instead.
The resolver tries the keys top-down, first match wins: `pinUserPeer` → `userPeerAliases[id]` → `runtimePeerPrefix + id` → raw runtime ID → `peerName` → session-key fallback.
:::warning Un-pinning orphans pooled memory
Flipping `pinUserPeer` from `true` to `false` does not migrate data — memory accumulated under `peerName` stays there, and platform users resolve to fresh, empty peers. To keep your own continuity, choose the **pooled** path so your runtime IDs alias back to `peerName`. The wizard offers this steer automatically when it detects the transition.
:::
:::note Deprecated key
`pinPeerName` is a legacy alias for `pinUserPeer` — still read for back-compat (`pinUserPeer` wins where both are set), never written. Re-running setup migrates it onto the canonical key.
:::
## Observation (Directional vs. Unified)
Honcho models a conversation as peers exchanging messages. Each peer has two observation toggles that map 1:1 to Honcho's `SessionPeerConfig`:
| Toggle | Effect |
|--------|--------|
| `observeMe` | Honcho builds a representation of this peer from its own messages |
| `observeOthers` | This peer observes the other peer's messages (feeds cross-peer reasoning) |
Two peers × two toggles = four flags. `observationMode` is a shorthand preset:
| Preset | User flags | AI flags | Semantics |
|--------|-----------|----------|-----------|
| `"directional"` (default) | me: on, others: on | me: on, others: on | Full mutual observation. Enables cross-peer dialectic — "what does the AI know about the user, based on what the user said and the AI replied." |
| `"unified"` | me: on, others: off | me: off, others: on | Shared-pool semantics — the AI observes the user's messages only, the user peer only self-models. Single-observer pool. |
Override the preset with an explicit `observation` block for per-peer control:
```json
"observation": {
"user": { "observeMe": true, "observeOthers": true },
"ai": { "observeMe": true, "observeOthers": false }
}
```
Common patterns:
| Intent | Config |
|--------|--------|
| Full observation (most users) | `"observationMode": "directional"` |
| AI shouldn't re-model the user from its own replies | `"ai": {"observeMe": true, "observeOthers": false}` |
| Strong persona the AI peer shouldn't update from self-observation | `"ai": {"observeMe": false, "observeOthers": true}` |
Server-side toggles set via the [Honcho dashboard](https://app.honcho.dev) win over local defaults — Hermes syncs them back at session init.
## Tools
When Honcho is active as the memory provider, five tools become available:
| Tool | Purpose |
|------|---------|
| `honcho_profile` | Read or update peer card — pass `card` (list of facts) to update, omit to read |
| `honcho_search` | Semantic search over context — raw excerpts, no LLM synthesis |
| `honcho_context` | Full session context — summary, representation, card, recent messages |
| `honcho_reasoning` | Synthesized answer from Honcho's LLM — pass `reasoning_level` (minimal/low/medium/high/max) to control depth |
| `honcho_conclude` | Create or delete conclusions — pass `conclusion` to create, `delete_id` to remove (PII only) |
## CLI Commands
The `hermes honcho` subcommand is **only registered when Honcho is the active memory provider** (`memory.provider: honcho` in `config.yaml`). On a fresh install, configure Honcho directly with `hermes memory setup honcho` (or run `hermes memory setup` and pick it from the list); the `hermes honcho` subcommand then appears on the next invocation.
```bash
hermes memory setup honcho # Configure Honcho directly (works before activation)
hermes honcho status # Connection status, config, and key settings
hermes honcho setup # Redirects to `hermes memory setup` (post-activation alias)
hermes honcho strategy # Show or set session strategy (per-session/per-directory/per-repo/global)
hermes honcho peer # Show or update peer names + dialectic reasoning level
hermes honcho mode # Show or set recall mode (hybrid/context/tools)
hermes honcho tokens # Show or set token budget for context and dialectic
hermes honcho identity # Seed or show the AI peer's Honcho identity
hermes honcho sync # Sync Honcho config to all existing profiles
hermes honcho peers # Show peer identities across all profiles
hermes honcho sessions # List known Honcho session mappings
hermes honcho map # Map current directory to a Honcho session name
hermes honcho enable # Enable Honcho for the active profile
hermes honcho disable # Disable Honcho for the active profile
hermes honcho migrate # Step-by-step migration guide from openclaw-honcho
```
## Migrating from `hermes honcho`
If you previously used the standalone `hermes honcho setup`:
1. Your existing configuration (`honcho.json` or `~/.honcho/config.json`) is preserved
2. Your server-side data (memories, conclusions, user profiles) is intact
3. Set `memory.provider: honcho` in config.yaml to reactivate
No re-login or re-setup needed. Run `hermes memory setup` and select "honcho" — the wizard detects your existing config.
## Full Documentation
See [Memory Providers — Honcho](./memory-providers.md#honcho) for the complete reference.
website/docs/user-guide/features/hooks.md
+1928
View File
File diff suppressed because it is too large Load Diff
website/docs/user-guide/features/image-generation.md
+284
View File
@@ -0,0 +1,284 @@
---
title: Image Generation
description: Generate images via FAL.ai — 11 models including FLUX 2, GPT Image (1.5 & 2), Nano Banana Pro, Ideogram, Recraft V4 Pro, Krea 2, and more, selectable via `hermes tools`.
sidebar_label: Image Generation
sidebar_position: 6
---
# Image Generation
Hermes Agent generates images from text prompts via FAL.ai. Eleven models are supported out of the box, each with different speed, quality, and cost tradeoffs. The active model is user-configurable via `hermes tools` and persists in `config.yaml`.
## Supported Models
| Model | Speed | Strengths | Price |
|---|---|---|---|
| `fal-ai/flux-2/klein/9b` *(default)* | `<1s` | Fast, crisp text | $0.006/MP |
| `fal-ai/flux-2-pro` | ~6s | Studio photorealism | $0.03/MP |
| `fal-ai/z-image/turbo` | ~2s | Bilingual EN/CN, 6B params | $0.005/MP |
| `fal-ai/nano-banana-pro` | ~8s | Gemini 3 Pro, reasoning depth, text rendering | $0.15/image (1K) |
| `fal-ai/gpt-image-1.5` | ~15s | Prompt adherence | $0.034/image |
| `fal-ai/gpt-image-2` | ~20s | SOTA text rendering + CJK, world-aware photorealism | $0.04–0.06/image |
| `fal-ai/ideogram/v3` | ~5s | Best typography | $0.03–0.09/image |
| `fal-ai/recraft/v4/pro/text-to-image` | ~8s | Design, brand systems, production-ready | $0.25/image |
| `fal-ai/qwen-image` | ~12s | LLM-based, complex text | $0.02/MP |
| `fal-ai/krea/v2/medium/text-to-image` | ~15-25s | Illustration, anime, painting, expressive/artistic styles | $0.030–0.035/image |
| `fal-ai/krea/v2/large/text-to-image` | ~25-60s | Photorealism, raw textured looks (motion blur, grain, film) | $0.060–0.065/image |
Prices are FAL's pricing at time of writing; check [fal.ai](https://fal.ai/) for current numbers.
## Setup
:::tip Nous Subscribers
If you have a paid [Nous Portal](https://portal.nousresearch.com) subscription, you can use image generation through the **[Tool Gateway](tool-gateway.md)** without a FAL API key. Your model selection persists across both paths. New installs can run `hermes setup --portal` to log in and turn on every gateway tool at once; existing installs can pick **Nous Subscription** as the image-gen backend via `hermes tools`.
If the managed gateway returns `HTTP 4xx` for a specific model, that model isn't yet proxied on the portal side — the agent will tell you so, with remediation steps (switch to FAL.ai in `hermes tools` with your own `FAL_KEY` for direct access, or pick a different model).
:::
### Get a FAL API Key
1. Sign up at [fal.ai](https://fal.ai/)
2. Generate an API key from your dashboard
### Configure and Pick a Model
Run the tools command:
```bash
hermes tools
```
Navigate to **🎨 Image Generation**, pick your backend (Nous Subscription or FAL.ai), then the picker shows all supported models in a column-aligned table — arrow keys to navigate, Enter to select:
```
Model Speed Strengths Price
fal-ai/flux-2/klein/9b <1s Fast, crisp text $0.006/MP ← currently in use
fal-ai/flux-2-pro ~6s Studio photorealism $0.03/MP
fal-ai/z-image/turbo ~2s Bilingual EN/CN, 6B $0.005/MP
...
```
Your selection is saved to `config.yaml`:
```yaml
image_gen:
provider: fal # `nous` if you picked Nous Subscription
model: fal-ai/flux-2/klein/9b
max_parallel_requests: 4 # concurrent images in one tool-call batch
```
`image_gen.provider` is the single selection key: `nous` routes through the managed Tool Gateway; a vendor name (`fal`, `openai`, `xai`, `krea`, ...) goes direct with your own key. The runtime always follows this stored selection — a `FAL_KEY` in `.env` is ignored while `provider: nous`, and `provider: fal` without `FAL_KEY` errors with `image_gen is configured to use fal (set via hermes tools), but FAL_KEY is not set. Run 'hermes tools' to change it.` rather than silently rerouting. Change providers via `hermes tools`, not by adding/removing keys. (The old `use_gateway` boolean is legacy — still read as `nous` when `true`, but never written anymore.)
`max_parallel_requests` defaults to `4`. Hermes clamps it to at least one and
to the global tool-worker limit, so image providers receive bounded parallel
requests without allowing an image batch to bypass the agent's concurrency cap.
### OpenRouter: the full Image API catalog
With `image_gen.provider: openrouter`, the model picker lists OpenRouter's
entire live image catalog — the dedicated
[Image API](https://openrouter.ai/docs/guides/overview/multimodal/image-generation)
models (Seedream, FLUX.2, Recraft, Qwen Image, MAI, Krea, Riverflow, Grok
Imagine, and more — 40+ ids) merged with the chat-completions image models.
The catalog is fetched live from `GET /images/models` and `GET /models`, so
new models appear in the picker as soon as OpenRouter serves them; no Hermes
update needed. Generation routes each model to the surface that serves it
(dedicated `POST /images/generations` vs chat-completions) automatically.
Nous Portal proxies the chat-completions protocol only, so its picker offers
the chat-served models.
Optional per-request knobs for Image API models go under the scoped config
section (or `OPENROUTER_IMAGE_API_*` env vars):
```yaml
image_gen:
provider: openrouter
model: bytedance-seed/seedream-4.5
openrouter:
resolution: 2K # model-dependent: 1K / 2K / 4K
quality: high # gpt-image models
output_format: png
```
### GPT-Image Quality
The `fal-ai/gpt-image-1.5` and `fal-ai/gpt-image-2` request quality is pinned to `medium` (~$0.034–$0.06/image at 1024×1024). We don't expose the `low` / `high` tiers as a user-facing option so that Nous Portal billing stays predictable across all users — the cost spread between tiers is 3–22×. If you want a cheaper option, pick Klein 9B or Z-Image Turbo; if you want higher quality, use Nano Banana Pro or Recraft V4 Pro.
### Meta Model API: Muse Image
With `image_gen.provider: meta-ai`, images are generated through the
[Meta Model API](https://api.meta.ai) (`https://api.meta.ai/v1`), the same
OpenAI-compatible endpoint that serves the Muse Spark chat models. It is the
image-gen companion to the bundled `meta-ai` chat provider.
| Model | Speed | Strengths | Price |
|---|---|---|---|
| `muse-image-1.0` *(default)* | ~10s | Meta Model API image generation | $0.01/image |
```yaml
image_gen:
provider: meta-ai
model: muse-image-1.0
```
Auth reuses the same env vars as the Meta chat provider — `MODEL_API_KEY`
(Meta's documented name), with `META_API_KEY` / `META_MODEL_API_KEY` accepted
as aliases. Set `META_BASE_URL` to point at a proxy or alternate host. Text-to-image
only for now; responses are saved to `$HERMES_HOME/cache/images/`.
## Usage
The agent-facing schema is intentionally minimal — the model picks up whatever you've configured:
```
Generate an image of a serene mountain landscape with cherry blossoms
```
```
Create a square portrait of a wise old owl — use the typography model
```
```
Make me a futuristic cityscape, landscape orientation
```
## Image-to-Image / Editing
The same `image_generate` tool also **edits existing images** when the active
model supports it — pass a source image and the backend routes to its editing
endpoint automatically (mirrors how `video_generate` handles image-to-video).
Omit the source image and it's plain text-to-image.
```
Take this photo and make it a rainy Tokyo street at night → <image>
```
```
Blend these two product shots into one hero image → <image1> <image2>
```
Two inputs drive the edit:
- **`image_url`** — the primary source image to edit/transform (public URL or local path).
- **`reference_image_urls`** — additional style/composition references (capped per-model).
### Which backends support editing
| Backend | Image-to-image | Reference cap | How |
|---|---|---|---|
| **FAL.ai** (edit-capable models below) | ✓ | up to 9 | routes to the model's `/edit` endpoint |
| **OpenAI** (`gpt-image-2`) | ✓ | up to 16 | `images.edit()` |
| **xAI** (Grok Imagine) | ✓ | 1 | `/v1/images/edits` (`grok-imagine-image-quality`) |
| **Krea** (`Krea 2`) | ✓ | up to 10 | reference-guided generation (`image_style_references`) |
| **OpenAI (Codex auth)** | ✓ | up to 16 | Codex Responses `image_generation` tool with `input_image` content parts |
| **OpenRouter** (Image API models) | ✓ | up to 14–16 (per model) | `input_references` on `POST /images/generations`; chat-served models use `image_url` content parts (up to 3) |
FAL models with an editing endpoint: `flux-2/klein/9b`, `flux-2-pro`,
`nano-banana-pro`, `gpt-image-1.5`, `gpt-image-2`, `ideogram/v3`, and
`qwen-image`. Pure text-to-image FAL models (`z-image/turbo`, `recraft`,
`krea/*`) reject image inputs with a clear error pointing you at an
edit-capable model.
:::note OpenAI (Codex auth) is best-effort
The Codex surface (`chatgpt.com/backend-api/codex`) hosts `image_generation`
as a tool the chat model may call, and Hermes cannot force the call — the
backend rejects every `tool_choice` shape for hosted tools, so the request
relies on instructions to steer the model. When the host model declines to
invoke the tool, the call fails with `empty_response`. Whether the hosted
image tool is reachable at all has also been reported to vary between
accounts. If you need image generation to work deterministically, configure
the **OpenAI** (API key), **FAL**, or **xAI** backend instead.
:::
The active model's editing capability is surfaced in the tool description at
runtime, so the agent knows whether `image_url` will be honored before it
calls the tool.
## Aspect Ratios
Every model accepts the same three aspect ratios from the agent's perspective. Internally, each model's native size spec is filled in automatically:
| Agent input | image_size (flux/z-image/qwen/recraft/ideogram) | aspect_ratio (nano-banana-pro) | image_size (gpt-image-1.5) | image_size (gpt-image-2) |
|---|---|---|---|---|
| `landscape` | `landscape_16_9` | `16:9` | `1536x1024` | `landscape_4_3` (1024×768) |
| `square` | `square_hd` | `1:1` | `1024x1024` | `square_hd` (1024×1024) |
| `portrait` | `portrait_16_9` | `9:16` | `1024x1536` | `portrait_4_3` (768×1024) |
GPT Image 2 maps to 4:3 presets rather than 16:9 because its minimum pixel count is 655,360 — the `landscape_16_9` preset (1024×576 = 589,824) would be rejected.
This translation happens in `_build_fal_payload()` — agent code never has to know about per-model schema differences.
## Upscaling
### Opt-in only
No model upscales by default. Modern image models emit their best quality
natively, and the available upscalers are *creative* enhancers (diffusion
passes) that can subtly redraw content — degrading rendered text, faces, and
fine detail. Upscaling only runs when the agent explicitly requests it.
### The `upscale` parameter (per-call opt-in)
- `upscale: true` — chain a high-resolution pass after generation:
| Backend | Upscaler |
|---|---|
| **FAL.ai** | Clarity Upscaler (2×, +$0.03/MP) |
| **Krea** | Krea Enhance (2×, up to 8K ceiling) |
| Other backends | no upscaler; native resolution returned |
- `upscale: false` / omitted — native resolution (the default)
`video_generate` also accepts `upscale: true` on the FAL backend, chaining
ByteDance's **SeedVR2** video upscaler (2×, $0.001/MP of output video) after
generation.
When the FAL image pass runs, it uses these settings:
| Setting | Value |
|---|---|
| Upscale factor | 2× |
| Creativity | 0.35 |
| Resemblance | 0.6 |
| Guidance scale | 4 |
| Inference steps | 18 |
If upscaling fails (network issue, rate limit), the original image is returned automatically. The response reports `upscaled: true/false` so the agent knows which resolution it got.
## How It Works Internally
1. **Model resolution** — `_resolve_fal_model()` reads `image_gen.model` from `config.yaml`, falls back to the `FAL_IMAGE_MODEL` env var, then to `fal-ai/flux-2/klein/9b`.
2. **Payload building** — `_build_fal_payload()` translates your `aspect_ratio` into the model's native format (preset enum, aspect-ratio enum, or GPT literal), merges the model's default params, applies any caller overrides, then filters to the model's `supports` whitelist so unsupported keys are never sent.
3. **Submission** — `_submit_fal_request()` routes via direct FAL credentials or the managed Nous gateway, according to the stored `image_gen.provider` selection.
4. **Upscaling** — runs only when the agent passed `upscale: true`; every model's catalog default is off.
5. **Delivery** — final image URL returned to the agent, which emits a `MEDIA:<url>` tag that platform adapters convert to native media.
## Debugging
Enable debug logging:
```bash
export IMAGE_TOOLS_DEBUG=true
```
Debug logs go to `./logs/image_tools_debug_<session_id>.json` with per-call details (model, parameters, timing, errors).
## Platform Delivery
| Platform | Delivery |
|---|---|
| **CLI** | Image URL printed as markdown `![](url)` — click to open |
| **Telegram** | Photo message with the prompt as caption |
| **Discord** | Embedded in a message |
| **Slack** | URL unfurled by Slack |
| **WhatsApp** | Media message |
| **Others** | URL in plain text |
## Limitations
- **Requires credentials** for the active backend (FAL `FAL_KEY` / Nous Subscription, `OPENAI_API_KEY`, xAI OAuth, `KREA_API_KEY`)
- **Editing is model-dependent** — image-to-image works only on edit-capable models (see the table above); text-to-image-only models reject image inputs with a clear error
- **Temporary URLs** — backends return hosted URLs that expire after hours/days; Hermes materializes them to the local cache so delivery still works after expiry
- **Per-model constraints** — some models don't support `seed`, `num_inference_steps`, etc. The `supports` / `edit_supports` filter silently drops unsupported params; this is expected behavior
website/docs/user-guide/features/kanban-tutorial.md
+310
View File
@@ -0,0 +1,310 @@
# Kanban tutorial
A walkthrough of the four use-cases the Hermes Kanban system was designed for, with the dashboard open in a browser. If you haven't read the [Kanban overview](./kanban) yet, start there — this assumes you know what a task, run, assignee, and dispatcher are.
## Setup
```bash
hermes kanban init # optional; first `hermes kanban <anything>` auto-inits
hermes dashboard # opens http://127.0.0.1:9119 in your browser
# click Kanban in the left nav
```
The dashboard is the most comfortable place for **you** to watch the system. Agent workers the dispatcher spawns never see the dashboard or the CLI — they drive the board through a dedicated `kanban_*` [toolset](./kanban#how-workers-interact-with-the-board) (`kanban_show`, `kanban_list`, `kanban_complete`, `kanban_block`, `kanban_heartbeat`, `kanban_comment`, `kanban_attach`, `kanban_attach_url`, `kanban_attachments`, `kanban_create`, `kanban_link`, `kanban_unblock`). All three surfaces — dashboard, CLI, worker tools — route through the same per-board SQLite DB (`~/.hermes/kanban.db` for the default board, `~/.hermes/kanban/boards/<slug>/kanban.db` for any board you create later), so each board is consistent no matter which side of the fence a change came from.
This tutorial uses the `default` board throughout. If you want multiple isolated queues (one per project / repo / domain), see [Boards (multi-project)](./kanban#boards-multi-project) in the overview — the same CLI / dashboard / worker flows apply per board, and workers physically cannot see tasks on other boards.
Throughout the tutorial, **code blocks labelled `bash` are commands *you* run.** Code blocks labelled `# worker tool calls` are what the spawned worker's model emits as tool calls — shown here so you can see the loop end-to-end, not because you'd ever run them yourself.
## The board at a glance
![Kanban board overview](/img/kanban-tutorial/01-board-overview.png)
Six columns, left to right:
- **Triage** — raw ideas. By default the dispatcher auto-runs the **decomposer** on tasks here: the built-in decomposer uses `auxiliary.kanban_decomposer`, reads your profile roster + descriptions, and produces a graph of child tasks routed to the best-fit specialists. The original task is held alive as the parent so its assignee (`kanban.orchestrator_profile`, or the active default profile when unset) wakes back up to judge completion when everything finishes. Flip the **Orchestration: Auto/Manual** pill at the top of the kanban page to switch modes. In Manual mode click **⚗ Decompose** on a card, or run `hermes kanban decompose <id>` / `/kanban decompose <id>`. For single tasks that don't need fan-out, **✨ Specify** does a one-shot spec rewrite (goal, approach, acceptance criteria) and promotes to `todo`. Configure the models under `auxiliary.kanban_decomposer` and `auxiliary.triage_specifier` in `config.yaml`. See [Auto vs Manual orchestration](./kanban#auto-vs-manual-orchestration) in the main Kanban guide.
- **Todo** — created but waiting on dependencies, or not yet assigned.
- **Ready** — assigned and waiting for the dispatcher to claim.
- **In progress** — a worker is actively running the task. With "Lanes by profile" on (the default), this column sub-groups by assignee so you can see at a glance what each worker is doing.
- **Blocked** — a worker asked for human input, or the circuit breaker tripped.
- **Done** — completed.
The top bar has filters for search, tenant, and assignee, plus a `Lanes by profile` toggle and a `Nudge dispatcher` button that runs one dispatch tick right now instead of waiting for the daemon's next interval. Clicking any card opens its drawer on the right.
### Flat view
If the profile lanes are noisy, toggle "Lanes by profile" off and the In Progress column collapses to a single flat list ordered by claim time:
![Board with lanes by profile off](/img/kanban-tutorial/02-board-flat.png)
## Story 1 — Solo dev shipping a feature
You're building a feature. Classic flow: design a schema, implement the API, write the tests. Three tasks with parent→child dependencies.
```bash
SCHEMA=$(hermes kanban create "Design auth schema" \
--assignee backend-dev --tenant auth-project --priority 2 \
--body "Design the user/session/token schema for the auth module." \
--json | jq -r .id)
API=$(hermes kanban create "Implement auth API endpoints" \
--assignee backend-dev --tenant auth-project --priority 2 \
--parent $SCHEMA \
--body "POST /register, POST /login, POST /refresh, POST /logout." \
--json | jq -r .id)
hermes kanban create "Write auth integration tests" \
--assignee qa-dev --tenant auth-project --priority 2 \
--parent $API \
--body "Cover happy path, wrong password, expired token, concurrent refresh."
```
Because `API` has `SCHEMA` as its parent, and `tests` has `API` as its parent, only `SCHEMA` starts in `ready`. The other two sit in `todo` until their parents complete. This is the dependency promotion engine doing its job — no other worker will pick up the test-writing until there's an API to test.
On the next dispatcher tick (60s by default, or immediately if you hit **Nudge dispatcher**) the `backend-dev` profile spawns as a worker with `HERMES_KANBAN_TASK=$SCHEMA` in its env. Here's what the worker's tool-call loop looks like from inside the agent:
```python
# worker tool calls — NOT commands you run
kanban_show()
# → returns title, body, worker_context, parents, prior attempts, comments
# (worker reads worker_context, uses terminal/file tools to design the schema,
# write migrations, run its own checks, commit — the real work happens here)
kanban_heartbeat(note="schema drafted, writing migrations now")
kanban_complete(
summary="users(id, email, pw_hash), sessions(id, user_id, jti, expires_at); "
"refresh tokens stored as sessions with type='refresh'",
metadata={
"changed_files": ["migrations/001_users.sql", "migrations/002_sessions.sql"],
"decisions": ["bcrypt for hashing", "JWT for session tokens",
"7-day refresh, 15-min access"],
},
)
```
`kanban_show` defaults `task_id` to `$HERMES_KANBAN_TASK`, so the worker doesn't need to know its own id. `kanban_complete` writes the summary + metadata onto the current `task_runs` row, closes that run, and transitions the task to `done` — all in one atomic hop through `kanban_db`.
When `SCHEMA` hits `done`, the dependency engine promotes `API` to `ready` automatically. The API worker, when it picks up, will call `kanban_show()` and see `SCHEMA`'s summary and metadata attached to the parent handoff — so it knows the schema decisions without re-reading a long design doc.
Click the completed schema task on the board and the drawer shows everything:
![Solo dev — completed schema task drawer](/img/kanban-tutorial/03-drawer-schema-task.png)
The Run History section at the bottom is the key addition. One attempt: outcome `completed`, worker `@backend-dev`, duration, timestamp, and the handoff summary in full. The metadata blob (`changed_files`, `decisions`) is stored on the run too and surfaced to any downstream worker that reads this parent.
You can inspect the same data from your terminal at any time — these commands are **you** peeking at the board, not the worker:
```bash
hermes kanban show $SCHEMA
hermes kanban runs $SCHEMA
# # OUTCOME PROFILE ELAPSED STARTED
# 1 completed backend-dev 0s 2026-04-27 19:34
# → users(id, email, pw_hash), sessions(id, user_id, jti, expires_at); refresh tokens ...
```
## Story 2 — Fleet farming
You have three workers (a translator, a transcriber, a copywriter) and a pile of independent tasks. You want all three pulling in parallel and making visible progress. This is the simplest kanban use-case and the one the original design optimized for.
Create the work:
```bash
for lang in Spanish French German; do
hermes kanban create "Translate homepage to $lang" \
--assignee translator --tenant content-ops
done
for i in 1 2 3 4 5; do
hermes kanban create "Transcribe Q3 customer call #$i" \
--assignee transcriber --tenant content-ops
done
for sku in 1001 1002 1003 1004; do
hermes kanban create "Generate product description: SKU-$sku" \
--assignee copywriter --tenant content-ops
done
```
Start the gateway and walk away — it hosts the embedded dispatcher
that picks up all three specialist profiles' tasks on the same
kanban.db:
```bash
hermes gateway start
```
Now filter the board to `content-ops` (or just search for "Transcribe") and you get this:
![Fleet view filtered to transcribe tasks](/img/kanban-tutorial/07-fleet-transcribes.png)
Two transcribes done, one running, two ready waiting for the next dispatcher tick. The In Progress column is grouped by profile (the "Lanes by profile" default) so you see each worker's active task without scanning a mixed list. The dispatcher will promote the next ready task to running as soon as the current one completes. With three daemons working on three assignee pools in parallel, the whole content queue drains without further human input.
**Everything Story 1 said about structured handoff still applies here.** A translator worker completing a call emits `kanban_complete(summary="translated 4 pages, style matched existing marketing voice", metadata={"duration_seconds": 720, "tokens_used": 2100})` — useful for analytics and for any downstream task that depends on this one.
## Story 3 — Role pipeline with retry
This is where Kanban earns its keep over a flat TODO list. A PM writes a spec. An engineer implements it. A reviewer rejects the first attempt. The engineer tries again with changes. The reviewer approves.
The dashboard view, filtered by `auth-project`:
![Pipeline view for a multi-role feature](/img/kanban-tutorial/08-pipeline-auth.png)
The screenshot uses the **pre-created downstream card** model: the implementation card has a dedicated reviewer child. In that model the engineer must call `kanban_complete` when implementation is ready so the reviewer child can leave `todo`. Never block the implementation parent merely to ask for review.
For workflows where the same card owns implementation and review, use the first-class review lifecycle instead. The full implement → review → changes → re-review choreography is:
```python
# --- Engineer: first implementation attempt ---
kanban_show()
# (write code, run tests, prepare the candidate)
kanban_request_review(
summary="implemented reset flow; candidate is ready for review",
metadata={"changed_files": ["auth/reset.py"], "tests_run": 8},
reviewer="reviewer",
)
# → the same card enters review; the implementation run closes as
# outcome='review_requested'
# --- Reviewer: request concrete changes ---
kanban_show()
# (inspect the handoff and candidate)
kanban_request_changes(
reason="Add password-strength validation and make reset tokens single-use."
)
# → the review run closes as outcome='changes_requested'; the card returns
# to backend-dev in ready/todo without touching block-loop accounting
# --- Engineer: second implementation attempt ---
kanban_show() # prior review evidence is in worker_context
# (apply feedback and re-run tests)
kanban_request_review(
summary="added zxcvbn validation and single-use reset tokens",
metadata={
"changed_files": [
"auth/reset.py",
"auth/tests/test_reset.py",
"migrations/003_single_use_reset_tokens.sql",
],
"tests_run": 11,
"review_iteration": 2,
},
reviewer="reviewer",
)
# --- Reviewer: approve ---
kanban_complete(summary="review passed; acceptance criteria verified")
# → done
```
The task's run history now records `review_requested → changes_requested → review_requested → completed`. Each attempt has its own actor, summary, metadata, and outcome, so the second engineer sees exactly what the reviewer rejected and the final approval remains auditable. `kanban_block` is reserved for a real external escalation (missing access, a product decision, unavailable infrastructure), not normal review feedback.
If you intentionally use the downstream-card model shown in the screenshot, the reviewer opens `Review password reset PR` after its implementation parent completes:
![Reviewer's drawer view of the pipeline](/img/kanban-tutorial/09-drawer-pipeline-review.png)
The reviewer card's `worker_context` includes the completed implementation handoff. That is a separate card workflow; do not combine it with same-card `kanban_request_review` or you will duplicate the review lane.
## Story 4 — Circuit breaker and crash recovery
Real workers fail. Missing credentials, OOM kills, transient network errors. The dispatcher has two lines of defense: a **circuit breaker** that auto-blocks after N consecutive failures so the board doesn't thrash forever, and **crash detection** that reclaims a task whose worker PID went away before its TTL expired.
### Circuit breaker — permanent-looking failure
A deploy task that can't spawn its worker because `AWS_ACCESS_KEY_ID` isn't set in the profile's environment:
```bash
hermes kanban create "Deploy to staging (missing creds)" \
--assignee deploy-bot --tenant ops \
--max-retries 3
```
The dispatcher tries to spawn the worker. Spawn fails (`RuntimeError: AWS_ACCESS_KEY_ID not set`). The dispatcher releases the claim, increments a failure counter, and tries again next tick. Because this example sets `--max-retries 3`, the circuit trips after three consecutive failures: the task goes to `blocked` with outcome `gave_up`. If you omit the flag, Hermes uses `kanban.failure_limit` (default: 2). No more retries until a human unblocks it.
Click the blocked task:
![Circuit breaker — 2 spawn_failed + 1 gave_up](/img/kanban-tutorial/11-drawer-gave-up.png)
Three runs, all with the same error on the `error` field. The first two are `spawn_failed` (retryable), the third is `gave_up` (terminal). The event log above shows the full sequence: `created → claimed → spawn_failed → claimed → spawn_failed → claimed → gave_up`.
On the terminal:
```bash
hermes kanban runs t_ef5d
# # OUTCOME PROFILE ELAPSED STARTED
# 1 spawn_failed deploy-bot 0s 2026-04-27 19:34
# ! AWS_ACCESS_KEY_ID not set in deploy-bot env
# 2 spawn_failed deploy-bot 0s 2026-04-27 19:34
# ! AWS_ACCESS_KEY_ID not set in deploy-bot env
# 3 gave_up deploy-bot 0s 2026-04-27 19:34
# ! AWS_ACCESS_KEY_ID not set in deploy-bot env
```
If Telegram / Discord / Slack is wired in, a gateway notification fires on the `gave_up` event so you hear about the outage without having to check the board.
### Crash recovery — worker dies mid-flight
Sometimes the spawn succeeds but the worker process dies later — segfault, OOM, `systemctl stop`. The dispatcher polls `kill(pid, 0)` and detects the dead pid; the claim releases, the task goes back to `ready`, and the next tick gives it to a fresh worker.
The example in the seed data is a migration that was running out of memory:
```bash
# Worker claims, starts scanning 2.4M rows, OOM kills it at ~2.3M
# Dispatcher detects dead pid, releases claim, increments attempt counter
# Retry with a chunked strategy succeeds
```
The drawer shows the full two-attempt history:
![Crash and recovery — 1 crashed + 1 completed](/img/kanban-tutorial/06-drawer-crash-recovery.png)
Run 1 — `crashed`, with the error `OOM kill at row 2.3M (process 99999 gone)`. Run 2 — `completed`, with `"strategy": "chunked with LIMIT + WHERE id > last_id"` in its metadata. The retrying worker saw the crash of run 1 in its context and picked a safer strategy; the metadata makes it obvious to a future observer (or postmortem writer) what changed.
## Structured handoff — why `summary` and `metadata` matter
In every story above, workers called `kanban_complete(summary=..., metadata=...)` at the end. That's not decoration — it's the primary handoff channel between stages of a workflow.
When a worker on task B is spawned and calls `kanban_show()`, the `worker_context` it gets back includes:
- B's **prior attempts** (previous runs: outcome, summary, error, metadata) so a retrying worker doesn't repeat a failed path.
- **Parent task results** — for each parent, the most-recent completed run's summary and metadata — so downstream workers see why and how the upstream work was done.
This replaces the "dig through comments and the work output" dance that plagues flat kanban systems. A PM writes acceptance criteria in the spec's metadata, and the engineer's worker sees them structurally in the parent handoff. An engineer records which tests they ran and how many passed, and the reviewer's worker has that list in hand before opening a diff.
The bulk-close guard exists because this data is per-run. `hermes kanban complete a b c --summary X` (you, from the CLI) is refused — copy-pasting the same summary to three tasks is almost always wrong. Bulk close without the handoff flags still works for the common "I finished a pile of admin tasks" case. The tool surface doesn't expose a bulk variant at all; `kanban_complete` is always single-task-at-a-time for the same reason.
## Follow-up on a done card — CI remediation via the parent link
Story 1's implementation card is `done`. Two hours later CI fails on the merged branch. Don't reopen the done card — completed cards are history, and their handoff flows forward. Create a remediation card with the done card as its **parent**:
```bash
hermes kanban create "Fix CI: test_backoff_jitter flakes on 3.11" \
--assignee backend-dev \
--parent t_impl \
--workspace worktree --branch wt/ci-fix-backoff \
--body "CI run #4812 failed after t_impl completed.
FAILED tests/test_retry.py::test_backoff_jitter - TimeoutError
Acceptance: tests/test_retry.py green on 3.11 and 3.12."
```
Three things make this work:
- **Immediate dispatch.** Because the parent is already `done`, the child is created straight into `ready` — the dispatcher can claim it on the next tick. (A child of a still-open parent would wait in `todo`.)
- **Inherited context.** The remediation worker's context includes a *Parent task results* section carrying `t_impl`'s completion summary and metadata — the changed files and decisions the original worker recorded — so it knows why the code is shaped the way it is before reading a line of it.
- **Fresh evidence in the body.** The CI log didn't exist when `t_impl` completed, so it can't be in the parent's handoff — it goes in the new card's body, alongside explicit acceptance criteria.
Prefer a fresh worktree/branch for the remediation card. Checking out the original branch gives the worker repo *state* but none of the *rationale* — the parent handoff carries that. Same assignee profile is usually right: the profile that wrote the code has the skills to fix it.
## Inspecting a task currently running
For completeness — here's the drawer of a task still in flight (the API implementation from Story 1, claimed by `backend-dev` but not yet complete):
![Claimed, in-flight task](/img/kanban-tutorial/10-drawer-in-flight.png)
Status is `Running`. The active run appears in the Run History section with outcome `active` and no `ended_at`. If this worker dies or times out, the dispatcher closes this run with the appropriate outcome and opens a new one on the next claim — the attempt row never disappears.
## Next steps
- [Kanban overview](./kanban) — the full data model, event vocabulary, and CLI reference.
- `hermes kanban --help` — every subcommand, every flag.
- `hermes kanban watch --kinds completed,gave_up,timed_out` — live stream terminal events across the whole board.
- `hermes kanban notify-subscribe <task> --platform telegram --chat-id <id>` — get a gateway ping when a specific task finishes.
website/docs/user-guide/features/kanban-worker-lanes.md
+117
View File
@@ -0,0 +1,117 @@
# Kanban worker lanes
A **worker lane** is a class of process that the kanban dispatcher can route tasks to. Each lane has an identity (the assignee string), a spawn mechanism, and a contract for what it must do with the task once spawned.
This page is the contract. It exists for two audiences:
- **Operators** picking which lanes to wire into a board (which profiles to create, which assignees to use).
- **Plugin / integration authors** wanting to add a new lane shape (a CLI worker that wraps Codex / Claude Code / OpenCode, a containerised review worker, a non-Hermes service that pulls tasks via the API).
If you're writing the worker code itself — the agent that runs *inside* a lane — the kanban lifecycle and reference details are injected into the worker's system prompt automatically (the `KANBAN_GUIDANCE` block in [`agent/prompt_builder.py`](https://github.com/NousResearch/hermes-agent/blob/main/agent/prompt_builder.py)).
## The hierarchy
```text
Hermes Kanban = canonical task lifecycle + audit trail
Worker lane = implementation executor for one assigned card
Reviewer = human or human-proxy that gates "done"
GitHub PR = upstreamable artifact (optional, for code lanes)
```
Hermes Kanban owns lifecycle truth — `ready` → `running` → `review` / `blocked` / `done` / `archived`. Worker lanes execute work but never own that truth; everything they do flows back through the kanban kernel via the `kanban_*` tools (or, for non-Hermes external workers, via the API). Reviewers gate the transition from "code change written" to "task done."
## What a lane provides
To be a kanban worker lane, an integration must provide three things:
### 1. An assignee string
The dispatcher matches `task.assignee` against either a Hermes profile name (the default lane shape) or a registered non-spawnable identifier (the plugin lane shape — see [Adding an external CLI worker lane](#adding-an-external-cli-worker-lane) below). Tasks whose assignee doesn't resolve are left on `ready` with a `skipped_nonspawnable` event so a board operator can fix them; they are not silently dropped or executed by an arbitrary fallback.
### 2. A spawn mechanism
For Hermes profile lanes, the dispatcher's `_default_spawn` runs `hermes -p <assignee> chat -q <prompt>` (or the equivalent module form when the `hermes` shim isn't on `$PATH`) inside the task's pinned workspace, with these env vars set:
| Variable | Carries |
|---|---|
| `HERMES_KANBAN_TASK` | the task id the worker is operating on |
| `HERMES_KANBAN_DB` | absolute path to the per-board SQLite file |
| `HERMES_KANBAN_BOARD` | board slug |
| `HERMES_KANBAN_WORKSPACES_ROOT` | root of the board's workspace tree |
| `HERMES_KANBAN_WORKSPACE` | absolute path to *this* task's workspace |
| `HERMES_KANBAN_RUN_ID` | the current run's id (for the lifecycle gate) |
| `HERMES_KANBAN_CLAIM_LOCK` | the claim lock string (`<host>:<pid>:<uuid>`) |
| `HERMES_PROFILE` | the worker's own profile name (for `kanban_comment` author attribution) |
| `HERMES_TENANT` | tenant namespace, if the task has one |
For non-Hermes lanes (registered via a plugin), the plugin supplies its own `spawn_fn` callable that gets `task`, `workspace`, and `board` and returns an optional pid for crash detection.
### 3. A lifecycle terminator
Every claim must end in exactly one of:
- `kanban_complete(summary=..., metadata=...)` — task succeeds, status flips to `done`.
- `kanban_request_review(summary=..., metadata=..., reviewer=...)` — same-card implementation is complete and enters first-class review; status flips to `review`. The dispatcher loads the bundled `sdlc-review` skill unless `kanban.review_dispatch` is disabled. A reviewer approves with `kanban_complete`, returns actionable rework with `kanban_request_changes`, or escalates a genuine external blocker with `kanban_block`.
- `kanban_block(reason=...)` — task waits for human input, status flips to `blocked`. The dispatcher respawns when `kanban_unblock` runs.
- The worker process exits without a tool call. The kernel reaps it and emits `crashed` (PID died) or `gave_up` (consecutive-failure breaker tripped) or `timed_out` (max_runtime exceeded). This is the failure path; healthy workers don't end here.
The kanban kernel enforces that exactly one of these terminates each run. A worker that calls neither and exits normally is treated as crashed.
## Outputs and the review handoff
For code-changing tasks, pick the review model encoded by the task graph:
- **Same-card review:** call `kanban_request_review(summary=..., metadata=..., reviewer=...)`. The task enters `review` without touching block recurrence accounting. The dispatcher claims it with the bundled `sdlc-review` skill by default. The reviewer approves with `kanban_complete`, calls `kanban_request_changes(reason=...)` to close the review run and route the task back to its original implementer, or blocks only for a genuine external escalation.
- **Pre-created downstream review/QA/release card:** `kanban_show` lists child IDs; inspect those cards with `kanban_show(task_id=...)` before choosing the terminal action. When a child is the downstream review/QA/release phase, call `kanban_complete` on the implementation phase. It cannot promote until this parent is `done`/`archived`. Do not additionally request same-card review and never sticky-block the parent with `review-required:` — either choice strands or duplicates the downstream lane.
- **Human-only boards:** set `kanban.review_dispatch: false`. A task can then remain in `review` until a human approves it or uses `reopen-review`/the dashboard to return it to `ready`/`todo`.
Both review models carry their structured handoff on the lifecycle transition itself. Do not place secrets, tokens, or raw PII in `summary` or `metadata`; run rows are durable.
The injected `KANBAN_GUIDANCE` covers both graph shapes, `kanban_complete`, the same-card review loop, and `kanban_block` for genuine blockers.
## Logs and audit trail
The dispatcher writes per-task worker stdout/stderr to `<board-root>/logs/<task_id>.log`. Logs are auditable from kanban metadata:
- `task_runs` rows carry the `log_path`, exit code (where available), summary, and metadata.
- `task_events` rows carry every state transition (`promoted`, `claimed`, `heartbeat`, `completed`, `blocked`, `review_requested`, `changes_requested`, `review_reopened`, `gave_up`, `crashed`, `timed_out`, `reclaimed`, `claim_extended`).
- `kanban_show` returns both, so a reviewer (or a follow-up worker) reading the task gets the full history without needing dashboard access.
The dashboard renders run history with summaries, metadata blocks, and exit-status badges. CLI users can run `hermes kanban tail <task_id>` to follow live, or `hermes kanban runs <task_id>` for the historical attempt list.
## Existing lane shapes
### Hermes profile lane (default)
The shape every kanban worker takes today: the assignee is a profile name, the dispatcher spawns `hermes -p <profile>`, the worker gets the `KANBAN_GUIDANCE` system-prompt block injected automatically, and uses the `kanban_*` tools to terminate the run. No setup beyond defining the profile.
When you create profiles for your fleet, choose names that match the *role* you want the orchestrator to route to. The orchestrator (when there is one) discovers your profile names via `hermes profile list` — there's no fixed roster the system assumes (the orchestrator side of the contract is part of the injected `KANBAN_GUIDANCE`).
### Orchestrator profile lane
A specialisation of the profile lane: an orchestrator is a Hermes profile whose toolset includes `kanban` but excludes `terminal` / `file` / `code` / `web` for implementation. Its job is decomposing a high-level goal into child tasks via `kanban_create` + `kanban_link` and stepping back. The orchestrator skill encodes the anti-temptation rules.
## Adding an external CLI worker lane
Wiring a non-Hermes CLI tool (Codex CLI, Claude Code CLI, OpenCode CLI, a local coding-model runner, etc.) as a kanban worker lane is *not yet a paved path*. The dispatcher's spawn function is pluggable (`spawn_fn` is a parameter on `dispatch_once`), and a plugin could register its own `spawn_fn` for a non-Hermes assignee, but the surrounding integration work — wrapping the CLI's exit code into `kanban_complete` / `kanban_block` calls, mapping the CLI's workspace/sandbox conventions onto the dispatcher's `HERMES_KANBAN_WORKSPACE` env, handling auth and per-CLI policy — is still per-integration design work.
If you're considering adding a CLI lane, open an issue describing the specific CLI and the workflow you're trying to enable. The contract above is the constraints any such lane must satisfy; the implementation shape (one plugin per CLI vs a generic CLI-runner plugin parameterised by config) is open.
The historical issue for this is [#19931](https://github.com/NousResearch/hermes-agent/issues/19931) and the closed-not-merged Codex-specific PR [#19924](https://github.com/NousResearch/hermes-agent/pull/19924) — those describe the original architecture proposal but didn't land a runner.
## Failure modes the dispatcher handles
So lane authors don't have to reimplement these:
- **Stale claim TTL** — a worker that claims and then never heartbeats / completes / blocks gets reclaimed after `DEFAULT_CLAIM_TTL_SECONDS` (15 min default) — but only if the worker process has actually died. A live worker (slow model spending 20+ min in one tool-free LLM call) gets the claim *extended* instead of killed; only a dead PID is reclaimed.
- **Crashed worker** — a worker whose host-local PID has vanished is detected by `detect_crashed_workers` and reaped; the task increments `consecutive_failures` and may auto-block when the breaker trips.
- **Run-level retry** — when a task is retried (post-block, post-crash, post-reclaim), the worker can use the `expected_run_id` parameter on terminating tools to fail fast if its own run was already superseded.
- **Per-task max runtime** — `task.max_runtime_seconds` hard-caps wall-clock time per run, regardless of PID liveness. Catches genuinely-deadlocked workers that the live-PID extension would otherwise keep running.
- **Stranded-task detection** — a ready task whose assignee never produces a claim within `kanban.stranded_threshold_seconds` (default 30 min) shows up in `hermes kanban diagnostics` as a `stranded_in_ready` warning. Severity escalates to error at 2x the threshold and critical at 6x. Catches typo'd assignees, deleted profiles, and down external worker pools in one signal — identity-agnostic, no per-board allowlist to curate.
- **Legacy review dependency deadlock** — a parent sticky-blocked with `review-required:` while one or more direct children remain dependency-gated in `todo` produces an immediate `review_dependency_deadlock` error. The diagnostic is read-only: it suggests completing the finished phase or unlinking the incorrect edge but never removes a user block automatically.
## Related
- [Kanban overview](./kanban) — the user-facing intro.
- [Kanban tutorial](./kanban-tutorial) — walkthrough with the dashboard open.
- [`KANBAN_GUIDANCE`](https://github.com/NousResearch/hermes-agent/blob/main/agent/prompt_builder.py) — the worker + orchestrator lifecycle injected into every kanban worker's system prompt.
website/docs/user-guide/features/kanban.md
+1171
View File
File diff suppressed because it is too large Load Diff
website/docs/user-guide/features/loops.md
+122
View File
@@ -0,0 +1,122 @@
---
sidebar_position: 17
title: "Recurring Loops"
description: "Re-run a prompt on a recurring interval inside your session — Hermes' take on Claude Code's /loop."
---
# Recurring Loops (`/loop`)
`/loop` re-runs a prompt (or a slash command) on a recurring cadence **inside your current session**. Each wakeup is a real agent turn: Hermes reads the current state fresh — the latest CI result, the newest queue depth, the file as it is now — does the work, reports back, and goes quiet until the next tick.
It's Hermes' take on **Claude Code's `/loop`** (and its `/proactive` alias, which works here too). Where [`/goal`](./goals.md) is judge-driven — "keep working until this objective is achieved" — `/loop` is timer-driven: "do this again every N minutes (or whenever it makes sense) until something says stop."
## When to use it
- **Polling external state.** "Watch the deploy / the CI run / the queue and tell me when it changes." The canonical use case.
- **Iterate-until-green.** "Run the tests, fix what fails, repeat until they pass."
- **Monitoring during a work session.** Keep an eye on error rates or a long job's progress while you do something else in the same conversation.
- **Periodic housekeeping.** Re-run a lint pass or a status summary every N minutes during a long session.
When the work should run **unattended** — overnight, on a real schedule, surviving restarts of your terminal — use a [cron job](./cron.md) instead. `/loop` lives inside a session; cron lives outside all of them. And when the task is a single objective with a definition of done, [`/goal`](./goals.md) is usually the better fit.
## Quick start
```
/loop 5m check the deploy status and tell me if it's live yet
```
What you'll see:
1. **Loop accepted** — `↻ Loop set (every 5m): check the deploy status…`
2. **First wakeup fires right away** — on the next idle poll (gateway: the next 15s watcher scan), Hermes injects the wakeup and runs a normal turn against current state.
3. **Repeat** — every 5 minutes after that, until a stop condition fires or you stop it.
Loop a slash command just as easily:
```
/loop 10m /recap
```
## The two cadence modes
**Fixed interval — you set the clock.** Give an interval (`30s`, `5m`, `2h`, `1h30m`) and the loop fires on that schedule. Use it when the thing you're watching changes on its own timeline:
```
/loop 2m poll the build at ci.example.com/job/42 and ping me the moment it finishes
```
**Self-paced — Hermes sets the clock.** Omit the interval and the loop paces itself: it starts at the floor (1 minute by default), and while the agent's replies stop changing it backs off exponentially — 2m, 4m, 8m, up to the ceiling (15 minutes by default). The moment a reply differs from the last one, cadence snaps back to the floor. Change detection is a local digest comparison (timestamps are ignored), so idle waits cost nothing extra:
```
/loop keep an eye on the migration and summarize progress
```
The rule of thumb: **fixed interval when an external clock drives the work; self-paced when the work drives the rhythm.**
## Stop conditions
A loop ends when any of these fires:
| Condition | How |
|---|---|
| The agent decides it's done | The wakeup prompt teaches the agent to end its reply with `LOOP_COMPLETE` on its own line when the task is finished or moot. |
| A run cap | `--times N` — stop after N wakeups. |
| An evidence-based condition | `--until <condition>` — after each wakeup, the same auxiliary judge that powers `/goal` checks the reply against your condition. If the judge rules the condition unachievable, the loop **pauses** with the reason instead of re-firing until the tick budget (fail-open: a broken judge never wedges the loop). |
| You | `/loop stop` (or `/loop pause` to keep it around). |
| The backstop budget | `loops.max_ticks` (default 100) pauses the loop so an unattended session can't burn tokens forever. `0` = unlimited. |
Examples:
```
/loop 2m poll CI --times 30
/loop 5m watch the queue --until queue depth reaches zero
```
## Commands
| Command | What it does |
|---|---|
| `/loop [interval] <prompt> [--times N] [--until <cond>]` | Start (or replace) the loop for this session. The first wakeup fires immediately; later ones follow the cadence. |
| `/loop` or `/loop status` | Show cadence, ticks fired, and time to the next wakeup. |
| `/loop pause` | Stop firing without losing the loop. |
| `/loop resume` | Pick it back up. |
| `/loop stop` | End the loop. |
| `/proactive …` | Alias for `/loop` (Claude Code parity). |
Works on the CLI, the TUI (`hermes --tui`), the web dashboard chat, the desktop app, and every gateway platform (Telegram, Discord, Slack, WhatsApp, …). On messaging platforms the gateway fires wakeups even between your messages — the loop belongs to the chat's session, and its results arrive as ordinary replies.
## Mixing with `/goal`
Both features inject synthetic turns at idle boundaries, so they follow one rule: **an active goal owns the session.** While a `/goal` is actively driving (judge saying "continue"), loop wakeups defer. The loop resumes using idle time as soon as the goal finishes, pauses, or parks itself on a wait barrier (`/goal wait`, or the judge's automatic WAIT verdict). A parked goal plus a `/loop` is a natural combo: the goal waits on the big async thing while the loop keeps a heartbeat on something else.
A real user message always wins over both — wakeups only fire while the session is idle and nothing of yours is queued.
## Behavior details
- **A wakeup is a normal user-role turn.** No system-prompt mutation, no toolset swap — prompt caching stays intact.
- **Survives `/resume` and compression.** Loop state persists per session and migrates across context-compression boundaries, same as `/goal`.
- **One loop per session.** Setting a new `/loop` replaces the old one. Run several loops by running several sessions (or use cron for a fleet of schedules).
- **Interrupting a wakeup turn (Ctrl+C) pauses the loop** — recoverable with `/loop resume`, so cancel actually means cancel.
- **Token cost scales with cadence.** Every tick is a full agent turn. Match the interval to how often the state actually changes; prefer self-pacing for idle waits.
## Configuration
```yaml
# ~/.hermes/config.yaml
loops:
min_interval_seconds: 30 # floor for fixed intervals
max_ticks: 100 # backstop budget (0 = unlimited)
self_paced_floor_seconds: 60 # self-paced starting cadence
self_paced_ceiling_seconds: 900 # self-paced max backoff
```
The `--until` judge routes through the `goal_judge` auxiliary task, so `auxiliary.goal_judge.*` overrides (provider, model, max_tokens) apply to loop conditions too.
## `/loop` vs `/goal` vs cron
| | `/loop` | `/goal` | cron |
|---|---|---|---|
| **Trigger** | Timer (or self-paced) | Judge verdict after each turn | Schedule, outside any session |
| **Lives in** | Your current session | Your current session | Its own session per run |
| **Ends when** | Stop condition / caps / you | Goal achieved / budget / you | You remove the job |
| **Best for** | Polling, monitoring, periodic re-runs | One objective, iterate until done | Unattended, long-horizon schedules |
website/docs/user-guide/features/lsp.md
+307
View File
@@ -0,0 +1,307 @@
---
sidebar_position: 16
title: "LSP — Semantic Diagnostics"
description: "Real language servers (pyright, gopls, rust-analyzer, …) wired into the post-write lint check used by write_file and patch."
---
# Language Server Protocol (LSP)
Hermes runs full language servers — pyright, gopls, rust-analyzer,
typescript-language-server, clangd, and ~20 more — as background
subprocesses and feeds their semantic diagnostics into the post-write
lint check used by `write_file` and `patch`. When the agent edits a
file, it sees exactly the errors that edit introduced — not just
syntax errors, but **type errors, undefined names, missing imports,
and project-wide semantic issues** the language server detects.
This is the same architecture top-tier coding agents use. Hermes
ships it self-contained: no editor host required, no plugins to
install, no separate daemon to manage.
## When LSP runs
LSP is gated on **git workspace detection**. When the agent's working
directory (or the file being edited) is inside a git repository, LSP
runs against that workspace. When neither is in a git repo, LSP
stays dormant — useful for messaging gateways where the cwd is the
user's home directory and there's no project to diagnose.
The check is layered: in-process syntax check first (microseconds),
then LSP diagnostics second when syntax is clean. A flaky or missing
language server can never break a write — every LSP failure path
falls back silently to the syntax-only result.
Concretely, on every successful `write_file` or `patch`:
1. Hermes captures a baseline of current diagnostics for the file.
2. Performs the write.
3. Re-queries the language server, filters out diagnostics that were
already in the baseline, and surfaces only the new ones.
The agent sees output like:
```
{
"bytes_written": 42,
"dirs_created": false,
"lint": {"status": "ok", "output": ""},
"lsp_diagnostics": "LSP diagnostics introduced by this edit:\n<diagnostics file=\"/path/to/foo.py\">\nERROR [42:5] Cannot find name 'foo' [reportUndefinedVariable] (Pyright)\nERROR [50:1] Argument of type \"str\" is not assignable to \"int\" [reportArgumentType] (Pyright)\n</diagnostics>"
}
```
The `lint` field carries the syntax-check result (microsecond
in-process parse via `ast.parse`, `json.loads`, etc.); the
`lsp_diagnostics` field carries the semantic diagnostics from the
real language server. Two channels, independent signals — the
agent sees a syntax-clean file with semantic problems as
``lint: ok`` plus a populated ``lsp_diagnostics``.
## Supported languages
| Language | Server | Auto-install |
|----------|--------|--------------|
| Python | `pyright-langserver` | npm |
| TypeScript / JavaScript / JSX / TSX | `typescript-language-server` | npm |
| Vue | `@vue/language-server` | npm |
| Svelte | `svelte-language-server` | npm |
| Astro | `@astrojs/language-server` | npm |
| Go | `gopls` | `go install` |
| Rust | `rust-analyzer` | manual (rustup) |
| C / C++ | `clangd` | manual (LLVM) |
| Bash / Zsh | `bash-language-server` | npm |
| YAML | `yaml-language-server` | npm |
| Lua | `lua-language-server` | manual (GitHub releases) |
| PHP | `intelephense` | npm |
| OCaml | `ocaml-lsp` | manual (opam) |
| Dockerfile | `dockerfile-language-server-nodejs` | npm |
| Terraform | `terraform-ls` | manual |
| Dart | `dart language-server` | manual (dart sdk) |
| Haskell | `haskell-language-server` | manual (ghcup) |
| Julia | `julia` + LanguageServer.jl | manual |
| Clojure | `clojure-lsp` | manual |
| Nix | `nixd` | manual |
| Zig | `zls` | manual |
| Gleam | `gleam lsp` | manual (gleam install) |
| Elixir | `elixir-ls` | manual |
| Prisma | `prisma language-server` | manual |
| Kotlin | `kotlin-language-server` | manual |
| Java | `jdtls` | manual |
| PowerShell | `PowerShellEditorServices` (`pwsh` host) | manual (release zip) |
For "manual" entries, install the server through whatever toolchain
manager makes sense for that language (rustup, ghcup, opam, brew,
…). Hermes auto-detects the binary on PATH or in
`<HERMES_HOME>/lsp/bin/`.
### PowerShell
PowerShellEditorServices isn't a single binary — it's a PowerShell
module bundle launched by a `pwsh` (PowerShell 7+) or `powershell`
host. Setup:
1. Install [PowerShell](https://github.com/PowerShell/PowerShell) so
`pwsh` (or Windows `powershell`) is on PATH.
2. Download the latest release zip from
[PowerShellEditorServices releases](https://github.com/PowerShell/PowerShellEditorServices/releases)
and extract it.
3. Point Hermes at the extracted bundle — the directory that contains
`PowerShellEditorServices/Start-EditorServices.ps1`. Either:
- set `lsp.servers.powershell.command: ["/path/to/bundle"]` in
`config.yaml`, or
- extract it to `<HERMES_HOME>/lsp/PowerShellEditorServices`, or
- export `PSES_BUNDLE_PATH=/path/to/bundle`.
`hermes lsp status` reports `installed` once `pwsh` is found; if the
bundle is missing you'll see a one-time warning in the logs with the
download link.
A few servers are installed alongside a peer dependency that npm
won't auto-pull. The current case is `typescript-language-server`,
which requires the `typescript` SDK importable from the same
`node_modules` tree — Hermes installs both packages together when you
run `hermes lsp install typescript` or auto-install fires on first
use.
## CLI
```
hermes lsp status # service state + per-server install status
hermes lsp list # registry, optionally --installed-only
hermes lsp install <id> # eagerly install one server
hermes lsp install-all # try every server with a known recipe
hermes lsp restart # tear down running clients
hermes lsp which <id> # print resolved binary path
```
`hermes lsp status` is the best starting point — it shows which
languages will get semantic diagnostics today and which need a
binary installed.
## Configuration
The defaults work for typical setups; nothing to set if the binaries
are on PATH.
```yaml
# config.yaml
lsp:
# Master toggle. Disabling skips the entire subsystem — no servers
# spawn, no background event loop runs.
enabled: true
# How long to wait for diagnostics after each write.
wait_mode: document # "document" or "full"
# Max seconds to wait for the server to re-check the file after an
# edit. Only *fresh* diagnostics (produced for the post-edit
# content) are ever reported; if the server doesn't finish within
# this budget, the edit reports "no LSP data" rather than stale
# errors from before the edit. Raise this for slow servers on big
# projects (tsserver, rust-analyzer mid-indexing).
wait_timeout: 5.0
# How to handle missing server binaries.
# auto — install via npm/pip/go install into <HERMES_HOME>/lsp/bin
# manual — only use binaries already on PATH
install_strategy: auto
# How long an unused language-server client stays alive (seconds).
# Idle servers are shut down automatically and respawned on the next
# relevant file operation. Set to 0 to disable idle reaping and keep
# servers alive for the life of the process. Values below 30s are
# clamped to 30 so a sweep can never reap a client mid-operation.
idle_timeout: 600
# Per-server overrides (all optional).
servers:
pyright:
disabled: false
command: ["/abs/path/to/pyright-langserver", "--stdio"]
env: { PYRIGHT_LOG_LEVEL: "info" }
initialization_options:
python:
analysis:
typeCheckingMode: "strict"
typescript:
disabled: true # skip TS even when its extensions match
```
### Per-server keys
* `disabled: true` — skip this server entirely even when its
extensions match a file.
* `command: [bin, ...args]` — pin a custom binary path. Bypasses
auto-install.
* `env: {KEY: value}` — extra env vars passed to the spawned process.
* `initialization_options: {...}` — merged into the LSP
`initializationOptions` payload sent in the `initialize`
handshake. Server-specific; consult the language server's docs.
## Installation locations
When `install_strategy: auto`, Hermes installs binaries into
`<HERMES_HOME>/lsp/bin/`. NPM packages land in
`<HERMES_HOME>/lsp/node_modules/` with bin symlinks one level up.
Go binaries come from `go install` with `GOBIN` pointed at the
staging dir.
Nothing is ever installed to `/usr/local/`, `~/.local/`, or any other
shared location — the staging dir is fully Hermes-owned and is
removed when you reset the profile.
## Performance characteristics
LSP servers are **lazy-spawned** on first use. Editing a Python file
in a project that's never seen `.py` traffic spawns pyright; the
spawn takes 1-3 seconds for most servers (rust-analyzer can take 10+
on a cold project). Subsequent edits in the same workspace re-use
the running server.
The LSP layer adds a few milliseconds to clean writes when no
diagnostics are emitted. When diagnostics are emitted, the wait
budget is `wait_timeout` seconds — typically the server responds in
tens of milliseconds for pyright/tsserver and a few seconds for
rust-analyzer mid-indexing.
Diagnostics are **freshness-gated**: a result only counts when the
server produced it for the content of the current edit (a
`publishDiagnostics` push at/after the change, or a pull request
answered after it). Slow servers that haven't re-checked yet result
in "no data" for that edit — never in yesterday's errors being
re-reported as current.
Servers are kept alive while they're being used and shut down after
`lsp.idle_timeout` seconds (default 600) with no file activity — a
long-running gateway that touches many worktrees no longer accumulates
one language-server process per workspace forever. A reaped server is
respawned automatically on the next relevant file operation. Set
`idle_timeout: 0` to disable reaping and hold every server's index warm
for the life of the process.
Servers that support multi-root workspaces (currently pyright) run as a
**single process** per Hermes process: the first Python project spawns
it, and every further project root — for example sibling git worktrees
edited by parallel subagents — is attached to that same server as an
additional workspace folder instead of starting another copy.
## Disabling
Set `lsp.enabled: false` in `config.yaml` to disable the entire
subsystem. The post-write check falls back to the in-process syntax
check (`ast.parse` for Python, `json.loads` for JSON, etc.) which
ships unchanged from earlier versions.
To disable a single language without disabling the whole layer:
```yaml
lsp:
servers:
rust-analyzer:
disabled: true
```
## Troubleshooting
**`hermes lsp status` shows a server as "missing"**
The binary isn't on PATH and isn't in `<HERMES_HOME>/lsp/bin/`. Run
`hermes lsp install <server_id>` to attempt an auto-install, or
install the binary manually through the language's normal toolchain.
**`Backend warnings` section in `hermes lsp status`**
Some servers ship as thin wrappers around an external CLI for actual
diagnostics — they spawn cleanly and accept requests but never emit
errors when the sidecar binary is missing. The most common case is
`bash-language-server`, which delegates diagnostics to `shellcheck`.
When `hermes lsp status` shows a `Backend warnings` section, install
the named tool through your OS package manager:
```
apt install shellcheck # Debian / Ubuntu
brew install shellcheck # macOS
scoop install shellcheck # Windows
```
The same warning is logged once at server spawn time in
`~/.hermes/logs/agent.log`.
**Server starts but never returns diagnostics**
Check `~/.hermes/logs/agent.log` for `[agent.lsp.client]` entries —
both stderr from the language server and protocol errors land
there. Some servers (rust-analyzer especially) need to finish a
project-wide index before they emit per-file diagnostics; the first
edit after server start may complete with no diagnostics, with
subsequent edits picking them up.
**Server crashed**
A crashed server is added to the broken-set and won't be retried for
the rest of the session. Run `hermes lsp restart` to clear the set;
the next edit re-spawns.
**Editing a file outside any git repo**
By design, LSP only runs inside a git repository. If the project isn't
yet initialized, run `git init` to enable LSP diagnostics. Otherwise the
in-process syntax-only fallback applies.
website/docs/user-guide/features/mcp.md
+936
View File
@@ -0,0 +1,936 @@
---
sidebar_position: 4
title: "MCP (Model Context Protocol)"
description: "Connect Hermes Agent to external tool servers via MCP — and control exactly which MCP tools Hermes loads"
---
# MCP (Model Context Protocol)
MCP lets Hermes Agent connect to external tool servers so the agent can use tools that live outside Hermes itself — GitHub, databases, file systems, browser stacks, internal APIs, and more.
If you have ever wanted Hermes to use a tool that already exists somewhere else, MCP is usually the cleanest way to do it.
:::tip Coming from Claude Code?
The `mcpServers` block in your `~/.claude.json` maps to `mcp_servers` in Hermes' `config.yaml` — and `hermes import-agent claude-code` migrates it (along with skills and instructions) automatically. See [Import from Other Agents](../import-from-other-agents.md).
:::
## What MCP gives you
- Access to external tool ecosystems without writing a native Hermes tool first
- Local stdio servers and remote HTTP MCP servers in the same config
- Automatic tool discovery and registration at startup
- Utility wrappers for MCP resources and prompts when supported by the server
- Per-server filtering so you can expose only the MCP tools you actually want Hermes to see
## Quick start
1. MCP support ships with the standard install — no extra step needed.
2. Add an MCP server to `~/.hermes/config.yaml`:
```yaml
mcp_servers:
filesystem:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/projects"]
```
3. Start Hermes:
```bash
hermes chat
```
4. Ask Hermes to use the MCP-backed capability.
For example:
```text
List the files in /home/user/projects and summarize the repo structure.
```
Hermes will discover the MCP server's tools and use them like any other tool.
## Catalog: one-click install for Nous-approved MCPs
Hermes ships a curated catalog of MCP servers that Nous staff has reviewed
and merged. They're disabled by default — install only what you actually
want.
```bash
hermes mcp # interactive picker (default)
hermes mcp catalog # plain-text list, scriptable
hermes mcp install n8n # install a catalog entry by name
```
The picker shows each entry with its current status:
```
n8n available Manage and inspect n8n workflows from Hermes
linear enabled Linear issue/project management (remote OAuth)
github installed (disabled) GitHub repo + PR tools
```
Hit `Enter` on a row to install (and walk through any required credentials),
enable, disable, or uninstall. Catalog entries are stored under
`optional-mcps/` in the hermes-agent repo — presence in that directory means
Nous approval. There is no community submission tier; entries are added by
merging a PR.
Catalog entries can require:
- **API key** — Hermes prompts at install time and writes the value to
`~/.hermes/.env`. Non-secret values (base URLs) go to the same file.
- **OAuth** (remote MCP) — written as `auth: oauth` in your config; the MCP
client opens a browser on first connection.
- **OAuth** (third-party provider like Google/GitHub) — Hermes points you at
`hermes auth <provider>` if you haven't authenticated already.
### Tool selection at install time
After credentials are configured, Hermes probes the MCP server to list every
tool it exposes and presents a checklist:
```
Select tools for 'linear' (SPACE toggle, ENTER confirm)
[x] find_issues Find issues matching a query
[x] get_issue Get a single issue
[x] create_issue Create a new issue
[ ] delete_workspace Delete a Linear workspace
...
```
The pre-checked rows come from:
1. **Your prior selection** if you've installed this entry before (reinstalls
preserve what you had — the manifest's defaults don't override it)
2. **The manifest's `tools.default_enabled`** if the entry declares one (some
catalog entries pre-prune mutating or rarely-useful tools)
3. **Everything** if neither applies
Some entries with very large auto-generated surfaces (e.g. `cloudflare`,
~3,300 OpenAPI endpoint tools) instead declare `tools.default_excluded` — a
curated block-list of names and glob patterns. Installing one of these skips
the checklist entirely and writes `tools.exclude`; everything not matched
stays enabled, including tools the server adds later. Edit
`mcp_servers.<name>.tools.exclude` in config.yaml to re-enable a family.
Submit the checklist with ENTER. Only the checked tools end up in
`mcp_servers.<name>.tools.include`. If you select everything, no filter is
written (cleanest config shape, identical behavior).
**If the probe fails** (server unreachable, OAuth not yet completed,
backing service not running), the install still succeeds: the manifest's
`tools.default_enabled` is applied directly (if declared), or no filter is
written (if not). Re-run `hermes mcp configure <name>` once the server is
reachable to refine.
### Trust model
Installing a catalog entry runs whatever the manifest specifies — `git clone`,
the entry's `bootstrap` commands (`pip install`, `npm install`, etc.), and
ultimately the MCP server's own code. Manifests are gated by PR review into
the hermes-agent repo, so Nous has reviewed each entry before it shipped —
**but you should still read the manifest before installing**, especially the
`source:` field's repository, the `install.bootstrap:` commands, and any
`transport.command:` invocation.
Manifests live at
[`optional-mcps/<name>/manifest.yaml`](https://github.com/NousResearch/hermes-agent/tree/main/optional-mcps)
on GitHub. The picker also prints the manifest's `source:` URL at install
time so you can quickly verify the upstream repo. The web dashboard's MCP
page surfaces the same detail per catalog entry — transport, auth type, the
endpoint URL (HTTP) or command + args (stdio), the git install source/ref and
bootstrap commands, and setup notes — with the `source:` rendered as a
clickable link, so you can inspect exactly what an entry connects to or runs
before clicking Install.
### Manifest version compatibility
Manifests pin a `manifest_version`. The catalog is forward-compatible: if a
PR adds an entry with a newer `manifest_version` than your installed Hermes
understands, the picker will surface a warning (`⚠ '<name>' requires a newer
Hermes`) for that entry instead of silently hiding it. Run `hermes update`
to install the latest Hermes when you see that.
### Runtime `${ENV_VAR}` substitution
Inside an entry's `transport.command`, `transport.args`, `transport.url`,
and `headers`, `${VAR}` placeholders are resolved at server-connect time
from environment variables (which include everything in `~/.hermes/.env`).
This is useful when a catalog entry wants to reference a value the user
configured elsewhere — e.g. `${HOME}/foo` or `${MY_PROVIDER_TOKEN}`.
Cursor-style context variables are also substituted (case-sensitive):
`${userHome}` (home directory), `${workspaceFolder}` (session workspace
root), `${workspaceFolderBasename}`, and `${pathSeparator}` / `${/}`
(the OS path separator). See the
[MCP config reference](/docs/reference/mcp-config-reference) for details.
Note this is distinct from `${INSTALL_DIR}` in catalog manifests, which is
substituted at install-time with the path the catalog cloned the entry's
repo into.
### Updating tool selection later
```bash
hermes mcp configure linear
```
Reopens the same checklist with your current selection pre-checked. Use this
when you want more tools enabled, or when the server has added new tools that
you want to opt into.
### Updating the catalog manifest
MCPs are never auto-updated. Re-run `hermes mcp install <name>` to refresh
after a Hermes update if a manifest version changed.
To add an MCP to the catalog, open a PR against
[`optional-mcps/`](https://github.com/NousResearch/hermes-agent/tree/main/optional-mcps).
### Suggestion metadata (`suggest:`)
A manifest may declare an optional `suggest:` block with `keywords:` and/or
`hosts:` lists. UI surfaces (currently the Desktop app's composer) use it to
offer a one-click "Add &lt;server&gt;" pill when your draft mentions one of the
keywords as a completed word, or contains a pasted link whose hostname ends
with one of the host suffixes. It is purely advisory — installs still flow
through the same validated catalog/config paths — and most hosted remote
entries (Atlassian, Sentry, Notion, Stripe, Vercel, Supabase, and friends)
declare it.
GitHub is deliberately **not** in the catalog: its hosted MCP requires each
client to bring its own OAuth app (generic dynamic client registration is
rejected), and Hermes's bundled `github/*` skills driving the `gh` CLI are a
more capable integration. On Desktop, GitHub mentions instead offer the
`github-auth` skill when `gh` isn't signed in yet.
## Two kinds of MCP servers
### Stdio servers
Stdio servers run as local subprocesses and talk over stdin/stdout.
```yaml
mcp_servers:
github:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-github"]
env:
GITHUB_PERSONAL_ACCESS_TOKEN: "***"
```
Use stdio servers when:
- the server is installed locally
- you want low-latency access to local resources
- you are following MCP server docs that show `command`, `args`, and `env`
### HTTP servers
HTTP MCP servers are remote endpoints Hermes connects to directly.
```yaml
mcp_servers:
remote_api:
url: "https://mcp.example.com/mcp"
headers:
Authorization: "Bearer ***"
```
Use HTTP servers when:
- the MCP server is hosted elsewhere
- your organization exposes internal MCP endpoints
- you do not want Hermes spawning a local subprocess for that integration
### OAuth-authenticated HTTP servers
Most hosted MCP servers (Cloudflare, Linear, Sentry, Atlassian, Asana, Figma, Stripe, …) require OAuth 2.1 instead of a static bearer token. Set `auth: oauth` and Hermes handles discovery, client identification, PKCE, token exchange, refresh, and step-up auth via the MCP Python SDK.
Hermes identifies itself with a [Client ID Metadata Document](../../reference/mcp-config-reference.md#client-identification-cimd-and-dcr) on servers that support one, and falls back to Dynamic Client Registration on those that don't. Both are automatic; there is nothing to configure.
:::tip Figma remote MCP
Figma's hosted endpoint (`https://mcp.figma.com/mcp`) allowlists Dynamic Client Registration by **exact `client_name`** — bare `"Hermes Agent"` 403s, while `"Claude Code"` and `"Codex"` succeed. Hermes auto-sets `oauth.client_name: "Claude Code"` for `mcp.figma.com` so install/login works without a special trick:
```yaml
mcp_servers:
figma:
url: "https://mcp.figma.com/mcp"
auth: oauth
```
Or: `hermes mcp install figma`, then `hermes mcp login figma`.
:::
```yaml
mcp_servers:
linear:
url: "https://mcp.linear.app/mcp"
auth: oauth
```
On first connect, Hermes prints an authorize URL, opens your browser when possible, and waits for the OAuth callback on a local loopback port. Tokens are cached at `~/.hermes/mcp-tokens/<server>.json` with 0o600 perms; subsequent runs reuse them silently until refresh fails.
**Remote / headless hosts.** When Hermes runs on a different machine than your browser, the loopback callback can't reach your laptop. Ways to complete the flow:
- **Hermes Desktop (automatic):** when you run the OAuth sign-in from the Desktop app's MCP setup UI against a remote backend, Desktop hosts the callback listener on *your* machine and relays the authorization back to the gateway automatically — no tunnel, paste, or proxy needed. Requires both the Desktop app and the backend to be up to date.
- **Paste-back (no setup):** on an interactive terminal Hermes prints "Or paste the redirect URL here…" alongside the authorize URL. Open the URL in your browser, approve, copy the full URL the browser ends up on (the redirect will show a connection error — that's expected), paste it at the prompt. Bare `?code=…&state=…` query strings work too.
- **SSH port forward:** `ssh -N -L <port>:127.0.0.1:<port> user@host` in a separate terminal, then let the redirect flow normally.
- **Proxied callback (`redirect_uri`):** when a public HTTPS endpoint forwards to the host (e.g. a Tailscale Funnel or reverse proxy pointed at the callback port), set `oauth.redirect_uri` and the browser redirect reaches Hermes on its own — no tunnel or paste needed:
```yaml
mcp_servers:
myserver:
url: "https://mcp.example.com/mcp"
auth: oauth
oauth:
redirect_port: 8765 # fixed port for the proxy to target
redirect_uri: "https://oauth.example.ts.net/callback"
```
For fully headless gateways (messaging bot, no interactive terminal at all), the optional [`mcp-oauth-remote-gateway` skill](../skills/optional/mcp/mcp-mcp-oauth-remote-gateway.md) walks the agent through completing the flow manually and writing tokens where Hermes expects them.
**Pitfall — WAF rejects `127.0.0.1` redirect URIs.** A few providers front their authorization server with a WAF that 403s any authorize request whose query string contains a literal `127.0.0.1` (Reclaim.ai's AWS API Gateway is a known example — every attempt returns `{"message":"Forbidden"}` before reaching the OAuth app). Set `oauth.redirect_host: localhost` to use `http://localhost:<port>/callback` instead; the callback listener still binds `127.0.0.1` either way.
See [OAuth over SSH / Remote Hosts](../../guides/oauth-over-ssh.md#mcp-servers) for the full walkthrough, including DCR-less servers (e.g. Slack), pre-registered `client_id`/`client_secret`, scope customization, and re-auth via `hermes mcp login <server>`.
**Pitfall — providers that don't support automatic registration (Google Drive, Atlassian).** Some servers reject the dynamic client registration step (RFC 7591) that bare `auth: oauth` relies on — Google's official Drive server (`https://drivemcp.googleapis.com/mcp/v1`) returns a `400 Bad Request`, so no OAuth client is created and no token is acquired. The symptom is subtle: these servers also serve `tools/list` *without* auth, so `hermes mcp login` can list the tools and look like it worked, but every real tool call later times out. `hermes mcp login` now detects this (it checks that a token actually landed on disk) and tells you to supply your own OAuth client. Create one in the provider's console and add it to config:
```yaml
mcp_servers:
googledrive:
url: "https://drivemcp.googleapis.com/mcp/v1"
auth: oauth
oauth:
client_id: "<your-oauth-client-id>"
client_secret: "<your-oauth-client-secret>"
```
Then run `hermes mcp login googledrive` — with the pre-registered client, Hermes skips registration and runs the normal browser authorization flow.
**Pitfall — config auto-reload race.** When you edit `~/.hermes/config.yaml` from inside a running Hermes session, the CLI auto-reloads MCP connections with a 30s timeout. That's not enough for an interactive OAuth flow. Add the entry, then run `hermes mcp login <server>` from a fresh terminal — it waits the full 5 minutes for you to complete auth.
## mTLS / client certificates
Remote HTTP MCP servers that require mutual TLS (client-certificate authentication) are supported via `client_cert` / `client_key`. Hermes passes the resolved certificate to the underlying HTTP client for the TLS handshake.
`client_cert` accepts three shapes:
- **A single combined PEM path** — one file holding both the certificate and the private key:
```yaml
mcp_servers:
internal_api:
url: "https://mcp.internal.example.com/mcp"
client_cert: "~/.certs/mcp-client.pem"
```
- **A `[cert, key]` 2-tuple** — certificate and key in separate files (equivalent to setting `client_cert` + `client_key`):
```yaml
mcp_servers:
internal_api:
url: "https://mcp.internal.example.com/mcp"
client_cert: ["~/.certs/mcp-client.crt", "~/.certs/mcp-client.key"]
```
- **A `[cert, key, password]` 3-tuple** — when the private key is encrypted, the third element is the key passphrase:
```yaml
mcp_servers:
internal_api:
url: "https://mcp.internal.example.com/mcp"
client_cert: ["~/.certs/mcp-client.crt", "~/.certs/mcp-client.key", "${MCP_KEY_PASSWORD}"]
```
You can also keep the cert and key fully separate via `client_cert` (combined PEM) plus an explicit `client_key`. Paths support `~` expansion; a missing file raises a clear, server-scoped error rather than an opaque TLS handshake failure.
## Per-user identity header
Remote HTTP/SSE MCP servers that key behavior on a caller identity (per-user rate limits, audit trails, multi-tenant routing) can be sent an identity header on every request via `identity_header`:
```yaml
mcp_servers:
team_api:
url: "https://mcp.team.example.com/mcp"
identity_header:
name: "X-User-Id"
value_from: "static" # "static" (default) or "profile"
value: "alice" # required for static
```
- `value_from: static` sends the literal `value` from config.yaml.
- `value_from: profile` sends the active Hermes profile name, resolved once at connect time — useful when multiple profiles on one machine talk to the same server and it needs to tell them apart.
An explicit entry in the server's `headers` mapping with the same name (any casing) always wins; the identity header never overrides your own header config. Invalid `identity_header` blocks are warned about and ignored — they never block the server from connecting. On stdio servers the key is ignored with a warning (stdio transports have no headers).
## Basic configuration reference
Hermes reads MCP config from `~/.hermes/config.yaml` under `mcp_servers`.
### Common keys
| Key | Type | Meaning |
|---|---|---|
| `command` | string | Executable for a stdio MCP server |
| `args` | list | Arguments for the stdio server |
| `env` | mapping | Environment variables passed to the stdio server |
| `url` | string | HTTP MCP endpoint |
| `headers` | mapping | HTTP headers for remote servers |
| `client_cert` | string \| list | Client certificate for mTLS — a combined PEM path, or `[cert, key]` / `[cert, key, password]` |
| `client_key` | string | Client private-key PEM path (when separate from `client_cert`) |
| `identity_header` | mapping | Optional per-user identity header for HTTP/SSE servers — `{name, value_from: static\|profile, value}` |
| `timeout` | number | Tool call timeout |
| `connect_timeout` | number | Initial connection timeout (also bounds the MCP `initialize` handshake) |
| `idle_timeout_seconds` | number | Recycle a stdio server after this many seconds without a tool call (`0` = never, default). The server restarts transparently on the next tool call. |
| `max_lifetime_seconds` | number | Recycle a stdio server after this total age (`0` = never, default). Restarts transparently on next use. |
| `enabled` | bool | If `false`, Hermes skips the server entirely |
| `supports_parallel_tool_calls` | bool | If `true`, tools from this server may run concurrently |
| `tools` | mapping | Per-server tool filtering and utility policy |
### Minimal stdio example
```yaml
mcp_servers:
filesystem:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
```
### Recycling memory-heavy stdio servers
Browser-based MCP servers (e.g. `@playwright/mcp`) keep a full Chromium
resident after their first tool call — hundreds of MB that never get
released. Opt in to automatic recycling and the server is torn down after
the idle/lifetime limit, then restarted transparently the next time one of
its tools is called (its tools stay registered the whole time):
```yaml
mcp_servers:
playwright:
command: "npx"
args: ["-y", "@playwright/mcp@latest", "--headless"]
idle_timeout_seconds: 900 # recycle after 15 min without a tool call
max_lifetime_seconds: 86400 # and at least once a day regardless
```
### Minimal HTTP example
```yaml
mcp_servers:
company_api:
url: "https://mcp.internal.example.com"
headers:
Authorization: "Bearer ***"
```
## Built-in presets
For well-known MCP servers, `hermes mcp add` accepts a `--preset` flag that fills in the transport details so you don't have to look up the command and args. The preset only supplies defaults — anything else (env vars, headers, filtering) you pass on the same command line still wins.
| Preset | What it wires up |
|---|---|
| `codex` | The Codex CLI's MCP server (`codex mcp-server` over stdio). Requires the `codex` CLI on PATH. |
```bash
# Add Codex CLI as an MCP server in one line
hermes mcp add codex --preset codex
```
That writes the equivalent of:
```yaml
mcp_servers:
codex:
command: "codex"
args: ["mcp-server"]
```
You can pick any local name (`hermes mcp add my-codex --preset codex` is fine); the preset only provides the `command`/`args` defaults.
## How Hermes registers MCP tools
Hermes prefixes MCP tools so they do not collide with built-in names:
```text
mcp_<server_name>_<tool_name>
```
Examples:
| Server | MCP tool | Registered name |
|---|---|---|
| `filesystem` | `read_file` | `mcp_filesystem_read_file` |
| `github` | `create-issue` | `mcp_github_create_issue` |
| `my-api` | `query.data` | `mcp_my_api_query_data` |
In practice, you usually do not need to call the prefixed name manually — Hermes sees the tool and chooses it during normal reasoning.
### Tool-result sanitization and `_meta`
Two behaviors apply to every MCP tool result before the model sees it:
- **Invisible Unicode TAG characters are stripped.** Characters in the U+E0000–U+E007F range render as nothing in terminals and chat UIs but are fully visible to the model — a classic prompt-injection smuggling channel for a malicious or compromised server. Hermes strips them from tool results, resource content, and tool descriptions. Legitimate emoji tag sequences (regional flags like 🏴󠁧󠁢󠁳󠁣󠁴󠁿) are preserved.
- **Vendor `_meta` is surfaced; protocol-reserved keys are not.** When a server attaches a `_meta` mapping to a tool result (vendor namespaces like `com.example/handoff`), Hermes passes it through to the model alongside the result content. Keys under protocol-reserved prefixes — a `modelcontextprotocol` or `mcp` label followed by another label, e.g. `modelcontextprotocol.io/...` or `tools.mcp.com/...` — are dropped, matching the MCP spec's key-name rules. If nothing model-facing remains, the `_meta` field is omitted entirely.
## MCP utility tools
When supported, Hermes also registers utility tools around MCP resources and prompts:
- `list_resources`
- `read_resource`
- `list_prompts`
- `get_prompt`
These are registered per server with the same prefix pattern, for example:
- `mcp_github_list_resources`
- `mcp_github_get_prompt`
### Important
These utility tools are now capability-aware:
- Hermes only registers resource utilities if the MCP session actually supports resource operations
- Hermes only registers prompt utilities if the MCP session actually supports prompt operations
So a server that exposes callable tools but no resources/prompts will not get those extra wrappers.
## Per-server filtering
You can control which tools each MCP server contributes to Hermes, allowing fine-grained management of your tool namespace.
### Disable a server entirely
```yaml
mcp_servers:
legacy:
url: "https://mcp.legacy.internal"
enabled: false
```
If `enabled: false`, Hermes skips the server completely and does not even attempt a connection.
### Whitelist server tools
```yaml
mcp_servers:
github:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-github"]
env:
GITHUB_PERSONAL_ACCESS_TOKEN: "***"
tools:
include: [create_issue, list_issues]
```
Only those MCP server tools are registered.
Entries in `include`/`exclude` may also be glob patterns (`*`, `?`, `[...]`,
matched case-sensitively): `include: ["*_dns_*"]` registers every tool whose
name contains `_dns_`. Plain entries without metacharacters stay exact-match.
Globs are the practical way to filter servers that expose thousands of
auto-generated endpoint tools by product family.
### Blacklist server tools
```yaml
mcp_servers:
stripe:
url: "https://mcp.stripe.com"
tools:
exclude: [delete_customer]
```
All server tools are registered except the excluded ones.
### Glob patterns
Both lists accept fnmatch-style globs alongside exact names — essential for
huge flat surfaces like Cloudflare's API MCP (`?codemode=false`, ~3,300
tools) where excluding product areas one endpoint at a time is impractical:
```yaml
mcp_servers:
cloudflare:
url: "https://mcp.cloudflare.com/mcp?codemode=false"
auth: oauth
tools:
exclude: ["*_radar_*", "*_accounts_dlp_*", "*_zones_web3_*"]
```
Entries without glob metacharacters (`*`, `?`, `[`) match exactly — `docs`
excludes only the tool named `docs`, never `docs_search`.
### Precedence rule
If both are present:
```yaml
tools:
include: [create_issue]
exclude: [create_issue, delete_issue]
```
`include` wins.
### Filter utility tools too
You can also separately disable Hermes-added utility wrappers:
```yaml
mcp_servers:
docs:
url: "https://mcp.docs.example.com"
tools:
prompts: false
resources: false
```
That means:
- `tools.resources: false` disables `list_resources` and `read_resource`
- `tools.prompts: false` disables `list_prompts` and `get_prompt`
### Full example
```yaml
mcp_servers:
github:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-github"]
env:
GITHUB_PERSONAL_ACCESS_TOKEN: "***"
tools:
include: [create_issue, list_issues, search_code]
prompts: false
stripe:
url: "https://mcp.stripe.com"
headers:
Authorization: "Bearer ***"
tools:
exclude: [delete_customer]
resources: false
legacy:
url: "https://mcp.legacy.internal"
enabled: false
```
## What happens if everything is filtered out?
If your config filters out all callable tools and disables or omits all supported utilities, Hermes does not create an empty runtime MCP toolset for that server.
That keeps the tool list clean.
## Runtime behavior
### Discovery time
Hermes discovers MCP servers at startup and registers their tools into the normal tool registry.
### Dynamic Tool Discovery
MCP servers can notify Hermes when their available tools change at runtime by sending a `notifications/tools/list_changed` notification. When Hermes receives this notification, it automatically re-fetches the server's tool list and updates the registry — no manual `/reload-mcp` required.
This is useful for MCP servers whose capabilities change dynamically (e.g. a server that adds tools when a new database schema is loaded, or removes tools when a service goes offline).
The refresh is lock-protected so rapid-fire notifications from the same server don't cause overlapping refreshes. Prompt and resource change notifications (`prompts/list_changed`, `resources/list_changed`) are received but not yet acted on.
### Reloading
If you change MCP config, use:
```text
/reload-mcp
```
This reloads MCP servers from config and refreshes the available tool list. It is also the explicit way to re-probe availability-gated tools (Docker, `HASS_TOKEN`, OAuth…): a session's tool set is otherwise frozen, so a credential or daemon that appears mid-session is only picked up on `/reload-mcp`, `/new`, or context compaction. For runtime tool changes pushed by the server itself, see [Dynamic Tool Discovery](#dynamic-tool-discovery) above.
### Toolsets
Each configured MCP server also creates a runtime toolset when it contributes at least one registered tool:
```text
mcp-<server>
```
That makes MCP servers easier to reason about at the toolset level.
## Security model
### Stdio env filtering
For stdio servers, Hermes does not blindly pass your full shell environment.
Only explicitly configured `env` plus a safe baseline are passed through. This reduces accidental secret leakage.
### Config-level exposure control
The new filtering support is also a security control:
- disable dangerous tools you do not want the model to see
- expose only a minimal whitelist for a sensitive server
- disable resource/prompt wrappers when you do not want that surface exposed
## Example use cases
### GitHub server with a minimal issue-management surface
```yaml
mcp_servers:
github:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-github"]
env:
GITHUB_PERSONAL_ACCESS_TOKEN: "***"
tools:
include: [list_issues, create_issue, update_issue]
prompts: false
resources: false
```
Use it like:
```text
Show me open issues labeled bug, then draft a new issue for the flaky MCP reconnection behavior.
```
### Stripe server with dangerous actions removed
```yaml
mcp_servers:
stripe:
url: "https://mcp.stripe.com"
headers:
Authorization: "Bearer ***"
tools:
exclude: [delete_customer, refund_payment]
```
Use it like:
```text
Look up the last 10 failed payments and summarize common failure reasons.
```
### Filesystem server for a single project root
```yaml
mcp_servers:
project_fs:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/my-project"]
```
Use it like:
```text
Inspect the project root and explain the directory layout.
```
## Troubleshooting
### MCP server not connecting
Check:
```bash
# Verify MCP deps are installed (already included in standard install)
cd ~/.hermes/hermes-agent && uv pip install -e ".[mcp]"
node --version
npx --version
```
Then verify your config and restart Hermes.
### Tools not appearing
Possible causes:
- the server failed to connect
- discovery failed
- your filter config excluded the tools
- the utility capability does not exist on that server
- the server is disabled with `enabled: false`
If you are intentionally filtering, this is expected.
### Why didn't resource or prompt utilities appear?
Because Hermes now only registers those wrappers when both are true:
1. your config allows them
2. the server session actually supports the capability
This is intentional and keeps the tool list honest.
## Parallel Tool Calls
By default, MCP tools run sequentially — one at a time. If your MCP server exposes tools that are safe to run concurrently (e.g. read-only queries, independent API calls), you can opt-in to parallel execution:
```yaml
mcp_servers:
docs:
command: "docs-server"
supports_parallel_tool_calls: true
```
When `supports_parallel_tool_calls` is `true`, Hermes may execute multiple tools from that server at the same time within a single tool-call batch, just like it does for built-in read-only tools (web_search, read_file, etc.).
:::caution
Only enable parallel calls for MCP servers whose tools are safe to run at the same time. If tools read and write shared state, files, databases, or external resources, review the read/write race conditions before enabling this setting.
:::
## MCP Sampling Support
MCP servers can request LLM inference from Hermes via the `sampling/createMessage` protocol. This allows an MCP server to ask Hermes to generate text on its behalf — useful for servers that need LLM capabilities but don't have their own model access.
Sampling is **enabled by default** for all MCP servers (when the MCP SDK supports it). Configure it per-server under the `sampling` key:
```yaml
mcp_servers:
my_server:
command: "my-mcp-server"
sampling:
enabled: true # Enable sampling (default: true)
model: "openai/gpt-4o" # Override model for sampling requests (optional)
max_tokens_cap: 4096 # Max tokens per sampling response (default: 4096)
timeout: 30 # Timeout in seconds per request (default: 30)
max_rpm: 10 # Rate limit: max requests per minute (default: 10)
max_tool_rounds: 5 # Max tool-use rounds in sampling loops (default: 5)
allowed_models: [] # Allowlist of model names the server may request (empty = any)
log_level: "info" # Audit log level: debug, info, or warning (default: info)
```
The sampling handler includes a sliding-window rate limiter, per-request timeouts, and tool-loop depth limits to prevent runaway usage. Metrics (request count, errors, tokens used) are tracked per server instance.
To disable sampling for a specific server:
```yaml
mcp_servers:
untrusted_server:
url: "https://mcp.example.com"
sampling:
enabled: false
```
## MCP Elicitation Support
MCP servers can ask the user for structured input mid-tool-call via the `elicitation/create` protocol (mcp Python SDK ≥ 1.11.0). Hermes routes **form-mode** elicitations through its existing approval surface — an interactive prompt in the CLI/TUI, or approval buttons on gateway platforms like Telegram and Slack — so the request reaches you wherever the session lives. **URL-mode** elicitations (where a server points you at an external URL) are declined as unsupported.
Elicitation is **enabled by default** per server. Configure it under the `elicitation` key:
```yaml
mcp_servers:
my_server:
command: "my-mcp-server"
elicitation:
enabled: true # default: true
timeout: 300 # seconds to wait for your answer (default: 300)
```
The 5-minute default timeout mirrors the gateway approval default so users on async surfaces have time to respond before the server gives up. Per-server metrics (requests, accepted, declined, errors) are tracked on the handler.
## Running Hermes as an MCP server
In addition to connecting **to** MCP servers, Hermes can also **be** an MCP server. This lets other MCP-capable agents (Claude Code, Cursor, Codex, or any MCP client) use Hermes's messaging capabilities — list conversations, read message history, and send messages across all your connected platforms.
### When to use this
- You want Claude Code, Cursor, or another coding agent to send and read Telegram/Discord/Slack messages through Hermes
- You want a single MCP server that bridges to all of Hermes's connected messaging platforms at once
- You already have a running Hermes gateway with connected platforms
### Quick start
```bash
hermes mcp serve
```
This starts a stdio MCP server. The MCP client (not you) manages the process lifecycle.
### MCP client configuration
Add Hermes to your MCP client config. For example, in Claude Code's `~/.claude/claude_desktop_config.json`:
```json
{
"mcpServers": {
"hermes": {
"command": "hermes",
"args": ["mcp", "serve"]
}
}
}
```
Or if you installed Hermes in a specific location:
```json
{
"mcpServers": {
"hermes": {
"command": "/home/user/.hermes/hermes-agent/venv/bin/hermes",
"args": ["mcp", "serve"]
}
}
}
```
### Available tools
The MCP server exposes 10 tools, matching OpenClaw's channel bridge surface plus a Hermes-specific channel browser:
| Tool | Description |
|------|-------------|
| `conversations_list` | List active messaging conversations. Filter by platform or search by name. |
| `conversation_get` | Get detailed info about one conversation by session key. |
| `messages_read` | Read recent message history for a conversation. |
| `attachments_fetch` | Extract non-text attachments (images, media) from a specific message. |
| `events_poll` | Poll for new conversation events since a cursor position. |
| `events_wait` | Long-poll / block until the next event arrives (near-real-time). |
| `messages_send` | Send a message through a platform (e.g. `telegram:123456`, `discord:#general`). |
| `channels_list` | List available messaging targets across all platforms. |
| `permissions_list_open` | List pending approval requests observed during this bridge session. |
| `permissions_respond` | Allow or deny a pending approval request. |
### Event system
The MCP server includes a live event bridge that polls Hermes's session database for new messages. This gives MCP clients near-real-time awareness of incoming conversations:
```
# Poll for new events (non-blocking)
events_poll(after_cursor=0)
# Wait for next event (blocks up to timeout)
events_wait(after_cursor=42, timeout_ms=30000)
```
Event types: `message`, `approval_requested`, `approval_resolved`
The event queue is in-memory and starts when the bridge connects. Older messages are available through `messages_read`.
### Options
```bash
hermes mcp serve # Normal mode
hermes mcp serve --verbose # Debug logging on stderr
```
### How it works
The MCP server reads conversation data directly from Hermes's session store — `~/.hermes/state.db` is the primary source, with `sessions.json` kept only as a legacy fallback. A background thread polls the database for new messages and maintains an in-memory event queue. For sending messages, it uses the same internal send engine (`tools/send_message_tool.py`) that powers cron delivery and the `hermes send` CLI.
The gateway does NOT need to be running for read operations (listing conversations, reading history, polling events). It DOES need to be running for send operations, since the platform adapters need active connections.
### Current limits
- The embedded `hermes mcp serve` exposes a **stdio-only** MCP server today. If you need an HTTP MCP server, run a separate adapter — or, much more commonly, use the MCP **client** side of Hermes, which already speaks both stdio and HTTP (`url` + `headers` in `mcp_servers.yaml` / `config.yaml`; see [HTTP servers](#http-servers) above).
- Event polling at ~200ms intervals via mtime-optimized DB polling (skips work when files are unchanged)
- No `claude/channel` push notification protocol yet
- Text-only sends (no media/attachment sending through `messages_send`)
## Related docs
- [Use MCP with Hermes](/guides/use-mcp-with-hermes)
- [CLI Commands](/reference/cli-commands)
- [Slash Commands](/reference/slash-commands)
- [FAQ](/reference/faq)
website/docs/user-guide/features/memory-providers.md
+698
View File
@@ -0,0 +1,698 @@
---
sidebar_position: 4
title: "Memory Providers"
description: "External memory provider plugins — Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, Supermemory"
---
# Memory Providers
Hermes Agent ships with 8 external memory provider plugins that give the agent persistent, cross-session knowledge beyond the built-in MEMORY.md and USER.md. Only **one** external provider can be active at a time — the built-in memory is always active alongside it.
## Quick Start
```bash
hermes memory setup # interactive picker + configuration
hermes memory status # check what's active
hermes memory off # disable external provider
```
You can also select the active memory provider via `hermes plugins` → Provider Plugins → Memory Provider.
Or set manually in `~/.hermes/config.yaml`:
```yaml
memory:
provider: openviking # or honcho, mem0, hindsight, holographic, retaindb, byterover, supermemory
```
## How It Works
When a memory provider is active, Hermes automatically:
1. **Injects provider context** into the system prompt (what the provider knows)
2. **Prefetches relevant memories** before each turn (background, non-blocking)
3. **Syncs conversation turns** to the provider after each response
4. **Extracts memories on session end** (for providers that support it)
5. **Mirrors built-in memory writes** to the external provider
6. **Adds provider-specific tools** so the agent can search, store, and manage memories
The built-in memory (MEMORY.md / USER.md) continues to work exactly as before. The external provider is additive.
## Available Providers
### Honcho
AI-native cross-session user modeling with dialectic reasoning, session-scoped context injection, semantic search, and persistent conclusions. Base context now includes the session summary alongside user representation and peer cards, giving the agent awareness of what has already been discussed.
| | |
|---|---|
| **Best for** | Multi-agent systems with cross-session context, user-agent alignment |
| **Requires** | `pip install honcho-ai` + [API key](https://app.honcho.dev) or self-hosted instance |
| **Data storage** | Honcho Cloud or self-hosted |
| **Cost** | Honcho pricing (cloud) / free (self-hosted) |
**Tools (5):** `honcho_profile` (read/update peer card), `honcho_search` (semantic search), `honcho_context` (session context — summary, representation, card, messages), `honcho_reasoning` (LLM-synthesized), `honcho_conclude` (create/delete conclusions)
**Architecture:** Two-layer context injection — a base layer (session summary + representation + peer card, refreshed on `contextCadence`) plus a dialectic supplement (LLM reasoning, refreshed on `dialecticCadence`). The dialectic automatically selects cold-start prompts (general user facts) vs. warm prompts (session-scoped context) based on whether base context exists.
**Three orthogonal config knobs** control cost and depth independently:
- `contextCadence` — how often the base layer refreshes (API call frequency)
- `dialecticCadence` — how often the dialectic LLM fires (LLM call frequency)
- `dialecticDepth` — how many `.chat()` passes per dialectic invocation (1–3, depth of reasoning)
The auto-injected dialectic also scales its reasoning level by query length (longer query → deeper reasoning, capped at `reasoningLevelCap`); see [Query-Adaptive Reasoning Level](./honcho.md#query-adaptive-reasoning-level).
**Setup Wizard:**
```bash
hermes memory setup # select "honcho" — runs the Honcho-specific post-setup
```
The legacy `hermes honcho setup` command still works (it now redirects to `hermes memory setup`), but is only registered after Honcho is selected as the active memory provider.
**Headless / remote machines:** for cloud auth on a box without a browser (SSH, remote VM), pick **device** at the wizard's auth-method prompt. The CLI prints a short code and a verification link; open the link in a browser on any other machine, approve, and setup completes — no API key copy-paste. The wizard defaults to this option automatically when it detects no usable local browser.
**Config:** `$HERMES_HOME/honcho.json` (profile-local) or `~/.honcho/config.json` (global). Resolution order: `$HERMES_HOME/honcho.json` > `~/.hermes/honcho.json` > `~/.honcho/config.json`. See the [config reference](https://github.com/NousResearch/hermes-agent/blob/main/plugins/memory/honcho/README.md) and the [Honcho integration guide](https://docs.honcho.dev/v3/guides/integrations/hermes).
<details>
<summary>Full config reference</summary>
| Key | Default | Description |
|-----|---------|-------------|
| `apiKey` | -- | API key from [app.honcho.dev](https://app.honcho.dev) |
| `baseUrl` | -- | Base URL for self-hosted Honcho |
| `peerName` | -- | User peer identity |
| `aiPeer` | host key | AI peer identity (one per profile) |
| `workspace` | host key | Shared workspace ID |
| `contextTokens` | `null` (uncapped) | Token budget for auto-injected context per turn. Truncates at word boundaries |
| `contextCadence` | `1` | Minimum turns between `context()` API calls (base layer refresh) |
| `dialecticCadence` | `2` | Minimum turns between `peer.chat()` LLM calls. Recommended 1–5. Only applies to `hybrid`/`context` modes |
| `dialecticDepth` | `1` | Number of `.chat()` passes per dialectic invocation. Clamped 1–3. Pass 0: cold/warm prompt, pass 1: self-audit, pass 2: reconciliation |
| `dialecticDepthLevels` | `null` | Optional array of reasoning levels per pass, e.g. `["minimal", "low", "medium"]`. Overrides proportional defaults |
| `dialecticReasoningLevel` | `'low'` | Base reasoning level: `minimal`, `low`, `medium`, `high`, `max` |
| `dialecticDynamic` | `true` | When `true`, model can override reasoning level per-call via tool param |
| `dialecticMaxChars` | `600` | Max chars of dialectic result injected into system prompt |
| `recallMode` | `'hybrid'` | `hybrid` (auto-inject + tools), `context` (inject only), `tools` (tools only) |
| `writeFrequency` | `'async'` | When to flush messages: `async` (background thread), `turn` (sync), `session` (batch on end), or integer N |
| `saveMessages` | `true` | Whether to persist messages to Honcho API |
| `observationMode` | `'directional'` | `directional` (all on) or `unified` (shared pool). Override with `observation` object |
| `messageMaxChars` | `25000` | Max chars per message (chunked if exceeded) |
| `dialecticMaxInputChars` | `10000` | Max chars for dialectic query input to `peer.chat()` |
| `sessionStrategy` | `'per-directory'` | `per-directory`, `per-repo`, `per-session`, `global` |
| `pinUserPeer` | `false` | Gateway only. When `true`, every non-agent gateway user collapses to `peerName`; the pin overrides all aliases |
| `userPeerAliases` | `{}` | Gateway only. Maps runtime IDs to peers (`{"7654321": "alice"}`). Many-to-one |
| `runtimePeerPrefix` | `""` | Gateway only. Namespaces unknown runtime IDs (`telegram_7654321`) when no alias matches |
</details>
<details>
<summary>Minimal honcho.json (cloud)</summary>
```json
{
"apiKey": "your-key-from-app.honcho.dev",
"hosts": {
"hermes": {
"enabled": true,
"aiPeer": "hermes",
"peerName": "your-name",
"workspace": "hermes"
}
}
}
```
</details>
<details>
<summary>Minimal honcho.json (self-hosted)</summary>
```json
{
"baseUrl": "http://localhost:8000",
"hosts": {
"hermes": {
"enabled": true,
"aiPeer": "hermes",
"peerName": "your-name",
"workspace": "hermes"
}
}
}
```
</details>
:::tip Migrating from `hermes honcho`
If you previously used `hermes honcho setup`, your config and all server-side data are intact. Just re-enable through the setup wizard again or manually set `memory.provider: honcho` to reactivate via the new system.
:::
**Multi-peer setup:**
Honcho models conversations as peers exchanging messages — one user peer plus one AI peer per Hermes profile, all sharing a workspace. The workspace is the shared environment: the user peer is global across profiles, each AI peer is its own identity. Every AI peer builds an independent representation / card from its own observations, so a `coder` profile stays code-oriented while a `writer` profile stays editorial against the same user.
The mapping:
| Concept | What it is |
|---------|-----------|
| **Workspace** | Shared environment. All Hermes profiles under one workspace see the same user identity. |
| **User peer** (`peerName`) | The human. Shared across profiles in the workspace. |
| **AI peer** (`aiPeer`) | One per Hermes profile. Host key `hermes` → default; `hermes.<profile>` for others. |
| **Observation** | Per-peer toggles controlling what Honcho models from whose messages. `directional` (default, all four on) or `unified` (single-observer pool). |
### New profile, fresh Honcho peer
```bash
hermes profile create coder --clone
```
`--clone` creates a `hermes.coder` host block in `honcho.json` with `aiPeer: "coder"`, shared `workspace`, inherited `peerName`, `recallMode`, `writeFrequency`, `observation`, etc. The AI peer is eagerly created in Honcho so it exists before the first message.
### Existing profiles, backfill Honcho peers
```bash
hermes honcho sync
```
Scans every Hermes profile, creates host blocks for any profile without one, inherits settings from the default `hermes` block, and creates the new AI peers eagerly. Idempotent — skips profiles that already have a host block.
### Per-profile observation
Each host block can override the observation config independently. Example: a code-focused profile where the AI peer observes the user but doesn't self-model:
```json
"hermes.coder": {
"aiPeer": "coder",
"observation": {
"user": { "observeMe": true, "observeOthers": true },
"ai": { "observeMe": false, "observeOthers": true }
}
}
```
**Observation toggles (one set per peer):**
| Toggle | Effect |
|--------|--------|
| `observeMe` | Honcho builds a representation of this peer from its own messages |
| `observeOthers` | This peer observes the other peer's messages (feeds cross-peer reasoning) |
Presets via `observationMode`:
- **`"directional"`** (default) — all four flags on. Full mutual observation; enables cross-peer dialectic.
- **`"unified"`** — user `observeMe: true`, AI `observeOthers: true`, rest false. Single-observer pool; AI models the user but not itself, user peer only self-models.
Server-side toggles set via the [Honcho dashboard](https://app.honcho.dev) win over local defaults — synced back at session init.
See the [Honcho page](./honcho.md#observation-directional-vs-unified) for the full observation reference.
### Gateway identity mapping
The peer model above covers CLI, TUI, and desktop sessions, where every conversation resolves to `peerName`. The [gateway](../../developer-guide/gateway-internals.md) adds a second axis: users arrive with platform-native runtime IDs (Telegram UID, Discord snowflake, Slack user), and three keys decide which peer each ID resolves to.
| Key | Effect |
|-----|--------|
| `pinUserPeer: true` | Every non-agent gateway user collapses to `peerName`. The pin is checked first, so it overrides all aliases — pick it only when no user-side identity needs its own peer |
| `userPeerAliases` | Maps specific runtime IDs to peers (`{"7654321": "alice"}`). The home for routing distinct identities — including agents that each carry their own peer |
| `runtimePeerPrefix` | Namespaces any unmapped runtime ID (`telegram_7654321`) so platforms with same-shaped IDs don't collide |
Off-gateway these keys do nothing. `hermes memory setup` only prompts for them when it detects a connected gateway platform. See the [Honcho page](./honcho.md#gateway-identity-mapping) for the resolver ladder and the setup flow.
<details>
<summary>Full honcho.json example (multi-profile)</summary>
```json
{
"apiKey": "your-key",
"workspace": "hermes",
"peerName": "eri",
"hosts": {
"hermes": {
"enabled": true,
"aiPeer": "hermes",
"workspace": "hermes",
"peerName": "eri",
"recallMode": "hybrid",
"writeFrequency": "async",
"sessionStrategy": "per-directory",
"observation": {
"user": { "observeMe": true, "observeOthers": true },
"ai": { "observeMe": true, "observeOthers": true }
},
"dialecticReasoningLevel": "low",
"dialecticDynamic": true,
"dialecticCadence": 2,
"dialecticDepth": 1,
"dialecticMaxChars": 600,
"contextCadence": 1,
"messageMaxChars": 25000,
"saveMessages": true
},
"hermes.coder": {
"enabled": true,
"aiPeer": "coder",
"workspace": "hermes",
"peerName": "eri",
"recallMode": "tools",
"observation": {
"user": { "observeMe": true, "observeOthers": false },
"ai": { "observeMe": true, "observeOthers": true }
}
},
"hermes.writer": {
"enabled": true,
"aiPeer": "writer",
"workspace": "hermes",
"peerName": "eri"
}
},
"sessions": {
"/home/user/myproject": "myproject-main"
}
}
```
</details>
See the [config reference](https://github.com/NousResearch/hermes-agent/blob/main/plugins/memory/honcho/README.md) and [Honcho integration guide](https://docs.honcho.dev/v3/guides/integrations/hermes).
---
### OpenViking
Context database by Volcengine (ByteDance) with filesystem-style knowledge hierarchy, tiered retrieval, and automatic memory extraction into 6 categories.
| | |
|---|---|
| **Best for** | Self-hosted knowledge management with structured browsing |
| **Requires** | OpenViking initialized, validated, and running |
| **Data storage** | Self-hosted (local or cloud) |
| **Cost** | Free (open-source, AGPL-3.0) |
**Tools (6):** `viking_search` (semantic search), `viking_read` (tiered: abstract/overview/full), `viking_browse` (filesystem navigation), `viking_remember` (store facts), `viking_forget` (delete a memory file by exact `viking://` URI), `viking_add_resource` (ingest URLs/docs)
**Setup:**
```bash
# Prepare OpenViking first
openviking-server init
openviking-server doctor
openviking-server
# Then configure Hermes
hermes memory setup # select "openviking"
# Or manually:
hermes config set memory.provider openviking
```
`hermes memory setup` can reuse or copy connection values from
`~/.openviking/ovcli.conf`. Manual setup uses the active profile's `.env` file;
for the default profile that is `~/.hermes/.env`, and for named profiles use
`~/.hermes/profiles/<profile>/.env`.
```text
OPENVIKING_ENDPOINT=http://127.0.0.1:1933
# OPENVIKING_API_KEY=...
# OPENVIKING_ACCOUNT=default
# OPENVIKING_USER=default
```
OpenViking server settings live in `ov.conf` (`--config`,
`OPENVIKING_CONFIG_FILE`, or `~/.openviking/ov.conf`). Client connection values
live in `ovcli.conf` (`OPENVIKING_CLI_CONFIG_FILE` or
`~/.openviking/ovcli.conf`).
**Key features:**
- Tiered context loading: L0 (~100 tokens) → L1 (~2k) → L2 (full)
- Automatic memory extraction on session commit (profile, preferences, entities, events, cases, patterns)
- `viking://` URI scheme for hierarchical knowledge browsing
`OPENVIKING_ACCOUNT` and `OPENVIKING_USER` are used for local/trusted mode.
Peer identity is optional. By default, Hermes sends no peer ID and writes
explicit memories to `viking://user/<user>/memories/...`. Setup does not ask
for a peer ID. For separate assistant context, set
`memory.openviking.agent: work-assistant` in `config.yaml`.
Existing non-empty peer settings keep their peer-scoped writes and recall.
This includes `OPENVIKING_AGENT` and `actor_peer_id` or legacy `agent_id` in a
linked OpenViking config. Existing memories are not moved or deleted.
With no peer ID, default search covers user memory and existing peer memories
under the same OpenViking user. Old peer memories remain searchable at their
existing paths. Ranking and result limits determine which memories are returned.
Set `memory.openviking.agent: hermes` to restore the old peer-scoped writes.
Memories written at user scope before this change stay there and remain
searchable. The setting changes future writes, not existing memory locations.
Hermes sends `User-Agent: openviking-memory-hermes/<version>` on OpenViking
requests. This standard harness identifier contains no per-user identifier and
does not add a separate request.
---
### Mem0
Server-side LLM fact extraction with semantic search, reranking, and automatic deduplication. Three connection modes: **Platform** (Mem0 Cloud), **self-hosted dashboard** (a Mem0 server you run via Docker), and **OSS** (Mem0 in-process with your own LLM + vector store).
| | |
|---|---|
| **Best for** | Hands-off memory management — Mem0 handles extraction automatically |
| **Requires** | `pip install mem0ai` + API key (platform), a running Mem0 server (self-hosted dashboard), or an LLM + vector store (OSS) |
| **Data storage** | Mem0 Cloud (platform), your own Mem0 server (self-hosted dashboard), or in-process (OSS) |
| **Cost** | Mem0 pricing (platform) / free (self-hosted or OSS) |
**Tools (4):** `mem0_search` (semantic search; optional reranking in platform mode, off by default), `mem0_add` (store verbatim facts), `mem0_update` (update by ID), `mem0_delete` (delete by ID)
**Setup (Platform):**
```bash
hermes memory setup # select "mem0" → "Platform"
# Or manually:
hermes config set memory.provider mem0
echo "MEM0_API_KEY=your-key" >> ~/.hermes/.env
```
**Setup (OSS):**
```bash
hermes memory setup # select "mem0" → "Open Source (self-hosted)"
# Or via flags:
hermes memory setup mem0 --mode oss --oss-llm openai --oss-llm-key sk-... --oss-vector qdrant
```
Preview without writing files:
```bash
hermes memory setup mem0 --mode oss --oss-llm-key sk-... --dry-run
```
**Setup (Self-Hosted Dashboard):** connect to a Mem0 server you run via Docker (the dashboard's REST API):
```bash
hermes memory setup # select "mem0" → "Self-hosted server"
# Or via flags:
hermes memory setup mem0 --mode selfhosted --host http://localhost:8888 --api-key your-admin-api-key
```
Or configure manually — either as env vars:
```bash
echo "MEM0_HOST=http://localhost:8888" >> ~/.hermes/.env
echo "MEM0_API_KEY=your-admin-api-key" >> ~/.hermes/.env
```
or in `mem0.json`:
```json
{ "host": "http://localhost:8888", "api_key": "your-admin-api-key" }
```
The plugin authenticates with `X-API-Key` and uses the server's `/search` / `/memories` routes. `api_key` is optional (omit only for `AUTH_DISABLED` servers). Don't set `mode: oss` — it takes precedence over `host`.
**Config:** `$HERMES_HOME/mem0.json` (behavioral settings). Only the secret `MEM0_API_KEY` belongs in `~/.hermes/.env`.
| Key | Default | Description |
|-----|---------|-------------|
| `mode` | `platform` | `platform` (Mem0 Cloud) or `oss` (self-managed, in-process) |
| `host` | — | Self-hosted Mem0 server URL (Docker dashboard). Routes over HTTP with `X-API-Key`; don't combine with `mode: oss` |
| `user_id` | `hermes-user` | User identifier |
| `agent_id` | `hermes` | Agent identifier |
| `rerank` | `false` | Rerank search results for relevance (platform mode only) |
**OSS supported providers:**
| Component | Providers |
|-----------|-----------|
| LLM | openai, ollama |
| Embedder | openai, ollama |
| Vector Store | qdrant (local/server), pgvector |
**Switching modes:** Re-run `hermes memory setup mem0 --mode <platform|selfhosted|oss>` or edit `mem0.json` directly.
---
### Hindsight
Long-term memory with knowledge graph, entity resolution, and multi-strategy retrieval. The `hindsight_reflect` tool provides cross-memory synthesis that no other provider offers. Automatically retains full conversation turns (including tool calls) with session-level document tracking.
| | |
|---|---|
| **Best for** | Knowledge graph-based recall with entity relationships |
| **Requires** | Cloud: API key from [ui.hindsight.vectorize.io](https://ui.hindsight.vectorize.io). Local: LLM API key (OpenAI, Groq, OpenRouter, etc.) |
| **Data storage** | Hindsight Cloud or local embedded PostgreSQL |
| **Cost** | Hindsight pricing (cloud) or free (local) |
**Tools:** `hindsight_retain` (store with entity extraction), `hindsight_recall` (multi-strategy search), `hindsight_reflect` (cross-memory synthesis)
**Setup:**
```bash
hermes memory setup # select "hindsight"
# Or manually:
hermes config set memory.provider hindsight
echo "HINDSIGHT_API_KEY=your-key" >> ~/.hermes/.env
```
The setup wizard installs dependencies automatically and only installs what's needed for the selected mode (`hindsight-client` for cloud, `hindsight-all` for local). Requires `hindsight-client >= 0.4.22` (auto-upgraded on session start if outdated).
**Local mode UI:** `hindsight-embed -p hermes ui start`
**Config:** `$HERMES_HOME/hindsight/config.json`
| Key | Default | Description |
|-----|---------|-------------|
| `mode` | `cloud` | `cloud` or `local` |
| `bank_id` | `hermes` | Memory bank identifier |
| `recall_budget` | `mid` | Recall thoroughness: `low` / `mid` / `high` |
| `memory_mode` | `hybrid` | `hybrid` (context + tools), `context` (auto-inject only), `tools` (tools only) |
| `auto_retain` | `true` | Automatically retain conversation turns |
| `auto_recall` | `true` | Automatically recall memories before each turn |
| `retain_async` | `true` | Process retain asynchronously on the server |
| `retain_context` | `conversation between Hermes Agent and the User` | Context label for retained memories |
| `retain_tags` | — | Default tags applied to retained memories; merged with per-call tool tags |
| `retain_source` | — | Optional `metadata.source` attached to retained memories |
| `retain_user_prefix` | `User` | Label used before user turns in auto-retained transcripts |
| `retain_assistant_prefix` | `Assistant` | Label used before assistant turns in auto-retained transcripts |
| `recall_tags` | — | Tags to filter on recall |
See [plugin README](https://github.com/NousResearch/hermes-agent/blob/main/plugins/memory/hindsight/README.md) for the full configuration reference.
---
### Holographic
Local SQLite fact store with FTS5 full-text search, trust scoring, and HRR (Holographic Reduced Representations) for compositional algebraic queries.
| | |
|---|---|
| **Best for** | Local-only memory with advanced retrieval, no external dependencies |
| **Requires** | Nothing (SQLite is always available). NumPy optional for HRR algebra. |
| **Data storage** | Local SQLite |
| **Cost** | Free |
**Tools:** `fact_store` (9 actions: add, search, probe, related, reason, contradict, update, remove, list), `fact_feedback` (helpful/unhelpful rating that trains trust scores)
**Setup:**
```bash
hermes memory setup # select "holographic"
# Or manually:
hermes config set memory.provider holographic
```
**Config:** `config.yaml` under `plugins.hermes-memory-store`
| Key | Default | Description |
|-----|---------|-------------|
| `db_path` | `$HERMES_HOME/memory_store.db` | SQLite database path |
| `auto_extract` | `false` | Auto-extract facts at session end |
| `default_trust` | `0.5` | Default trust score (0.0–1.0) |
**Unique capabilities:**
- `probe` — entity-specific algebraic recall (all facts about a person/thing)
- `reason` — compositional AND queries across multiple entities
- `contradict` — automated detection of conflicting facts
- Trust scoring with asymmetric feedback (+0.05 helpful / -0.10 unhelpful)
---
### RetainDB
Cloud memory API with hybrid search (Vector + BM25 + Reranking), 7 memory types, and delta compression.
| | |
|---|---|
| **Best for** | Teams already using RetainDB's infrastructure |
| **Requires** | RetainDB account + API key |
| **Data storage** | RetainDB Cloud |
| **Cost** | $20/month |
**Tools (10):** `retaindb_profile` (user profile), `retaindb_search` (semantic search), `retaindb_context` (task-relevant context), `retaindb_remember` (store with type + importance), `retaindb_forget` (delete memories), plus file tools: `retaindb_upload_file`, `retaindb_list_files`, `retaindb_read_file`, `retaindb_ingest_file`, `retaindb_delete_file`
**Setup:**
```bash
hermes memory setup # select "retaindb"
# Or manually:
hermes config set memory.provider retaindb
echo "RETAINDB_API_KEY=your-key" >> ~/.hermes/.env
```
---
### ByteRover
Persistent memory via the `brv` CLI — hierarchical knowledge tree with tiered retrieval (fuzzy text → LLM-driven search). Local-first with optional cloud sync.
| | |
|---|---|
| **Best for** | Developers who want portable, local-first memory with a CLI |
| **Requires** | ByteRover CLI (`npm install -g byterover-cli` or [install script](https://byterover.dev)) |
| **Data storage** | Local (default) or ByteRover Cloud (optional sync) |
| **Cost** | Free (local) or ByteRover pricing (cloud) |
**Tools:** `brv_query` (search knowledge tree), `brv_curate` (store facts/decisions/patterns), `brv_status` (CLI version + tree stats)
**Setup:**
```bash
# Install the CLI first
curl -fsSL https://byterover.dev/install.sh | sh
# Then configure Hermes
hermes memory setup # select "byterover"
# Or manually:
hermes config set memory.provider byterover
```
**Key features:**
- Automatic pre-compression extraction (saves insights before context compression discards them)
- Knowledge tree stored at `$HERMES_HOME/byterover/` (profile-scoped)
- SOC2 Type II certified cloud sync (optional)
---
### Supermemory
Semantic long-term memory with profile recall, semantic search, explicit memory tools, and session-end conversation ingest via the Supermemory graph API.
| | |
|---|---|
| **Best for** | Semantic recall with user profiling and session-level graph building |
| **Requires** | `pip install supermemory` + [cloud API key](http://app.supermemory.ai/integrations?connect=hermes), or a [self-hosted server](https://supermemory.ai/docs/self-hosting/overview) |
| **Data storage** | Supermemory Cloud or self-hosted |
| **Cost** | Supermemory pricing (cloud) / free (self-hosted) |
**Tools:** `supermemory_store` (save explicit memories), `supermemory_search` (semantic similarity search), `supermemory_forget` (forget by ID or best-match query), `supermemory_profile` (persistent profile + recent context)
**Setup:**
```bash
hermes memory setup # select "supermemory"
# Or manually:
hermes config set memory.provider supermemory
echo 'SUPERMEMORY_API_KEY=***' >> ~/.hermes/.env
```
Self-hosted setup:
```bash
npx supermemory local
```
Before running `hermes memory setup`, set `base_url` in
`$HERMES_HOME/supermemory.json`:
```json
{
"base_url": "http://localhost:6767"
}
```
Then run `hermes memory setup` and enter the API key printed by the local
server. Configuring the endpoint first ensures the setup connection probe also
stays local.
**Config:** `$HERMES_HOME/supermemory.json`
| Key | Default | Description |
|-----|---------|-------------|
| `base_url` | `https://api.supermemory.ai` | API endpoint for hosted or self-hosted Supermemory. Takes priority over `SUPERMEMORY_BASE_URL`. |
| `container_tag` | `hermes` | Container tag used for search and writes. Supports `{identity}` template for profile-scoped tags. |
| `auto_recall` | `true` | Inject relevant memory context before turns |
| `auto_capture` | `true` | Store cleaned user-assistant turns after each response |
| `max_recall_results` | `10` | Max recalled items to format into context |
| `profile_frequency` | `50` | Include profile facts on first turn and every N turns |
| `capture_mode` | `all` | Skip tiny or trivial turns by default |
| `search_mode` | `hybrid` | Search mode: `hybrid`, `memories`, or `documents` |
| `api_timeout` | `5.0` | Timeout for SDK and ingest requests |
**Environment variables:** `SUPERMEMORY_API_KEY` (required), `SUPERMEMORY_BASE_URL` (compatibility fallback when `base_url` is not configured), `SUPERMEMORY_CONTAINER_TAG` (overrides config).
Base URL precedence is `supermemory.json` → `SUPERMEMORY_BASE_URL` → `https://api.supermemory.ai`. SDK operations, setup/status probes, and conversation ingest all use the resolved endpoint.
**Key features:**
- Automatic context fencing — strips recalled memories from captured turns to prevent recursive memory pollution
- Full-session ingest — the entire conversation is sent once at session boundaries
- Session-end conversation ingest (to `/v4/conversations`) for richer profile + graph building in Supermemory
- End-to-end self-hosted routing — SDK, probe, and conversation-ingest requests use the same configured endpoint
- Profile facts injected on first turn and at configurable intervals
- **Profile-scoped containers** — use `{identity}` in `container_tag` (e.g. `hermes-{identity}` → `hermes-coder`) to isolate memories per Hermes profile
- **Multi-container mode** — enable `enable_custom_container_tags` with a `custom_containers` list to let the agent read/write across named containers. Automatic operations stay on the primary container.
<details>
<summary>Multi-container example</summary>
```json
{
"container_tag": "hermes",
"enable_custom_container_tags": true,
"custom_containers": ["project-alpha", "shared-knowledge"],
"custom_container_instructions": "Use project-alpha for coding context."
}
```
</details>
**Support:** [Discord](https://supermemory.link/discord) · [support@supermemory.com](mailto:support@supermemory.com)
### Memori
Structured long-term memory using Memori Cloud, with background completed-turn capture, tool-aware turn context, and explicit recall tools for facts, summaries, quota, signup, and feedback.
| | |
|---|---|
| **Best for** | Agent-controlled recall with structured project and session attribution |
| **Requires** | `pip install hermes-memori` + `hermes-memori install` + [Memori API key](https://app.memorilabs.ai/signup) |
| **Data storage** | Memori Cloud |
| **Cost** | Memori pricing |
**Tools:** `memori_recall` (search long-term memory), `memori_recall_summary` (summarized context), `memori_quota` (usage/quota), `memori_signup` (request signup email), `memori_feedback` (send integration feedback)
**Setup:**
```bash
pip install hermes-memori
hermes-memori install
hermes config set memory.provider memori
hermes memory setup
```
---
## Provider Comparison
| Provider | Storage | Cost | Tools | Dependencies | Unique Feature |
|----------|---------|------|-------|-------------|----------------|
| **Honcho** | Cloud | Paid | 5 | `honcho-ai` | Dialectic user modeling + session-scoped context |
| **OpenViking** | Self-hosted | Free | 6 | `openviking` + server | Filesystem hierarchy + tiered loading |
| **Mem0** | Cloud/Self-hosted | Free/Paid | 4 | `mem0ai` | Server-side LLM extraction + self-hosted/OSS modes |
| **Hindsight** | Cloud/Local | Free/Paid | 3 | `hindsight-client` | Knowledge graph + reflect synthesis |
| **Holographic** | Local | Free | 2 | None | HRR algebra + trust scoring |
| **RetainDB** | Cloud | $20/mo | 10 | `requests` | Delta compression |
| **ByteRover** | Local/Cloud | Free/Paid | 3 | `brv` CLI | Pre-compression extraction |
| **Supermemory** | Cloud/Self-hosted | Free/Paid | 4 | `supermemory` | Context fencing + session graph ingest + multi-container |
| **Memori** | Cloud | Free/Paid | 5 | `hermes-memori` | Tool-aware memory + structured recall |
## Profile Isolation
Each provider's data is isolated per [profile](/user-guide/profiles):
- **Local storage providers** (Holographic, ByteRover) use `$HERMES_HOME/` paths which differ per profile
- **Config file providers** (Honcho, Mem0, Hindsight, Supermemory) store config in `$HERMES_HOME/` so each profile has its own credentials
- **Cloud providers** (RetainDB) auto-derive profile-scoped project names
- **Env var providers** (OpenViking) are configured via each profile's `.env` file
## Building a Memory Provider
See the [Developer Guide: Memory Provider Plugins](/developer-guide/memory-provider-plugin) for how to create your own.
website/docs/user-guide/features/memory.md
+443
View File
@@ -0,0 +1,443 @@
---
sidebar_position: 3
title: "Persistent Memory"
description: "How Hermes Agent remembers across sessions — MEMORY.md, USER.md, and session search"
---
# Persistent Memory
Hermes Agent has bounded, curated memory that persists across sessions. This lets it remember your preferences, your projects, your environment, and things it has learned.
## How It Works
Two files make up the agent's memory:
| File | Purpose | Char Limit |
|------|---------|------------|
| **MEMORY.md** | Agent's personal notes — environment facts, conventions, things learned | 2,200 chars (~800 tokens) |
| **USER.md** | User profile — your preferences, communication style, expectations | 1,375 chars (~500 tokens) |
Both are stored in `~/.hermes/memories/` and are injected into the system prompt as a frozen snapshot at session start. The agent manages its own memory via the `memory` tool — it can add, replace, or remove entries.
:::caution One agent per Hermes home
Don't point two agent processes at the same Hermes home directory. Memory writes are automatic and load back into the system prompt at session start, so two writers sharing one home will compound each other's entries into state neither of them (nor you) authored. Memory is scoped per [profile](/user-guide/profiles) by design — give a second agent its own profile, and if they need shared memory, use an [external memory provider](/user-guide/features/memory-providers) instead.
:::
:::info
Character limits keep memory focused. Memory does **not** auto-compact: when a
write would exceed the limit, the `memory` tool returns an error instead of
silently dropping entries. The agent then makes room itself — consolidating or
removing entries in the same turn before retrying (see [What Happens When Memory
is Full](#what-happens-when-memory-is-full)). Note that `replace` is also bound
by the limit: swapping an entry for a longer one can still overflow, so the new
content must be shortened (or another entry removed) to fit.
:::
## How Memory Appears in the System Prompt
At the start of every session, memory entries are loaded from disk and rendered into the system prompt as a frozen block:
```
══════════════════════════════════════════════
MEMORY (your personal notes) [67% — 1,474/2,200 chars]
══════════════════════════════════════════════
User's project is a Rust web service at ~/code/myapi using Axum + SQLx
§
This machine runs Ubuntu 22.04, has Docker and Podman installed
§
User prefers concise responses, dislikes verbose explanations
```
The format includes:
- A header showing which store (MEMORY or USER PROFILE)
- Usage percentage and character counts so the agent knows capacity
- Individual entries separated by `§` (section sign) delimiters
- Entries can be multiline
**Frozen snapshot pattern:** The system prompt injection is captured once at session start and never changes mid-session. This is intentional — it preserves the LLM's prefix cache for performance. When the agent adds/removes memory entries during a session, the changes are persisted to disk immediately but won't appear in the system prompt until the next session starts. Tool responses always show the live state.
## Memory Tool Actions
The agent uses the `memory` tool with these actions:
- **add** — Add a new memory entry
- **replace** — Replace an existing entry with updated content (uses substring matching via `old_text`)
- **remove** — Remove an entry that's no longer relevant (uses substring matching via `old_text`)
There is no `read` action — memory content is automatically injected into the system prompt at session start. The agent sees its memories as part of its conversation context.
### Substring Matching
The `replace` and `remove` actions use short unique substring matching — you don't need the full entry text. The `old_text` parameter just needs to be a unique substring that identifies exactly one entry:
```python
# If memory contains "User prefers dark mode in all editors"
memory(action="replace", target="memory",
old_text="dark mode",
content="User prefers light mode in VS Code, dark mode in terminal")
```
If the substring matches multiple entries, an error is returned asking for a more specific match.
## Two Targets Explained
### `memory` — Agent's Personal Notes
For information the agent needs to remember about the environment, workflows, and lessons learned:
- Environment facts (OS, tools, project structure)
- Project conventions and configuration
- Tool quirks and workarounds discovered
- Completed task diary entries
- Skills and techniques that worked
### `user` — User Profile
For information about the user's identity, preferences, and communication style:
- Name, role, timezone
- Communication preferences (concise vs detailed, format preferences)
- Pet peeves and things to avoid
- Workflow habits
- Technical skill level
## What to Save vs Skip
### Save These (Proactively)
The agent saves automatically — you don't need to ask. It saves when it learns:
- **User preferences:** "I prefer TypeScript over JavaScript" → save to `user`
- **Environment facts:** "This server runs Debian 12 with PostgreSQL 16" → save to `memory`
- **Corrections:** "Don't use `sudo` for Docker commands, user is in docker group" → save to `memory`
- **Conventions:** "Project uses tabs, 120-char line width, Google-style docstrings" → save to `memory`
- **Completed work:** "Migrated database from MySQL to PostgreSQL on 2026-01-15" → save to `memory`
- **Explicit requests:** "Remember that my API key rotation happens monthly" → save to `memory`
### Skip These
- **Trivial/obvious info:** "User asked about Python" — too vague to be useful
- **Easily re-discovered facts:** "Python 3.12 supports f-string nesting" — can web search this
- **Raw data dumps:** Large code blocks, log files, data tables — too big for memory
- **Session-specific ephemera:** Temporary file paths, one-off debugging context
- **Information already in context files:** SOUL.md and AGENTS.md content
## Capacity Management
Memory has strict character limits to keep system prompts bounded:
| Store | Limit | Typical entries |
|-------|-------|----------------|
| memory | 2,200 chars | 8-15 entries |
| user | 1,375 chars | 5-10 entries |
### What Happens When Memory is Full
When you try to add an entry that would exceed the limit, the tool returns an error:
```json
{
"success": false,
"error": "Memory at 2,100/2,200 chars. Adding this entry (250 chars) would exceed the limit. Consolidate now: use 'replace' to merge overlapping entries into shorter ones or 'remove' stale or less important entries (see current_entries below), then retry this add — all in this turn.",
"current_entries": ["..."],
"usage": "2,100/2,200"
}
```
The agent should then:
1. Read the current entries (shown in the error response)
2. Identify entries that can be removed or consolidated
3. Use `replace` to merge related entries into shorter versions
4. Then `add` the new entry
**Best practice:** When memory is above 80% capacity (visible in the system prompt header), consolidate entries before adding new ones. For example, merge three separate "project uses X" entries into one comprehensive project description entry.
### Practical Examples of Good Memory Entries
**Compact, information-dense entries work best:**
```
# Good: Packs multiple related facts
User runs macOS 14 Sonoma, uses Homebrew, has Docker Desktop and Podman. Shell: zsh with oh-my-zsh. Editor: VS Code with Vim keybindings.
# Good: Specific, actionable convention
Project ~/code/api uses Go 1.22, sqlc for DB queries, chi router. Run tests with 'make test'. CI via GitHub Actions.
# Good: Lesson learned with context
The staging server (10.0.1.50) needs SSH port 2222, not 22. Key is at ~/.ssh/staging_ed25519.
# Bad: Too vague
User has a project.
# Bad: Too verbose
On January 5th, 2026, the user asked me to look at their project which is
located at ~/code/api. I discovered it uses Go version 1.22 and...
```
## Duplicate Prevention
The memory system automatically rejects exact duplicate entries. If you try to add content that already exists, it returns success with a "no duplicate added" message.
## Security Scanning
Memory entries are scanned for injection and exfiltration patterns before being accepted, since they're injected into the system prompt. Content matching threat patterns (prompt injection, credential exfiltration, SSH backdoors) or containing invisible Unicode characters is blocked.
## Session Search
Beyond MEMORY.md and USER.md, the agent can search its past conversations using the `session_search` tool:
- All CLI and messaging sessions are stored in SQLite (`~/.hermes/state.db`) with FTS5 full-text search
- Search queries return actual messages from the DB — no LLM summarization, no truncation
- The agent can find things it discussed weeks ago, even if they're not in its active memory
- The agent can also scroll forward/backward inside any session it finds
```bash
hermes sessions list # Browse past sessions
```
See [Session Search Tool](/user-guide/sessions#session-search-tool) for the three calling shapes (discovery / scroll / browse) and the response format.
### session_search vs memory
| Feature | Persistent Memory | Session Search |
|---------|------------------|----------------|
| **Capacity** | ~1,300 tokens total | Unlimited (all sessions) |
| **Speed** | Instant (in system prompt) | ~20ms FTS5 query, ~1ms scroll |
| **Cost** | Token cost in every prompt | Free — no LLM calls |
| **Use case** | Key facts always available | Finding specific past conversations |
| **Management** | Manually curated by agent | Automatic — all sessions stored |
| **Token cost** | Fixed per session (~1,300 tokens) | On-demand (searched when needed) |
**Memory** is for critical facts that should always be in context. **Session search** is for "did we discuss X last week?" queries where the agent needs to recall specifics from past conversations.
## Learning Journey (`/journey`)
The learning journey is a timeline view of everything Hermes has learned — saved skills and memory entries plotted over time (oldest at top, newest at bottom), with a playable "constellation" scrubber that replays the build-up. The same graph data drives three surfaces:
- **Classic CLI / standalone** — `hermes journey` (aliases: `hermes learning`, `hermes memory-graph`) renders the timeline in the terminal. Flags: `--play` animates the build-up (`--fps` to tune it), `--width`/`--height` override the render size, `--no-color` disables color, and `--json` dumps the raw graph payload.
- **TUI** — `/journey` (aliases: `/learning`, `/memory-graph`) opens the timeline as an overlay.
- **Desktop app** — `/journey` opens the Star Map / memory-graph panel, an interactive visual of the same nodes.
Beyond viewing, the journey is also where you **prune and correct** what Hermes has learned:
| Command | What it does |
|---------|--------------|
| `hermes journey list` | List node ids — skill names and `memory:<source>:<index>` ids for memory chunks. |
| `hermes journey delete <node> [-y]` | Delete a node. Skills are **archived** (restorable), memory chunks are removed. `-y` skips the confirmation. |
| `hermes journey edit <node>` | Open the node's content (a skill's `SKILL.md` or the memory chunk) in `$EDITOR`. |
The same `list` / `delete <id>` / `edit <id>` subcommands work from the in-chat `/journey` command on the CLI, and the desktop panel offers edit/delete on nodes directly.
## Configuration
```yaml
# In ~/.hermes/config.yaml
memory:
memory_enabled: true
user_profile_enabled: true
memory_char_limit: 2200 # ~800 tokens
user_char_limit: 1375 # ~500 tokens
write_approval: false # false = write freely (default) | true = require approval
```
Setting **both** `memory_enabled` and `user_profile_enabled` to `false` turns the
built-in stores off completely: the `memory` tool is dropped from the schema and
its guidance block is dropped from the system prompt, so the model is never told
about a tool it cannot use. An external provider set via `memory.provider`
(Hindsight, Mem0, Honcho, …) is unaffected and keeps its own tools — use this
when you want a third-party memory backend *instead of* the built-in files.
Listing `memory` under `agent.disabled_toolsets` is the heavier switch: it hides
external provider tools too.
With only `memory_enabled: false` (user profile still on), the tool stays —
it backs the profile store — but the system prompt swaps the full memory
guidance for a narrower profile-only block. The tool schema advertises only the
`user` target, and direct or staged writes to disabled `MEMORY.md` are rejected.
The inverse configuration advertises only `memory` and rejects `USER.md` writes.
## Controlling memory writes (`write_approval`)
By default the agent saves memory freely — including from the background
self-improvement review that runs after a turn. If you'd rather approve saves
first, set `memory.write_approval: true`. It's a simple on/off gate applied to
**both** foreground turns and the background review:
| `write_approval` | Behaviour |
|------------------|-----------|
| `false` (default) | Write freely — the gate is off (the pre-gate behaviour). |
| `true` | Require approval before anything is saved. In the interactive CLI, foreground writes prompt you inline (entries are small enough to read in full). Everywhere else — messaging platforms, scripts, and the background self-improvement review — writes are **staged** for review with `/memory pending`. |
> To turn memory off entirely (not just gate it), set both `memory_enabled: false` and `user_profile_enabled: false`. When both built-in stores are disabled, the built-in `memory` tool is automatically hidden.
Review staged writes from the CLI or any messaging platform:
```
/memory pending # list staged memory writes (auto ones tagged [auto])
/memory approve <id> # apply one (or 'all')
/memory reject <id> # drop one (or 'all')
/memory approval on # turn the gate on (or 'off') and persist it
```
This is the answer to "the agent saved a wrong assumption about me": set
`write_approval: true`, and every save — especially the unprompted background
ones — waits for your yes/no before it ever enters your profile.
## Background review notifications (`display.memory_notifications`)
After a turn, the background self-improvement review may quietly save a memory
or update a skill. This is Hermes' consent-aware learning loop: repeated
corrections and durable workflow lessons become compact memory entries or
procedural skills, while `write_approval` can stage those writes for review
before they affect future sessions. By default it surfaces a short
`💾 Memory updated` line in chat so you know it happened. Control how chatty
that is:
```yaml
display:
memory_notifications: on # off | on (default) | verbose
```
| Value | Behaviour |
|-------|-----------|
| `off` | No chat notification. The review still runs and still writes — you just don't see a line for it. |
| `on` (default) | Generic line, e.g. `💾 Memory updated`, `💾 Skill 'foo' patched`. |
| `verbose` | Includes a compact preview of what changed, e.g. `💾 Memory ➕ User prefers terse replies` or a `"old" → "new"` skill diff snippet. |
> This only governs the **gateway** chat notification. The review itself, and
> writes to your memory/skill stores, are unaffected by this setting. Set it
> per-platform via `display.platforms.<platform>.memory_notifications`.
## Running the review on a cheaper model (`auxiliary.background_review`)
The review runs on your **main chat model** by default, replaying the
conversation — which is already warm in the prompt cache, so it's cheap cache
reads. On an expensive main model you can run the review on a cheaper model
instead:
```yaml
auxiliary:
background_review:
provider: openrouter
model: google/gemini-3-flash-preview # auto (default) = main chat model
```
When you point it at a model **different** from your main one, the review runs
there for substantially lower cost (~3–5× in benchmarks). Because a different
model can't reuse your main model's prompt cache anyway, the fork automatically
replays a compact **digest** of the conversation (recent turns verbatim + a
summary of older ones) rather than the full transcript — minimizing what it
writes to the new cache. Capture holds: in testing, memory capture was
identical and skill capture near-identical to the main-model review.
Leave it at `auto` (or set it to your main model) and nothing changes — the
review keeps running on the main model with the full warm-cache replay.
### Disabling automatic reviews (`enabled`)
The review fork can burn a meaningful share of total tokens on busy hosts.
Operators can disable it without zeroing nudge intervals:
```yaml
auxiliary:
background_review:
enabled: true # false = skip automatic post-turn forks
```
With `enabled: false`, automatic post-turn forks do not spawn; manual
`/refine` still works.
Fork usage is persisted in `session_model_usage` with `task='background_review'`
and a completion line is written to `agent.log`
(`Background review complete: thread=bg-review calls=… in=… out=… result=…`).
### Allowing a narrowly scoped extra review tool (`extra_tools`)
Background review can use memory, skill-management, and read-only file tools
by default. If a profile provides another tool that is safe for unattended
review, opt it in by name:
```yaml
auxiliary:
background_review:
extra_tools:
- propose_shared_memory
```
The tool must already be available to the parent agent; this setting only adds
it to the review fork's runtime whitelist. It does not enable arbitrary tools,
and tools not listed here remain denied. Keep the list narrow and prefer tools
that stage a proposal for human review rather than applying external or
destructive changes directly. The default is an empty list.
### Local models: reviews wait for an idle GPU (`defer`)
On a cloud provider the review finishes in seconds and runs alongside
whatever you do next. When the review's runtime is the **managed local
llama-server** (Settings → Local models), the same fork occupies the GPU your
next prompt needs — for minutes on a large model — and sending a new prompt
cancels it, discarding the learning. So on the managed local runtime, reviews
are **deferred by default**: queued at turn end and executed once the machine
has been quiet for a short settle window. Nothing about the review itself
changes — same model, same full-transcript replay, same writes — only the
execution moment moves.
```yaml
auxiliary:
background_review:
defer: auto # auto (default) | never
defer_max_age_s: 1800 # run a queued review anyway after this long
```
| Value | Behaviour |
|-------|-----------|
| `auto` (default) | Reviews whose runtime resolves to the managed local server are queued and run at idle; every other runtime (cloud, external servers) spawns immediately as before. |
| `never` | Old behavior everywhere: spawn immediately at turn end, even on the managed local GPU. |
Queued reviews coalesce per session (a newer turn's snapshot replaces the
older one — the review replays the whole conversation, so nothing is lost),
a review preempted by a new prompt is re-queued instead of discarded, and a
review that has waited longer than `defer_max_age_s` runs even if the machine
never goes idle. Explicit `/refine` always runs immediately. The queue is
in-memory: reviews still pending when the app exits are dropped, same as an
in-flight fork would have been.
## Controlling skill writes (`skills.write_approval`)
Skills use the same on/off gate, but the review UX differs because a
`SKILL.md` is far too large to read in a chat bubble:
```yaml
skills:
write_approval: false # false = write freely (default) | true = require approval
```
When `write_approval: true`, skill writes (create / edit / patch / write_file /
delete) always **stage** regardless of origin. You review the one-line gist
inline, but the full diff stays out-of-band:
```
/skills pending # list staged skill writes + a one-line gist each
/skills diff <id> # full unified diff (best viewed in CLI or dashboard)
/skills approve <id> # apply it (or 'all')
/skills reject <id> # drop it (or 'all')
/skills approval on # turn the gate on (or 'off') and persist it
```
On a messaging platform, approve a skill from its gist + metadata, or open
`/skills diff` on the CLI / dashboard / the staged file under
`~/.hermes/pending/skills/<id>.json` when you want to read the whole change.
Full details in [Gating agent skill writes](/user-guide/features/skills#gating-agent-skill-writes-skillswrite_approval).
## External Memory Providers
For deeper, persistent memory that goes beyond MEMORY.md and USER.md, Hermes ships with 8 external memory provider plugins — including Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, and Supermemory.
External providers run **alongside** built-in memory (never replacing it) and add capabilities like knowledge graphs, semantic search, automatic fact extraction, and cross-session user modeling.
```bash
hermes memory setup # pick a provider and configure it
hermes memory status # check what's active
```
See the [Memory Providers](./memory-providers.md) guide for full details on each provider, setup instructions, and comparison.
website/docs/user-guide/features/mixture-of-agents.md
+271
View File
@@ -0,0 +1,271 @@
---
sidebar_position: 7
title: "Mixture of Agents"
description: "Create named MoA presets that appear as selectable models under the Mixture of Agents provider"
---
# Mixture of Agents
Mixture of Agents is a virtual model provider. Each named MoA preset appears as a selectable model under the `moa` provider.
When you select a MoA preset, the preset's aggregator is the acting model. It is the model that writes the assistant response and emits tool calls. Reference models run first and provide analysis for the aggregator to use.
Use MoA when a hard task benefits from multiple model perspectives but still needs Hermes' normal agent loop: tool calls, follow-up iterations, interrupts, transcript persistence, and the same session context as any other message.
## Select a MoA preset as your model
You can select a preset through the normal model picker surfaces:
```bash
/model default --provider moa
/model review --provider moa
```
MoA presets are selectable on **every Hermes surface**, because MoA is a normal provider in the model system:
- **CLI / gateway / TUI `/model`** — `/model <preset> --provider moa`, or `/model --provider moa` for the default preset. A bare `/model <preset>` also works when the name exactly matches a configured preset.
- **`hermes model`** and the **Dashboard model picker** — a `Mixture of Agents` provider row appears with your preset names as its models.
- **Desktop GUI app** — the model dropdown shows an `MoA presets` section; selecting one (`MoA: <preset>`) switches the active model to that preset. The Desktop settings panel also creates and edits presets.
Configured presets therefore show up wherever you would pick any other model.
## Slash command shortcut
`/moa` is one-shot convenience sugar. It runs a single prompt through the **default** MoA preset, then restores whatever model you were on:
```bash
/moa design and implement a migration plan for this flaky test cluster
```
Hermes temporarily switches to the default MoA preset for that one turn, sends the prompt, then restores your previous model afterward. The whole argument is the prompt — `/moa` no longer interprets it as a preset name.
```bash
/moa
```
Bare `/moa` (no prompt) just prints usage.
To **switch** to a MoA preset for the rest of the session, select it from the model picker — MoA presets appear under a `Mixture of Agents` provider in every model-selection surface (see above). `/moa` is deliberately not a model switch, so a normal prompt can never accidentally change your model.
## How it works in the agent loop
For each main model call when provider `moa` is selected, Hermes:
1. resolves the selected preset by name;
2. runs the configured reference models without tool schemas (they receive only the conversation's user/assistant text — not the Hermes system prompt or tool-call transcript — so reference calls stay cheap and avoid strict-provider rejections);
3. appends the reference outputs as private context for the aggregator;
4. calls the configured aggregator with the normal Hermes tool schema;
5. treats the aggregator response as the real model response;
6. if the aggregator calls tools, Hermes executes those tools normally;
7. on the next model iteration, the same MoA process runs again over the updated conversation, including tool results.
Because MoA is selected through the normal model system, it composes automatically with `/goal`, gateway sessions, TUI sessions, and Desktop chat.
## Configure presets
You can configure named MoA presets from:
- Dashboard → Models → Model Settings → Mixture of Agents
- Desktop app → Settings → Model → Mixture of Agents
- `hermes moa configure [name]`
- `config.yaml`
The config stores explicit provider/model pairs, so you can mix providers and use multiple models from the same provider:
```yaml
moa:
default_preset: default
presets:
default:
reference_models:
- provider: openai-codex
model: gpt-5.5
- provider: openrouter
model: deepseek/deepseek-v4-pro
aggregator:
provider: openrouter
model: anthropic/claude-opus-4.8
# Optional: pin sampling temperatures. When omitted (the default),
# temperature is NOT sent and each model uses its provider default —
# the same behavior as a single-model Hermes agent.
# reference_temperature: 0.6
# aggregator_temperature: 0.4
max_tokens: 4096
enabled: true
```
Default preset:
- reference: `openai-codex:gpt-5.5`
- reference: `openrouter:deepseek/deepseek-v4-pro`
- aggregator / acting model: `openrouter:anthropic/claude-opus-4.8`
### Tuning advisor speed with `reference_max_tokens`
Each turn, MoA runs the reference models (advisors) in parallel and then the
aggregator acts. Advisor generation is the dominant per-turn latency — turn
wall time correlates strongly with how many tokens the advisors emit, because
the turn waits for the slowest advisor to finish writing. By default advisors
are **uncapped** (`reference_max_tokens` unset), so they may write long,
essay-length advice.
Set `reference_max_tokens` on a preset to cap advisor output and give concise
advice instead. The aggregator only needs the gist of each advisor's
judgement, so a cap (e.g. `600`) measurably cuts per-turn wall time with little
quality impact. It caps **advisors only** — the acting aggregator's output (the
user-visible answer) is never capped.
```yaml
moa:
presets:
fast:
reference_models:
- provider: openrouter
model: anthropic/claude-opus-4.8
- provider: openrouter
model: openai/gpt-5.5
aggregator:
provider: openrouter
model: anthropic/claude-opus-4.8
reference_max_tokens: 600 # concise advice → faster turns
```
Leave it unset (or `0`/blank) to keep the prior uncapped behavior.
### Advisor cadence with `fanout`
By default the advisors run **once per user turn** (`fanout: user_turn`) —
they synthesize plan-level advice on the first message of the turn, then the
acting aggregator works through the rest of the tool loop alone. This is the
cheapest cadence: advisor cost does not multiply with the number of tool
calls in a turn. Two alternative cadences trade cost for advice freshness:
- `fanout: per_iteration` — advisors re-run on **every tool iteration**, so
their advice always tracks the latest tool results — at the cost of
multiplying advisor latency and spend by the number of tool calls in a
turn.
- `fanout: every_n:3` — the middle ground: advisors run on the **first**
iteration of each user turn and then every **3rd** tool iteration (any
`N >= 2` works). Iterations in between reuse the cached guidance from the
last advisor run, so the aggregator still gets advice on every step — it is
just refreshed every N steps instead of every step. The counter resets on
each new user message, so every turn starts with fresh advice. The mapping
form `fanout: {mode: every_n, n: 3}` is also accepted and normalized to
the string form.
```yaml
moa:
presets:
fresh:
reference_models:
- provider: openrouter
model: anthropic/claude-opus-4.8
aggregator:
provider: openrouter
model: openai/gpt-5.5
fanout: per_iteration # advisors refresh on every tool iteration
```
Unknown or malformed values fall back to `user_turn`.
:::note Default change
Prior to July 2026 the default cadence was `per_iteration`. The default is
now `user_turn` — the cheapest, lowest-impact cadence — until per-mode
benchmarks justify a costlier default. Presets that want per-step advising
back set `fanout: per_iteration` explicitly.
:::
### Privacy filter for advisor outputs
Advisor outputs can echo sensitive data from the conversation — emails,
formatted phone numbers, API keys, JWTs — into the reference blocks shown in
the UI, saved MoA traces, and the aggregator prompt. `moa.privacy_filter`
(off by default) redacts those surfaces:
```yaml
moa:
privacy_filter: display # or: full
```
- `display` — redacts **user-visible surfaces only**: the labelled reference
blocks rendered in the UI and the records written by `save_traces`. The
aggregator still receives the raw advisor text, so answer quality is
unaffected.
- `full` — additionally redacts the advisor text injected into the
aggregator prompt (and the one-shot `/moa` synthesis input).
Credential shapes (API-key prefixes, JWTs, private keys, DB connection
strings) are masked by Hermes' central secret redactor; the MoA filter adds
email and clearly formatted phone-number redaction on top. Patterns are
deliberately conservative for code-review-style advice: bare digit runs, line
numbers, timestamps, git SHAs, and IP addresses are never touched — only
delimited phone formats like `(555) 123-4567` or `555-123-4567` match.
### Per-slot reasoning effort
Reference and aggregator slots may also set `reasoning_effort`. Use this when
you want the same model to contribute at different depths, or when the
aggregator should think harder than the advisory references. Valid values match
Hermes' normal reasoning controls: `none`, `minimal`, `low`, `medium`, `high`,
`xhigh`, `max`, and `ultra`.
```yaml
moa:
presets:
deep_review:
reference_models:
- provider: openai-codex
model: gpt-5.6-sol
reasoning_effort: low
- provider: openai-codex
model: gpt-5.6-sol
reasoning_effort: xhigh
- provider: xai-oauth
model: grok-4.5
aggregator:
provider: openai-codex
model: gpt-5.6-sol
reasoning_effort: high
```
Omit `reasoning_effort` to use the provider/Hermes default for that slot.
## Terminal preset management
```bash
hermes moa list
hermes moa configure # update the default preset
hermes moa configure review # create or update a named preset
hermes moa delete review
```
## Benchmarks
On HermesBench, a two-model MoA preset — `claude-opus-4.8` aggregating over a `gpt-5.5` reference — outscores either model run on its own:
| Model | HermesBench score |
|---|---|
| **Opus aggregator (opus-4.8 + gpt-5.5 reference) — MoA** | **0.8202** |
| `anthropic/claude-opus-4.8` | 0.7607 |
| `openai/gpt-5.5` | 0.7412 |
The MoA configuration beats its strongest component (opus-4.8) by ~6 points, confirming that aggregating a second perspective lifts quality on hard tasks rather than just averaging the two.
## Prompt caching
MoA is built so the **main conversation's prompt cache is never broken**. Selecting a MoA preset is a normal model selection: it does not mutate past context, swap toolsets, or rebuild the system prompt mid-conversation. Your conversation history, system prompt, and tool schema stay byte-stable, so the cached prefix every other model relies on is preserved exactly as it would be for a plain model. Switching to or away from a MoA preset costs the same cache invalidation as any other `/model` switch — no more.
Both internal call types cache normally:
- **Reference models** receive a trimmed, deterministic view of the conversation (system prompt and tool transcript stripped — see the loop above). Because that view is a stable function of the stable history, a reference model's prompt prefix repeats across iterations and caches normally. References are short advisory calls with no tools.
- **The aggregator** is the acting model. The reference outputs are appended to the *end* of the latest user turn as private guidance. Because that text sits at the tail — below the entire stable prefix (system prompt + prior history) — it does not invalidate any cached prefix: the aggregator gets a cache hit on everything above the injection, and only the freshly appended tail is new. That is exactly how every normal turn behaves, where each new user message is also uncached tail tokens.
So MoA does not sacrifice prompt caching on either call type. Its only real cost is the extra reference calls per iteration — you pay for multiple model perspectives, not for broken caches. The long-lived conversation prefix shared with the rest of Hermes is fully intact.
## Notes
- MoA is no longer listed under `hermes tools`; there is no `moa` toolset to enable.
- Setting `enabled: false` on a preset disables the reference fan-out for that preset: the aggregator acts alone, exactly as if you selected it as a plain model. This is the per-preset off switch surfaced in the dashboard and desktop settings.
- A preset's aggregator cannot be another MoA preset. Recursive MoA trees are intentionally blocked.
- Credential failures on one reference model do not abort the turn. Hermes includes the failure in the reference context and continues with whatever models returned.
- MoA increases model-call count. A single model iteration can involve multiple reference calls plus the aggregator call.
website/docs/user-guide/features/overview.md
+57
View File
@@ -0,0 +1,57 @@
---
title: "Features Overview"
sidebar_label: "Overview"
sidebar_position: 1
---
# Features Overview
Hermes Agent includes a rich set of capabilities that extend far beyond basic chat. From persistent memory and file-aware context to browser automation and voice conversations, these features work together to make Hermes a powerful autonomous assistant.
:::tip Don't know where to start?
`hermes setup --portal` covers a model provider plus all four Tool Gateway tools (web search, image generation, TTS, browser) in one command. See [Nous Portal](/integrations/nous-portal).
:::
## Core
- **[Tools & Toolsets](tools.md)** — Tools are functions that extend the agent's capabilities. They're organized into logical toolsets that can be enabled or disabled per platform, covering web search, terminal execution, file editing, memory, delegation, and more.
- **[Skills System](skills.md)** — On-demand knowledge documents the agent can load when needed. Skills follow a progressive disclosure pattern to minimize token usage and are compatible with the [agentskills.io](https://agentskills.io/specification) open standard.
- **[Persistent Memory](memory.md)** — Bounded, curated memory that persists across sessions. Hermes remembers your preferences, projects, environment, and things it has learned via `MEMORY.md` and `USER.md`.
- **[Context Files](context-files.md)** — Hermes automatically discovers and loads project context files (`.hermes.md`, `AGENTS.md`, `CLAUDE.md`, `SOUL.md`, `.cursorrules`) that shape how it behaves in your project.
- **[Context References](context-references.md)** — Type `@` followed by a reference to inject files, folders, git diffs, and URLs directly into your messages. Hermes expands the reference inline and appends the content automatically.
- **[Checkpoints](../checkpoints-and-rollback.md)** — Hermes automatically snapshots your working directory before making file changes, giving you a safety net to roll back with `/rollback` if something goes wrong.
## Automation
- **[Scheduled Tasks (Cron)](cron.md)** — Schedule tasks to run automatically with natural language or cron expressions. Jobs can attach skills, deliver results to any platform, and support pause/resume/edit operations.
- **[Subagent Delegation](delegation.md)** — The `delegate_task` tool spawns child agent instances with isolated context, restricted toolsets, and their own terminal sessions. Run 3 concurrent subagents by default (configurable) for parallel workstreams.
- **[Code Execution](code-execution.md)** — The `execute_code` tool lets the agent write Python scripts that call Hermes tools programmatically, collapsing multi-step workflows into a single LLM turn via sandboxed RPC execution.
- **[Event Hooks](hooks.md)** — Run custom code at key lifecycle points. Gateway hooks handle logging, alerts, and webhooks; plugin hooks handle tool interception, metrics, and guardrails.
- **[Batch Processing](batch-processing.md)** — Run the Hermes agent across hundreds or thousands of prompts in parallel, generating structured ShareGPT-format trajectory data for training data generation or evaluation.
## Media & Web
- **[Voice Mode](voice-mode.md)** — Full voice interaction across CLI and messaging platforms. Talk to the agent using your microphone, hear spoken replies, and have live voice conversations in Discord voice channels.
- **[Wake Word](wake-word.md)** — Hands-free "Hey Hermes" trigger for the CLI, TUI, and desktop app. An on-device hotword listener starts a voice session when you speak the wake phrase.
- **[Browser Automation](browser.md)** — Full browser automation with multiple backends: Browserbase cloud, Browser Use cloud, local Chrome/Brave/Chromium/Edge via CDP, or local Chromium. Navigate websites, fill forms, and extract information.
- **[Vision & Image Paste](vision.md)** — Multimodal vision support. Paste images from your clipboard into the CLI and ask the agent to analyze, describe, or work with them using any vision-capable model.
- **[Image Generation](image-generation.md)** — Generate images from text prompts using FAL.ai. Eleven models supported (FLUX 2 Klein/Pro, GPT-Image 1.5/2, Nano Banana Pro, Ideogram V3, Recraft V4 Pro, Qwen, Z-Image Turbo, Krea V2 Medium/Large); pick one via `hermes tools`.
- **[Voice & TTS](tts.md)** — Text-to-speech output and voice message transcription across all messaging platforms, with ten native provider options: Edge TTS (free), ElevenLabs, OpenAI TTS, MiniMax, Mistral Voxtral, Google Gemini, xAI, NeuTTS, KittenTTS, and Piper — plus custom command providers for any local TTS CLI.
## Integrations
- **[MCP Integration](mcp.md)** — Connect to any MCP server via stdio or HTTP transport. Access external tools from GitHub, databases, file systems, and internal APIs without writing native Hermes tools. Includes per-server tool filtering and sampling support.
- **[Provider Routing](provider-routing.md)** — Fine-grained control over which AI providers handle your requests. Optimize for cost, speed, or quality with sorting, whitelists, blacklists, and priority ordering.
- **[Fallback Providers](fallback-providers.md)** — Automatic failover to backup LLM providers when your primary model encounters errors, including independent fallback for auxiliary tasks like vision and compression.
- **[Credential Pools](credential-pools.md)** — Distribute API calls across multiple keys for the same provider. Automatic rotation on rate limits or failures.
- **[Prompt caching](../configuration#prompt-caching)** — Built-in cross-session 1-hour prefix cache for Claude on native Anthropic, OpenRouter, and Nous Portal. Always-on; no configuration required.
- **[Memory Providers](memory-providers.md)** — Plug in external memory backends (Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, Supermemory) for cross-session user modeling and personalization beyond the built-in memory system.
- **[API Server](api-server.md)** — Expose Hermes as an OpenAI-compatible HTTP endpoint. Connect any frontend that speaks the OpenAI format — Open WebUI, LobeChat, LibreChat, and more.
- **[IDE Integration (ACP)](acp.md)** — Use Hermes inside ACP-compatible editors such as VS Code, Zed, and JetBrains. Chat, tool activity, file diffs, and terminal commands render inside your editor.
- **[Batch Processing](batch-processing.md)** — Run the agent over many prompts or tasks in parallel from the CLI, with structured outputs and trajectory capture suitable for evals or downstream training pipelines.
## Customization
- **[Personality & SOUL.md](personality.md)** — Fully customizable agent personality. `SOUL.md` is the primary identity file — the first thing in the system prompt — and you can swap in built-in or custom `/personality` presets per session.
- **[Skins & Themes](skins.md)** — Customize the CLI's visual presentation: banner colors, spinner faces and verbs, response-box labels, branding text, and the tool activity prefix.
- **[Plugins](plugins.md)** — Add custom tools, hooks, and integrations without modifying core code. Three plugin types: general plugins (tools/hooks), memory providers (cross-session knowledge), and context engines (alternative context management). Managed via the unified `hermes plugins` interactive UI.
website/docs/user-guide/features/personality.md
+289
View File
@@ -0,0 +1,289 @@
---
sidebar_position: 9
title: "Personality & SOUL.md"
description: "Customize Hermes Agent's personality with a global SOUL.md, built-in personalities, and custom persona definitions"
---
# Personality & SOUL.md
Hermes Agent's personality is fully customizable. `SOUL.md` is the **primary identity** — it's the first thing in the system prompt and defines who the agent is.
- `SOUL.md` — a durable persona file that lives in `HERMES_HOME` and serves as the agent's identity (slot #1 in the system prompt)
- built-in or custom `/personality` presets — session-level system-prompt overlays
If you want to change who Hermes is — or replace it with an entirely different agent persona — edit `SOUL.md`.
## How SOUL.md works now
Hermes now seeds a default `SOUL.md` automatically in:
```text
~/.hermes/SOUL.md
```
More precisely, it uses the current instance's `HERMES_HOME`, so if you run Hermes with a custom home directory, it will use:
```text
$HERMES_HOME/SOUL.md
```
### Important behavior
- **SOUL.md is the agent's primary identity.** It occupies slot #1 in the system prompt, replacing the hardcoded default identity.
- Hermes creates a starter `SOUL.md` automatically if one does not exist yet
- Existing user `SOUL.md` files are never overwritten
- Hermes loads `SOUL.md` only from `HERMES_HOME`
- Hermes does not look in the current working directory for `SOUL.md`
- If `SOUL.md` exists but is empty, or cannot be loaded, Hermes falls back to a built-in default identity
- If `SOUL.md` has content, that content is injected verbatim after security scanning and truncation
- SOUL.md is **not** duplicated in the context files section — it appears only once, as the identity
That makes `SOUL.md` a true per-user or per-instance identity, not just an additive layer.
## Why this design
This keeps personality predictable.
If Hermes loaded `SOUL.md` from whatever directory you happened to launch it in, your personality could change unexpectedly between projects. By loading only from `HERMES_HOME`, the personality belongs to the Hermes instance itself.
That also makes it easier to teach users:
- "Edit `~/.hermes/SOUL.md` to change Hermes' default personality."
## Where to edit it
For most users:
```bash
~/.hermes/SOUL.md
```
If you use a custom home:
```bash
$HERMES_HOME/SOUL.md
```
## What should go in SOUL.md?
Use it for durable voice and personality guidance, such as:
- tone
- communication style
- level of directness
- default interaction style
- what to avoid stylistically
- how Hermes should handle uncertainty, disagreement, or ambiguity
Use it less for:
- one-off project instructions
- file paths
- repo conventions
- temporary workflow details
Those belong in `AGENTS.md`, not `SOUL.md`.
## Good SOUL.md content
A good SOUL file is:
- stable across contexts
- broad enough to apply in many conversations
- specific enough to materially shape the voice
- focused on communication and identity, not task-specific instructions
### Example
```markdown
# Personality
You are a pragmatic senior engineer with strong taste.
You optimize for truth, clarity, and usefulness over politeness theater.
## Style
- Be direct without being cold
- Prefer substance over filler
- Push back when something is a bad idea
- Admit uncertainty plainly
- Keep explanations compact unless depth is useful
## What to avoid
- Sycophancy
- Hype language
- Repeating the user's framing if it's wrong
- Overexplaining obvious things
## Technical posture
- Prefer simple systems over clever systems
- Care about operational reality, not idealized architecture
- Treat edge cases as part of the design, not cleanup
```
## What Hermes injects into the prompt
`SOUL.md` content goes directly into slot #1 of the system prompt — the agent identity position. No wrapper language is added around it.
The content goes through:
- prompt-injection scanning
- truncation if it is too large
If the file is empty, whitespace-only, or cannot be read, Hermes falls back to a built-in default identity ("You are Hermes Agent, built by Nous Research. Be direct: match the length of your reply to the weight of the ask..."). This fallback also applies when `skip_context_files` is set (e.g., in subagent/delegation contexts).
## Security scanning
`SOUL.md` is scanned like other context-bearing files for prompt injection patterns before inclusion.
That means you should still keep it focused on persona/voice rather than trying to sneak in strange meta-instructions.
## SOUL.md vs AGENTS.md
This is the most important distinction.
### SOUL.md
Use for:
- identity
- tone
- style
- communication defaults
- personality-level behavior
### AGENTS.md
Use for:
- project architecture
- coding conventions
- tool preferences
- repo-specific workflows
- commands, ports, paths, deployment notes
A useful rule:
- if it should follow you everywhere, it belongs in `SOUL.md`
- if it belongs to a project, it belongs in `AGENTS.md`
## SOUL.md vs `/personality`
`SOUL.md` is your durable default personality.
`/personality` is a session-level overlay that changes or supplements the current system prompt.
So:
- `SOUL.md` = baseline voice
- `/personality` = temporary mode switch
Examples:
- keep a pragmatic default SOUL, then use `/personality teacher` for a tutoring conversation
- keep a concise SOUL, then use `/personality creative` for brainstorming
## Built-in personalities
Hermes ships with built-in personalities you can switch to with `/personality`.
| Name | Description |
|------|-------------|
| **helpful** | Friendly, general-purpose assistant |
| **concise** | Brief, to-the-point responses |
| **technical** | Detailed, accurate technical expert |
| **creative** | Innovative, outside-the-box thinking |
| **teacher** | Patient educator with clear examples |
| **kawaii** | Cute expressions, sparkles, and enthusiasm ★ |
| **catgirl** | Neko-chan with cat-like expressions, nya~ |
| **pirate** | Captain Hermes, tech-savvy buccaneer |
| **shakespeare** | Bardic prose with dramatic flair |
| **surfer** | Totally chill bro vibes |
| **noir** | Hard-boiled detective narration |
| **uwu** | Maximum cute with uwu-speak |
| **philosopher** | Deep contemplation on every query |
| **hype** | MAXIMUM ENERGY AND ENTHUSIASM!!! |
## Switching personalities with commands
### CLI
```text
/personality
/personality concise
/personality technical
```
### Messaging platforms
```text
/personality teacher
```
These are convenient overlays, but your global `SOUL.md` still gives Hermes its persistent default personality unless the overlay meaningfully changes it.
## Custom personalities in config
Built-in personalities are always available on every surface (CLI, messaging platforms, TUI, and the desktop app). You can add your own — or override a built-in by reusing its name — in `~/.hermes/config.yaml` under `agent.personalities`.
```yaml
agent:
personalities:
codereviewer: >
You are a meticulous code reviewer. Identify bugs, security issues,
performance concerns, and unclear design choices. Be precise and constructive.
```
Then switch to it with:
```text
/personality codereviewer
```
Your selection is stored as a name in `display.personality`. Personalities never touch `agent.system_prompt` — that field is reserved for a manual system prompt you write yourself, and it applies only when no personality is selected.
## Resetting to the default
To cancel the active personality overlay and return to base behavior (your `SOUL.md` persona, plus `agent.system_prompt` if you set one), use any of:
```text
/personality none
/personality default
/personality neutral
```
All three clear the selection (`display.personality`) and the change takes effect on your next message. Running `/personality` with no arguments also lists `none` alongside the available presets and marks the active one.
:::note One-time reset on upgrade
Older Hermes versions saved personality state inconsistently across surfaces, which could re-enable a personality you had previously turned off. On your first run after upgrading, any saved personality selection is reset to `none` once (the migration prints which personality was cleared). Re-enable it with `/personality <name>` if you still want it. Manual `agent.system_prompt` text is never touched.
:::
## Recommended workflow
A strong default setup is:
1. Keep a thoughtful global `SOUL.md` in `~/.hermes/SOUL.md`
2. Put project instructions in `AGENTS.md`
3. Use `/personality` only when you want a temporary mode shift
That gives you:
- a stable voice
- project-specific behavior where it belongs
- temporary control when needed
## How personality interacts with the full prompt
At a high level, the prompt stack includes:
1. **SOUL.md** (agent identity — or built-in fallback if SOUL.md is unavailable)
2. tool-aware behavior guidance
3. memory/user context
4. skills guidance
5. context files (`AGENTS.md`, `.cursorrules`)
6. timestamp
7. platform-specific formatting hints
8. optional system-prompt overlays such as `/personality`
`SOUL.md` is the foundation — everything else builds on top of it.
## Related docs
- [Context Files](/user-guide/features/context-files)
- [Configuration](/user-guide/configuration)
- [Tips & Best Practices](/guides/tips)
- [SOUL.md Guide](/guides/use-soul-with-hermes)
## CLI appearance vs conversational personality
Conversational personality and CLI appearance are separate:
- `SOUL.md`, `agent.system_prompt`, and `/personality` affect how Hermes speaks
- `display.skin` and `/skin` affect how Hermes looks in the terminal
For terminal appearance, see [Skins & Themes](./skins.md).
website/docs/user-guide/features/pets.md
+236
View File
@@ -0,0 +1,236 @@
---
sidebar_position: 11
title: "Pets (Petdex Mascots)"
description: "Adopt an animated mascot that reacts to agent activity across the CLI, TUI, and desktop app"
---
# Pets
Hermes can show an animated **pet** — a small mascot sprite that reacts to what
the agent is doing (idle, running a tool, thinking, finishing, failing) across
the **CLI**, **TUI**, and **desktop app**. Pets come from the public
[petdex](https://github.com/crafter-station/petdex) gallery.
Pets are purely cosmetic. They have **no effect on prompt caching, tokens, or
the agent's behavior** — the sprite is a display concern only. The feature is
**off by default** and stays dormant until you install and select a pet.
## How it works
- Pets are installed into your profile's `pets/` directory
(`<HERMES_HOME>/pets/<slug>/`), so each [profile](../profiles.md) keeps its
own set.
- Selecting a pet writes `display.pet.slug` and `display.pet.enabled` to
`config.yaml` — nothing is stored as a secret or env var.
- Each surface watches the activity it already tracks and maps it to one of six
animation states. The mapping lives in one place so every surface behaves the
same:
| Agent activity | Pet state |
| --- | --- |
| A tool/turn just failed | `failed` |
| A plan finished (all todos done) | `jump` (celebrate) |
| A turn finished cleanly | `wave` |
| A tool is executing | `run` |
| The model is thinking/reading | `review` |
| Turn in flight (unspecified) | `run` |
| Blocked on you (a clarify/approval prompt is open) | `waiting` (falls back to `idle` on legacy 8-row sheets) |
| Nothing happening | `idle` |
## Rendering
In the terminal (CLI/TUI), Hermes renders the sprite at full fidelity when your
terminal supports a graphics protocol (**kitty**, **Ghostty**, **WezTerm**,
**iTerm2**, or **sixel**). Otherwise it falls back automatically to a truecolor
Unicode **half-block** rendering. Inside a pipe or redirect (no TTY), terminal
rendering is disabled by design.
The desktop app draws the pet as a floating sprite on a canvas and toggles it
from **Settings → Appearance**.
## Quick start (CLI)
```bash
# Browse the gallery (filter by substring)
hermes pets list
hermes pets list cat
# Install a pet and make it active in one step
hermes pets install boba --select
# Preview / animate it in your terminal (Ctrl+C to stop)
hermes pets show
# Check your setup
hermes pets doctor
```
## `hermes pets` commands
| Goal | Command |
| --- | --- |
| Browse the gallery | `hermes pets list [query] [--limit N]` |
| List installed pets | `hermes pets list --installed` |
| Install a pet | `hermes pets install <slug> [--select] [--force]` |
| Set the active pet | `hermes pets select [slug]` (omit slug for a picker) |
| Resize the pet everywhere | `hermes pets scale <factor>` (e.g. `0.5`, clamped 0.1–3.0) |
| Preview/animate | `hermes pets show [slug] [--state <s>] [--cycle] [--once] [--mode <m>] [--scale <f>]` |
| Disable the pet | `hermes pets off` |
| Remove an installed pet | `hermes pets remove <slug>` |
| Diagnose setup | `hermes pets doctor` |
`hermes pets show` flags:
- `--state` — play a single state (`idle`, `wave`, `run`, `failed`, `review`,
`jump`).
- `--cycle` — cycle through every state.
- `--once` — play once instead of looping.
- `--mode` — override the render protocol (`kitty`, `iterm`, `sixel`,
`unicode`, `auto`).
- `--scale` — override the on-screen scale (`0` = use config).
## `/pet` slash command
Inside the CLI and TUI you can manage the pet without leaving the session:
- `/pet` — toggle the pet on/off (adopts the first installed pet if none is
active).
- `/pet list` — browse the gallery.
- `/pet scale <factor>` — resize the pet everywhere (e.g. `/pet scale 0.5`).
- `/pet <slug>` — adopt a specific pet.
- `/pet off` — disable the pet.
In the TUI, `/pet list` opens an interactive picker overlay; in the desktop app
it opens the Cmd+K pet palette.
## Generating a pet (`/hatch`)
Beyond installing pre-made pets from the gallery, Hermes can **generate a brand-new pet** from a text description — its own AI sprite-generation pipeline.
- CLI/TUI: `/hatch <description>` (alias `/generate-pet`), or `hermes pets` → the generate flow.
- Desktop app: the Pokédex-style **generate** UI — an animated egg, hatch FX, and a draft picker.
How generation works (a two-step, cost-bounded flow):
1. **Base drafts** — a handful of cheap, prompt-only "what should this pet look like" variants are generated. You pick one, or remix/retry for a fresh round.
2. **Hatch** — the chosen base is used as a reference image to generate one grounded animation row per Hermes state (idle, thinking, tool use, etc.), which are deterministically sliced into frames and packed into a standard petdex/Codex atlas (8×9 grid of 192×208 cells). The result is a valid spritesheet you keep — and could `petdex submit`.
### Image backend
Generation uses the active [image-generation provider](/user-guide/features/image-generation), but it requires **reference-image grounding** so each animation row stays the same character as the base. Reference-capable backends: **Nous Portal**, **OpenRouter**, **OpenAI** (`gpt-image-2`), and **Krea**. OpenRouter/Nous run a quality-first model chain by default.
- Resolution order prefers Nous Portal → OpenAI → OpenRouter.
- If no reference-capable backend is configured, generation surfaces an actionable error pointing you to `hermes tools` → Image Generation. (Installing/adopting existing gallery pets needs no image backend.)
- Override the backend with the `HERMES_PET_IMAGE_PROVIDER` env var (e.g. `HERMES_PET_IMAGE_PROVIDER=openrouter`).
## Desktop app
In the desktop app you can manage the pet two ways:
- **Cmd+K → "Pets…"** — browse, search, adopt, and toggle pets without leaving
the keyboard (mirrors the theme picker).
- **Settings → Appearance** — the same gallery plus a **size slider** that
resizes the floating mascot live as you drag.
Both adopt/toggle/resize the floating mascot in place — size changes apply
instantly; adopting a new pet lights it up within a moment.
### Roaming
Settings → Appearance has a **Roam** toggle: when enabled, the pet wanders the
window on its own while the agent is idle — walking surfaces, pausing, and
hopping between spots. Roaming only runs while the pet is in-window, active,
and the agent is at rest; any agent-driven state (working, celebrating)
immediately takes over. The toggle is off by default and persists across
restarts.
### Alt+wheel resizing
Hold **Alt** and scroll the mouse wheel over the pet to resize it in place —
in the app window and on the popped-out overlay alike. The overlay zooms
toward the cursor position and the resulting scale is persisted, so it
survives restarts and stays in sync with the in-app pet.
### Vibe reactions
Say something nice to the agent — "good bot", "thank you", "ily", `<3`, or a
heart emoji — and the pet reacts with floating hearts (desktop) or a heart
flash (CLI/TUI). Detection is a curated, token-free lexicon matched locally on
each user message (no model call); it fires on affection and gratitude aimed at
the agent, not general positive sentiment. All surfaces — CLI pet, TUI, desktop
floating pet, and the pop-out overlay — react off the same signal.
### Pop-out overlay
**Shift-click** the floating pet to pop it out into its own transparent,
always-on-top desktop window. Out there it stays visible while Hermes is
minimized (Codex-style), so a glance tells you what the agent is doing.
Gestures once it's popped out:
| Gesture | Action |
| --- | --- |
| **Drag** | Move the pet anywhere on screen, even outside the app. Its spot and in/out state persist across restarts. |
| **Single-click** | Open a mini composer to send a prompt to the most recent session — without surfacing the app. |
| **Double-click** | Toggle the app window: minimize it if it's up front, restore it if it's hidden. |
| **Shift-click** | Pop the pet back into the window. |
| **Mail icon** | Appears only when a turn finished while you were away; click to raise the app on the most recent thread (and mark it read). |
Only the popped-out pet shows a **speech bubble** (`working…`, `thinking…`,
`your turn`, …) — in-window the app itself is the surface, so the pet stays
quiet there.
The overlay is a pure puppet of the in-app pet — it carries no separate gateway
connection and never appears in the dock or app switcher.
## Configuration
All settings live under `display.pet` in `config.yaml`:
```yaml
display:
pet:
enabled: false # master on/off (true once you select a pet)
slug: "" # active pet; empty = first installed
render_mode: auto # auto | kitty | iterm | sixel | unicode | off
scale: 0.33 # master size knob (relative to native 192x208 frames)
unicode_cols: 0 # hard override for terminal width (0 = derive from scale)
```
- **`scale`** is the single master size knob. One number shrinks every surface:
the desktop canvas scales its pixels by it, and the CLI/TUI derive their
terminal column width from it. The half-block fallback clamps to a legibility
floor — it can't shrink as far as true-pixel kitty/GUI rendering without
turning to mush, so the same `scale` looks crisp under kitty but is floored in
half-blocks.
- **`render_mode: auto`** detects kitty/iTerm2/sixel and falls back to unicode
half-blocks. Set it explicitly to force a protocol or `off` to disable
terminal rendering while keeping the pet on the desktop.
- **`unicode_cols`** pins the terminal column width independently of `scale`;
leave it at `0` to derive width from `scale`.
## Troubleshooting
Run `hermes pets doctor` — it reports:
- the pets directory and which pets are installed,
- `display.pet.enabled`, `display.pet.slug`, and the resolved active pet,
- the configured `render_mode`, the detected terminal graphics protocol, and the
effective mode for a TTY,
- whether Pillow (used for sprite decoding) is importable.
It prints `✓ ready` once a pet is installed, selected, enabled, and Pillow is
available.
Common gotchas:
- A pet only shows once one is **installed AND selected** (`enabled: true`).
- Inside a pipe/redirect (no TTY), terminal rendering is disabled by design.
- The petdex npm CLI installs to `~/.codex/pets`; Hermes uses its own
profile-scoped `<HERMES_HOME>/pets/` instead — install through `hermes pets`.
## See also
- The [`hermes-agent` skill](../skills/bundled/autonomous-ai-agents/autonomous-ai-agents-hermes-agent.md)
lets the agent install and switch pets for you on request (see its
`references/petdex.md`).
website/docs/user-guide/features/plugins.md
+806
View File
@@ -0,0 +1,806 @@
---
sidebar_position: 11
sidebar_label: "Plugins"
title: "Plugins"
description: "Extend Hermes with custom tools, hooks, and integrations via the plugin system"
---
# Plugins
Hermes has a plugin system for adding custom tools, hooks, and integrations without modifying core code.
If you want to create a custom tool for yourself, your team, or one project,
this is usually the right path. The developer guide's
[Adding Tools](/developer-guide/adding-tools) page is for built-in Hermes
core tools that live in `tools/` and `toolsets.py`.
**→ [Build a Hermes Plugin](/developer-guide/plugins)** — step-by-step guide with a complete working example.
## Quick overview
Drop a directory into `~/.hermes/plugins/` with a `plugin.yaml` and Python code:
```
~/.hermes/plugins/my-plugin/
├── plugin.yaml # manifest
├── __init__.py # register() — wires schemas to handlers
├── schemas.py # tool schemas (what the LLM sees)
└── tools.py # tool handlers (what runs when called)
```
Start Hermes — your tools appear alongside built-in tools. The model can call them immediately.
### Minimal working example
Here is a complete plugin that adds a `hello_world` tool and logs every tool call via a hook.
**`~/.hermes/plugins/hello-world/plugin.yaml`**
```yaml
name: hello-world
version: "1.0"
description: A minimal example plugin
```
**`~/.hermes/plugins/hello-world/__init__.py`**
```python
"""Minimal Hermes plugin — registers a tool and a hook."""
import json
def register(ctx):
# --- Tool: hello_world ---
schema = {
"name": "hello_world",
"description": "Returns a friendly greeting for the given name.",
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Name to greet",
}
},
"required": ["name"],
},
}
def handle_hello(params, **kwargs):
del kwargs
name = params.get("name", "World")
return json.dumps({"success": True, "greeting": f"Hello, {name}!"})
ctx.register_tool(
name="hello_world",
toolset="hello_world",
schema=schema,
handler=handle_hello,
)
# --- Hook: log every tool call ---
def on_tool_call(tool_name, params, result):
print(f"[hello-world] tool called: {tool_name}")
ctx.register_hook("post_tool_call", on_tool_call)
```
Drop both files into `~/.hermes/plugins/hello-world/`, restart Hermes, and the model can immediately call `hello_world`. The hook prints a log line after every tool invocation.
The model-facing tool description belongs in `schema["description"]`. The optional `ctx.register_tool(description=...)` value is separate `ToolEntry` registry metadata: when omitted, it defaults to the schema description, but Hermes does not copy it back into a schema that lacks `description`. Prefer defining the text once in the schema. If you provide both values, keep them synchronized; the model sees the schema value.
Project-local plugins under `./.hermes/plugins/` are disabled by default. Enable them only for trusted repositories by setting `HERMES_ENABLE_PROJECT_PLUGINS=true` before starting Hermes.
## What plugins can do
Every `ctx.*` API below is available inside a plugin's `register(ctx)` function.
| Capability | How |
|-----------|-----|
| Add tools | `ctx.register_tool(name=..., toolset=..., schema=..., handler=...)` |
| Add hooks | `ctx.register_hook("post_tool_call", callback)` |
| Add slash commands | `ctx.register_command(name, handler, description)` — adds `/name` in CLI and gateway sessions |
| Dispatch tools from commands | `ctx.dispatch_tool(name, args)` — invokes a registered tool with parent-agent context auto-wired |
| Add CLI commands | `ctx.register_cli_command(name, help, setup_fn, handler_fn)` — adds `hermes <plugin> <subcommand>` |
| Inject messages | `ctx.inject_message(content, role="user", session_key=...)` - see [Injecting Messages](#injecting-messages) |
| Ship data files | `Path(__file__).parent / "data" / "file.yaml"` |
| Bundle skills | `ctx.register_skill(name, path)` — namespaced as `plugin:skill`, loaded via `skill_view("plugin:skill")` |
| Gate on env vars | `requires_env: [API_KEY]` in plugin.yaml — prompted during `hermes plugins install` |
| Distribute via pip | `[project.entry-points."hermes_agent.plugins"]` |
| Register a gateway platform (Discord, Telegram, IRC, …) | `ctx.register_platform(name, label, adapter_factory, check_fn, ...)` — see [Adding Platform Adapters](/developer-guide/adding-platform-adapters) |
| Register an image-generation backend | `ctx.register_image_gen_provider(provider)` — see [Image Generation Provider Plugins](/developer-guide/image-gen-provider-plugin) |
| Register a video-generation backend | `ctx.register_video_gen_provider(provider)` — see [Video Generation Provider Plugins](/developer-guide/video-gen-provider-plugin) |
| Register a context-compression engine | `ctx.register_context_engine(engine)` — see [Context Engine Plugins](/developer-guide/context-engine-plugin) |
| Register a terminal execution backend (cloud sandbox) | `ctx.register_terminal_environment_provider(provider)` — see [Terminal Environment Plugins](/developer-guide/terminal-environment-plugin) |
| Route human approval prompts | `ctx.register_approval_transport(name, present_fn)` — see [Approval transports](#approval-transports) |
| Register a memory backend | Subclass `MemoryProvider` in `plugins/memory/<name>/__init__.py` — see [Memory Provider Plugins](/developer-guide/memory-provider-plugin) (uses a separate discovery system) |
| Run a host-owned LLM call | `ctx.llm.complete(...)` / `ctx.llm.complete_structured(...)` — borrow the user's active model + auth for a one-shot completion with optional JSON schema validation. See [Plugin LLM Access](/developer-guide/plugin-llm-access) |
| Call an MCP tool (capability-gated) | `ctx.call_mcp(server, tool, arguments, timeout=30)` — see [Calling MCP servers from plugins](#calling-mcp-servers-from-plugins) |
| Register an inference backend (LLM provider) | `register_provider(ProviderProfile(...))` in `plugins/model-providers/<name>/__init__.py` — see [Model Provider Plugins](/developer-guide/model-provider-plugin) (uses a separate discovery system) |
## Plugin discovery
| Source | Path | Use case |
|--------|------|----------|
| Bundled | `<repo>/plugins/` | Ships with Hermes — see [Built-in Plugins](/user-guide/features/built-in-plugins) |
| User | `~/.hermes/plugins/` | Personal plugins |
| Project | `.hermes/plugins/` | Project-specific plugins (requires `HERMES_ENABLE_PROJECT_PLUGINS=true`) |
| pip | `hermes_agent.plugins` entry_points | Distributed packages |
| Nix | `services.hermes-agent.extraPlugins` / `extraPythonPackages` | NixOS declarative installs — see [Nix Setup](/getting-started/nix-setup#plugins) |
Later sources override earlier ones on name collision, so a user plugin with the same name as a bundled plugin replaces it.
### Plugin sub-categories
Within each source, Hermes also recognizes sub-category directories that route plugins to specialized discovery systems:
| Sub-directory | What it holds | Discovery system |
|---|---|---|
| `plugins/` (root) | General plugins — tools, hooks, slash commands, CLI commands, bundled skills | `PluginManager` (kind: `standalone` or `backend`) |
| `plugins/platforms/<name>/` | Gateway channel adapters (`ctx.register_platform()`) | `PluginManager` (kind: `platform`, one level deeper) |
| `plugins/image_gen/<name>/` | Image-generation backends (`ctx.register_image_gen_provider()`) | `PluginManager` (kind: `backend`, one level deeper) |
| `plugins/memory/<name>/` | Memory providers (subclass `MemoryProvider`) | **Own loader** in `plugins/memory/__init__.py` (kind: `exclusive` — one active at a time) |
| `plugins/context_engine/<name>/` | Context-compression engines (`ctx.register_context_engine()`) | **Own loader** in `plugins/context_engine/__init__.py` (one active at a time) |
| `plugins/model-providers/<name>/` | LLM provider profiles (`register_provider(ProviderProfile(...))`) | **Own loader** in `providers/__init__.py` (lazily scanned on first `get_provider_profile()` call) |
User plugins at `~/.hermes/plugins/model-providers/<name>/` and `~/.hermes/plugins/memory/<name>/` override bundled plugins of the same name — last-writer-wins in `register_provider()` / `register_memory_provider()`. Drop a directory in, and it replaces the built-in without any repo edits.
## Plugins are opt-in (with a few exceptions)
**General plugins and user-installed backends are disabled by default** — discovery finds them (so they show up in `hermes plugins` and `/plugins`), but nothing with hooks or tools loads until you add the plugin's name to `plugins.enabled` in `~/.hermes/config.yaml`. This stops third-party code from running without your explicit consent.
```yaml
plugins:
enabled:
- my-tool-plugin
- disk-cleanup
disabled: # optional deny-list — always wins if a name appears in both
- noisy-plugin
# Optional: wall-clock cap (seconds) for timeout-bounded in-process Python
# plugin hook callbacks (hot-path observers + pre_tool_call). Default 30;
# set 0 to disable; values above 600 are clamped. Timed-out pre_tool_call
# callbacks fail closed (block the tool). Caller-thread hooks such as
# subagent_stop are never moved onto a timeout worker.
# Shell hooks keep their own per-entry timeout under the top-level hooks: key.
hook_callback_timeout: 30
```
Three ways to flip state:
```bash
hermes plugins # interactive toggle (space to check/uncheck)
hermes plugins enable <name> # add to allow-list
hermes plugins disable <name> # remove from allow-list + add to disabled
```
After `hermes plugins install owner/repo`, you're asked `Enable 'name' now? [y/N]` — defaults to no. Skip the prompt for scripted installs with `--enable` or `--no-enable`.
For a reproducible install, pin a full immutable commit (tags, branches, and
abbreviated SHAs are not accepted):
```bash
hermes plugins install owner/repo --ref 0123456789abcdef0123456789abcdef01234567
```
Hermes checks out the commit detached, verifies that `HEAD` exactly matches the
requested SHA, and records the canonical source, installed revision, and pin
status in the current profile. `hermes plugins update` refuses to move a pinned
plugin; choose a new exact commit explicitly with
`hermes plugins install <source> --force --ref <new-commit>`. The
profile-local install metadata contains no config values, environment values,
secrets, or capability grants.
### What the allow-list does NOT gate
Several categories of plugin bypass `plugins.enabled` — they're part of Hermes' built-in surface and would break basic functionality if gated off by default:
| Plugin kind | How it's activated instead |
|---|---|
| **Bundled platform plugins** (IRC, Teams, etc. under `plugins/platforms/`) | Auto-loaded so every shipped gateway channel is available. The actual channel turns on via `gateway.platforms.<name>.enabled` in `config.yaml`. |
| **Bundled backends** (image-gen providers under `plugins/image_gen/`, etc.) | Auto-loaded so the default backend "just works". Selection happens via `<category>.provider` in `config.yaml` (e.g. `image_gen.provider: openai`). |
| **Memory providers** (`plugins/memory/`) | All discovered; exactly one is active, chosen by `memory.provider` in `config.yaml`. |
| **Context engines** (`plugins/context_engine/`) | All discovered; one is active, chosen by `context.engine` in `config.yaml`. |
| **Model providers** (`plugins/model-providers/`) | All bundled providers under `plugins/model-providers/` discover and register at the first `get_provider_profile()` call. The user picks one at a time via `--provider` or `config.yaml`. |
| **Pip-installed `backend` plugins** | Opt-in via `plugins.enabled` (same as general plugins). |
| **User-installed platforms** (under `~/.hermes/plugins/platforms/`) | Opt-in via `plugins.enabled` — third-party gateway adapters need explicit consent. |
In short: **bundled "always-works" infrastructure loads automatically; third-party general plugins are opt-in.** The `plugins.enabled` allow-list is the gate specifically for arbitrary code a user drops into `~/.hermes/plugins/`.
### Approval transports
An approval transport changes **where a human sees and answers** an existing
Hermes tool-approval request. It does not decide whether a command needs
approval and it is not an authorization-policy API.
```python
def present(request):
# Deliver request.command and request.description to your UI, wait for
# its authenticated human response, then return a request-bound decision.
choice = send_to_my_ui_and_wait(request) # once/session/always/deny
return request.respond(choice)
def register(ctx):
ctx.register_approval_transport("my-ui", present)
```
`present` may be synchronous or async. Hermes runs it on a bounded worker and
enforces the canonical `approvals.timeout` even if the plugin does not. The
request is immutable and contains redacted display text, its host presentation
class (`cli` or `gateway`), the host timeout, allowed choices, and an opaque
request ID/digest.
Return the result of
`request.respond(choice)`; unbound dictionaries and stale or changed request
IDs/digests are rejected. A plugin cannot return a scope that the host did not
offer (for example, `always` on a once-only request).
Registration alone does nothing. Enabling the plugin and explicitly selecting
its transport are separate consent steps:
```yaml
plugins:
enabled: [my-approval-plugin]
security:
approval:
transport: my-ui
transport_fallback: deny # default
```
Transport exceptions, timeouts, unavailable registrations, invalid choices,
and stale responses deny by default. To deliberately show the prompt on the
ordinary CLI/TUI/gateway/ACP surface when the selected transport fails, set
`transport_fallback: builtin`. Without that exact opt-in, Hermes never
materializes the prompt on another surface.
Hermes still owns hardline blocks, sudo-stdin protection, user deny rules,
request binding, allowed scopes, persistence, hooks, and final authorization.
Hardline commands are blocked before any transport callback. There is
intentionally **no plugin approval policy, auto-allow callback, or required
`pre_tool_call` policy** in this interface. A future approval-policy capability
may use the plugin capability-consent model, but transport selection does not
grant it.
### Migration for existing users
When you upgrade to a version of Hermes that has opt-in plugins (config schema v21+), any user plugins already installed under `~/.hermes/plugins/` that weren't already in `plugins.disabled` are **automatically grandfathered** into `plugins.enabled`. Your existing setup keeps working. Bundled standalone plugins are NOT grandfathered — even existing users have to opt in explicitly. (Bundled platform/backend plugins never needed grandfathering because they were never gated.)
## Available hooks
Plugins can register the 26 lifecycle events currently accepted by `hermes_cli.plugins.VALID_HOOKS`. The **[Event Hooks catalog](/user-guide/features/hooks#shipped-plugin-hook-catalog)** is canonical for exact timing, return handling, payload fields, and privacy notes.
| Descriptive category | Shipped hooks |
|---|---|
| **Directive/control** | `pre_tool_call`, `pre_llm_call`, `pre_verify`, `pre_gateway_dispatch` |
| **Transform** | `transform_tool_result`, `transform_terminal_output`, `transform_llm_output`, `pre_transcription` |
| **Observer** | `post_tool_call`, `post_llm_call`, `pre_api_request`, `post_api_request`, `api_request_error`, `on_stream_start`, `on_stream_delta`, `on_stream_end`, `on_interim_message`, `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `on_skill_lifecycle`, `subagent_start`, `subagent_stop`, `pre_approval_request`, `post_approval_response`, `pre_command`, `kanban_task_claimed`, `kanban_task_completed`, `kanban_task_blocked` |
These categories describe current behavior rather than defining future naming rules. Plugin middleware remains a separate registry/surface.
## Plugin types
Hermes has four kinds of plugins:
| Type | What it does | Selection | Location |
|------|-------------|-----------|----------|
| **General plugins** | Add tools, hooks, slash commands, CLI commands | Multi-select (enable/disable) | `~/.hermes/plugins/` |
| **Memory providers** | Replace or augment built-in memory | Single-select (one active) | `plugins/memory/` |
| **Context engines** | Replace the built-in context compressor | Single-select (one active) | `plugins/context_engine/` |
| **Model providers** | Declare an inference backend (OpenRouter, Anthropic, …) | Multi-register, picked by `--provider` / `config.yaml` | `plugins/model-providers/` |
Memory providers and context engines are **provider plugins** — only one of each type can be active at a time. Model providers are also plugins, but many load simultaneously; the user picks one at a time via `--provider` or `config.yaml`. General plugins can be enabled in any combination.
## Pluggable interfaces — where to go for each
The table above shows the four plugin categories, but within "General plugins" the `PluginContext` exposes several distinct extension points — and Hermes also accepts extensions outside the Python plugin system (config-driven backends, shell-hooked commands, external servers, etc.). Use this table to find the right doc for what you want to build:
| Want to add… | How | Authoring guide |
|---|---|---|
| A **tool** the LLM can call | Python plugin — `ctx.register_tool()` | [Build a Hermes Plugin](/developer-guide/plugins) · [Adding Tools](/developer-guide/adding-tools) |
| A **lifecycle hook** (pre/post LLM, session start/end, tool filter) | Python plugin — `ctx.register_hook()` | [Hooks reference](/user-guide/features/hooks) · [Build a Hermes Plugin](/developer-guide/plugins) |
| A **slash command** for the CLI / gateway | Python plugin — `ctx.register_command()` | [Build a Hermes Plugin](/developer-guide/plugins) · [Extending the CLI](/developer-guide/extending-the-cli) |
| A **subcommand** for `hermes <thing>` | Python plugin — `ctx.register_cli_command()` | [Extending the CLI](/developer-guide/extending-the-cli) |
| A bundled **skill** that your plugin ships | Python plugin — `ctx.register_skill()` | [Creating Skills](/developer-guide/creating-skills) |
| An **inference backend** (LLM provider: OpenAI-compat, Codex, Anthropic-Messages, Bedrock) | Provider plugin — `register_provider(ProviderProfile(...))` in `plugins/model-providers/<name>/` | **[Model Provider Plugins](/developer-guide/model-provider-plugin)** · [Adding Providers](/developer-guide/adding-providers) |
| A **gateway channel** (Discord / Telegram / IRC / Teams / etc.) | Platform plugin — `ctx.register_platform()` in `plugins/platforms/<name>/` | [Adding Platform Adapters](/developer-guide/adding-platform-adapters) |
| A **memory backend** (Honcho, Mem0, Supermemory, …) | Memory plugin — subclass `MemoryProvider` in `plugins/memory/<name>/` | [Memory Provider Plugins](/developer-guide/memory-provider-plugin) |
| A **context-compression strategy** | Context-engine plugin — `ctx.register_context_engine()` | [Context Engine Plugins](/developer-guide/context-engine-plugin) |
| An **image-generation backend** (DALL·E, SDXL, …) | Backend plugin — `ctx.register_image_gen_provider()` | [Image Generation Provider Plugins](/developer-guide/image-gen-provider-plugin) |
| A **video-generation backend** (Veo, Kling, Pixverse, Grok-Imagine, Runway, …) | Backend plugin — `ctx.register_video_gen_provider()` | [Video Generation Provider Plugins](/developer-guide/video-gen-provider-plugin) |
| A **TTS backend** (any CLI — Piper, VoxCPM, Kokoro, xtts, voice-cloning scripts, …) | Config-driven (recommended) — declare under `tts.providers.<name>` with `type: command` in `config.yaml`. OR Python backend plugin — `ctx.register_tts_provider()` for Python-SDK / streaming engines that need more than a shell template. | [TTS Setup](/user-guide/features/tts#custom-command-providers) · [Python plugin guide](/user-guide/features/tts#python-plugin-providers) |
| An **STT backend** (any CLI — whisper.cpp, custom whisper binary, local ASR CLI) | Config-driven (recommended) — declare under `stt.providers.<name>` with `type: command` in `config.yaml`, or set `HERMES_LOCAL_STT_COMMAND` for the legacy single-command escape hatch. OR Python backend plugin — `ctx.register_transcription_provider()` for Python-SDK engines (OpenRouter, SenseAudio, Gemini-STT, etc.). | [STT Setup](/user-guide/features/tts#stt-custom-command-providers) · [Python plugin guide](/user-guide/features/tts#python-plugin-providers-stt) |
| **External tools via MCP** (filesystem, GitHub, Linear, Notion, any MCP server) | Config-driven — declare `mcp_servers.<name>` with `command:` / `url:` in `config.yaml`. Hermes auto-discovers the server's tools and registers them alongside built-ins. | [MCP](/user-guide/features/mcp) |
| **Additional skill sources** (custom GitHub repos, private skill indexes) | CLI — `hermes skills tap add <repo>` | [Skills Hub](/user-guide/features/skills#skills-hub) · [Publishing a custom tap](/user-guide/features/skills#publishing-a-custom-skill-tap) |
| **Gateway event hooks** (fire on `gateway:startup`, `session:start`, `agent:end`, `command:*`) | Drop `HOOK.yaml` + `handler.py` into `~/.hermes/hooks/<name>/` | [Event Hooks](/user-guide/features/hooks#gateway-event-hooks) |
| **Shell hooks** (run a shell command on events — notifications, audit logs, desktop alerts) | Config-driven — declare under `hooks:` in `config.yaml` | [Shell Hooks](/user-guide/features/hooks#shell-hooks) |
:::note
Not everything is a Python plugin. Some extension surfaces intentionally use **config-driven shell commands** (TTS, STT, shell hooks) so any CLI you already have becomes a plugin without writing Python. Others are **external servers** (MCP) the agent connects to and auto-registers tools from. And some are **drop-in directories** (gateway hooks) with their own manifest format. Pick the right surface for the integration style that fits your use case; the authoring guides in the table above each cover placeholders, discovery, and examples.
:::
## NixOS declarative plugins
On NixOS, plugins can be installed declaratively via the module options — no `hermes plugins install` needed. See the **[Nix Setup guide](/getting-started/nix-setup#plugins)** for full details.
```nix
services.hermes-agent = {
# Directory plugin (source tree with plugin.yaml)
extraPlugins = [ (pkgs.fetchFromGitHub { ... }) ];
# Entry-point plugin (pip package)
extraPythonPackages = [ (pkgs.python312Packages.buildPythonPackage { ... }) ];
# Enable in config
settings.plugins.enabled = [ "my-plugin" ];
};
```
Declarative plugins are symlinked with a `nix-managed-` prefix — they coexist with manually installed plugins and are cleaned up automatically when removed from the Nix config.
## Managing plugins
```bash
hermes plugins # unified interactive UI
hermes plugins list # table: enabled / disabled / not enabled
hermes plugins search <term> # search the community plugin index
hermes plugins install <name> # install by index name (resolved to repo @ pinned ref)
hermes plugins install user/repo # install from Git, then prompt Enable? [y/N]
hermes plugins install user/repo --enable # install AND enable (no prompt)
hermes plugins install user/repo --no-enable # install but leave disabled (no prompt)
hermes plugins update my-plugin # pull latest (local edits are autostashed and re-applied)
hermes plugins remove my-plugin # uninstall
hermes plugins enable my-plugin # add to allow-list
hermes plugins disable my-plugin # remove from allow-list + add to disabled
hermes plugins capabilities [my-plugin] # declared vs granted capabilities
```
### One-click install links (Desktop)
Hermes Desktop registers the `hermes://` URL scheme, so a website, README, or
chat message can link straight to a plugin install:
```
hermes://plugin/install?repo=owner/repo # main install link
hermes://plugin/install?repo=owner/repo&enable=1 # enable the agent plugin after install
hermes://plugin/install?repo=owner/repo&force=1 # replace an existing install
```
Clicking one opens Hermes and shows a **confirmation dialog** — the repo id,
a "Before you install" note, and GitHub browse + clone links — then
shallow-clones the repo to detect what it ships (an **agent plugin** —
backend Python, a **desktop plugin** — app UI, or both). You pick the
components with checkboxes and confirm. Nothing is installed until you do;
deep links never auto-install, and agent-plugin installs go through the same
[install-time security scanning](#install-time-security-scanning) as
`hermes plugins install`.
Hybrid repos (agent + desktop halves in one repo) use one link and one
dialog. The same modal is reachable without a link via **Settings → Plugins →
Install from Git**. Legacy `hermes://plugin-agent/…` and
`hermes://plugin-desktop/…` URLs route into the same dialog. In dev builds
(`npm run dev`) the scheme is `hermes-dev://`.
Websites need no SDK — a normal anchor works:
```html
<a href="hermes://plugin/install?repo=owner/repo&enable=1">Install in Hermes</a>
```
MCP servers have the equivalent link form — see
[Add to Hermes link](/reference/mcp-config-reference#add-to-hermes-link).
### Plugin capabilities and consent
Plugins can declare the privileged host surfaces they want in their
`plugin.yaml`:
```yaml
name: my-plugin
capabilities:
- tools.override # replace built-in tools
- llm.model_override # pick the model for host-owned LLM calls
```
When a plugin declares capabilities, `hermes plugins install` (and
`hermes plugins enable`) shows the list with one-line risk descriptions and
asks once. Consenting records the grant under
`plugins.entries.<id>.granted_capabilities` together with a consent hash and
timestamp. Declining leaves the plugin enabled with those capabilities off —
a well-behaved plugin probes with `ctx.has_capability()` and degrades
gracefully.
**Update re-consent:** if a plugin update declares capabilities you haven't
granted, `hermes plugins update` surfaces the additions and asks again. New
capabilities stay off until you consent — a plugin update can never silently
widen its access.
**Non-interactive sessions fail closed:** installing or updating without a
TTY completes the install, but declared capabilities are *not* granted. Run
`hermes plugins enable <id>` interactively to grant them later.
Inspect the state at any time:
```bash
hermes plugins capabilities # all plugins with declared/granted capabilities
hermes plugins capabilities my-plugin # one plugin, declared vs granted
```
Capability ids map 1:1 to the older per-feature config gates, which keep
working but are **deprecated** in favor of the consent flow:
| Capability | Legacy key (`plugins.entries.<id>.…`) |
|---|---|
| `tools.override` | `allow_tool_override` |
| `llm.provider_override` | `llm.allow_provider_override` |
| `llm.model_override` | `llm.allow_model_override` |
| `llm.agent_id_override` | `llm.allow_agent_id_override` |
| `llm.profile_override` | `llm.allow_profile_override` |
| `llm.task_override` | `llm.allow_task_override` |
| `gateway.platform_actions` | `allow_platform_actions` |
A gate is open when *either* the capability is granted *or* the legacy key is
set — existing configs keep working unchanged.
:::warning Not a sandbox
Capabilities are a **consent and audit layer**, not isolation. Plugins run as
regular in-process Python: a malicious plugin can ignore every gate here.
Granting a capability is a statement of trust in the plugin author — it is
not a code audit, and Hermes has not reviewed the plugin's code. Only install
plugins from sources you trust.
:::
### Platform actions
`ctx.platform_actions` gives a plugin a minimal, capability-gated verb set for
acting on connected chat platforms through the live gateway adapter registry —
the sanctioned alternative to monkeypatching an adapter. **It is off by
default**: every call re-checks the `gateway.platform_actions` capability
(legacy key `plugins.entries.<id>.allow_platform_actions`), and an ungranted
call returns a structured error instead of acting.
v1 verbs (both `async`, both return a plain dict, and neither ever raises into
hook dispatch):
```python
result = await ctx.platform_actions.add_reaction(
platform="telegram", chat_id="-100123", message_id="456", emoji="👍",
)
result = await ctx.platform_actions.set_thread_title(
platform="discord", chat_id="123", thread_id="456", title="New title",
)
if not result["ok"]:
print(result["error"], result.get("detail"))
```
Success is `{"ok": True, "action": <verb>}`. Failures are
`{"ok": False, "error": <code>, "detail": <str>}` with stable error codes:
`capability_not_granted`, `invalid_argument`, `gateway_unavailable`,
`unknown_platform`, `adapter_not_registered`, `adapter_disconnected`,
`unsupported_platform_action`, `action_failed`. Actions validate that the
target adapter exists and is connected before acting; a disconnected or
missing adapter degrades to a structured error, never an exception.
Platforms supported in v1: Telegram and Discord. Telegram's `add_reaction`
*sets* the bot's reaction (the Bot API replaces a previous bot reaction rather
than stacking). Every action — allowed or denied — is written to the log with
the plugin id, verb, platform, and outcome.
:::warning Security note
Platform actions are a **messaging-as-the-bot power**: a granted plugin can
react and rename threads in any chat the gateway bot can reach, not just the
chat that triggered the hook. Grant `gateway.platform_actions` only to plugins
you trust, and prefer plugins that document exactly which actions they take.
Raw platform SDK payload/handle access is deliberately **not** part of this
surface — per the #64176 round-2 design correction it requires its own
capability (`gateway.raw_events`) with a "no stability guarantee" label and a
separate design, and has not shipped.
:::
### Discovering community plugins
`hermes plugins search <term>` searches the **community plugin index** — a
static, machine-readable JSON catalog of community plugins. Matching is fuzzy
across name, description, and tags:
```bash
hermes plugins search telegram # fuzzy search
hermes plugins search # browse the whole index
hermes plugins search --capability platform # filter by declared capability
hermes plugins search media --json # machine-readable output
hermes plugins search --refresh # bypass the 24h local cache
```
Once you've found a plugin, install it by bare name — the name is resolved
through the index to its `owner/repo` plus the index-pinned commit:
```bash
hermes plugins install hermes-media-studio
```
If a name matches more than one entry, the candidates are listed and nothing
is installed. Explicit `owner/repo` or Git-URL identifiers never touch the
index and keep working exactly as before. An explicit `--ref <sha>` always
overrides the index pin.
**How the index is fetched.** The index lives at a canonical URL
(`https://raw.githubusercontent.com/NousResearch/hermes-plugin-index/main/index.json`,
overridable via `hermes config set plugins.index_url <url>`). Fetches are
cached under `~/.hermes/cache/plugin_index.json` for 24 hours; when the
remote is unreachable the stale cache is used, and when there is no cache at
all a bundled seed copy ships with Hermes — so search works fully offline.
**Index entry format.** Each entry is a JSON object:
```json
{
"name": "hermes-media-studio",
"description": "Generative media workspace plugin.",
"author": "NousResearch",
"tags": ["media", "image-gen"],
"repo": "NousResearch/hermes-media-studio",
"ref": "<40-char commit SHA>",
"subdir": null,
"homepage": "https://github.com/NousResearch/hermes-media-studio",
"capabilities": ["tools", "dashboard"],
"api_version": 1,
"added_at": "2026-08-12"
}
```
`repo` is the `owner/name` GitHub identifier, `ref` pins an immutable commit
SHA, and optional `subdir` supports monorepos. The bundled seed file
(`hermes_cli/data/plugin_index.json` in the repo) is the format reference.
**Submitting a plugin.** The index is maintained as a plain JSON file —
submit a pull request to the
[hermes-plugin-index](https://github.com/NousResearch/hermes-plugin-index)
repository adding your entry (name, description, author, tags, `owner/repo`,
and a pinned commit SHA). Review covers the entry's *metadata* only.
:::warning Indexed ≠ audited
Inclusion in the community index means the entry's metadata was reviewed —
**it is not a code audit**. Installing still goes through the normal
consent/review flow (plugins install disabled by default, enabling is an
explicit step, and tool-override rights require a separate grant). Review a
plugin's source before enabling it.
:::
### Plugin packs
A **plugin pack** is a declarative, shareable YAML file (`hermes-pack.yaml`)
that pins a set of plugins — like sharing a modpack. Installing a pack fans
out to ordinary pinned installs; nothing new exists at runtime.
```yaml
name: voice-assistant-pack
description: STT + streaming TTS + approval relay
author: hyper
version: 1.0.0
plugins:
- name: hermes-media-studio # bare community-index name…
ref: e8d59971d2b7901405b39dac7b03bdd616272d0d
- repo: owner/approval-relay # …or explicit owner/repo (or git URL)
ref: 8f3c2d1a9b4e5f6071829304a5b6c7d8e9f00112
subdir: plugins/relay # optional monorepo path
config: # optional, non-secret seeds only
hermes-media-studio:
default_model: flux-3
skills: [] # declared list only (not auto-installed yet)
```
```bash
hermes plugins pack show ./hermes-pack.yaml # dry-run review
hermes plugins pack install ./hermes-pack.yaml # review → confirm → install
hermes plugins pack export > hermes-pack.yaml # snapshot the current install
hermes plugins pack export --enabled-only # only plugins.enabled
```
**Supply-chain posture.** Every entry's `ref` must be an exact 40-character
commit SHA — tags and branch names are rejected with an error naming the
entry, the same rule as the community index. Pack installs ride the exact
same pinned install path as `hermes plugins install --ref <sha>` and record
the same provenance in `plugins/.install-metadata.json`, so two installs of
the same pack resolve identically. Packs build on the
[manifest v2 fields](/developer-guide/plugins) (`manifest_version`,
`api_version`, `requires_plugins`) — each plugin's own manifest still
validates through the normal install path.
**Consent is never bulk-granted.** `pack install` shows a mandatory review
screen (every plugin, source, pinned ref, and the capabilities it declares),
then asks **one** confirmation for the pack contents. After that, each
plugin's declared capabilities go through the standard per-plugin
capability-consent prompt — identical to a single `hermes plugins install`.
There is no `--yes`, and non-interactive sessions cannot install packs.
**Secrets never travel in packs.** `config:` seeds are limited to
non-secret `plugins.entries.<id>` keys — secret-shaped key names
(`*token*`, `*key*`, `*password*`, …), capability grants, and the deprecated
`allow_*` trust gates are rejected on install and stripped on export.
Plugins that need secrets declare them in their own `requires_env`, which
prompts during install as usual. Existing user values in
`plugins.entries.<id>` always win over pack seeds.
**Partial failure.** Each plugin installs independently; failures are
reported per plugin, the rest continue, and the command exits non-zero if
any plugin failed.
**Export caveats.** `pack export` only includes plugins with known Git
provenance (installed via `hermes plugins install`). Local-only plugins are
listed as warning comments in the emitted YAML, not as installable entries.
The `skills:` list is parsed and displayed at install time but not yet
auto-installed — install those manually for now (`hermes skills`). Wiring
skill-hub ids into pack install is a documented follow-up seam.
### Install-time security scanning
Every `hermes plugins install` and `hermes plugins update` runs a static
security scan over the plugin tree before it is activated (inspired by
Claude Cowork's skill & plugin security scanning). The scanner reuses the
same threat-pattern engine as the [Skills Hub guard](/user-guide/features/skills)
— exfiltration of credential stores, reverse shells, destructive commands,
persistence mechanisms, obfuscated execution, and prompt injection in
documentation files — with plugin-aware exemptions: a provider plugin
reading its **own** API key from the environment (the documented
`requires_env` pattern) is not flagged.
Three verdicts, matching Cowork's pass/warn/fail:
| Verdict | Behavior |
|---|---|
| **safe** | Installs normally, no extra output |
| **caution** | Findings are shown; you confirm `Install anyway? [y/N]` (or pass `--force`) |
| **dangerous** | Blocked. `--force` does **not** override |
On `hermes plugins update`, a dangerous verdict on the updated tree
disables the plugin until you review the findings and re-enable it.
Scanning is on by default; disable it in `config.yaml`:
```yaml
plugins:
scan_on_install: false
```
### Interactive UI
Running `hermes plugins` with no arguments opens a composite interactive screen:
```
Plugins
↑↓ navigate SPACE toggle ENTER configure/confirm ESC done
General Plugins
→ [✓] my-tool-plugin — Custom search tool
[ ] webhook-notifier — Event hooks
[ ] disk-cleanup — Auto-cleanup of ephemeral files [bundled]
Provider Plugins
Memory Provider ▸ honcho
Context Engine ▸ compressor
```
- **General Plugins section** — checkboxes, toggle with SPACE. Checked = in `plugins.enabled`, unchecked = in `plugins.disabled` (explicit off).
- **Provider Plugins section** — shows current selection. Press ENTER to drill into a radio picker where you choose one active provider.
- Bundled plugins appear in the same list with a `[bundled]` tag.
Provider plugin selections are saved to `config.yaml`:
```yaml
memory:
provider: "honcho" # empty string = built-in only
context:
engine: "compressor" # default built-in compressor
```
### Enabled vs. disabled vs. neither
Plugins occupy one of three states:
| State | Meaning | In `plugins.enabled`? | In `plugins.disabled`? |
|---|---|---|---|
| `enabled` | Loaded on next session | Yes | No |
| `disabled` | Explicitly off — won't load even if also in `enabled` | (irrelevant) | Yes |
| `not enabled` | Discovered but never opted in | No | No |
The default for a newly-installed or bundled plugin is `not enabled`. `hermes plugins list` shows all three distinct states so you can tell what's been explicitly turned off vs. what's just waiting to be enabled.
In a running session, `/plugins` shows which plugins are currently loaded.
## Injecting Messages
Plugins can inject messages into a CLI conversation or a known gateway session using `ctx.inject_message()`:
```python
# Active CLI conversation
ctx.inject_message("New data arrived from the webhook", role="user")
# Existing gateway conversation
ctx.inject_message(
"New data arrived from the webhook",
role="user",
session_key="agent:main:telegram:dm:123456789",
)
```
**Signature:** `ctx.inject_message(content: str, role: str = "user", *, session_key: str | None = None) -> bool`
In CLI mode:
- If the agent is **idle** (waiting for user input), the message is queued as the next input and starts a new turn.
- If the agent is **mid-turn** (actively running), the message interrupts the current operation — the same as a user typing a new message and pressing Enter.
- For non-`"user"` roles, the content is prefixed with `[role]` (e.g. `[system] ...`).
- Returns `True` if the message was queued successfully.
In gateway mode:
- `session_key` is required and must identify an existing gateway session. It is the stable routing key, not the CLI session ID.
- Hermes reuses that session's stored platform, chat, thread, profile, and conversation history. Plugins cannot supply a new chat route through this API.
- Hermes rechecks the stored route against the gateway's current authorisation rules before dispatch.
- Routes that relied only on an adapter-time or upstream authorisation decision are rejected unless Hermes can revalidate them from current core allowlists, pairing, or explicit allow-all configuration.
- Injected text is always conversational input. It cannot invoke slash commands, approve tools, or resolve pending confirmation and clarification prompts.
- The route and conversation are pinned while dispatch is pending. Hermes drops the request if topic recovery changes the route or the session rotates before handling starts.
- The request enters the platform adapter's normal message path. Active sessions use the existing busy-session queue rather than starting a competing turn.
- Returns `True` when the live gateway accepts the request for asynchronous dispatch. This does not confirm that the agent turn or platform delivery has completed.
- Returns `False` when `session_key` is omitted, the permission is not granted, or no live gateway can accept the request. Unknown or unroutable session keys discovered after asynchronous acceptance are written to the gateway log.
This enables plugins like remote control viewers, messaging bridges, or webhook receivers to feed messages into the conversation from external sources.
Gateway injection can send an agent response to an external messaging platform. It is disabled by default for every plugin. Grant it per plugin in `config.yaml`:
```yaml
plugins:
entries:
my-plugin:
allow_gateway_injection: true
```
:::warning
Only grant gateway injection to plugins you trust. Hermes checks this host API permission and restricts it to existing session routes, but Python plugins run in-process and this setting is not a sandbox.
:::
:::note
This plugin API does not expose a public HTTP endpoint or CLI command for external processes. The plugin must already know the target gateway `session_key`, for example from its own trusted configuration or previously retained session state.
:::
## Calling MCP servers from plugins
`ctx.call_mcp()` lets a plugin call a tool on one of the user's configured MCP servers — synchronously, from any hook or tool handler — routing through Hermes' existing native MCP client (same connections, trust-tier gates, circuit breaker, and reconnect logic as model-invoked MCP tools; never a parallel client).
```python
result = ctx.call_mcp(
"knowledge_rag", # server name from mcp.servers
"query_knowledge", # tool on that server
{"query": "deploy runbook"},
timeout=30, # seconds; clamped to 1–600
)
if result["ok"]:
print(result["result"])
else:
print("MCP error:", result["error"])
```
**Signature:** `ctx.call_mcp(server: str, tool: str, arguments: dict | None = None, timeout: float = 30) -> dict`
Returns a stable envelope: `{"ok": True, "result": ...}` (plus `structuredContent` when the server provides it) or `{"ok": False, "error": "..."}`. Results over ~64 KB are truncated and flagged with `"truncated": True`.
### Security: default-off, per-server allowlist
A plugin has **no MCP access by default**. The operator must grant each server explicitly in `config.yaml`:
```yaml
plugins:
entries:
my-plugin:
mcp_allowlist: ["knowledge_rag", "github"]
```
- Calling a server not in the list raises `PermissionError` naming the exact config key to set.
- The grant is per-server and per-plugin — never ambient authority over every configured server, and `"*"` wildcards are not honored.
- Every call has an enforced timeout (default 30 s) so a hung MCP server cannot stall the hook or tool pipeline that invoked it.
- MCP servers return untrusted content. Treat `result` as data, not instructions — don't feed it into privileged decisions (approvals, command execution) without validation.
:::warning
Granting `mcp_allowlist` gives the plugin the same access to that MCP server as the model has — including any write-capable tools the server exposes (subject to the server's `trust` tier gates). Grant only servers the plugin genuinely needs.
:::
See the **[full guide](/developer-guide/plugins)** for handler contracts, schema format, hook behavior, error handling, and common mistakes.
website/docs/user-guide/features/provider-routing.md
+204
View File
@@ -0,0 +1,204 @@
---
title: Provider Routing
description: Configure OpenRouter or Nous Portal provider preferences to optimize for cost, speed, or quality.
sidebar_label: Provider Routing
sidebar_position: 7
---
# Provider Routing
When using [OpenRouter](https://openrouter.ai) or [Nous Portal](/integrations/nous-portal) as your LLM provider, Hermes Agent supports **provider routing** — fine-grained control over which underlying AI providers handle your requests and how they're prioritized.
OpenRouter routes requests to many providers (e.g., Anthropic, Google, AWS Bedrock, Together AI). Provider routing lets you optimize for cost, speed, quality, or enforce specific provider requirements.
:::tip
Traffic routed through Nous Portal respects the same provider preferences — and Portal subscribers get 10% off token-billed providers.
:::
## Configuration
Add a `provider_routing` section to your `~/.hermes/config.yaml`:
```yaml
provider_routing:
sort: "price" # How to rank providers
only: [] # Whitelist: only use these providers
ignore: [] # Blacklist: never use these providers
order: [] # Explicit provider priority order
require_parameters: false # Only use providers that support all parameters
data_collection: null # Control data collection ("allow" or "deny")
```
:::info
Provider routing only applies when using OpenRouter or Nous Portal. It has no effect with direct provider connections (e.g., connecting directly to the Anthropic API).
:::
## Options
### `sort`
Controls how OpenRouter ranks available providers for your request.
| Value | Description |
|-------|-------------|
| `"price"` | Cheapest provider first |
| `"throughput"` | Fastest tokens-per-second first |
| `"latency"` | Lowest time-to-first-token first |
```yaml
provider_routing:
sort: "price"
```
### `only`
Whitelist of provider slugs. When set, **only** these providers will be used. All others are excluded. Use the lowercase slug shown by OpenRouter for each provider.
```yaml
provider_routing:
only:
- "anthropic"
- "google"
```
### `ignore`
Blacklist of provider names. These providers will **never** be used, even if they offer the cheapest or fastest option.
```yaml
provider_routing:
ignore:
- "together"
- "deepinfra"
```
### `order`
Explicit priority order. Providers listed first are preferred. Unlisted providers are used as fallbacks.
```yaml
provider_routing:
order:
- "anthropic"
- "google"
- "amazon-bedrock"
```
### `require_parameters`
When `true`, OpenRouter will only route to providers that support **all** parameters in your request (like `temperature`, `top_p`, `tools`, etc.). This avoids silent parameter drops.
```yaml
provider_routing:
require_parameters: true
```
### `data_collection`
Controls whether providers can use your prompts for training. Options are `"allow"` or `"deny"`.
```yaml
provider_routing:
data_collection: "deny"
```
## Practical Examples
### Optimize for Cost
Route to the cheapest available provider. Good for high-volume usage and development:
```yaml
provider_routing:
sort: "price"
```
### Optimize for Speed
Prioritize low-latency providers for interactive use:
```yaml
provider_routing:
sort: "latency"
```
### Optimize for Throughput
Best for long-form generation where tokens-per-second matters:
```yaml
provider_routing:
sort: "throughput"
```
### Lock to Specific Providers
Ensure all requests go through a specific provider for consistency:
```yaml
provider_routing:
only:
- "anthropic"
```
### Avoid Specific Providers
Exclude providers you don't want to use (e.g., for data privacy):
```yaml
provider_routing:
ignore:
- "together"
- "lepton"
data_collection: "deny"
```
### Preferred Order with Fallbacks
Try your preferred providers first, fall back to others if unavailable:
```yaml
provider_routing:
order:
- "anthropic"
- "google"
require_parameters: true
```
## How It Works
Provider routing preferences are passed to OpenRouter or Nous Portal on agent chat requests and iteration-limit summaries via the `extra_body.provider` field. (`extra_body` is the OpenAI Python SDK argument; it becomes the top-level `provider` object in the JSON request.) Auxiliary tasks such as compression and title generation are configured independently under `auxiliary.<task>.extra_body`.
- **CLI mode** — configured in `~/.hermes/config.yaml`, loaded at startup
- **Gateway mode** — same config file, loaded when the gateway starts
The routing config is read from `config.yaml` and passed as parameters when creating the `AIAgent`:
```
providers_allowed ← from provider_routing.only
providers_ignored ← from provider_routing.ignore
providers_order ← from provider_routing.order
provider_sort ← from provider_routing.sort
provider_require_parameters ← from provider_routing.require_parameters
provider_data_collection ← from provider_routing.data_collection
```
:::tip
You can combine multiple options. For example, sort by price but exclude certain providers and require parameter support:
```yaml
provider_routing:
sort: "price"
ignore: ["together"]
require_parameters: true
data_collection: "deny"
```
:::
## Default Behavior
When no `provider_routing` section is configured (the default), the aggregator uses its own default routing logic, which generally balances cost and availability automatically.
:::tip Provider Routing vs. Fallback Models
Provider routing controls which **sub-providers behind OpenRouter or Nous Portal** handle your requests. For automatic failover to an entirely different provider when your primary model fails, see [Fallback Providers](/user-guide/features/fallback-providers).
:::
website/docs/user-guide/features/skills.md
+1037
View File
File diff suppressed because it is too large Load Diff
website/docs/user-guide/features/skins.md
+271
View File
@@ -0,0 +1,271 @@
---
sidebar_position: 10
title: "Skins & Themes"
description: "Customize the Hermes CLI with built-in and user-defined skins"
---
# Skins & Themes
Skins control the **visual presentation** of the Hermes CLI: banner colors, spinner faces and verbs, response-box labels, branding text, and the tool activity prefix.
Conversational style and visual style are separate concepts:
- **Personality** changes the agent's tone and wording.
- **Skin** changes the CLI's appearance.
## Change skins
```bash
/skin # show the current skin and list available skins
/skin ares # switch to a built-in skin
/skin mytheme # switch to a custom skin from ~/.hermes/skins/mytheme.yaml
```
Or set the default skin in `~/.hermes/config.yaml`:
```yaml
display:
skin: default
```
## Built-in skins
| Skin | Description | Agent branding | Visual character |
|------|-------------|----------------|------------------|
| `default` | Classic Hermes — gold and kawaii | `Hermes Agent` | Warm gold borders, cornsilk text, kawaii faces in spinners. The familiar caduceus banner. Clean and inviting. |
| `ares` | War-god theme — crimson and bronze | `Ares Agent` | Deep crimson borders with bronze accents. Aggressive spinner verbs ("forging", "marching", "tempering steel"). Custom sword-and-shield ASCII art banner. |
| `mono` | Monochrome — clean grayscale | `Hermes Agent` | All grays — no color. Borders are `#555555`, text is `#c9d1d9`. Ideal for minimal terminal setups or screen recordings. |
| `slate` | Cool blue — developer-focused | `Hermes Agent` | Royal blue borders (`#4169e1`), soft blue text. Calm and professional. No custom spinner — uses default faces. |
| `daylight` | Light theme for bright terminals with dark text and cool blue accents | `Hermes Agent` | Designed for white or bright terminals. Dark slate text with blue borders, pale status surfaces, and a light completion menu that stays readable in light terminal profiles. |
| `warm-lightmode` | Warm brown/gold text for light terminal backgrounds | `Hermes Agent` | Warm parchment tones for light terminals. Dark brown text with saddle-brown accents, cream-colored status surfaces. An earthy alternative to the cooler daylight theme. |
| `poseidon` | Ocean-god theme — deep blue and seafoam | `Poseidon Agent` | Deep blue to seafoam gradient. Ocean-themed spinners ("charting currents", "sounding the depth"). Trident ASCII art banner. |
| `sisyphus` | Sisyphean theme — austere grayscale with persistence | `Sisyphus Agent` | Light grays with stark contrast. Boulder-themed spinners ("pushing uphill", "resetting the boulder", "enduring the loop"). Boulder-and-hill ASCII art banner. |
| `charizard` | Volcanic theme — burnt orange and ember | `Charizard Agent` | Warm burnt orange to ember gradient. Fire-themed spinners ("banking into the draft", "measuring burn"). Dragon-silhouette ASCII art banner. |
## Complete list of configurable keys
### Colors (`colors:`)
Controls all color values throughout the CLI. Values are hex color strings.
| Key | Description | Default (`default` skin) |
|-----|-------------|--------------------------|
| `banner_border` | Panel border around the startup banner | `#CD7F32` (bronze) |
| `banner_title` | Title text color in the banner | `#FFD700` (gold) |
| `banner_accent` | Section headers in the banner (Available Tools, etc.) | `#FFBF00` (amber) |
| `banner_dim` | Muted text in the banner (separators, secondary labels) | `#B8860B` (dark goldenrod) |
| `banner_text` | Body text in the banner (tool names, skill names) | `#FFF8DC` (cornsilk) |
| `ui_accent` | General UI accent color (highlights, active elements) | `#FFBF00` |
| `ui_label` | UI labels and tags | `#DAA520` (goldenrod) |
| `ui_ok` | Success indicators (checkmarks, completion) | `#4caf50` (green) |
| `ui_error` | Error indicators (failures, blocked) | `#ef5350` (red) |
| `ui_warn` | Warning indicators (caution, approval prompts) | `#ffa726` (orange) |
| `prompt` | Interactive prompt text color | `#FFF8DC` |
| `input_rule` | Horizontal rule above the input area | `#CD7F32` |
| `response_border` | Border around the agent's response box (ANSI escape) | `#FFD700` |
| `session_label` | Session label color | `#DAA520` |
| `session_border` | Session ID dim border color | `#8B8682` |
| `status_bar_bg` | Background color for the TUI status / usage bar | `#1a1a2e` |
| `voice_status_bg` | Background color for the voice-mode status badge | `#1a1a2e` |
| `selection_bg` | Background color for the TUI mouse-selection highlighter. Falls back to `completion_menu_current_bg` when unset. | `#3a3a55` |
| `completion_menu_bg` | Background color for the completion menu list | `#1a1a2e` |
| `completion_menu_current_bg` | Background color for the active completion row | `#333355` |
| `completion_menu_meta_bg` | Background color for the completion meta column | `#1a1a2e` |
| `completion_menu_meta_current_bg` | Background color for the active completion meta column | `#333355` |
### Spinner (`spinner:`)
Controls the animated spinner shown while waiting for API responses.
| Key | Type | Description | Example |
|-----|------|-------------|---------|
| `waiting_faces` | list of strings | Faces cycled while waiting for API response | `["(⚔)", "(⛨)", "(▲)"]` |
| `thinking_faces` | list of strings | Faces cycled during model reasoning | `["(⚔)", "(⌁)", "(<>)"]` |
| `thinking_verbs` | list of strings | Verbs shown in spinner messages | `["forging", "plotting", "hammering plans"]` |
| `wings` | list of [left, right] pairs | Decorative brackets around the spinner | `[["⟪⚔", "⚔⟫"], ["⟪▲", "▲⟫"]]` |
When spinner values are empty (like in `default` and `mono`), hardcoded defaults from `display.py` are used.
### Branding (`branding:`)
Text strings used throughout the CLI interface.
| Key | Description | Default |
|-----|-------------|---------|
| `agent_name` | Name shown in banner title and status display | `Hermes Agent` |
| `welcome` | Welcome message shown at CLI startup | `Welcome to Hermes Agent! Type your message or /help for commands.` |
| `goodbye` | Message shown on exit | `Goodbye! ⚕` |
| `response_label` | Label on the response box header | ` ⚕ Hermes ` |
| `prompt_symbol` | Symbol before the user input prompt (bare token, renderers add a trailing space) | `❯` |
| `help_header` | Header text for the `/help` command output | `(^_^)? Available Commands` |
### Other top-level keys
| Key | Type | Description | Default |
|-----|------|-------------|---------|
| `tool_prefix` | string | Character prefixed to tool output lines in the CLI | `┊` |
| `tool_emojis` | dict | Per-tool emoji overrides for spinners and progress (`{tool_name: emoji}`) | `{}` |
| `banner_logo` | string | Rich-markup ASCII art logo (replaces the default HERMES_AGENT banner) | `""` |
| `banner_hero` | string | Rich-markup hero art (replaces the default caduceus art) | `""` |
## Custom skins
Create YAML files under `~/.hermes/skins/`. User skins inherit missing values from the built-in `default` skin, so you only need to specify the keys you want to change.
### Full custom skin YAML template
```yaml
# ~/.hermes/skins/mytheme.yaml
# Complete skin template — all keys shown. Delete any you don't need;
# missing values automatically inherit from the 'default' skin.
name: mytheme
description: My custom theme
colors:
banner_border: "#CD7F32"
banner_title: "#FFD700"
banner_accent: "#FFBF00"
banner_dim: "#B8860B"
banner_text: "#FFF8DC"
ui_accent: "#FFBF00"
ui_label: "#4dd0e1"
ui_ok: "#4caf50"
ui_error: "#ef5350"
ui_warn: "#ffa726"
prompt: "#FFF8DC"
input_rule: "#CD7F32"
response_border: "#FFD700"
session_label: "#DAA520"
session_border: "#8B8682"
status_bar_bg: "#1a1a2e"
voice_status_bg: "#1a1a2e"
selection_bg: "#333355"
completion_menu_bg: "#1a1a2e"
completion_menu_current_bg: "#333355"
completion_menu_meta_bg: "#1a1a2e"
completion_menu_meta_current_bg: "#333355"
spinner:
waiting_faces:
- "(⚔)"
- "(⛨)"
- "(▲)"
thinking_faces:
- "(⚔)"
- "(⌁)"
- "(<>)"
thinking_verbs:
- "processing"
- "analyzing"
- "computing"
- "evaluating"
wings:
- ["⟪⚡", "⚡⟫"]
- ["⟪●", "●⟫"]
branding:
agent_name: "My Agent"
welcome: "Welcome to My Agent! Type your message or /help for commands."
goodbye: "See you later! ⚡"
response_label: " ⚡ My Agent "
prompt_symbol: "⚡"
help_header: "(⚡) Available Commands"
tool_prefix: "┊"
# Per-tool emoji overrides (optional)
tool_emojis:
terminal: "⚔"
web_search: "🔮"
read_file: "📄"
# Custom ASCII art banners (optional, Rich markup supported)
# banner_logo: |
# [bold #FFD700] MY AGENT [/]
# banner_hero: |
# [#FFD700] Custom art here [/]
```
### Minimal custom skin example
Since everything inherits from `default`, a minimal skin only needs to change what's different:
```yaml
name: cyberpunk
description: Neon terminal theme
colors:
banner_border: "#FF00FF"
banner_title: "#00FFFF"
banner_accent: "#FF1493"
spinner:
thinking_verbs: ["jacking in", "decrypting", "uploading"]
wings:
- ["⟨⚡", "⚡⟩"]
branding:
agent_name: "Cyber Agent"
response_label: " ⚡ Cyber "
tool_prefix: "▏"
```
## Hermes Mod — Visual Skin Editor
[Hermes Mod](https://github.com/cocktailpeanut/hermes-mod) is a community-built web UI for creating and managing skins visually. Instead of writing YAML by hand, you get a point-and-click editor with live preview.
![Hermes Mod skin editor](https://raw.githubusercontent.com/cocktailpeanut/hermes-mod/master/nous.png)
**What it does:**
- Lists all built-in and custom skins
- Opens any skin into a visual editor with all Hermes skin fields (colors, spinner, branding, tool prefix, tool emojis)
- Generates `banner_logo` text art from a text prompt
- Converts uploaded images (PNG, JPG, GIF, WEBP) into `banner_hero` ASCII art with multiple render styles (braille, ASCII ramp, blocks, dots)
- Saves directly to `~/.hermes/skins/`
- Activates a skin by updating `~/.hermes/config.yaml`
- Shows the generated YAML and a live preview
### Install
**Option 1 — Pinokio (1-click):**
Find it on [pinokio.computer](https://pinokio.computer) and install with one click.
**Option 2 — npx (quickest from terminal):**
```bash
npx -y hermes-mod
```
**Option 3 — Manual:**
```bash
git clone https://github.com/cocktailpeanut/hermes-mod.git
cd hermes-mod/app
npm install
npm start
```
### Usage
1. Start the app (via Pinokio or terminal).
2. Open **Skin Studio**.
3. Choose a built-in or custom skin to edit.
4. Generate a logo from text and/or upload an image for hero art. Pick a render style and width.
5. Edit colors, spinner, branding, and other fields.
6. Click **Save** to write the skin YAML to `~/.hermes/skins/`.
7. Click **Activate** to set it as the current skin (updates `display.skin` in `config.yaml`).
Hermes Mod respects the `HERMES_HOME` environment variable, so it works with [profiles](/user-guide/profiles) too.
## Operational notes
- Built-in skins load from `hermes_cli/skin_engine.py`.
- Unknown skins automatically fall back to `default`.
- `/skin` updates the active CLI theme immediately for the current session.
- User skins in `~/.hermes/skins/` take precedence over built-in skins with the same name.
- Skin changes via `/skin` are session-only. To make a skin your permanent default, set it in `config.yaml`.
- The `banner_logo` and `banner_hero` fields support Rich console markup (e.g., `[bold #FF0000]text[/]`) for colored ASCII art.
website/docs/user-guide/features/spotify.md
+279
View File
@@ -0,0 +1,279 @@
# Spotify
Hermes can control Spotify directly — playback, queue, search, playlists, saved tracks/albums, and listening history — using Spotify's official Web API with PKCE OAuth. Tokens are stored in `~/.hermes/auth.json` and refreshed automatically on 401; you only log in once per machine (refresh tokens expire after ~6 months; re-run `hermes auth spotify` when they do).
Unlike Hermes' built-in OAuth integrations (Google, GitHub Copilot, Codex), Spotify requires every user to register their own lightweight developer app. Spotify does not let third parties ship a public OAuth app that anyone can use. It takes about two minutes and `hermes auth spotify` walks you through it.
## Prerequisites
- A Spotify account. **Free** works for search, playlist, library, and activity tools. **Premium** is required for playback control (play, pause, skip, seek, volume, queue add, transfer).
- Hermes Agent installed and running.
- For playback tools: an **active Spotify Connect device** — the Spotify app must be open on at least one device (phone, desktop, web player, speaker) so the Web API has something to control. If nothing is active you'll get a `403 Forbidden` with a "no active device" message; open Spotify on any device and retry.
## Setup
### One-shot: `hermes tools` or first-run setup
The fastest path. Run:
```bash
hermes tools
```
Scroll to `🎵 Spotify`, press space to toggle it on, then `s` to save. The same toggle is also available during the first-run `hermes setup` / `hermes setup tools` flow. Spotify stays opt-in, so enabling it there runs the same provider-aware configuration as `hermes tools`.
Hermes drops you straight into the OAuth flow — if you don't have a Spotify app yet, it walks you through creating one inline. Once you finish, the toolset is enabled AND authenticated in one pass.
If you prefer to do the steps separately (or you're re-authing later), use the two-step flow below.
### Two-step flow
#### 1. Enable the toolset
```bash
hermes tools
```
Toggle `🎵 Spotify` on, save, and when the inline wizard opens, dismiss it (Ctrl+C). The toolset stays on; only the auth step is deferred.
#### 2. Run the login wizard
```bash
hermes auth spotify
```
The 7 Spotify tools only appear in the agent's toolset after step 1 — they're off by default so users who don't want them don't ship extra tool schemas on every API call.
If no `HERMES_SPOTIFY_CLIENT_ID` is set, Hermes walks you through the app registration inline:
1. Opens `https://developer.spotify.com/dashboard` in your browser
2. Prints the exact values to paste into Spotify's "Create app" form
3. Prompts you for the Client ID you get back
4. Saves it to `~/.hermes/.env` so future runs skip this step
5. Continues straight into the OAuth consent flow
After you approve, tokens are written under `providers.spotify` in `~/.hermes/auth.json`. The active inference provider is NOT changed — Spotify auth is independent of your LLM provider.
### Creating the Spotify app (what the wizard asks for)
When the dashboard opens, click **Create app** and fill in:
| Field | Value |
|-------|-------|
| App name | anything (e.g. `hermes-agent`) |
| App description | anything (e.g. `personal Hermes integration`) |
| Website | leave blank |
| Redirect URI | `http://127.0.0.1:43827/spotify/callback` |
| Which API/SDKs? | check **Web API** |
Agree to the terms and click **Save**. On the next page click **Settings** → copy the **Client ID** and paste it into the Hermes prompt. That's the only value Hermes needs — PKCE doesn't use a client secret.
### Running over SSH / in a headless environment
If `SSH_CLIENT` or `SSH_TTY` is set, Hermes skips the automatic browser open during both the wizard and the OAuth step. Copy the dashboard URL and the authorization URL Hermes prints, open them in a browser on your local machine, and proceed normally — the local HTTP listener still runs on the remote host on port `43827`. Your laptop's browser can't reach the remote loopback without an SSH local-forward:
```bash
ssh -N -L 43827:127.0.0.1:43827 user@remote-host
```
For jump-box / bastion setups and other gotchas (mosh, tmux, port conflicts), see [OAuth over SSH / Remote Hosts](../../guides/oauth-over-ssh.md).
## Verify
```bash
hermes auth status spotify
```
Shows whether tokens are present and when the access token expires. Refresh is automatic: when any Spotify API call returns 401, the client exchanges the refresh token and retries once. Refresh tokens persist across Hermes restarts, so you only re-auth if you revoke the app in your Spotify account settings or run `hermes auth logout spotify`.
## Using it
Once logged in, the agent has access to 7 Spotify tools. You talk to the agent naturally — it picks the right tool and action. For the best behavior, the agent loads a companion skill that teaches canonical usage patterns (single-search-then-play, when not to preflight `get_state`, etc.).
```
> play some miles davis
> what am I listening to
> add this track to my Late Night Jazz playlist
> skip to the next song
> make a new playlist called "Focus 2026" and add the last three songs I played
> which of my saved albums are by Radiohead
> search for acoustic covers of Blackbird
> transfer playback to my kitchen speaker
```
### Tool reference
All playback-mutating actions accept an optional `device_id` to target a specific device. If omitted, Spotify uses the currently active device.
#### `spotify_playback`
Control and inspect playback, plus fetch recently played history.
| Action | Purpose | Premium? |
|--------|---------|----------|
| `get_state` | Full playback state (track, device, progress, shuffle/repeat) | No |
| `get_currently_playing` | Just the current track (returns empty on 204 — see below) | No |
| `play` | Start/resume playback. Optional: `context_uri`, `uris`, `offset`, `position_ms` | Yes |
| `pause` | Pause playback | Yes |
| `next` / `previous` | Skip track | Yes |
| `seek` | Jump to `position_ms` | Yes |
| `set_repeat` | `state` = `track` / `context` / `off` | Yes |
| `set_shuffle` | `state` = `true` / `false` | Yes |
| `set_volume` | `volume_percent` = 0-100 | Yes |
| `recently_played` | Last played tracks. Optional `limit`, `before`, `after` (Unix ms) | No |
#### `spotify_devices`
| Action | Purpose |
|--------|---------|
| `list` | Every Spotify Connect device visible to your account |
| `transfer` | Move playback to `device_id`. Optional `play: true` starts playback on transfer |
### Home Assistant-managed speakers
If Home Assistant manages speakers that already support Spotify Connect (for example Sonos, Echo, Nest, or other Connect-capable speakers), they appear in `spotify_devices list` automatically whenever Spotify can see them. Hermes does not need a Home Assistant ↔ Spotify bridge for this path — Spotify handles the device routing natively.
Ask Hermes to transfer playback by the speaker's display name (for example, “transfer Spotify to the kitchen speaker”), or call `spotify_devices list` and pass the exact `device_id` to `spotify_devices transfer` when scripting. If the speaker is missing, open the Spotify app or the speaker's Spotify integration once so Spotify registers it as an active Connect target.
#### `spotify_queue`
| Action | Purpose | Premium? |
|--------|---------|----------|
| `get` | Currently queued tracks | No |
| `add` | Append `uri` to the queue | Yes |
#### `spotify_search`
Search the catalog. `query` is required. Optional: `types` (array of `track` / `album` / `artist` / `playlist` / `show` / `episode`), `limit`, `offset`, `market`.
#### `spotify_playlists`
| Action | Purpose | Required args |
|--------|---------|---------------|
| `list` | User's playlists | — |
| `get` | One playlist + tracks | `playlist_id` |
| `create` | New playlist | `name` (+ optional `description`, `public`, `collaborative`) |
| `add_items` | Add tracks | `playlist_id`, `uris` (optional `position`) |
| `remove_items` | Remove tracks | `playlist_id`, `uris` (+ optional `snapshot_id`) |
| `update_details` | Rename / edit | `playlist_id` + any of `name`, `description`, `public`, `collaborative` |
#### `spotify_albums`
| Action | Purpose | Required args |
|--------|---------|---------------|
| `get` | Album metadata | `album_id` |
| `tracks` | Album track list | `album_id` |
#### `spotify_library`
Unified access to saved tracks and saved albums. Pick the collection with the `kind` arg.
| Action | Purpose |
|--------|---------|
| `list` | Paginated library listing |
| `save` | Add `ids` / `uris` to library |
| `remove` | Remove `ids` / `uris` from library |
Required: `kind` = `tracks` or `albums`, plus `action`.
### Feature matrix: Free vs Premium
Read-only tools work on Free accounts. Anything that mutates playback or the queue requires Premium.
| Works on Free | Premium required |
|---------------|------------------|
| `spotify_search` (all) | `spotify_playback` — play, pause, next, previous, seek, set_repeat, set_shuffle, set_volume |
| `spotify_playback` — get_state, get_currently_playing, recently_played | `spotify_queue` — add |
| `spotify_devices` — list | `spotify_devices` — transfer |
| `spotify_queue` — get | |
| `spotify_playlists` (all) | |
| `spotify_albums` (all) | |
| `spotify_library` (all) | |
## Scheduling: Spotify + cron
Because Spotify tools are regular Hermes tools, a cron job running in a Hermes session can trigger playback on any schedule. No new code needed.
### Morning wake-up playlist
```bash
hermes cron add \
--name "morning-commute" \
"0 7 * * 1-5" \
"Transfer playback to my kitchen speaker and start my 'Morning Commute' playlist. Volume to 40. Shuffle on."
```
What happens at 7am every weekday:
1. Cron spins up a headless Hermes session.
2. Agent reads the prompt, calls `spotify_devices list` to find "kitchen speaker" by name, then `spotify_devices transfer` → `spotify_playback set_volume` → `spotify_playback set_shuffle` → `spotify_search` + `spotify_playback play`.
3. Music starts on the target speaker. Total cost: one session, a few tool calls, no human input.
### Wind-down at night
```bash
hermes cron add \
--name "wind-down" \
"30 22 * * *" \
"Pause Spotify. Then set volume to 20 so it's quiet when I start it again tomorrow."
```
### Gotchas
- **An active device must exist when the cron fires.** If no Spotify client is running (phone/desktop/Connect speaker), playback actions return `403 no active device`. For morning playlists, the trick is to target a device that's always on (Sonos, Echo, a smart speaker) rather than your phone.
- **Premium required for anything that mutates playback** — play, pause, skip, volume, transfer. Read-only cron jobs (scheduled "email me my recently played tracks") work fine on Free.
- **The cron agent inherits your active toolsets.** Spotify must be enabled in `hermes tools` for the cron session to see the Spotify tools.
- **Cron jobs run with `skip_memory=True`** so they don't write to your memory store.
Full cron reference: [Cron Jobs](./cron).
## Sign out
```bash
hermes auth logout spotify
```
Removes tokens from `~/.hermes/auth.json`. To also clear the app config, delete `HERMES_SPOTIFY_CLIENT_ID` (and `HERMES_SPOTIFY_REDIRECT_URI` if you set it) from `~/.hermes/.env`, or run the wizard again.
To revoke the app on Spotify's side, visit [Apps connected to your account](https://www.spotify.com/account/apps/) and click **REMOVE ACCESS**.
## Troubleshooting
**`403 Forbidden — Player command failed: No active device found`** — You need Spotify running on at least one device. Open the Spotify app on your phone, desktop, or web player, start any track for a second to register it, and retry. `spotify_devices list` shows what's currently visible.
**`403 Forbidden — Premium required`** — You're on a Free account trying to use a playback-mutating action. See the feature matrix above.
**`204 No Content` on `get_currently_playing`** — nothing is currently playing on any device. This is Spotify's normal response, not an error; Hermes surfaces it as an explanatory empty result (`is_playing: false`).
**`INVALID_CLIENT: Invalid redirect URI`** — the redirect URI in your Spotify app settings doesn't match what Hermes is using. The default is `http://127.0.0.1:43827/spotify/callback`. Either add that to your app's allowed redirect URIs, or set `HERMES_SPOTIFY_REDIRECT_URI` in `~/.hermes/.env` to whatever you registered.
**`429 Too Many Requests`** — Spotify's rate limit. Hermes returns a friendly error; wait a minute and retry. If this persists, you're probably running a tight loop in a script — Spotify's quota resets roughly every 30 seconds.
**`401 Unauthorized` keeps coming back** — Your refresh token was revoked (usually because you removed the app from your account, or the app was deleted). Run `hermes auth spotify` again.
**Wizard doesn't open the browser** — If you're over SSH or in a container without a display, Hermes detects it and skips the auto-open. Copy the dashboard URL it prints and open it manually.
## Advanced: custom scopes
By default Hermes requests the scopes needed for every shipped tool. Override if you want to restrict access:
```bash
hermes auth spotify --scope "user-read-playback-state user-modify-playback-state playlist-read-private"
```
Scope reference: [Spotify Web API scopes](https://developer.spotify.com/documentation/web-api/concepts/scopes). If you request fewer scopes than a tool needs, that tool's calls will fail with 403.
## Advanced: custom client ID / redirect URI
```bash
hermes auth spotify --client-id <id> --redirect-uri http://localhost:3000/callback
```
Or set them permanently in `~/.hermes/.env`:
```
HERMES_SPOTIFY_CLIENT_ID=<your_id>
HERMES_SPOTIFY_REDIRECT_URI=http://localhost:3000/callback
```
The redirect URI must be allow-listed in your Spotify app's settings. The default works for almost everyone — only change it if port 43827 is taken.
## Where things live
| File | Contents |
|------|----------|
| `~/.hermes/auth.json` → `providers.spotify` | access token, refresh token, expiry, scope, redirect URI |
| `~/.hermes/.env` | `HERMES_SPOTIFY_CLIENT_ID`, optional `HERMES_SPOTIFY_REDIRECT_URI` |
| Spotify app | owned by you at [developer.spotify.com/dashboard](https://developer.spotify.com/dashboard); contains the Client ID and the redirect URI allow-list |
website/docs/user-guide/features/subscription-proxy.md
+203
View File
@@ -0,0 +1,203 @@
---
sidebar_position: 15
title: "Subscription Proxy"
description: "Use your Nous Portal subscription (or other OAuth provider) as an OpenAI-compatible endpoint for external apps"
---
# Subscription Proxy
The subscription proxy is a local HTTP server that lets external apps —
OpenViking, Karakeep, Open WebUI, anything that speaks OpenAI-compatible
chat completions — use your Hermes-managed provider subscription as their
LLM endpoint. The proxy attaches the right credentials (refreshing them
automatically) so the app never needs a static API key.
This is different from the [API server](./api-server.md):
| | API server | Subscription proxy |
|---|---|---|
| What it serves | Your agent (full toolset, memory, skills) | Raw model inference |
| Use case | "Use Hermes as a chat backend" | "Use my Portal sub from another app" |
| Auth | Your `API_SERVER_KEY` | Any bearer (proxy attaches the real one) |
| Tool calls | Yes — the agent runs tools | No — passthrough only |
Use the API server when you want the **agent** as a backend. Use the
proxy when you just want **the model** through your subscription.
## Quick Start
### 1. Log into your provider (one-time)
```bash
hermes portal
```
This opens your browser for the Nous Portal OAuth flow. Hermes stores
the refresh token in `~/.hermes/auth.json` — the same place all Hermes
provider logins live.
### 2. Start the proxy
```bash
hermes proxy start
```
```
Starting Hermes proxy for Nous Portal
Listening on: http://127.0.0.1:8645/v1
Forwarding to: (resolved per-request from your subscription)
Use any bearer token in the client — the proxy attaches your real credential.
```
Leave this running in the foreground. Use `tmux`, `nohup`, or a systemd
unit if you want it to survive logout.
### 3. Point your app at it
Any OpenAI-compatible app config takes the same triple:
```
Base URL: http://127.0.0.1:8645/v1
API key: anything (e.g. "sk-unused")
Model: Hermes-4-70B # or Hermes-4.3-36B, Hermes-4-405B
```
The proxy ignores the `Authorization` header from your app and attaches
your real Portal credential to the upstream request. Refreshes happen
automatically when the bearer approaches expiry.
## Available providers
```bash
hermes proxy providers
```
Currently shipped: `nous` (Nous Portal) and `xai` (xAI / Grok). More
OAuth providers can be added by implementing the `UpstreamAdapter`
interface in `hermes_cli/proxy/adapters/`.
## Check status
```bash
hermes proxy status
```
```
Hermes proxy upstream adapters
[nous ] Nous Portal — ready (bearer expires 2026-05-15T06:43:21Z)
```
If you see `not logged in`, run `hermes portal`. If you see
`credentials need attention`, your refresh token was revoked (rare —
happens if you signed out from the Portal web UI) — just re-run
`hermes portal`.
## Allowed paths
The proxy only forwards paths the upstream actually serves. For Nous
Portal:
| Path | Purpose |
|------|---------|
| `/v1/chat/completions` | Chat completions (streaming + non-streaming) |
| `/v1/completions` | Legacy text completions |
| `/v1/embeddings` | Embeddings |
| `/v1/models` | Model list |
Other paths (`/v1/images/generations`, `/v1/audio/speech`, etc.) return
404 with a clear error pointing at the allowed paths. This keeps stray
clients from leaking weird requests to the upstream.
## Configuring OpenViking to use Portal
[OpenViking](https://github.com/volcengine/OpenViking) is a context
database that needs an LLM provider for its VLM (vision/language model
used to extract memories) and embedding model. With the proxy, you can
point its `vlm.api_base` at your local proxy:
Edit `~/.openviking/ov.conf`:
```json
{
"vlm": {
"provider": "openai",
"model": "Hermes-4-70B",
"api_base": "http://127.0.0.1:8645/v1",
"api_key": "unused-proxy-attaches-real-creds"
}
}
```
Then start your proxy in a terminal alongside `openviking-server`:
```bash
# Terminal 1
hermes proxy start
# Terminal 2
openviking-server
```
OpenViking's VLM calls now flow through your Portal subscription. The
embedding model side still needs its own provider — Portal does serve
`/v1/embeddings` but the model selection depends on what your tier
supports; check `portal.nousresearch.com/models`.
## Configuring Karakeep (or any bookmark/summarizer app)
[Karakeep](https://karakeep.app/) takes an OpenAI-compatible API for
bookmark summarization. In its config:
```bash
# Karakeep .env
OPENAI_API_BASE_URL=http://127.0.0.1:8645/v1
OPENAI_API_KEY=any-non-empty-string
INFERENCE_TEXT_MODEL=Hermes-4-70B
```
Same pattern works for Open WebUI, LobeChat, NextChat, or any other
OpenAI-compatible client.
## Exposing on LAN
By default the proxy binds `127.0.0.1` (localhost only). To let other
machines on your network use it:
```bash
hermes proxy start --host 0.0.0.0 --port 8645
```
⚠ **Be aware:** anyone on your network can now use your Portal
subscription. The proxy has no auth of its own — it accepts any bearer.
Use a firewall, VPN, or reverse proxy with proper auth if you expose
this beyond your trusted network.
## Rate limits
Your Portal tier's RPM/TPM limits apply across the whole proxy. The
proxy doesn't fan out or pool — it's a single bearer with your full
subscription quota. Monitor usage at
[portal.nousresearch.com](https://portal.nousresearch.com).
## Architecture
The proxy is intentionally minimal. Per request:
1. Receive `POST /v1/chat/completions` from your app
2. Look up the adapter's current credential (refresh if expiring)
3. Forward the request body verbatim, with `Authorization: Bearer <minted-key>`
4. Stream the response back unchanged (SSE preserved)
No transformation. No logging of request bodies. No agent loop. The
proxy is a credential-attaching pass-through.
## Future: more OAuth providers
The adapter system is pluggable. Adding a new provider (e.g.
HuggingFace, GitHub Copilot's chat endpoint, Anthropic via OAuth)
requires implementing `UpstreamAdapter` in
`hermes_cli/proxy/adapters/<provider>.py` and registering it in
`adapters/__init__.py`. Providers that aren't OpenAI-compatible at the
protocol level (Anthropic Messages API, for example) would need a
transformation layer, which is out of scope for the current shape.
website/docs/user-guide/features/tool-gateway.md
+215
View File
@@ -0,0 +1,215 @@
---
title: "Nous Tool Gateway"
description: "One subscription, every tool. Web search, image generation, TTS, and cloud browsers — all routed through Nous Portal with no extra API keys."
sidebar_label: "Tool Gateway"
sidebar_position: 2
---
# Nous Tool Gateway
**One subscription. Every tool built in.**
The Tool Gateway is included with every paid [Nous Portal](https://portal.nousresearch.com) subscription. It routes Hermes' tool calls — web search, image generation, text-to-speech, and cloud browser automation — through infrastructure Nous already runs, so you don't have to sign up with Firecrawl, FAL, OpenAI, Browser Use, or anyone else just to make your agent useful.
<div style={{display: 'flex', gap: '1rem', flexWrap: 'wrap', margin: '1.5rem 0'}}>
<a href="https://portal.nousresearch.com/manage-subscription" style={{background: 'var(--ifm-color-primary)', color: 'white', padding: '0.75rem 1.5rem', borderRadius: '6px', textDecoration: 'none', fontWeight: 'bold'}}>Start or manage subscription →</a>
</div>
## What's included
| | Tool | What you get |
|---|---|---|
| 🔍 | **Web search & extract** | Agent-grade web search and full-page extraction via Firecrawl. No rate limits to worry about — the gateway handles scaling. |
| 🎨 | **Image generation** | Nine models under one endpoint: **FLUX 2 Klein 9B**, **FLUX 2 Pro**, **Z-Image Turbo**, **Nano Banana Pro** (Gemini 3 Pro Image), **GPT Image 1.5**, **GPT Image 2**, **Ideogram V3**, **Recraft V4 Pro**, **Qwen Image**. Pick per-generation with a flag, or let Hermes default to FLUX 2 Klein. |
| 🔊 | **Text-to-speech** | OpenAI TTS voices wired into the `text_to_speech` tool. Drop voice notes into Telegram, generate audio for pipelines, narrate anything. |
| 🌐 | **Cloud browser automation** | Headless Chromium sessions via Browser Use. `browser_navigate`, `browser_click`, `browser_type`, `browser_vision` — all the agent-driving primitives, no Browserbase account required. |
All four are pay-as-you-use billed against your Nous subscription. Use any combination — run the gateway for web and images while keeping your own ElevenLabs key for TTS, or route everything through Nous.
## Why it's here
Building an agent that can actually *do things* means stitching together 5+ API subscriptions — each with their own signup, rate limits, billing, and quirks. The gateway collapses that into one account:
- **One bill.** Pay Nous; we handle the rest.
- **One signup.** No Firecrawl, FAL, Browser Use, or OpenAI audio accounts to manage.
- **One key.** Your Nous Portal OAuth covers every tool.
- **Same quality.** Same backends the direct-key route uses — just fronted by us.
Bring your own keys anytime — per-tool, whenever you want to. The gateway isn't a lock-in, it's a shortcut.
## Get started
There are three ways in — pick whichever fits where you are:
```bash
hermes setup --portal # Fresh install: Nous OAuth + set Nous as provider + turn on the Tool Gateway in one go
```
```bash
hermes model # Switch your inference provider to Nous Portal — Hermes then offers to turn on the gateway for all tools
```
```bash
hermes tools # Enable the gateway per-tool — pick "Nous Subscription" for any tool you want
```
`hermes setup --portal` and `hermes model` are the all-at-once paths: log in once, optionally flip every tool to the gateway. `hermes tools` is the à la carte path — turn on just the tools you want, one at a time.
**You don't have to log in first.** With `hermes tools`, the Nous-managed backends (Web search, Image, Video, TTS, Browser) are always listed, even if you've never signed into Nous Portal. Select one and Hermes runs the Portal login right there if you aren't already authenticated — no need to run `hermes model` beforehand. If your Nous OAuth is already active, selecting the backend enables it immediately with no extra prompt. This path only logs you in and turns on the one tool you picked — it does **not** switch your inference provider, and it does **not** prompt you to enable the gateway for every other tool.
Check what's active at any time:
```bash
hermes portal info # Portal auth + Tool Gateway routing summary
hermes portal tools # Gateway catalog with current routing per tool
hermes status # Full system status (Tool Gateway is one section)
```
`hermes portal info` shows a section like:
```
◆ Nous Tool Gateway
Nous Portal ✓ managed tools available
Web tools ✓ active via Nous subscription
Image gen ✓ active via Nous subscription
TTS ✓ active via Nous subscription
Browser ○ active via Browser Use key
```
Tools marked "active via Nous subscription" are going through the gateway. Anything else is using your own keys.
## Eligibility
The Tool Gateway is a **paid-subscription** feature. Free-tier Nous accounts can use Portal for inference but don't include managed tools — [upgrade your plan](https://portal.nousresearch.com/manage-subscription) to unlock the gateway.
Some accounts are also entitled to a **free tool pool** — a small managed-tool allowance that covers gateway tool calls without a paid subscription. When a free pool is available, the gateway surfaces it and shows a setup prompt on first use, so you can opt in and start using managed tools right away.
## The enablement checklist
Picking a Nous model (`hermes model`) offers a per-tool checklist of gateway backends. Its behavior respects your existing setup:
- Tools you've explicitly pointed at another backend (e.g. `web.backend: searxng`, `browser.cloud_provider: camofox`) are **never offered** — your selection can't be accidentally overwritten.
- Tools configured via environment variables alone (e.g. `SEARXNG_URL`, `CAMOFOX_URL`) are offered **unchecked**, labeled to keep your own backend.
- Only genuinely unconfigured tools come pre-checked.
- Declines stick: if you submit the checklist with a tool unchecked, it won't be pre-checked on future Nous model swaps (stored in `tool_gateway_declined_tools` in `config.yaml`; checking it later clears the decline).
## Mix and match
The gateway is per-tool. Turn it on for just what you want:
- **All tools through Nous** — easiest; one subscription, done.
- **Gateway for web + images, bring your own TTS** — keep your ElevenLabs voice, let Nous handle the rest.
- **Gateway only for things you don't have keys for** — "I already pay for Browserbase, but I don't want a Firecrawl account" works fine.
Switch any tool at any time via:
```bash
hermes tools # Interactive picker for each tool category
```
Select the tool, pick **Nous Subscription** as the provider (or any direct provider you prefer). No config editing required. If you aren't logged into Nous Portal yet, picking **Nous Subscription** kicks off the Portal login inline — you don't need to authenticate through `hermes model` first.
## Using individual image models
Image generation defaults to FLUX 2 Klein 9B for speed. Override per-call by passing the model ID to the `image_generate` tool:
| Model | ID | Best for |
|---|---|---|
| FLUX 2 Klein 9B | `fal-ai/flux-2/klein/9b` | Fast, good default |
| FLUX 2 Pro | `fal-ai/flux-2-pro` | Higher fidelity FLUX |
| Z-Image Turbo | `fal-ai/z-image/turbo` | Stylized, fast |
| Nano Banana Pro | `fal-ai/nano-banana-pro` | Google Gemini 3 Pro Image |
| GPT Image 1.5 | `fal-ai/gpt-image-1.5` | OpenAI image gen, text+image |
| GPT Image 2 | `fal-ai/gpt-image-2` | OpenAI latest |
| Ideogram V3 | `fal-ai/ideogram/v3` | Strong prompt adherence + typography |
| Recraft V4 Pro | `fal-ai/recraft/v4/pro/text-to-image` | Vector-style, graphic design |
| Qwen Image | `fal-ai/qwen-image` | Alibaba multimodal |
The set evolves — `hermes tools` → Image Generation shows the current live list.
---
## Configuration reference
Most users never need to touch this — `hermes model` and `hermes tools` cover every workflow interactively. This section is for writing config.yaml directly or scripting setups.
### One selection key per tool category
Each tool category has a single provider-selection key, written by the `hermes tools` picker (or the desktop GUI). Picking the **Nous Subscription** row stores the value `nous`, which routes that category through the managed Tool Gateway. Picking a BYOK row stores the vendor name (`fal`, `openai`, `firecrawl`, `browser-use`, ...), which goes direct with your own credentials:
```yaml
web:
backend: nous # web search/extract via the Tool Gateway
image_gen:
provider: nous # image generation via the Tool Gateway
tts:
provider: nous # TTS via the Tool Gateway
stt:
provider: nous # speech-to-text via the Tool Gateway
browser:
cloud_provider: nous # cloud browser via the Tool Gateway
```
The runtime **always uses the stored selection** — credential presence never selects or reroutes a category. A `FAL_KEY` sitting in `.env` is ignored while `image_gen.provider: nous`; conversely, `image_gen.provider: fal` with no `FAL_KEY` set produces a clear error instead of silently falling back to the gateway:
```
image_gen is configured to use fal (set via hermes tools), but FAL_KEY is not set. Run 'hermes tools' to change it.
```
Categories you have **never configured** (no selection key ever written) autodetect from available credentials, same as before. But once a selection exists, adding a key to `.env` does not change the route — only `hermes tools` (or editing the selection key) does.
### Switching back to your own keys
```bash
hermes tools # pick the tool → choose a direct provider (e.g. Firecrawl)
```
Or set the selection key directly:
```yaml
web:
backend: firecrawl # Hermes now uses FIRECRAWL_API_KEY from .env
```
### Legacy `use_gateway` flag (deprecated)
Older Hermes versions used a per-tool `use_gateway: true` boolean to route through the gateway. That flag is **legacy**: it is never written anymore, and the `hermes tools` picker removes it from a category's config when it rewrites the selection. Old configs that still contain `use_gateway: true` are interpreted at read time as the `nous` selection, so existing setups keep working. Don't set `use_gateway` in new configs — select the provider in `hermes tools` instead.
### Self-hosted gateway (advanced)
Running your own Nous-compatible gateway? Override endpoints in `~/.hermes/.env`:
```bash
TOOL_GATEWAY_DOMAIN=your-domain.example.com
TOOL_GATEWAY_SCHEME=https
TOOL_GATEWAY_USER_TOKEN=your-token # normally auto-populated from Portal login
FIRECRAWL_GATEWAY_URL=https://... # override one endpoint specifically
```
These knobs exist for custom infrastructure setups (enterprise deployments, dev environments). Regular subscribers never set them.
## FAQ
### Does it work with Telegram / Discord / the other messaging gateways?
Yes. Tool Gateway operates at the tool-execution layer, not the CLI. Every interface that can call a tool — CLI, Telegram, Discord, Slack, IRC, Teams, the API server, anything — benefits from it transparently.
### What happens if my subscription expires?
Tools routed through the gateway stop working until you renew or swap in direct API keys via `hermes tools`. Hermes shows a clear error pointing at the portal.
### Can I see usage or costs per tool?
Yes — the [Nous Portal dashboard](https://portal.nousresearch.com) breaks usage down by tool so you can see what's driving your bill.
### Is Modal (serverless terminal) included?
Modal is available as an **optional add-on** through the Nous subscription, not part of the default Tool Gateway bundle. Configure it via `hermes setup terminal` or directly in `config.yaml` when you want a remote sandbox for shell execution.
### Do I need to delete my existing API keys when I enable the gateway?
No — keep them in `.env`. While a tool's selection is **Nous Subscription**, direct keys for that tool are simply ignored. Pick the direct provider again in `hermes tools` and your keys become the source again. The gateway isn't a lock-in.
website/docs/user-guide/features/tool-search.md
+213
View File
@@ -0,0 +1,213 @@
---
title: Tool Search
sidebar_position: 95
---
# Tool Search
When you have many MCP servers or non-core plugin tools attached to a
session, their JSON schemas can consume a substantial fraction of the
context window on every turn — even when only a few of them are relevant
to what the user actually asked for.
**Tool Search** is Hermes' opt-in progressive-disclosure layer for that
problem. When activated, MCP and plugin tools are replaced in the
model-visible tools array by three bridge tools, and the model loads each
specific tool's schema on demand.
:::info Built-in Hermes tools never defer
The tools that make up Hermes' core capability set (`terminal`,
`read_file`, `write_file`, `patch`, `search_files`, `todo`, `memory`,
`browser_*`, `web_search`, `web_extract`, `clarify`, `execute_code`,
`delegate_task`, `session_search`, and the rest of
`_HERMES_CORE_TOOLS`) are *always* loaded directly. Only MCP tools and
non-core plugin tools are eligible for deferral.
:::
## How it works
When Tool Search activates for a turn, the model sees three new tools in
place of the deferred ones:
```
tool_search(queries, limit?) — search the deferred-tool catalog (one or more queries)
tool_describe(names) — load the full schemas for one or more tools
tool_call(name, arguments) — invoke a deferred tool
```
A typical interaction looks like:
```
Model: tool_search(["create a github issue", "send a slack message"])
→ { results: [ { query: "create a github issue",
matches: ["mcp_github_create_issue", ...] },
{ query: "send a slack message",
matches: ["mcp_slack_post_message", ...] } ],
tools: { mcp_github_create_issue: { description: "...",
required: ["title"], ... },
mcp_slack_post_message: { ... } } }
Model: tool_describe(["mcp_github_create_issue", "mcp_slack_post_message"])
→ { tools: { mcp_github_create_issue: { parameters: { ... } },
mcp_slack_post_message: { parameters: { ... } } } }
Model: tool_call("mcp_github_create_issue", { title: "...", body: "..." })
→ { ok: true, issue_number: 42 }
```
Each query in a `tool_search` call is searched independently against the
same catalog (`limit` applies per query); the per-query groups carry tool
names only, while the shared `tools` map holds each matched tool's
description and required parameter names once. Queries are stemmed, so
"issues" finds `create_issue`. Each query group that returns no matches
includes an `available_sources` summary of the connected servers so a lexical
miss is not mistaken for a missing capability.
`tool_describe` resolves every requested name in one call; unknown names
are reported in `not_found` without failing the rest of the batch.
When the model invokes `tool_call`, Hermes **unwraps the bridge** and
dispatches the underlying tool exactly as if the model had called it
directly. Pre-tool-call hooks, guardrails, approval prompts, and
post-tool-call hooks all run against the real tool name — not against
`tool_call`. The activity feed in the CLI and gateway also unwraps so you
see the underlying tool, not the bridge.
## When does it activate?
Tool Search uses **tiered disclosure**: the presence of *any* deferrable
(MCP/plugin) tool activates the bridge; what scales with catalog size is
how much of the catalog stays visible, not whether schemas defer.
| Tier | Condition | What the model sees |
| --- | --- | --- |
| **0** | No MCP/plugin tools | Every tool eager, no bridge. Pass-through. |
| **1** | Deferred catalog's listing fits the budget | Bridge + a skills-style manifest of every deferred tool (name + short description, degrading to names-only when over budget). Degradation is **per server**: when one oversized server (Cloudflare) is attached alongside small ones (Linear), the small servers keep their per-tool listings and only the oversized server collapses to a summary line. |
| **2** | Per-tool listing exceeds the budget even names-only for every server (e.g. Cloudflare's flat API surface alone: ~3,300 tools whose names are ~32K tokens) | Bare bridge + a one-line-per-server summary (server name + tool count), so the model knows which domains are reachable; individual tools are discoverable only through `tool_search`. |
The listing budget is `min(threshold_pct% of context, listing_max_tokens)`.
The decision is re-evaluated every time the tools array is built, so
adding or removing MCP servers mid-session moves the session between
tiers on the next assembly.
## Configuration
```yaml
tools:
tool_search:
enabled: auto # auto (default), on, or off
threshold_pct: 5 # listing budget as a percentage of context
search_default_limit: 5
max_search_limit: 25
listing: auto # embed a grouped name+description catalog manifest
listing_max_tokens: 4000
```
| Key | Default | Meaning |
| --- | --- | --- |
| `enabled` | `auto` | `auto`/`on` activate whenever at least one deferrable tool exists; `off` disables entirely (everything stays eager). `auto` is currently an alias of `on` — it is reserved for a future mode that inlines schemas when they fit the context and defers only when they don't. Pin `on` or `off` if you want today's behavior guaranteed across upgrades. |
| `threshold_pct` | `5` | Listing budget as a percentage of the active model's context length. Range 0–100. |
| `search_default_limit` | `5` | Hits returned per query when the model calls `tool_search` without a `limit`. |
| `max_search_limit` | `25` | Hard upper bound the model can request via `limit` (per query). Range 1–50. |
| `listing` | `auto` | Embed a skills-style manifest of every deferred tool (name + first sentence of its description, ≤60 chars, grouped by MCP server) in the `tool_search` bridge description. `auto` includes it when it fits the budget (falling back to names-only, then to the tier-2 server summary); `on`/`off` force either way. |
| `listing_max_tokens` | `4000` | Absolute cap on the embedded listing, regardless of context size. Range 200–60000. Large catalogs degrade to names-only or per-server summaries, keeping full schemas available through search. |
Per-call array caps are internal safety bounds, not configuration. Over-cap
calls return an error so the model can retry with a smaller batch.
### Why the listing exists
Without it, deferred capabilities are *invisible* — live benchmarking showed
models substituting visible core tools (running `gh` in the terminal instead
of searching for the deferred GitHub tool) or declaring a capability
nonexistent instead of calling `tool_search`. The listing applies the skills
pattern to tools: every capability stays discoverable by name at all times,
while full parameter schemas remain deferred. If the model sees the exact
tool name in the listing, it can skip `tool_search` and go straight to
`tool_describe`, saving a round trip.
You can also flip the legacy boolean shape:
```yaml
tools:
tool_search: true # equivalent to {enabled: auto}
```
## When NOT to use it
Tool Search trades a fixed per-turn token cost (the three bridge tool
schemas plus the catalog listing) and at least one extra round trip on
cold tools (describe → call) for the savings on the deferred schemas.
At tier 1 the listing keeps every capability visible, so the discovery
round trip usually disappears — the model goes straight to
`tool_describe`. Live benchmarking showed the listing mode matching
eager loading's task success while costing less than the bare bridge.
If you want the old always-eager behavior for a small toolset, set
`enabled: off`.
## Trade-offs that don't go away
These come from the prompt-cache integrity invariant — they are inherent
to any progressive-disclosure design, not specific to this implementation:
- **One extra round trip on cold tools.** The first time the model needs
a deferred tool, it spends one or two extra model calls to find and
load the schema. The token savings on the static side are real, but a
portion is paid back at runtime.
- **No cache benefit on deferred schemas.** A loaded `tool_describe`
result enters the conversation history (so it does get cached on
subsequent turns) but it never benefits from the system-prompt cache
prefix.
- **No provider-native validation for deferred schemas.** `tool_describe`
lets the model read a deferred tool's schema, but the provider still sees
only the generic `tool_call.arguments` object. Hermes therefore coerces and
validates the underlying arguments locally before dispatch; the concrete
tool or MCP server remains responsible for schemas Hermes cannot safely
validate, such as malformed schemas or external references.
- **Model-quality dependence.** Tool Search assumes the model can write a
reasonable search query for the tool it wants. Smaller models do this
less well; the published Anthropic numbers (49% → 74% on Opus 4 with
vs. without tool search) show the upside but also that ~26 points of
accuracy is still retrieval failure.
- **Toolset edits invalidate cache.** Adding or removing a tool mid-
session changes the bridge tools' descriptions (which include the
count of deferred tools) and the catalog, so the prompt cache is
invalidated. This is the same trade-off as any toolset edit.
## Implementation details
- **Retrieval:** BM25 over tokenized tool name, source name (the MCP
server or plugin toolset the tool belongs to, so searching `"linear"`
finds that server's tools even when a tool's own name doesn't carry
the service), description, and parameter names, with Snowball
stemming (English) applied to both the index and the query so
morphological variants match ("issues" finds `create_issue`). Falls
back to a literal substring match on the tool name when no query
token matches any document (e.g. searching `"hub"` where the token is
`github`).
- **Parallel execution unwraps the bridge.** The batch planner decides
concurrency on the *underlying* tool of a `tool_call`, not on the
literal bridge name — so an MCP server opted in via
`supports_parallel_tool_calls: true` keeps its concurrency when its
tools are called through the bridge, and `tool_search` /
`tool_describe` lookups batch concurrently like any read-only tool.
- **Catalog is stateless across turns.** It rebuilds from the current
tool-defs list every assembly — no session-keyed `Map`. This avoids
the class of bug where a stored catalog drifts out of sync with the
live tool registry.
- **The catalog is scoped to the session's toolsets.** `tool_search`,
`tool_describe`, and `tool_call` only ever see and invoke tools the
session was actually granted. A subagent, kanban worker, or gateway
session restricted to a subset of toolsets cannot use the bridge to
discover or call a tool outside that subset — the deferred catalog is
the deferrable slice of the session's own enabled/disabled toolsets,
not the whole process registry.
- **No JS sandbox.** Hermes uses the simpler "structured tools" mode
(search / describe / call as plain functions). The JS-sandbox "code
mode" some other implementations offer is a large surface area; we
skip it.
## See also
- `tools/tool_search.py` — the implementation
- `tests/tools/test_tool_search.py` — the regression suite
- The `openclaw-tool-search-report` PDF in the original implementation
PR for the research that shaped the design
website/docs/user-guide/features/tools.md
+240
View File
@@ -0,0 +1,240 @@
---
sidebar_position: 1
title: "Tools & Toolsets"
description: "Overview of Hermes Agent's tools — what's available, how toolsets work, and terminal backends"
---
# Tools & Toolsets
Tools are functions that extend the agent's capabilities. They're organized into logical **toolsets** that can be enabled or disabled per platform.
## Available Tools
Hermes ships with a broad built-in tool registry covering web search, browser automation, terminal execution, file editing, memory, delegation, scheduled tasks, Home Assistant, and more.
:::note
**Honcho cross-session memory** is available as a memory provider plugin (`plugins/memory/honcho/`), not as a built-in toolset. See [Plugins](./plugins.md) for installation.
:::
High-level categories:
| Category | Examples | Description |
|----------|----------|-------------|
| **Web** | `web_search`, `web_extract` | Search the web and extract page content. |
| **X Search** | `x_search` | Search X (Twitter) posts and threads via xAI's built-in `x_search` Responses tool — gated on xAI credentials (SuperGrok OAuth or `XAI_API_KEY`); off by default, opt in via `hermes tools` → 🐦 X (Twitter) Search. |
| **Terminal & Files** | `terminal`, `process`, `read_file`, `patch` | Execute commands and manipulate files. |
| **Browser** | `browser_navigate`, `browser_snapshot`, `browser_vision` | Interactive browser automation with text and vision support. |
| **Media** | `vision_analyze`, `image_generate`, `text_to_speech` | Multimodal analysis and generation. |
| **Agent orchestration** | `todo`, `clarify`, `execute_code`, `delegate_task` | Planning, clarification, code execution, and subagent delegation. |
| **Memory & recall** | `memory`, `session_search` | Persistent memory and session search. |
| **Automation** | `cronjob` | Scheduled tasks with create/list/update/pause/resume/run/remove actions. Outbound delivery is handled by cron's own delivery, the `hermes send` CLI, and the gateway notifier — not by an agent-callable tool. |
| **Integrations** | `ha_*`, MCP server tools | Home Assistant, MCP, and other integrations. |
For the authoritative code-derived registry, see [Built-in Tools Reference](/reference/tools-reference) and [Toolsets Reference](/reference/toolsets-reference).
:::tip Nous Tool Gateway
Paid [Nous Portal](https://portal.nousresearch.com) subscribers can use web search, image generation, TTS, and browser automation through the **[Tool Gateway](tool-gateway.md)** — no separate API keys needed. Run `hermes model` to enable it, or configure individual tools with `hermes tools`.
:::
## Using Toolsets
```bash
# Use specific toolsets
hermes chat --toolsets "web,terminal"
# See all available tools
hermes tools
# Configure tools per platform (interactive)
hermes tools
```
Common toolsets include `web`, `search`, `terminal`, `file`, `browser`, `vision`, `image_gen`, `skills`, `tts`, `todo`, `memory`, `session_search`, `cronjob`, `code_execution`, `delegation`, `clarify`, `homeassistant`, `messaging`, `spotify`, `discord`, `discord_admin`, `debugging`, and `safe`.
See [Toolsets Reference](/reference/toolsets-reference) for the full set, including platform presets such as `hermes-cli`, `hermes-telegram`, and dynamic MCP toolsets like `mcp-<server>`.
## Tool result annotations
A few tool behaviors are worth knowing when you read agent transcripts:
- **Signal deaths are explained.** When a terminal command is killed by a signal, the result carries a human-readable note instead of a bare numeric code — e.g. exit `-9`/`137` becomes "terminated by signal 9: SIGKILL — often the kernel OOM killer on memory exhaustion, or an explicit kill -9", and segfaults, aborts, SIGTERM, broken pipes, and CPU/file-size limits are labeled the same way. Negative codes (subprocess semantics) are stated definitively; the shell's `128+signum` convention is hedged with "usually" since an application can legitimately exit with those codes.
- **UTF-16 text files are transcoded, not refused.** `read_file` detects UTF-16 (BOM or byte-pattern heuristic, either endianness — common for Windows Notepad files and PowerShell `>` redirects) and transcodes it to UTF-8 for display instead of flagging the file as binary. The result includes a hint disclosing the conversion; edits via `patch`/`write_file` re-encode as UTF-8. Files over 10 MB and genuinely binary files still get the binary-file refusal.
## Terminal Backends
The terminal tool can execute commands in different environments:
| Backend | Description | Use Case |
|---------|-------------|----------|
| `local` | Run on your machine (default) | Development, trusted tasks |
| `docker` | Isolated containers | Security, reproducibility |
| `ssh` | Remote server | Sandboxing, keep agent away from its own code |
| `singularity` | HPC containers | Cluster computing, rootless |
| `modal` | Cloud execution | Serverless, scale |
| `daytona` | Cloud sandbox workspace | Persistent remote dev environments |
| `vercel_sandbox` | Vercel Sandbox cloud microVM | Cloud execution with snapshot-backed filesystem persistence |
### Configuration
```yaml
# In ~/.hermes/config.yaml
terminal:
backend: local # or: docker, ssh, singularity, modal, daytona, vercel_sandbox
cwd: "." # Working directory
timeout: 180 # Command timeout in seconds
```
### Shell startup files and non-interactive commands
Agent terminal calls run your shell **non-interactively** — there is no TTY and no human at the prompt. Heavy or interactive shell initialisation that you never notice in a normal terminal can break or badly slow every command the agent runs:
- **Slow init (`nvm`, version managers, network-touching prompts):** the classic `nvm.sh` sourcing adds noticeable latency to *every* shell start, and the agent starts many shells. Multi-second rc files turn a quick `git status` into a timeout risk.
- **TTY-expecting blocks:** anything in `.bashrc`/`.zshrc` that prompts, runs `tmux`/`screen` attach, calls `read`, or prints a menu will hang a non-interactive shell — the command appears to run forever and then times out.
- **Unconditional output:** rc files that `echo` banners pollute every command's output the agent has to parse.
The fix is the standard guard most distros already ship at the top of `.bashrc` — return early when the shell is non-interactive, and keep anything heavy or interactive below it:
```bash
# ~/.bashrc — keep this guard near the top
case $- in
*i*) ;; # interactive: continue
*) return;; # non-interactive: stop here
esac
# heavy/interactive init goes BELOW the guard
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
```
Zsh users: put login-only setup in `.zprofile` and interactive-only setup in `.zshrc`; keep `.zshenv` minimal, since it runs for every shell including non-interactive ones. If the agent genuinely needs a tool that only your rc file puts on `PATH`, export the `PATH` change *above* the guard (path exports are cheap) or symlink the binary into `~/.local/bin`.
If agent terminal commands hang or time out immediately after working in your own terminal, your shell init is the first suspect.
### Docker Backend
```yaml
terminal:
backend: docker
docker_image: python:3.11-slim
```
**One persistent container, shared across the whole process.** Hermes starts a single long-lived container on first use (`docker run -d ... sleep infinity`) and routes every terminal, file, and `execute_code` call through `docker exec` into that same container. Working-directory changes, installed packages, environment tweaks, and files written to `/workspace` all carry over from one tool call to the next, across `/new`, `/reset`, and `delegate_task` subagents, for the lifetime of the Hermes process. The container is stopped and removed on shutdown.
This means the Docker backend behaves like a persistent sandbox VM, not a fresh container per command. If you `pip install foo` once, it's there for the rest of the session. If you `cd /workspace/project`, subsequent `ls` calls see that directory. See [Configuration → Docker Backend](../configuration.md#docker-backend) for the full lifecycle details and the `container_persistent` flag that controls whether `/workspace` and `/root` survive across Hermes restarts.
### SSH Backend
Recommended for security — agent can't modify its own code:
```yaml
terminal:
backend: ssh
```
```bash
# Set credentials in ~/.hermes/.env
TERMINAL_SSH_HOST=my-server.example.com
TERMINAL_SSH_USER=myuser
TERMINAL_SSH_KEY=~/.ssh/id_rsa
```
### Singularity/Apptainer
```bash
# Pre-build SIF for parallel workers
apptainer build ~/python.sif docker://python:3.11-slim
# Configure
hermes config set terminal.backend singularity
hermes config set terminal.singularity_image ~/python.sif
```
### Modal (Serverless Cloud)
```bash
uv pip install modal
modal setup
hermes config set terminal.backend modal
```
### Vercel Sandbox
```bash
pip install 'hermes-agent[vercel]'
hermes config set terminal.backend vercel_sandbox
hermes config set terminal.vercel_runtime node24
```
Authenticate with all three of `VERCEL_TOKEN`, `VERCEL_PROJECT_ID`, and `VERCEL_TEAM_ID`. This access-token setup is the supported path for deployments and normal long-running Hermes processes on Render, Railway, Docker, and similar hosts. Supported runtimes are `node24`, `node22`, and `python3.13`; Hermes defaults to `/vercel/sandbox` as the remote workspace root.
For one-off local development, Hermes also accepts short-lived Vercel OIDC tokens:
```bash
VERCEL_OIDC_TOKEN="$(vc project token <project-name>)" hermes chat
```
From a linked Vercel project directory:
```bash
VERCEL_OIDC_TOKEN="$(vc project token)" hermes chat
```
With `container_persistent: true`, Hermes uses Vercel snapshots to preserve filesystem state across sandbox recreation for the same task. This can include Hermes-synced credentials, skills, and cache files inside the sandbox. Snapshots do not preserve live processes, PID space, or the same live sandbox identity.
Background terminal commands use Hermes' generic non-local process flow: spawn, poll, wait, log, and kill work through the normal process tool while the sandbox is alive, but Hermes does not provide native Vercel detached-process recovery after cleanup or restart.
Leave `container_disk` unset or at the shared default `51200`; custom disk sizing is unsupported for Vercel Sandbox and will fail diagnostics/backend creation.
### Container Resources
Configure CPU, memory, disk, and persistence for all container backends:
```yaml
terminal:
backend: docker # or singularity, modal, daytona, vercel_sandbox
container_cpu: 1 # CPU cores (default: 1)
container_memory: 5120 # Memory in MB (default: 5GB)
container_disk: 51200 # Disk in MB (default: 50GB)
container_persistent: true # Persist filesystem across sessions (default: true)
```
When `container_persistent: true`, installed packages, files, and config survive across sessions.
### Container Security
All container backends run with security hardening:
- Read-only root filesystem (Docker)
- All Linux capabilities dropped
- No privilege escalation
- PID limits (256 processes)
- Full namespace isolation
- Persistent workspace via volumes, not writable root layer
Docker can optionally receive an explicit env allowlist via `terminal.docker_forward_env`, but forwarded variables are visible to commands inside the container and should be treated as exposed to that session.
## Background Process Management
Start background processes and manage them:
```python
terminal(command="pytest -v tests/", background=true)
# Returns: {"session_id": "proc_abc123", "pid": 12345}
# Then manage with the process tool:
process(action="list") # Show all running processes
process(action="poll", session_id="proc_abc123") # Check status
process(action="wait", session_id="proc_abc123") # Block until done
process(action="log", session_id="proc_abc123") # Full output
process(action="kill", session_id="proc_abc123") # Terminate
process(action="write", session_id="proc_abc123", data="y") # Send input
```
PTY mode (`pty=true`) enables interactive CLI tools like Codex and Claude Code.
## Sudo Support
If a command needs sudo, you'll be prompted for your password (cached for the session). Or set `SUDO_PASSWORD` in `~/.hermes/.env`.
:::warning
On messaging platforms, if sudo fails, the output includes a tip to add `SUDO_PASSWORD` to `~/.hermes/.env`.
:::
website/docs/user-guide/features/tts.md
+735
View File
@@ -0,0 +1,735 @@
---
sidebar_position: 9
title: "Voice & TTS"
description: "Text-to-speech and voice message transcription across all platforms"
---
# Voice & TTS
Hermes Agent supports both text-to-speech output and voice message transcription across all messaging platforms.
:::tip Nous Subscribers
If you have a paid [Nous Portal](https://portal.nousresearch.com) subscription, OpenAI TTS is available through the **[Tool Gateway](tool-gateway.md)** without a separate OpenAI API key. New installs can run `hermes setup --portal` to log in and turn on every gateway tool at once; existing installs can pick **Nous Subscription** for just TTS via `hermes model` or `hermes tools`.
:::
## Text-to-Speech
Convert text to speech with eleven providers:
| Provider | Quality | Cost | API Key |
|----------|---------|------|---------|
| **Edge TTS** (default) | Good | Free | None needed |
| **ElevenLabs** | Excellent | Paid | `ELEVENLABS_API_KEY` |
| **OpenAI TTS** | Good | Paid | `VOICE_TOOLS_OPENAI_KEY` |
| **MiniMax TTS** | Excellent | Paid | `MINIMAX_API_KEY` or `MINIMAX_CN_API_KEY` |
| **Mistral (Voxtral TTS)** | Excellent | Paid | `MISTRAL_API_KEY` |
| **Google Gemini TTS** | Excellent | Free tier | `GEMINI_API_KEY` |
| **xAI TTS** | Excellent | Paid | `XAI_API_KEY` |
| **DeepInfra TTS** | Good | Paid | `DEEPINFRA_API_KEY` |
| **NeuTTS** | Good | Free (local) | None needed |
| **KittenTTS** | Good | Free (local) | None needed |
| **Piper** | Good | Free (local) | None needed |
### Platform Delivery
| Platform | Delivery | Format |
|----------|----------|--------|
| Telegram | Voice bubble (plays inline) | Opus `.ogg` |
| Discord | Voice bubble (Opus/OGG), falls back to file attachment | Opus/MP3 |
| WhatsApp | Audio file attachment | MP3 |
| CLI | Saved to `~/.hermes/audio_cache/` | MP3 |
### Configuration
```yaml
# In ~/.hermes/config.yaml
tts:
provider: "edge" # "edge" | "elevenlabs" | "openai" | "minimax" | "mistral" | "gemini" | "xai" | "deepinfra" | "neutts" | "kittentts" | "piper" — or "nous" for the managed Tool Gateway (written when you pick Nous Subscription in `hermes tools`)
speed: 1.0 # Global speed multiplier (provider-specific settings override this)
edge:
voice: "en-US-AriaNeural" # 322 voices, 74 languages
speed: 1.0 # Converted to rate percentage (+/-%)
elevenlabs:
voice_id: "pNInz6obpgDQGcFmaJgB" # Adam
model_id: "eleven_multilingual_v2"
openai:
model: "gpt-4o-mini-tts"
voice: "alloy" # alloy, echo, fable, onyx, nova, shimmer
base_url: "https://api.openai.com/v1" # Override for OpenAI-compatible TTS endpoints
speed: 1.0 # 0.25 - 4.0
# language: "es" # Sent as lang_code — only for OpenAI-compatible endpoints that support it (e.g. Kokoro)
minimax:
region: "global" # "global" or "cn"; see selection rules below
model: "speech-02-hd" # speech-02-hd (default), speech-02-turbo
voice_id: "English_expressive_narrator" # See https://platform.minimax.io/faq/system-voice-id
speed: 1 # 0.5 - 2.0
vol: 1 # 0 - 10
pitch: 0 # -12 - 12
# base_url: "https://tts.example/v1/t2a_v2" # Optional endpoint override for the selected region
mistral:
model: "voxtral-mini-tts-2603"
voice_id: "c69964a6-ab8b-4f8a-9465-ec0925096ec8" # Paul - Neutral (default)
gemini:
model: "gemini-2.5-flash-preview-tts" # or gemini-3.1-flash-tts-preview
voice: "Kore" # 30 prebuilt voices: Zephyr, Puck, Kore, Enceladus, Gacrux, etc.
audio_tags: false # Enable hidden Gemini 3.1 TTS audio-tag insertion
persona_prompt_file: "" # Optional Markdown/text file with Gemini voice direction
xai:
voice_id: "eve" # or a custom voice ID — see docs below
language: "en" # BCP-47 code (e.g. "en", "pt-BR") or "auto" for detection
speed: 1.0 # 0.7–1.5, playback speed (default: 1.0)
auto_speech_tags: false # insert expressive audio tags via LLM rewrite
text_normalization: false # normalize numbers/abbreviations/symbols to spoken form
optimize_streaming_latency: 0 # 0–2, trades quality for lower latency (default: 0)
sample_rate: 24000 # 22050 / 24000 (default) / 44100 / 48000
bit_rate: 128000 # MP3 bitrate; only applies when codec=mp3
# base_url: "https://api.x.ai/v1" # Override via XAI_BASE_URL env var
neutts:
ref_audio: ''
ref_text: ''
model: neuphonic/neutts-air-q4-gguf
device: cpu
kittentts:
model: KittenML/kitten-tts-nano-0.8-int8 # 25MB int8; also: kitten-tts-micro-0.8 (41MB), kitten-tts-mini-0.8 (80MB)
voice: Jasper # Jasper, Bella, Luna, Bruno, Rosie, Hugo, Kiki, Leo
speed: 1.0 # 0.5 - 2.0
clean_text: true # Expand numbers, currencies, units
piper:
voice: en_US-lessac-medium # voice name (auto-downloaded) OR absolute path to .onnx
# voices_dir: '' # default: ~/.hermes/cache/piper-voices/
# use_cuda: false # requires onnxruntime-gpu
# length_scale: 1.0 # 2.0 = twice as slow
# noise_scale: 0.667
# noise_w_scale: 0.8
# volume: 1.0 # 0.5 = half as loud
# normalize_audio: true
```
MiniMax TTS selects its region, endpoint, and credential together:
- `region: "global"` uses `https://api.minimax.io/v1/t2a_v2` with `MINIMAX_API_KEY`.
- `region: "cn"` uses `https://api.minimaxi.com/v1/t2a_v2` with `MINIMAX_CN_API_KEY`.
- If `region` is omitted, `MINIMAX_API_KEY` keeps precedence for backward compatibility. If only `MINIMAX_CN_API_KEY` is configured, Hermes selects `cn`.
- An explicitly selected region must have its matching credential. Hermes never borrows the other region's key. A `base_url` override does not change the selected credential, and an override pointing at the other region's official endpoint is rejected.
**Speed control**: The global `tts.speed` value applies to all providers by default. Each provider can override it with its own `speed` setting (e.g., `tts.openai.speed: 1.5`). Provider-specific speed takes precedence over the global value. Default is `1.0` (normal speed).
### Gemini Persona Prompts
Gemini TTS can follow natural-language performance direction. Set `tts.gemini.persona_prompt_file` to a local Markdown or text file that describes the voice persona. The file can include Gemini-style sections such as `AUDIO PROFILE`, `SCENE`, `DIRECTOR'S NOTES`, `SAMPLE CONTEXT`, and `TRANSCRIPT`.
If the file contains `{transcript}` or `{{ transcript }}`, Hermes replaces that placeholder with the live TTS text. Otherwise, Hermes appends a labeled `TRANSCRIPT` section automatically. The persona prompt stays local and is not shown in the chat reply.
```yaml
tts:
provider: gemini
gemini:
voice: Algieba
persona_prompt_file: ~/.hermes/tts/butler-voice.md
```
### Audio Tags (Gemini, xAI)
Google's Gemini 3.1 Flash TTS and xAI's Grok TTS support freeform square-bracket audio tags such as `[whispers]`, `[excitedly]`, `[very slow]`, `[laughs]`, and other expressive delivery notes. Enable `tts.gemini.audio_tags` or `tts.xai.auto_speech_tags` to have Hermes run a hidden rewrite pass before TTS. The rewrite inserts inline tags into the TTS script only; the visible chat reply stays unchanged.
```yaml
tts:
provider: gemini
gemini:
model: gemini-3.1-flash-tts-preview
audio_tags: true
xai:
auto_speech_tags: true
```
The rewrite uses `auxiliary.tts_audio_tags` and defaults to your main chat model. Override that auxiliary task if you want tag insertion handled by a cheaper or faster model.
**Language (OpenAI-compatible endpoints)**: `tts.openai.language` is forwarded to the endpoint as a `lang_code` request parameter. It is intended for OpenAI-compatible TTS servers that support `lang_code` — for example [Kokoro-FastAPI](https://github.com/remsky/Kokoro-FastAPI), where `language: "es"` selects the Spanish phonemizer instead of the English default. Leave it unset when using the official OpenAI API, which does not accept this parameter. When unset, nothing extra is sent.
### Input length limits
Each provider has a documented per-request input-character cap. Hermes splits longer replies into ordered, sentence-aware chunks before calling the provider, so the full normalized text is preserved instead of silently truncated:
| Provider | Default cap (chars) |
|----------|---------------------|
| Edge TTS | 5000 |
| OpenAI | 4096 |
| xAI | 15000 |
| MiniMax | 10000 |
| Mistral | 4000 |
| Google Gemini | 32000 |
| ElevenLabs | Model-aware (see below) |
| NeuTTS | 2000 |
| KittenTTS | 2000 |
| Piper | 5000 |
**ElevenLabs** picks a cap from the configured `model_id`:
| `model_id` | Cap (chars) |
|------------|-------------|
| `eleven_flash_v2_5` | 40000 |
| `eleven_flash_v2` | 30000 |
| `eleven_multilingual_v2` (default), `eleven_multilingual_v1`, `eleven_english_sts_v2`, `eleven_english_sts_v1` | 10000 |
| `eleven_v3`, `eleven_ttv_v3` | 5000 |
| Unknown model | Falls back to provider default (10000) |
**Override per provider** with `max_text_length:` under the provider section of your TTS config:
```yaml
tts:
openai:
max_text_length: 8192 # raise or lower the provider cap
```
Only positive integers are honored. Zero, negative, non-numeric, or boolean values fall through to the provider default, so a broken config can't accidentally bypass the provider request limit.
### Telegram Voice Bubbles & ffmpeg
Telegram voice bubbles require Opus/OGG audio format:
- **OpenAI, ElevenLabs, and Mistral** produce Opus natively — no extra setup
- **Edge TTS** (default) outputs MP3 and needs **ffmpeg** to convert:
- **MiniMax TTS** outputs MP3 and needs **ffmpeg** to convert for Telegram voice bubbles
- **Google Gemini TTS** outputs raw PCM and uses **ffmpeg** to encode Opus directly for Telegram voice bubbles
- **xAI TTS** outputs MP3 and needs **ffmpeg** to convert for Telegram voice bubbles
- **NeuTTS** outputs WAV and also needs **ffmpeg** to convert for Telegram voice bubbles
- **KittenTTS** outputs WAV and also needs **ffmpeg** to convert for Telegram voice bubbles
- **Piper** outputs WAV and also needs **ffmpeg** to convert for Telegram voice bubbles
```bash
# Ubuntu/Debian
sudo apt install ffmpeg
# macOS
brew install ffmpeg
# Fedora
sudo dnf install ffmpeg
```
Without ffmpeg, Edge TTS, MiniMax TTS, NeuTTS, KittenTTS, and Piper audio are sent as regular audio files (playable, but shown as a rectangular player instead of a voice bubble).
:::tip
If you want voice bubbles without installing ffmpeg, switch to the OpenAI, ElevenLabs, or Mistral provider.
:::
### xAI Custom Voices (voice cloning)
xAI supports cloning your voice and using it with TTS. Create a custom voice in the [xAI Console](https://console.x.ai/team/default/voice/voice-library), then set the resulting `voice_id` in your config:
```yaml
tts:
provider: xai
xai:
voice_id: "nlbqfwie" # your custom voice ID
```
See the [xAI Custom Voices docs](https://docs.x.ai/developers/model-capabilities/audio/custom-voices) for details on recording, supported formats, and limits.
### Piper (local, 44 languages)
Piper is a fast, local neural TTS engine from the Open Home Foundation (the Home Assistant maintainers). It runs entirely on CPU, supports **44 languages** with pre-trained voices, and needs no API key.
**Install via `hermes tools`** → Voice & TTS → Piper — Hermes runs `pip install piper-tts` for you. Or install manually: `pip install piper-tts`.
**Switch to Piper:**
```yaml
tts:
provider: piper
piper:
voice: en_US-lessac-medium
```
On the first TTS call for a voice that isn't cached locally, Hermes runs `python -m piper.download_voices <name>` and downloads the model (~20-90MB depending on quality tier) into `~/.hermes/cache/piper-voices/`. Subsequent calls reuse the cached model.
**Picking a voice.** The [full voice catalog](https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/VOICES.md) covers English, Spanish, French, German, Italian, Dutch, Portuguese, Russian, Polish, Turkish, Chinese, Arabic, Hindi, and more — each with `x_low` / `low` / `medium` / `high` quality tiers. Sample voices at [rhasspy.github.io/piper-samples](https://rhasspy.github.io/piper-samples/).
**Using a pre-downloaded voice.** Set `tts.piper.voice` to an absolute path ending in `.onnx`:
```yaml
tts:
piper:
voice: /path/to/my-custom-voice.onnx
```
**Advanced knobs** (`tts.piper.length_scale` / `noise_scale` / `noise_w_scale` / `volume` / `normalize_audio`, `use_cuda`) correspond 1:1 to Piper's `SynthesisConfig`. They're ignored on older `piper-tts` versions.
### Warm-up and unload via speech toggles (local engines)
Local engines (Piper, KittenTTS) load their model lazily, so without help the *first* spoken reply after you turn speech on pays the whole model load — and on a fresh install the voice download — as silence before the first word. Hermes treats the speech-output toggles as the signal that TTS is about to be needed:
- **Desktop** — turning on **Read replies aloud**, or starting a **voice conversation**, pre-loads the configured engine in the background right away. Turning both off again unloads the resident model (a Piper voice is tens of MB; KittenTTS up to ~80MB) so it isn't parked in RAM for nothing.
- **CLI / TUI** — `/voice tts` (and `/voice on` when `voice.auto_tts` is set) do the same; `/voice off` releases.
Each toggle holds a *lease* on the engine; the model is only unloaded when the last lease across surfaces is released, so switching off read-aloud in one Desktop window never pulls the voice out from under a conversation running in another. For cloud providers there is no model to hold — the toggle only makes sure a lazily-installed SDK (edge-tts, ElevenLabs, Mistral) is present. Warm-up is best-effort: if the engine can't load, the toggle still succeeds and the first reply falls back to loading on demand as before.
The Desktop calls `POST /api/audio/tts-lease` with `{"lease": "<name>", "active": true|false}`; other frontends can use the same endpoint.
The same lease also reaches user-declared providers, so a self-hosted TTS server can preload and unload its model on the toggles: a [command provider](#custom-command-providers) runs its optional `warm_command` / `release_command`, and a [Python plugin provider](#python-plugin-providers) gets `warm()` / `release()`.
### Custom command providers
If a TTS engine you want isn't natively supported (VoxCPM, MLX-Kokoro, XTTS CLI, a voice-cloning script, anything else that exposes a CLI), you can wire it in as a **command-type provider** without writing any Python. Hermes writes the input text to a temp UTF-8 file, runs your shell command, and reads the audio file the command produced.
Declare one or more providers under `tts.providers.<name>` and switch between them with `tts.provider: <name>` — the same way you switch between built-ins like `edge` and `openai`.
```yaml
tts:
provider: voxcpm # pick any name under tts.providers
providers:
voxcpm:
type: command
command: "voxcpm --ref ~/voice.wav --text-file {input_path} --out {output_path}"
output_format: mp3
timeout: 180
voice_compatible: true # try to deliver as a Telegram voice bubble
mlx-kokoro:
type: command
command: "python -m mlx_kokoro --in {input_path} --out {output_path} --voice {voice}"
voice: af_sky
output_format: wav
piper-custom: # native Piper also supports custom .onnx via tts.piper.voice
type: command
command: "piper -m /path/to/custom.onnx -f {output_path} < {input_path}"
output_format: wav
```
**Supported `output_format` values:** `mp3` (default), `wav`, `ogg`, `flac`, `m4a`, `aac`, `amr`, `opus`. Your command must actually produce that format (e.g. via `ffmpeg`); Hermes only validates the declared value and names the output file accordingly. An unknown value falls back to `mp3`. The chosen format is also exposed to the command as the `{format}` placeholder.
**Subprocess environment:** command providers (TTS and STT) run with Hermes secrets scrubbed from the child environment — gateway bot tokens, LLM provider API keys, and internal relay credentials are removed; `PATH`, `HOME`, locale, and other normal variables are kept. If your command template needs its own API key from the environment (e.g. a `curl` one-liner), list the variable names under `env_passthrough` in the provider config:
```yaml
tts:
providers:
mycloud:
type: command
command: 'curl -s -H "Authorization: Bearer $MYCLOUD_API_KEY" ... -o {output_path}'
env_passthrough: [MYCLOUD_API_KEY]
```
#### Example: Doubao (Chinese seed-tts-2.0)
For high-quality Chinese TTS via ByteDance's [seed-tts-2.0](https://www.volcengine.com/docs/6561/1257544) bidirectional-streaming API, install the [`doubao-speech`](https://pypi.org/project/doubao-speech/) PyPI package and wire it in as a command provider:
```bash
pip install doubao-speech
export VOLCENGINE_APP_ID="your-app-id"
export VOLCENGINE_ACCESS_TOKEN="your-access-token"
```
```yaml
tts:
provider: doubao
providers:
doubao:
type: command
command: "doubao-speech say --text-file {input_path} --out {output_path}"
output_format: mp3
max_text_length: 1024
timeout: 30
```
Credentials come from your shell environment (`VOLCENGINE_APP_ID` / `VOLCENGINE_ACCESS_TOKEN`) or `~/.doubao-speech/config.yaml`. Pick a voice by adding `--voice zh-female-warm` (or any other alias from `doubao-speech list-voices`) to the command. `doubao-speech` also bundles streaming ASR — see the [STT section below](#example-doubao--volcengine-asr) for Hermes integration. Source and full docs: [github.com/Hypnus-Yuan/doubao-speech](https://github.com/Hypnus-Yuan/doubao-speech).
#### Placeholders
Your command template can reference these placeholders. Hermes substitutes them at render time and shell-quotes each value for the surrounding context (bare / single-quoted / double-quoted), so paths with spaces and other shell-sensitive characters are safe.
| Placeholder | Meaning |
|------------------|------------------------------------------------------|
| `{input_path}` | Path to the temp UTF-8 text file Hermes wrote |
| `{text_path}` | Alias for `{input_path}` |
| `{output_path}` | Path the command must write audio to |
| `{format}` | `mp3` / `wav` / `ogg` / `flac` |
| `{voice}` | `tts.providers.<name>.voice`, empty when unset |
| `{model}` | `tts.providers.<name>.model` |
| `{speed}` | Resolved speed multiplier (provider or global) |
Use `{{` and `}}` for literal braces.
#### Optional keys
| Key | Default | Meaning |
|--------------------|---------|------------------------------------------------------------------------------------------------------------|
| `timeout` | `120` | Idle seconds; stdout or stderr output resets the deadline. The process tree is killed after inactivity (Unix `killpg`, Windows `taskkill /T`). |
| `output_format` | `mp3` | One of `mp3` / `wav` / `ogg` / `flac`. Auto-inferred from the output extension if Hermes picks a path. |
| `voice_compatible` | `false` | When `true`, Hermes converts MP3/WAV output to Opus/OGG via ffmpeg so Telegram renders a voice bubble. |
| `max_text_length` | `5000` | Maximum input characters per command invocation; longer text is split into ordered chunks. |
| `voice` / `model` | empty | Passed to the command as placeholder values only. |
| `warm_command` / `release_command` | unset | Shell commands run when a surface toggles speech output on / when the last lease across surfaces is released — e.g. `curl -s localhost:5002/load?model={model}` to preload a local TTS server, and its `unload` counterpart. Best-effort and non-blocking: run in the background with the same `timeout`, `env_passthrough` and `{voice}` / `{model}` / `{speed}` placeholders as `command`; output is discarded and failures are only logged at debug. |
#### Behavior notes
- **Built-in names always win.** A `tts.providers.openai` entry never shadows the native OpenAI provider, so no user config can silently replace a built-in.
- **Default delivery is a document.** Command providers deliver as regular audio attachments on every platform. Opt in to voice-bubble delivery per-provider with `voice_compatible: true`.
- **Command failures surface to the agent.** Non-zero exit, empty output, or timeout all return an error with the command's stderr/stdout included so you can debug the provider from the conversation.
- **`type: command` is the default when `command:` is set.** Writing `type: command` explicitly is good practice but not required; an entry with a non-empty `command` string is treated as a command provider.
- **`{input_path}` / `{text_path}` are interchangeable.** Use whichever reads better in your command.
#### Security
Command-type providers run whatever shell command you configure, with your user's permissions. Hermes quotes placeholder values and enforces the configured timeout, but the command template itself is trusted local input — treat it the same way you would a shell script on your PATH.
### Python plugin providers
For TTS engines that can't be expressed as a single shell command — Python SDKs without a CLI, streaming engines, voice-listing APIs, OAuth-refreshing auth — register a Python plugin via `ctx.register_tts_provider()`. The plugin **coexists with** (does not replace) the [Custom command providers](#custom-command-providers) registry; pick the surface that fits your engine.
#### When to pick which
| Your backend has… | Use |
|---|---|
| A single CLI reading text from a file/stdin and writing audio to a file/stdout | **Command provider** (no Python needed) |
| Two or three CLIs chained with shell pipes | **Command provider** |
| A Python SDK only — no CLI | **Plugin** |
| Streaming bytes you want to deliver chunked (mid-generation voice bubbles) | **Plugin** (override `stream()`) |
| A voice-listing API used by `hermes setup` | **Plugin** (override `list_voices()`) |
| OAuth refresh flow (not a static bearer token) | **Plugin** |
Built-ins always win, and command providers win over a same-name plugin — so plugins are safe to register against any non-built-in name without worrying about shadowing your existing config.
#### Minimal plugin
Drop this in `~/.hermes/plugins/my-tts/`:
`plugin.yaml`:
```yaml
name: my-tts
version: 0.1.0
description: "My custom Python TTS backend"
```
`__init__.py`:
```python
from agent.tts_provider import TTSProvider
class MyTTSProvider(TTSProvider):
@property
def name(self) -> str:
return "my-tts" # what tts.provider matches against
@property
def display_name(self) -> str:
return "My Custom TTS"
def is_available(self) -> bool:
# Return False when credentials/deps are missing — picker skips
# this row but the dispatcher still routes here on explicit config.
import os
return bool(os.environ.get("MY_TTS_API_KEY"))
def synthesize(self, text, output_path, *, voice=None, model=None,
speed=None, format="mp3", **extra) -> str:
# Write audio bytes to output_path, return the path.
# Raise on failure — the dispatcher converts exceptions to a
# standard error envelope.
import my_tts_sdk
client = my_tts_sdk.Client()
audio_bytes = client.synthesize(text=text, voice=voice or "default")
with open(output_path, "wb") as f:
f.write(audio_bytes)
return output_path
def register(ctx):
ctx.register_tts_provider(MyTTSProvider())
```
Enable it (`hermes plugins enable my-tts`), point `tts.provider` at it (`tts.provider: my-tts` in `config.yaml`), and the `text_to_speech` tool will route through your plugin.
#### Optional hooks
Override these on your provider class for richer integration:
- `list_voices()` → list of `{id, display, language, gender, preview_url}` dicts shown in `hermes tools`.
- `list_models()` → list of `{id, display, languages, max_text_length}` dicts.
- `get_setup_schema()` → return `{name, badge, tag, env_vars: [{key, prompt, url}]}` to power the picker row in `hermes tools` / `hermes setup`. Without this, the plugin still works but its row in the picker is minimal.
- `stream(text, *, voice, model, format, **extra)` → iterator yielding audio bytes for streaming delivery (default raises `NotImplementedError`).
- `voice_compatible` property → set `True` if your output is Opus-compatible and the gateway should deliver it as a voice bubble (default `False` = regular audio attachment).
- `warm()` / `release()` → called when a surface toggles speech output on / when the last lease across surfaces is released, while your provider is the configured `tts.provider` — preload or unload a local model server here. Both default to no-ops; exceptions are logged at debug and never fail the toggle.
See `agent/tts_provider.py` for the full ABC including docstrings.
## Voice Message Transcription (STT)
Voice messages sent on Telegram, Discord, WhatsApp, Slack, or Signal are automatically transcribed and injected as text into the conversation. The agent sees the transcript as normal text.
| Provider | Quality | Cost | API Key |
|----------|---------|------|---------|
| **Local Whisper** (default) | Good | Free | None needed |
| **Groq Whisper API** | Good–Best | Free tier | `GROQ_API_KEY` |
| **OpenAI Whisper API** | Good–Best | Paid | `VOICE_TOOLS_OPENAI_KEY` or `OPENAI_API_KEY` |
:::info Zero Config
Local transcription works out of the box when `faster-whisper` is installed. If that's unavailable, Hermes can also use a local `whisper` CLI from common install locations (like `/opt/homebrew/bin`) or a custom command via `HERMES_LOCAL_STT_COMMAND`.
:::
### Configuration
```yaml
# In ~/.hermes/config.yaml
stt:
provider: "local" # "local" | "groq" | "openai" | "mistral" | "xai" | "elevenlabs" | "deepinfra"
language: "en" # Global language hint applied to every provider unless a per-provider language overrides it; set "" to restore auto-detect
local:
model: "base" # tiny, base, small, medium, large-v3
language: "" # optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else auto-detect
groq:
language: "" # optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else auto-detect
openai:
model: "whisper-1" # whisper-1, gpt-4o-mini-transcribe, gpt-4o-transcribe, gpt-transcribe
mistral:
model: "voxtral-mini-latest" # voxtral-mini-latest, voxtral-mini-2602
xai:
model: "grok-stt" # xAI Grok STT
language: "" # optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else "en"
```
### Provider Details
**Local (faster-whisper)** — Runs Whisper locally via [faster-whisper](https://github.com/SYSTRAN/faster-whisper). Uses CPU by default, GPU if available. Model sizes:
| Model | Size | Speed | Quality |
|-------|------|-------|---------|
| `tiny` | ~75 MB | Fastest | Basic |
| `base` | ~150 MB | Fast | Good (default) |
| `small` | ~500 MB | Medium | Better |
| `medium` | ~1.5 GB | Slower | Great |
| `large-v3` | ~3 GB | Slowest | Best |
**Groq API** — Requires `GROQ_API_KEY`. Good cloud fallback when you want a free hosted STT option. Set `stt.groq.language` (or the global `HERMES_LOCAL_STT_LANGUAGE` env var) to skip Whisper's auto-detect and reduce latency on known-language audio.
**OpenAI API** — Accepts `VOICE_TOOLS_OPENAI_KEY` first and falls back to `OPENAI_API_KEY`. Supports `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-transcribe`, and `gpt-transcribe`.
**Mistral API (Voxtral Transcribe)** — Requires `MISTRAL_API_KEY`. Uses Mistral's [Voxtral Transcribe](https://docs.mistral.ai/capabilities/audio/speech_to_text/) models. Supports 13 languages, speaker diarization, and word-level timestamps. Install with `cd ~/.hermes/hermes-agent && uv pip install -e ".[mistral]"`.
**xAI Grok STT** — Requires `XAI_API_KEY`. Posts to `https://api.x.ai/v1/stt` as multipart/form-data. Good choice if you're already using xAI for chat or TTS and want one API key for everything. Auto-detection order puts it after Groq — explicitly set `stt.provider: xai` to force it.
**Custom local CLI fallback** — Set `HERMES_LOCAL_STT_COMMAND` if you want Hermes to call a local transcription command directly. The command template supports `{input_path}`, `{output_dir}`, `{language}`, and `{model}` placeholders. Hermes tokenizes the rendered template into an argument list and executes it without a shell, so operators such as `|`, `>`, `&&`, and `;` are passed as literal arguments. Your command must write a `.txt` transcript somewhere under `{output_dir}`.
#### Example: Doubao / Volcengine ASR
If you use [`doubao-speech`](https://pypi.org/project/doubao-speech/) for Doubao TTS (see [above](#example-doubao-chinese-seed-tts-20)), the same package handles speech-to-text via the local-command STT surface:
```bash
pip install doubao-speech
export VOLCENGINE_APP_ID="your-app-id"
export VOLCENGINE_ACCESS_TOKEN="your-access-token"
export HERMES_LOCAL_STT_COMMAND='doubao-speech transcribe {input_path} --out {output_dir}/transcript.txt'
```
If a trusted local template intentionally needs pipes, redirects, or another shell feature, invoke the shell explicitly. Keep dynamic paths outside the shell program and pass them as positional arguments:
```bash
export HERMES_LOCAL_STT_COMMAND='sh -c '\''whisper "$1" --output_format txt --output_dir "$2" | tee "$2/whisper.log"'\'' _ {input_path} {output_dir}'
```
On Windows, use an explicit `cmd /c` or PowerShell wrapper instead. An explicit wrapper makes shell interpretation an opt-in part of the configured argv rather than an implicit property of every local STT template.
```yaml
stt:
provider: local_command
```
Hermes writes the incoming voice message to `{input_path}`, runs the command, and reads the `.txt` file produced under `{output_dir}`. Language is auto-detected by the Volcengine bigmodel endpoint.
### Fallback Behavior
An **explicit** `stt.provider` selection (written in `config.yaml`, e.g. via `hermes tools`) is honored strictly — if that provider can't run, transcription fails with a clear error (`stt is configured to use <provider> (set via hermes tools), but <failure>. Run 'hermes tools' to change it.`) instead of silently switching engines. Note that `stt.provider: local` written in your config counts as an explicit selection.
When **no provider has ever been selected**, Hermes auto-detects from what's available:
- **Local faster-whisper unavailable** → Tries a local `whisper` CLI or `HERMES_LOCAL_STT_COMMAND` before cloud providers
- **Groq key not set** → Skipped; next available provider
- **OpenAI key not set** → Skipped; next available provider
- **Mistral key/SDK not set** → Skipped in auto-detect; falls through to next available provider
- **Nothing available** → Voice messages pass through with an accurate note to the user
### STT custom command providers
If the STT engine you want isn't natively supported (Doubao ASR, NVIDIA Parakeet, a whisper.cpp build, an open-source SenseVoice CLI, anything else that exposes a shell command), wire it in as a **command-type provider** without writing any Python. Hermes runs your shell command against the audio file and reads back the transcript.
Declare one or more providers under `stt.providers.<name>` and switch between them with `stt.provider: <name>` — same shape as the TTS [command-provider registry](#custom-command-providers), adapted for the input=audio → output=transcript direction.
```yaml
stt:
provider: parakeet # pick any name under stt.providers
providers:
parakeet:
type: command
command: "parakeet-asr --model nvidia/parakeet-tdt-0.6b-v2 --in {input_path} --out {output_path}"
format: txt
language: en
timeout: 300
whispercpp:
type: command
command: "whisper-cli -m ~/models/ggml-large-v3.bin -f {input_path} -otxt -of {output_dir}/transcript"
format: txt
sensevoice:
type: command
command: "sensevoice-cli {input_path} --json | tee {output_path}"
format: json
```
This complements the legacy `HERMES_LOCAL_STT_COMMAND` escape hatch via the built-in `local_command` path. Unlike the shell-driven command-provider registry, the legacy template is tokenized into argv and runs without implicit shell interpretation. Use `stt.providers.<name>` when you want **multiple** shell-driven STT engines, a name you can pick via `stt.provider`, or anything that needs per-provider `language` / `model` / `timeout`.
#### STT placeholders
Your command template can reference these placeholders. Hermes substitutes them at render time and shell-quotes each value for the surrounding context (bare / single-quoted / double-quoted), so paths with spaces are safe.
| Placeholder | Meaning |
|-------------------|----------------------------------------------------------------------|
| `{input_path}` | Absolute path to the input audio file (original location, read-only) |
| `{output_path}` | Absolute path the command should write the transcript to |
| `{output_dir}` | Parent directory of `{output_path}` (handy for whisper-style tools) |
| `{format}` | Configured output format: `txt` / `json` / `srt` / `vtt` |
| `{language}` | Configured language code (defaults to `en`) |
| `{model}` | `stt.providers.<name>.model`, empty when unset |
Use `{{` and `}}` for literal braces (handy when embedding JSON snippets in the command).
#### How the transcript is read back
After your command exits successfully:
1. If `{output_path}` exists and is non-empty → Hermes reads it as UTF-8 text.
2. Otherwise, if the command wrote to stdout → Hermes uses that.
3. Otherwise → error: "Command STT provider wrote no output file and produced no stdout".
This lets you use the registry for both file-writing CLIs (`whisper-cli`, `parakeet-asr`) and curl-style one-liners that emit transcript to stdout (`curl … | jq -r .text`).
For `format: json` / `srt` / `vtt`, Hermes returns the raw file content as the `transcript` field. Extracting `.text` from JSON is out of scope for the runner — either configure `format: txt`, or post-process JSON downstream.
#### STT command-provider optional keys
| Key | Default | Meaning |
|-----------------|---------|------------------------------------------------------------------------------------------------------|
| `timeout` | `300` | Seconds; the process tree is killed on expiry (Unix `start_new_session`, Windows `taskkill /T`). |
| `format` | `txt` | One of `txt` / `json` / `srt` / `vtt`. Sets the extension of `{output_path}`. |
| `language` | `en` | Forwarded to `{language}`. Defaults to `stt.language` then `en`. |
| `model` | empty | Forwarded to `{model}`. The `model=` argument to `transcribe_audio()` overrides this. |
#### STT command-provider behavior notes
- **Built-ins always win.** Declaring `stt.providers.openai: type: command` does NOT override the real OpenAI Whisper handler. The built-in name is short-circuited before the command-provider resolver runs.
- **Process-tree cleanup.** A command running over `timeout` has its entire process tree killed, not just the shell wrapper. Long-running ASR pipelines that fork model-loading subprocesses are reaped reliably.
- **Shell-quoting is automatic.** Placeholders inside `'…'` get single-quote-safe escaping; inside `"…"` get `$`/`` ` ``/`"` escaping; outside quotes get `shlex.quote`. Don't pre-quote placeholder values.
#### STT command-provider security
The shell command runs under the same user as Hermes with full filesystem access — same trust model as `tts.providers.<name>: type: command` and `HERMES_LOCAL_STT_COMMAND`. Only declare command providers from sources you trust.
### Python plugin providers (STT)
For STT engines that aren't built-in AND can't be expressed as a shell command (need a Python SDK, OAuth-refreshing auth, streaming chunks, etc.), register a Python plugin via `ctx.register_transcription_provider()`. The plugin **coexists with** the 8 built-in providers (`local`, `local_command`, `groq`, `openai`, `mistral`, `xai`, `elevenlabs`, `deepinfra`) and the `stt.providers.<name>: type: command` registry — built-ins keep their native implementations and always win on name collision; command providers win over plugins of the same name (config is more local than plugin install).
#### When to pick which (STT)
| Backend has… | Use |
|--------------------------------------------------------------|------------------------------------------------------------------|
| A single shell command that takes an audio file and emits text | `stt.providers.<name>: type: command` (no Python needed) |
| Only the legacy single-command escape hatch is wanted | `HERMES_LOCAL_STT_COMMAND` env var (argv-tokenized; no implicit shell) |
| A Python SDK with no CLI | `register_transcription_provider()` plugin |
| OAuth-refreshing auth, streaming chunks, voice-list metadata | `register_transcription_provider()` plugin |
| A built-in already covers it (`local`, `groq`, `openai`, …) | Set `stt.provider: <name>` — built-ins are inline |
#### Resolution order
1. **`stt.provider` is a built-in name** → built-in dispatch. **Always wins.**
2. **`stt.provider` matches `stt.providers.<name>` with `command:` set** → command-provider runner (see [STT custom command providers](#stt-custom-command-providers)). Wins over a same-name plugin.
3. **`stt.provider` matches a plugin-registered `TranscriptionProvider`** → plugin dispatch:
- if the plugin's `is_available()` returns `False` (missing creds or SDK), the call surfaces an unavailability error envelope identifying the plugin — **not** the generic "No STT provider available" message.
- otherwise the plugin's `transcribe()` is called with `model` (from the public `model=` arg, falling back to `stt.<provider>.model`) and `language` (from `stt.<provider>.language`).
4. **No match** → "No STT provider available" error.
#### Per-provider config namespace
Plugins read their per-provider configuration from `stt.<provider>` in `config.yaml`, mirroring how built-ins read `stt.openai.model` / `stt.mistral.model`:
```yaml
stt:
provider: my-stt
my-stt:
model: whisper-large-v3
language: ja # forwarded as language= to transcribe()
# any other plugin-specific keys go here; read them via your
# own config.yaml access in __init__/is_available/transcribe
```
The dispatcher forwards `model` and `language` from this section; everything else, the plugin can read itself.
#### Minimal plugin
Drop this in `~/.hermes/plugins/my-stt/`:
`plugin.yaml`:
```yaml
name: my-stt
version: 0.1.0
description: "My custom Python STT backend"
```
`__init__.py`:
```python
from agent.transcription_provider import TranscriptionProvider
class MySTTProvider(TranscriptionProvider):
@property
def name(self) -> str:
return "my-stt" # what stt.provider matches against
@property
def display_name(self) -> str:
return "My Custom STT"
def is_available(self) -> bool:
# Return False when credentials/deps are missing — picker skips
# this row but the dispatcher still routes here on explicit config.
import os
return bool(os.environ.get("MY_STT_API_KEY"))
def transcribe(self, file_path, *, model=None, language=None, **extra):
# Return the standard transcribe envelope:
# {"success": bool, "transcript": str, "provider": str, "error": str}
# Do NOT raise — convert exceptions to the error envelope so the
# gateway/CLI caller sees a consistent shape on failure.
try:
import my_stt_sdk
client = my_stt_sdk.Client()
text = client.transcribe(open(file_path, "rb"))
return {
"success": True,
"transcript": text,
"provider": "my-stt",
}
except Exception as exc:
return {
"success": False,
"transcript": "",
"error": f"my-stt failed: {exc}",
"provider": "my-stt",
}
def register(ctx):
ctx.register_transcription_provider(MySTTProvider())
```
Enable it (`hermes plugins enable my-stt`), set `stt.provider: my-stt` in `config.yaml`, and voice-message transcription will route through your plugin.
#### Optional hooks
Override these on your provider class for richer integration:
- `list_models()` → list of `{id, display, languages, max_audio_seconds}` dicts.
- `default_model()` → string returned when the user doesn't override the model.
- `get_setup_schema()` → return `{name, badge, tag, env_vars: [{key, prompt, url}]}` to power picker rows in `hermes tools` / `hermes setup` (the picker category for STT is not yet shipped — this metadata is available to plugins for forward compatibility).
See `agent/transcription_provider.py` for the full ABC including docstrings.
website/docs/user-guide/features/vision.md
+214
View File
@@ -0,0 +1,214 @@
---
title: Vision & Image Paste
description: Paste images from your clipboard into the Hermes CLI for multimodal vision analysis.
sidebar_label: Vision & Image Paste
sidebar_position: 7
---
# Vision & Image Paste
Hermes Agent supports **multimodal vision** — you can paste images from your clipboard directly into the CLI and ask the agent to analyze, describe, or work with them. Images are sent to the model as base64-encoded content blocks, so any vision-capable model can process them.
:::tip
Portal subscribers get vision-capable models (Claude, GPT-5, Gemini) in the same catalog — no extra credentials needed. See [Nous Portal](/integrations/nous-portal).
:::
## How It Works
1. Copy an image to your clipboard (screenshot, browser image, etc.)
2. Attach it using one of the methods below
3. Type your question and press Enter
4. The image appears as a `[📎 Image #1]` badge above the input
5. On submit, the image is sent to the model as a vision content block
You can attach multiple images before sending — each gets its own badge. Press `Ctrl+C` to clear all attached images.
Images are saved to `~/.hermes/images/` as PNG files with timestamped filenames.
## Paste Methods
How you attach an image depends on your terminal environment. Not all methods work everywhere — here's the full breakdown:
### `/paste` Command
**The most reliable explicit image-attach fallback.**
```
/paste
```
Type `/paste` and press Enter. Hermes checks your clipboard for an image and attaches it. This is the safest option when your terminal rewrites `Cmd+V`/`Ctrl+V`, or when you copied only an image and there is no bracketed-paste text payload to inspect.
### Ctrl+V / Cmd+V
Hermes now treats paste as a layered flow:
- normal text paste first
- native clipboard / OSC52 text fallback if the terminal did not deliver text cleanly
- image attach when the clipboard or pasted payload resolves to an image or image path
This means pasted macOS screenshot temp paths and `file://...` image URIs can attach immediately instead of sitting in the composer as raw text.
:::warning
If your clipboard has **only an image** (no text), terminals still cannot send binary image bytes directly. Use `/paste` as the explicit image-attach fallback.
:::
### `/terminal-setup` for VS Code / Cursor / Windsurf
If you run the TUI inside a local VS Code-family integrated terminal on macOS, Hermes can install the recommended `workbench.action.terminal.sendSequence` bindings for better multiline and undo/redo parity:
```text
/terminal-setup
```
This is especially useful when `Cmd+Enter`, `Cmd+Z`, or `Shift+Cmd+Z` are being intercepted by the IDE. Run it on the local machine only — not inside an SSH session.
## Platform Compatibility
| Environment | `/paste` | Cmd/Ctrl+V | `/terminal-setup` | Notes |
|---|:---:|:---:|:---:|---|
| **macOS Terminal / iTerm2** | ✅ | ✅ | n/a | Best experience — native clipboard + screenshot-path recovery |
| **Apple Terminal** | ✅ | ✅ | n/a | If Cmd+←/→/⌫ gets rewritten, use Ctrl+A / Ctrl+E / Ctrl+U fallbacks |
| **Linux X11 desktop** | ✅ | ✅ | n/a | Requires `xclip` (`apt install xclip`) |
| **Linux Wayland desktop** | ✅ | ✅ | n/a | Requires `wl-paste` (`apt install wl-clipboard`) |
| **WSL2 (Windows Terminal)** | ✅ | ✅ | n/a | Uses `powershell.exe` — no extra install needed |
| **VS Code / Cursor / Windsurf (local)** | ✅ | ✅ | ✅ | Recommended for better Cmd+Enter / undo / redo parity |
| **VS Code / Cursor / Windsurf (SSH)** | ❌² | ❌² | ❌³ | Run `/terminal-setup` on the local machine instead |
| **SSH terminal (any)** | ❌² | ❌² | n/a | Remote clipboard not accessible |
² See [SSH & Remote Sessions](#ssh--remote-sessions) below
³ The command writes local IDE keybindings and should not be run from the remote host
## Platform-Specific Setup
### macOS
**No setup required.** Hermes uses `osascript` (built into macOS) to read the clipboard. For faster performance, optionally install `pngpaste`:
```bash
brew install pngpaste
```
### Linux (X11)
Install `xclip`:
```bash
# Ubuntu/Debian
sudo apt install xclip
# Fedora
sudo dnf install xclip
# Arch
sudo pacman -S xclip
```
### Linux (Wayland)
Modern Linux desktops (Ubuntu 22.04+, Fedora 34+) often use Wayland by default. Install `wl-clipboard`:
```bash
# Ubuntu/Debian
sudo apt install wl-clipboard
# Fedora
sudo dnf install wl-clipboard
# Arch
sudo pacman -S wl-clipboard
```
:::tip How to check if you're on Wayland
```bash
echo $XDG_SESSION_TYPE
# "wayland" = Wayland, "x11" = X11, "tty" = no display server
```
:::
### WSL2
**No extra setup required.** Hermes detects WSL2 automatically (via `/proc/version`) and uses `powershell.exe` to access the Windows clipboard through .NET's `System.Windows.Forms.Clipboard`. This is built into WSL2's Windows interop — `powershell.exe` is available by default.
The clipboard data is transferred as base64-encoded PNG over stdout, so no file path conversion or temp files are needed.
:::info WSLg Note
If you're running WSLg (WSL2 with GUI support), Hermes tries the PowerShell path first, then falls back to `wl-paste`. WSLg's clipboard bridge only supports BMP format for images — Hermes auto-converts BMP to PNG using Pillow (if installed) or ImageMagick's `convert` command.
:::
#### Verify WSL2 clipboard access
```bash
# 1. Check WSL detection
grep -i microsoft /proc/version
# 2. Check PowerShell is accessible
which powershell.exe
# 3. Copy an image, then check
powershell.exe -NoProfile -Command "Add-Type -AssemblyName System.Windows.Forms; [System.Windows.Forms.Clipboard]::ContainsImage()"
# Should print "True"
```
## SSH & Remote Sessions
**Clipboard image paste does not fully work over SSH.** When you SSH into a remote machine, the Hermes CLI runs on the remote host. Clipboard tools (`xclip`, `wl-paste`, `powershell.exe`, `osascript`) read the clipboard of the machine they run on — which is the remote server, not your local machine. Your local clipboard image is therefore inaccessible from the remote side.
Text can sometimes still bridge through terminal paste or OSC52, but image clipboard access and local screenshot temp paths remain tied to the machine running Hermes.
### Workarounds for SSH
1. **Upload the image file** — Save the image locally, upload it to the remote server via `scp`, VSCode's file explorer (drag-and-drop), or any file transfer method. Then reference it by path. *(A `/attach <filepath>` command is planned for a future release.)*
2. **Use a URL** — If the image is accessible online, just paste the URL in your message. The agent can use `vision_analyze` to look at any image URL directly.
3. **X11 forwarding** — Connect with `ssh -X` to forward X11. This lets `xclip` on the remote machine access your local X11 clipboard. Requires an X server running locally (XQuartz on macOS, built-in on Linux X11 desktops). Slow for large images.
4. **Use a messaging platform** — Send images to Hermes via Telegram, Discord, Slack, or WhatsApp. These platforms handle image upload natively and are not affected by clipboard/terminal limitations.
## Why Terminals Can't Paste Images
This is a common source of confusion, so here's the technical explanation:
Terminals are **text-based** interfaces. When you press Ctrl+V (or Cmd+V), the terminal emulator:
1. Reads the clipboard for **text content**
2. Wraps it in [bracketed paste](https://en.wikipedia.org/wiki/Bracketed-paste) escape sequences
3. Sends it to the application through the terminal's text stream
If the clipboard contains only an image (no text), the terminal has nothing to send. There is no standard terminal escape sequence for binary image data. The terminal simply does nothing.
This is why Hermes uses a separate clipboard check — instead of receiving image data through the terminal paste event, it calls OS-level tools (`osascript`, `powershell.exe`, `xclip`, `wl-paste`) directly via subprocess to read the clipboard independently.
## Supported Models
Image paste works with any vision-capable model. The image is sent as a base64-encoded data URL in the OpenAI vision content format:
```json
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,..."
}
}
```
Most modern models support this format, including GPT-4 Vision, Claude (with vision), Gemini, and open-source multimodal models served through OpenRouter.
## Image Routing (Vision-Capable vs Text-Only Models)
When a user attaches an image — from the CLI clipboard, the gateway (Telegram/Discord photo), or any other entry point — Hermes routes it based on whether your current model actually supports vision:
| Your model | What happens to the image |
|---|---|
| **Vision-capable** (GPT-4V, Claude with vision, Gemini, Qwen-VL, MiMo-VL, etc.) | Sent as **real pixels** using the provider's native image content format above. No text summary layer. |
| **Text-only** (DeepSeek V3, smaller open-source models, older chat-only endpoints) | Routed through the `vision_analyze` auxiliary tool — an auxiliary vision model describes the image, and the text description is injected into the conversation. |
You don't configure this — Hermes looks up your current model's capability in the provider metadata and picks the right path automatically. The practical effect: you can switch between vision and non-vision models mid-session and image handling "just works" without changing your workflow. Text-only models get coherent context about the image rather than a broken multimodal payload they'd have to reject.
Which auxiliary model handles the text-description path is configurable under `auxiliary.vision` — see [Auxiliary Models](/user-guide/configuration#auxiliary-models).
### `vision_analyze` has the same dual behavior
The `vision_analyze` tool itself follows the same routing. When the active main model is vision-capable **and** its provider supports image content inside tool results (currently the Anthropic, OpenAI, Azure-OpenAI, and Gemini 3.x stacks), `vision_analyze` short-circuits the auxiliary describer and returns the raw image pixels as a multimodal tool-result envelope. The main model sees the image natively on its next turn — no aux call, no text-summary information loss, no extra latency.
For text-only main models (or providers whose tool-result channel doesn't carry images), `vision_analyze` falls back to the legacy path: it asks the configured auxiliary vision model to describe the image and returns the description as plain text. Either way the calling tool signature is the same — the tool decides which path to take at runtime based on the active model.
website/docs/user-guide/features/voice-mode.md
+581
View File
@@ -0,0 +1,581 @@
---
sidebar_position: 10
title: "Voice Mode"
description: "Real-time voice conversations with Hermes Agent — CLI, Telegram, Discord (DMs, text channels, and voice channels)"
---
# Voice Mode
Hermes Agent supports full voice interaction across CLI and messaging platforms. Talk to the agent using your microphone, hear spoken replies, and have live voice conversations in Discord voice channels.
If you want a practical setup walkthrough with recommended configurations and real usage patterns, see [Use Voice Mode with Hermes](../../guides/use-voice-mode-with-hermes.md).
For hands-free session start — saying "hey hermes" (or any phrase) to open a fresh voice session on the CLI, TUI, or desktop app — see [Wake Word](/user-guide/features/wake-word).
## Prerequisites
Before using voice features, make sure you have:
1. **Hermes Agent installed** — via the install script (see [Installation](/getting-started/installation))
2. **An LLM provider configured** — run `hermes model` or set your preferred provider credentials in `~/.hermes/.env`
3. **A working base setup** — run `hermes` to verify the agent responds to text before enabling voice
:::tip
The `~/.hermes/` directory and default `config.yaml` are created automatically the first time you run `hermes`. You only need to create `~/.hermes/.env` manually for API keys.
:::
:::tip Nous Portal covers both
A paid [Nous Portal](/user-guide/features/tool-gateway) subscription supplies the LLM (step 2) **and** OpenAI TTS via the Tool Gateway — no separate OpenAI key needed. On a fresh install, `hermes setup --portal` wires both up at once.
:::
## Overview
| Feature | Platform | Description |
|---------|----------|-------------|
| **Interactive Voice** | CLI | Press Ctrl+B to record, agent auto-detects silence and responds |
| **Auto Voice Reply** | Telegram, Discord | Agent sends spoken audio alongside text responses |
| **Voice Channel** | Discord | Bot joins VC, listens to users speaking, speaks replies back |
## Requirements
### Python Packages
```bash
# CLI voice mode (microphone + audio playback)
cd ~/.hermes/hermes-agent && uv pip install -e ".[voice]"
# Discord + Telegram messaging (includes discord.py[voice] for VC support)
cd ~/.hermes/hermes-agent && uv pip install -e ".[messaging]"
# Premium TTS (ElevenLabs)
cd ~/.hermes/hermes-agent && uv pip install -e ".[tts-premium]"
# Local TTS (NeuTTS, optional)
python -m pip install -U neutts[all]
# Everything at once
cd ~/.hermes/hermes-agent && uv pip install -e ".[all]"
```
| Extra | Packages | Required For |
|-------|----------|-------------|
| `voice` | `sounddevice`, `numpy` | CLI voice mode |
| `messaging` | `discord.py[voice]`, `python-telegram-bot`, `aiohttp` | Discord & Telegram bots |
| `tts-premium` | `elevenlabs` | ElevenLabs TTS provider |
Optional local TTS provider: install `neutts` separately with `python -m pip install -U neutts[all]`. On first use it downloads the model automatically.
:::info
`discord.py[voice]` installs **PyNaCl** (for voice encryption) and **opus bindings** automatically. This is required for Discord voice channel support.
:::
### System Dependencies
```bash
# macOS
brew install portaudio ffmpeg opus
brew install espeak-ng # for NeuTTS
# Ubuntu/Debian
sudo apt install portaudio19-dev ffmpeg libopus0
sudo apt install espeak-ng # for NeuTTS
```
| Dependency | Purpose | Required For |
|-----------|---------|-------------|
| **PortAudio** | Microphone input and audio playback | CLI voice mode |
| **ffmpeg** | Audio format conversion (MP3 → Opus, PCM → WAV) | All platforms |
| **Opus** | Discord voice codec | Discord voice channels |
| **espeak-ng** | Phonemizer backend | Local NeuTTS provider |
### API Keys
Add to `~/.hermes/.env`:
```bash
# Speech-to-Text — local provider needs NO key at all
# pip install faster-whisper # Free, runs locally, recommended
GROQ_API_KEY=your-key # Groq Whisper — fast, free tier (cloud)
VOICE_TOOLS_OPENAI_KEY=your-key # OpenAI Whisper — paid (cloud)
# Text-to-Speech (optional — Edge TTS and NeuTTS work without any key)
ELEVENLABS_API_KEY=*** # ElevenLabs — premium quality
# VOICE_TOOLS_OPENAI_KEY above also enables OpenAI TTS
```
:::tip
If `faster-whisper` is installed, voice mode works with **zero API keys** for STT. The model (~150 MB for `base`) downloads automatically on first use.
:::
---
## CLI Voice Mode
Voice mode is available in both the **classic CLI** (`hermes chat`) and the **TUI** (`hermes --tui`). Behavior is identical across both — same slash commands, same VAD silence detection, same streaming TTS, same hallucination filter. The TUI additionally forwards crash-forensic logs to `~/.hermes/logs/` so push-to-talk failures on exotic audio backends can be reported with a full stack trace rather than disappearing silently.
### Quick Start
Start the CLI and enable voice mode:
```bash
hermes # Start the interactive CLI
```
Then use these commands inside the CLI:
```
/voice Toggle voice mode on/off
/voice on Enable voice mode
/voice off Disable voice mode
/voice tts Toggle TTS output
/voice status Show current state
```
### How It Works
1. Start the CLI with `hermes` and enable voice mode with `/voice on`
2. **Press Ctrl+B** — a beep plays (880Hz), recording starts
3. **Speak** — a live audio level bar shows your input: `● [▁▂▃▅▇▇▅▂] ❯`
4. **Stop speaking** — after 3 seconds of silence, recording auto-stops
5. **Two beeps** play (660Hz) confirming the recording ended
6. Audio is transcribed via Whisper and sent to the agent
7. If TTS is enabled, the agent's reply is spoken aloud
8. Recording **automatically restarts** — speak again without pressing any key
This loop continues until you press **Ctrl+B** during recording (exits continuous mode) or 3 consecutive recordings detect no speech.
:::tip
The record key is configurable via `voice.record_key` in `~/.hermes/config.yaml` (default: `ctrl+b`).
:::
### Silence Detection
Two-stage algorithm detects when you've finished speaking:
1. **Speech confirmation** — waits for audio above the RMS threshold (200) for at least 0.3s, tolerating brief dips between syllables
2. **End detection** — once speech is confirmed, triggers after 3.0 seconds of continuous silence
If no speech is detected at all for 15 seconds, recording stops automatically.
Both `silence_threshold` and `silence_duration` are configurable in `config.yaml`. You can also disable the record start/stop beeps with `voice.beep_enabled: false`.
### Ending a voice chat by voice
Say **"stop"** — and nothing else — to end the voice conversation hands-free. The match is deliberately strict: the whole utterance (case-insensitive, surrounding punctuation ignored) must equal a configured phrase, so "stop doing that and try X instead" still reaches the agent normally. Customize the phrase list with `voice.stop_phrases` in `config.yaml` (e.g. `["stop", "goodbye hermes"]`), or set it to `[]` to disable. A voice chat also ends on its own after three consecutive silent cycles (no speech detected).
**Typing** a bare stop phrase while a voice chat is active works the same way on every surface (CLI, TUI, desktop): the message ends the voice chat instead of being sent to the agent. Outside a voice chat, typed "stop" is an ordinary message.
### Streaming TTS
When TTS is enabled, the agent speaks its reply **sentence-by-sentence** as it generates text — you don't wait for the full response. This works with **every TTS provider**:
1. Buffers text deltas into complete sentences (min 20 chars)
2. Strips markdown formatting, emoji, and `<think>` blocks
3. Plays audio per sentence in real-time — providers with a chunked PCM API (ElevenLabs, OpenAI) stream raw audio for the lowest time-to-first-word; every other provider (including the default Edge) synthesizes and plays each sentence as it completes
The same pipeline runs in the classic CLI, the TUI, and the desktop app. In a desktop voice conversation the reply text is fed **live** into a per-reply speech WebSocket as the model generates it, so speech overlaps generation — one socket and one audio clock per reply, no per-sentence connection gaps.
### Desktop remote: client-direct voice (lowest-hop path)
When Hermes Desktop is connected to a **remote gateway**, audio does not need to be relayed through the gateway at all. At voice-session start the desktop fetches the active profile's resolved STT/TTS settings (provider, model, language/voice, and credential) from the gateway over the authenticated REST channel (`GET /api/audio/voice-config`) and then calls the providers **directly**:
- **Dictation / voice input:** the mic recording goes straight from your desktop to the profile's STT provider; only the resulting *text* is sent to the gateway as the prompt.
- **Spoken replies:** the reply text is already streaming to the desktop over the chat socket, so the desktop synthesizes it locally with the profile's TTS provider and plays it — the gateway link never carries audio.
There is nothing to configure on the client: the profile you're talking to is the single source of truth for providers and keys, exactly as if the gateway had done the work itself. Keys are held in the desktop's memory for the session only — never written to disk on the client.
Providers that can only run on the gateway host (local whisper, `edge` TTS, command providers, plugins) automatically fall back to the relay path (`/api/audio/transcribe` and the speech WebSocket), as does any older backend without the endpoint. To force the relay for every provider, set:
```yaml
voice:
client_direct: false
```
Client-direct wire support: OpenAI (incl. Nous-managed audio), Groq, Mistral, and DeepInfra via the OpenAI-compatible shapes, xAI Grok STT, and ElevenLabs STT + TTS. xAI configured through OAuth stays on the relay (the OAuth bearer refreshes server-side).
### Barge-in
You can interrupt the agent at ANY point in its turn — the microphone stays live from the moment you finish speaking until the reply has fully played (full duplex):
- **Interject while it's thinking** — in continuous voice mode, speaking during LLM generation (before any audio plays) interrupts the in-flight turn and your interjection becomes the next message, the same as typing over a running turn.
- **Talk over it** — speaking while the agent's reply plays cuts playback the moment you start talking and submits what you said. The detector calibrates its noise floor against the *quiet room* at turn start (never against the playback itself), so speaker bleed can't deafen it and normal speech reliably trips it.
- **Type or press the record key** — sending a new message or hitting the push-to-talk key stops playback instantly on every surface.
- **Say "stop"** — the stop phrase works in both phases: mid-generation it interrupts the turn AND ends the voice chat; mid-playback it cuts the speech and ends the chat.
Tuning (config.yaml): `voice.barge_in: false` disables it; `voice.barge_in_threshold_multiplier` (default `3.0`) scales the speech trigger over the quiet-room floor; `voice.barge_in_grace_seconds` (default `0.5`) suppresses trips right after playback starts. Set `HERMES_VOICE_DEBUG=1` to stream per-block VAD diagnostics (calibrated floor, RMS, trip decisions) to stderr for live tuning.
The agent **knows** it was interrupted: the next message carries a short note telling the model its spoken reply was cut off, so it can react naturally ("rude!") or pick up where it left off instead of being oblivious.
### Hallucination Filter
Whisper sometimes generates phantom text from silence or background noise ("Thank you for watching", "Subscribe", etc.). The agent filters these out using a set of 26 known hallucination phrases across multiple languages, plus a regex pattern that catches repetitive variations.
---
## Gateway Voice Reply (Telegram & Discord)
If you haven't set up your messaging bots yet, see the platform-specific guides:
- [Telegram Setup Guide](../messaging/telegram.md)
- [Discord Setup Guide](../messaging/discord.md)
Start the gateway to connect to your messaging platforms:
```bash
hermes gateway # Start the gateway (connects to configured platforms)
hermes gateway setup # Interactive setup wizard for first-time configuration
```
### Discord: Channels vs DMs
The bot supports two interaction modes on Discord:
| Mode | How to Talk | Mention Required | Setup |
|------|------------|-----------------|-------|
| **Direct Message (DM)** | Open the bot's profile → "Message" | No | Works immediately |
| **Server Channel** | Type in a text channel where the bot is present | Yes (`@botname`) | Bot must be invited to the server |
**DM (recommended for personal use):** Just open a DM with the bot and type — no @mention needed. Voice replies and all commands work the same as in channels.
**Server channels:** The bot only responds when you @mention it (e.g. `@hermesbyt4 hello`). Make sure you select the **bot user** from the mention popup, not the role with the same name.
:::tip
To disable the mention requirement in server channels, add to `~/.hermes/.env`:
```bash
DISCORD_REQUIRE_MENTION=false
```
Or set specific channels as free-response (no mention needed):
```bash
DISCORD_FREE_RESPONSE_CHANNELS=123456789,987654321
```
:::
### Commands
These work in both Telegram and Discord (DMs and text channels):
```
/voice Toggle voice mode on/off
/voice on Voice replies only when you send a voice message
/voice tts Voice replies for ALL messages
/voice off Disable voice replies
/voice status Show current setting
```
### Modes
| Mode | Command | Behavior |
|------|---------|----------|
| `off` | `/voice off` | Text only (default) |
| `voice_only` | `/voice on` | Speaks reply only when you send a voice message |
| `all` | `/voice tts` | Speaks reply to every message |
Voice mode setting is persisted across gateway restarts.
### Platform Delivery
| Platform | Format | Notes |
|----------|--------|-------|
| **Telegram** | Voice bubble (Opus/OGG) | Plays inline in chat. ffmpeg converts MP3 → Opus if needed |
| **Discord** | Native voice bubble (Opus/OGG) | Plays inline like a user voice message. Falls back to file attachment if voice bubble API fails |
---
## Discord Voice Channels
The most immersive voice feature: the bot joins a Discord voice channel, listens to users speaking, transcribes their speech, processes through the agent, and speaks the reply back in the voice channel.
### Setup
#### 1. Discord Bot Permissions
If you already have a Discord bot set up for text (see [Discord Setup Guide](../messaging/discord.md)), you need to add voice permissions.
Go to the [Discord Developer Portal](https://discord.com/developers/applications) → your application → **Installation** → **Default Install Settings** → **Guild Install**:
**Add these permissions to the existing text permissions:**
| Permission | Purpose | Required |
|-----------|---------|----------|
| **Connect** | Join voice channels | Yes |
| **Speak** | Play TTS audio in voice channels | Yes |
| **Use Voice Activity** | Detect when users are speaking | Recommended |
**Updated Permissions Integer:**
| Level | Integer | What's Included |
|-------|---------|----------------|
| Text only | `309237763136` | View Channels, Send Messages, Read History, Embeds, Attachments, Threads, Reactions, Create Public Threads |
| Text + Voice | `309240908864` | All above + Connect, Speak |
**Re-invite the bot** with the updated permissions URL:
```
https://discord.com/oauth2/authorize?client_id=YOUR_APP_ID&scope=bot+applications.commands&permissions=309240908864
```
Replace `YOUR_APP_ID` with your Application ID from the Developer Portal.
:::warning
Re-inviting the bot to a server it's already in will update its permissions without removing it. You won't lose any data or configuration.
:::
#### 2. Privileged Gateway Intents
In the [Developer Portal](https://discord.com/developers/applications) → your application → **Bot** → **Privileged Gateway Intents**, enable all three:
| Intent | Purpose |
|--------|---------|
| **Presence Intent** | Detect user online/offline status |
| **Server Members Intent** | Resolve usernames in `DISCORD_ALLOWED_USERS` to numeric IDs (conditional) |
| **Message Content Intent** | Read text message content in channels |
**Message Content Intent** is required. **Server Members Intent** is only needed if your `DISCORD_ALLOWED_USERS` list uses usernames — if you use numeric user IDs, you can leave it OFF. Voice-channel SSRC → user_id mapping comes from Discord's SPEAKING opcode on the voice websocket and does **not** require the Server Members Intent.
#### 3. Opus Codec
The Opus codec library must be installed on the machine running the gateway:
```bash
# macOS (Homebrew)
brew install opus
# Ubuntu/Debian
sudo apt install libopus0
```
The bot auto-loads the codec from:
- **macOS:** `/opt/homebrew/lib/libopus.dylib`
- **Linux:** `libopus.so.0`
#### 4. Environment Variables
```bash
# ~/.hermes/.env
# Discord bot (already configured for text)
DISCORD_BOT_TOKEN=your-bot-token
DISCORD_ALLOWED_USERS=your-user-id
# STT — local provider needs no key (pip install faster-whisper)
# GROQ_API_KEY=your-key # Alternative: cloud-based, fast, free tier
# TTS — optional. Edge TTS and NeuTTS need no key.
# ELEVENLABS_API_KEY=*** # Premium quality
# VOICE_TOOLS_OPENAI_KEY=*** # OpenAI TTS / Whisper
```
### Start the Gateway
```bash
hermes gateway # Start with existing configuration
```
The bot should come online in Discord within a few seconds.
### Commands
Use these in the Discord text channel where the bot is present:
```
/voice join Bot joins your current voice channel
/voice channel Alias for /voice join
/voice leave Bot disconnects from voice channel
/voice status Show voice mode and connected channel
```
:::info
You must be in a voice channel before running `/voice join`. The bot joins the same VC you're in.
:::
### How It Works
When the bot joins a voice channel, it:
1. **Listens** to each user's audio stream independently
2. **Detects silence** — 1.5s of silence after at least 0.5s of speech triggers processing
3. **Transcribes** the audio via Whisper STT (local, Groq, or OpenAI)
4. **Processes** through the full agent pipeline (session, tools, memory)
5. **Speaks** the reply back in the voice channel via TTS
### Text Channel Integration
When the bot is in a voice channel:
- Transcripts appear in the text channel: `[Voice] @user: what you said`
- Agent responses are sent as text in the channel AND spoken in the VC
- The text channel is the one where `/voice join` was issued
### Echo Prevention
The bot automatically pauses its audio listener while playing TTS replies, preventing it from hearing and re-processing its own output.
### Access Control
Only users listed in `DISCORD_ALLOWED_USERS` can interact via voice. Other users' audio is silently ignored.
```bash
# ~/.hermes/.env
DISCORD_ALLOWED_USERS=284102345871466496
```
---
## Configuration Reference
### config.yaml
```yaml
# Voice recording (CLI)
voice:
record_key: "ctrl+b" # Key to start/stop recording
max_recording_seconds: 120 # Maximum recording length
auto_tts: false # Auto-enable TTS when voice mode starts
beep_enabled: true # Play record start/stop beeps
silence_threshold: 200 # RMS level (0-32767) below which counts as silence
silence_duration: 3.0 # Seconds of silence before auto-stop
stop_phrases: ["stop"] # Saying exactly one of these ends the voice chat; [] disables
# Speech-to-Text
stt:
enabled: true # set to false to skip auto-transcription —
# the gateway still caches the audio file and
# passes its path to the agent as part of the
# inbound message, useful for custom pipelines
# (diarization, alignment, archival, etc.)
provider: "local" # "local" (free) | "groq" | "openai" | "mistral" | "xai"
local:
model: "base" # tiny, base, small, medium, large-v3
language: "" # optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else auto-detect
groq:
language: "" # optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else auto-detect
# model: "whisper-1" # Legacy: used when provider is not set
# Text-to-Speech
tts:
provider: "edge" # "edge" (free) | "elevenlabs" | "openai" | "neutts" | "minimax" | "mistral" | "gemini" | "xai" | "kittentts" | "piper"
edge:
voice: "en-US-AriaNeural" # 322 voices, 74 languages
elevenlabs:
voice_id: "pNInz6obpgDQGcFmaJgB" # Adam
model_id: "eleven_multilingual_v2"
openai:
model: "gpt-4o-mini-tts"
voice: "alloy" # alloy, echo, fable, onyx, nova, shimmer
base_url: "https://api.openai.com/v1" # optional: override for self-hosted or OpenAI-compatible endpoints
# The `text_to_speech` tool accepts an optional per-call `instructions`
# argument (tone, emotion, pacing, accent, whispering) that is forwarded
# to `gpt-4o-mini-tts` and to OpenAI-compatible voice-design servers
# (e.g. Qwen3-TTS-VoiceDesign via oMLX). See OpenAI's voice-design guide:
# https://platform.openai.com/docs/guides/text-to-speech
neutts:
ref_audio: ''
ref_text: ''
model: neuphonic/neutts-air-q4-gguf
device: cpu
```
### Environment Variables
```bash
# Speech-to-Text providers (local needs no key)
# pip install faster-whisper # Free local STT — no API key needed
GROQ_API_KEY=... # Groq Whisper (fast, free tier)
VOICE_TOOLS_OPENAI_KEY=... # OpenAI Whisper (paid)
# STT advanced overrides (optional)
STT_GROQ_MODEL=whisper-large-v3-turbo # Override default Groq STT model
STT_OPENAI_MODEL=whisper-1 # Override default OpenAI STT model
GROQ_BASE_URL=https://api.groq.com/openai/v1 # Custom Groq endpoint
STT_OPENAI_BASE_URL=https://api.openai.com/v1 # Custom OpenAI STT endpoint
# Text-to-Speech providers (Edge TTS and NeuTTS need no key)
ELEVENLABS_API_KEY=*** # ElevenLabs (premium quality)
# VOICE_TOOLS_OPENAI_KEY above also enables OpenAI TTS
# Discord voice channel
DISCORD_BOT_TOKEN=...
DISCORD_ALLOWED_USERS=...
```
### STT Provider Comparison
| Provider | Model | Speed | Quality | Cost | API Key |
|----------|-------|-------|---------|------|---------|
| **Local** | `base` | Fast (depends on CPU/GPU) | Good | Free | No |
| **Local** | `small` | Medium | Better | Free | No |
| **Local** | `large-v3` | Slow | Best | Free | No |
| **Groq** | `whisper-large-v3-turbo` | Very fast (~0.5s) | Good | Free tier | Yes |
| **Groq** | `whisper-large-v3` | Fast (~1s) | Better | Free tier | Yes |
| **OpenAI** | `whisper-1` | Fast (~1s) | Good | Paid | Yes |
| **OpenAI** | `gpt-4o-transcribe` | Medium (~2s) | Best | Paid | Yes |
| **OpenAI** | `gpt-transcribe` | Fast | Best | Paid ($0.0045/min) | Yes |
| **Mistral** | `voxtral-mini-latest` | Fast | Good | Paid | Yes |
| **xAI** | `grok-stt` | Fast | Good | Paid | Yes |
Provider priority (automatic fallback): **local** > **groq** > **openai**
### TTS Provider Comparison
| Provider | Quality | Cost | Latency | Key Required |
|----------|---------|------|---------|-------------|
| **Edge TTS** | Good | Free | ~1s | No |
| **ElevenLabs** | Excellent | Paid | ~2s | Yes |
| **OpenAI TTS** | Good | Paid | ~1.5s | Yes |
| **NeuTTS** | Good | Free | Depends on CPU/GPU | No |
NeuTTS uses the `tts.neutts` config block above.
For `openai`, the `text_to_speech` tool accepts an optional `instructions`
argument that unlocks `gpt-4o-mini-tts`'s voice-design capability (tone,
emotion, pacing, accent, whispering). The same field also routes to
OpenAI-compatible voice-design servers mounted via `tts.openai.base_url`
(e.g. Qwen3-TTS-VoiceDesign via oMLX).
---
## Troubleshooting
### "No audio device found" (CLI)
PortAudio is not installed:
```bash
brew install portaudio # macOS
sudo apt install portaudio19-dev # Ubuntu
```
If you are running Hermes inside Docker on a Linux desktop, the container also needs access to your host audio socket. See the [Docker audio bridge](/user-guide/docker#optional-linux-desktop-audio-bridge) notes for a PulseAudio/PipeWire-compatible setup.
### Bot doesn't respond in Discord server channels
The bot requires an @mention by default in server channels. Make sure you:
1. Type `@` and select the **bot user** (with the #discriminator), not the **role** with the same name
2. Or use DMs instead — no mention needed
3. Or set `DISCORD_REQUIRE_MENTION=false` in `~/.hermes/.env`
### Bot joins VC but doesn't hear me
- Check your Discord user ID is in `DISCORD_ALLOWED_USERS`
- Make sure you're not muted in Discord
- The bot needs a SPEAKING event from Discord before it can map your audio — start speaking within a few seconds of joining
### Bot hears me but doesn't respond
- Verify STT is available: install `faster-whisper` (no key needed) or set `GROQ_API_KEY` / `VOICE_TOOLS_OPENAI_KEY`
- Check the LLM model is configured and accessible
- Review gateway logs: `tail -f ~/.hermes/logs/gateway.log`
### Bot responds in text but not in voice channel
- TTS provider may be failing — check API key and quota
- Edge TTS (free, no key) is the default fallback
- Check logs for TTS errors
### Whisper returns garbage text
The hallucination filter catches most cases automatically. If you're still getting phantom transcripts:
- Use a quieter environment
- Adjust `silence_threshold` in config (higher = less sensitive)
- Try a different STT model
website/docs/user-guide/features/wake-word.md
+341
View File
@@ -0,0 +1,341 @@
---
sidebar_position: 11
title: "Wake Word"
description: "Hands-free 'Hey Hermes' wake word — start a voice session by speaking, the 'Hey Siri' way"
---
# Wake Word ("Hey Hermes")
The wake word turns Hermes into a hands-free assistant across the CLI, TUI, and
desktop app: with one setting on, Hermes listens in the background for a spoken
trigger phrase. Say it, and Hermes starts a fresh session, opens the microphone,
captures your command via the normal [voice pipeline](/user-guide/features/voice-mode),
and answers — exactly like "Hey Siri" or "Alexa". Use `surface` to pick which
one listens.
Detection runs **entirely on-device**. The always-on listener only watches for
the wake phrase; no audio leaves your machine until you actually speak a command
to the agent.
## How it works
1. With `wake_word.enabled: true` (or after `/wake on`), a lightweight hotword
detector listens on your configured input device, or the process default
microphone when `wake_word.input_device` is unset.
2. When it hears the wake phrase it pauses itself (freeing the mic), starts a new
session, and records one utterance with voice mode's silence detection.
3. Your speech is transcribed and sent to the agent. After it replies, the
listener resumes automatically and waits for the next wake word.
It is **off by default** — nothing listens until you turn it on.
On the desktop app, a hands-free voice conversation can be ended by simply
**saying "stop"** (or "never mind", "goodbye", "cancel", "that's all") — the
spoken command ends the conversation instead of being sent to the agent. Only a
whole-utterance stop command matches, so a real request like "stop the docker
container" still goes through normally.
## Remote desktop (client capture)
When the desktop app connects to a **remote** Hermes backend (for example a
headless Docker host or a machine in another room), the backend often has **no
microphone**. Server-side PortAudio then fails with “Failed to open the
wake-word microphone.”
Hermes supports **client capture** for that case:
1. The desktop arms wake with `capture: client` (automatic for the GUI when the
backend has no local input device, or set explicitly below).
2. openWakeWord still runs **on the backend** (same engines, same models).
3. The desktop opens the **local Mac/PC microphone**, resamples to 16 kHz mono
int16, and streams short frames via the `wake.feed` RPC.
4. On detection the backend emits `wake.detected` as usual; the desktop starts
the normal voice pipeline on the client mic.
```yaml
wake_word:
enabled: true
capture: auto # auto | local | client
# auto — local PortAudio unless the desktop arms with client_capture
# local — always open the backend mic (CLI/TUI default)
# client — always expect wake.feed PCM from the desktop (remote-friendly)
```
The desktop GUI always passes `client_capture: true` on `wake.start`, so remote
backends without a mic arm in client mode automatically. CLI and TUI keep local
capture unless you set `capture: client` explicitly.
Privacy note: with client capture, wake PCM travels over the authenticated
desktop↔backend WebSocket (same channel as the rest of the session). Detection
still does not send audio to third-party wake APIs; the engine is local to the
backend process.
## Engines
| Engine | Cost | API key | Notes |
|--------|------|---------|-------|
| **openWakeWord** (default) | Free | None | Local ONNX models. Ships a bundled **"hey hermes"** model (default); also supports `hey_jarvis`, `alexa`, `hey_mycroft`, … and custom models |
| **sherpa** | Free | None | **Open vocabulary** — detects ANY typed phrase with zero training. Small English model auto-downloads on first use (~13 MB) |
| **Porcupine** | Free tier / paid | `PORCUPINE_ACCESS_KEY` | Picovoice engine; built-in keywords + custom `.ppn` files |
By default the phrase is **"hey hermes"** — a model for it ships with Hermes, so
it works out of the box with no training. (On first use, openWakeWord downloads
its shared feature-extraction models — a small one-time fetch.)
Both are lazy-installed the first time you enable the wake word (desktop
installs made with `--include-desktop` pre-install them, so the ear works
instantly). To install ahead of time:
```bash
cd ~/.hermes/hermes-agent && uv pip install -e ".[wake]"
```
## Quick start
```bash
# In an interactive `hermes` session:
/wake on # start listening (installs the engine on first use)
/wake status # show phrase, provider, and state
/wake off # stop listening
```
In the desktop app, click the ear icon in the composer.
The toggle IS the setting: turning the wake word on or off — via `/wake` or the
desktop ear button — also writes `wake_word.enabled` to `~/.hermes/config.yaml`,
so your choice persists across sessions. You can also flip it by hand:
```yaml
wake_word:
enabled: true
```
## Configuration
```yaml
wake_word:
enabled: false
surface: auto # eligible surface: "auto" | "cli" | "tui" | "gui"
input_device: null # PortAudio input index or device-name substring; null = process default
capture: auto # auto | local | client — where PCM is captured (see Remote desktop)
provider: openwakeword # "openwakeword" (free, local) | "sherpa" (free, any phrase) | "porcupine"
phrase: "hey hermes" # cosmetic label only — detection is keyed by the model/keyword below
sensitivity: 0.6 # 0.0-1.0 — higher = stricter (fewer false triggers), consistent across all engines
confirmation_frames: 3 # openWakeWord only — consecutive over-threshold frames required to fire
start_new_session: true # start a fresh session on wake vs. continue the current one
openwakeword:
model: hey_hermes # bundled default; OR a built-in name OR a path to a custom .onnx/.tflite
inference_framework: "" # "" (auto) | "onnx" | "tflite"
porcupine:
keyword: jarvis # built-in keyword OR path to a custom .ppn
```
`sensitivity`, `phrase`, and `start_new_session` apply to both engines. The
`openwakeword` and `porcupine` blocks select the actual detection model.
`input_device` is passed directly to the wake listener's PortAudio
(`sounddevice`) stream. Use either a numeric device index or an unambiguous
device-name substring. This setting only changes wake-word capture; desktop
push-to-talk still uses the desktop application's microphone path.
### Reducing false triggers on ambient speech
openWakeWord scores one short (~80ms) audio frame at a time, so a stray phoneme
in background conversation can occasionally spike a single frame over the
threshold and fire the wake word unintentionally. Two knobs control this:
- **`confirmation_frames`** (default `3`, openWakeWord only) — how many
*consecutive* over-threshold frames are required before the wake fires. A real
"hey hermes" holds a high score across several frames; an ambient blip spikes
just one. Raise it (e.g. `4`–`5`) if you still get false triggers in a noisy
room; the cost is a few tens of milliseconds of extra latency. `1` restores
the old fire-on-first-frame behavior.
- **`sensitivity`** (default `0.6`) — the detection threshold, `0.0`–`1.0`.
Higher is stricter (fewer false triggers). This direction is consistent across
**all** engines — for openWakeWord it's the raw per-frame score threshold, for
sherpa it maps onto the keyword threshold, and for Porcupine it's inverted
internally so "higher = stricter" holds there too. The `0.6` default sits
above openWakeWord's permissive `0.5` baseline, which let near-misses like
"hey hor" through; raise toward `0.8` if you still get false fires, lower it
if real "hey hermes" utterances are missed.
The `sherpa` and `porcupine` engines decode the whole phrase internally, so they
don't have the single-frame-spike problem and ignore `confirmation_frames`
(but they still honor `sensitivity`).
`inference_framework` picks the openWakeWord backend. Leave it empty (the
default) to let Hermes choose per platform: **tflite on Apple Silicon**, onnx
everywhere else. openWakeWord's onnx backend returns near-zero scores on macOS
ARM64 ([openWakeWord#336](https://github.com/dscripka/openWakeWord/issues/336)),
so a listener pinned to `onnx` there will arm, show as listening, and never
fire. The tflite backend needs `ai-edge-litert` on macOS, which Hermes installs
on demand alongside the other wake-word deps.
### Surfaces (CLI, TUI, GUI)
The wake word works in all three Hermes surfaces, and `surface` picks which one
owns the listener and opens the new session when it fires:
| `surface` | Behavior |
|-----------|----------|
| `auto` (default) | All local surfaces are eligible; the first one to arm owns the listener. |
| `cli` | Only the classic `hermes` CLI. |
| `tui` | Only `hermes --tui`. |
| `gui` | Only the desktop app. |
The detector is on-device and single-mic, so only one surface listens at a time,
including when Hermes surfaces run in separate processes. Ownership is sticky:
the first eligible claimant keeps the listener until it stops, disconnects, or
its process exits. Hermes does not silently fail over to another open surface.
Set `surface` when you want to pin ownership instead of using first-claim wins.
The TUI and desktop GUI share the same Python backend (`tui_gateway`), which
runs the detector server-side and yields the mic to voice capture while a
command records.
## Using a different phrase
"Hey Hermes" works out of the box — the bundled openWakeWord model
(`model: hey_hermes`) is the default. To wake on something else, the easiest
path is the open-vocabulary engine:
### Option A — sherpa (any phrase, zero training)
Type the phrase you want; it's tokenized at runtime — "hey coder",
"computer", "wake up neo", anything:
```yaml
wake_word:
enabled: true
provider: sherpa
phrase: "hey coder" # detection key — just type your phrase
```
The small English KWS model (~13 MB) downloads once on first use. Each
profile can set its own phrase — "hey \<profile\>" for every profile you run.
### Waking a specific profile (desktop)
With the sherpa engine, ONE listener can wake ANY profile. Every profile
whose config has `wake_word.enabled: true` is enrolled automatically; its
phrase defaults to `hey <profile name>` when unset. Say a profile's phrase
and the desktop app live-switches to that profile, opens a fresh session
there, and starts hands-free voice:
- "hey hermes" → default profile
- "hey coder" → the `coder` profile
- "hey trader" → the `trader` profile
Set `wake_word.profile_routing: false` on the listener's profile to opt out
and listen only for its own phrase. The CLI and TUI are single-profile
processes: a wake phrase belonging to another profile prints the switch
command (`hermes -p <profile>`) instead of routing.
Names are matched acoustically by their English subword sounds: two-word
phrases with distinct, 2+ syllable names work best. Very short names, heavy
non-English phonology, or two profiles with similar-sounding names will
degrade accuracy — tune per-profile `sensitivity` if needed.
### Option B — openWakeWord (free, trained model)
Name a built-in model (`hey_jarvis`, `alexa`, `hey_mycroft`, …), or train a
custom model (≈75–90 min on a free/Colab GPU) for maximum robustness, drop
the `.onnx` file somewhere, and reference it:
```yaml
wake_word:
enabled: true
provider: openwakeword
phrase: "computer"
openwakeword:
model: ~/.hermes/wakewords/computer.onnx # or a built-in name like hey_jarvis
```
Training references:
- [openWakeWord](https://github.com/dscripka/openWakeWord)
- [2026 training Colab](https://github.com/alfiedennen/openwakeword-colab-2026)
:::tip Pick a distinctive phrase
Wake phrases that don't collide with everyday speech generalize best. Two
syllables with an uncommon word ("hermes" qualifies) beat common words like
"hello" or "stop".
:::
### Option C — Porcupine (custom keyword in seconds)
Create a "Hey Hermes" keyword in the [Picovoice Console](https://console.picovoice.ai/),
download the `.ppn`, and:
```yaml
wake_word:
enabled: true
provider: porcupine
phrase: "hey hermes"
porcupine:
keyword: ~/.hermes/wakewords/hey_hermes.ppn
```
Set your access key in `~/.hermes/.env`:
```bash
PORCUPINE_ACCESS_KEY=your-key-here
```
## Requirements
- A working microphone and the `sounddevice` + `numpy` audio stack (shared with
voice mode).
- An STT provider for transcribing the spoken command — local `faster-whisper`
works out of the box; see [Voice Mode](/user-guide/features/voice-mode) for the
full provider list.
- A TTS provider for speaking the reply (the default `edge-tts` works with no
key). The wake flow is fully hands-free, so the toggle refuses to arm until
both STT and TTS are ready — `hermes tools` (Voice section) sets them up.
- The wake engine deps (auto-installed, or `hermes-agent[wake]`).
`/wake status` reports exactly what's missing if the listener won't start.
### "Listening" but never wakes (macOS)
macOS grants microphone access per **process**. STT working in the desktop app
proves the *renderer* has mic access — the wake listener runs in the Python
*backend*, which needs its own grant. Without it, CoreAudio hands the backend a
"working" stream that only ever delivers silence, so the ear shows listening
but the phrase never fires. Hermes detects this (`/wake status` shows
"mic delivers only silence"; the desktop ear tooltip carries the same hint).
Fix: System Settings → Privacy & Security → Microphone → enable the Hermes
backend (it may appear as your terminal, `python`, or Hermes), then toggle the
wake word off and on.
### "Listening" but receives silence (Windows)
Desktop push-to-talk and wake-word capture use different microphone paths.
Push-to-talk uses the desktop application's browser capture, while the
wake-word listener opens a PortAudio stream in the Python backend. One can work
while the other selects a silent or unusable Windows input.
`/wake status` reports the selected input device and Windows audio host API.
When it reports silence, set `wake_word.input_device` to the numeric index or an
unambiguous name of the working PortAudio input, then toggle the wake word:
```bash
hermes config set wake_word.input_device "Microphone Array"
```
Use `null` to return to the process default:
```bash
hermes config set wake_word.input_device null
```
## Notes & limits
- **Local surfaces only.** The wake word runs in the CLI, TUI, and desktop GUI —
wherever a local microphone is available. It does not run in the messaging
gateway (Telegram, Discord, …), which has no mic.
- **One mic at a time.** The detector releases the microphone while a command is
recording and reclaims it once the turn ends, so it won't fight voice capture.
- **Privacy.** Hotword detection is local. Set `sensitivity` higher if you get
false triggers, lower if it misses you.
website/docs/user-guide/features/web-dashboard.md
+1205
View File
File diff suppressed because it is too large Load Diff
website/docs/user-guide/features/web-search.md
+489
View File
@@ -0,0 +1,489 @@
---
title: Web Search & Extract
description: Search the web and extract page content with multiple backend providers — including free self-hosted SearXNG.
sidebar_label: Web Search
sidebar_position: 6
---
# Web Search & Extract
Hermes Agent includes two model-callable web tools backed by multiple providers:
- **`web_search`** — search the web and return ranked results
- **`web_extract`** — fetch and extract readable content from one or more URLs
Both are configured through a single backend selection. Providers are chosen via `hermes tools` or set directly in `config.yaml`.
## Backends
| Provider | Env Var | Search | Extract | Free tier |
|----------|---------|--------|---------|-----------|
| **Firecrawl** (default) | `FIRECRAWL_API_KEY` (optional — keyless when selected) | ✔ | ✔ | 500 credits/mo · keyless cloud when selected |
| **SearXNG** | `SEARXNG_URL` | ✔ | — | ✔ Free (self-hosted) |
| **Brave Search (free tier)** | `BRAVE_SEARCH_API_KEY` | ✔ | — | 2 000 queries/mo |
| **DDGS (DuckDuckGo)** | — (no key) | ✔ | — | ✔ Free |
| **Exa** | `EXA_API_KEY` (optional) | ✔ | ✔ | ✔ Keyless ring member · 1 000 searches/mo with key |
| **Parallel** | `PARALLEL_API_KEY` (optional) | ✔ | ✔ | ✔ Keyless ring member · paid with key |
| **Tavily** | `TAVILY_API_KEY` (optional) | ✔ | ✔ | ✔ Opt-in keyless when selected |
| **Keenable** | `KEENABLE_API_KEY` (optional) | ✔ | ✔ | ✔ Keyless ring member · paid with key |
| **xAI (Grok)** | `XAI_API_KEY` or `hermes auth add xai-oauth` | ✔ | — | Paid (SuperGrok or per-token) |
Brave Search, DDGS, and xAI are **search-only** — pair any of them with Firecrawl/Tavily/Keenable/Exa/Parallel when you also need `web_extract`. DDGS uses the [`ddgs` Python package](https://pypi.org/project/ddgs/) under the hood; if it isn't already installed, run `pip install ddgs` (or let Hermes lazy-install it on first use). xAI runs Grok's server-side `web_search` tool on the Responses API — results are LLM-generated rather than index-backed, so titles, descriptions, and URL choice are all model output (see the [trust-model caveat](#xai-grok) below).
**Per-capability split:** you can use different providers for search and extract independently — for example SearXNG (free) for search and Firecrawl for extract. See [Per-capability configuration](#per-capability-configuration) below.
:::info Works out of the box — keyless free-tier rotation
A fresh install with **no web credentials at all** gets working `web_search` and `web_extract` out of the box: requests rotate round-robin across the ring vendors' public free tiers — **Exa, Parallel, Firecrawl, and Keenable** — spreading load evenly, and a rate-limited request automatically retries on the next vendor in the ring (multi-hop, until one serves or all are throttled). No signup, no key. This tier is strictly last-resort — any configured backend or present API key always wins — and requests carry no user identifiers (only a random per-process session id, rotated on restart). For guaranteed, unthrottled service, set up a keyed provider. Disable the keyless tier entirely with `web.keyless_fallback: false`.
:::
**Choosing free vs paid explicitly:** in `hermes tools`, Exa, Parallel, and Keenable each appear as two rows — **Free (keyless)** and **Paid (API key)**. Picking Free pins that vendor's anonymous endpoint (even if you later add a key); picking Paid pins the keyed path (a missing key then errors instead of silently downgrading to the free tier). The selection is stored as `web.provider_tier.<name>: free|paid`; leave it unset for auto (key present → paid, otherwise the keyless ring).
:::tip Nous Subscribers
If you have a paid [Nous Portal](https://portal.nousresearch.com) subscription, web search and extract are available through the **[Tool Gateway](tool-gateway.md)** via managed Firecrawl — no API key needed. New installs can run `hermes setup --portal` to log in and turn on all gateway tools at once; existing installs can flip just web via `hermes tools`.
:::
---
## How `web_extract` handles long pages
Backends return raw page markdown, which can be huge (forum threads, docs sites, news articles with embedded comments). To keep your context window usable, `web_extract` applies a **deterministic character budget** — no LLM summarization is involved:
| Page size (characters) | What happens |
|------------------------|--------------|
| At or under the budget (default 15 000) | Returned whole — full markdown reaches the agent |
| Over the budget | Head+tail window (~75% head / ~25% tail, cut on markdown line boundaries) plus an explicit `[TRUNCATED]` footer. The full clean text is stored to disk and the footer tells the agent the file path and the exact `read_file` call to page through the omitted middle |
| Over 2 000 000 | Stored text is capped at 2 MB |
The per-page budget is configurable via `web.extract_char_limit` in `config.yaml` (default `15000`, clamped to 2 000–500 000), and the agent can raise it per-call with the tool's `char_limit` argument.
### When truncation gets in the way
If you specifically need the live DOM rather than extracted markdown — for example, a JS-heavy page where extraction returns little content — use `browser_navigate` + `browser_snapshot` instead. The browser tool returns the live accessibility tree (subject to its own snapshot cap on huge pages).
---
## Result caching
Repeat web calls within a short window are served from cache instead of the paid backend — this saves credits and latency in the two patterns where duplicates are common: subagent fan-outs (several delegated agents researching the same topic) and the agent re-checking a page it read minutes ago.
| Call | Cache | Scope |
|------|-------|-------|
| `web_search` — same query (case/whitespace-insensitive), same provider | In-memory memo | Per process |
| `web_extract` — same URL, same format, same provider | Full text stored under `~/.hermes/cache/web/` | Shared across CLI, gateway, cron, and subagent processes |
Concurrent identical searches (a parallel subagent fan-out firing the same query at once) are **coalesced into a single backend request** — the first caller pays; the rest share the response. Requested search limits are bucketed up to 10/20/50/100 so near-identical requests (`limit=5` vs `limit=8`) share one entry, with each caller receiving its requested count.
Only successful responses are cached. Failures always retry the backend, responses served by the one-shot keyless rescue are never cached (the next call attempts your chosen backend again), and URLs matched by your `security.website_blocklist` are never served from cache. Cached extracts re-run the normal truncation pipeline, so a different `char_limit` on the second call works off the same stored scrape.
**Local development URLs are never cached.** Anything on `localhost`, `127.0.0.1`, `*.local`, single-label LAN hostnames, or private/link-local IP ranges (`192.168.*`, `10.*`, `172.16-31.*`) bypasses the extract cache entirely — dev servers, hot-reload builds, and chat-GUI artifact previews change on every save, and a cached copy would show you a stale build. Every fetch of a local page is live. (These URLs are only reachable at all when `security.allow_private_urls` is enabled.)
**Testing over the public internet?** Staging deploys and tunnel URLs are public DNS, so the local-dev rule can't catch them — list them in `web.cache_exempt_hosts` and they're always fetched live too. Entries match exactly, as a `*.` wildcard, or as a domain suffix (`mysite.dev` also covers `preview.mysite.dev`):
```yaml
# ~/.hermes/config.yaml
web:
cache_exempt_hosts:
- mysite.vercel.app
- "*.ngrok-free.app"
```
```yaml
# ~/.hermes/config.yaml
web:
cache_enabled: true # default; set false to disable both caches
cache_ttl_minutes: 20 # freshness window, clamped 1–1440
```
If you're researching genuinely live data (scores, prices, breaking news) and need every call fresh, lower the TTL or set `web.cache_enabled: false`.
---
## Setup
### Quick setup via `hermes tools`
Run `hermes tools`, navigate to **Web Search & Extract**, and pick a provider. The wizard prompts for the required URL or API key and writes it to your config.
```bash
hermes tools
```
---
### Firecrawl (default)
Full-featured search and extract. Recommended for most users.
```bash
# ~/.hermes/.env
FIRECRAWL_API_KEY=fc-your-key-here
```
Get a key at [firecrawl.dev](https://firecrawl.dev). The free tier includes 500 credits/month.
**Self-hosted Firecrawl:** Point at your own instance instead of the cloud API:
```bash
# ~/.hermes/.env
FIRECRAWL_API_URL=http://localhost:3002
```
When `FIRECRAWL_API_URL` is set, the API key is optional (disable server auth with `USE_DB_AUTHENTICATION=false`).
---
### SearXNG (free, self-hosted)
SearXNG is a privacy-respecting, open-source metasearch engine that aggregates results from 70+ search engines. **No API key required** — just point Hermes at a running SearXNG instance.
SearXNG is **search-only** — `web_extract` requires a separate extract provider.
#### Option A — Self-host with Docker (recommended)
This gives you a private instance with no rate limits.
**1. Create a working directory:**
```bash
mkdir -p ~/searxng/searxng
cd ~/searxng
```
**2. Write a `docker-compose.yml`:**
```yaml
# ~/searxng/docker-compose.yml
services:
searxng:
image: searxng/searxng:latest
container_name: searxng
ports:
- "8888:8080"
volumes:
- ./searxng:/etc/searxng:rw
environment:
- SEARXNG_BASE_URL=http://localhost:8888/
restart: unless-stopped
```
**3. Start the container:**
```bash
docker compose up -d
```
**4. Enable the JSON API format:**
SearXNG ships with JSON output disabled by default. Copy the generated config and enable it:
```bash
# Copy the auto-generated config out of the container
docker cp searxng:/etc/searxng/settings.yml ~/searxng/searxng/settings.yml
```
Open `~/searxng/searxng/settings.yml`.
If `use_default_settings: true` is present, the file only contains your overrides. All other settings are inherited from the built-in defaults.
To enable JSON responses for Hermes, add the following override:
```yaml
search:
formats:
- html
- json
```
Your `settings.yml` should look similar to:
```yaml
# Read the documentation before extending the defaults:
# https://docs.searxng.org/admin/settings/
use_default_settings: true
server:
secret_key: "abcdef12345678"
image_proxy: true
search:
formats:
- html
- json
```
**5. Restart to apply:**
```bash
docker cp ~/searxng/searxng/settings.yml searxng:/etc/searxng/settings.yml
docker restart searxng
```
**6. Verify it works:**
```bash
curl -s "http://localhost:8888/search?q=test&format=json" | python3 -c \
"import sys,json; d=json.load(sys.stdin); print(f'{len(d[\"results\"])} results')"
```
You should see something like `10 results`. If you get a `403 Forbidden`, JSON format is still disabled — recheck step 4.
**7. Configure Hermes:**
```bash
# ~/.hermes/.env
SEARXNG_URL=http://localhost:8888
```
Then select SearXNG as the search backend in `~/.hermes/config.yaml`:
```yaml
web:
search_backend: "searxng"
```
Or set via `hermes tools` → Web Search & Extract → SearXNG.
---
#### Option B — Use a public instance
Public SearXNG instances are listed at [searx.space](https://searx.space/). Filter by instances that have **JSON format enabled** (shown in the table).
```bash
# ~/.hermes/.env
SEARXNG_URL=https://searx.example.com
```
:::caution Public instances
Public instances have rate limits, variable uptime, and may disable JSON format at any time. For production use, self-hosting is strongly recommended.
:::
---
#### Pair SearXNG with an extract provider
SearXNG handles search; you need a separate provider for `web_extract`. Use the per-capability keys:
```yaml
# ~/.hermes/config.yaml
web:
search_backend: "searxng"
extract_backend: "firecrawl" # or tavily, keenable, exa, parallel
```
With this config, Hermes uses SearXNG for all search queries and Firecrawl for URL extraction — combining free search with high-quality extraction.
---
### Tavily
AI-optimised search and extract. Select Tavily in `hermes tools` (or set `web.backend: tavily`) to use it **keyless** with no account (rate-limited). Set an API key when you want higher limits.
```bash
# optional — skip this for keyless access after selecting Tavily
# ~/.hermes/.env
TAVILY_API_KEY=tvly-your-key-here
```
Get a key at [app.tavily.com](https://app.tavily.com/home). See [Tavily keyless](https://docs.tavily.com/documentation/keyless).
---
### Exa
Neural search with semantic understanding. Good for research and finding conceptually related content.
```bash
# ~/.hermes/.env
EXA_API_KEY=your-exa-key-here
```
Get a key at [exa.ai](https://exa.ai). The free tier includes 1 000 searches/month.
---
### Parallel
AI-native search and extraction with deep research capabilities.
```bash
# ~/.hermes/.env
PARALLEL_API_KEY=your-parallel-key-here
```
Get access at [parallel.ai](https://parallel.ai).
---
### xAI (Grok) {#xai-grok}
Routes `web_search` through Grok's server-side [web_search tool](https://docs.x.ai/developers/tools/web-search) on the Responses API. Grok runs the actual searching and returns the top results as structured JSON.
Works with either credential path — no new env vars, no new setup wizard:
```bash
# ~/.hermes/.env (env-var path)
XAI_API_KEY=sk-xai-your-key-here
```
or for SuperGrok subscribers:
```bash
hermes auth add xai-oauth
```
Then select xAI as the search backend:
```yaml
# ~/.hermes/config.yaml
web:
backend: "xai"
```
**Optional knobs:**
```yaml
web:
backend: "xai"
xai:
model: grok-build-0.1 # reasoning model required by web_search (default)
allowed_domains: # optional, max 5 — mutex with excluded_domains
- arxiv.org
excluded_domains: # optional, max 5
- example-spam.com
timeout: 90 # seconds (default)
```
**Search-only** — pair with Firecrawl / Tavily / Keenable / Exa / Parallel if you also need `web_extract`. On 401 the provider performs a single forced OAuth-token refresh and retries (covers mid-window revocation and opaque tokens the proactive expiry check can't decode); env-var credentials skip the retry.
:::caution Trust model
Unlike index-backed providers (Brave, Tavily, Exa) which return verbatim search-engine results, xAI is an LLM choosing which URLs to surface and writing the titles and descriptions itself. The *content* of the query influences the output, so a maliciously crafted query (e.g. injected via untrusted upstream input the agent picked up) can in principle steer Grok into emitting attacker-chosen URLs. Treat returned URLs the same way you'd treat any model-generated link — validate before fetching, especially if the query came from untrusted input.
:::
---
## Configuration
### Single backend
Set one provider for all web capabilities:
```yaml
# ~/.hermes/config.yaml
web:
backend: "searxng" # firecrawl | searxng | brave-free | ddgs | tavily | keenable | exa | parallel | xai
```
### Per-capability configuration
Use different providers for search vs extract. This lets you combine free search (SearXNG) with a paid extract provider, or vice versa:
```yaml
# ~/.hermes/config.yaml
web:
search_backend: "searxng" # used by web_search
extract_backend: "firecrawl" # used by web_extract
```
When per-capability keys are empty, both fall through to `web.backend`. Only when no web selection has ever been written is the backend auto-detected from whichever API key/URL is present — once a selection exists, the runtime always uses it, and adding a key to `.env` does not reroute web traffic.
**Priority order (per capability):**
1. `web.search_backend` / `web.extract_backend` (explicit per-capability)
2. `web.backend` (shared fallback; `nous` = managed Tool Gateway)
3. Auto-detect from environment variables (never-configured setups only)
### Auto-detection
If no backend has **ever** been selected (no `web.backend` / per-capability key written by you or `hermes tools`), Hermes picks the first available one based on which credentials are set:
| Credential present | Auto-selected backend |
|--------------------|-----------------------|
| `TAVILY_API_KEY` | tavily |
| `EXA_API_KEY` | exa |
| `PARALLEL_API_KEY` | parallel |
| `FIRECRAWL_API_KEY` or `FIRECRAWL_API_URL` (or the Nous Tool Gateway is ready) | firecrawl |
| `SEARXNG_URL` | searxng |
| `BRAVE_SEARCH_API_KEY` | brave-free |
| `ddgs` package importable | ddgs |
| *(nothing set at all)* | keyless ring: exa / parallel / firecrawl / keenable (round-robin) |
**Keyless free-tier ring:** when *no* credential above is present, requests rotate across the ring vendors' public free tiers (Exa, Parallel, Firecrawl, Keenable) so web tools work on a fresh install with zero setup — and a rate-limited request fails over to the next vendor in the ring automatically. Pin one vendor in `hermes tools` to stop the rotation (the ring is then only used as failover succession on throttles). All free tiers are vendor-rate-limited under burst load; sustained normal usage goes through fine. Set `web.keyless_fallback: false` to turn the tier off — with it off and no credentials, web tools are unavailable until a provider is configured.
**One-shot keyless rescue for keyed backends:** when your chosen/keyed backend fails a call (bad key, outage, upstream 5xx), that single call automatically retries on the keyless free-tier ring instead of erroring — the result notes which vendor served it and why (`rescued_from` / `backend_error`). The failover is never sticky: the very next `web_search`/`web_extract` call attempts your chosen backend again. Disable with `web.keyless_rescue: false` (also off whenever `keyless_fallback` is off).
xAI Web Search is **not** in the auto-detection chain — having `XAI_API_KEY` set (or being signed in via xAI Grok OAuth) does not automatically route web traffic through xAI, since those credentials are also used for inference / TTS / image gen and the user may want a different backend for web. Opt in explicitly with `web.backend: "xai"`.
---
## Verify your setup
Run `hermes setup` to see which web backend is detected:
```
✅ Web Search & Extract (searxng)
```
Or check via the CLI:
```bash
# Activate the venv and run the web tools module directly
source ~/.hermes/hermes-agent/.venv/bin/activate
python -m tools.web_tools
```
This prints the active backend and its status:
```
✅ Web backend: searxng
Using SearXNG (search only): http://localhost:8888
```
---
## Troubleshooting
### `web_search` returns `{"success": false}`
- Check `SEARXNG_URL` is reachable: `curl -s "http://localhost:8888/search?q=test&format=json"`
- If you get HTTP 403, JSON format is disabled — add `json` to the `formats` list in `settings.yml` and restart
- If you get a connection error, the container may not be running: `docker ps | grep searxng`
### `web_extract` says "search-only backend"
SearXNG cannot extract URL content. Set `web.extract_backend` to a provider that supports extraction:
```yaml
web:
search_backend: "searxng"
extract_backend: "firecrawl" # or tavily / keenable / exa / parallel
```
### SearXNG returns 0 results
Some public instances disable certain search engines or categories. Try:
- A different query
- A different public instance from [searx.space](https://searx.space/)
- Self-hosting your own instance for reliable results
### Rate limited on a public instance
Switch to a self-hosted instance (see [Option A](#option-a--self-host-with-docker-recommended) above). With Docker, your own instance has no rate limits.
### `web_extract` returns truncated content with a `[TRUNCATED]` footer
That's expected for pages over the character budget. The footer names the on-disk file holding the full clean text and the exact `read_file` call to page through the omitted middle. To see more inline, raise `web.extract_char_limit` in `config.yaml` or pass a larger `char_limit` on the call.
---
## Optional skill: `searxng-search`
For agents that need to use SearXNG via `curl` directly (e.g. as a fallback when the web toolset isn't available), install the `searxng-search` optional skill:
```bash
hermes skills install official/research/searxng-search
```
This adds a skill that teaches the agent how to:
- Call the SearXNG JSON API via `curl` or Python
- Filter by category (`general`, `news`, `science`, etc.)
- Handle pagination and error cases
- Fall back gracefully when SearXNG is unreachable
website/docs/user-guide/features/x-search.md
+167
View File
@@ -0,0 +1,167 @@
---
title: X (Twitter) Search
description: Search X (Twitter) posts and threads from within the agent using xAI's built-in x_search Responses tool — works with either a SuperGrok OAuth login or an XAI_API_KEY.
sidebar_label: X (Twitter) Search
sidebar_position: 7
---
# X (Twitter) Search
The `x_search` tool lets the agent search X (Twitter) posts, profiles, and threads directly. It's backed by xAI's built-in `x_search` tool on the Responses API at `https://api.x.ai/v1/responses` — Grok itself runs the search server-side and returns synthesized results with citations to the originating posts.
**Use this instead of `web_search`** when you specifically want current discussion, reactions, or claims **on X**. For general web pages, keep using `web_search` / `web_extract`.
## `x_search` vs `xurl`
Hermes can expose two different X surfaces:
| Surface | Use it for | Do not use it for |
|---------|------------|-------------------|
| `x_search` | Read-only public X discovery: current discussion, reactions, claims, profiles, threads, and synthesized answers with citations. | Posting, replying, liking, DMs, media upload, deleting, or proving that an authenticated X account changed state. |
| `xurl` skill | Exact or authenticated X API work: `post`, `reply`, `read`, `like`, `dm`, timelines, mentions, media upload, account-specific reads, and raw v2 endpoints. | Broad Grok-synthesized public X research when `x_search` is available and no authenticated account context is needed. |
For mixed workflows, use `x_search` to discover candidate public posts, then switch to `xurl read` or another exact `xurl` command after the target post/user/action is clear. Any state-changing X action must be confirmed by `xurl` output or the X API response; an `x_search` answer is never evidence that a write happened.
:::tip
If you're paying Portal for an xAI model anyway, Live Search calls bill against the same xAI key configured for chat. See [Nous Portal](/integrations/nous-portal).
:::
## Authentication
`x_search` registers when **either** xAI credential path is available:
| Credential | Source | Setup |
|------------|--------|-------|
| **SuperGrok / X Premium+ OAuth** | Browser login at `accounts.x.ai`, refreshed automatically | `hermes auth add xai-oauth` — see [xAI Grok OAuth (SuperGrok / X Premium+)](../../guides/xai-grok-oauth.md) |
| **`XAI_API_KEY`** (preferred) | Paid xAI API key | Set in `~/.hermes/.env` |
Both hit the same endpoint with the same payload — the only difference is the bearer token. **When both are configured, the explicit `XAI_API_KEY` wins** — the subscription OAuth bearer authorizes `/v1/responses` but answers x_search in a degraded Grok explanatory mode with no citations, while the API key returns real posts. Note this means x_search runs against metered API billing when a key is set; remove `XAI_API_KEY` to fall back to your subscription quota (with the degraded-answer caveat).
The tool's `check_fn` runs the xAI credential resolver every time the model's tool list is rebuilt. A `True` return means the bearer is fetchable AND non-empty AND (if it had expired) successfully refreshed. Revoked tokens with a failed refresh hide the tool from the schema; the model simply can't see it.
## Enabling the tool
Auto-enables when xAI credentials (OAuth token or `XAI_API_KEY`) are present. Disable explicitly via `hermes tools` → Search → x_search if you don't want this.
```bash
hermes tools
# → 🐦 X (Twitter) Search (press space to toggle on)
```
The picker offers two credential choices:
1. **xAI Grok OAuth (SuperGrok / Premium+)** — opens the browser to `accounts.x.ai` if you're not already logged in
2. **xAI API key** — prompts for `XAI_API_KEY`
Either choice satisfies the gating. You can pick whichever credentials you already have; the tool works identically with both. If both end up configured, OAuth is preferred at call time.
## Configuration
```yaml
# ~/.hermes/config.yaml
x_search:
# xAI model used for the Responses call.
# grok-4.5 is the recommended default; any Grok model
# with x_search tool access works.
model: grok-4.5
# Optional reasoning effort: low, medium, high, or xhigh. When omitted,
# the selected model's default applies. xhigh is supported only by
# models that document it, such as grok-4.20-multi-agent.
# reasoning_effort: low
# Request timeout in seconds. x_search can take 60–120s for
# complex queries — the default is generous. Minimum: 30.
timeout_seconds: 180
# Number of automatic retries on 5xx / ReadTimeout / ConnectionError.
# Each retry backs off (1.5x attempt seconds, capped at 5s).
retries: 2
```
`reasoning_effort` is sent to the xAI Responses API as
`reasoning: {effort: ...}`. Leave it unset for models that do not support
configurable reasoning. Invalid values fail before an API request is made.
## Tool parameters
The agent calls `x_search` with these arguments:
| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | string (required) | What to look up on X. |
| `allowed_x_handles` | string array | Optional list of handles to include **exclusively** (max 10). Leading `@` is stripped. |
| `excluded_x_handles` | string array | Optional list of handles to exclude (max 10). Mutually exclusive with `allowed_x_handles`. |
| `from_date` | string | Optional `YYYY-MM-DD` start date. |
| `to_date` | string | Optional `YYYY-MM-DD` end date. |
| `enable_image_understanding` | boolean | Ask xAI to analyze images attached to matching posts. |
| `enable_video_understanding` | boolean | Ask xAI to analyze videos attached to matching posts. |
The tool returns JSON with:
- `answer` — synthesized text response from Grok
- `citations` — citations returned by the Responses API top-level field
- `inline_citations` — `url_citation` annotations extracted from the message body (each with `url`, `title`, `start_index`, `end_index`)
- `degraded` — `true` when any narrowing filter (`allowed_x_handles`, `excluded_x_handles`, `from_date`, `to_date`) was set AND both citation channels came back empty. In that case the `answer` was synthesized from the model's own knowledge rather than the X index, so treat it as unsourced. `false` otherwise (including the "no filters set" case — a broad unsourced answer is just an answer, not a filter miss)
- `degraded_reason` — short string naming which filters were active, or `null` when `degraded` is `false`
- `credential_source` — `"xai-oauth"` if OAuth resolved, `"xai"` if API key resolved
- `model`, `query`, `provider`, `tool`, `success`
### Date validation
`from_date` / `to_date` are validated client-side before the HTTP call:
- Both, if provided, must parse as `YYYY-MM-DD`.
- When both are set, `from_date` must be on or before `to_date`.
- `from_date` must not be later than today UTC — no posts can exist in a window that hasn't started yet, so the call would be guaranteed to return zero citations.
- `to_date` in the future is allowed (callers may legitimately request "from yesterday to tomorrow" to catch posts as they arrive).
Validation failures surface as a structured `{"error": "..."}` tool result, never as an HTTP call to xAI.
## Example
Talking to the agent:
> What are people on X saying about the new Grok image features? Focus on responses from @xai.
The agent will:
1. Call `x_search` with `query="reactions to new Grok image features"`, `allowed_x_handles=["xai"]`
2. Get back a synthesized answer plus a list of citations linking to specific posts
3. Reply with the answer and references
If the next user request is "reply to the best one" or "like that post", the agent should switch to the `xurl` skill, confirm the exact target post, and use the X API action. `x_search` remains a discovery tool.
## Troubleshooting
### "No xAI credentials available"
The tool surfaces this when both auth paths fail. Either set `XAI_API_KEY` in `~/.hermes/.env` or run `hermes auth add xai-oauth` and complete the browser login. Then restart your session so the agent re-reads the tool registry.
### "`x_search` is not enabled for this model"
The configured `x_search.model` doesn't have access to the server-side `x_search` tool. Switch to `grok-4.5` (the default) or another Grok model that supports it. Check the [xAI documentation](https://docs.x.ai/) for the current list.
### Tool doesn't appear in the schema
Two possible causes:
1. **Toolset not enabled.** Run `hermes tools` and confirm `🐦 X (Twitter) Search` is checked.
2. **No xAI credentials.** The check_fn returns False, so the schema stays hidden. Run `hermes auth status` to confirm xai-oauth login state, and check that `XAI_API_KEY` is set (if you're using the API-key path).
### `degraded: true` — answer with no citations
When you used `allowed_x_handles`, `excluded_x_handles`, or a date range and the response comes back with `degraded: true`, xAI's X index returned no matching posts but Grok still produced a synthesized answer from its own training data. The answer is unsourced — do not treat it as a real X result.
Causes worth checking:
- **Typo in the handle.** Strip the `@`, double-check spelling, and confirm the account exists.
- **Date range too narrow** or sliding past today's posts; widen and retry.
- **xAI index gap.** Some active accounts intermittently fail to surface in `x_search` even when they post regularly. Retry after a few minutes, or use the `xurl` skill for direct X API reads when you need an exact handle's timeline.
## See Also
- [xAI Grok OAuth (SuperGrok / Premium+)](../../guides/xai-grok-oauth.md) — the OAuth setup guide
- [xurl skill](../skills/bundled/social-media/social-media-xurl.md) — official X API CLI for authenticated account actions
- [Web Search & Extract](web-search.md) — for general (non-X) web search
- [Tools Reference](../../reference/tools-reference.md) — full tool catalog