Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license
This commit is contained in:
@@ -0,0 +1,325 @@
|
||||
# Hermes Observer Hooks
|
||||
|
||||
Hermes observer hooks are the read-only telemetry contract for plugins that
|
||||
need to reconstruct agent execution without changing runtime behavior. This
|
||||
contract supports trace, metrics, audit, replay, and export integrations such
|
||||
as Langfuse, OpenTelemetry-style collectors, and NeMo Relay.
|
||||
|
||||
Observer hooks are intentionally backend-neutral. They expose stable lifecycle
|
||||
events, correlation IDs, sanitized payloads, timing, status, and error fields.
|
||||
They do not replace Hermes' planner, model providers, memory, tool registry,
|
||||
approval UX, CLI, gateway behavior, or execution semantics.
|
||||
|
||||
Behavior-changing request or execution wrappers are outside this observer
|
||||
contract. Observer hooks should report what happened; they should not replace
|
||||
provider requests, tool arguments, or execution callbacks.
|
||||
|
||||
Hermes also has a first-party NeMo Relay shared-metrics path. It uses these
|
||||
lifecycle boundaries directly and does not require enabling an observability
|
||||
plugin. See [Relay shared metrics](relay-shared-metrics.md).
|
||||
|
||||
## Contract
|
||||
|
||||
Plugins register observer callbacks from `register(ctx)`:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_hook("pre_api_request", on_pre_api_request)
|
||||
ctx.register_hook("post_api_request", on_post_api_request)
|
||||
ctx.register_hook("pre_tool_call", on_pre_tool_call)
|
||||
ctx.register_hook("post_tool_call", on_post_tool_call)
|
||||
```
|
||||
|
||||
Every hook callback receives keyword arguments. Plugins should accept
|
||||
`**kwargs` so additive fields remain backward-compatible:
|
||||
|
||||
```python
|
||||
def on_post_tool_call(**kwargs):
|
||||
tool_name = kwargs.get("tool_name")
|
||||
status = kwargs.get("status")
|
||||
result = kwargs.get("result")
|
||||
```
|
||||
|
||||
The plugin manager injects this field into every hook payload:
|
||||
|
||||
```text
|
||||
telemetry_schema_version = "hermes.observer.v1"
|
||||
```
|
||||
|
||||
Hook callbacks are fail-open. Hermes catches callback exceptions, logs a
|
||||
warning, and keeps the agent loop running.
|
||||
|
||||
Most observer hook return values are ignored. The exceptions are older
|
||||
behavior-affecting hooks:
|
||||
|
||||
| Hook | Return behavior |
|
||||
| --- | --- |
|
||||
| `pre_llm_call` | May return a string or `{"context": "..."}` to inject ephemeral context into the current user message. |
|
||||
| `pre_tool_call` | May return `{"action": "block", "message": "..."}` to block a tool before execution, or `{"action": "modify", "args": {...}}` to transform the tool's input arguments. |
|
||||
| `transform_tool_result` | May return a replacement tool result string after `post_tool_call`. |
|
||||
| `transform_llm_output` | May return a replacement final assistant text string. |
|
||||
|
||||
Telemetry plugins should treat these behavior-affecting returns as optional
|
||||
compatibility features, not as observability requirements.
|
||||
|
||||
## Correlation IDs
|
||||
|
||||
Observer payloads use stable IDs so plugins can join events without relying on
|
||||
callback order alone.
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `session_id` | Conversation/session identity. |
|
||||
| `task_id` | Task identity, especially useful for subagents and isolated execution. |
|
||||
| `turn_id` | User-turn identity shared by API attempts and tool calls in a turn. |
|
||||
| `api_request_id` | Opaque provider-attempt identity. Do not parse its string format. |
|
||||
| `api_call_count` | Numeric API attempt count within the agent loop. |
|
||||
| `tool_call_id` | Provider-supplied tool call ID when available. |
|
||||
| `parent_session_id` / `child_session_id` | Session link for delegated subagents. |
|
||||
| `parent_subagent_id` / `child_subagent_id` | Subagent link when available. |
|
||||
| `parent_turn_id` | Parent turn that spawned delegated work. |
|
||||
|
||||
Consumers should prefer explicit fields over parsing compound IDs. In
|
||||
particular, `api_request_id` is an opaque correlation value.
|
||||
|
||||
## Event Families
|
||||
|
||||
### Session Lifecycle
|
||||
|
||||
Session hooks describe conversation boundaries and resets:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `on_session_start` | A brand-new session starts after the system prompt is built. |
|
||||
| `on_session_end` | A `run_conversation` call ends, including interrupted or incomplete turns. |
|
||||
| `on_session_finalize` | CLI or gateway tears down an active session identity. |
|
||||
| `on_session_reset` | CLI or gateway moves from an old session identity to a new one. |
|
||||
|
||||
Common fields include `session_id`, `completed`, `interrupted`, `reason`,
|
||||
`old_session_id`, and `new_session_id` where available.
|
||||
|
||||
`on_session_end` is turn/run scoped. It is not necessarily the final lifetime
|
||||
boundary for a chat identity. Use `on_session_finalize` and `on_session_reset`
|
||||
for lifecycle cleanup that must happen once per session identity.
|
||||
|
||||
### Turn-Scoped LLM Hooks
|
||||
|
||||
These hooks frame the user turn, not individual provider API attempts:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_llm_call` | Before the tool loop begins for a user turn. |
|
||||
| `post_llm_call` | After the turn completes with final assistant output. |
|
||||
|
||||
Common `pre_llm_call` fields include `session_id`, `turn_id`,
|
||||
`user_message`, `conversation_history`, `is_first_turn`, `model`, `platform`,
|
||||
and `sender_id`.
|
||||
|
||||
Common `post_llm_call` fields include `session_id`, `turn_id`,
|
||||
`user_message`, `assistant_response`, `conversation_history`, `model`, and
|
||||
`platform`.
|
||||
|
||||
Use request-scoped API hooks for LLM span telemetry. Use `pre_llm_call` and
|
||||
`post_llm_call` for turn-level context, compatibility, and final turn summary.
|
||||
|
||||
### Request-Scoped API Hooks
|
||||
|
||||
API hooks describe provider attempts inside the agent loop:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_api_request` | Immediately before a provider API request. |
|
||||
| `post_api_request` | After a successful provider response. |
|
||||
| `api_request_error` | After a failed provider request or retryable error path. |
|
||||
|
||||
`pre_api_request` includes:
|
||||
|
||||
- identity: `session_id`, `task_id`, `turn_id`, `api_request_id`
|
||||
- runtime: `platform`, `model`, `provider`, `base_url`, `api_mode`
|
||||
- attempt metadata: `api_call_count`, `message_count`, `tool_count`,
|
||||
`approx_input_tokens`, `request_char_count`, `max_tokens`
|
||||
- timing: `started_at`
|
||||
- sanitized request payload: `request`
|
||||
|
||||
`post_api_request` includes the same identity/runtime fields plus:
|
||||
|
||||
- `api_duration`, `started_at`, `ended_at`
|
||||
- `finish_reason`, `message_count`, `response_model`
|
||||
- `usage`
|
||||
- `assistant_content_chars`, `assistant_tool_call_count`
|
||||
- sanitized response payload: `response`
|
||||
- compatibility object: `assistant_message`
|
||||
|
||||
`api_request_error` includes the same identity/runtime fields plus:
|
||||
|
||||
- `api_duration`, `started_at`, `ended_at`
|
||||
- `status_code`, `retry_count`, `max_retries`, `retryable`, `reason`
|
||||
- structured `error = {"type": ..., "message": ...}`
|
||||
- sanitized failed request payload: `request`
|
||||
|
||||
The sanitized `request`, `response`, and `error` fields are the canonical
|
||||
observer inputs for new consumers.
|
||||
|
||||
### Tool Lifecycle
|
||||
|
||||
Tool hooks describe individual tool calls:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_tool_call` | Before guardrail-approved tool dispatch. |
|
||||
| `post_tool_call` | After tool dispatch, cancellation, block, or error completion. |
|
||||
| `transform_tool_result` | After `post_tool_call`, before the result is appended to model context. |
|
||||
|
||||
`pre_tool_call` includes `tool_name`, `args`, `task_id`, `session_id`,
|
||||
`tool_call_id`, `turn_id`, and `api_request_id`.
|
||||
|
||||
`post_tool_call` includes the same identity fields plus `result`,
|
||||
`duration_ms`, `status`, `error_type`, and `error_message`.
|
||||
|
||||
`status` is the observer-grade lifecycle outcome. Common values include:
|
||||
|
||||
| Status | Meaning |
|
||||
| --- | --- |
|
||||
| `ok` | Tool completed normally. |
|
||||
| `error` | Tool ran and returned or raised an error outcome. |
|
||||
| `blocked` | A `pre_tool_call` hook blocked execution. |
|
||||
| `cancelled` | Execution was cancelled before normal completion. |
|
||||
|
||||
`post_tool_call` is emitted for blocked and cancelled paths so telemetry
|
||||
plugins can close spans cleanly.
|
||||
|
||||
### Approval Lifecycle
|
||||
|
||||
Approval hooks describe dangerous-command approval prompts:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_approval_request` | Before the approval request is shown or sent. |
|
||||
| `post_approval_response` | After the user responds or the request times out. |
|
||||
|
||||
Common fields include `command`, `description`, `pattern_key`,
|
||||
`pattern_keys`, `session_key`, and `surface`.
|
||||
|
||||
`post_approval_response` also includes `choice`, with values such as `once`,
|
||||
`session`, `always`, `deny`, and `timeout`.
|
||||
|
||||
Approval hooks are observer-only. Plugins cannot pre-answer or veto approvals
|
||||
from these hooks. To prevent a tool from reaching approval, use
|
||||
`pre_tool_call` blocking.
|
||||
|
||||
### Subagent Lifecycle
|
||||
|
||||
Subagent hooks describe delegated child-agent work:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `subagent_start` | A delegated child agent is created. |
|
||||
| `subagent_stop` | A delegated child agent returns or fails. |
|
||||
|
||||
`subagent_start` fields include `parent_session_id`, `parent_turn_id`,
|
||||
`parent_subagent_id`, `child_session_id`, `child_subagent_id`, `child_role`,
|
||||
and `child_goal`.
|
||||
|
||||
`subagent_stop` fields include parent/child session IDs, role/status fields,
|
||||
`child_summary`, `duration_ms`, and a metadata-only `tool_call_history`. Each
|
||||
history entry contains the tool name, argument names, bounded side-effect
|
||||
targets, input/output byte counts, and outcome. URL query strings and fragments
|
||||
are removed; raw arguments, prompts, commands, contents, headers, and results
|
||||
are intentionally excluded.
|
||||
|
||||
Observers can use these hooks to model nested trajectories while keeping child
|
||||
agent execution linked to the parent turn that spawned it.
|
||||
|
||||
## Payload Safety
|
||||
|
||||
Observer payloads are designed for telemetry consumers, not raw object access.
|
||||
New consumers should use the sanitized API payloads:
|
||||
|
||||
- `pre_api_request.request`
|
||||
- `post_api_request.response`
|
||||
- `api_request_error.request`
|
||||
- `api_request_error.error`
|
||||
|
||||
Sanitization converts provider objects to JSON-compatible structures, bounds
|
||||
large payloads, redacts sensitive keys, and avoids exposing raw response
|
||||
objects in sanitized fields.
|
||||
|
||||
Legacy compatibility fields such as `request_messages`, `conversation_history`,
|
||||
and `assistant_message` may still be present for existing plugins. New
|
||||
observability consumers should prefer the sanitized payloads.
|
||||
|
||||
## Performance
|
||||
|
||||
The default uninstrumented path should stay cheap. Expensive request/response
|
||||
payload construction is gated behind `has_hook(...)`, so Hermes only builds
|
||||
sanitized API telemetry payloads when at least one plugin registered the
|
||||
relevant hook.
|
||||
|
||||
Plugin authors should preserve this property:
|
||||
|
||||
- Register only hooks the plugin actually consumes.
|
||||
- Avoid deep-copying or re-sanitizing already sanitized payloads.
|
||||
- Keep hook callbacks fast and fail-open.
|
||||
- Offload network export or batch writes when practical.
|
||||
|
||||
## Writing An Observer Plugin
|
||||
|
||||
Minimal observer plugin:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_hook("pre_api_request", on_pre_api_request)
|
||||
ctx.register_hook("post_api_request", on_post_api_request)
|
||||
ctx.register_hook("pre_tool_call", on_pre_tool_call)
|
||||
ctx.register_hook("post_tool_call", on_post_tool_call)
|
||||
|
||||
|
||||
def on_pre_api_request(**kwargs):
|
||||
start_llm_span(
|
||||
request_id=kwargs.get("api_request_id"),
|
||||
turn_id=kwargs.get("turn_id"),
|
||||
request=kwargs.get("request"),
|
||||
model=kwargs.get("model"),
|
||||
)
|
||||
|
||||
|
||||
def on_post_api_request(**kwargs):
|
||||
finish_llm_span(
|
||||
request_id=kwargs.get("api_request_id"),
|
||||
response=kwargs.get("response"),
|
||||
usage=kwargs.get("usage"),
|
||||
duration=kwargs.get("api_duration"),
|
||||
)
|
||||
|
||||
|
||||
def on_pre_tool_call(**kwargs):
|
||||
start_tool_span(
|
||||
call_id=kwargs.get("tool_call_id"),
|
||||
name=kwargs.get("tool_name"),
|
||||
args=kwargs.get("args"),
|
||||
)
|
||||
|
||||
|
||||
def on_post_tool_call(**kwargs):
|
||||
finish_tool_span(
|
||||
call_id=kwargs.get("tool_call_id"),
|
||||
result=kwargs.get("result"),
|
||||
status=kwargs.get("status"),
|
||||
duration_ms=kwargs.get("duration_ms"),
|
||||
)
|
||||
```
|
||||
|
||||
Use `session_id`, `turn_id`, `api_request_id`, and `tool_call_id` for span
|
||||
correlation. Use subagent and approval hooks when the export format supports
|
||||
nested agent work or security lifecycle events.
|
||||
|
||||
## Existing Consumers
|
||||
|
||||
The bundled Langfuse plugin demonstrates direct hook-based observability for
|
||||
turns, provider requests, and tool calls.
|
||||
|
||||
The native NeMo Relay SDK integration maps Hermes session, turn, LLM, and tool
|
||||
lifecycles to Relay. Explicit Relay plugin configuration can add
|
||||
[ATOF, ATIF, or OTEL](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/about)
|
||||
exporters and execution middleware; see
|
||||
[Relay shared metrics](relay-shared-metrics.md).
|
||||
@@ -0,0 +1,304 @@
|
||||
# Gateway Monitoring
|
||||
|
||||
Service health monitoring plus structured operational diagnostics for the
|
||||
Hermes gateway daemon, exported over OTLP/HTTP to an operator-configured
|
||||
endpoint (OpenTelemetry Collector, DataDog, or any OTLP receiver).
|
||||
|
||||
This plane is content-free by construction. It exports gateway and cron
|
||||
lifecycle state, platform connector health, and content-free warning/error
|
||||
diagnostics. It never exports prompts, messages, tool arguments or results,
|
||||
job names, destinations, schedules, raw errors, session history, usage
|
||||
analytics, audit logs, or detailed execution traces. Run/model/tool trajectory
|
||||
capture is a separate plane served by Hermes's native NeMo Relay SDK
|
||||
integration and explicitly configured Relay subscribers or exporters.
|
||||
|
||||
## What gets exported
|
||||
|
||||
| Signal | OTLP route | Content |
|
||||
| --- | --- | --- |
|
||||
| Gateway gauges | `/v1/metrics` | `hermes.gateway.up/state/busy/drainable/active_agents/background_work/background_delegations/restart_requested`, `hermes.platform.up/degraded` with bounded `error_code` attributes |
|
||||
| Health/lifecycle events | `/v1/traces` | `gateway.lifecycle` state transitions (`starting -> running -> draining -> stopped`, `startup_failed`, exit), `gateway.health_snapshot`, platform state changes |
|
||||
| Diagnostics | `/v1/logs` | Warning/error gateway events with a constant body and bounded subsystem, severity, error class, and error code attributes; rendered log messages are never exported |
|
||||
| Cron scheduler gauges | `/v1/metrics` | Ticker heartbeat and last-success age (omitted when unavailable), a monotonic catch-up-occurrence count from the scheduler's stale-window branch, enabled/running job counts, and overdue count derived from persisted `next_run_at` plus the scheduler's existing grace rule |
|
||||
| Cron execution lifecycle | `/v1/traces` | Durable `claimed/running/completed/failed/unknown` states, bounded source and error class, opaque hashed job key, elapsed duration when timestamps exist, and delivery outcome when the scheduler knows it; terminal states make a fail-open flush attempt that can delay completion by up to one second |
|
||||
|
||||
Signals carry `service.name`, version, supervision mode, and a stable one-way
|
||||
hash of the install id so an operator can distinguish instances without
|
||||
exporting account/profile identity or the raw install identifier.
|
||||
|
||||
`hermes.gateway.active_agents`, `hermes.gateway.background_work`, and
|
||||
`hermes.gateway.background_delegations` are complementary. `active_agents`
|
||||
counts foreground message turns plus in-flight cron jobs plus API runs — the
|
||||
work the gateway drains on shutdown. `background_work` counts detached work that
|
||||
`active_agents` never includes: backgrounded `delegate_task` subagents,
|
||||
`terminal(background=true)` processes, and kanban workers; it is
|
||||
**task-granular** — a fan-out batch of N subagents counts as N — so it reflects
|
||||
real concurrent subagent load. `background_delegations` counts only async
|
||||
delegation **units** (each `delegate_task` dispatch is one, a fan-out batch is
|
||||
one), matching the async pool's capacity accounting; alert it against
|
||||
`delegation.max_concurrent_children` to see slot pressure. Sum `active_agents`
|
||||
and `background_work` for total live work per instance; use
|
||||
`background_delegations` for pool-saturation.
|
||||
|
||||
## Enabling
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
monitoring:
|
||||
gateway_health_export:
|
||||
enabled: true
|
||||
export:
|
||||
otlp:
|
||||
enabled: true
|
||||
endpoint: http://collector-host:4318/v1/traces # metrics/logs derive
|
||||
headers_env: {} # header name -> ENV VAR NAME (values never stored)
|
||||
```
|
||||
|
||||
Check the posture any time:
|
||||
|
||||
```bash
|
||||
hermes monitoring status
|
||||
```
|
||||
|
||||
The OpenTelemetry SDK is an optional extra (`pip install 'hermes-agent[otlp]'`),
|
||||
lazy-installed on first use. When the SDK is missing or the endpoint is down,
|
||||
the gateway runs unaffected: metric collection and ordinary event export stay
|
||||
off the hot path, while terminal cron events make one bounded fail-open flush
|
||||
attempt of up to one second so the final state is less likely to be lost.
|
||||
|
||||
Works identically under systemd/launchd/s6 supervision, containers, tmux, or
|
||||
a plain `hermes gateway run`: the exporter lives in the gateway process, so
|
||||
no sidecar, agent, or collector is required on the host.
|
||||
|
||||
## Collecting into DataDog
|
||||
|
||||
Run a customer-owned OpenTelemetry Collector and forward:
|
||||
|
||||
```yaml
|
||||
# otel-collector config
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
exporters:
|
||||
datadog:
|
||||
api:
|
||||
key: ${env:DD_API_KEY}
|
||||
service:
|
||||
pipelines:
|
||||
metrics: {receivers: [otlp], exporters: [datadog]}
|
||||
traces: {receivers: [otlp], exporters: [datadog]}
|
||||
logs: {receivers: [otlp], exporters: [datadog]}
|
||||
```
|
||||
|
||||
Point `monitoring.export.otlp.endpoint` at the collector. Alerts belong on
|
||||
`hermes.gateway.up`, `hermes.platform.up`, and `hermes.platform.degraded`.
|
||||
|
||||
## Generic fleet queries and alerts
|
||||
|
||||
The exact syntax depends on the customer's observability backend. The examples
|
||||
below use PromQL-style expressions and intentionally avoid vendor-specific
|
||||
routing, destinations, or customer inventory.
|
||||
|
||||
Group fleet views by the opaque `service.instance.id` resource attribute. A
|
||||
process that has died cannot emit its own zero, so every deployment needs both
|
||||
explicit-state and missing-series detection.
|
||||
|
||||
```promql
|
||||
# Explicit gateway failure.
|
||||
hermes_gateway_up == 0
|
||||
|
||||
# Box disappeared or stopped exporting. Choose a window longer than the
|
||||
# configured export interval and collector retry allowance.
|
||||
absent_over_time(hermes_gateway_up[5m])
|
||||
|
||||
# Locally owned bridge is explicitly down.
|
||||
hermes_platform_up == 0
|
||||
|
||||
# Scheduler thread is stale even though the gateway may still be alive.
|
||||
hermes_cron_scheduler_heartbeat_age_seconds > 180
|
||||
|
||||
# Ticker loops but has not completed a successful tick recently.
|
||||
hermes_cron_scheduler_last_success_age_seconds > 300
|
||||
|
||||
# One or more jobs are beyond their existing scheduler grace window.
|
||||
hermes_cron_jobs_overdue > 0
|
||||
|
||||
# Catch-up counter increased, proving at least one stale occurrence was
|
||||
# collapsed and run once after a delay.
|
||||
increase(hermes_cron_scheduler_catch_up_occurrences[15m]) > 0
|
||||
```
|
||||
|
||||
Cron execution lifecycle records arrive as `hermes.cron_execution` spans.
|
||||
Alert or derive events from bounded attributes such as:
|
||||
|
||||
```text
|
||||
hermes.status = failed|unknown
|
||||
hermes.delivery_outcome = failed|not_configured
|
||||
hermes.error_class = auth_failed|rate_limited|timeout|network_error|
|
||||
dispatch_failed|interrupted|empty_response|
|
||||
invalid_config|unknown
|
||||
```
|
||||
|
||||
Recommended operator views:
|
||||
|
||||
1. one row per `service.instance.id` with gateway and configured local-platform
|
||||
state;
|
||||
2. scheduler heartbeat, last-success age, running count, overdue count, and
|
||||
catch-up increase;
|
||||
3. a cron lifecycle feed keyed only by opaque `hermes.job_key`;
|
||||
4. separate alerts for box absence, local bridge down, scheduler stale, cron
|
||||
failed/unknown, delivery failure, and overdue/catch-up activity.
|
||||
|
||||
Keep alert thresholds and routing in deployment-owned configuration. Do not add
|
||||
job names, prompts, outputs, schedules, destinations, raw errors, profile names,
|
||||
or account identity merely to make a dashboard easier to read.
|
||||
|
||||
## Release-validation scenarios
|
||||
|
||||
Before accepting a deployment, force and verify all five cases through the real
|
||||
collector and backend:
|
||||
|
||||
1. **Cron success:** observe `claimed -> running -> completed`, duration, and a
|
||||
truthful delivery outcome.
|
||||
2. **Cron failure:** observe `failed` plus a bounded error class, with no raw
|
||||
exception or content in the decoded OTLP payload.
|
||||
3. **Cron interruption:** stop the owning gateway during execution, restart it,
|
||||
and observe recovery to `unknown`.
|
||||
4. **Locally owned bridge outage:** break one native connector, observe its
|
||||
bounded down/retrying/fatal state and recovery, and verify unaffected boxes
|
||||
remain healthy.
|
||||
5. **Killed gateway:** terminate one canary, verify missing-series detection,
|
||||
restart it, and confirm the same opaque instance identity returns.
|
||||
|
||||
Hermes Agent-owned Relay transport health remains in scope. A separate gateway
|
||||
or connector service remains authoritative for any shared connected-platform
|
||||
state that it owns and should export that state through its own telemetry path.
|
||||
|
||||
For every scenario, verify the signal and alert clear on recovery, other boxes
|
||||
remain unaffected, collector failure stays fail-open, and decoded metrics,
|
||||
spans, logs, and resource attributes remain content-free.
|
||||
|
||||
## Local smoke test (no Docker)
|
||||
|
||||
```bash
|
||||
# terminal 1: capture collector on :4318
|
||||
python scripts/observability/otel_capture_collector.py \
|
||||
--host 127.0.0.1 --port 4318 --log /tmp/hermes_otel_capture.jsonl
|
||||
|
||||
# terminal 2: drive the real exporter through lifecycle transitions,
|
||||
# a fatal platform, and a structured warning event, then flush
|
||||
python scripts/observability/gateway_health_export_probe.py \
|
||||
--endpoint http://127.0.0.1:4318/v1/traces \
|
||||
--log /tmp/hermes_otel_capture.jsonl --wait 8
|
||||
# exit 0 prints: {"requests": 6, "paths": ["/v1/logs", "/v1/metrics", "/v1/traces"]}
|
||||
```
|
||||
|
||||
## Maintaining and extending this plane
|
||||
|
||||
This plane is a **fixed, enumerated, content-free vocabulary** by design. Adding
|
||||
a signal is not just "emit a new metric" — every new name and attribute must be
|
||||
declared in each layer that enforces the bounded vocabulary, or it is silently
|
||||
dropped downstream. Follow the checklist for the change you are making. The
|
||||
golden rule: **a new signal that is emitted but not declared in every layer
|
||||
looks like a code bug but is a vocabulary-registration bug — nothing errors, the
|
||||
signal just never arrives.**
|
||||
|
||||
### Content-free invariant (applies to every change)
|
||||
|
||||
Before adding anything, confirm it cannot carry content. Numbers, booleans,
|
||||
ages, durations, monotonic counts, and one-way hashes are safe. **Never** add an
|
||||
attribute that can hold a job name, prompt, output, schedule, destination, raw
|
||||
exception text, file path, profile name, account id, or free-form string. When
|
||||
you must key a record to a job/entity, hash it (`sha256(...)[:24]`, see
|
||||
`_job_key` in `agent/monitoring/cron_health.py`) — never emit the raw id. All
|
||||
string attributes that could touch user input must pass through
|
||||
`redaction.redact_for_export` and be truncated (see `_span_attrs` in
|
||||
`agent/monitoring/otlp_exporter.py`).
|
||||
|
||||
### Adding a new gauge/metric
|
||||
|
||||
1. Emit it in the snapshot builder (`agent/monitoring/gateway_health.py`
|
||||
`build_gateway_health_snapshot`, `cron_health.py` `build_cron_health_snapshot`,
|
||||
or a sibling reader wired into `_read_runtime_snapshot` in
|
||||
`gateway_health_export.py`). Best-effort: never let a reader raise into the
|
||||
collection loop — wrap it and log a **content-free WARNING with the exception
|
||||
TYPE name only** (the pattern the cron and background-work readers use), so a
|
||||
future regression is visible instead of silently dropping the signal.
|
||||
2. Register the dotted metric name in the observable-gauge `metric_names` list in
|
||||
`gateway_health_export.py::_start_metric_provider`. **A gauge that is emitted
|
||||
in the snapshot but not registered here is never observed.**
|
||||
3. Add the export-table row and an alert example in this file.
|
||||
4. If the deployment fronts the exporter with an OpenTelemetry Collector that
|
||||
uses a metric-name allowlist (a `filter/...` processor with `name != "..."`
|
||||
guards), add the new name there too — otherwise the collector drops it before
|
||||
the backend. This is not repo code, but it is the single most common reason a
|
||||
correctly-emitted new metric never appears; call it out in the PR so the
|
||||
deploying operator updates their collector config.
|
||||
|
||||
### Adding a new subsystem (a new family of signals)
|
||||
|
||||
Mirror the cron pattern (`cron_health.py` + its wiring): put the read/projection
|
||||
logic in its own module, expose one `build_<subsystem>_health_snapshot()` that
|
||||
returns bounded `GatewayMetric`s (and events if any), and extend it into
|
||||
`_read_runtime_snapshot` with the same best-effort try/except-WARNING guard.
|
||||
Then do the "adding a metric" checklist for each new name, and the "adding an
|
||||
attribute" checklist for each new event attribute. Add a release-validation
|
||||
scenario below for the subsystem's failure mode.
|
||||
|
||||
### Extending the error-class / status / source / state vocabularies
|
||||
|
||||
These are the closed enums that keep the plane bounded. Extend the SET, then the
|
||||
classifier, never one without the other:
|
||||
|
||||
- **Cron** (`agent/monitoring/cron_health.py`): `_KNOWN_STATUSES`,
|
||||
`_KNOWN_SOURCES`, `_KNOWN_DELIVERY_OUTCOMES`, and the `classify_cron_error`
|
||||
keyword buckets. Anything not in the set is coerced to `unknown` on the way
|
||||
out, so a new value that is not added to the set is invisible.
|
||||
- **Gateway/platform** (`agent/monitoring/gateway_health.py`):
|
||||
`_KNOWN_GATEWAY_STATES`, `_KNOWN_PLATFORM_STATES`, and `classify_gateway_error`.
|
||||
|
||||
Rules: keep the vocabulary SMALL and operationally meaningful (an error class
|
||||
should map to an operator action, not to an exception subclass); a new bucket
|
||||
must match on a stable keyword, not on message text that could vary; update the
|
||||
`hermes.error_class = ...` list in this file's alert section and the enum's unit
|
||||
test so the contract is asserted, not frozen as a count.
|
||||
|
||||
### Adding a content-free attribute to an existing event/span
|
||||
|
||||
Add the key to the emitter's per-kind `keep_by_kind` allowlist in
|
||||
`agent/monitoring/otlp_exporter.py::_span_attrs` (unlisted keys are dropped), run
|
||||
it through redaction if it is ever string-shaped, and — as with metrics — if the
|
||||
deployment's collector has a span-attribute `keep_keys(...)` allowlist, add the
|
||||
attribute there too or it is stripped in transit.
|
||||
|
||||
### Verify the whole chain, not just emission
|
||||
|
||||
Emitting is necessary but not sufficient. Confirm the signal survives all the
|
||||
way to the backend, because the enums, the `metric_names` registration, the
|
||||
emitter attribute allowlist, and any collector allowlist each drop unlisted
|
||||
values with no error:
|
||||
|
||||
```bash
|
||||
hermes monitoring status # posture
|
||||
python scripts/observability/gateway_health_export_probe.py \
|
||||
--endpoint http://127.0.0.1:4318/v1/traces \
|
||||
--log /tmp/cap.jsonl --wait 8 # drive the real exporter
|
||||
```
|
||||
|
||||
Decode the captured OTLP payload and assert the new name/attribute is present
|
||||
AND that no content leaked. When a real collector sits in front, add its
|
||||
allowlist entries and re-verify against the backend, not just the local capture.
|
||||
|
||||
## Boundaries and roadmap
|
||||
|
||||
The `hermes monitoring` CLI intentionally exposes `status` only. This first
|
||||
release covers only Hermes Agent-owned service-health and operational-diagnostic
|
||||
signals, including Hermes Agent-owned Relay transport health. Team Gateway's
|
||||
authoritative shared connector/platform state is explicitly out of scope, as
|
||||
are product analytics, audit/quality reporting, and detailed execution traces.
|
||||
Shared client usage metrics and enterprise trace telemetry are being designed on
|
||||
the NeMo Relay integration with their own consent, policy, and export
|
||||
boundaries; this monitoring plane stays narrow so an operator can enable it
|
||||
without touching any content-bearing signal. The telemetry surface may be
|
||||
reorganized as that lands.
|
||||
@@ -0,0 +1,482 @@
|
||||
# NeMo Relay Shared Metrics
|
||||
|
||||
Hermes includes NeMo Relay as a normal runtime dependency on platforms for
|
||||
which Relay publishes a native wheel. The shared-metrics integration is built
|
||||
into Hermes and does not require a Hermes observability plugin. Hermes remains
|
||||
importable without Relay on other native targets. Those targets use an
|
||||
explicit reduced-capability no-op host:
|
||||
Hermes execution remains available, while Relay scopes, middleware, plugins,
|
||||
and subscribers are unavailable. The `hermes-agent[nemo-relay]` extra remains
|
||||
as a no-op compatibility alias for existing installation commands.
|
||||
|
||||
> [!WARNING]
|
||||
> This removes the Hermes `observability/nemo_relay` plugin. Existing users
|
||||
> must remove `observability/nemo_relay` (or its legacy `nemo_relay` alias)
|
||||
> from `plugins.enabled` and move exporter configuration into a Relay
|
||||
> `plugins.toml` selected with `HERMES_NEMO_RELAY_PLUGINS_TOML`. The legacy
|
||||
> `HERMES_NEMO_RELAY_ATOF_*` and `HERMES_NEMO_RELAY_ATIF_*` variables no
|
||||
> longer activate exporters. Without the new variable, Hermes does not run
|
||||
> Relay plugin discovery, configuration layering, middleware, or exporters.
|
||||
|
||||
Hermes requires NeMo Relay 0.8.3 or later within the 0.8 release line. That
|
||||
line provides the provider-codec and canonical tool-result contracts Hermes
|
||||
uses for managed provider and tool calls.
|
||||
|
||||
## Runtime Dependency and Data Boundary
|
||||
|
||||
Hermes installs the platform-specific `nemo-relay` native wheel from the
|
||||
bounded `>=0.8.3,<0.9` dependency range. The published package is built from
|
||||
the [NVIDIA NeMo Relay repository](https://github.com/NVIDIA/NeMo-Relay).
|
||||
Unsupported platforms use the explicit no-op runtime described above rather
|
||||
than downloading a different implementation.
|
||||
|
||||
Operator-supplied typed native plugins must be rebuilt for Relay 0.8. `grpc-v1`
|
||||
workers must be regenerated and rebuilt when they use tool callbacks, tool
|
||||
execution intercepts, or manual tool-end APIs.
|
||||
|
||||
When Relay managed execution is active, the provider request and response pass
|
||||
through that native module in the Hermes process so configured interceptors can
|
||||
operate on the real call. This is separate from the shared-metrics data
|
||||
contract. Shared-metrics mode installs no rich-observability network exporter,
|
||||
and its subscriber
|
||||
accepts only the versioned, allowlisted projection described below. The
|
||||
opt-in package sender described in Appendix A is the only outbound path, it
|
||||
transmits nothing unless the user enables both `enabled` and `send`, and it
|
||||
sends whole packages rather than live spans. Enabling a
|
||||
separately configured rich-observability or dynamic plugin can create a
|
||||
different data path and requires its own policy review.
|
||||
|
||||
Collection remains off unless Hermes policy enables it:
|
||||
|
||||
```yaml
|
||||
telemetry:
|
||||
shared_metrics:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
This choice is read from the profile's own `config.yaml`. A machine-managed
|
||||
configuration overlay cannot enable or disable shared metrics on the profile's
|
||||
behalf.
|
||||
|
||||
Relay plugin activation is owned by the native runtime and remains explicitly
|
||||
opt-in. Set `HERMES_NEMO_RELAY_PLUGINS_TOML` to a selected `plugins.toml` to
|
||||
activate configured middleware, exporters, or dynamic plugins. When the
|
||||
variable is unset, Hermes does not invoke Relay's plugin initializer, so Relay
|
||||
does not perform plugin configuration discovery or layering. When it is set
|
||||
and the selected file loads successfully, Relay discovers supported user and
|
||||
system `plugins.toml` files and layers the selected static configuration over
|
||||
them. Repository-local `.nemo-relay/plugins.toml` files are ignored. Dynamic
|
||||
`[[plugins.dynamic]]` records are loaded from the selected file only. If the
|
||||
selected file cannot be loaded, Hermes reports the error and does not invoke
|
||||
Relay initialization or fall back to ambient discovery.
|
||||
|
||||
## Session-Span Segmentation for Continuous Sessions
|
||||
|
||||
Relay exports a span when its scope closes. A continuous gateway session can
|
||||
remain open for days, so its session span remains open even though each turn
|
||||
span is exported normally. Optional segmentation rotates only the session
|
||||
scope at a turn boundary:
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
telemetry:
|
||||
session_segments:
|
||||
on_compaction: false # rotate after context compaction
|
||||
max_turns: 0 # 0 = unlimited; N = turns per segment
|
||||
```
|
||||
|
||||
| Key | Default | Behavior |
|
||||
|---|---:|---|
|
||||
| `on_compaction` | `false` | Rotate after compaction completes, at the next turn boundary. |
|
||||
| `max_turns` | `0` | Rotate after every N completed turns; `0` disables the cap. |
|
||||
|
||||
Both defaults preserve one session scope for the full session. Rotated spans
|
||||
retain the same `session_id` and add `hermes.session.segment` plus
|
||||
`hermes.session.segment_reason` (`compaction` or `max_turns`).
|
||||
|
||||
## Process-Wide Plugin Policy and Profile Isolation
|
||||
|
||||
Relay plugin configuration is a process-level deployment choice, not a Hermes
|
||||
profile setting. The first hosted profile triggers lazy initialization, and
|
||||
every additional profile hosted by that Hermes process shares the resulting
|
||||
static middleware, dynamic plugins, subscribers, exporters, and guardrail
|
||||
policy. After initialization succeeds, Hermes logs:
|
||||
|
||||
```text
|
||||
Relay plugins are active process-wide and apply to all profiles hosted by this Hermes process.
|
||||
```
|
||||
|
||||
Profile scopes still preserve causal isolation inside that shared policy.
|
||||
ATIF groups events by their top-level Agent scope, so simultaneous profile
|
||||
sessions produce separate trajectories rather than one mixed trajectory.
|
||||
ATOF and other global subscribers observe events from every hosted profile.
|
||||
Static and dynamic middleware likewise runs for managed calls from every
|
||||
profile.
|
||||
|
||||
A worker plugin running in a separate worker process does not create a
|
||||
per-profile security boundary. One process-wide activation dispatches calls
|
||||
from all hosted profiles to that worker while preserving the invoking
|
||||
profile's Relay scope stack. Native dynamic plugins are loaded into the Hermes
|
||||
process and share the same policy boundary.
|
||||
|
||||
Run profiles in separate Hermes processes when they require different trust
|
||||
levels, plugin credentials, exporter destinations, or guardrail policies.
|
||||
This process-wide plugin contract does not change each profile's independent
|
||||
shared-metrics consent, local SQLite state, or ATIF trajectory grouping.
|
||||
|
||||
Hermes core owns one Relay host and one isolated Relay session scope per Hermes
|
||||
session. Core lifecycle producers use
|
||||
`hermes_cli.observability.relay_runtime` to obtain the shared session handle or
|
||||
run Relay scope, LLM, tool, and mark APIs in that session context. New product
|
||||
marks do not require Hermes plugin registration. Shared-metrics marks must
|
||||
still contain only fields approved by the versioned allowlist; the hard
|
||||
dependency does not change the collection or privacy policy.
|
||||
|
||||
## Current Slices
|
||||
|
||||
The current vertical slices record pseudonymous profile activity, logical
|
||||
model calls, top-level task runs, tool and approval outcomes, and skill
|
||||
lifecycle and reuse:
|
||||
|
||||
```text
|
||||
Hermes turn, API, tool, and approval hooks
|
||||
-> Relay session, task, LLM, tool, and mark lifecycle
|
||||
-> Hermes shared-metrics subscriber
|
||||
-> SQLite counters
|
||||
-> immutable JSON delta package
|
||||
```
|
||||
|
||||
Hermes sends an empty `LLMRequest` into the metrics-owned lifecycle. This does
|
||||
not describe the separate managed-execution call through the native runtime
|
||||
documented above. The terminal metrics event contains the model identifier and
|
||||
provider route that Hermes used for the logical call, such as
|
||||
`nvidia/nemotron-3-ultra` through `openrouter`. These identifiers are
|
||||
lowercased and structurally bounded, but they are not normalized through a
|
||||
checked-in model catalog. Pricing and model-family classification belong to
|
||||
the metrics backend. Prompts, responses, endpoints, errors, session IDs, task
|
||||
IDs, and request IDs are not included in the metrics event or package.
|
||||
New calls use `hermes.model_route.count`. The previous
|
||||
`hermes.model_call.count` contract remains readable only so pending local
|
||||
counters created by older builds can be exported without losing data.
|
||||
|
||||
The first consented session start emits an empty `hermes.client.active` Relay
|
||||
mark. The profile-scoped subscriber creates a random UUID install identity and
|
||||
uses a transactional compare-and-set to record at most one client-active
|
||||
counter in any rolling 24-hour window. The metric has no dimensions; Hermes
|
||||
version, OS family, architecture, and install method remain bounded package
|
||||
resources. Concurrent Hermes processes share the SQLite latch, so simultaneous
|
||||
starts cannot double-count one install. A later session or task can attempt the
|
||||
mark again, but the subscriber suppresses it until the rolling window expires.
|
||||
|
||||
Each task run is a Relay `Function` scope named `hermes.task_run`, parented to
|
||||
the owning Hermes session. The start counter contains only bounded execution
|
||||
surface and entrypoint values. The terminal counter contains bounded outcome,
|
||||
end reason, termination status, duration, logical model-call count, terminal
|
||||
tool-call count, and provider-retry count buckets. Retries are additional
|
||||
provider attempts for the same Hermes API request ID; they do not inflate the
|
||||
logical model-call count. Tool calls are deduplicated by their Hermes tool-call
|
||||
ID after a terminal tool result is observed. The outer `AIAgent` execution
|
||||
boundary closes the task for normal returns, early returns, exceptions, and
|
||||
cancellations. Active task ownership follows the task ID if Hermes rotates its
|
||||
conversation session during context compression.
|
||||
|
||||
Each tool invocation is represented by a Relay tool lifecycle named
|
||||
`hermes.tool_call`. The terminal counter contains only bounded tool category,
|
||||
outcome, approval outcome, latency, and explicit retry-count buckets. Hermes
|
||||
derives the category from the toolset already declared in its runtime registry;
|
||||
custom and unrecognized toolsets collapse to `other` rather than exporting
|
||||
tool or plugin names. Hermes does not infer retries from repeated tool names or
|
||||
adjacent calls; when the
|
||||
hook does not provide an explicit retry relationship, the retry bucket is
|
||||
`unknown`. Approval decisions are emitted as `hermes.tool_approval` marks and
|
||||
recorded as attributed to a tool call or explicitly `unattributed`. Tool names,
|
||||
call IDs, arguments, results, commands, descriptions, and error text are not
|
||||
included in shared-metrics events or packages. A started tool that is still
|
||||
open when its task terminates is closed as failed, timed out, or cancelled and
|
||||
remains in the task's tool-count bucket.
|
||||
|
||||
Successful skill mutations emit `hermes.skill.lifecycle` marks with only a
|
||||
bounded action and provenance. Successful loads emit `hermes.skill.load`
|
||||
marks with bounded provenance, first-use or reuse state, reuse-after-patch
|
||||
state, and a use-count bucket. Hermes derives reuse and patch-generation
|
||||
continuity transactionally in its existing `skills/.usage.json` state; skill
|
||||
names and exact counts or generations never enter Relay metrics events,
|
||||
SQLite dimensions, or packages. A use after a new patch is counted once as
|
||||
`reused_after_patch`; later uses remain ordinary reuse until another patch.
|
||||
Task-outcome attribution after a patch remains deferred until its window and
|
||||
multi-skill semantics are defined.
|
||||
|
||||
Local state is written under:
|
||||
|
||||
```text
|
||||
$HERMES_HOME/telemetry/shared_metrics/metrics.sqlite3
|
||||
$HERMES_HOME/telemetry/shared_metrics/outbox/*.json
|
||||
```
|
||||
|
||||
The database keeps transactional aggregate and package-outbox state. Package
|
||||
files are immutable delta documents that conform to a closed JSON schema and
|
||||
are written with atomic replacement. Each package records the Hermes version,
|
||||
OS family, architecture, and install method as bounded client resources.
|
||||
Unrecognized platform or installation values are exported as `unknown`; raw
|
||||
platform strings, hostnames, and paths are never included. Fully packaged
|
||||
aggregate rows and successfully exported package rows and files are retained
|
||||
locally for 30 days. Pending package rows and counters with unexported deltas
|
||||
are never pruned.
|
||||
Package schema v1 remains unchanged for existing outbox files. New packages
|
||||
use v2, which accepts both the retired model-call contract and the current
|
||||
model-route contract so upgrades can drain pending counters safely.
|
||||
|
||||
Each package contains an `install_id` generated as a random UUID. Despite the
|
||||
schema field name, its current scope is one `HERMES_HOME`, so it is more
|
||||
precisely a persistent pseudonymous profile identifier. It is not derived from
|
||||
hardware, account, host, path, or credential data. It remains stable across
|
||||
packages from that profile and can therefore link those local packages.
|
||||
Deleting `$HERMES_HOME/telemetry/shared_metrics` resets the identifier together
|
||||
with all aggregates and package files.
|
||||
|
||||
Remote delivery is opt-in and off by default. Reusing the persistent local
|
||||
identifier remotely required a separate product and privacy decision covering
|
||||
consent, identity scope, reset behavior, retention, and deletion — that
|
||||
decision has been made.
|
||||
|
||||
> Those decisions are recorded in
|
||||
> [Appendix A](#appendix-a-remote-exporter-decisions-phase-2), and the exporter
|
||||
> implementing them has shipped. Collection alone still transmits nothing: the
|
||||
> sender runs only when `telemetry.shared_metrics.send` is also true. Each
|
||||
> transmitted package carries the stable `install_id` as-is (product decision,
|
||||
> 2026-08-27 — see A.2 for the record, including the superseded
|
||||
> HMAC-pseudonym design).
|
||||
|
||||
The install identity is scoped to one `HERMES_HOME`. To reset it, stop Hermes
|
||||
processes and remove `$HERMES_HOME/telemetry/shared_metrics`. This deliberately
|
||||
removes the old identity, aggregate database, and queued local packages
|
||||
together; the next consented session creates a new identity. Disabling shared
|
||||
metrics stops new collection but does not silently delete previously collected
|
||||
local state.
|
||||
|
||||
## Smoke Test
|
||||
|
||||
Run a real Hermes CLI turn against the deterministic local model server:
|
||||
|
||||
```bash
|
||||
./.venv/bin/python scripts/smoke_nemo_relay_shared_metrics.py
|
||||
```
|
||||
|
||||
The script uses the installed `nemo-relay` dependency by default. Pass
|
||||
`--relay-python ../nemo-relay/python` only when testing a locally built Relay
|
||||
binding.
|
||||
|
||||
The smoke has the local model request a real `read_file` tool call before its
|
||||
final response, then drives create, load, reuse, patch, edit, stale, archive,
|
||||
restore, and install skill transitions through the installed Relay binding. It
|
||||
verifies model, provider, task, tool, and skill counters in SQLite, validates
|
||||
all exported delta packages against the closed schema, verifies the
|
||||
pseudonymous client-active counter, and checks that prompt, response, tool-call
|
||||
ID, tool-result, and skill-name canaries are absent from the packages.
|
||||
|
||||
## Appendix A: Remote Exporter Decisions (Phase 2)
|
||||
|
||||
Status: **implemented.** This appendix answers the product and
|
||||
privacy questions that "Current Slices" defers to a future remote exporter. It
|
||||
records what was decided and why, so the reasoning survives the implementation.
|
||||
|
||||
Sending is off by default and requires both `telemetry.shared_metrics.enabled`
|
||||
and `telemetry.shared_metrics.send`.
|
||||
|
||||
The exporter sends the package files already written under
|
||||
`$HERMES_HOME/telemetry/shared_metrics/outbox/` to the Hermes telemetry ingest
|
||||
service. That service validates only the envelope (`schema_version` plus a UUID
|
||||
`package_id`) and stores the body verbatim in S3.
|
||||
|
||||
### A.1 Consent
|
||||
|
||||
Transmission is a **separate opt-in** from collection, under a new config key:
|
||||
|
||||
```yaml
|
||||
telemetry:
|
||||
shared_metrics:
|
||||
enabled: false # collect locally
|
||||
send: false # NEW: transmit to the Nous telemetry service
|
||||
```
|
||||
|
||||
- `send` defaults to **false**. Collection alone never transmits.
|
||||
- `send` requires `enabled`. It does **not** imply it: a transmission flag must
|
||||
not silently switch on collection. `send: true` with `enabled: false` warns
|
||||
and does nothing.
|
||||
- Like `enabled`, `send` is profile-owned and is not overridden by
|
||||
managed-scope configuration.
|
||||
|
||||
**A package is only sent when its whole period falls inside a recorded
|
||||
consent window.** Consent is stored as explicit intervals in the shared-
|
||||
metrics SQLite store (`send_consent_windows`): a window opens when `send:
|
||||
true` is first observed, is confirmed forward by every later observation,
|
||||
and closes — at the last *confirmed* moment, never at the wall clock — when
|
||||
`send: false` is observed. A single reconciler derives this table from the
|
||||
config on every process start, so wizard changes, hand-edits to
|
||||
`config.yaml`, and mid-pass revocations all take the same path, and no
|
||||
transition can be missed by any of them.
|
||||
|
||||
Any package whose period predates the first window, falls between windows,
|
||||
or runs past the newest confirmed moment is excluded — the gate fails
|
||||
closed. A fresh package therefore waits at most one process start after its
|
||||
period completes before becoming eligible.
|
||||
|
||||
The gate is on the **period**, not on the package's creation time. One period
|
||||
is split across several packages created on different days: a day's first
|
||||
package is written that day, and a tail package for the same period typically
|
||||
follows the next day. Gating on creation time would send a period's tail while
|
||||
dropping its head, reporting a **silently undercounted** day. Gating on the
|
||||
period keeps consent forward-only and every transmitted period complete.
|
||||
|
||||
Local history can be up to 30 days old, and that data was collected under a
|
||||
promise that nothing is uploaded. Honouring consent forward-only costs at most
|
||||
30 days of backlog we never had permission to send.
|
||||
|
||||
### A.2 Identity scope — the stable install_id is transmitted as-is
|
||||
|
||||
**Decision record.** The original design of this exporter (and revisions 1–8
|
||||
of this appendix) transmitted a keyed pseudonym instead of the identifier:
|
||||
`HMAC-SHA256(key = locally-held rotating salt, message = install_id)`, with
|
||||
the salt rotating every 30 days. On **2026-08-27**, before the feature
|
||||
shipped (zero consented users, zero production transmissions), the product
|
||||
owner decided the analytical need is a **stable cross-window identity** —
|
||||
retention curves, longitudinal install behaviour — which rotation by design
|
||||
destroys. The pseudonymization layer was removed in full rather than
|
||||
weakened in place.
|
||||
|
||||
What is transmitted now:
|
||||
|
||||
- Each package carries `install_id` verbatim: the persistent, profile-scoped
|
||||
random UUID described above.
|
||||
- It is generated locally (`uuid4`), contains no hardware, account, user, or
|
||||
machine-derived information, and identifies a *profile*, not a person.
|
||||
- It is stable until the user deletes the shared-metrics directory, which
|
||||
regenerates it (see A.4).
|
||||
|
||||
Consequences stated plainly rather than papered over:
|
||||
|
||||
- Packages from one profile correlate **indefinitely**, not per-window.
|
||||
Long-term linkability of one install's daily envelope sequence is now the
|
||||
designed behaviour, not a residue.
|
||||
- The A.3 residue analysis of the old design (stable `resource` tuple +
|
||||
contiguous periods bridging rotation windows) is moot — there is no window
|
||||
boundary left to bridge.
|
||||
- The setup wizard's consent language states this identity model explicitly;
|
||||
it was updated in the same change that removed the derivation, so no
|
||||
consent was ever collected under the old wording in any shipped build.
|
||||
|
||||
**Byte-identical resends still hold.** The transmitted id is recorded on the
|
||||
row (`sent_install_id`) when the package is first prepared, and the wire body
|
||||
is always rebuilt from that recorded value, so a retry rebuilds identical
|
||||
bytes. The contract requires this: resending a `package_id` with different
|
||||
content is undefined behaviour. (With a stable id the recorded copy is no
|
||||
longer load-bearing against rotation — it remains as the audit column and as
|
||||
cheap insurance against any future change to identity semantics.)
|
||||
|
||||
### A.3 Rotation — removed (decision record)
|
||||
|
||||
Salt rotation was deleted together with the derivation (product decision,
|
||||
2026-08-27). This section is retained as a record of what the earlier design
|
||||
did and why the removal was accepted:
|
||||
|
||||
- Rotation existed to bound long-term linkability: one identity per 30-day
|
||||
window, unrelated identities across windows.
|
||||
- The documented residue (see git history for the full analysis): the
|
||||
envelope's stable, low-entropy `resource` tuple plus contiguous daily
|
||||
periods could plausibly bridge windows for rare configurations anyway, so
|
||||
the boundary was a cost-raiser, not a wall.
|
||||
- The product need that killed it: cross-window continuity is precisely what
|
||||
retention analysis requires. A boundary that mostly inconveniences honest
|
||||
analysis while only raising costs for a determined correlator was judged
|
||||
the wrong trade once stable identity became a requirement.
|
||||
|
||||
There is no salt in the store, no rotation schedule, and no derived
|
||||
identifier anywhere in the pipeline.
|
||||
|
||||
### A.4 Reset behavior
|
||||
|
||||
Removing `$HERMES_HOME/telemetry/shared_metrics` still resets local identity,
|
||||
aggregates, and package files, exactly as documented above. Two honest
|
||||
qualifications now apply:
|
||||
|
||||
- Reset regenerates `install_id`, so subsequent packages transmit a **new**
|
||||
identity. Local reset does give a new remote identity.
|
||||
- Reset **cannot unsend**. Packages already transmitted remain in the ingest
|
||||
service's storage under the identifier they were sent with. There is no
|
||||
read-back or delete API in the v1 contract.
|
||||
|
||||
Setting `send: false` stops transmission immediately: consent is re-read
|
||||
before every package, so a pass already in flight stops after the package it
|
||||
is currently sending rather than draining its whole batch. It does not delete
|
||||
previously transmitted packages, and it does not stop local collection.
|
||||
|
||||
Turning sending off also **closes the consent window** — at the last moment
|
||||
consent was actually observed, not at the wall clock. Packages whose periods
|
||||
fall between one window and the next are never transmitted, even if sending
|
||||
is later re-enabled, and this holds for any number of on/off cycles, across
|
||||
hand-edits with no process running, and under a clock that jumps in either
|
||||
direction (window opens are clamped above every timestamp already in the
|
||||
store; observation marks advance by a bounded step per call, so one glitched
|
||||
forward sample cannot drag the confirmation horizon years ahead; a close
|
||||
never lands after the closing observation's own clock).
|
||||
Unlike the earlier single moving opt-in date, closing and reopening does NOT
|
||||
discard the still-undelivered backlog from a previous consented window —
|
||||
those packages stay inside their own interval and remain eligible.
|
||||
|
||||
One deliberate upgrade-path consequence: packages exported under the
|
||||
pre-interval consent model (before `send_consent_windows` existed) predate
|
||||
the first recorded window and are therefore never transmitted after an
|
||||
upgrade. This is the fail-closed direction — re-importing the old moving
|
||||
day-stamp to release them would re-import the semantics five review rounds
|
||||
showed to be unsound — and it costs at most the undelivered backlog, never
|
||||
collected data.
|
||||
|
||||
### A.5 Retention
|
||||
|
||||
- **Local:** unchanged — 30 days for successfully exported history, and pending
|
||||
deltas are kept until exported. Send state does **not** extend local
|
||||
retention: a package that could never be sent is still pruned at 30 days.
|
||||
Unbounded local growth against a permanently unreachable endpoint is a worse
|
||||
failure than losing metrics from an install that has been broken for a month.
|
||||
- **Remote:** raw packages are retained in S3 without expiry in production and
|
||||
for 30 days in staging.
|
||||
|
||||
### A.6 Deletion
|
||||
|
||||
There is no remote deletion path in the v1 contract, and this appendix does not
|
||||
invent one. What a user can do:
|
||||
|
||||
| Action | Effect |
|
||||
|---|---|
|
||||
| `send: false` | No further packages leave the machine |
|
||||
| `enabled: false` | Collection stops; existing local state remains |
|
||||
| Remove `.../shared_metrics` | Local identity, aggregates, and files reset; future sends use a new install_id |
|
||||
| Delete already-sent data | Not self-service — requires an operator acting on the S3 bucket |
|
||||
|
||||
If a deletion-on-request obligation is ever taken on, the lookup path is now
|
||||
direct: the user's `install_id` (readable from their local store) is the key
|
||||
their data is stored under. Building the service-side delete API remains a
|
||||
new product decision, not an implementation detail.
|
||||
|
||||
### A.7 What the outbox directory is
|
||||
|
||||
Recorded because it was misread once during Phase 2 planning, in a way that
|
||||
would have deleted user data.
|
||||
|
||||
The directory is **local history, not a send-queue**. `package_outbox` is the
|
||||
SQLite table; its `exported_at` column means "written to disk", not "sent".
|
||||
Files are immutable and pruned **by age alone**.
|
||||
|
||||
The ingest contract says senders should delete a package from their outbox on
|
||||
`202`. **The exporter does not do this.** Deleting on acknowledgement would
|
||||
repurpose the user's 30-day local history as a transmission queue and destroy
|
||||
state they were promised. Send state lives in new columns on the
|
||||
`package_outbox` table instead; the files are untouched by transmission.
|
||||
|
||||
### A.8 Scope note
|
||||
|
||||
The `install_id` field inside the package body is transmitted as the
|
||||
generator wrote it (rewritten from the row's frozen `sent_install_id`, which
|
||||
records the same value). No other payload field changes, nothing is added,
|
||||
and the service treats the whole body as opaque. Payload schema evolution
|
||||
therefore stays a sender-side concern, as before.
|
||||
Reference in New Issue
Block a user