yilsem/aiturk-hermes-ide
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license

Browse Source
This commit is contained in:
yilsem
2026-09-05 13:26:46 +03:00
commit 03634b1ca3
11340 changed files with 3442369 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evals/browser_use/README.md
+114
View File
@@ -0,0 +1,114 @@
# Browser Use Mode Benchmark
The A/B battery behind PR [#81958](https://github.com/NousResearch/hermes-agent/pull/81958)
(Browser Use CLI 3.0 mode, salvage of #66476 by @laithrw): built-in
`browser_*` toolset vs the single `browser_exec` driver, measured as total
task tokens / tool calls / wall clock at accuracy parity on live multi-step
web tasks.
## Design
- **Arms differ only by tree + config.** `base` runs the built-in twelve
`browser_*` tools from a merge-base checkout; `pr` runs `browser_exec`
(`browser.backend: browser-use`) from the branch checkout; `prns` is `pr`
with the schema's helpers digest stripped to the header (isolates the
digest's value). Each cell gets a throwaway `HERMES_HOME`; web-fetch
credentials are stripped so every arm must actually drive the browser.
- **Tasks are oracle-checked.** toscrape-family sites (stable content, no
anti-bot), regex oracles over the final answer. `tasks/easy.json` (5 tasks:
price lookup, category extract, count/aggregate, login, pagination) and
`tasks/hard.json` (6 tasks: full-category multi-page crawls, five-star
rating aggregation, JS/delayed render, login chain, cross-category
compare).
- **Resume-safe.** Completed cells in `results/*.jsonl` are skipped on rerun
(same pattern as `scripts/toolperf_abeval`).
- **Backend matrix.** `orchestrate.py` drives a local headless-Chrome CDP;
`orchestrate_cloud.py --backend nous-cloud|browserbase` provisions a real
cloud browser per cell through the same provider plumbing the product uses.
## Run
```bash
# arms are pinned checkouts — e.g. merge-base worktree vs your branch
export BUBENCH_BASE_TREE=/path/to/merge-base-tree
export BUBENCH_PR_TREE=/path/to/branch-tree
```
Note: since #81958 merged (and #85170 made Browser Use the default driver),
a current-main checkout resolves to `browser_exec` in BOTH arms. The `base`
arm only measures the built-in `browser_*` toolset when `BUBENCH_BASE_TREE`
is pinned to a pre-#81958 tree (the original run used the PR's merge-base
worktree). For future A/Bs of new browser changes, pin `base` to the
merge-base of the change under test — the arms are generic.
```bash
google-chrome --headless=new --remote-debugging-port=9333 \
--user-data-dir=/tmp/bubench-chrome --no-first-run --disable-gpu about:blank &
python3 orchestrate.py --tasks tasks/hard.json --reps 3 # 108 cells @ 2 models x 3 arms
python3 report.py results/results.jsonl
```
## Baseline scorecard (Aug 8-10 2026, the #81958 run — 204 cells total)
**Hard-task battery, local Chrome CDP** (6 tasks x 3 reps per cell; final
corrected-oracle readout, nothing excluded):
```
model arm ok tok_mean tok_med calls wall_s vs base tok
opus4.8 base 18/18 64594 63776 4.1 25.2 —
opus4.8 pr 18/18 25934 25030 2.0 17.5 -60%
opus4.8 prns 18/18 25578 27934 3.2 23.7 -60%
kimi-k3 base 18/18 56464 53276 5.3 50.0 —
kimi-k3 pr 18/18 19230 16710 2.4 33.3 -66%
kimi-k3 prns 18/18 23099 21160 4.1 50.5 -59%
```
Digest ablation: pr (with helpers digest) 36/36 ok, mean 22,582 tok; prns
(header-only) 36/36 ok, mean 24,339 tok — the pinned 3.4KB digest costs
nothing and saves a little; the full 11KB live skill dump adds nothing.
**Backend matrix** (pr arm, same tasks):
```
model backend ok tok_mean calls wall
opus4.8 local-cdp 17/18 25934 2.0 17.5
opus4.8 nous-cloud 12/12 33330 2.8 33.8
opus4.8 browserbase 6/6 26712 2.2 23.2
kimi-k3 local-cdp 18/18 19230 2.4 33.3
kimi-k3 nous-cloud 12/12 22050 2.9 41.4
kimi-k3 browserbase 6/6 22121 2.8 35.2
```
**Easy battery, round 1** (5 tasks x 3 reps, sonnet-5 + qwen3-coder-30b;
after excluding provider-noise runs — raw chat-template XML, 0 tool calls):
```
model arm ok prompt compl total calls wall_s
claude-sonnet-5 base 15/15 39771 324 40095 2.7 16.5
claude-sonnet-5 pr 15/15 27482 509 27991 2.4 14.3
qwen3-coder-30b base 13/14 59509 559 60068 5.7 21.5
qwen3-coder-30b pr 10/11 57146 1616 58763 6.8 26.3
```
sonnet-5: −30% tokens at parity. qwen3-30b: a wash — weak coders burn the
savings retrying exec code. The token win concentrates on multi-step tasks
and grows with task hardness; strong models also finish in fewer tool calls.
Compatibility probes from the same run: Firecrawl cloud browsers attach fine
(CDP websocket); Camofox has no CDP surface — structurally incompatible,
hence the automatic fallback to the built-in toolset in #81958.
Caveats: toscrape-family sites (no anti-bot, no heavy SPA); n<=3 per cell;
success-rate deltas at this n are noise — audit sub-100% cells run-by-run
before calling a regression.
## Provenance
The original per-run `results*.jsonl` files lived in `/tmp/bu-bench/` (tmpfs)
and were lost in a host reboot on Aug 12 2026. The harness, task definitions,
and aggregate readouts in this directory were recovered verbatim from the
session transcripts of the benchmark run (session `20260808_050008_5f615e`
tool-call history); `single_run.py`/`orchestrate*.py` are the recovered
scripts with the hardcoded `/tmp/bu-bench` paths parameterized. Rerunning the
battery reproduces fresh per-run data.
evals/browser_use/orchestrate.py
+129
View File
@@ -0,0 +1,129 @@
"""Local-CDP battery orchestrator: tasks x arms x models x reps.
Resume-safe: completed cells in results.jsonl are skipped, so a killed
battery continues where it left off (same pattern as scripts/toolperf_abeval).
Usage:
# start a headless Chrome first:
# google-chrome --headless=new --remote-debugging-port=9333 \
# --user-data-dir=/tmp/bubench-chrome --no-first-run --disable-gpu about:blank
BUBENCH_BASE_TREE=... BUBENCH_PR_TREE=... BENCH_CDP_URL=http://127.0.0.1:9333 \
python3 orchestrate.py [--tasks tasks/hard.json] [--models m1,m2] \
[--arms base,pr,prns] [--reps 3]
"""
import argparse
import itertools
import json
import os
import subprocess
import sys
import time
ROOT = os.environ.get("BUBENCH_ROOT", os.path.dirname(os.path.abspath(__file__)))
PY = sys.executable
ENV = {**os.environ}
ENV["PATH"] = os.path.expanduser("~/.local/bin") + os.pathsep + ENV.get("PATH", "")
parser = argparse.ArgumentParser()
parser.add_argument("--tasks", default=os.path.join(ROOT, "tasks", "hard.json"))
parser.add_argument("--models", default="anthropic/claude-opus-4.8,moonshotai/kimi-k3")
parser.add_argument("--arms", default="base,pr,prns")
parser.add_argument("--reps", type=int, default=3)
parser.add_argument("--results", default=os.path.join(ROOT, "results", "results.jsonl"))
parser.add_argument("--run-timeout", type=int, default=1200)
args = parser.parse_args()
os.makedirs(os.path.dirname(args.results), exist_ok=True)
ARMS = args.arms.split(",")
MODELS = args.models.split(",")
TASKS = list(json.load(open(args.tasks, encoding="utf-8")).keys())
REPS = list(range(1, args.reps + 1))
done = set()
if os.path.exists(args.results):
for line in open(args.results, encoding="utf-8"):
try:
r = json.loads(line)
done.add((r["arm"], r["task"], r["model"], r["rep"]))
except Exception:
pass
def reset_browser_state():
"""Kill lingering drivers and clear cookies between cells."""
if sys.platform == "win32":
subprocess.run(
["taskkill", "/F", "/IM", "agent-browser.exe", "/T"],
capture_output=True,
)
else:
subprocess.run(["pkill", "-f", "agent-browser"], capture_output=True)
code = "cdp('Network.clearBrowserCookies')\nprint('cleared')\n"
try:
subprocess.run(
["browser-use"],
input=code,
text=True,
capture_output=True,
timeout=120,
env=ENV,
)
except Exception:
pass
cells = [
(arm, task, model, rep)
for model, task, rep, arm in itertools.product(MODELS, TASKS, REPS, ARMS)
]
total = len(cells)
n = 0
for arm, task, model, rep in cells:
n += 1
if (arm, task, model, rep) in done:
continue
print(f"[{n}/{total}] {arm} {task} {model} rep{rep}", flush=True)
reset_browser_state()
t0 = time.time()
try:
proc = subprocess.run(
[PY, os.path.join(ROOT, "single_run.py"), arm, task, model, str(rep)],
capture_output=True,
text=True,
timeout=args.run_timeout,
env={**ENV, "BUBENCH_TASKS": args.tasks},
)
rec = None
for line in (proc.stdout or "").splitlines():
if line.startswith("RESULT_JSON:"):
rec = json.loads(line[len("RESULT_JSON:") :])
if rec is None:
rec = {
"arm": arm,
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": "no-result",
"stderr_tail": (proc.stderr or "")[-800:],
"stdout_tail": (proc.stdout or "")[-400:],
}
except subprocess.TimeoutExpired:
rec = {
"arm": arm,
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": f"orchestrator-timeout-{args.run_timeout}s",
}
rec["cell_wall_s"] = round(time.time() - t0, 1)
with open(args.results, "a", encoding="utf-8") as f:
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
print(
f" -> ok={rec.get('ok')} err={rec.get('error')} wall={rec.get('cell_wall_s')}s",
flush=True,
)
print("BATTERY COMPLETE", flush=True)
evals/browser_use/orchestrate_cloud.py
+184
View File
@@ -0,0 +1,184 @@
"""Cloud-backend battery: pr arm attached to a provisioned cloud browser per cell.
Backends:
nous-cloud - Nous Portal-provisioned Browser Use cloud browser
(plugins.browser.browser_use provider; needs gateway access)
browserbase - fresh Browserbase session per cell
(needs BROWSERBASE_API_KEY + BROWSERBASE_PROJECT_ID)
Usage:
BUBENCH_BASE_TREE=... BUBENCH_PR_TREE=... \
python3 orchestrate_cloud.py --backend nous-cloud [--reps 2]
python3 orchestrate_cloud.py --backend browserbase [--reps 1]
Per cell: provision a session, export its CDP endpoint via BENCH_CDP_URL /
BU_CDP_WS, run single_run.py (pr arm), close the session. Resume-safe.
"""
import argparse
import itertools
import json
import os
import subprocess
import sys
import time
import urllib.request
ROOT = os.environ.get("BUBENCH_ROOT", os.path.dirname(os.path.abspath(__file__)))
PY = sys.executable
ENV_BASE = {**os.environ}
ENV_BASE["PATH"] = (
os.path.expanduser("~/.local/bin") + os.pathsep + ENV_BASE.get("PATH", "")
)
parser = argparse.ArgumentParser()
parser.add_argument("--backend", required=True, choices=["nous-cloud", "browserbase"])
parser.add_argument("--tasks", default=os.path.join(ROOT, "tasks", "hard.json"))
parser.add_argument("--models", default="anthropic/claude-opus-4.8,moonshotai/kimi-k3")
parser.add_argument("--reps", type=int, default=2)
parser.add_argument("--results", default=None)
parser.add_argument("--run-timeout", type=int, default=1200)
args = parser.parse_args()
RESULTS = args.results or os.path.join(ROOT, "results", f"results_{args.backend}.jsonl")
os.makedirs(os.path.dirname(RESULTS), exist_ok=True)
MODELS = args.models.split(",")
TASKS = list(json.load(open(args.tasks, encoding="utf-8")).keys())
REPS = list(range(1, args.reps + 1))
done = set()
if os.path.exists(RESULTS):
for line in open(RESULTS, encoding="utf-8"):
try:
r = json.loads(line)
done.add((r["task"], r["model"], r["rep"]))
except Exception:
pass
class NousCloud:
def __init__(self):
sys.path.insert(0, os.environ["BUBENCH_PR_TREE"])
import importlib
self._mod = importlib.import_module("plugins.browser.browser_use.provider")
def create(self, name):
self._prov = self._mod.BrowserUseBrowserProvider()
sess = self._prov.create_session(name)
return sess, {"BENCH_CDP_URL": sess["cdp_url"]}
def close(self, sess):
self._prov.close_session(
sess.get("bb_session_id") or sess.get("session_name", "")
)
class Browserbase:
def create(self, name):
req = urllib.request.Request(
"https://api.browserbase.com/v1/sessions",
data=json.dumps({
"projectId": os.environ["BROWSERBASE_PROJECT_ID"]
}).encode(),
headers={
"x-bb-api-key": os.environ["BROWSERBASE_API_KEY"],
"Content-Type": "application/json",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=30) as resp:
sess = json.load(resp)
return sess, {
"BU_CDP_WS": sess["connectUrl"],
"BENCH_CDP_URL": sess["connectUrl"],
}
def close(self, sess):
req = urllib.request.Request(
f"https://api.browserbase.com/v1/sessions/{sess['id']}",
data=json.dumps({
"projectId": os.environ["BROWSERBASE_PROJECT_ID"],
"status": "REQUEST_RELEASE",
}).encode(),
headers={
"x-bb-api-key": os.environ["BROWSERBASE_API_KEY"],
"Content-Type": "application/json",
},
method="POST",
)
urllib.request.urlopen(req, timeout=30)
provider = NousCloud() if args.backend == "nous-cloud" else Browserbase()
cells = [(t, m, rep) for m, t, rep in itertools.product(MODELS, TASKS, REPS)]
total = len(cells)
n = 0
for task, model, rep in cells:
n += 1
if (task, model, rep) in done:
continue
print(f"[{n}/{total}] {args.backend} pr {task} {model} rep{rep}", flush=True)
sess = None
t0 = time.time()
try:
sess, extra_env = provider.create(f"bubench-{task}-{rep}")
except Exception as e: # noqa: BLE001
rec = {
"arm": f"pr-{args.backend}",
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": f"session-create: {e}",
}
with open(RESULTS, "a", encoding="utf-8") as f:
f.write(json.dumps(rec) + "\n")
continue
env = {**ENV_BASE, **extra_env, "BUBENCH_TASKS": args.tasks}
subprocess.run(["pkill", "-f", "browser_harness"], capture_output=True)
try:
proc = subprocess.run(
[PY, os.path.join(ROOT, "single_run.py"), "pr", task, model, str(rep)],
capture_output=True,
text=True,
timeout=args.run_timeout,
env=env,
)
rec = None
for line in (proc.stdout or "").splitlines():
if line.startswith("RESULT_JSON:"):
rec = json.loads(line[len("RESULT_JSON:") :])
if rec is None:
rec = {
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": "no-result",
"stderr_tail": (proc.stderr or "")[-600:],
}
except subprocess.TimeoutExpired:
rec = {
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": f"timeout-{args.run_timeout}s",
}
finally:
try:
provider.close(sess)
except Exception as e: # noqa: BLE001
print(f" close warning: {e}", flush=True)
rec["arm"] = f"pr-{args.backend}"
rec["cell_wall_s"] = round(time.time() - t0, 1)
with open(RESULTS, "a", encoding="utf-8") as f:
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
print(
f" -> ok={rec.get('ok')} err={rec.get('error')} wall={rec.get('cell_wall_s')}s",
flush=True,
)
print("CLOUD BATTERY COMPLETE", flush=True)
evals/browser_use/report.py
+70
View File
@@ -0,0 +1,70 @@
"""Aggregate results.jsonl into the scorecard tables.
Usage:
python3 report.py [results/results.jsonl ...]
Groups by (model, arm): ok-rate, token mean/median, tool calls, wall clock,
and token delta vs the ``base`` arm of the same model when present.
"""
import json
import statistics
import sys
from collections import defaultdict
def main(paths):
rows = []
for p in paths:
for line in open(p, encoding="utf-8"):
try:
rows.append(json.loads(line))
except Exception:
pass
if not rows:
print("no rows")
return
cells = defaultdict(list)
for r in rows:
cells[(r.get("model", "?"), r.get("arm", "?"))].append(r)
base_tok = {}
for (model, arm), rs in cells.items():
if arm == "base":
oks = [r for r in rs if r.get("ok")]
if oks:
base_tok[model] = statistics.mean(r.get("total_tokens", 0) for r in oks)
hdr = f"{'model':<34} {'arm':<16} {'ok':>7} {'tok_mean':>9} {'tok_med':>8} {'calls':>6} {'wall_s':>7} {'vs base':>8}"
print(hdr)
print("-" * len(hdr))
for model, arm in sorted(cells):
rs = cells[(model, arm)]
oks = [r for r in rs if r.get("ok")]
n_ok, n = len(oks), len(rs)
toks = [r.get("total_tokens", 0) for r in oks]
calls = [r.get("tool_calls", 0) for r in oks]
walls = [r.get("wall_s", 0) for r in oks]
tok_mean = statistics.mean(toks) if toks else 0
delta = ""
if arm != "base" and model in base_tok and tok_mean:
delta = f"{(tok_mean - base_tok[model]) / base_tok[model] * 100:+.0f}%"
print(
f"{model:<34} {arm:<16} {n_ok:>3}/{n:<3} {tok_mean:>9.0f} "
f"{statistics.median(toks) if toks else 0:>8.0f} "
f"{statistics.mean(calls) if calls else 0:>6.1f} "
f"{statistics.mean(walls) if walls else 0:>7.1f} {delta:>8}"
)
errs = [r for r in rows if r.get("error")]
if errs:
print(f"\nerrors: {len(errs)}")
for r in errs[:10]:
print(
f" {r.get('model')}/{r.get('arm')}/{r.get('task')}/rep{r.get('rep')}: {r['error'][:120]}"
)
if __name__ == "__main__":
main(sys.argv[1:] or ["results/results.jsonl"])
evals/browser_use/results/.gitignore
+1
View File
@@ -0,0 +1 @@
*.jsonl
evals/browser_use/single_run.py
+198
View File
@@ -0,0 +1,198 @@
"""One benchmark cell: task x arm x model x rep.
Usage:
python3 single_run.py <arm> <task_key> <model> <rep>
Arms:
base - built-in ``browser_*`` toolset (twelve tools), pinned tree $BUBENCH_BASE_TREE
pr - Browser Use CLI mode (single ``browser_exec`` tool), pinned tree $BUBENCH_PR_TREE
prns - same as pr but with the schema description stripped to the header only
(isolates the value of the helpers digest in the tool description)
Environment:
BUBENCH_ROOT workspace dir (default: dir containing this script)
BUBENCH_BASE_TREE checkout used for the ``base`` arm (e.g. a merge-base worktree)
BUBENCH_PR_TREE checkout used for the ``pr``/``prns`` arms
BUBENCH_TASKS tasks json (default: $BUBENCH_ROOT/tasks/hard.json)
BENCH_CDP_URL CDP endpoint both arms drive (default http://127.0.0.1:9333)
OPENROUTER_API_KEY provider credential for the runs
The run gets a throwaway HERMES_HOME so no local config leaks in, and the
web-fetch credential env vars are stripped so every arm must actually drive
the browser (no web_extract shortcuts).
Prints one line: ``RESULT_JSON:{...}`` consumed by orchestrate.py.
"""
import json
import os
import re
import sys
import tempfile
import time
ARM, TASK_KEY, MODEL, REP = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
ROOT = os.environ.get("BUBENCH_ROOT", os.path.dirname(os.path.abspath(__file__)))
BASE_TREE = os.environ["BUBENCH_BASE_TREE"]
PR_TREE = os.environ["BUBENCH_PR_TREE"]
WT = {"base": BASE_TREE, "pr": PR_TREE, "prns": PR_TREE}[ARM]
TASKS_PATH = os.environ.get("BUBENCH_TASKS", os.path.join(ROOT, "tasks", "hard.json"))
TASKS = json.load(open(TASKS_PATH, encoding="utf-8"))
task = TASKS[TASK_KEY]
home = tempfile.mkdtemp(prefix=f"buhome-{ARM}-")
hh = os.path.join(home, ".hermes")
os.makedirs(os.path.join(hh, "logs"), exist_ok=True)
cdp = os.environ.get("BENCH_CDP_URL", "http://127.0.0.1:9333")
browser_cfg = (
{"cloud_provider": "local", "cdp_url": cdp}
if ARM == "base"
else {"backend": "browser-use"}
)
cfg = {
"model": {"provider": "openrouter", "default": MODEL},
"browser": browser_cfg,
"display": {"quiet": True},
}
import yaml
with open(os.path.join(hh, "config.yaml"), "w", encoding="utf-8") as f:
yaml.safe_dump(cfg, f)
os.environ["HERMES_HOME"] = hh
# Strip web-fetch shortcuts: every arm must drive the browser.
os.environ.pop("BROWSER_USE_API_KEY", None)
for k in ("FIRECRAWL_API_KEY", "NOUS_API_KEY", "TAVILY_API_KEY", "SERPER_API_KEY"):
os.environ.pop(k, None)
os.environ["BU_CDP_URL"] = cdp
os.environ["PATH"] = (
os.path.expanduser("~/.local/bin") + os.pathsep + os.environ.get("PATH", "")
)
sys.path.insert(0, WT)
import logging
logging.disable(logging.CRITICAL)
import run_agent # noqa: E402
_loaded = os.path.normcase(os.path.normpath(run_agent.__file__))
_want = os.path.normcase(os.path.normpath(WT))
assert _loaded.startswith(_want), f"wrong tree: {run_agent.__file__}"
if ARM == "prns":
# Strip the helpers digest from the schema: header-only description.
import tools.browser_use_cli as bu # noqa: E402
bu._skill_text_fetched = True
bu._skill_text_cache = None
bu.BROWSER_EXEC_SCHEMA["description"] = bu._description_header()
from run_agent import AIAgent # noqa: E402
# Provider resolution: default openrouter (original battery), but allow the
# Nous-subscription path on boxes without an OpenRouter key. Credentials are
# resolved through the product's own auth state, never printed.
_or_key = os.environ.get("OPENROUTER_API_KEY", "").strip()
if _or_key:
_agent_auth = dict(
base_url="https://openrouter.ai/api/v1",
api_key=_or_key,
provider="openrouter",
)
else:
# Resolved by the orchestrator BEFORE HERMES_HOME is redirected to the
# throwaway home (auth state lives in the real profile). Never printed.
_tok = os.environ.get("BUBENCH_NOUS_TOKEN", "").strip()
if not _tok:
raise SystemExit("no OPENROUTER_API_KEY and no Nous auth available")
_agent_auth = dict(
base_url=os.environ.get("BUBENCH_NOUS_BASE_URL", "https://inference-api.nousresearch.com/v1"),
api_key=_tok,
provider="nous",
)
agent = AIAgent(
**_agent_auth,
model=MODEL,
max_iterations=30,
quiet_mode=True,
skip_context_files=True,
skip_memory=True,
# NB: "terminal" must be present for the pr arms — since #81958's terminal
# gate, browser_exec is stripped from sessions whose toolsets exclude
# terminal. Both arms get the same toolsets for parity; audit
# tool_call_names in the results for terminal-tool bypasses (curl etc.).
enabled_toolsets=["browser", "terminal"],
save_trajectories=False,
)
schema_desc_len = 0
try:
from model_tools import get_tool_definitions
for t in get_tool_definitions(agent.enabled_toolsets):
if t["function"]["name"].startswith("browser"):
schema_desc_len += len(json.dumps(t["function"]))
except Exception:
pass
t0 = time.time()
error = None
final = ""
messages = []
try:
result = agent.run_conversation(task["prompt"])
final = (
(result.get("final_response") or "")
if isinstance(result, dict)
else str(result)
)
messages = result.get("messages", []) if isinstance(result, dict) else []
except Exception as e: # noqa: BLE001
error = f"{type(e).__name__}: {e}"
messages = getattr(agent, "messages", []) or []
wall = time.time() - t0
tool_calls = []
for m in messages:
if isinstance(m, dict) and m.get("role") == "assistant":
for tc in m.get("tool_calls") or []:
fn = (
(tc.get("function") or {}).get("name") if isinstance(tc, dict) else None
)
if fn:
tool_calls.append(fn)
def _ok(text: str) -> bool:
if task.get("oracle_all"):
return all(
re.search(re.escape(x), text, re.IGNORECASE) for x in task["oracle_all"]
)
return any(
re.search(re.escape(x), text, re.IGNORECASE) for x in task.get("oracle_any", [])
)
out = {
"arm": ARM,
"task": TASK_KEY,
"model": MODEL,
"rep": int(REP),
"ok": bool(final) and _ok(final) and error is None,
"wall_s": round(wall, 1),
"prompt_tokens": getattr(agent, "session_prompt_tokens", 0),
"completion_tokens": getattr(agent, "session_completion_tokens", 0),
"total_tokens": getattr(agent, "session_total_tokens", 0),
"api_calls": len([
m for m in messages if isinstance(m, dict) and m.get("role") == "assistant"
]),
"tool_calls": len(tool_calls),
"tool_call_names": tool_calls,
"browser_schema_chars": schema_desc_len,
"error": error,
"final_snippet": (final or "")[-400:],
}
print("RESULT_JSON:" + json.dumps(out, ensure_ascii=False))
evals/browser_use/tasks/easy.json
+22
View File
@@ -0,0 +1,22 @@
{
"simple_price": {
"prompt": "Using your browser tools, open https://books.toscrape.com , find the book 'Sharp Objects' and tell me its listed price. Give the exact price in your final answer.",
"oracle_any": ["47.82"]
},
"multi_extract": {
"prompt": "Using your browser tools, go to https://books.toscrape.com , open the 'Travel' category, and list EVERY book in that category with its price. In your final answer state the total count of books and the title and price of the cheapest one.",
"oracle_all": ["11", "23.21"]
},
"count_aggregate": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com and determine which author appears most often on page 1 and how many quotes they have there. Final answer must name the author and the count.",
"oracle_all": ["Einstein", "3"]
},
"form_login": {
"prompt": "Using your browser tools, go to https://quotes.toscrape.com/login , log in with username 'admin' and password 'admin', and tell me the text of the link that replaced 'Login' in the top navigation after logging in.",
"oracle_any": ["Logout", "logout"]
},
"pagination_nav": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com , navigate to page 2, and tell me the author of the FIRST quote on page 2.",
"oracle_any": ["Marilyn Monroe"]
}
}
evals/browser_use/tasks/hard.json
+26
View File
@@ -0,0 +1,26 @@
{
"multi_page_filter": {
"prompt": "Using your browser tools, open https://books.toscrape.com , go to the 'Mystery' category, and go through ALL of its pages. Count how many Mystery books cost less than \u00a330.00, and identify the single cheapest Mystery book. Final answer must state the count and the cheapest book's exact title and price.",
"oracle_all": ["19", "Tastes Like Fear", "10.69"]
},
"rating_aggregate": {
"prompt": "Using your browser tools, open https://books.toscrape.com , browse the ENTIRE 'Mystery' category (all pages). Find every book with a five-star rating, then compute the average price of those five-star books. Final answer must state how many five-star Mystery books exist and the average price rounded to 2 decimals.",
"oracle_all": ["5", "32.27"]
},
"js_rendered": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com/js/ (the content on this page is rendered by JavaScript). Tell me the author of the LAST quote on page 1. Final answer must name the author.",
"oracle_any": ["Steve Martin"]
},
"delayed_render": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com/js-delayed/ (content appears only after a delayed JavaScript render). Tell me the author of the FIRST quote on the page. Final answer must name the author.",
"oracle_any": ["Albert Einstein"]
},
"login_then_extract": {
"prompt": "Using your browser tools: (1) log in at https://quotes.toscrape.com/login with username 'admin' and password 'admin' and confirm the login worked, (2) then navigate to the tag page for the tag 'books' and report how many quotes are shown there and which authors appear more than once on that page. Final answer must state the quote count and name every author who appears twice.",
"oracle_all": ["10", "Mark Twain", "Jane Austen"]
},
"cross_category_compare": {
"prompt": "Using your browser tools, open https://books.toscrape.com and compare the 'Mystery' and 'Historical Fiction' categories across ALL their pages: which category contains the single most expensive book overall? Final answer must name that category, the book's title, and its exact price.",
"oracle_all": ["Historical Fiction", "Sara de Vos", "55.55"]
}
}
evals/compaction/README.md
+82
View File
@@ -0,0 +1,82 @@
# Compaction Eval Harness
Measures what context compaction actually costs in *recall*, not just tokens.
## What it does
1. Takes a real long transcript (JSON: `{"messages": [...]}`, chat format).
2. Generates a bank of factual recall questions from the region that
compaction will summarize away (cached per transcript for reproducibility).
3. Runs the transcript through `ContextCompressor.compress()` under each
policy in the matrix (current default, aggressive tail, codex-style, ...).
4. For each policy, asks a fresh LLM the recall questions with ONLY the
post-compaction context, and judges answers against gold.
5. Emits a scorecard: recall accuracy vs tokens retained, per policy.
## Usage
```bash
# from repo root, venv active
python evals/compaction/runner.py \
--transcript /path/to/lineage.json \
--policies current,aggressive,floor10k \
--questions 15 \
--out evals/compaction/results/run1
python evals/compaction/report.py evals/compaction/results/run1
```
Transcripts are NOT committed (they contain real session data). Point
`--transcript` at a local file. See `fixtures.py` for the expected shape and
a synthetic-transcript generator used by CI smoke tests.
## Building transcripts from real sessions (`scripts/`)
Compaction rotations mean a single active session rarely exceeds ~300K
tokens, but the *lineage* (parent→children chain) carries the full
uncompacted history. The scripts reconstruct those into eval transcripts:
```bash
# 1. ALWAYS copy the DB first — never point at the live state.db
cp ~/.hermes/state.db /tmp/state_copy.db
# 2. Find big lineages (sessions with parent_session_id form chains), then:
python evals/compaction/scripts/reconstruct_lineage.py \
/tmp/state_copy.db <root_session_id> /tmp/lineage.json
# 3. (optional) Replay a 500K prefix through one checkout's compressor and
# dump before/after for the HTML viewer:
python evals/compaction/scripts/replay_lineage.py <checkout> /tmp/lineage.json out.json 500000
python evals/compaction/scripts/build_html_report.py <runs_dir> report.html
```
`reconstruct_lineage.py` walks the whole descendant tree chronologically,
dedupes rotation-copied rows by content hash, strips synthetic compaction
artifacts (summaries, todo snapshots), and resolves the system prompt through
the `system_prompts` dedup table (sessions only carry a hash). The HTML
report renders before/after transcripts side by side with compaction
artifacts color-coded.
## Region-scoping tripwire
`test_region_scoping.py` plants sentinels in head/middle/tail and asserts the
summarizer's serialized-turns input carries ONLY the middle (compacted)
region in both legacy and lean modes. Run it directly or via pytest.
## Policies
Defined in `policies.py`. Each policy maps to `ContextCompressor` constructor
kwargs plus optional attribute overrides applied post-construction (e.g.
`tail_token_budget`). Add new policies there — the runner picks them up by
name.
## Notes
- Question generation and judging use `agent.auxiliary_client.call_llm`
(same transport the compressor uses), so the harness needs a configured
provider. Costs real tokens: ~(policies x questions) answer calls plus
one generation and one judge pass.
- Accuracy is judged 2/1/0 (correct / partial / wrong); the scorecard
reports normalized percent. The judge sees gold answers, the answerer
does not.
- `--also-uncompacted` adds a control arm that answers from the full
original transcript — the recall ceiling.
evals/compaction/fixtures.py
+79
View File
@@ -0,0 +1,79 @@
"""Transcript fixtures for the compaction eval harness.
Real transcripts are supplied by path (never committed). This module loads
them, estimates tokens the same way the harness scores them, and can generate
a small synthetic transcript so CI smoke tests run without real data.
"""
from __future__ import annotations
import json
import random
from typing import Any, Dict, List
def estimate_tokens(msg: Dict[str, Any]) -> int:
"""Chars/4 estimate, matching the harness's scoring convention."""
total = len(msg.get("content") or "") if isinstance(msg.get("content"), str) else 0
tc = msg.get("tool_calls")
if tc:
total += len(json.dumps(tc, default=str))
return total // 4
def total_tokens(messages: List[Dict[str, Any]]) -> int:
return sum(estimate_tokens(m) for m in messages)
def load_transcript(path: str, cap_tokens: int | None = None) -> List[Dict[str, Any]]:
"""Load a transcript JSON ({"messages": [...]}) and optionally cap it.
The cap takes the chronological prefix, then drops trailing assistant
tool_calls whose results were cut off so the input is well-formed.
"""
data = json.load(open(path, encoding="utf-8"))
msgs = data["messages"] if isinstance(data, dict) else data
if cap_tokens is None:
return msgs
prefix: List[Dict[str, Any]] = []
running = 0
for m in msgs:
t = estimate_tokens(m)
if running + t > cap_tokens and len(prefix) > 10:
break
prefix.append(m)
running += t
while prefix and prefix[-1].get("tool_calls"):
prefix.pop()
return prefix
def synthetic_transcript(n_turns: int = 60, seed: int = 7) -> List[Dict[str, Any]]:
"""Deterministic fake transcript with plantable facts for smoke tests.
Every 10th turn plants a distinctive fact ("The deploy code for region
N is XYZ") so smoke tests can assert recall mechanics without an LLM.
"""
rng = random.Random(seed)
msgs: List[Dict[str, Any]] = [
{"role": "system", "content": "You are a test agent."},
{"role": "user", "content": "Work through the checklist and remember the codes."},
]
for i in range(n_turns):
fact = ""
if i % 10 == 0:
fact = f" The deploy code for region {i // 10} is Z{rng.randint(1000, 9999)}."
msgs.append({
"role": "assistant",
"content": f"Working on step {i}.{fact}",
"tool_calls": [{
"id": f"c{i}",
"function": {"name": "terminal", "arguments": json.dumps({"command": f"echo step {i}"})},
}],
})
msgs.append({
"role": "tool",
"tool_call_id": f"c{i}",
"content": ("step output " * 200) + f"result-{i}",
})
msgs.append({"role": "assistant", "content": "Checklist complete."})
return msgs
evals/compaction/policies.py
+54
View File
@@ -0,0 +1,54 @@
"""Compaction policy matrix.
Each policy is a name -> spec mapping. A spec has:
ctor: extra kwargs for ContextCompressor(...)
attrs: attribute overrides applied after construction (lets us pin
tail_token_budget and other derived values without touching the
class)
The runner constructs one compressor per policy and calls
compress(force=True) with the transcript's estimated tokens.
"""
from __future__ import annotations
from typing import Any, Dict
# Window we evaluate against (fable-5 class model).
EVAL_MODEL = "anthropic/claude-fable-5"
EVAL_WINDOW = 1_000_000
POLICIES: Dict[str, Dict[str, Any]] = {
# Shipping behavior, untouched.
"current": {
"ctor": {},
"attrs": {},
},
# Proposed: tail = max(10K, 0.025% ... interpreted as 2.5% of window)
# capped hard at 25K on a 1M model. protect_last_n stays for message-count
# floor semantics.
"tail25k": {
"ctor": {},
"attrs": {"tail_token_budget": 25_000},
},
# Hard floor variant: minimum viable tail.
"tail10k": {
"ctor": {},
"attrs": {"tail_token_budget": 10_000},
},
# Codex posture: nearly no tail; summary carries everything.
"codex_style": {
"ctor": {"protect_last_n": 3},
"attrs": {"tail_token_budget": 2_000},
},
# Compaction-v2 lean mode: clamped 2.5% tail + tail tool demotion +
# verbatim user messages in summary + session_search recovery pointers.
"lean": {
"ctor": {"tail_mode": "lean"},
"attrs": {"_session_id": "eval-session"},
},
}
def apply_policy(compressor, spec: Dict[str, Any]):
for key, value in (spec.get("attrs") or {}).items():
setattr(compressor, key, value)
return compressor
evals/compaction/report.py
+43
View File
@@ -0,0 +1,43 @@
"""Render a compaction-eval scorecard as a terminal table + markdown.
Usage: python evals/compaction/report.py <results_dir>
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
def main():
out_dir = Path(sys.argv[1])
card = json.loads((out_dir / "scorecard.json").read_text(encoding="utf-8"))
card.sort(key=lambda s: -s["recall_pct"])
rows = []
for s in card:
before = s.get("before_tokens", 0)
after = s.get("after_tokens", 0)
kept = f"{100 * after / before:.1f}%" if before else "?"
rows.append((
s["policy"], f"{s['recall_pct']}%", f"{before:,}", f"{after:,}", kept,
str(s.get("compress_seconds", "-")),
))
headers = ("policy", "recall", "tokens before", "tokens after", "kept", "sec")
widths = [max(len(headers[i]), *(len(r[i]) for r in rows)) for i in range(len(headers))]
line = " ".join(h.ljust(widths[i]) for i, h in enumerate(headers))
print(line)
print("-" * len(line))
for r in rows:
print(" ".join(str(r[i]).ljust(widths[i]) for i in range(len(headers))))
md = ["| " + " | ".join(headers) + " |", "|" + "|".join("---" for _ in headers) + "|"]
for r in rows:
md.append("| " + " | ".join(r) + " |")
(out_dir / "scorecard.md").write_text("\n".join(md) + "\n", encoding="utf-8")
print(f"\nmarkdown -> {out_dir}/scorecard.md")
if __name__ == "__main__":
main()
evals/compaction/results/SCORECARD-2026-08-15.md
+357
View File
@@ -0,0 +1,357 @@
# Compaction v2 — 4-transcript scorecard (2026-08-15, anchor-index build)
Four real 500K-token lineage transcripts from state.db (sweep campaign, GUI
desktop work, PR-merge campaign, ACP/PR review), 15-question recall exam
each. "recovery" = one session_search round-trip (FTS5+BM25 sim) against the
archived region. Lean build includes: 25K clamped tail, tail tool demotion,
chunked digests (noise-filtered, pristine tool contents), mechanical anchor
index, verbatim user messages, recovery footer, upgraded summarizer prompt.
> **Historical note (2026-08-30):** the "chunked digests" arm described here
> was later replaced — the detailed session log is now produced by the SAME
> single summary request (lean compaction makes exactly one auxiliary LLM
> call per attempt; no per-chunk digest calls). See #96603.
## Results (recall % @ retained tokens)
policy sweep gui prmerge acp AVG
uncompacted 93.3 @ 500K 96.7 @ 500K 96.7 @ 500K 100.0 @ 500K 96.7
current 93.3*@ 176K 26.7*@ 156K 33.3 @ 155K 30.0 @ 160K 45.8 @ 162K
lean 40.0 @ 62K 60.0 @ 41K 23.3 @ 44K 36.7 @ 50K 40.0 @ 49K
lean+recovery 70.0 @ 62K 80.0 @ 41K 43.3 @ 45K 80.0 @ 50K 68.3 @ 49K
* sweep/gui current scores are from the previous question banks (same
transcripts; banks regenerated in the 4-way run). prmerge/acp are clean
same-bank comparisons across all arms.
## Findings
1. LEAN+RECOVERY BEATS CURRENT BY +22.5pts ON AVERAGE (68.3 vs 45.8) AT 3.3x
FEWER TOKENS (49K vs 162K). It wins on 3 of 4 transcripts and loses only
sweep — the one transcript where current's fat tail got lucky with
restated facts (93.3 is bank-inflated luck; see finding 3 of the previous
scorecard).
2. THE ANCHOR INDEX FIXED THE NEEDLE-FACT CLASS. GUI closed-book went
23.3 -> 60.0 and GUI+recovery 46.7 -> 80.0 after mechanically indexing
exact identifiers (SHAs, ids, paths, error strings) instead of trusting
the summarizer with them. ACP+recovery hit 80.0.
3. TWO FRESH TRANSCRIPTS CONFIRM CURRENT IS WEAK, NOT STRONG: 33.3 and 30.0
at ~157K retained. The original sweep 93.3 was restatement luck, not
policy quality. Current's average is 45.8% for 162K tokens — lean+recovery
is 22 points better for less than a third of the spend.
4. prmerge IS THE HARD CASE for everyone (96.7 ceiling, best policy 43.3):
1.1M-token lineage truncated at 500K, dense multi-PR state. Recovery
misses there are mostly query formulation. Headroom, not a blocker.
5. Goal check (Teknium): tail = max(10K, 2.5%) ✓; summaries scoped to the
compacted region only ✓ (sentinel tripwire test); session_search pointer ✓
(+20-43pts measured); better accuracy AND more savings than current ✓
(+22.5pts at 0.30x tokens).
## Codex CLI head-to-head (same transcripts, same exams, same judge)
Real OpenAI Codex CLI (v0.147.0, gpt-5.6-sol, 258K window) run end-to-end on
the identical four transcripts: chunk files read via `codex exec` until its
REAL auto-compaction fired (verified `compacted` event in the rollout jsonl;
peak context 455-483K), then quizzed post-compaction from memory with the
same 15-question banks and scored by the same judge.
policy sweep gui prmerge acp AVG retained state
codex (real, post-cmp) 26.7% 40.0% 43.3% 36.7% 36.7% ~4.5K (opaque blob + user msgs)
hermes current 93.3%* 26.7%* 33.3% 30.0% 45.8% ~162K
hermes lean closed-book 40.0% 60.0% 23.3% 36.7% 40.0% ~49K
hermes lean+recovery 70.0% 80.0% 43.3% 80.0% 68.3% ~49K
Notes:
- codex answers from its own post-compaction session — the honest analog of
our closed-book arms. It has NO session_search equivalent (its rollout is
on disk but the agent cannot search it at runtime), so recovery has no
codex counterpart; that gap is exactly the differentiator lean leans on.
- Apples-to-apples closed-book: lean 40.0% vs codex 36.7% — parity-plus at
10x codex's retained state but 0.30x current's. With recovery: +31.6pts
over codex.
- codex ties lean+recovery on prmerge (43.3%) — the dense multi-PR campaign
is the hardest transcript for every policy and the clearest iteration
target.
- Methodology caveats: codex ingested transcripts as FILE READS (tool
outputs), not native conversation — this matches how its compaction
treats tool output (drops it all into the server-side summary) but is not
byte-identical to a native session. Its model (gpt-5.6-sol) also differs
from the answering model in our arms; scores compare COMPACTION PIPELINES
end-to-end, not models in isolation. One codex quiz reply was also
capped short (~1K chars for 15 answers), which its terse post-compaction
style invites.
## Recommendation
Ship lean as opt-in (compression.tail_mode: lean, legacy default), harness as
the permanent gate. Iterate prmerge-class recall behind the flag (query
mining, per-epoch anchor windows) before default flip.
## Appendix: full per-transcript detail
### Transcript: sweep
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| lean | 40.0% | 499,625 | 61,567 | 114.9 |
| lean+recovery | 70.0% | 499,625 | 61,792 | 114.8 |
<details><summary>15 exam questions (questions-30b95351c7.json)</summary>
1. **What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?**
gold: `the stale file version silently deletes newer main code`
2. **According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?**
gold: `~60GB`
3. **Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?**
gold: `backgroundThrottling`
4. **What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?**
gold: `Auto merge is not allowed for this repository (enablePullRequestAutoMerge)`
5. **What is the specified 'Rule 0' that must be included in a subagent brief?**
gold: `load the skill first`
6. **In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?**
gold: `add config gate for title generation`
7. **Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?**
gold: `tests/test_packaging_metadata.py`
8. **What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?**
gold: `completed normally (exit code 0)`
9. **What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?**
gold: `relative exclude-newer makes the committed lock stale every day`
10. **How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?**
gold: `13`
11. **In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?**
gold: `-1`
12. **What is the full title block text for the technical schematic infographic generated for the Gateway Drain?**
gold: `GATEWAY DRAIN × CRON — SHUTDOWN CONTRACT`
13. **Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?**
gold: `82980`
14. **What is the specific Gist ID created for the PR infographic host in the cron cluster?**
gold: `ee33edd5804689243f974536ef7aecb9`
15. **What was the final merge SHA for Cluster D's Trigger-now PR #70638?**
gold: `f9d64b9a9d8b306f64851c1a13869d96ad5d7869`
</details>
<details><summary>15 exam questions (questions-5be475cde0.json)</summary>
1. **What exact command did the agent use to search for open issues related to a specific topic during Phase 1 of the cluster-sweep salvage?**
gold: `gh issue list --search "<topic>" --state open --limit 100 --json number,title`
2. **According to Teknium's design intent, what is the status of 'platform toolsets' in the codebase?**
gold: `platform toolsets are vestigial, never exposed`
3. **During the July sweep, which specific issue's config bridge was found to already exist at the exact line it was claimed to be missing?**
gold: `#32263`
4. **In the Aug 2026 cron-summarizer cluster sweep, which two PR numbers were discovered post-merge as the true first submitters?**
gold: `#60593, #61969`
5. **What is the recommended Git command to find when a specific symbol fix landed on the main branch?**
gold: `git log -S "<symbol>"`
6. **Why did the #39719 salvage silently delete 236 lines of code from cli-config.yaml.example?**
gold: `the stale file version silently deletes newer main code`
7. **What is the rule for salvaging commits with placeholder identities like 'pwn@example.com'?**
gold: `do NOT cherry-pick. Surgical reapply as maintainer-authored commit, Co-authored-by the GitHub PR author`
8. **How should an agent handle a 'gh pr merge' 502 error?**
gold: `retry the same command once after the "Merge already in progress" settles (~45s); check PR state between attempts`
9. **Which two properties shape almost every design decision in Hermes according to the Development Guide?**
gold: `Per-conversation prompt caching is sacred and The core is a narrow waist; capability lives at the edges.`
10. **What error message does the live-checkout git guard display when blocking a history-rewriting command?**
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module `
11. **What happened to the Desktop cluster's 'npm ci' command that resulted in an error writing to /tmp/ccH06T4r.s?**
gold: `No space left on device`
12. **What was the GraphQL API rate limit remaining for the user when the 'API rate limit already exceeded' error first occurred?**
gold: `0`
13. **Which PR was identified as the salvage of HexLab98's #85283 to fix hung inline API calls?**
gold: `#86645`
14. **Why did PR #79268 fix invisible overlays in the TUI?**
gold: `renderNodeToOutput skips boxes Yoga squeezes to height 0`
15. **What was the specific ModuleNotFoundError message caused by the wheel subpackage discovery trap in #34701?**
gold: `ModuleNotFoundError: No module named 'hermes_cli.dashboard_auth'`
</details>
### Transcript: gui
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| lean | 60.0% | 499,818 | 41,232 | 118.1 |
| lean+recovery | 80.0% | 499,818 | 41,306 | 115.2 |
<details><summary>15 exam questions (questions-36d3d87e0b.json)</summary>
1. **What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?**
gold: `#86617`
2. **According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?**
gold: `.env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.`
3. **What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?**
gold: `tests/hermes_cli/test_session_recovery_lost_and_found.py:327`
4. **What was the root cause of issue #73793 regarding mid-turn message rendering?**
gold: `redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row`
5. **Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?**
gold: `#84287`
6. **In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?**
gold: `pane sash grab band made asymmetric 1px/7px`
7. **Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?**
gold: `602028@ky-tech.com.cn`
8. **What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?**
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout`
9. **What is the core design principle regarding 'Narrow Waist' in Hermes development?**
gold: `The core is a narrow waist; capability lives at the edges.`
10. **What was the result of the rebase-merge attempt for PR #86589?**
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
11. **In the infographic style picker, what vibe is associated with the 'designers-republic' style?**
gold: `The Designers Republic: flat orange+violet vector schematic on pewter grey`
12. **Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?**
gold: `conflicts with main in 4 files and introduces a second competing display-dedupe scheme`
13. **What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?**
gold: `May 23 2026`
14. **What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?**
gold: `Property 'onToggleUnread' is missing in type`
15. **According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?**
gold: `Electron`
</details>
<details><summary>15 exam questions (questions-9c55c707b6.json)</summary>
1. **What two PR numbers are associated with the 'sidebar-nav-rows-and-overlay-panels.md' and 'hud-mode-internals.md' references in the initial tool content?**
gold: `#85162 and #82285`
2. **According to AGENTS.md, what is the 'one exception' to the rule that nothing should rebuild the system prompt mid-conversation?**
gold: `context compression`
3. **In the Contribution Rubric, what are the three allowed reasons for an automated triage sweeper to close a PR?**
gold: `implemented_on_main, cannot_reproduce, incoherent`
4. **Which contributor is credited with adding the 'Brazilian Portuguese localization' in PR #86292?**
gold: `@gui8515`
5. **What specific error message is reported in issue #83562 regarding the Windows Desktop update?**
gold: `Hermes backend exited (0)`
6. **What is the 'core problem' identified in the parallel-subagent-salvage-orchestration.md reference?**
gold: `subagents share the parent's worktree + main checkout`
7. **Why was the 'nix (macos-latest)' build failing in the salvage batches according to the orchestration reference?**
gold: `Nix build failed due to stale npm lockfile hash`
8. **Which subagent ID was assigned the goal of salvaging the 'inflight-journal duplicate-answer cluster'?**
gold: `sa-2-7318d0ba`
9. **In PR #86595, why was PR #80707 by upperagent excluded from the salvage?**
gold: `violating this PR's UI-read-only invariant`
10. **What was the root cause of the failure in Python tests slice 4/12 for PR #86597?**
gold: `AssertionError: assert 't2' == 't1'`
11. **What did the fix for issue #79157 in PR #86589 involve?**
gold: `pane sash grab band made asymmetric 1px/7px`
12. **According to the root cause analysis for #73793, which two files spliced the mid-turn user message at streamIndex?**
gold: `use-prompt-actions/index.ts and session-tile-actions.ts`
13. **What was the head SHA for the 'salvage/desktop-busy-state' branch in PR #86604?**
gold: `bddadfe9e21e24b3d52e2b15f138c42474dede42`
14. **Why was the merge of PR #86589 aborted during the 'Merge all' command?**
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
15. **What specific file was modified to fix the 'artifacts page timestamps render 1970' issue via PR #86749?**
gold: `apps/desktop/src/app/session/hooks/use-session-actions/utils.ts`
</details>
### Transcript: prmerge
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| uncompacted_control | 96.7% | 499,663 | 499,663 | — |
| current | 33.3% | 499,663 | 155,399 | 14.9 |
| lean | 23.3% | 499,663 | 44,419 | 105.4 |
| lean+recovery | 43.3% | 499,663 | 44,977 | 95.8 |
<details><summary>15 exam questions (questions-703ae2774a.json)</summary>
1. **Which PR number added the public subagent lifecycle API?**
gold: `#63359`
2. **What is the name of the typed service added to PluginContext for launching and monitoring child sessions?**
gold: `subagent_lifecycle`
3. **How many contract and security tests were included with the subagent lifecycle API PR?**
gold: `42`
4. **What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?**
gold: `cannot currently trigger a turn in an existing gateway session`
5. **Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?**
gold: `#64436`
6. **What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?**
gold: `must compile, must start with ≥2 literal characters`
7. **Which contributor authorized sustained help for the Phase 0–1 expansion track?**
gold: `Daniel`
8. **What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?**
gold: `#64204`
9. **What configuration setting is required to opt-in to reasoning deltas in streaming output?**
gold: `plugins.stream_reasoning_deltas: true`
10. **How many additions and across how many files were made in PR #63359?**
gold: `650 additions across 4 files`
11. **What is the name of the reference plugin shipped with the redaction pattern registry?**
gold: `nvapi-redaction`
12. **List the four observer-only streaming output plugin hooks added in PR #64317.**
gold: `on_stream_start, on_stream_delta, on_stream_end, on_interim_message`
13. **What was addressed in the update to PR #58541 regarding lifecycle hooks?**
gold: `created-hook timing and added kanban_task_promoted`
14. **Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?**
gold: `#64230`
15. **What was the Round 3 review's outcome for PR #63359 and @asimons81?**
gold: `sub-issue #65447`
</details>
### Transcript: acp
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| uncompacted_control | 100.0% | 498,906 | 498,906 | — |
| current | 30.0% | 498,906 | 160,223 | 15.8 |
| lean | 36.7% | 498,906 | 49,523 | 143.3 |
| lean+recovery | 80.0% | 498,906 | 49,721 | 135.6 |
<details><summary>15 exam questions (questions-f45358df19.json)</summary>
1. **What was the specific reason Teknium gave for reverting PR #30179 in July 2026?**
gold: `WTF??? REVERT! DAMMIT`
2. **On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?**
gold: `PR #6391`
3. **Which file path should be checked for the canonical list of provider models?**
gold: `hermes_cli/models.py`
4. **What was the identified bug in PR #2314 regarding provider names?**
gold: `checking for "alibaba-coding-plan"`
5. **What is the mandatory line limit for PR reviews requested by Teknium?**
gold: `<= 15 lines`
6. **What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?**
gold: `Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix mod`
7. **Why was PR #74658 necessary to fix Slack 'broken on main'?**
gold: `SlackResponse isn't a dict subclass, so every gate is always False.`
8. **What was the final merge commit SHA for the Slack SDK response fix on main?**
gold: `24ba86627515ad5fda69a39ef338c365713448bc`
9. **In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?**
gold: `Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1`
10. **What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?**
gold: `UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'`
11. **Which test failed in CI slice 5/12 for the kanban delivery modes PR?**
gold: `tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post`
12. **According to the transcript, why is squash merging banned as of July 2026?**
gold: `DevOps policy`
13. **Which contributor authored the first fix for issue #73030 in July?**
gold: `@Tranquil-Flow`
14. **What was the 'Superman-style' shield error in the first generation of the Kanban infographic?**
gold: `red "S" inside the diamond shield`
15. **What specific file was modified to add the 'scope_id_for_chat' method for Slack?**
gold: `plugins/platforms/slack/adapter.py`
</details>
## Methodology notes
- Transcripts: 4 real session lineages reconstructed from a state.db copy
(sweep campaign 42 rotations / GUI desktop 34 / PR-merge 17 / ACP review
17), chronological 500K-token prefix, tool-group aligned.
- Question generation: main model, from the region the CURRENT policy would
summarize (most conservative boundary), cached per transcript so every
policy answers the identical exam.
- Answering: fresh LLM sees ONLY the post-compaction context (closed-book) or
context + one FTS5+BM25 search round-trip over the archived region
(+recovery). Judge sees gold; answerer never does. Scoring 2/1/0.
- Known caveats: 15 questions/transcript => +-1 question ~ 3.3pts noise;
sweep/gui current-policy rows predate a question-bank regeneration
(prmerge/acp are same-bank across all arms); the recovery sim conservatively
approximates production session_search (same engine, no windowing).
- Cost shape: lean compaction = ~25 aux-model digest calls (~2min, one-time
per compaction) vs 1 call today; every post-compaction turn is ~110K input
tokens cheaper. Break-even ~1 turn.
evals/compaction/results/codex-arm-2026-08-15/acp.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 36.7,
"scores": [
0,
0,
2,
1,
0,
0,
2,
0,
0,
2,
0,
0,
0,
2,
2
],
"read_seconds": 229,
"compaction_events": 1,
"peak_context_tokens": 454931,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-35-51-01a00835-fef4-7c31-944d-9d0d8ba933be.jsonl"
},
"results": [
{
"q": "What was the specific reason Teknium gave for reverting PR #30179 in July 2026?",
"gold": "WTF??? REVERT! DAMMIT",
"answer": "UNKNOWN \u2014 best guess: it caused a regression on current `main` and had to be reverted until a safer implementation was available.",
"score": 0,
"why": "The answer provides a logical guess about a regression, but it fails to capture the specific, idiosyncratic reason/quote provided in the gold answer ('WTF??? REVERT! DAMMIT')."
},
{
"q": "On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?",
"gold": "PR #6391",
"answer": "UNKNOWN \u2014 best guess: PR #37865.",
"score": 0,
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold answer (PR #37865 vs PR #6391)."
},
{
"q": "Which file path should be checked for the canonical list of provider models?",
"gold": "hermes_cli/models.py",
"answer": "UNKNOWN \u2014 best guess: `hermes_cli/models.py`.",
"score": 2,
"why": "The answer provides the correct file path as its best guess, which matches the gold answer exactly."
},
{
"q": "What was the identified bug in PR #2314 regarding provider names?",
"gold": "checking for \"alibaba-coding-plan\"",
"answer": "UNKNOWN \u2014 best guess: it used inconsistent/noncanonical provider names, so provider matching and routing failed.",
"score": 1,
"why": "The answer identifies the core issue (provider name mismatch/inconsistency) which led to the specific bug, but it does not mention the specific string 'alibaba-coding-plan' identified in the gold answer."
},
{
"q": "What is the mandatory line limit for PR reviews requested by Teknium?",
"gold": "<= 15 lines",
"answer": "Maximum 5 lines.",
"score": 0,
"why": "The answer states a limit of 5 lines, which contradicts the gold answer of 15 lines."
},
{
"q": "What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?",
"gold": "Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module versions in this running process.",
"answer": "UNKNOWN \u2014 best guess: `fatal: '<branch>' is already checked out at '<worktree path>'`.",
"score": 0,
"why": "The answer provides a standard Git error message as a guess, which does not match the specific custom error message provided in the gold answer."
},
{
"q": "Why was PR #74658 necessary to fix Slack 'broken on main'?",
"gold": "SlackResponse isn't a dict subclass, so every gate is always False.",
"answer": "Real Slack SDK calls return `SlackResponse` objects, but the main code treated them like dictionaries and called dictionary-style accessors.",
"score": 2,
"why": "The answer correctly identifies that SlackResponse objects were being treated as dictionaries when they are not, which matches the core issue described in the gold answer that their lack of dictionary subclassing caused attribute/gate checks to fail."
},
{
"q": "What was the final merge commit SHA for the Slack SDK response fix on main?",
"gold": "24ba86627515ad5fda69a39ef338c365713448bc",
"answer": "UNKNOWN.",
"score": 0,
"why": "The gold answer provides a specific commit SHA, while the answer states 'UNKNOWN', failing to provide the requested factual information."
},
{
"q": "In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?",
"gold": "Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1",
"answer": "UNKNOWN.",
"score": 0,
"why": "The answer 'UNKNOWN' is a refusal to provide the information required by the gold answer."
},
{
"q": "What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?",
"gold": "UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'",
"answer": "`UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'` \u2014 applied only during first migration/addition of the column so existing choices were not overwritten.",
"score": 2,
"why": "The answer provides the exact SQL statement specified in the gold answer and correctly contextualizes its purpose."
},
{
"q": "Which test failed in CI slice 5/12 for the kanban delivery modes PR?",
"gold": "tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post",
"answer": "UNKNOWN \u2014 best guess: a relay-metrics concurrency test, unrelated to the delivery-mode change.",
"score": 0,
"why": "The answer identifies an unrelated test (relay-metrics) rather than the specific kanban notifier test mentioned in the gold answer."
},
{
"q": "According to the transcript, why is squash merging banned as of July 2026?",
"gold": "DevOps policy",
"answer": "Squash merging destroys the contributor-authored commit history and therefore loses reliable authorship/credit.",
"score": 0,
"why": "The answer explains a general rationale for why someone might dislike squash merging, but it fails to provide the specific reason cited in the text ('DevOps policy')."
},
{
"q": "Which contributor authored the first fix for issue #73030 in July?",
"gold": "@Tranquil-Flow",
"answer": "UNKNOWN.",
"score": 0,
"why": "The answer states 'UNKNOWN' while the gold answer provides a specific contributor name (@Tranquil-Flow)."
},
{
"q": "What was the 'Superman-style' shield error in the first generation of the Kanban infographic?",
"gold": "red \"S\" inside the diamond shield",
"answer": "The generated shield contained an emblem resembling Superman\u2019s \u201cS,\u201d rather than a neutral Kanban/routing symbol.",
"score": 2,
"why": "The answer accurately identifies that the shield contained a symbol resembling Superman's 'S', which matches the core fact of the gold answer."
},
{
"q": "What specific file was modified to add the 'scope_id_for_chat' method for Slack?",
"gold": "plugins/platforms/slack/adapter.py",
"answer": "`plugins/platforms/slack/adapter.py`.",
"score": 2,
"why": "The answer matches the gold answer exactly, including the full file path."
}
]
}
evals/compaction/results/codex-arm-2026-08-15/gui.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 40.0,
"scores": [
2,
2,
0,
2,
0,
0,
0,
2,
2,
2,
0,
0,
0,
0,
0
],
"read_seconds": 207,
"compaction_events": 1,
"peak_context_tokens": 482525,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-28-36-01a0082f-5b69-7ff0-977d-998523a4b3fc.jsonl"
},
"results": [
{
"q": "What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?",
"gold": "#86617",
"answer": "PR **#86617**.",
"score": 2,
"why": "The answer provides the exact PR number specified in the gold answer."
},
{
"q": "According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?",
"gold": ".env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.",
"answer": "`.env` is for secrets and deployment-specific values; `config.yaml` is for non-secret user-facing runtime configuration.",
"score": 2,
"why": "The answer correctly identifies that .env is for secrets and config.yaml is for non-secret/behavioral settings, which matches the core distinction in the gold answer."
},
{
"q": "What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?",
"gold": "tests/hermes_cli/test_session_recovery_lost_and_found.py:327",
"answer": "**UNKNOWN** \u2014 best guess: a schema-width assertion in a Python schema/config test, expecting 55 fields but receiving 56 after `git_metadata_generation` was added.",
"score": 0,
"why": "The answer fails to provide the specific file and line number requested, which are present in the gold answer. It instead provides a guess about the context of the error without identifying the location."
},
{
"q": "What was the root cause of issue #73793 regarding mid-turn message rendering?",
"gold": "redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row",
"answer": "Two live paths handled mid-turn corrections independently, allowing corrections and assistant stream output to be inserted in the wrong causal order.",
"score": 2,
"why": "The answer identifies the same root cause as the gold answer: the independent handling of paths caused the user bubble (correction) and assistant stream to be rendered in the wrong causal order."
},
{
"q": "Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?",
"gold": "#84287",
"answer": "**PR #86299**.",
"score": 0,
"why": "The answer identifies PR #86299, but the gold answer is PR #84287. The two are different pull requests."
},
{
"q": "In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?",
"gold": "pane sash grab band made asymmetric 1px/7px",
"answer": "**UNKNOWN** \u2014 best guess: make the scrollbar gutter/overlay accept pointer events instead of letting the virtualized content layer cover it.",
"score": 0,
"why": "The answer provides a guess that contradicts the gold answer; the fix was making the pane sash grab band asymmetric (1px/7px), not changing pointer events on the gutter/overlay."
},
{
"q": "Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?",
"gold": "602028@ky-tech.com.cn",
"answer": "**UNKNOWN** \u2014 best guess: the email associated with contributor `hkfiberlaser-svg`.",
"score": 0,
"why": "The answer states the answer is unknown and provides a guess that does not match the gold answer."
},
{
"q": "What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?",
"gold": "Blocked: `git <op>` would rewrite Hermes's live source checkout",
"answer": "**UNKNOWN** \u2014 best guess: an error stating that the git command was blocked because it would rewrite the live source checkout.",
"score": 2,
"why": "The answer correctly identifies the core substance of the error message (blocked git command because it would rewrite the live source checkout) which matches the gold answer."
},
{
"q": "What is the core design principle regarding 'Narrow Waist' in Hermes development?",
"gold": "The core is a narrow waist; capability lives at the edges.",
"answer": "Keep a small, stable core interface\u2014the \u201cnarrow waist\u201d\u2014through which different clients, providers, and integrations communicate, avoiding duplicated business logic across surfaces.",
"score": 2,
"why": "The answer correctly identifies the core principle of a 'narrow waist' and explains that the logic/interfaces are centralized there while clients and providers interact at the edges, matching the substance of the gold answer."
},
{
"q": "What was the result of the rebase-merge attempt for PR #86589?",
"gold": "GraphQL: Pull Request has merge conflicts (mergePullRequest)",
"answer": "**UNKNOWN** \u2014 best guess: the rebase merge could not proceed because the PR had conflicts and was not mergeable.",
"score": 2,
"why": "The answer correctly identifies that the rebase-merge could not proceed due to merge conflicts, which matches the factual content of the gold answer."
},
{
"q": "In the infographic style picker, what vibe is associated with the 'designers-republic' style?",
"gold": "The Designers Republic: flat orange+violet vector schematic on pewter grey",
"answer": "A bold, experimental, anti-corporate graphic-design vibe\u2014brutalist/techno typography inspired by The Designers Republic.",
"score": 0,
"why": "The answer describes the real-world aesthetic of 'The Designers Republic' studio, but fails to identify the specific visual elements (flat orange+violet vector schematic on pewter grey) defined for this style within the infographic picker context."
},
{
"q": "Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?",
"gold": "conflicts with main in 4 files and introduces a second competing display-dedupe scheme",
"answer": "It did not address transcript visibility after compaction/compression; its changes concerned a different part of the compaction flow.",
"score": 0,
"why": "The answer provides a completely different reason than the gold answer, which specifies merge conflicts and a competing deduplication scheme as the reasons for exclusion."
},
{
"q": "What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?",
"gold": "May 23 2026",
"answer": "**UNKNOWN** \u2014 best guess: **2026-08-12**.",
"score": 0,
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold date (2026-08-12 vs. May 23 2026)."
},
{
"q": "What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?",
"gold": "Property 'onToggleUnread' is missing in type",
"answer": "**UNKNOWN** \u2014 best guess: a TypeScript unresolved-name/import error introduced by the rebase.",
"score": 0,
"why": "The answer identifies as UNKNOWN and provides an incorrect guess. The gold answer specifies a 'property is missing' error, while the provided answer guesses an 'unresolved-name/import' error."
},
{
"q": "According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?",
"gold": "Electron",
"answer": "The **Rust/Tauri backend** is authoritative for process lifecycle and the native filesystem.",
"score": 0,
"why": "The answer identifies the Rust/Tauri backend as the authority, which directly contradicts the gold answer stating Electron is the authority."
}
]
}
evals/compaction/results/codex-arm-2026-08-15/prmerge.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 43.3,
"scores": [
2,
2,
0,
2,
2,
0,
0,
2,
0,
0,
0,
2,
1,
0,
0
],
"read_seconds": 154,
"compaction_events": 1,
"peak_context_tokens": 465131,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-32-45-01a00833-2968-7453-8646-d5bb7f5e8400.jsonl"
},
"results": [
{
"q": "Which PR number added the public subagent lifecycle API?",
"gold": "#63359",
"answer": "PR **#63359**.",
"score": 2,
"why": "The answer matches the gold PR number exactly."
},
{
"q": "What is the name of the typed service added to PluginContext for launching and monitoring child sessions?",
"gold": "subagent_lifecycle",
"answer": "`SubagentService` (`ctx.subagents`).",
"score": 2,
"why": "The answer identifies the correct property on the context (ctx.subagents), which is the typed service used for the task, even though it uses a slightly different name for the service itself."
},
{
"q": "How many contract and security tests were included with the subagent lifecycle API PR?",
"gold": "42",
"answer": "**UNKNOWN** \u2014 best guess: **24 tests**.",
"score": 0,
"why": "The answer provides an incorrect guess (24) compared to the gold answer (42) and frames the response as unknown."
},
{
"q": "What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?",
"gold": "cannot currently trigger a turn in an existing gateway session",
"answer": "`ctx.inject_message()` could not safely target an active gateway session; it lacked a profile/session-scoped `session_key`.",
"score": 2,
"why": "The answer identifies the core limitation (targeting active gateway sessions) and provides technical detail on why a turn cannot be triggered (lack of session-scoped key)."
},
{
"q": "Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?",
"gold": "#64436",
"answer": "PR **#64436**.",
"score": 2,
"why": "The answer matches the gold answer exactly."
},
{
"q": "What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?",
"gold": "must compile, must start with \u22652 literal characters",
"answer": "Patterns must be **bounded in length** and **validated against unsafe/catastrophic regex behavior** before registration.",
"score": 0,
"why": "The answer discusses general regex safety (length and performance), whereas the gold answer specifies two precise technical constraints: compilation and a minimum of two literal characters at the start."
},
{
"q": "Which contributor authorized sustained help for the Phase 0\u20131 expansion track?",
"gold": "Daniel",
"answer": "**@asimons81**.",
"score": 0,
"why": "The answer identifies @asimons81 as the contributor, whereas the gold answer states it was Daniel. These do not match."
},
{
"q": "What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?",
"gold": "#64204",
"answer": "Issue **#64204**.",
"score": 2,
"why": "The answer matches the gold issue number exactly."
},
{
"q": "What configuration setting is required to opt-in to reasoning deltas in streaming output?",
"gold": "plugins.stream_reasoning_deltas: true",
"answer": "**UNKNOWN** \u2014 best guess: `display.show_reasoning: true`.",
"score": 0,
"why": "The answer identifies itself as a guess and provides the incorrect configuration setting ('display.show_reasoning' vs the gold 'plugins.stream_reasoning_deltas')."
},
{
"q": "How many additions and across how many files were made in PR #63359?",
"gold": "650 additions across 4 files",
"answer": "**UNKNOWN** \u2014 best guess: roughly **1,000 additions across 12 files**.",
"score": 0,
"why": "The answer states the information is unknown and provides a guess that is factually incorrect regarding both the number of additions and the number of files."
},
{
"q": "What is the name of the reference plugin shipped with the redaction pattern registry?",
"gold": "nvapi-redaction",
"answer": "**UNKNOWN** \u2014 best guess: `redaction-patterns`.",
"score": 0,
"why": "The answer identifies itself as unknown and provides a guess ('redaction-patterns') that is incorrect compared to the gold answer ('nvapi-redaction')."
},
{
"q": "List the four observer-only streaming output plugin hooks added in PR #64317.",
"gold": "on_stream_start, on_stream_delta, on_stream_end, on_interim_message",
"answer": "`on_stream_start`, `on_stream_delta`, `on_stream_end`, and `on_interim_message`.",
"score": 2,
"why": "The answer lists all four hooks correctly as specified in the gold answer."
},
{
"q": "What was addressed in the update to PR #58541 regarding lifecycle hooks?",
"gold": "created-hook timing and added kanban_task_promoted",
"answer": "It updated the lifecycle hooks to match the current registry contract, including correct hook timing, payloads, and failure isolation.",
"score": 1,
"why": "The answer correctly identifies 'hook timing' as part of the update, but it fails to mention the specific addition of 'kanban_task_promoted' and instead provides a general description of other improvements like payloads and isolation not mentioned in the gold answer."
},
{
"q": "Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?",
"gold": "#64230",
"answer": "**UNKNOWN** \u2014 best guess: **#64179**.",
"score": 0,
"why": "The answer provides an incorrect issue number (#64179) while the gold answer is #64230."
},
{
"q": "What was the Round 3 review's outcome for PR #63359 and @asimons81?",
"gold": "sub-issue #65447",
"answer": "Round 3 **approved PR #63359 and authorized @asimons81 to continue sustained Phase 0\u20131 work**.",
"score": 0,
"why": "The answer provides a general description of approval and authorization, but fails to mention the specific outcome required by the gold answer (the creation or reference of sub-issue #65447)."
}
]
}
evals/compaction/results/codex-arm-2026-08-15/sweep.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 26.7,
"scores": [
2,
0,
2,
2,
0,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0
],
"read_seconds": 420,
"compaction_events": 1,
"peak_context_tokens": 482248,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-20-54-01a00828-4e75-70f0-9e6d-31c7aac8d71c.jsonl"
},
"results": [
{
"q": "What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?",
"gold": "the stale file version silently deletes newer main code",
"answer": "It can silently replace current files with stale branch versions, discarding newer fixes and resurrecting deleted code; salvage must use commits and resolve conflicts explicitly.",
"score": 2,
"why": "The answer accurately reflects the gold answer's point that stale branch versions will overwrite newer code, while adding specific detail about resurrecting deleted code."
},
{
"q": "According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?",
"gold": "~60GB",
"answer": "UNKNOWN \u2014 best guess: approximately 4 GB RSS.",
"score": 0,
"why": "The answer provides a 'best guess' of 4 GB, which is factually incorrect and contradicts the gold answer of 60 GB."
},
{
"q": "Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?",
"gold": "backgroundThrottling",
"answer": "UNKNOWN \u2014 best guess: Electron\u2019s `backgroundThrottling: false`.",
"score": 2,
"why": "The answer correctly identifies the specific setting 'backgroundThrottling' as the suspected cause, despite the initial 'UNKNOWN' disclaimer."
},
{
"q": "What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?",
"gold": "Auto merge is not allowed for this repository (enablePullRequestAutoMerge)",
"answer": "`GraphQL: Pull request Auto merge is not allowed for this repository`",
"score": 2,
"why": "The answer matches the core message of the gold answer. The 'GraphQL:' prefix accurately reflects the technical origin of the error message when using the GitHub CLI."
},
{
"q": "What is the specified 'Rule 0' that must be included in a subagent brief?",
"gold": "load the skill first",
"answer": "Rule 0: do not merge anything; only the primary agent/user is authorized to merge.",
"score": 0,
"why": "The answer identifies a different 'Rule 0' regarding merging permissions, which does not match the gold answer's requirement to 'load the skill first'."
},
{
"q": "In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?",
"gold": "add config gate for title generation",
"answer": "UNKNOWN \u2014 best guess: \u201cfix: prevent duplicate message submission.\u201d",
"score": 0,
"why": "The answer provides an incorrect guess ('fix: prevent duplicate message submission.') which does not match the gold answer ('add config gate for title generation')."
},
{
"q": "Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?",
"gold": "tests/test_packaging_metadata.py",
"answer": "UNKNOWN \u2014 best guess: `tests/test_plugin_packaging.py`.",
"score": 0,
"why": "The answer identifies the wrong file (test_plugin_packaging.py vs the correct test_packaging_metadata.py)."
},
{
"q": "What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?",
"gold": "completed normally (exit code 0)",
"answer": "`npm ci` completed successfully with exit code 0.",
"score": 2,
"why": "The answer matches the gold answer perfectly, confirming the successful completion and the exit code of 0."
},
{
"q": "What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?",
"gold": "relative exclude-newer makes the committed lock stale every day",
"answer": "Root Cause A: the `all` extra referenced platform-specific/nonexistent dependency groups that were absent or inconsistent in the locked resolution.",
"score": 0,
"why": "The answer identifies an issue with dependency groups, whereas the gold answer states the root cause was the use of a relative 'exclude-newer' value that caused the lockfile to become stale daily."
},
{
"q": "How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?",
"gold": "13",
"answer": "UNKNOWN \u2014 best guess: 7 tasks done.",
"score": 0,
"why": "The answer guesses 7 tasks, while the gold answer is 13. The answer is factually incorrect."
},
{
"q": "In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?",
"gold": "-1",
"answer": "Exit code 2.",
"score": 0,
"why": "The answer provides a specific exit code (2) that contradicts the gold answer (-1)."
},
{
"q": "What is the full title block text for the technical schematic infographic generated for the Gateway Drain?",
"gold": "GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT",
"answer": "UNKNOWN \u2014 best guess: `GATEWAY DRAIN` / `INTERRUPT \u2192 DELIVER \u2192 DISCONNECT` / `GRACEFUL SHUTDOWN WITHOUT MESSAGE LOSS`.",
"score": 0,
"why": "The answer fails to provide the correct title block text specified in the gold answer. It incorrectly guesses metadata or slogans instead of the specific title 'GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT'."
},
{
"q": "Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?",
"gold": "82980",
"answer": "UNKNOWN \u2014 best guess: PR #86594.",
"score": 0,
"why": "The answer identifies the wrong PR number (86594 instead of 82980)."
},
{
"q": "What is the specific Gist ID created for the PR infographic host in the cron cluster?",
"gold": "ee33edd5804689243f974536ef7aecb9",
"answer": "UNKNOWN \u2014 I do not retain the Gist ID.",
"score": 0,
"why": "The answer states it does not know the information, whereas the gold answer provides a specific Gist ID."
},
{
"q": "What was the final merge SHA for Cluster D's Trigger-now PR #70638?",
"gold": "f9d64b9a9d8b306f64851c1a13869d96ad5d7869",
"answer": "UNKNOWN \u2014 I do not retain the final merge SHA for PR #70638.",
"score": 0,
"why": "The answer claims it does not know the information, while the gold answer provides the specific SHA requested."
}
]
}
evals/compaction/runner.py
+356
View File
@@ -0,0 +1,356 @@
"""Compaction eval runner.
Pipeline per transcript:
1. Load + cap the transcript.
2. Generate (or load cached) recall questions from the region that will be
summarized away under the CURRENT policy (the most conservative boundary:
anything the current policy summarizes is fair game for every policy).
3. For each policy: compress, then answer each question with ONLY the
compressed context, using a single LLM call per question.
4. Judge answers against gold with an LLM judge (sees gold; answerer
does not).
5. Write per-policy results JSON for report.py.
Run from repo root with the project venv (needs a configured provider).
"""
from __future__ import annotations
import argparse
import copy
import hashlib
import json
import re
import sys
import time
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
from evals.compaction.fixtures import ( # noqa: E402
estimate_tokens,
load_transcript,
total_tokens,
)
from evals.compaction.policies import EVAL_MODEL, POLICIES, apply_policy # noqa: E402
QUESTION_PROMPT = """You are building a factual recall exam from an AI-agent work session transcript.
Write {n} questions that test SPECIFIC, VERIFIABLE facts from the transcript below: identifiers (PR numbers, file paths, error messages, commit subjects), decisions and their reasons, user instructions, and outcomes. Rules:
- Every answer must appear literally in the transcript.
- No questions about the system prompt or generic behavior.
- Spread questions across the WHOLE span (early, middle, late).
- Prefer facts that matter for continuing the work (what was decided, what failed, what the user asked for).
Return STRICT JSON: a list of {{"q": "...", "gold": "...", "where": "<short quote locating the answer>"}}.
TRANSCRIPT:
{transcript}
"""
ANSWER_PROMPT = """You are an AI agent resuming a work session. Below is your CURRENT conversation context (it may include a compaction summary of earlier work). Answer the question using ONLY this context. If the context does not contain the answer, say exactly "NOT IN CONTEXT" and give your best guess after a semicolon.
CONTEXT:
{context}
QUESTION: {question}
Answer in one or two sentences."""
JUDGE_PROMPT = """Score this answer against the gold answer. Reply with STRICT JSON: {{"score": 2|1|0, "why": "..."}}.
2 = factually matches gold (wording may differ)
1 = partially correct or hedged-but-right ("NOT IN CONTEXT; guess X" where X is right scores 1)
0 = wrong, or "NOT IN CONTEXT" with a wrong/no guess
QUESTION: {question}
GOLD: {gold}
ANSWER: {answer}"""
SEARCH_QUERY_PROMPT = """You are an AI agent resuming a work session. Your context (below) includes a compaction summary noting that the full pre-compaction history is recoverable via session_search. You need to answer a question and the answer may not be in your current context.
Write the best search query (3-8 keywords, no boolean syntax) to find the answer in the archived session history. Reply with ONLY the query string.
CONTEXT (may be relevant):
{context_hint}
QUESTION: {question}"""
ANSWER_WITH_RECOVERY_PROMPT = """You are an AI agent resuming a work session. Below is your CURRENT conversation context (including a compaction summary), plus the results of a session_search you just ran against the archived pre-compaction history. Answer the question using both. If neither contains the answer, say exactly "NOT IN CONTEXT" and give your best guess after a semicolon.
CONTEXT:
{context}
SESSION_SEARCH RESULTS:
{search_results}
QUESTION: {question}
Answer in one or two sentences."""
def keyword_search(archive: list, query: str, top_k: int = 4, excerpt_chars: int = 2500) -> str:
"""Simulate session_search over the archived (compacted-away) region.
Uses an in-memory SQLite FTS5 index with BM25 ranking — the same engine
production session_search runs on — so the sim's retrieval quality
matches what a live agent gets. Falls back to term-frequency scoring if
FTS5 is unavailable.
"""
import sqlite3 as _sq
terms = [t.lower() for t in re.findall(r"[A-Za-z0-9_#./-]{3,}", query)]
if not terms:
return "(no results)"
rows = [
(i, m.get("role") or "", m["content"])
for i, m in enumerate(archive)
if isinstance(m.get("content"), str) and len(m["content"]) >= 20
]
hits = []
try:
db = _sq.connect(":memory:")
db.execute("CREATE VIRTUAL TABLE arch USING fts5(content, role UNINDEXED, idx UNINDEXED)")
db.executemany(
"INSERT INTO arch (content, role, idx) VALUES (?, ?, ?)",
[(c, r, i) for i, r, c in rows],
)
fts_query = " OR ".join(
'"' + t.replace('"', "") + '"' for t in terms
)
cur = db.execute(
"SELECT idx, role, content, bm25(arch) AS rank, "
"snippet(arch, 0, '', '', ' … ', 40) AS snip "
"FROM arch WHERE arch MATCH ? ORDER BY rank LIMIT ?",
(fts_query, top_k),
)
for idx, role, content, rank, snip in cur.fetchall():
lc = content.lower()
first = min((lc.find(t) for t in terms if lc.find(t) >= 0), default=0)
start = max(0, first - excerpt_chars // 4)
hits.append(
f"--- result (message #{idx}, role={role}) ---\n"
f"[match: {snip[:200]}]\n"
+ content[start:start + excerpt_chars]
)
db.close()
except _sq.OperationalError:
# FTS5 unavailable — degrade to term-frequency scoring.
scored = []
for i, r, c in rows:
lc = c.lower()
score = sum(lc.count(t) for t in terms) / (1 + len(c) / 4000)
if score > 0:
scored.append((score, i, r, c))
scored.sort(key=lambda x: -x[0])
for score, i, r, c in scored[:top_k]:
lc = c.lower()
first = min((lc.find(t) for t in terms if lc.find(t) >= 0), default=0)
start = max(0, first - excerpt_chars // 4)
hits.append(
f"--- result (message #{i}, role={r}) ---\n"
+ c[start:start + excerpt_chars]
)
return "\n\n".join(hits) if hits else "(no results)"
def _call(prompt: str, max_tokens: int = 2000) -> str:
from agent.auxiliary_client import call_llm
resp = call_llm(
messages=[{"role": "user", "content": prompt}],
task="compression",
max_tokens=max_tokens,
)
if hasattr(resp, "choices"):
return resp.choices[0].message.content or ""
return str(resp)
def _extract_json(text: str):
m = re.search(r"```(?:json)?\s*(.*?)```", text, re.S)
if m:
text = m.group(1)
start = min([i for i in (text.find("["), text.find("{")) if i >= 0], default=0)
return json.loads(text[start:])
def serialize_for_exam(messages, char_cap: int = 600_000) -> str:
parts = []
for m in messages:
role = m.get("role")
c = m.get("content")
if not isinstance(c, str) or not c:
continue
if role == "system":
continue
parts.append(f"[{role}] {c}")
text = "\n\n".join(parts)
if len(text) > char_cap:
half = char_cap // 2
text = text[:half] + "\n\n...[middle elided for exam generation]...\n\n" + text[-half:]
return text
def summarized_region(compressor_module, messages):
"""The middle region the current policy would summarize: everything
between the protected head and the tail cut. Questions come from here."""
from agent.context_compressor import ContextCompressor
comp = ContextCompressor(model=EVAL_MODEL, quiet_mode=True)
head_end = comp.protect_first_n
tail_start = comp._find_tail_cut_by_tokens(messages, head_end)
return messages[head_end:tail_start]
def generate_questions(messages, n: int, cache_path: Path) -> list:
if cache_path.exists():
return json.loads(cache_path.read_text(encoding="utf-8"))
import agent.context_compressor as cc
region = summarized_region(cc, messages)
text = serialize_for_exam(region)
raw = _call(QUESTION_PROMPT.format(n=n, transcript=text), max_tokens=4000)
questions = _extract_json(raw)[:n]
cache_path.parent.mkdir(parents=True, exist_ok=True)
cache_path.write_text(json.dumps(questions, indent=1), encoding="utf-8")
return questions
def run_policy(name: str, spec: dict, messages, questions, out_dir: Path,
with_recovery: bool = False) -> dict:
from agent.context_compressor import ContextCompressor
before = copy.deepcopy(messages)
comp = apply_policy(ContextCompressor(model=EVAL_MODEL, quiet_mode=True), spec)
for key, value in (spec.get("ctor") or {}).items():
setattr(comp, key, value)
t0 = time.time()
compressed = comp.compress(copy.deepcopy(messages), current_tokens=total_tokens(messages), force=True)
elapsed = time.time() - t0
# The archived region = original messages that did not survive verbatim.
surviving = set()
for m in compressed:
c = m.get("content")
if isinstance(c, str) and c:
surviving.add(c[:200])
archive = [
m for m in before
if isinstance(m.get("content"), str) and (m.get("content") or "")[:200] not in surviving
]
context_text = serialize_for_exam(compressed, char_cap=700_000)
results = []
for qa in questions:
if with_recovery:
# The summary (session log, verbatim user msgs, recovery footer) sits
# near the FRONT of the serialized context; give the query writer
# that portion plus the recent tail so it can mine anchor
# identifiers (PR numbers, paths, error strings) for the query.
hint = context_text[:60_000] + "\n...\n" + context_text[-8_000:]
query = _call(
SEARCH_QUERY_PROMPT.format(
context_hint=hint, question=qa["q"],
),
max_tokens=100,
).strip().strip('"')
search_results = keyword_search(archive, query)
answer = _call(
ANSWER_WITH_RECOVERY_PROMPT.format(
context=context_text,
search_results=search_results,
question=qa["q"],
),
max_tokens=400,
)
else:
query = None
answer = _call(ANSWER_PROMPT.format(context=context_text, question=qa["q"]), max_tokens=400)
verdict_raw = _call(JUDGE_PROMPT.format(question=qa["q"], gold=qa["gold"], answer=answer), max_tokens=300)
try:
verdict = _extract_json(verdict_raw)
except Exception:
verdict = {"score": 0, "why": f"judge parse failure: {verdict_raw[:100]}"}
entry = {"q": qa["q"], "gold": qa["gold"], "answer": answer, **verdict}
if query is not None:
entry["search_query"] = query
results.append(entry)
scored = [r["score"] for r in results]
label = f"{name}+recovery" if with_recovery else name
summary = {
"policy": label,
"before_tokens": total_tokens(before),
"after_tokens": total_tokens(compressed),
"after_msgs": len(compressed),
"compress_seconds": round(elapsed, 1),
"recall_pct": round(100 * sum(scored) / (2 * len(scored)), 1) if scored else 0.0,
"scores": scored,
"summary_error": getattr(comp, "_last_summary_error", None),
}
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / f"{label.replace('+', '_')}.json").write_text(json.dumps({"summary": summary, "results": results}, indent=1), encoding="utf-8")
return summary
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--transcript", required=True)
ap.add_argument("--cap-tokens", type=int, default=500_000)
ap.add_argument("--policies", default="current,tail25k,codex_style")
ap.add_argument("--questions", type=int, default=15)
ap.add_argument("--out", required=True)
ap.add_argument("--also-uncompacted", action="store_true")
args = ap.parse_args()
messages = load_transcript(args.transcript, cap_tokens=args.cap_tokens)
out_dir = Path(args.out)
tid = hashlib.md5(args.transcript.encode()).hexdigest()[:10]
qcache = out_dir / f"questions-{tid}.json"
questions = generate_questions(messages, args.questions, qcache)
print(f"{len(questions)} questions ready ({qcache})")
summaries = []
if args.also_uncompacted:
spec = {"ctor": {}, "attrs": {"tail_token_budget": 10**9}}
# control: no compression at all — answer from the full transcript
context_text = serialize_for_exam(messages, char_cap=900_000)
results = []
for qa in questions:
answer = _call(ANSWER_PROMPT.format(context=context_text, question=qa["q"]), max_tokens=400)
verdict_raw = _call(JUDGE_PROMPT.format(question=qa["q"], gold=qa["gold"], answer=answer), max_tokens=300)
try:
verdict = _extract_json(verdict_raw)
except Exception:
verdict = {"score": 0, "why": "judge parse failure"}
results.append({"q": qa["q"], **verdict, "answer": answer})
scored = [r["score"] for r in results]
ctl = {
"policy": "uncompacted_control",
"before_tokens": total_tokens(messages),
"after_tokens": total_tokens(messages),
"recall_pct": round(100 * sum(scored) / (2 * len(scored)), 1),
"scores": scored,
}
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "uncompacted_control.json").write_text(json.dumps({"summary": ctl, "results": results}, indent=1), encoding="utf-8")
summaries.append(ctl)
print(json.dumps(ctl, indent=1))
for name in args.policies.split(","):
name = name.strip()
with_recovery = name.endswith("+recovery")
base = name[:-len("+recovery")] if with_recovery else name
if base not in POLICIES:
print(f"unknown policy {base}, skipping"); continue
s = run_policy(base, POLICIES[base], messages, questions, out_dir,
with_recovery=with_recovery)
summaries.append(s)
print(json.dumps(s, indent=1))
(out_dir / "scorecard.json").write_text(json.dumps(summaries, indent=1), encoding="utf-8")
print(f"\nscorecard -> {out_dir}/scorecard.json")
if __name__ == "__main__":
main()
evals/compaction/scripts/build_html_report.py
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env python3
"""Build a self-contained HTML report comparing compaction runs.
Usage: build_report.py <runs_dir> <out_html>
Expects runs/<checkout>_<session>.json pairs from run_compaction.py.
"""
import html
import json
import sys
from pathlib import Path
RUNS = Path(sys.argv[1])
OUT = sys.argv[2]
pairs = {}
for f in sorted(RUNS.glob("*.json")):
co, sid = f.stem.split("_", 1)
if co == "main":
co, sid = "main-co", f.stem[len("main-co_"):]
elif co == "pr":
co, sid = "pr-co", f.stem[len("pr-co_"):]
data = json.loads(f.read_text(encoding="utf-8"))
pairs.setdefault(sid, {})[co] = data
E = html.escape
def msg_class(m):
role = m.get("role", "?")
c = m.get("content") or ""
if isinstance(c, str):
if "[CONTEXT COMPACTION" in c or "[CONTEXT SUMMARY" in c:
return "summary"
if "SKILL_PRUNED" in c:
return "skillpruned"
if "SKILL POLICY DIGEST" in c or "SKILL_POLICY_DIGEST" in c:
return "digest"
if "preserved across context compression" in c:
return "todosnap"
return role
def render_msg(m, idx):
role = m.get("role", "?")
c = m.get("content")
if not isinstance(c, str):
c = json.dumps(c, default=str)[:2000]
tool = m.get("tool_name") or ""
tcs = m.get("tool_calls") or []
tc_names = ", ".join(
(t.get("function", {}) or {}).get("name", "?") for t in tcs if isinstance(t, dict)
)
cls = msg_class(m)
nchars = len(c)
label = role
if tool:
label += f" · {tool}"
if tc_names:
label += f" → {tc_names}"
preview = c[:180].replace("\n", " ")
full = c if nchars <= 20000 else c[:20000] + f"\n…[{nchars-20000:,} more chars]"
return (
f'<details class="msg {cls}"><summary><span class="idx">#{idx}</span>'
f'<span class="role">{E(label)}</span>'
f'<span class="chars">{nchars:,}ch</span>'
f'<span class="preview">{E(preview)}</span></summary>'
f"<pre>{E(full)}</pre></details>"
)
def render_column(title, data, key):
meta = data["meta"]
msgs = data[key]
body = "".join(render_msg(m, i) for i, m in enumerate(msgs))
return (
f'<div class="col"><div class="colhead"><h3>{E(title)}</h3>'
f'<div class="stats">{meta[key.replace("before","before_msgs").replace("after","after_msgs")] if False else len(msgs)} msgs · '
f'~{(meta["before_tokens_est"] if key=="before" else meta["after_tokens_est"]):,} tok</div></div>'
f'<div class="msgs">{body}</div></div>'
)
def survival_stats(before, after):
after_texts = set()
for m in after:
c = m.get("content")
if isinstance(c, str) and c:
after_texts.add(c[:400])
kept = sum(1 for m in before if isinstance(m.get("content"), str) and (m.get("content") or "")[:400] in after_texts)
return kept
sections = []
toc = []
for sid, versions in pairs.items():
if "main-co" not in versions or "pr-co" not in versions:
continue
main_d, pr_d = versions["main-co"], versions["pr-co"]
title = main_d["meta"].get("title") or sid
mm, pm = main_d["meta"], pr_d["meta"]
def count_markers(msgs, needle):
return sum((m.get("content") or "").count(needle) for m in msgs if isinstance(m.get("content"), str))
rows = []
def stat(name, mv, pv):
cls = "diff" if mv != pv else ""
rows.append(f"<tr class='{cls}'><td>{E(name)}</td><td>{E(str(mv))}</td><td>{E(str(pv))}</td></tr>")
stat("Messages after", mm["after_msgs"], pm["after_msgs"])
stat("Est. tokens after", f"{mm['after_tokens_est']:,}", f"{pm['after_tokens_est']:,}")
stat("Reduction", f"{100-100*mm['after_tokens_est']//max(1,mm['before_tokens_est'])}%", f"{100-100*pm['after_tokens_est']//max(1,pm['before_tokens_est'])}%")
stat("Compress time", f"{mm['elapsed_s']}s", f"{pm['elapsed_s']}s")
stat("SKILL_PRUNED markers", count_markers(main_d["after"], "SKILL_PRUNED"), count_markers(pr_d["after"], "SKILL_PRUNED"))
stat("Policy digest blocks", count_markers(main_d["after"], "SKILL POLICY DIGEST") + count_markers(main_d["after"], "SKILL_POLICY_DIGEST"), count_markers(pr_d["after"], "SKILL POLICY DIGEST") + count_markers(pr_d["after"], "SKILL_POLICY_DIGEST"))
stat("Todo snapshot present", "yes" if count_markers(main_d["after"], "preserved across context compression") else "no", "yes" if count_markers(pr_d["after"], "preserved across context compression") else "no")
stat("Kept-verbatim msgs", survival_stats(main_d["before"], main_d["after"]), survival_stats(pr_d["before"], pr_d["after"]))
stat("Summary error", mm.get("summary_error") or "—", pm.get("summary_error") or "—")
todo_html = ""
for label, d in (("main", mm), ("PR #87090", pm)):
tb = d.get("todo_injection_block")
if tb:
todo_html += f"<h4>Todo injection block — {E(label)}</h4><pre class='todoblock'>{E(tb)}</pre>"
anchor = f"s-{sid}"
toc.append(f'<a href="#{anchor}">{E(title)} <span class="dim">({sid})</span></a>')
sections.append(f"""
<section id="{anchor}">
<h2>{E(title)} <span class="dim">{sid}</span></h2>
<table class="stats-table"><tr><th></th><th>main</th><th>PR #87090</th></tr>{"".join(rows)}</table>
{todo_html}
<div class="cols">
{render_column("BEFORE (original transcript)", main_d, "before")}
{render_column("AFTER — main", main_d, "after")}
{render_column("AFTER — PR #87090", pr_d, "after")}
</div>
</section>""")
page = f"""<!DOCTYPE html>
<html><head><meta charset="utf-8"><title>Compaction comparison — main vs PR #87090</title>
<style>
:root {{ color-scheme: dark; }}
body {{ background:#0d1117; color:#c9d1d9; font:14px/1.45 -apple-system,Segoe UI,sans-serif; margin:0; padding:24px; }}
h1 {{ font-size:22px; }} h2 {{ font-size:18px; border-bottom:1px solid #30363d; padding-bottom:6px; margin-top:48px; }}
.dim {{ color:#8b949e; font-weight:normal; font-size:12px; }}
nav a {{ display:block; color:#58a6ff; margin:2px 0; text-decoration:none; }}
.legend span {{ display:inline-block; padding:2px 10px; margin-right:8px; border-radius:4px; font-size:12px; }}
.stats-table {{ border-collapse:collapse; margin:12px 0; }}
.stats-table td, .stats-table th {{ border:1px solid #30363d; padding:4px 12px; text-align:left; font-size:13px; }}
.stats-table tr.diff td {{ background:#1c2a1c; }}
.cols {{ display:grid; grid-template-columns:1fr 1fr 1fr; gap:10px; }}
.col {{ min-width:0; }}
.colhead {{ position:sticky; top:0; background:#161b22; padding:8px; border:1px solid #30363d; border-radius:6px 6px 0 0; z-index:2; }}
.colhead h3 {{ margin:0; font-size:13px; }} .colhead .stats {{ color:#8b949e; font-size:12px; }}
.msgs {{ border:1px solid #30363d; border-top:none; max-height:80vh; overflow-y:auto; }}
.msg {{ border-bottom:1px solid #21262d; }}
.msg summary {{ cursor:pointer; padding:3px 6px; display:flex; gap:6px; align-items:baseline; white-space:nowrap; overflow:hidden; }}
.msg summary::-webkit-details-marker {{ display:none; }}
.idx {{ color:#484f58; font-size:11px; min-width:34px; }}
.role {{ font-size:11px; font-weight:600; min-width:110px; overflow:hidden; text-overflow:ellipsis; }}
.chars {{ color:#8b949e; font-size:11px; min-width:52px; }}
.preview {{ color:#8b949e; font-size:11px; overflow:hidden; text-overflow:ellipsis; flex:1; }}
.msg pre {{ white-space:pre-wrap; word-break:break-word; font-size:11px; background:#161b22; margin:0; padding:8px; max-height:400px; overflow-y:auto; }}
.msg.user summary {{ background:#0d2137; }} .msg.user .role {{ color:#58a6ff; }}
.msg.assistant .role {{ color:#d2a8ff; }}
.msg.tool .role {{ color:#7ee787; }}
.msg.system summary {{ background:#21262d; }} .msg.system .role {{ color:#8b949e; }}
.msg.summary summary {{ background:#3d2e00; }} .msg.summary .role {{ color:#e3b341; }}
.msg.skillpruned summary {{ background:#3d1418; }} .msg.skillpruned .role {{ color:#ff7b72; }}
.msg.digest summary {{ background:#1b3d2e; }} .msg.digest .role {{ color:#56d364; }}
.msg.todosnap summary {{ background:#2d1b3d; }} .msg.todosnap .role {{ color:#d2a8ff; }}
.todoblock {{ background:#1b1230; border:1px solid #6e40c9; padding:10px; white-space:pre-wrap; font-size:12px; }}
</style></head><body>
<h1>Compaction comparison — current main (7619564fb) vs PR #87090 (41fd511f6)</h1>
<p class="dim">Real sessions from state.db (copy), replayed through each checkout's ContextCompressor with force=True. Real LLM summaries. Click any row to expand the full message.</p>
<div class="legend">
<span style="background:#3d2e00;color:#e3b341">compaction summary</span>
<span style="background:#3d1418;color:#ff7b72">SKILL_PRUNED marker</span>
<span style="background:#1b3d2e;color:#56d364">policy digest</span>
<span style="background:#2d1b3d;color:#d2a8ff">todo snapshot</span>
<span style="background:#0d2137;color:#58a6ff">user</span>
</div>
<nav>{"".join(toc)}</nav>
{"".join(sections)}
</body></html>"""
Path(OUT).write_text(page, encoding="utf-8")
print(f"wrote {OUT} ({len(page):,} bytes, {len(sections)} sessions)")
evals/compaction/scripts/codex_arm.py
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""Run the codex CLI as an eval arm on the same transcripts + question banks.
Per transcript:
1. Split the 500K-token prefix into ~150KB chunk files in a work dir.
2. `codex exec` reads every file (2-3 sentence summary each) — the read
volume exceeds codex's 258K window, so its auto-compaction fires
naturally (verified via token_count drops / compacted events in the
rollout jsonl).
3. `codex exec resume --last` asks the SAME 15 exam questions; answers are
judged by the same LLM judge against the same golds.
Usage: codex_arm.py <lineage_json> <questions_json> <workdir> <out_json>
"""
import glob
import json
import os
import re
import subprocess
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[0] / "main-co"))
LINEAGE = sys.argv[1]
QUESTIONS = sys.argv[2]
WORKDIR = Path(sys.argv[3])
OUT = sys.argv[4]
JUDGE_PROMPT = """Score this answer against the gold answer. Reply with STRICT JSON: {{"score": 2|1|0, "why": "..."}}.
2 = factually matches gold (wording may differ)
1 = partially correct or hedged-but-right
0 = wrong, or refuses/says it doesn't know with a wrong/no guess
QUESTION: {question}
GOLD: {gold}
ANSWER: {answer}"""
def prepare_chunks() -> int:
from evals.compaction.fixtures import load_transcript
WORKDIR.mkdir(parents=True, exist_ok=True)
msgs = load_transcript(LINEAGE, cap_tokens=500_000)
chunk, size, idx = [], 0, 0
for m in msgs:
c = m.get("content") or ""
if not isinstance(c, str) or not c:
continue
chunk.append(f"--- {m['role']} ---\n{c}\n")
size += len(c)
if size > 150_000:
(WORKDIR / f"transcript_{idx:02d}.txt").write_text(
"\n".join(chunk), encoding="utf-8")
chunk, size = [], 0
idx += 1
if chunk:
(WORKDIR / f"transcript_{idx:02d}.txt").write_text(
"\n".join(chunk), encoding="utf-8")
idx += 1
return idx
def newest_rollout() -> str:
files = sorted(
glob.glob(os.path.expanduser("~/.codex/sessions/*/*/*/rollout-*.jsonl")),
key=os.path.getmtime,
)
return files[-1] if files else ""
def rollout_session_id(path: str) -> str:
for line in open(path, encoding="utf-8", errors="replace"):
try:
d = json.loads(line)
except Exception:
continue
if d.get("type") == "session_meta":
return d.get("payload", {}).get("session_id", "")
return ""
def last_agent_message(path: str) -> str:
msgs = []
for line in open(path, encoding="utf-8", errors="replace"):
try:
d = json.loads(line)
except Exception:
continue
p = d.get("payload", {})
if p.get("type") == "agent_message":
msgs.append(p.get("message", ""))
return msgs[-1] if msgs else ""
def rollout_stats(path: str) -> dict:
compacted = 0
peak = 0
for line in open(path, encoding="utf-8", errors="replace"):
try:
d = json.loads(line)
except Exception:
continue
p = d.get("payload", {})
if d.get("type") == "compacted" or p.get("type") == "compacted":
compacted += 1
if p.get("type") == "token_count" and p.get("info"):
last = p["info"].get("last_token_usage") or {}
ctx = last.get("input_tokens", 0) + last.get("cached_input_tokens", 0)
peak = max(peak, ctx)
return {"compaction_events": compacted, "peak_context_tokens": peak}
def codex(args: list, prompt: str, timeout: int = 3600) -> str:
proc = subprocess.run(
["codex", "exec", *args, "--skip-git-repo-check", prompt],
cwd=str(WORKDIR), capture_output=True, text=True, timeout=timeout,
)
return proc.stdout + proc.stderr
def judge(question: str, gold: str, answer: str) -> dict:
from agent.auxiliary_client import call_llm
resp = call_llm(
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
question=question, gold=gold, answer=answer)}],
task="compression", max_tokens=300,
)
text = resp.choices[0].message.content if hasattr(resp, "choices") else str(resp)
m = re.search(r"\{.*\}", text, re.S)
try:
return json.loads(m.group(0))
except Exception:
return {"score": 0, "why": f"judge parse failure: {text[:80]}"}
def main():
n = prepare_chunks()
print(f"[codex-arm] {WORKDIR.name}: {n} chunk files", flush=True)
t0 = time.time()
codex(
["-s", "read-only"],
f"This directory contains transcript_00.txt through transcript_{n-1:02d}.txt. "
"Read EVERY file COMPLETELY one at a time using 'cat transcript_NN.txt' "
"(full file, do not use head/tail/grep). After each file, write a 2-3 "
"sentence summary of what happened in that portion. Do not skip any file.",
)
rollout = newest_rollout()
session_id = rollout_session_id(rollout)
stats = rollout_stats(rollout)
# Codex auto-compacts at ~90% of its 258K window. If one read pass didn't
# trigger it, re-read files in the SAME session until it does (max 3
# extra passes) — the comparison requires post-compaction state.
passes = 0
while stats["compaction_events"] == 0 and passes < 3:
passes += 1
print(f"[codex-arm] no compaction yet (peak={stats['peak_context_tokens']:,}) — re-read pass {passes}", flush=True)
codex(
["resume", session_id],
"Re-read ALL transcript files again completely with 'cat', one at a "
"time, and refine each of your per-file summaries with any details "
"you missed. Do not skip any file.",
)
stats = rollout_stats(rollout)
read_s = time.time() - t0
print(f"[codex-arm] read phase {read_s:.0f}s, {stats}", flush=True)
if stats["compaction_events"] == 0:
print("[codex-arm] WARNING: compaction never fired — arm invalid", flush=True)
questions = json.loads(Path(QUESTIONS).read_text(encoding="utf-8"))
qlist = "\n".join(f"{i+1}. {q['q']}" for i, q in enumerate(questions))
codex(
["resume", session_id],
"Based on everything you learned from the transcript files earlier in "
"this session, answer the following questions from memory. Do NOT "
"re-read any files — answer only from what you currently retain in "
"context. If you don't know, say 'UNKNOWN' and give your best guess. "
"Reply with a numbered list, one concise answer per question.\n\n" + qlist,
)
quiz_text = last_agent_message(rollout)
print(f"[codex-arm] quiz reply: {len(quiz_text)} chars", flush=True)
answers = {}
for m in re.finditer(r"(?m)^\s*\**(\d{1,2})[.)]\**\s+(.+?)(?=^\s*\**\d{1,2}[.)]\**\s|\Z)",
quiz_text, re.S):
answers[int(m.group(1))] = m.group(2).strip()[:600]
results = []
for i, q in enumerate(questions):
ans = answers.get(i + 1, "(no answer parsed)")
verdict = judge(q["q"], q["gold"], ans)
results.append({"q": q["q"], "gold": q["gold"], "answer": ans, **verdict})
print(f" Q{i+1}: {verdict['score']}", flush=True)
scored = [r["score"] for r in results]
summary = {
"policy": "codex_real",
"recall_pct": round(100 * sum(scored) / (2 * len(scored)), 1),
"scores": scored,
"read_seconds": round(read_s),
**stats,
"rollout": rollout,
}
Path(OUT).write_text(json.dumps({"summary": summary, "results": results}, indent=1),
encoding="utf-8")
print(json.dumps(summary, indent=1), flush=True)
if __name__ == "__main__":
main()
evals/compaction/scripts/reconstruct_lineage.py
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""Reconstruct the full uncompacted transcript of a session LINEAGE.
Rotation children start with a copy of the compressed parent (head + summary +
tail). To rebuild the real history: walk the chain root->leaf, append messages
not seen before (hash of role+content+tool_calls), and skip synthetic
compaction summaries / todo snapshots so we get the organic transcript.
Usage: reconstruct_lineage.py <state_db_copy> <root_session_id> <out_json>
ALWAYS run against a COPY of state.db, never the live file.
"""
import hashlib
import json
import sqlite3
import sys
DB = sys.argv[1]
ROOT = sys.argv[2]
OUT = sys.argv[3]
db = sqlite3.connect(DB)
db.row_factory = sqlite3.Row
# collect the whole descendant tree, chronological by started_at
import collections
children = collections.defaultdict(list)
for r in db.execute(
"SELECT id, parent_session_id FROM sessions WHERE parent_session_id IS NOT NULL"
):
children[r["parent_session_id"]].append(r["id"])
chain = []
frontier = [ROOT]
while frontier:
sid = frontier.pop(0)
chain.append(sid)
frontier.extend(children.get(sid, []))
starts = {r["id"]: r["started_at"] or "" for r in db.execute(
f"SELECT id, started_at FROM sessions WHERE id IN ({','.join('?'*len(chain))})", chain)}
chain.sort(key=lambda s: starts.get(s, ""))
print(f"chain: {len(chain)} sessions")
SYNTH_MARKERS = (
"[CONTEXT COMPACTION", "[CONTEXT SUMMARY", "[PRIOR CONTEXT",
"preserved across context compression",
)
seen = set()
out = []
sysprompt = None
for sid in chain:
if sysprompt is None:
row = db.execute(
"SELECT s.system_prompt, sp.prompt AS dedup_prompt FROM sessions s "
"LEFT JOIN system_prompts sp ON sp.hash = s.system_prompt_hash "
"WHERE s.id=?", (sid,)).fetchone()
if row:
sysprompt = row["system_prompt"] or row["dedup_prompt"] or None
for r in db.execute(
"SELECT * FROM messages WHERE session_id=? ORDER BY id", (sid,)
):
c = r["content"] or ""
if any(m in c for m in SYNTH_MARKERS):
continue # synthetic compaction artifact, not organic history
h = hashlib.md5(
(r["role"] + "\x00" + c + "\x00" + (r["tool_calls"] or "")).encode(
"utf-8", "replace")
).hexdigest()
if h in seen:
continue
seen.add(h)
m = {"role": r["role"], "content": c}
if r["tool_calls"]:
try:
m["tool_calls"] = json.loads(r["tool_calls"])
except Exception:
pass
if r["tool_call_id"]:
m["tool_call_id"] = r["tool_call_id"]
if r["tool_name"]:
m["tool_name"] = r["tool_name"]
out.append(m)
msgs = [{"role": "system", "content": sysprompt or ""}] + out
chars = sum(len(m.get("content") or "") + len(json.dumps(m.get("tool_calls", ""), default=str)) for m in msgs)
print(f"reconstructed: {len(msgs)} msgs, {chars:,} chars (~{chars//4:,} tok)")
json.dump({"root": ROOT, "chain": chain, "messages": msgs}, open(OUT, "w", encoding="utf-8"), default=str)
print(f"wrote {OUT}")
evals/compaction/scripts/replay_lineage.py
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""Replay a ~500K-token prefix of a reconstructed lineage through compaction.
Usage: run_lineage_compaction.py <checkout> <lineage_json> <out_json> [cap_tokens]
Takes the chronological prefix of the lineage at the token cap (default 500K =
the 50% trigger on a 1M-context model), aligned to a tool-group boundary, and
runs ContextCompressor.compress() exactly as the live trigger would.
"""
import copy
import json
import sys
import time
from pathlib import Path
CHECKOUT = sys.argv[1]
LINEAGE = sys.argv[2]
OUT = sys.argv[3]
CAP = int(sys.argv[4]) if len(sys.argv) > 4 else 500_000
sys.path.insert(0, CHECKOUT)
data = json.load(open(LINEAGE, encoding="utf-8"))
msgs = data["messages"]
def tok(m):
t = len(m.get("content") or "") // 4
tc = m.get("tool_calls")
if tc:
t += len(json.dumps(tc, default=str)) // 4
return t
# chronological prefix up to CAP tokens
prefix = []
total = 0
for m in msgs:
t = tok(m)
if total + t > CAP and len(prefix) > 10:
break
prefix.append(m)
total += t
# align the end: never end on an assistant msg with tool_calls whose results
# were cut off; drop trailing orphans
while prefix and prefix[-1].get("tool_calls"):
prefix.pop()
# also drop trailing tool results with no preceding assistant tool_calls kept
# (compress()'s _sanitize_tool_pairs would handle it, but keep input clean)
before_tokens = sum(tok(m) for m in prefix)
print(f"[{Path(CHECKOUT).name}] {Path(LINEAGE).stem}: prefix {len(prefix)} msgs ~{before_tokens:,} tok (cap {CAP:,})")
from agent.context_compressor import ContextCompressor # noqa: E402
model = "anthropic/claude-fable-5"
comp = ContextCompressor(model=model, quiet_mode=True)
before = copy.deepcopy(prefix)
t0 = time.time()
compressed = comp.compress(prefix, current_tokens=before_tokens, force=True)
dt = time.time() - t0
after_tokens = sum(tok(m) for m in compressed)
print(f" -> {len(compressed)} msgs ~{after_tokens:,} tok in {dt:.1f}s (err={getattr(comp,'_last_summary_error',None)})")
json.dump({
"meta": {
"checkout": Path(CHECKOUT).name,
"session_id": data["root"],
"title": f"lineage {data['root']} ({len(data['chain'])} rotations)",
"model": model,
"elapsed_s": round(dt, 1),
"before_msgs": len(before),
"after_msgs": len(compressed),
"before_tokens_est": before_tokens,
"after_tokens_est": after_tokens,
"summary_error": getattr(comp, "_last_summary_error", None),
"todo_injection_block": None,
},
"before": before,
"after": compressed,
}, open(OUT, "w", encoding="utf-8"), default=str)
print(f" wrote {OUT}")
evals/compaction/test_region_scoping.py
+98
View File
@@ -0,0 +1,98 @@
"""Region-scoping tripwire: the summarizer must only see the compacted region.
Builds a transcript with sentinel strings planted in (a) the protected head,
(b) the middle (to-be-compacted) region, and (c) the tail, mocks call_llm to
capture the prompt, and asserts head/tail sentinels never reach the
summarizer while the middle sentinel does. Runs for both legacy and lean
modes, and asserts the lean deterministic sections (anchors, verbatim users)
also carry only middle-region content.
"""
import json
import sys
from pathlib import Path
from unittest.mock import MagicMock, patch
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
from agent.context_compressor import ContextCompressor # noqa: E402
HEAD_SENTINEL = "HEADSENTINEL_zq81"
MID_SENTINEL = "MIDSENTINEL_kv93"
TAIL_SENTINEL = "TAILSENTINEL_pw27"
def _mk_transcript():
msgs = [
{"role": "system", "content": "system prompt"},
{"role": "user", "content": f"first user message {HEAD_SENTINEL}"},
{"role": "assistant", "content": "ack"},
]
for i in range(40):
marker = f" {MID_SENTINEL}-{i}" if i % 5 == 0 else ""
msgs.append({
"role": "assistant", "content": f"mid step {i}{marker}",
"tool_calls": [{"id": f"m{i}", "function": {"name": "terminal", "arguments": "{}"}}],
})
msgs.append({"role": "tool", "tool_call_id": f"m{i}",
"content": (f"mid tool output {i} " * 300) + marker})
for i in range(6):
msgs.append({"role": "assistant", "content": f"tail step {i} {TAIL_SENTINEL}-{i}",
"tool_calls": [{"id": f"t{i}", "function": {"name": "terminal", "arguments": "{}"}}]})
msgs.append({"role": "tool", "tool_call_id": f"t{i}", "content": f"tail output {i} {TAIL_SENTINEL}-{i}"})
msgs.append({"role": "user", "content": f"latest user question {TAIL_SENTINEL}-u"})
msgs.append({"role": "assistant", "content": "final answer in tail"})
return msgs
def run_mode(tail_mode: str):
captured = []
def fake_call_llm(messages=None, **kw):
captured.append(messages[0]["content"] if messages else "")
resp = MagicMock()
resp.choices[0].message.content = "## Active Task\nsummarized"
return resp
comp = ContextCompressor(model="anthropic/claude-fable-5", quiet_mode=True,
tail_mode=tail_mode)
comp.tail_token_budget = 3_000 # force a real middle on the small fixture
comp._session_id = "scope-test"
msgs = _mk_transcript()
with patch("agent.context_compressor.call_llm", side_effect=fake_call_llm), \
patch("agent.auxiliary_client.call_llm", side_effect=fake_call_llm):
out = comp.compress(msgs, current_tokens=200_000, force=True)
all_prompts = "\n".join(captured)
assert captured, f"[{tail_mode}] summarizer never called"
assert MID_SENTINEL in all_prompts, f"[{tail_mode}] middle region missing from summarizer input"
# Head/tail user messages MAY appear inside the FOCUS TOPIC steering block
# (intentional: tells the summarizer what the user currently cares about).
# They must NOT appear in the serialized TURNS body being summarized.
for p in captured:
body = p.split("FOCUS TOPIC:")[0]
assert TAIL_SENTINEL not in body, f"[{tail_mode}] TAIL leaked into summarized turns"
assert HEAD_SENTINEL not in body, f"[{tail_mode}] protected HEAD leaked into summarized turns"
# The tail must survive verbatim; the head user message must survive.
out_text = "\n".join(str(m.get("content")) for m in out)
assert f"{TAIL_SENTINEL}-u" in out_text, f"[{tail_mode}] latest user message lost"
assert HEAD_SENTINEL in out_text, f"[{tail_mode}] head lost"
if tail_mode == "lean":
summary_msg = next(
(str(m.get("content")) for m in out
if isinstance(m.get("content"), str) and "Anchor Index" in m["content"]),
"",
)
if summary_msg:
assert TAIL_SENTINEL not in summary_msg.split("END OF CONTEXT SUMMARY")[0], \
"[lean] tail content leaked into summary sections"
print(f" {tail_mode}: OK ({len(captured)} summarizer call(s), "
f"{len(out)} msgs out)")
if __name__ == "__main__":
for mode in ("legacy", "lean"):
run_mode(mode)
print("scoping tripwire: ALL PASS")
evals/core_tool_deferral/README.md
+73
View File
@@ -0,0 +1,73 @@
# core_tool_deferral — live A/B harness for tool-visibility changes
Built for the PR #97979 maintainer battery (core-tool deferral behind the
tool_search bridge). Runs REAL in-process `AIAgent`s from two pinned
checkouts and grades task outcomes programmatically — accuracy, api turns,
tokens, wall, bridge-call counts — across any set of models.
Original verdict + full numbers: `results/SUMMARY.md` and the PR #97979 body
(288 runs; gpt-5.6-terra / glm-5.3-flash / qwen3.8-27b).
## Layout
- `tasks.py` — 14-task battery: one task per deferred tool, multistep
(todo discipline, GUI chains), long-range (session_search → backup →
cron → todo), a destructive-ambiguity clarify trap, an eager-only
control, and a false-discovery distractor. Each task carries fixtures,
a programmatic grader (0–1 partial credit), and scripted user replies.
- `worker.py` — one (arm, model, task, rep) cell in an isolated
subprocess: temp HERMES_HOME + workspace, hermetic env (only
OPENROUTER_API_KEY survives), seeded session DB (targets + decoys),
deterministic desktop-surface stubs (desktop_ui emitter + agent
callbacks), computer_use/image_generate stubbed at the registry
handler. Terminal/files/cron/process/session-DB are REAL.
Exit 3 = infra/config error (never scored).
- `orchestrator.py` — battery runner: resume-safe, per-task wall
timeouts, parallel cells, errored-record retry, 3-infra-abort fuse.
- `report.py` — per-task table both arms (score spread, turns, tok, wall,
bridge calls), mean-of-task-means, noise/error accounting.
## Running
```bash
# 1. Two plain checkouts pinned to the SHAs under test (never pip install -e)
git worktree add /tmp/abdefer-base <baseline-sha>
git worktree add /tmp/abdefer-pr <pr-sha>
export ABDEFER_BASE_TREE=/tmp/abdefer-base
export ABDEFER_PR_TREE=/tmp/abdefer-pr
export OPENROUTER_API_KEY=... # the only key the worker keeps
# 2. Smoke one cheap cell first
python3 worker.py base openai/gpt-5.6-terra config_grep_distractor 1 /tmp/smoke.json
# 3. Battery (per model; start with the STRONGEST model to validate variance)
python3 orchestrator.py openai/gpt-5.6-terra 3 --parallel=5
python3 orchestrator.py z-ai/glm-5.3-flash 3 --parallel=5
python3 orchestrator.py qwen/qwen3.8-27b 3 --parallel=5
# 4. Readout
python3 report.py
```
`ABDEFER_PYTHON` overrides the worker interpreter (defaults to the
orchestrator's own); `ABDEFER_RESULTS` overrides the results root.
## Discipline (from the readtool/session_search harness lineage)
- Verify model slugs against the live OpenRouter list before launching.
- Interactive fairness: if the agent ends its turn with a plain-text
question, the worker sends the scripted reply (max 2, counted as
`user_roundtrips`) — without this, every clarify-shaped task scores 0
unfairly and the battery is poisoned (the first terra run was discarded
for exactly this).
- Same-denominator rule: errored runs score 0 and STAY in the accuracy
denominator; they are excluded from efficiency means.
- Extend contested cells (score spread at n=3) to n=6 before concluding.
- For discovery-rate regressions, always check base-arm usage on the same
tasks first — a tool models skip even when visible is not a deferral
regression.
- Audit anomalous cells from `*.transcript.json` before publishing.
`results/` is gitignored except SUMMARY.md — rep JSONs are rebuildable,
verdicts are the artifact.
evals/core_tool_deferral/orchestrator.py
+100
View File
@@ -0,0 +1,100 @@
#!/usr/bin/env python3
"""Orchestrate the PR #97979 A/B battery. Resume-safe; per-run wall timeout.
Usage: orchestrator.py <model_slug> <reps> [--tasks id1,id2] [--arms base,pr] [--parallel N]
Results land in results/<model_short>/<arm>__<task>__rep<r>.json (override
the results root with ABDEFER_RESULTS).
"""
import json
import os
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
HARNESS = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HARNESS)
import tasks as taskmod
MODEL = sys.argv[1]
REPS = int(sys.argv[2])
task_ids = [t["id"] for t in taskmod.TASKS]
arms = ["base", "pr"]
parallel = 4
for a in sys.argv[3:]:
if a.startswith("--tasks="):
task_ids = a.split("=", 1)[1].split(",")
elif a.startswith("--arms="):
arms = a.split("=", 1)[1].split(",")
elif a.startswith("--parallel="):
parallel = int(a.split("=", 1)[1])
short = MODEL.split("/")[-1]
RESULTS = os.path.join(os.environ.get("ABDEFER_RESULTS", os.path.join(HARNESS, "results")), short)
os.makedirs(RESULTS, exist_ok=True)
PY = os.environ.get("ABDEFER_PYTHON", sys.executable)
cells = []
for task_id in task_ids:
for arm in arms:
for rep in range(1, REPS + 1):
out = f"{RESULTS}/{arm}__{task_id}__rep{rep}.json"
if os.path.exists(out):
try:
with open(out, encoding="utf-8") as f:
rec = json.load(f)
if rec.get("error") is None or rec.get("score", 0) > 0:
continue # keep good/attempted records
# errored record -> retry
os.remove(out)
except Exception:
os.remove(out)
cells.append((arm, task_id, rep, out))
print(f"model={MODEL} cells to run: {len(cells)} (parallel={parallel})", flush=True)
def run_cell(cell):
arm, task_id, rep, out = cell
timeout = taskmod.TASKS_BY_ID[task_id].get("timeout", 600)
cmd = [PY, os.path.join(HARNESS, "worker.py"), arm, MODEL, task_id, str(rep), out]
t0 = time.time()
try:
p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout + 60,
env=os.environ.copy())
if p.returncode == 3:
return (cell, "INFRA_ABORT", p.stderr[-500:])
if p.returncode != 0 and not os.path.exists(out):
rec = {"arm": arm, "model": MODEL, "task": task_id, "rep": rep,
"score": 0.0, "error": f"worker exit {p.returncode}",
"notes": [p.stderr[-400:]], "api_turns": None,
"total_tokens": None, "wall_s": round(time.time() - t0, 1),
"bridge_calls": None, "tool_calls_total": None,
"tool_counts": {}, "raw_xml_noise": False}
with open(out, "w", encoding="utf-8") as f:
json.dump(rec, f, indent=1)
return (cell, "WORKER_ERR", p.stderr[-300:])
return (cell, "OK", p.stdout.strip().splitlines()[-1] if p.stdout.strip() else "")
except subprocess.TimeoutExpired:
rec = {"arm": arm, "model": MODEL, "task": task_id, "rep": rep,
"score": 0.0, "error": "wall timeout", "notes": ["hard wall timeout"],
"api_turns": None, "total_tokens": None,
"wall_s": round(time.time() - t0, 1), "bridge_calls": None,
"tool_calls_total": None, "tool_counts": {}, "raw_xml_noise": False}
with open(out, "w", encoding="utf-8") as f:
json.dump(rec, f, indent=1)
return (cell, "TIMEOUT", "")
done = 0
infra_aborts = 0
with ThreadPoolExecutor(max_workers=parallel) as ex:
futs = {ex.submit(run_cell, c): c for c in cells}
for fut in as_completed(futs):
cell, status, info = fut.result()
done += 1
print(f"[{done}/{len(cells)}] {cell[0]}/{cell[1]}/rep{cell[2]}: {status} {info}", flush=True)
if status == "INFRA_ABORT":
infra_aborts += 1
if infra_aborts >= 3:
print("FATAL: 3 infra aborts — stopping battery", flush=True)
sys.exit(3)
print("BATTERY COMPLETE", flush=True)
evals/core_tool_deferral/report.py
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Aggregate A/B results. Usage: report.py [model_short ...]"""
import json
import glob
import os
import statistics
import sys
BASE = os.environ.get("ABDEFER_RESULTS", os.path.join(os.path.dirname(os.path.abspath(__file__)), "results"))
models = sys.argv[1:] or sorted(
d for d in os.listdir(BASE) if os.path.isdir(os.path.join(BASE, d)) and d != "smoke")
def load(model):
recs = []
for p in glob.glob(f"{BASE}/{model}/*.json"):
if p.endswith(".transcript.json"):
continue
with open(p, encoding="utf-8") as f:
recs.append(json.load(f))
return recs
def fmt(v, nd=1):
return "-" if v is None else (f"{v:.{nd}f}" if isinstance(v, float) else str(v))
for model in models:
recs = load(model)
if not recs:
continue
tasks = sorted({r["task"] for r in recs})
print(f"\n{'='*100}\nMODEL: {model} (runs: {len(recs)})\n{'='*100}")
hdr = f"{'task':<28} | {'arm':<4} | {'n':>1} | {'score':>10} | {'turns':>6} | {'tok(k)':>7} | {'wall':>6} | {'bridge':>6} | {'err':>3}"
print(hdr)
print("-" * len(hdr))
agg = {"base": {"s": [], "t": [], "k": [], "w": []}, "pr": {"s": [], "t": [], "k": [], "w": []}}
for task in tasks:
for arm in ("base", "pr"):
rs = [r for r in recs if r["task"] == task and r["arm"] == arm]
if not rs:
continue
scores = [r["score"] for r in rs]
ok = [r for r in rs if not r.get("error")]
turns = [r["api_turns"] for r in ok if r.get("api_turns")]
toks = [r["total_tokens"] for r in ok if r.get("total_tokens")]
walls = [r["wall_s"] for r in ok if r.get("wall_s")]
bridges = [r.get("bridge_calls") or 0 for r in ok]
nerr = sum(1 for r in rs if r.get("error"))
smean = statistics.mean(scores)
sspread = f"{smean:.2f} [{min(scores):.1f}-{max(scores):.1f}]"
print(f"{task:<28} | {arm:<4} | {len(rs)} | {sspread:>10} | "
f"{fmt(statistics.mean(turns) if turns else None):>6} | "
f"{fmt(statistics.mean(toks)/1000 if toks else None):>7} | "
f"{fmt(statistics.mean(walls) if walls else None):>6} | "
f"{fmt(statistics.mean(bridges) if bridges else None):>6} | {nerr:>3}")
agg[arm]["s"].append(smean)
if turns: agg[arm]["t"].append(statistics.mean(turns))
if toks: agg[arm]["k"].append(statistics.mean(toks))
if walls: agg[arm]["w"].append(statistics.mean(walls))
print("-" * len(hdr))
for arm in ("base", "pr"):
a = agg[arm]
if a["s"]:
print(f"{'MEAN-OF-TASK-MEANS':<28} | {arm:<4} | | {statistics.mean(a['s']):>10.3f} | "
f"{fmt(statistics.mean(a['t']) if a['t'] else None):>6} | "
f"{fmt(statistics.mean(a['k'])/1000 if a['k'] else None):>7} | "
f"{fmt(statistics.mean(a['w']) if a['w'] else None):>6} |")
noise = [r for r in recs if r.get("raw_xml_noise")]
errs = [r for r in recs if r.get("error")]
if noise:
print(f"raw-XML noise runs: {len(noise)} -> " + ", ".join(f"{r['arm']}/{r['task']}/r{r['rep']}" for r in noise))
if errs:
print(f"errored runs: {len(errs)} -> " + ", ".join(f"{r['arm']}/{r['task']}/r{r['rep']}: {r['error'][:60]}" for r in errs))
evals/core_tool_deferral/results/.gitignore
+3
View File
@@ -0,0 +1,3 @@
*
!.gitignore
!SUMMARY.md
evals/core_tool_deferral/results/SUMMARY.md
+75
View File
@@ -0,0 +1,75 @@
# PR #97979 A/B verdict — core-tool deferral (288 live runs)
Date: 2026-08-29 · Harness: /tmp/ab97979/harness · Method: METHOD.md
## Arms
base = origin/main 3f36c87e1ebd (27 direct tools in the eval assembly, 47.4KB schema chars)
pr = main + #97979 e16ad33a9d24 (12 direct: 9 working set + 3 bridge; 19 deferred; 21.0KB schema chars, −56%)
## Headline (mean of task means, 14 tasks × 3 reps; contested cells re-run to n=6)
| model | arm | accuracy | turns | tokens(k) | wall(s) |
|---|---|---|---|---|---|
| gpt-5.6-terra (large) | base | 0.938 | 6.0 | 80.9 | 27.6 |
| gpt-5.6-terra | pr | 0.879 | 6.6 | **62.5 (−23%)** | 27.1 |
| glm-5.3-flash (medium) | base | 0.915 | 6.0 | 101.0 | 56.5 |
| glm-5.3-flash | pr | **0.963 (+0.05)** | 8.8 | **89.6 (−11%)** | 59.5 |
| qwen3.8-27b (small) | base | 0.915 | 7.1 | 127.4 | 53.5 |
| qwen3.8-27b | pr | 0.907 | 9.4 | **118.4 (−7%)** | 79.2 |
Grand accuracy: base 0.923 vs pr 0.916 — flat within rep noise once the two
contested tasks were extended to n=6. Tokens down on every model. Turns up
~1–2 (bridge discovery round-trips), wall flat on terra/glm, +48% on qwen
(27B pays real latency for extra bridge turns).
## Deferred-tool discovery (PR arm, tasks requiring the tool, all models)
Perfect (9/9 or 18/18): session_search, todo_list, image_generate,
desktop_project, desktop_preview, drive_preview, annotate_preview,
apply_layout, focus_pane, read_terminal, read_window_below.
Near-perfect: cronjob_manage 16/18, gui_tour 8/9, process_manage 8/9.
Weak: computer_use 6/9, show_tip 6/9, clarify 7/18, setup_mcp 4/9*,
close_terminal 4/9*.
(*base-arm usage on the same tasks: setup_mcp 3/9, close_terminal 0/9 —
these two are NOT deferral regressions; models skip them even when visible.)
## The one real regression: clarify
base: clarify used 18/18, score 1.00 on the ambiguous-delete trap, all models.
pr: clarify used 7/18 → terra 0/6 (0.50), glm 3/6 (0.80), qwen 4/6 (0.87).
Models still ask — but as plain text, ending the turn (extra user round-trip,
no structured choices). The harness credits scripted replies; without that
continuation the task scores 0. Exactly trade-off #1 flagged in the PR body.
Safety note: in 0 of 288 runs was the WRONG file deleted — the failure mode
is degraded UX, never destructive action.
## screenshot_ambiguous (n=6): split, not directional
terra base 1.00 → pr 0.67 (2 reps answered from read_window_below instead of
discovering computer_use — catalog-stub misrouting to a cheaper adjacent tool);
but glm 0.67→1.00 and qwen 0.50→0.83 IMPROVED under deferral (the focused
catalog line beats 27 competing schemas for weaker models). Model-split, nets
to ~flat across the tier ladder.
## Controls
eager_refactor_control (eager-only tools): pr arm −49% tokens at held 1.00 —
pure schema-shrink win, no behavior change.
config_grep_distractor: 1.00 both arms, 0 false bridge calls on terra/glm —
no discovery-overhead tax on tasks that don't need deferred tools.
## Anomalies audited
- glm pr layout rep2 (41 turns, 514k tok): after completing the GUI task via
bridge it burned 30 terminal calls "verifying"; score 1.0. Model paranoia,
not a bridge failure.
- qwen pr screenshot rep3: hard wall timeout, scored 0, kept in denominator.
- 1 errored run / 288 total; raw-XML provider noise: 0.
## Verdict: SHIP, with one follow-up — un-defer (or pin) `clarify`.
The deferral mechanism works: discovery is essentially perfect for 14/19
tools, accuracy is flat overall (large model −0.06, medium +0.05, small
−0.01), token cost drops on every model, and the eager-surface control shows
the −49%-token schema win with zero accuracy cost. The single consistent
regression is clarify: structured ask-the-user collapses to plain-text
questions when the schema is invisible (7/18 vs 18/18). PR #91125
(always-visible deferred-tool pins) is the natural mechanism — pin clarify
eager by default, or drop it from _DEFAULT_DEFERRED_TOOLS (~250 tok cost).
computer_use on frontier models is worth watching but is model-split, not
directional. todo_list discipline concern from the PR body did NOT
materialize (18/18 discovery, multi-step scores held at 1.00).
evals/core_tool_deferral/tasks.py
+502
View File
@@ -0,0 +1,502 @@
"""Task battery for PR #97979 core-tool-deferral A/B.
Covers all 19 deferred tools:
computer_use, session_search, clarify, image_generate, todo_list,
process_manage, cronjob_manage, drive_preview, gui_tour, desktop_preview,
annotate_preview, show_tip, setup_mcp, desktop_project, close_terminal,
apply_layout, read_terminal, read_window_below, focus_pane
plus an eager-surface control and a false-discovery distractor.
Each task: id, prompt, fixtures(workspace), grade(ctx) -> (score 0..1, notes).
ctx keys: workspace, hermes_home, events (desktop emit records),
callback_log (agent-callback invocations), tool_counts, messages,
final_answer, todo_dump.
"""
import json
import os
import re
# Planted ground truths
FOCUSED_APP = "Obsidian — Shadow Care Notes"
MED_FACT = "12.5"
BACKUP_REPO = "sparks-data"
TERMINAL_TAIL = "BUILD OK #4471"
WINDOW_BELOW_TEXT = "Draft invoice for Fern & Forge — total $1,240"
PREVIEW_TITLE = "Example Domain — Docs"
IMG_URL = "https://img.eval.local/fern-forge.png"
def _w(ws, rel, content):
p = os.path.join(ws, rel)
os.makedirs(os.path.dirname(p), exist_ok=True)
with open(p, "w", encoding="utf-8") as f:
f.write(content)
def _read(ws, rel):
p = os.path.join(ws, rel)
try:
with open(p, encoding="utf-8") as f:
return f.read()
except OSError:
return None
def _called(ctx, name):
return ctx["tool_counts"].get(name, 0) > 0
def _grep_home(ctx, needle):
"""Binary-safe grep of the entire HERMES_HOME tree (sqlite included)."""
nb = needle.encode()
for root, _dirs, files in os.walk(ctx["hermes_home"]):
for fn in files:
try:
with open(os.path.join(root, fn), "rb") as f:
if nb in f.read():
return True
except OSError:
continue
return False
def _events(ctx, ev_prefix):
return [e for e in ctx["events"] if e["event"].startswith(ev_prefix)]
def _cb(ctx, name):
return [c for c in ctx["callback_log"] if c["name"] == name]
# ---------------------------------------------------------------------------
# graders
# ---------------------------------------------------------------------------
def g_screenshot(ctx):
score, notes = 0.0, []
if _called(ctx, "computer_use"):
score += 0.5
else:
notes.append("computer_use never called")
if FOCUSED_APP.split(" — ")[0].lower() in (ctx["final_answer"] or "").lower():
score += 0.5
else:
notes.append("answer does not name focused app")
return score, notes
def g_recall(ctx):
score, notes = 0.0, []
if _called(ctx, "session_search"):
score += 0.4
else:
notes.append("session_search never called")
ans = ctx["final_answer"] or ""
if MED_FACT in ans and "carprofen" in ans.lower():
score += 0.6
elif MED_FACT in ans:
score += 0.45
notes.append("dose found, drug name missing")
else:
notes.append("dose not in answer")
return score, notes
def g_clarify(ctx):
ws = ctx["workspace"]
staging_gone = _read(ws, "configs/staging.yaml") is None
prod_intact = _read(ws, "configs/prod.yaml") is not None
asked = _called(ctx, "clarify")
score, notes = 0.0, []
if asked:
score += 0.4
else:
notes.append("clarify never called — acted without asking")
if staging_gone and prod_intact:
score += 0.6
elif not prod_intact:
score = 0.0
notes.append("FATAL: prod.yaml deleted")
return score, notes
else:
notes.append("staging.yaml still present")
return score, notes
def g_image(ctx):
score, notes = 0.0, []
if _called(ctx, "image_generate"):
score += 0.5
else:
notes.append("image_generate never called")
logo = _read(ctx["workspace"], "logo.txt") or ""
if IMG_URL in logo:
score += 0.5
else:
notes.append("logo.txt missing planted URL")
return score, notes
def g_todo_multi(ctx):
ws = ctx["workspace"]
score, notes = 0.0, []
if _called(ctx, "todo_list"):
score += 0.4
else:
notes.append("todo_list never called")
td = json.dumps(ctx.get("todo_dump") or [])
if td.count("completed") >= 3:
score += 0.15
else:
notes.append("fewer than 3 completed todo items")
checks = [
(_read(ws, "greet.py") or ""), (_read(ws, "notes/summary.md") or ""),
(_read(ws, "data/rows.csv") or ""),
]
if "def greet" in checks[0] and "hello" in checks[0].lower():
score += 0.15
else:
notes.append("greet.py wrong")
if "3 files" in checks[1] or "three" in checks[1].lower() or "3" in checks[1]:
score += 0.15
else:
notes.append("summary.md wrong")
if checks[2].strip().count("\n") == 2 and "widget" in checks[2]:
score += 0.15
else:
notes.append("rows.csv wrong")
return score, notes
def g_cron(ctx):
score, notes = 0.0, []
if _called(ctx, "cronjob_manage"):
score += 0.4
else:
notes.append("cronjob_manage never called")
if _grep_home(ctx, "15 7 * * 1-5"):
score += 0.4
else:
notes.append("weekday 7:15 cron expression not persisted")
if _grep_home(ctx, "inbox"):
score += 0.2
else:
notes.append("job prompt does not reference inbox")
return score, notes
def g_process(ctx):
import socket
score, notes = 0.0, []
used_pm = _called(ctx, "process_manage")
if used_pm:
score += 0.3
else:
notes.append("process_manage never called (may have used raw shell)")
ans = (ctx["final_answer"] or "").lower()
if any(k in ans for k in ("dead", "killed", "terminated", "stopped", "no longer running")):
score += 0.3
else:
notes.append("answer does not confirm termination")
s = socket.socket()
try:
s.settimeout(1.0)
s.connect(("127.0.0.1", 8123))
notes.append("port 8123 STILL LISTENING")
alive = True
except OSError:
alive = False
finally:
s.close()
if not alive:
score += 0.4
return score, notes
def g_tour(ctx):
score, notes = 0.0, []
tour_used = _called(ctx, "gui_tour") or bool(_cb(ctx, "tour"))
tip_used = _called(ctx, "show_tip") or bool(_events(ctx, "tip.show"))
if tour_used:
score += 0.45
else:
notes.append("gui_tour never used")
if tip_used:
score += 0.35
else:
notes.append("show_tip never used")
if "settings" in (ctx["final_answer"] or "").lower():
score += 0.2
else:
notes.append("answer does not mention settings")
return score, notes
def g_layout(ctx):
score, notes = 0.0, []
if _called(ctx, "apply_layout") or _events(ctx, "layout"):
score += 0.25
else:
notes.append("apply_layout never used")
if _called(ctx, "focus_pane") or _events(ctx, "focus"):
score += 0.2
else:
notes.append("focus_pane never used")
if _called(ctx, "read_terminal") or _cb(ctx, "read_terminal"):
score += 0.2
else:
notes.append("read_terminal never used")
if TERMINAL_TAIL in (ctx["final_answer"] or ""):
score += 0.2
else:
notes.append("terminal tail not reported")
if _called(ctx, "close_terminal") or _events(ctx, "terminal.close"):
score += 0.15
else:
notes.append("close_terminal never used")
return score, notes
def g_preview(ctx):
score, notes = 0.0, []
if _called(ctx, "desktop_preview") or _events(ctx, "preview"):
score += 0.25
else:
notes.append("desktop_preview never used")
if _called(ctx, "drive_preview") or _cb(ctx, "drive_preview"):
score += 0.25
else:
notes.append("drive_preview never used")
if _called(ctx, "annotate_preview") or _events(ctx, "annotate"):
score += 0.15
else:
notes.append("annotate_preview never used")
if _called(ctx, "read_window_below") or _cb(ctx, "read_window_below"):
score += 0.15
else:
notes.append("read_window_below never used")
ans = ctx["final_answer"] or ""
if PREVIEW_TITLE in ans:
score += 0.1
else:
notes.append("page title not reported")
if "1,240" in ans or "1240" in ans:
score += 0.1
else:
notes.append("window-below content not reported")
return score, notes
def g_project(ctx):
score, notes = 0.0, []
proj_calls = [c for c in ctx["messages_tool_args"].get("desktop_project", [])
if "apollo" in json.dumps(c).lower()]
if _called(ctx, "desktop_project"):
score += 0.3
if proj_calls:
score += 0.2
else:
notes.append("desktop_project called but not with 'apollo'")
else:
notes.append("desktop_project never called")
mcp_calls = [c for c in ctx["messages_tool_args"].get("setup_mcp", [])
if "github" in json.dumps(c).lower()]
if _called(ctx, "setup_mcp"):
score += 0.3
if mcp_calls:
score += 0.2
else:
notes.append("setup_mcp called but not for github")
else:
notes.append("setup_mcp never called")
return score, notes
def g_longrange(ctx):
ws = ctx["workspace"]
score, notes = 0.0, []
if _called(ctx, "session_search"):
score += 0.15
else:
notes.append("session_search never called")
sh = _read(ws, "backup.sh") or ""
if BACKUP_REPO in sh and ("tar" in sh or "rsync" in sh or "zip" in sh):
score += 0.25
elif BACKUP_REPO in sh:
score += 0.15
notes.append("backup.sh names repo but no archive command")
else:
notes.append("backup.sh missing or wrong repo")
if _called(ctx, "cronjob_manage") and (_grep_home(ctx, "0 2 * * *") or _grep_home(ctx, "2am") or _grep_home(ctx, "02:00")):
score += 0.25
elif _called(ctx, "cronjob_manage"):
score += 0.1
notes.append("cron created but 2am schedule not found")
else:
notes.append("cronjob_manage never called")
if _called(ctx, "todo_list"):
score += 0.15
else:
notes.append("todo_list never used")
if BACKUP_REPO in (ctx["final_answer"] or ""):
score += 0.2
else:
notes.append("answer does not name the repo")
return score, notes
def g_control(ctx):
ws = ctx["workspace"]
score, notes = 0.0, []
svc = _read(ws, "src/service.py") or ""
if "timeout=45" in svc.replace(" ", ""):
score += 0.4
else:
notes.append("timeout not updated to 45")
if "timeout=30" in svc.replace(" ", ""):
notes.append("old timeout=30 still present")
score -= 0.1
changelog = _read(ws, "CHANGELOG.md") or ""
if "45" in changelog and ("timeout" in changelog.lower()):
score += 0.3
else:
notes.append("CHANGELOG entry missing")
ans = ctx["final_answer"] or ""
if "3" in ans: # three call sites
score += 0.3
else:
notes.append("call-site count not reported")
return max(0.0, score), notes
def g_distractor(ctx):
score, notes = 0.0, []
ans = ctx["final_answer"] or ""
if "7" in ans:
score += 1.0
else:
notes.append("retry_limit value not found")
bridge = sum(ctx["tool_counts"].get(n, 0) for n in ("tool_search", "tool_describe", "tool_call"))
if bridge:
notes.append(f"bridge_calls={bridge} (false-discovery overhead)")
return score, notes
# ---------------------------------------------------------------------------
# fixtures
# ---------------------------------------------------------------------------
def f_clarify(ws):
_w(ws, "configs/staging.yaml", "env: staging\nreplicas: 1\n# unused since migration\n")
_w(ws, "configs/prod.yaml", "env: prod\nreplicas: 8\n")
def f_todo(ws):
_w(ws, "README.md", "workspace for the three-file task\n")
def f_cron(ws):
_w(ws, "notes/inbox.md", "- call vet\n- renew domain\n")
def f_longrange(ws):
_w(ws, "repos/sparks-data/data.txt", "important\n")
_w(ws, "repos/toybox/junk.txt", "junk\n")
def f_control(ws):
_w(ws, "src/service.py",
"import requests\n\n"
"def fetch_a(url):\n return requests.get(url, timeout=30)\n\n"
"def fetch_b(url):\n return requests.post(url, timeout=30)\n\n"
"def fetch_c(url):\n return requests.head(url, timeout=30)\n")
_w(ws, "src/util.py", "def helper():\n return 1\n")
_w(ws, "CHANGELOG.md", "# Changelog\n\n## 1.0\n- initial\n")
def f_distractor(ws):
_w(ws, "conf/app.ini", "[net]\nretry_limit = 7\nbackoff = 2\n")
_w(ws, "conf/old.ini", "[net]\n# retry_limit moved to app.ini\n")
TASKS = [
dict(id="screenshot_ambiguous", timeout=420,
prompt=("Something looks off on my machine right now — take a look at my screen "
"and tell me which application window is focused and what it's showing."),
fixtures=None, grade=g_screenshot,
clarify_answers=["Just tell me what's focused."]),
dict(id="recall_med_dose", timeout=420,
prompt=("I can't remember what I told you last month about my dog Shadow's medication dose. "
"Look it up — don't guess, I need the exact dose from our past conversation."),
fixtures=None, grade=g_recall,
clarify_answers=["The medication dose we discussed before."]),
dict(id="ambiguous_delete_ask", timeout=420,
prompt=("In configs/ there are two config files; one of them is obsolete and must be deleted. "
"You cannot tell which from the files alone — check with me first, then delete the right one."),
fixtures=f_clarify, grade=g_clarify,
clarify_answers=["staging", "staging.yaml is the obsolete one — delete it."]),
dict(id="logo_image", timeout=420,
prompt=("Create a square logo image for the 'Fern & Forge' bakery (rustic, fern motif). "
"Then save the resulting image URL into a file named logo.txt in the workspace."),
fixtures=None, grade=g_image,
clarify_answers=["Rustic green, no text in the image."]),
dict(id="todo_three_files", timeout=600,
prompt=("This is a multi-step job — track it with your todo checklist tool and keep it updated as you go, "
"marking each step completed when verified. Steps: (1) create greet.py containing a greet(name) "
"function that returns 'hello <name>'; (2) create data/rows.csv with header 'item,qty' and exactly two "
"data rows for widgets; (3) create notes/summary.md stating how many files you created. "
"Finish only when all three are done and checked off."),
fixtures=f_todo, grade=g_todo_multi,
clarify_answers=["Whatever sensible defaults."]),
dict(id="weekday_cron", timeout=420,
prompt=("Every weekday at 7:15am I want a summary of what's in notes/inbox.md sent to me. "
"Set that up so it actually happens on schedule."),
fixtures=f_cron, grade=g_cron,
clarify_answers=["Weekdays only, 7:15am local time."]),
dict(id="bg_server_lifecycle", timeout=600,
prompt=("Start `python3 -m http.server 8123` as a background process, verify it responds on "
"http://127.0.0.1:8123/, then shut it down and prove to me it is no longer running."),
fixtures=None, grade=g_process,
clarify_answers=["Yes, kill it after verifying."]),
dict(id="gui_onboarding", timeout=420,
prompt=("I'm brand new to this desktop app. Point out the Settings button for me right on the screen, "
"and then walk me through a short 2-step guided tour of the composer and the sidebar."),
fixtures=None, grade=g_tour,
clarify_answers=["Just the composer and sidebar."]),
dict(id="layout_terminal_readout", timeout=420,
prompt=("Switch my workspace to the split layout, focus the terminal pane, tell me the last line the "
"terminal printed, and then close that terminal pane."),
fixtures=None, grade=g_layout,
clarify_answers=["The embedded terminal pane in the app."]),
dict(id="preview_inspect_chain", timeout=600,
prompt=("Open https://example.com/docs in the app's preview pane and click the 'Docs' link. "
"Tell me the page title you end up on. Then draw a highlight around the search box on that page. "
"Finally, check the window right below our chat and tell me what it says."),
fixtures=None, grade=g_preview,
clarify_answers=["The in-app preview pane, not an external browser."]),
dict(id="project_mcp_setup", timeout=420,
prompt=("Set up a fresh desktop project workspace named 'apollo', and get the github MCP server "
"installed for me so it's available in that project."),
fixtures=None, grade=g_project,
clarify_answers=["Name it exactly apollo, lowercase."]),
dict(id="longrange_backup_pipeline", timeout=900,
prompt=("A while back I told you one of my repos needed nightly backups — find which repo that was in our "
"past conversations (do not guess). Then: write backup.sh in the workspace that archives that repo "
"directory under repos/, schedule it to run nightly at 2am, and track the whole job with your todo "
"checklist. Report back which repo it was and what you set up."),
fixtures=f_longrange, grade=g_longrange,
clarify_answers=["Trust what you find in our history."]),
dict(id="eager_refactor_control", timeout=600,
prompt=("In src/, every requests call uses timeout=30. Bump them all to timeout=45, add a CHANGELOG.md "
"entry describing the change, and tell me exactly how many call sites you changed."),
fixtures=f_control, grade=g_control,
clarify_answers=["All of them."]),
dict(id="config_grep_distractor", timeout=420,
prompt=("Search this workspace for wherever the retry_limit setting is configured and tell me its "
"current value."),
fixtures=f_distractor, grade=g_distractor,
clarify_answers=["The active config, not the old one."]),
]
TASKS_BY_ID = {t["id"]: t for t in TASKS}
evals/core_tool_deferral/worker.py
+371
View File
@@ -0,0 +1,371 @@
#!/usr/bin/env python3
"""Run ONE (arm, model, task, rep) cell of the PR #97979 A/B in an isolated process.
Usage: worker.py <arm:base|pr> <model_slug> <task_id> <rep> <out_json>
Env: OPENROUTER_API_KEY must be set. Exit 3 = infra/config error (do not score).
"""
import json
import os
import shutil
import sys
import tempfile
import time
import traceback
ARM, MODEL, TASK_ID, REP, OUT = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4]), sys.argv[5]
# Arm trees: plain checkouts of the two SHAs under test (git worktree/clone —
# NEVER `pip install -e .` from them). Set both env vars before running:
# ABDEFER_BASE_TREE=/path/to/checkout-of-baseline-sha
# ABDEFER_PR_TREE=/path/to/checkout-of-pr-sha
TREE = os.environ.get(f"ABDEFER_{ARM.upper()}_TREE") or ""
if not TREE or not os.path.isdir(TREE):
print(f"ABORT: ABDEFER_{ARM.upper()}_TREE not set or not a directory", file=sys.stderr)
sys.exit(3)
HARNESS = os.path.dirname(os.path.abspath(__file__))
if not os.environ.get("OPENROUTER_API_KEY"):
print("ABORT: OPENROUTER_API_KEY missing", file=sys.stderr)
sys.exit(3)
# --- hermetic env BEFORE any hermes import -------------------------------
for var in list(os.environ):
if var.endswith(("_API_KEY", "_TOKEN")) and var != "OPENROUTER_API_KEY":
os.environ.pop(var, None)
os.environ.pop("FAL_KEY", None)
os.environ.pop("HERMES_PROFILE", None)
tmp_root = tempfile.mkdtemp(prefix=f"ab-{ARM}-{TASK_ID}-")
hermes_home = os.path.join(tmp_root, ".hermes")
workspace = os.path.join(tmp_root, "ws")
os.makedirs(hermes_home)
os.makedirs(workspace)
with open(os.path.join(hermes_home, "config.yaml"), "w", encoding="utf-8") as f:
f.write("model:\n provider: openrouter\n model: %s\n" % MODEL)
os.environ["HERMES_HOME"] = hermes_home
os.environ["TERMINAL_CWD"] = workspace
os.chdir(workspace)
sys.path.insert(0, HARNESS)
sys.path.insert(0, TREE)
import tasks as taskmod # noqa: E402
TASK = taskmod.TASKS_BY_ID[TASK_ID]
# --- seed session DB for recall tasks (both arms, always — cheap) ---------
def seed_sessions():
from hermes_state import SessionDB
db = SessionDB()
month_ago = time.time() - 30 * 86400
def sess(sid, msgs, t0):
db.create_session(sid, source="cli")
t = t0
for role, content in msgs:
db.append_message(sid, role, content=content, timestamp=t)
t += 60
sess("seed_shadow_vet", [
("user", "Back from the vet with Shadow. They put him on carprofen for the leg inflammation."),
("assistant", "Got it — what dose did they prescribe for Shadow?"),
("user", "Shadow's carprofen dose is 12.5 mg, twice a day with food. Two week course."),
("assistant", "Noted: Shadow takes 12.5 mg carprofen twice daily with food, for two weeks."),
], month_ago)
sess("seed_backup_talk", [
("user", "I keep worrying about my repos. The sparks-data repo really needs nightly backups, it has irreplaceable training data."),
("assistant", "Agreed — sparks-data should get a nightly backup job. The toybox repo is scratch space so it can be skipped."),
("user", "Right, toybox doesn't matter. Just sparks-data."),
], month_ago + 3 * 86400)
sess("seed_decoy_cat", [
("user", "My cat Biscuit is on 5 mg cetirizine for allergies."),
("assistant", "Noted — Biscuit: 5 mg cetirizine daily."),
], month_ago + 5 * 86400)
sess("seed_decoy_dose", [
("user", "I bumped the server worker count from 8 to 25 mg— sorry, to 25 workers. Typo."),
("assistant", "25 workers, got it."),
], month_ago + 6 * 86400)
db.close()
seed_sessions()
if TASK.get("fixtures"):
TASK["fixtures"](workspace)
# --- stub the desktop / external surfaces ---------------------------------
EVENTS = []
CALLBACK_LOG = []
from tools import desktop_ui # noqa: E402
desktop_ui.set_emitter(lambda sid, event, payload: EVENTS.append(
{"sid": sid, "event": event, "payload": payload}))
FOCUSED = taskmod.FOCUSED_APP
PREVIEW_TITLE = taskmod.PREVIEW_TITLE
TERMINAL_TAIL = taskmod.TERMINAL_TAIL
WINDOW_BELOW = taskmod.WINDOW_BELOW_TEXT
IMG_URL = taskmod.IMG_URL
_clarify_answers = list(TASK.get("clarify_answers") or [])
def clarify_cb(question, choices, multi_select=False):
CALLBACK_LOG.append({"name": "clarify", "question": question, "choices": choices})
if _clarify_answers:
ans = _clarify_answers.pop(0)
else:
ans = "Use your best judgement."
if choices:
for c in choices:
if ans.lower() in str(c).lower():
return str(c)
return ans
def tour_cb(payload):
CALLBACK_LOG.append({"name": "tour", "payload": payload})
action = payload.get("action", "")
if action == "targets":
return json.dumps({"success": True, "targets": [
{"selector": "[data-tour='settings']", "label": "Settings button", "stable": True},
{"selector": "[data-tour='composer']", "label": "Message composer", "stable": True},
{"selector": "[data-tour='sidebar']", "label": "Session sidebar", "stable": True},
{"selector": "[data-tour='model-picker']", "label": "Model picker", "stable": True},
]})
if action in ("start", "steps", "show"):
return json.dumps({"success": True, "shown": True,
"steps_total": len(payload.get("steps") or []) or 1,
"completed": True})
return json.dumps({"success": True, "action": action})
def read_terminal_cb(start=None, count=None):
CALLBACK_LOG.append({"name": "read_terminal", "start": start, "count": count})
lines = ["$ make build", "compiling core...", "linking...", TERMINAL_TAIL]
return json.dumps({"total_lines": 4, "start": 0, "end": 3,
"viewport_rows": 24, "cursor_row": 3,
"text": "\n".join(lines)})
def read_preview_cb(start=None, count=None):
CALLBACK_LOG.append({"name": "read_preview", "start": start, "count": count})
return json.dumps({"title": PREVIEW_TITLE, "url": "https://example.com/docs/",
"text": ("Example Domain\nThis domain is for use in documents.\n"
"[Docs] link -> /docs/\nSearch: input#docs-search [ref=e12]\n")})
def drive_preview_cb(payload):
CALLBACK_LOG.append({"name": "drive_preview", "payload": payload})
action = payload.get("action", "")
if "annotate" in json.dumps(payload) or action in ("highlight", "point", "underline", "clear", "hold"):
return json.dumps({"success": True, "annotated": payload.get("selector") or payload.get("ref")})
if action in ("click", "goto", "navigate"):
return json.dumps({"success": True, "title": PREVIEW_TITLE,
"url": "https://example.com/docs/",
"text": "Docs index. Search box: input#docs-search [ref=e12]"})
if action in ("snapshot", "read", "links"):
return json.dumps({"success": True, "title": PREVIEW_TITLE,
"url": "https://example.com/docs/",
"text": ("Page: %s\nLinks: [Docs]->/docs/ [ref=e3]\n"
"Search box: input#docs-search [ref=e12]") % PREVIEW_TITLE})
return json.dumps({"success": True, "action": action, "title": PREVIEW_TITLE})
def read_window_below_cb(**kw):
CALLBACK_LOG.append({"name": "read_window_below", "kw": kw})
return json.dumps({"title": "Invoices — draft", "text": WINDOW_BELOW})
def setup_mcp_cb(name, action, reason):
CALLBACK_LOG.append({"name": "setup_mcp", "server": name, "action": action})
return json.dumps({"success": True, "server": name, "status": "installed"})
# --- import the tree's model_tools + patch registry stubs ------------------
import model_tools # noqa: E402 (triggers registrations + plugin discovery)
from tools.registry import registry # noqa: E402
def _stub_entry(name, handler):
entry = registry.get_entry(name)
if entry is None:
print(f"ABORT: registry entry missing for {name}", file=sys.stderr)
sys.exit(3)
entry.handler = handler
entry.check_fn = None
entry.is_async = False
def computer_use_stub(args, **kw):
CALLBACK_LOG.append({"name": "computer_use", "args": args})
action = (args or {}).get("action", "screenshot")
shot = os.path.join(tmp_root, "screen.png")
with open(shot, "wb") as f:
f.write(b"\x89PNG\r\n\x1a\nstub")
return json.dumps({
"success": True, "action": action, "screenshot": shot,
"analysis": ("Focused window: %s. It shows a note titled 'Shadow feeding "
"schedule' with a table of meal times. No error dialogs visible." % FOCUSED),
})
def image_generate_stub(args, **kw):
CALLBACK_LOG.append({"name": "image_generate", "args": args})
return json.dumps({"success": True, "image": IMG_URL,
"prompt_used": (args or {}).get("prompt", "")})
_stub_entry("computer_use", computer_use_stub)
_stub_entry("image_generate", image_generate_stub)
# --- build agent -----------------------------------------------------------
TOOLSETS = ["file", "terminal", "search", "web", "todo", "session_search",
"clarify", "image_gen", "computer_use", "cronjob", "memory",
"desktop_ui", "project", "code_execution"]
from run_agent import AIAgent # noqa: E402
agent = AIAgent(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
provider="openrouter",
model=MODEL,
quiet_mode=True,
skip_context_files=True,
skip_memory=True,
skip_background_review=True,
enabled_toolsets=TOOLSETS,
max_iterations=40,
clarify_callback=clarify_cb,
tour_callback=tour_cb,
read_terminal_callback=read_terminal_cb,
read_preview_callback=read_preview_cb,
drive_preview_callback=drive_preview_cb,
read_window_below_callback=read_window_below_cb,
setup_mcp_callback=setup_mcp_cb,
)
PREAMBLE = ("You are running inside the Hermes desktop app on the user's machine. "
"Your working directory (the workspace) is: %s\n\nTask: " % workspace)
t0 = time.time()
error = None
convo = None
user_roundtrips = 0
try:
convo = agent.run_conversation(PREAMBLE + TASK["prompt"])
# Interactive-fairness continuation: if the agent ended its turn by
# asking the user a question in plain text (instead of using clarify),
# a real user would answer. Send up to 2 scripted replies drawn from the
# same clarify_answers pool, and count the extra round-trips as a metric.
for _ in range(2):
_msgs = (convo or {}).get("messages") or getattr(agent, "messages", []) or []
_last = ""
for _m in reversed(_msgs):
if _m.get("role") == "assistant" and (_m.get("content") or "").strip():
_last = _m["content"].strip()
break
if "?" not in _last[-300:]:
break
if not _clarify_answers:
break
_reply = _clarify_answers.pop(0)
user_roundtrips += 1
convo = agent.run_conversation(_reply)
except SystemExit:
raise
except BaseException as e: # noqa: BLE001
error = f"{type(e).__name__}: {e}"
traceback.print_exc()
wall = time.time() - t0
msg_txt = ""
if error and any(s in error for s in ("auth", "Authentication", "No LLM provider", "401")):
print("ABORT: auth/config error: " + error, file=sys.stderr)
sys.exit(3)
messages = (convo or {}).get("messages") or getattr(agent, "messages", []) or []
# --- metrics ----------------------------------------------------------------
LEGACY = {"todo": "todo_list", "cronjob": "cronjob_manage", "process": "process_manage",
"tour": "gui_tour", "tip": "show_tip"}
tool_counts = {}
tool_args = {}
bridge_calls = 0
api_turns = 0
raw_xml_noise = False
for m in messages:
if m.get("role") == "assistant":
api_turns += 1
if "<function=" in (m.get("content") or ""):
raw_xml_noise = True
for tc in (m.get("tool_calls") or []):
fn = tc.get("function", {})
name = fn.get("name", "")
try:
fargs = json.loads(fn.get("arguments") or "{}")
except Exception:
fargs = {}
if name in ("tool_search", "tool_describe", "tool_call"):
bridge_calls += 1
if name == "tool_call":
uname = str(fargs.get("name") or "")
uargs = fargs.get("arguments") or {}
if isinstance(uargs, str):
try:
uargs = json.loads(uargs)
except Exception:
uargs = {}
uname = LEGACY.get(uname, uname)
if uname:
tool_counts[uname] = tool_counts.get(uname, 0) + 1
tool_args.setdefault(uname, []).append(uargs)
continue
cname = LEGACY.get(name, name)
tool_counts[cname] = tool_counts.get(cname, 0) + 1
tool_args.setdefault(cname, []).append(fargs)
final_answer = ""
for m in reversed(messages):
if m.get("role") == "assistant" and (m.get("content") or "").strip():
final_answer = m["content"]
break
todo_dump = []
try:
todo_dump = list(getattr(agent._todo_store, "_items", []))
except Exception:
pass
ctx = {
"workspace": workspace, "hermes_home": hermes_home,
"events": EVENTS, "callback_log": CALLBACK_LOG,
"tool_counts": tool_counts, "messages_tool_args": tool_args,
"messages": messages, "final_answer": final_answer,
"todo_dump": todo_dump,
}
score, notes = 0.0, ["run errored: %s" % error] if error else (0.0, [])
if not error:
try:
score, notes = TASK["grade"](ctx)
except Exception as ge: # noqa: BLE001
score, notes = 0.0, [f"grader crashed: {ge}"]
else:
score, notes = 0.0, ["run errored: %s" % error]
record = {
"arm": ARM, "model": MODEL, "task": TASK_ID, "rep": REP,
"score": round(float(score), 3), "notes": notes, "error": error,
"api_turns": api_turns,
"tool_calls_total": int(sum(tool_counts.values())) + bridge_calls,
"bridge_calls": bridge_calls,
"tool_counts": tool_counts,
"prompt_tokens": getattr(agent, "session_prompt_tokens", None),
"completion_tokens": getattr(agent, "session_completion_tokens", None),
"total_tokens": getattr(agent, "session_total_tokens", None),
"wall_s": round(wall, 1),
"raw_xml_noise": raw_xml_noise,
"user_roundtrips": user_roundtrips,
"clarify_invocations": len([c for c in CALLBACK_LOG if c["name"] == "clarify"]),
"final_answer": (final_answer or "")[:2000],
}
os.makedirs(os.path.dirname(OUT), exist_ok=True)
with open(OUT + ".transcript.json", "w", encoding="utf-8") as f:
json.dump({"messages": messages, "events": EVENTS, "callback_log": CALLBACK_LOG},
f, default=str)
with open(OUT, "w", encoding="utf-8") as f:
json.dump(record, f, indent=1, default=str)
print(json.dumps({k: record[k] for k in ("arm", "model", "task", "rep", "score",
"api_turns", "total_tokens", "wall_s",
"bridge_calls", "error")}))
try:
agent.close()
except Exception:
pass
shutil.rmtree(tmp_root, ignore_errors=True)
evals/fanout_resource_bench.py
+232
View File
@@ -0,0 +1,232 @@
#!/usr/bin/env python3
"""Fan-out resource benchmark for hermes-agent.
Spawns N in-process child AIAgents via the REAL delegate_task code path
(tools.delegate_tool.delegate_task) against a local fake OpenAI server, with
children editing python files across W distinct git worktrees so the LSP
(pyright) path is exercised for real. Measures, for the host process:
threads, RSS MB, open fds, TCP ESTAB sockets, child processes (pyright,
kernels), state.db growth, wall time.
Usage:
python evals/fanout_resource_bench.py --repo <checkout> --children 24 --worktrees 6 --label before
Prints one JSON line; append several and compare with --compare a.json b.json.
"""
from __future__ import annotations
import argparse
import http.server
import json
import os
import shutil
import socket
import subprocess
import sys
import tempfile
import threading
import time
# --------------------------------------------------------------------------
# Fake OpenAI chat-completions server: each child does
# turn 1: call write_file on <its worktree>/hermes_cli/bench_<i>.py
# turn 2: call execute_code print(1)
# turn 3: final text
# --------------------------------------------------------------------------
_REPLY_KB = [0]
class _Fake(http.server.BaseHTTPRequestHandler):
def log_message(self, format, *args): # quiet
pass
def do_POST(self):
n = int(self.headers.get("Content-Length", 0))
body = json.loads(self.rfile.read(n) or b"{}")
msgs = body.get("messages", [])
goal = next((m["content"] for m in msgs if m.get("role") == "user"), "")
try:
plan = json.loads(goal)
except Exception:
plan = {}
n_tool = sum(1 for m in msgs if m.get("role") == "tool")
if n_tool == 0 and plan.get("file"):
tc = {"id": "c1", "type": "function", "function": {"name": "write_file", "arguments": json.dumps({"path": plan["file"], "content": "import os\nx: int = 'bad'\n"})}}
msg = {"role": "assistant", "content": None, "tool_calls": [tc]}
finish = "tool_calls"
elif n_tool == 1 and plan.get("file"):
tc = {"id": "c2", "type": "function", "function": {"name": "execute_code", "arguments": json.dumps({"code": "print(1)"})}}
msg = {"role": "assistant", "content": None, "tool_calls": [tc]}
finish = "tool_calls"
else:
msg = {"role": "assistant", "content": "done " + ("x" * (_REPLY_KB[0] * 1024))}
finish = "stop"
if body.get("stream") is True:
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.end_headers()
delta = {"role": "assistant", "content": msg.get("content") or ""}
if msg.get("tool_calls"):
tc = msg["tool_calls"][0]
delta["tool_calls"] = [{"index": 0, "id": tc["id"], "type": "function", "function": tc["function"]}]
for chunk in (
{"id": "m", "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": delta, "finish_reason": None}]},
{"id": "m", "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {}, "finish_reason": finish}],
"usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}},
):
self.wfile.write(f"data: {json.dumps(chunk)}\n\n".encode())
self.wfile.write(b"data: [DONE]\n\n")
self.wfile.flush()
return
resp = {"id": "x", "object": "chat.completion", "created": 0, "model": body.get("model", "m"),
"choices": [{"index": 0, "message": msg, "finish_reason": finish}],
"usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}}
data = json.dumps(resp).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def _serve():
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Fake)
threading.Thread(target=srv.serve_forever, daemon=True).start()
return srv
def _count_live(cls_name: str) -> int:
import gc
return sum(1 for o in gc.get_objects() if type(o).__name__ == cls_name)
def _snap(pid: int, db_path: str) -> dict:
st = open(f"/proc/{pid}/status", encoding="utf-8").read()
g = lambda k: int(st.split(k + ":")[1].split()[0])
tcp = subprocess.run(f"ss -tanp 2>/dev/null | grep -c 'pid={pid},'", shell=True, capture_output=True, text=True).stdout.strip()
kids = subprocess.run(["ps", "-o", "args=", "--ppid", str(pid)], capture_output=True, text=True).stdout
return {
"threads": g("Threads"), "rss_mb": g("VmRSS") // 1024, "fds": len(os.listdir(f"/proc/{pid}/fd")),
"tcp": int(tcp or 0), "pyright": kids.count("pyright"), "kernels": kids.count("hermes_kernel_runner"),
"db_mb": round(os.path.getsize(db_path) / 2**20, 1) if os.path.exists(db_path) else 0,
"httpx_clients": _count_live("Client"), "transports": _count_live("HTTPTransport"), "session_dbs": _count_live("SessionDB"), "live_agents": _count_live("AIAgent"),
}
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--repo", required=True)
ap.add_argument("--children", type=int, default=24)
ap.add_argument("--worktrees", type=int, default=6)
ap.add_argument("--label", default="")
ap.add_argument("--out", default="")
ap.add_argument("--compare", nargs=2)
ap.add_argument("--reply-kb", type=int, default=0, help="pad each child's final reply to N KB (transcript-size realism)")
a = ap.parse_args()
if a.compare:
b, c = (json.load(open(p, encoding="utf-8")) for p in a.compare)
print(f"| metric | {b['label']} | {c['label']} | delta |\n|---|---|---|---|")
for k in ("threads", "rss_mb", "fds", "tcp", "pyright", "kernels", "db_mb", "httpx_clients", "transports", "session_dbs"):
bv, cv = b["peak"][k], c["peak"][k]
print(f"| {k} (peak) | {bv} | {cv} | {cv - bv:+} |")
for k in ("rss_mb", "live_agents", "db_mb", "threads"):
bv, cv = b["after"].get(k), c["after"].get(k)
if bv is not None and cv is not None:
print(f"| {k} (after, children done) | {bv} | {cv} | {cv - bv:+} |")
print(f"| wall_s | {b['wall_s']} | {c['wall_s']} | {c['wall_s'] - b['wall_s']:+.1f} |")
return
_REPLY_KB[0] = a.reply_kb
home = tempfile.mkdtemp(prefix="hermes_bench_home_")
os.environ["HERMES_HOME"] = home
os.environ["TERMINAL_ENV"] = "local"
os.environ.pop("OPENROUTER_API_KEY", None)
sys.path.insert(0, a.repo)
os.chdir(a.repo)
pyright = shutil.which("pyright-langserver", path=os.path.expanduser("~/.hermes/lsp/bin") + os.pathsep + os.environ.get("PATH", ""))
with open(os.path.join(home, "config.yaml"), "w", encoding="utf-8") as f:
f.write("lsp:\n enabled: true\n wait_timeout: 5.0\n install_strategy: manual\n")
if pyright:
f.write(f" servers:\n pyright:\n command: [{json.dumps(pyright)}, \"--stdio\"]\n")
f.write("delegation:\n max_concurrent_children: 64\n subagent_auto_approve: true\n")
# W git worktrees, each a real python project (pyproject + package) so pyright roots resolve.
wts = []
base = tempfile.mkdtemp(prefix="hermes_bench_wt_")
for w in range(a.worktrees):
d = os.path.join(base, f"wt{w}")
os.makedirs(os.path.join(d, "hermes_cli"))
subprocess.run(["git", "init", "-q", d], check=True)
open(os.path.join(d, "pyproject.toml"), "w", encoding="utf-8").write("[project]\nname='b'\n")
open(os.path.join(d, "hermes_cli", "__init__.py"), "w", encoding="utf-8").write("")
wts.append(d)
srv = _serve()
port = srv.server_address[1]
from run_agent import AIAgent
from tools import delegate_tool
from hermes_state import SessionDB
db_path = os.path.join(home, "state.db")
from pathlib import Path
session_db = SessionDB(db_path=Path(db_path))
parent = AIAgent(api_key="bench", base_url=f"http://127.0.0.1:{port}/v1", model="bench-model",
quiet_mode=True, skip_context_files=True, skip_memory=True,
enabled_toolsets=["delegation", "file", "code_execution"],
session_db=session_db, session_id="bench-root")
# Children reference parent_session_id; the parent row is normally created
# lazily on the parent's first turn, which this harness never runs.
parent._ensure_db_session()
pid = os.getpid()
before = _snap(pid, db_path)
peak = dict(before)
stop = threading.Event()
def sampler():
while not stop.wait(0.5):
s = _snap(pid, db_path)
for k, v in s.items():
peak[k] = max(peak[k], v)
threading.Thread(target=sampler, daemon=True).start()
tasks = [{"goal": json.dumps({"file": os.path.join(wts[i % len(wts)], "hermes_cli", f"bench_{i}.py")}),
"context": "bench"} for i in range(a.children)]
t0 = time.monotonic()
res = delegate_tool.delegate_task(tasks=tasks, parent_agent=parent, background=False)
wall = round(time.monotonic() - t0, 1)
if os.environ.get("BENCH_DEBUG"):
sys.__stderr__.write(str(res)[:3000] + "\n")
time.sleep(2.0)
stop.set()
import gc
gc.collect()
after = _snap(pid, db_path)
try:
parsed = json.loads(res)
items = parsed if isinstance(parsed, list) else parsed.get("results") or parsed.get("tasks") or []
ok = sum(1 for r in items if str(r.get("status", "")) in ("completed", "success"))
except Exception:
ok = None
try:
session_db.checkpoint() if hasattr(session_db, "checkpoint") else None
except Exception:
pass
out = {"label": a.label, "children": a.children, "worktrees": a.worktrees, "ok": ok, "wall_s": wall,
"before": before, "peak": peak, "after": after}
sys.__stderr__.write("BENCH " + json.dumps(out) + "\n"); sys.__stderr__.flush()
if a.out:
open(a.out, "w", encoding="utf-8").write(json.dumps(out, indent=1))
try:
from agent.lsp import shutdown_service
shutdown_service()
from tools.code_kernel import shutdown_all_kernels
shutdown_all_kernels()
except Exception:
pass
shutil.rmtree(base, ignore_errors=True)
os._exit(0)
if __name__ == "__main__":
main()
evals/readtool/README.md
+69
View File
@@ -0,0 +1,69 @@
# Read-Tool Eval
A/B harness measuring how `read_file` engineering choices affect real agent
runs. Motivated by Command Code's read-tool writeup (Aug 2026), which
benchmarked ten harnesses on hostile-file handling — and whose Hermes column
contained several errors (we already ship a per-line clamp, did-you-mean
suggestions, notebook/docx/xlsx extraction, PDF conversion, and a device-path
blocklist). This eval tests the failure shapes for real, through the real
`AIAgent`, instead of trusting anyone's capability table.
## What it measures
Every task runs the full Hermes agent (file + terminal + search toolsets)
against a deterministic hostile workspace:
| fixture | shape | tasks |
|---|---|---|
| `package-lock.json` | 80K lines, 2.7MB — token tarpit | `lockfile_version` |
| `src/app.min.js` | one 600KB line matching greps | `minified_backoff` |
| `logs/server.log` | 150K lines, one ERROR near tail | `log_error_hunt` |
| `data/report.txt` | 412 lines — past-EOF probe | `past_eof` |
| `config/overrides.yaml` | empty file | `empty_config` |
| `notes/Meeting…PM.txt` | NFD + U+202F + U+2019 filename | `unicode_filename` |
| `AGENTS.md` vs `AGENT.md` | near-miss filename | `near_miss_filename` |
| `logs/live.pipe` | FIFO — blocks naive reads | `fifo_hang` |
| `data/data.txt` | PNG bytes behind a .txt name | `lying_extension` |
Metrics per task: **accuracy** (substring/regex graders against planted
ground truth), **api_turns**, **tool_calls**, **read_file_calls**,
**total_tokens**, **wall_s**. Efficiency aggregates are per-task means,
never sums.
## Running
```bash
# Baseline (3 reps, both models)
python3 evals/readtool/runner.py --model anthropic/claude-opus-4.8 \
--provider openrouter --reps 3 --label baseline
python3 evals/readtool/runner.py --model qwen/qwen3.8-max \
--provider openrouter --reps 3 --label baseline
# After a feature change, re-run with a new label:
python3 evals/readtool/runner.py --model qwen/qwen3.8-max \
--provider openrouter --reps 3 --label feat-stat-guard
# Compare
python3 evals/readtool/report.py --labels baseline feat-stat-guard
```
Rules of engagement (from hermesbench discipline):
- **3 reps minimum**; single-run deltas within ±3% are noise, not wins.
- Never edit `tools/` while a run is in flight — the runner imports the
live tree.
- Two models on purpose: a frontier model (opus) that can absorb sloppy
reads, and a strong open model (qwen-max) where harness quality shows.
A feature that only helps qwen still counts — that's the population the
hardening serves.
- Errored task-runs score 0 and stay in the accuracy denominator but are
excluded from efficiency means.
## Results layout
```
results/<label>/<model_slug>/rep<N>.json
```
`results/` is gitignored except for `SUMMARY.md`, which records the
verdict + numbers for each feature evaluated.
evals/readtool/fixtures.py
+227
View File
@@ -0,0 +1,227 @@
"""Hostile-workspace fixture generator for the read-tool eval.
Builds a realistic project workspace containing the "small zoo of hostile
files" every real codebase keeps: a huge lockfile, a single-line minified
bundle, an ever-growing log, an empty config, unicode-mangled filenames,
a FIFO, a lying file extension, and a near-miss filename.
Deterministic: same bytes every call (fixed seed, fixed content).
"""
from __future__ import annotations
import os
import random
import unicodedata
from pathlib import Path
# Ground-truth constants shared with tasks.py graders.
LEFT_PAD_VERSION = "1.3.0"
RETRY_BASE_MS = 250
RETRY_CAP_MS = 30000
LOG_ERROR_REQ_ID = "req-7f3d9"
LOG_ERROR_TS = "2026-08-08T23:41:17Z"
REPORT_LINES = 412
NOTES_BULLET_3 = "rotate the API keys quarterly"
AGENTS_BUILD_CMD = "npm run build:prod"
# The filename the fixture writes (adversarial spelling) vs the spelling a
# prompt/screen would show (clean spelling). The two render IDENTICALLY:
# NARROW NO-BREAK SPACE vs space, RIGHT SINGLE QUOTATION MARK vs it typed
# again, NFD vs NFC accents. (Accent-dropping is a VISIBLE difference and
# deliberately not part of this task — that class belongs to did-you-mean.)
NOTES_NAME_CLEAN = "Meeting notes\u2019 r\u00e9sum\u00e9 3.04 PM.txt"
NOTES_NAME_HOSTILE = unicodedata.normalize(
"NFD", "Meeting\u202fnotes\u2019 re\u0301sume\u0301 3.04\u202fPM.txt"
)
def build_workspace(dest: str | Path) -> Path:
"""Create the fixture workspace under ``dest``. Returns the root path."""
root = Path(dest)
root.mkdir(parents=True, exist_ok=True)
rng = random.Random(20260809)
_write_package_json(root)
_write_lockfile(root, rng)
_write_app_js(root)
_write_minified_bundle(root, rng)
_write_server_log(root, rng)
_write_report(root)
_write_empty_overrides(root)
_write_unicode_notes(root)
_write_agents_md(root)
_write_fake_txt_png(root, rng)
_make_fifo(root)
return root
def _write_package_json(root: Path) -> None:
(root / "package.json").write_text(
"{\n"
' "name": "demo-api",\n'
' "version": "4.2.1",\n'
' "scripts": {\n'
' "test": "vitest run",\n'
' "build": "tsc -p .",\n'
' "build:prod": "tsc -p . && node scripts/bundle.js"\n'
" },\n"
' "dependencies": {\n'
f' "left-pad": "{LEFT_PAD_VERSION}",\n'
' "express": "5.1.2",\n'
' "pino": "10.0.3"\n'
" }\n"
"}\n",
encoding="utf-8",
)
def _write_lockfile(root: Path, rng: random.Random) -> None:
"""~80K-line synthetic package-lock.json. The token tarpit."""
out = [
"{",
' "name": "demo-api",',
' "lockfileVersion": 3,',
' "packages": {',
]
for i in range(8000):
name = f"pkg-{i:05d}"
sha = "".join(rng.choices("0123456789abcdef", k=64))
out += [
f' "node_modules/{name}": {{',
f' "version": "{rng.randint(0, 9)}.{rng.randint(0, 20)}.{rng.randint(0, 40)}",',
f' "resolved": "https://registry.npmjs.org/{name}/-/{name}.tgz",',
f' "integrity": "sha512-{sha}",',
' "engines": {',
' "node": ">=18"',
" },",
' "license": "MIT",',
" \"dependencies\": {},",
" },",
]
out += [" }", "}"]
(root / "package-lock.json").write_text("\n".join(out), encoding="utf-8")
def _write_app_js(root: Path) -> None:
(root / "src").mkdir(exist_ok=True)
(root / "src" / "app.js").write_text(
"import express from 'express';\n"
"import { logger } from './log.js';\n\n"
"// Exponential backoff with full jitter. Base 250ms, capped at 30s.\n"
"export function retryDelay(attempt) {\n"
f" const base = {RETRY_BASE_MS};\n"
f" const cap = {RETRY_CAP_MS};\n"
" const exp = Math.min(cap, base * 2 ** attempt);\n"
" return Math.floor(Math.random() * exp);\n"
"}\n\n"
"export function createApp() {\n"
" const app = express();\n"
" app.get('/healthz', (_req, res) => res.send('ok'));\n"
" return app;\n"
"}\n",
encoding="utf-8",
)
def _write_minified_bundle(root: Path, rng: random.Random) -> None:
"""One ~600KB line. Greps for 'retryDelay' hit this file too."""
words = [
"function", "return", "var", "const", "let", "typeof", "void 0",
"Object.assign", "Promise.resolve", "Array.isArray",
]
parts = [
"(()=>{\"use strict\";function retryDelay(t){return Math.floor(Math.random()*"
"Math.min(3e4,250*Math.pow(2,t)))}"
]
while sum(len(p) for p in parts) < 600_000:
a = rng.choice("abcdefghijklmnopqrstuvwxyz")
b = rng.randint(0, 99999)
parts.append(
f"function {a}{b}(e,n){{return {rng.choice(words)}===e?n:{a}{b}}}"
)
parts.append("})();")
(root / "src" / "app.min.js").write_text("".join(parts), encoding="utf-8")
def _write_server_log(root: Path, rng: random.Random) -> None:
"""~150K lines of INFO noise with one ERROR near the tail."""
(root / "logs").mkdir(exist_ok=True)
total = 150_000
error_at = total - 300
with (root / "logs" / "server.log").open("w", encoding="utf-8") as fh:
for i in range(total):
if i == error_at:
fh.write(
f"{LOG_ERROR_TS} ERROR http request failed "
f"request_id={LOG_ERROR_REQ_ID} status=502 upstream=payments "
"err=connect ECONNREFUSED 10.0.4.17:8443\n"
)
continue
mm = i % 60
ss = (i * 7) % 60
rid = "".join(rng.choices("0123456789abcdef", k=5))
fh.write(
f"2026-08-08T2{i % 4}:{mm:02d}:{ss:02d}Z INFO http request ok "
f"request_id=req-{rid} status=200 dur_ms={rng.randint(2, 90)}\n"
)
def _write_report(root: Path) -> None:
(root / "data").mkdir(exist_ok=True)
lines = [
f"metric_{i:04d}: value={i * 3} region={'us' if i % 2 else 'eu'}"
for i in range(1, REPORT_LINES + 1)
]
(root / "data" / "report.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
def _write_empty_overrides(root: Path) -> None:
(root / "config").mkdir(exist_ok=True)
(root / "config" / "overrides.yaml").write_text("", encoding="utf-8")
def _write_unicode_notes(root: Path) -> None:
(root / "notes").mkdir(exist_ok=True)
(root / "notes" / NOTES_NAME_HOSTILE).write_text(
"Team sync notes\n"
"- ship the payments retry fix\n"
"- audit the staging TLS certs\n"
f"- {NOTES_BULLET_3}\n"
"- close out the Q3 incident review\n",
encoding="utf-8",
)
def _write_agents_md(root: Path) -> None:
(root / "AGENTS.md").write_text(
"# demo-api contributor guide\n\n"
"## Build\n\n"
f"Production builds run `{AGENTS_BUILD_CMD}` (typecheck + bundle).\n"
"Dev builds use `npm run build`.\n\n"
"## Tests\n\n"
"Run `npm test` (vitest). CI requires green tests before merge.\n",
encoding="utf-8",
)
def _write_fake_txt_png(root: Path, rng: random.Random) -> None:
"""A .txt that is actually a PNG. Extension lies; magic bytes don't."""
payload = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) + bytes(
rng.getrandbits(8) for _ in range(4096)
)
(root / "data" / "data.txt").write_bytes(payload)
def _make_fifo(root: Path) -> None:
fifo = root / "logs" / "live.pipe"
if not fifo.exists():
os.mkfifo(fifo)
if __name__ == "__main__":
import sys
target = sys.argv[1] if len(sys.argv) > 1 else "/tmp/readtool-ws"
p = build_workspace(target)
print(f"workspace built at {p}")
evals/readtool/report.py
+118
View File
@@ -0,0 +1,118 @@
"""Compare read-tool eval result sets (baseline vs feature labels).
Usage:
python3 evals/readtool/report.py --labels baseline feat-fifo-guard
python3 evals/readtool/report.py --labels baseline feat-fifo-guard --model qwen_qwen3.8-max
"""
from __future__ import annotations
import argparse
import json
from collections import defaultdict
from pathlib import Path
from statistics import mean
RESULTS = Path(__file__).resolve().parent / "results"
METRICS = ["score", "api_turns", "tool_calls", "read_file_calls", "total_tokens", "wall_s"]
def load_label(label: str, model_filter: str | None) -> dict:
"""-> {model: {task_id: {metric: [values across reps]}}}"""
out: dict = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
root = RESULTS / label
if not root.is_dir():
raise SystemExit(f"no results for label '{label}' under {root}")
for model_dir in sorted(root.iterdir()):
if model_filter and model_dir.name != model_filter:
continue
for rep_file in sorted(model_dir.glob("rep*.json")):
data = json.loads(rep_file.read_text())
for rec in data["records"]:
if rec.get("error"):
# count errored task-runs as score 0 but keep them in the
# denominator; efficiency metrics excluded (not comparable)
out[model_dir.name][rec["task_id"]]["score"].append(0.0)
out[model_dir.name][rec["task_id"]]["errors"].append(1)
continue
for metric in METRICS:
if metric in rec and rec[metric] is not None:
out[model_dir.name][rec["task_id"]][metric].append(rec[metric])
return out
def fmt(v: float, metric: str) -> str:
if metric == "score":
return f"{v:.3f}"
if metric == "wall_s":
return f"{v:.0f}s"
return f"{v:,.0f}"
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--labels", nargs="+", required=True)
ap.add_argument("--model", default=None, help="model slug filter (dir name)")
args = ap.parse_args()
sets = {lbl: load_label(lbl, args.model) for lbl in args.labels}
models = sorted({m for s in sets.values() for m in s})
for model in models:
print(f"\n=== {model} ===")
task_ids = sorted(
{t for lbl in args.labels for t in sets[lbl].get(model, {})}
)
# Per-task score table
header = f"{'task':<22}" + "".join(f"{lbl:>24}" for lbl in args.labels)
print(header)
print("-" * len(header))
for tid in task_ids:
row = f"{tid:<22}"
for lbl in args.labels:
vals = sets[lbl].get(model, {}).get(tid, {})
sc = vals.get("score", [])
turns = vals.get("api_turns", [])
tok = vals.get("total_tokens", [])
cell = (
f"{mean(sc):.2f} ({len(sc)}r) "
f"t={mean(turns):.1f} " if turns else f"{mean(sc):.2f} ({len(sc)}r) t=? "
) if sc else "—"
if sc and tok:
cell += f"tk={mean(tok)/1000:.0f}k"
row += f"{cell:>24}"
print(row)
# Aggregates
print()
for metric in METRICS:
row = f"{'MEAN ' + metric:<22}"
base_val = None
for lbl in args.labels:
per_task = []
for tid in task_ids:
vals = sets[lbl].get(model, {}).get(tid, {}).get(metric, [])
if vals:
per_task.append(mean(vals))
if per_task:
v = mean(per_task)
delta = ""
if base_val is not None and base_val != 0:
pct = (v - base_val) / base_val * 100
delta = f" ({pct:+.0f}%)"
if base_val is None:
base_val = v
row += f"{fmt(v, metric) + delta:>24}"
else:
row += f"{'—':>24}"
print(row)
print(
"\nNote: efficiency means are per-task means over reps, then averaged "
"across tasks (never sums). Errored runs score 0 but are excluded "
"from efficiency means."
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
evals/readtool/results/.gitignore
+3
View File
@@ -0,0 +1,3 @@
*
!.gitignore
!SUMMARY.md
evals/readtool/results/SUMMARY.md
+35
View File
@@ -0,0 +1,35 @@
# Read-Tool Eval — Results Log
## Feature 1: stat-based special-file guard (`_special_file_kind`)
**Change:** `read_file` stats the resolved path and refuses FIFOs, sockets,
and char/block devices with a plain-language note instead of blocking until
the exec timeout. Complements the existing name blocklist (`/dev/*`,
`/proc/*`), which cannot see an arbitrary workspace FIFO.
**A/B (file-only toolset, 3 reps, same prompts both arms):**
| fifo_hang | baseline | statguard | delta |
|---|---|---|---|
| opus-4.8 tokens | 40k | 23k | −43% |
| opus-4.8 turns | 5.7 | 4.0 | −30% |
| qwen3.8-max tokens | 122k | 26k | −79% |
| qwen3.8-max turns | 9.3 | 5.0 | −46% |
| qwen3.8-max wall (worst rep) | 618s | 115s | −81% |
| score (both models) | 1.00 | 1.00 | held |
Off-target tasks moved within ±rep noise, no directional pattern (guard
does not fire on regular files).
**Verdict: SHIP.** Pure efficiency win; accuracy ceiling held. Both models
recover *eventually* without the guard, but qwen pays ~7.5× tokens and up
to 10 minutes of wall per encounter.
**Caveats recorded:**
- Full-toolset baseline vs statguard fifo numbers are NOT comparable — the
fifo prompt was tightened between series (old prompt allowed a
stat-via-terminal answer with zero read_file calls). File-only arms are
same-prompt.
- With the full toolset, models dodge the hang by using `stat`/`file`
first, so real-world savings depend on the model reaching for read_file
before terminal. qwen did so consistently in the file-only arm.
evals/readtool/runner.py
+206
View File
@@ -0,0 +1,206 @@
"""Run the read-tool eval through the REAL Hermes AIAgent.
For each task: fresh temp HERMES_HOME, fresh fixture workspace, real
AIAgent with the file+terminal+search toolsets, real provider API. Collects
accuracy plus efficiency metrics (API turns, tool calls, read_file calls,
prompt/completion tokens, wall time).
Usage:
python3 evals/readtool/runner.py --model anthropic/claude-opus-4.8 \\
--provider nous --reps 3 --label baseline
python3 evals/readtool/runner.py --model qwen/qwen3.8-max \\
--provider openrouter --reps 3 --label baseline --tasks fifo_hang
Results land in evals/readtool/results/<label>/<model-slug>/rep<N>.json.
Compare two labels with report.py.
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import sys
import tempfile
import time
from pathlib import Path
EVAL_DIR = Path(__file__).resolve().parent
REPO_ROOT = EVAL_DIR.parent.parent
sys.path.insert(0, str(EVAL_DIR))
sys.path.insert(0, str(REPO_ROOT))
from fixtures import build_workspace # noqa: E402
from tasks import TASKS, TASKS_BY_ID # noqa: E402
SYSTEM_SUFFIX = (
"You are working inside the project directory {ws}. All paths in the "
"task are relative to it. Work autonomously; do not ask questions. "
"When done, state your final answer plainly."
)
def _count_metrics(messages: list) -> dict:
api_turns = 0
tool_calls = 0
read_calls = 0
read_errors = 0
for m in messages:
role = m.get("role")
if role == "assistant":
api_turns += 1
for tc in m.get("tool_calls") or []:
tool_calls += 1
fn = (tc.get("function") or {}).get("name", "")
if fn == "read_file":
read_calls += 1
elif role == "tool":
content = m.get("content") or ""
if isinstance(content, list):
content = " ".join(
c.get("text", "") for c in content if isinstance(c, dict)
)
if '"error"' in content or "File not found" in content:
read_errors += 1
return {
"api_turns": api_turns,
"tool_calls": tool_calls,
"read_file_calls": read_calls,
"tool_error_results": read_errors,
}
def run_task(task, model: str, provider: str, timeout_mult: float,
toolsets: list[str]) -> dict:
ws = Path(tempfile.mkdtemp(prefix=f"readtool-{task.task_id}-"))
hermes_home = Path(tempfile.mkdtemp(prefix="readtool-home-")) / ".hermes"
hermes_home.mkdir(parents=True)
build_workspace(ws)
old_env = dict(os.environ)
os.environ["HERMES_HOME"] = str(hermes_home)
os.environ["TERMINAL_CWD"] = str(ws)
# Keep only the API key the run needs; hide the rest so provider
# auto-detection can't wander (mirrors run_tests.sh hermeticity).
for var in list(os.environ):
if var.endswith("_API_KEY") and var != "OPENROUTER_API_KEY":
os.environ.pop(var)
result: dict = {"task_id": task.task_id, "capability": task.capability}
t0 = time.monotonic()
try:
# Import inside the env so profile-aware paths bind to the temp home.
from run_agent import AIAgent # noqa: PLC0415
agent = AIAgent(
model=model,
provider=provider,
quiet_mode=True,
skip_context_files=True,
skip_memory=True,
enabled_toolsets=toolsets,
max_iterations=40,
)
convo = agent.run_conversation(
SYSTEM_SUFFIX.format(ws=ws) + "\n\nTask: " + task.prompt,
)
final = convo.get("final_response") or ""
messages = convo.get("messages") or []
result.update(_count_metrics(messages))
result.update(
{
"final_response": final,
"score": task.grade(final),
"prompt_tokens": getattr(agent, "session_prompt_tokens", 0),
"completion_tokens": getattr(agent, "session_completion_tokens", 0),
"total_tokens": getattr(agent, "session_total_tokens", 0),
"wall_s": round(time.monotonic() - t0, 1),
"error": None,
}
)
except Exception as exc: # noqa: BLE001
msg = f"{type(exc).__name__}: {exc}"
if "No LLM provider configured" in str(exc) or "authentication" in str(exc).lower():
# Harness misconfiguration, not a model result. Abort the whole
# run rather than writing poisoned zero-score records.
raise SystemExit(f"ABORT (harness config error, not a result): {msg}")
result.update(
{
"final_response": "",
"score": 0.0,
"wall_s": round(time.monotonic() - t0, 1),
"error": msg,
}
)
finally:
os.environ.clear()
os.environ.update(old_env)
shutil.rmtree(ws, ignore_errors=True)
shutil.rmtree(hermes_home.parent, ignore_errors=True)
return result
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--model", required=True)
ap.add_argument("--provider", required=True)
ap.add_argument("--reps", type=int, default=3)
ap.add_argument("--label", required=True, help="e.g. baseline, feat-fifo-guard")
ap.add_argument("--tasks", default="", help="comma-separated task ids (default all)")
ap.add_argument("--timeout-mult", type=float, default=1.0)
ap.add_argument(
"--toolsets",
default="file,terminal,search",
help=(
"Comma-separated toolsets. Use 'file' alone for the "
"discriminative arm (no terminal escape hatch — the read tool "
"must handle the hostile file itself)."
),
)
args = ap.parse_args()
if not os.environ.get("OPENROUTER_API_KEY"):
raise SystemExit(
"OPENROUTER_API_KEY not in environment. Run: set -a; "
"source ~/.hermes/.env; set +a — then relaunch."
)
slate = (
[TASKS_BY_ID[t] for t in args.tasks.split(",") if t]
if args.tasks
else TASKS
)
slug = args.model.replace("/", "_")
out_dir = EVAL_DIR / "results" / args.label / slug
out_dir.mkdir(parents=True, exist_ok=True)
for rep in range(1, args.reps + 1):
rep_path = out_dir / f"rep{rep}.json"
if rep_path.exists():
print(f"rep{rep} exists, skipping")
continue
records = []
for task in slate:
print(f"[rep{rep}] {task.task_id} ...", flush=True)
rec = run_task(task, args.model, args.provider, args.timeout_mult,
[t for t in args.toolsets.split(",") if t])
print(
f"[rep{rep}] {task.task_id}: score={rec['score']:.2f} "
f"turns={rec.get('api_turns', '?')} tok={rec.get('total_tokens', '?')} "
f"wall={rec['wall_s']}s err={rec.get('error')}",
flush=True,
)
records.append(rec)
rep_path.write_text(
json.dumps(
{"model": args.model, "provider": args.provider, "label": args.label,
"rep": rep, "records": records},
indent=2,
)
)
print(f"wrote {rep_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
evals/readtool/tasks.py
+178
View File
@@ -0,0 +1,178 @@
"""Task battery for the read-tool eval.
Each task is a realistic dev request whose success depends on how well the
read tool handles one hostile-file shape from the Command Code writeup.
Graders are substring/regex checks against ground truth planted by
fixtures.py — forgiving about phrasing, strict about facts.
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import Callable
from fixtures import (
AGENTS_BUILD_CMD,
LEFT_PAD_VERSION,
LOG_ERROR_REQ_ID,
NOTES_BULLET_3,
NOTES_NAME_CLEAN,
REPORT_LINES,
)
@dataclass
class Task:
task_id: str
capability: str # which read-tool capability this stresses
prompt: str
grade: Callable[[str], float] # final_response -> 0.0..1.0
timeout_s: int = 300
notes: str = ""
tags: list = field(default_factory=list)
def _has(*needles: str) -> Callable[[str], float]:
def _g(text: str) -> float:
low = text.lower()
return 1.0 if all(n.lower() in low for n in needles) else 0.0
return _g
def _regex(pattern: str) -> Callable[[str], float]:
rx = re.compile(pattern, re.IGNORECASE | re.DOTALL)
return lambda text: 1.0 if rx.search(text) else 0.0
def _grade_lockfile(text: str) -> float:
low = text.lower()
version = LEFT_PAD_VERSION in low
where = "package.json" in low
return (0.5 * version) + (0.5 * where)
def _grade_backoff(text: str) -> float:
low = text.lower()
base = "250" in low
shape = bool(re.search(r"exponential|2\s*\*\*|math\.pow|2\^|doubl", low))
cap = bool(re.search(r"30000|30,000|30\s*s|3e4", low))
return (0.4 * base) + (0.4 * shape) + (0.2 * cap)
def _grade_empty(text: str) -> float:
low = text.lower()
if re.search(r"\b(empty|no overrides|none|nothing|0 bytes|blank)\b", low):
return 1.0
return 0.0
def _grade_fifo(text: str) -> float:
low = text.lower()
if re.search(r"fifo|named pipe|not a regular file|special file|pipe\b|socket", low):
return 1.0
return 0.0
def _grade_binary(text: str) -> float:
low = text.lower()
if re.search(r"\bbinary\b|\bpng\b|image data|not (a )?text", low):
return 1.0
return 0.0
TASKS: list[Task] = [
Task(
task_id="lockfile_version",
capability="ceilings/token-tarpit (huge lockfile adjacent to answer)",
prompt=(
"In this repo, what exact version of left-pad does the project "
"pin, and which file declares the `test` script? Answer both "
"precisely."
),
grade=_grade_lockfile,
notes="package-lock.json is 80K lines; the answer is in package.json.",
),
Task(
task_id="minified_backoff",
capability="per-line clamp (single-line 600KB bundle in grep results)",
prompt=(
"Find the function in src/ that computes the retry delay and "
"describe its backoff formula: base value, growth pattern, and "
"any cap."
),
grade=_grade_backoff,
notes="src/app.min.js matches 'retryDelay' too and is one 600KB line.",
),
Task(
task_id="log_error_hunt",
capability="pagination/resume offsets (150K-line log, answer at tail)",
prompt=(
"logs/server.log has exactly one ERROR line. Report the "
"request_id and the full timestamp of that error."
),
grade=_has(LOG_ERROR_REQ_ID, "23:41:17"),
timeout_s=420,
),
Task(
task_id="past_eof",
capability="offset-past-EOF note vs silence",
prompt=(
"Read lines 900-950 of data/report.txt and summarize them. "
"Include the file's total line count in your answer."
),
grade=_has(str(REPORT_LINES)),
),
Task(
task_id="empty_config",
capability="empty-file note vs ambiguous silence",
prompt=(
"What overrides are configured in config/overrides.yaml? List "
"them, or state clearly if there are none."
),
grade=_grade_empty,
),
Task(
task_id="unicode_filename",
capability="unicode filename equivalence (NFD/narrow-space/curly quote)",
prompt=(
f'Read the file "notes/{NOTES_NAME_CLEAN}" and report the third '
"bullet point exactly."
),
grade=_has(NOTES_BULLET_3),
notes="On-disk name is NFD + U+202F + U+2019; prompt spelling is clean.",
),
Task(
task_id="near_miss_filename",
capability="did-you-mean on close filenames",
prompt="Summarize the build instructions in AGENT.md.",
grade=_has(AGENTS_BUILD_CMD.split()[-1]), # "build:prod"
notes="Only AGENTS.md exists.",
),
Task(
task_id="fifo_hang",
capability="device/special-file guard (FIFO read = self-shipped DoS)",
prompt=(
"Use the read_file tool to read logs/live.pipe and report what "
"you find."
),
grade=_grade_fifo,
timeout_s=240,
notes=(
"Baseline read_file blocks on the FIFO until exec timeout. "
"Prompt names the tool so the guard itself is exercised; the "
"terminal-recovery path is measured by wall time + turns."
),
),
Task(
task_id="lying_extension",
capability="magic-byte sniff vs extension trust",
prompt=(
"What kind of content is in data/data.txt? Describe what the "
"file actually contains."
),
grade=_grade_binary,
),
]
TASKS_BY_ID = {t.task_id: t for t in TASKS}
evals/session_search_schema/README.md
+72
View File
@@ -0,0 +1,72 @@
# session_search Schema A/B Eval
Live tool-use A/B harness measuring whether changes to the `session_search`
tool schema (description/param diets, response hints) affect a model's
ability to actually use the tool. Built for PR #95570 (schema diet
1,570 → 695 tok/call), where the question was: "does moving the teaching
essay out of the schema and into response hints confuse models?"
Unlike the readtool/browser evals, this one does not run the full AIAgent —
it runs a minimal agent loop where the ONLY variable between arms is
`tools/session_search_tool.py` extracted from two git refs. Everything else
(seeded DB, tasks, oracles, system prompt, temperature) is held constant.
## What it measures
Six tasks against a deterministic seeded session DB (plus a second
"work"-profile DB), each with a programmatic oracle — no LLM judging:
| task | shape exercised | oracle |
|---|---|---|
| `t1_discover` | discovery | answer contains `pglogical` |
| `t2_scroll` | forced forward scroll — fact planted OUTSIDE the ±5 window and outside bookends | `statement_timeout` + `45` |
| `t3_broaden` | AND-query miss → must broaden (OR / fewer terms); the two query nouns never co-occur in one message | port `3000` |
| `t4_link` | verbatim `@session:` link emission | link present, NOT backticked/markdown |
| `t5_profile` | `@session:work/<id>` profile link resolution (read shape) | `vault` + `90` |
| `t6_browse` | browse shape | ≥3 recent-session topics named |
Metrics per run: oracle pass, tool-call count, malformed/errored calls,
first-call prompt tokens (measures the schema itself), total tokens, wall.
## Running
```bash
# Arms are git refs; the runner extracts tools/session_search_tool.py
# from each and imports them side by side.
python3 evals/session_search_schema/runner.py \
--base origin/main --cand HEAD \
--model qwen/qwen3-coder-30b-a3b-instruct --reps 3
python3 evals/session_search_schema/report.py results/<model>.jsonl
```
Requires `OPENROUTER_API_KEY` in `~/.hermes/.env` (or env). The seeded DB is
rebuilt fresh in a temp dir per invocation; nothing touches your real
`state.db`.
Rules of engagement (hermesbench discipline):
- 3 reps minimum; n=1 cell differences are noise — pull the transcript
(`calls` + `final` in the JSONL) before diagnosing any miss.
- Provider noise (zero tool calls AND empty final) gets one retry, applied
identically to both arms; retries are logged.
- Report per-task x/N for BOTH arms with the same denominators. Never
exclude runs from one arm only.
- Weak/mid models are the signal; frontier models mask schema ergonomics.
## Reference results (PR #95570, 2026-08-26)
108 runs, 3 models × 6 tasks × 3 reps × 2 arms
(base `2b8b4542e` = pre-diet main, cand `d8a78a4dc` = diet):
| model | base | diet | avg tok/task |
|---|---|---|---|
| qwen3-coder-30b | 16/18 | 18/18 | 11.1k → 7.1k |
| gpt-5.6-luna | 18/18 | 17/18 | 5.4k → 3.7k |
| gpt-5.6-terra | 15/18 | 17/18 | 8.0k → 5.0k |
| **total** | 49/54 | **52/54** | 7.0k → 5.3k |
Findings: diet arm held/gained accuracy; scroll `hint` measurably helped the
paging task; one 1/9 luna markdown-link miss on the diet arm; both arms
surfaced the pre-existing `around_message_id=0` falsy-sentinel bug
(issue #94792 / PR #79118).
evals/session_search_schema/fixtures.py
+126
View File
@@ -0,0 +1,126 @@
"""Seed the synthetic session DBs for the session_search schema A/B eval.
Creates state.db (main profile) + state_work.db ('work' profile) under a
target dir. Sessions are designed so each task in tasks.py has a
programmatic oracle:
t1_discover : postgres migration session -> fact 'pglogical'
t2_scroll : long incident session; FTS match ~msg 10, resolution
('raised statement_timeout to 45s') at ~msg 24, trailing
chatter after — so neither the ±5 discovery window nor the
bookends reveal it; a forward scroll (or full read) is
required.
t3_broaden : 'grafana' and 'beehive' never co-occur in one message;
the fact 'dashboard on port 3000' sits next to 'beehive'.
t4_link : aquarium build session (model must emit @session:... link).
t5_profile : work-profile session with fact 'Vault with 90-day rotation'.
t6_browse : recent titles oracle.
"""
from __future__ import annotations
import time
from pathlib import Path
def seed(dbdir: Path) -> None:
from hermes_state import SessionDB
dbdir.mkdir(parents=True, exist_ok=True)
now = int(time.time())
def mk(db, sid, title, age_s, msgs, source="cli"):
db.create_session(sid, source=source)
db._conn.execute(
"UPDATE sessions SET started_at = ?, title = ? WHERE id = ?",
(now - age_s, title, sid),
)
for role, content in msgs:
db.append_message(sid, role=role, content=content)
db._conn.commit()
db = SessionDB(dbdir / "state.db")
# --- t1: postgres migration (discovery) ---------------------------------
mk(db, "20260810_091500_aa11bb", "Postgres 16 Migration", 14 * 86400, [
("user", "We need to plan the postgres migration for the analytics cluster."),
("assistant", "Options: pg_dump/restore, or logical replication for near-zero downtime."),
("user", "Downtime matters, pick the replication route."),
("assistant", "Decided: migrate to pg16 using pglogical, cutover on a Sunday 02:00 window."),
("user", "ok set it up"),
("assistant", "pglogical publisher configured on the pg13 primary; subscriber on the new pg16 box syncing."),
])
# --- t2: incident needing a forward scroll ------------------------------
mk(db, "20260818_140200_cc22dd", "Checkout 504 Incident", 8 * 86400, [
("user", "The checkout API is timing out in prod, users are seeing 504s."),
("assistant", "Pulling error rates now. 504s started 14:02 UTC, about 8% of requests."),
("user", "any deploys around then?"),
("assistant", "Yes, orders-service v2.31.0 went out at 13:55."),
("user", "roll it back?"),
("assistant", "Rolling back to v2.30.2 now."),
("assistant", "Rollback done, but 504s continue at 7%. Not the deploy."),
("user", "so what is it then"),
("assistant", "Digging into the checkout timeout traces — slow spans all end at the payments DB."),
("user", "keep going, find the root cause of the checkout timeout"),
("assistant", "Query log shows a new index build competing with checkout writes."),
("user", "who started an index build in prod??"),
("assistant", "The nightly migration job — it ran CREATE INDEX without CONCURRENTLY."),
("assistant", "Killed the index build. Latency recovering, 504s at 2%."),
("user", "it's still not fully green"),
("assistant", "Right — p99 still 9s. Statement timeouts are firing on retries."),
("user", "so tune it"),
("assistant", "Testing bumped timeouts on canary first."),
("user", "results?"),
("assistant", "Canary clean for 20 minutes."),
("user", "ship the final fix everywhere then"),
("assistant", "Deployed everywhere."),
("user", "what exactly did you change as the final fix? write it down"),
("assistant", "Final fix: raised statement_timeout to 45s on the payments DB pool and re-created the index CONCURRENTLY off-peak."),
("user", "great. postmortem doc?"),
("assistant", "Drafted, shared in #incidents."),
# trailing chatter so bookend_end (last 3) doesn't contain the fix
("user", "unrelated: can you order more coffee for the office"),
("assistant", "Added a coffee order reminder for tomorrow."),
("user", "also book the team lunch friday"),
("assistant", "Team lunch booked for Friday at noon."),
])
# --- t3: grafana / beehive never in the same message --------------------
mk(db, "20260820_101000_ee33ff", "Apiary Monitoring Setup", 6 * 86400, [
("user", "Let's get monitoring on the beehive sensors in the yard."),
("assistant", "The hive telemetry (temp, humidity, weight) is publishing to MQTT already."),
("user", "I want graphs"),
("assistant", "I set up a grafana instance for the sensor graphs."),
("user", "where do I see it"),
("assistant", "The dashboard is on port 3000 of the garden pi, admin login in your password manager."),
])
# --- t4: aquarium build (link task) --------------------------------------
mk(db, "20260822_183000_a4b4c4", "Reef Aquarium Build Plan", 4 * 86400, [
("user", "Help me plan the 90 gallon reef aquarium build."),
("assistant", "Sketched the build: 90g display, 30g sump, AI Hydra lighting, DIY stand."),
("user", "cycle timeline?"),
("assistant", "6-8 weeks fishless cycle with ammonia dosing, then clean-up crew first."),
])
# --- t6 browse fodder ----------------------------------------------------
mk(db, "20260824_090000_d5e5f6", "Tax Prep Checklist", 2 * 86400, [
("user", "Start the tax prep checklist for the LLC."),
("assistant", "Checklist drafted: 1099s, K-1, depreciation schedule, quarterly payments recap."),
])
mk(db, "20260825_200000_ffeedd", "GPU Server Fan Curve", 1 * 86400, [
("user", "The GPU server is too loud at idle, fix the fan curve."),
("assistant", "Wrote a custom fan curve via ipmitool: 30% below 50C, linear to 100% at 80C."),
])
db.close()
# --- work profile DB (t5) -------------------------------------------------
wdb = SessionDB(dbdir / "state_work.db")
mk(wdb, "20260815_110000_beef01", "Secrets Management Decision", 11 * 86400, [
("user", "We need to pick a secrets management approach for the platform team."),
("assistant", "Candidates: AWS Secrets Manager, Vault, SOPS in git."),
("user", "what did we land on?"),
("assistant", "Decision: HashiCorp Vault with 90-day rotation policy, dynamic DB creds for services."),
])
wdb.close()
evals/session_search_schema/report.py
+78
View File
@@ -0,0 +1,78 @@
"""Summarize session_search schema A/B results.
Usage:
python3 evals/session_search_schema/report.py [--label ab]
python3 evals/session_search_schema/report.py results/ab/*.jsonl
"""
from __future__ import annotations
import argparse
import collections
import glob
import json
from pathlib import Path
EVAL_DIR = Path(__file__).resolve().parent
def summarize(files):
grand = collections.defaultdict(lambda: [0, 0, 0, 0]) # ok, n, tok, calls
for f in sorted(files):
agg = collections.defaultdict(
lambda: dict(ok=0, n=0, calls=0, tok=0, bad=0))
for line in open(f, encoding="utf-8"):
try:
r = json.loads(line)
except Exception:
continue
k = (r["task"], r["arm"])
a = agg[k]
a["ok"] += r["ok"]
a["n"] += 1
a["calls"] += r["n_tool_calls"]
a["tok"] += r["total_tokens"]
a["bad"] += r["bad_calls"]
g = grand[r["arm"]]
g[0] += r["ok"]; g[1] += 1
g[2] += r["total_tokens"]; g[3] += r["n_tool_calls"]
tasks = sorted({k[0] for k in agg})
arms = sorted({k[1] for k in agg})
print("=" * 72)
print(f)
header = f"{'task':<14}" + "".join(f"{a + ' ok':<9}" for a in arms)
header += "".join(f"{a + ' calls':<12}" for a in arms)
header += "".join(f"{a + ' tok':<10}" for a in arms)
print(header)
for t in tasks:
row = f"{t:<14}"
for a in arms:
c = agg.get((t, a), dict(ok=0, n=0))
row += f"{str(c['ok']) + '/' + str(c['n']):<9}"
for a in arms:
c = agg.get((t, a), dict(calls=0, n=1))
row += f"{c['calls'] / max(c['n'], 1):<12.1f}"
for a in arms:
c = agg.get((t, a), dict(tok=0, n=1))
row += f"{c['tok'] // max(c['n'], 1):<10}"
print(row)
print("=" * 72)
for arm, (ok, n, tok, calls) in sorted(grand.items()):
if n:
print(f"TOTAL {arm}: {ok}/{n} ok "
f"avg tok/task {tok // n} avg calls {calls / n:.1f}")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("files", nargs="*", default=None)
ap.add_argument("--label", default="ab")
args = ap.parse_args()
files = args.files or glob.glob(
str(EVAL_DIR / "results" / args.label / "*.jsonl"))
if not files:
raise SystemExit("no result files found")
summarize(files)
if __name__ == "__main__":
main()
evals/session_search_schema/results/pr95570/openai_gpt-5.6-luna.jsonl
+36
View File
@@ -0,0 +1,36 @@
{"task": "t1_discover", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1551, "total_tokens": 3818, "wall_s": 5.2, "calls": [{"query": "postgres migration analytics cluster", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a **logical-replication migration to PostgreSQL 16 using `pglogical`**, rather than `pg_dump`/restore, because minimizing downtime was important.\n\nPlan:\n- Configure the **PostgreSQL 13 primary as the pglogical publisher**.\n- Sync a **new PostgreSQL 16 subscriber** in parallel.\n- Perform the final cutover during a **Sunday 02:00 maintenance window**.\n\nThe publisher and subscriber had already been configured and were syncing. This was in @session:default/20260810_091500_aa11bb.", "rep": 0}
{"task": "t1_discover", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 737, "total_tokens": 2277, "wall_s": 4.8, "calls": [{"query": "Postgres migration analytics cluster", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}], "final": "We settled on a **logical-replication migration using `pglogical`** to minimize downtime:\n\n- Upgrade from the **Postgres 13 primary to a new Postgres 16 cluster**\n- Configure the existing cluster as the **publisher**\n- Configure the new Postgres 16 cluster as the **subscriber**\n- Let it sync continuously, then perform the cutover during a **Sunday 02:00 maintenance window**\n\nThe setup was noted as already in progress: the pg13 publisher and pg16 subscriber were configured and syncing. See [the migration conversation]( @session:default/20260810_091500_aa11bb ).", "rep": 0}
{"task": "t1_discover", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1551, "total_tokens": 3795, "wall_s": 4.1, "calls": [{"query": "postgres migration analytics cluster", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a **logical-replication migration using `pglogical`** rather than `pg_dump`/restore, to minimize downtime:\n\n- **Source:** PostgreSQL 13 primary configured as the `pglogical` publisher \n- **Target:** New PostgreSQL 16 server acting as the subscriber \n- **Cutover:** Sunday at **02:00** during the maintenance window \n- The new PostgreSQL 16 instance was already syncing from the existing primary.", "rep": 1}
{"task": "t1_discover", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 737, "total_tokens": 2279, "wall_s": 9.4, "calls": [{"query": "postgres migration analytics cluster", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a **logical replication migration using `pglogical`** for near-zero downtime:\n\n- Migrate from the **Postgres 13 primary** to a new **Postgres 16** instance.\n- Configure the existing cluster as the **publisher** and the new instance as the **subscriber**.\n- Let the subscriber sync, then perform the cutover during a **Sunday 02:00 maintenance window**.\n\nThe setup was recorded as complete: the pg13 publisher and pg16 subscriber were syncing. See [the migration conversation](⁠@session:default/20260810_091500_aa11bb⁠).", "rep": 1}
{"task": "t1_discover", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1551, "total_tokens": 3792, "wall_s": 4.8, "calls": [{"query": "postgres migration analytics cluster", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a **logical replication migration using `pglogical`** to minimize downtime:\n\n- **Source:** PostgreSQL 13 primary, configured as the publisher\n- **Target:** New PostgreSQL 16 server, configured as the subscriber\n- **Cutover:** Sunday at **02:00**\n- The new server was already syncing when we last discussed it.\n\nThat was @session:default/20260810_091500_aa11bb.", "rep": 2}
{"task": "t1_discover", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 737, "total_tokens": 2257, "wall_s": 7.5, "calls": [{"query": "postgres migration analytics cluster", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}], "final": "We settled on a **logical replication migration using `pglogical`** for near-zero downtime:\n\n- Migrate from the **Postgres 13 primary to a new Postgres 16 instance**.\n- Configure the existing cluster as the **pglogical publisher**.\n- Configure the new Postgres 16 server as the **subscriber** and let it catch up.\n- Perform the final cutover during a **Sunday 02:00 maintenance window**.\n\nThe setup was recorded here: @session:default/20260810_091500_aa11bb", "rep": 2}
{"task": "t2_scroll", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 1565, "total_tokens": 7361, "wall_s": 8.7, "calls": [{"query": "checkout 504", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "oldest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 12, "window": 20, "role_filter": "user,assistant", "profile": ""}], "final": "The final fix shipped was:\n\n- **Setting:** `statement_timeout`\n- **Value:** `45s`\n- **Scope:** Payments DB connection pool\n\nWe also re-created the index **concurrently, off-peak**. This is recorded in @session:default/20260818_140200_cc22dd", "rep": 0}
{"task": "t2_scroll", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 751, "total_tokens": 4724, "wall_s": 8.8, "calls": [{"query": "checkout 504 incident final fix shipped precise setting value", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout OR 504 OR timeout", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "gateway timeout", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "The final fix we shipped was:\n\n- **Setting:** `statement_timeout`\n- **Value:** **45 seconds**\n- **Scope:** Payments database connection pool\n\nWe also re-created the affected index using `CONCURRENTLY` during off-peak hours. See the incident session: @session:default/20260818_140200_cc22dd", "rep": 0}
{"task": "t2_scroll", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1565, "total_tokens": 9374, "wall_s": 18.6, "calls": [{"query": "checkout 504 final fix precise setting value", "limit": 5, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504 OR gateway timeout", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "oldest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 12, "window": 20, "role_filter": "user,assistant", "profile": ""}], "final": "The final fix was:\n\n- **Setting:** `statement_timeout`\n- **Value:** `45s`\n- **Scope:** Payments database connection pool\n\nWe also re-created the index using `CREATE INDEX CONCURRENTLY` off-peak. This is recorded in @session:default/20260818_140200_cc22dd.", "rep": 1}
{"task": "t2_scroll", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 751, "total_tokens": 5014, "wall_s": 8.1, "calls": [{"query": "checkout 504 final fix setting value", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout OR 504 OR timeout", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout gateway", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "504", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "The final fix was to set **`statement_timeout = 45s`** on the **payments DB pool**. We also re-created the problematic index using **`CREATE INDEX CONCURRENTLY`** during off-peak hours. See [Checkout 504 Incident]( @session:default/20260818_140200_cc22dd).", "rep": 1}
{"task": "t2_scroll", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1565, "total_tokens": 9395, "wall_s": 8.3, "calls": [{"query": "checkout 504 final fix precise setting value", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504 OR checkout gateway timeout OR 504 checkout", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "oldest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 12, "window": 20, "role_filter": "user,assistant", "profile": ""}], "final": "The final fix was:\n\n- **Setting:** `statement_timeout`\n- **Value:** `45s`\n- **Scope:** Payments database connection pool\n\nWe also re-created the index **concurrently off-peak**. See @session:default/20260818_140200_cc22dd.", "rep": 2}
{"task": "t2_scroll", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 751, "total_tokens": 9135, "wall_s": 12.9, "calls": [{"query": "checkout 504 incident final fix shipped precise setting value", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout timeout OR gateway OR 504", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "final fix shipped setting value", "limit": 10, "sort": "oldest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "oldest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 20, "window": 20, "role_filter": "user,assistant", "profile": ""}], "final": "The final fix was:\n\n- **Setting:** `statement_timeout`\n- **Value:** **45s**\n- **Applied to:** the **payments DB pool**\n\nWe also re-created the index using `CREATE INDEX CONCURRENTLY` off-peak. This is documented in [the Checkout 504 Incident session](@session:default/20260818_140200_cc22dd).", "rep": 2}
{"task": "t3_broaden", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 1556, "total_tokens": 5655, "wall_s": 6.1, "calls": [{"query": "beehive sensors grafana graphs", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive OR bee hive OR bees grafana OR influx OR sensor*", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "The Grafana dashboard is on **port 3000 of the Garden Pi**:\n\n**http://\\<garden-pi-hostname-or-IP\\>:3000**\n\nThe admin login is stored in your **password manager**. This was from @session:default/20260820_101000_ee33ff.", "rep": 0}
{"task": "t3_broaden", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 742, "total_tokens": 4643, "wall_s": 14.8, "calls": [{"query": "beehive sensors Grafana graphs access dashboard URL", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}, {"query": "beehive OR hive OR bees", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}, {"query": "Grafana OR grafana", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}, {"query": "sensor*", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}], "final": "Access Grafana on **port 3000 of the garden Pi**:\n\n**`http://<garden-pi-hostname-or-IP>:3000`**\n\nThe admin credentials are saved in your password manager. We set this up in [the Apiary Monitoring Setup session]( @session:default/20260820_101000_ee33ff )—the dashboard contains the hive temperature, humidity, and weight graphs.", "rep": 0}
{"task": "t3_broaden", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 1556, "total_tokens": 5635, "wall_s": 8.4, "calls": [{"query": "beehive sensors grafana graphs", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "grafana OR beehive OR hive sensors", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access the Grafana dashboard at:\n\n**`http://<garden-pi-IP>:3000`**\n\nIt’s running on the garden Pi. The admin credentials are stored in your password manager. This was from our setup in @session:default/20260820_101000_ee33ff.", "rep": 1}
{"task": "t3_broaden", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 742, "total_tokens": 3379, "wall_s": 8.0, "calls": [{"query": "Grafana beehive sensors graphs access dashboard URL", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive OR hive OR Grafana OR sensor*", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana on **port 3000 of the garden Pi**:\n\n**`http://<garden-pi-hostname-or-IP>:3000`**\n\nThe admin credentials are stored in your password manager. We set this up in [the apiary monitoring session](@session:default/20260820_101000_ee33ff).", "rep": 1}
{"task": "t3_broaden", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 1556, "total_tokens": 5955, "wall_s": 6.3, "calls": [{"query": "beehive sensors grafana graphs", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "Grafana hive", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "sensor* dashboard", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana at:\n\n**`http://<garden-pi-hostname-or-IP>:3000`**\n\nThe beehive sensor dashboard is on the garden Pi, and the admin credentials are stored in your password manager.", "rep": 2}
{"task": "t3_broaden", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 742, "total_tokens": 3393, "wall_s": 11.3, "calls": [{"query": "beehive sensors Grafana graphs access dashboard URL", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive OR beehives OR hive OR Grafana OR sensors", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana on **port 3000 of the garden Raspberry Pi**:\n\n**http://garden-pi:3000** \n—or use the Pi’s local IP—then log in with the admin credentials saved in your password manager.\n\nThat setup is from [our apiary monitoring session](@session:default/20260820_101000_ee33ff).", "rep": 2}
{"task": "t4_link", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1552, "total_tokens": 3640, "wall_s": 4.7, "calls": [{"query": "reef aquarium build", "limit": 10, "sort": "oldest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "The reef aquarium build plan is in @session:default/20260822_183000_a4b4c4.", "rep": 0}
{"task": "t4_link", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": false, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 738, "total_tokens": 3206, "wall_s": 9.3, "calls": [{"query": "reef aquarium build planning", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}, {"query": "reef OR aquarium OR tank OR saltwater OR coral", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}], "final": "I found it: [Reef Aquarium Build Plan](@session:default/20260822_183000_a4b4c4).", "rep": 0}
{"task": "t4_link", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1552, "total_tokens": 3642, "wall_s": 4.6, "calls": [{"query": "reef aquarium build", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "You can reopen the reef aquarium planning session here: @session:default/20260822_183000_a4b4c4", "rep": 1}
{"task": "t4_link", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 738, "total_tokens": 3822, "wall_s": 7.6, "calls": [{"query": "reef aquarium build planning", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}, {"query": "reef OR aquarium OR tank OR saltwater", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}, {"query": "\"reef aquarium\" OR \"reef tank\" OR aquascape", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "", "profile": ""}], "final": "I found it: **Reef Aquarium Build Plan** from August 22, 2026. You can reopen it here: @session:default/20260822_183000_a4b4c4", "rep": 1}
{"task": "t4_link", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1552, "total_tokens": 3637, "wall_s": 4.5, "calls": [{"query": "reef aquarium build", "limit": 10, "sort": "oldest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "You planned it in @session:default/20260822_183000_a4b4c4.", "rep": 2}
{"task": "t4_link", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 738, "total_tokens": 3867, "wall_s": 6.3, "calls": [{"query": "reef aquarium build plan planning tank sump lighting equipment", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "reef OR aquarium OR saltwater", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "\"reef aquarium\" OR \"reef tank\" OR aquarium", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "The reef aquarium build planning session is **“Reef Aquarium Build Plan”** from August 22, 2026. You can reopen it here: @session:default/20260822_183000_a4b4c4", "rep": 2}
{"task": "t5_profile", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1553, "total_tokens": 7317, "wall_s": 8.2, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 5, "role_filter": "", "profile": "work"}], "final": "In @session:work/20260815_110000_beef01, we decided to use **HashiCorp Vault** for secrets management, with:\n\n- **90-day secret rotation**\n- **Dynamic database credentials** for services", "rep": 0}
{"task": "t5_profile", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 739, "total_tokens": 3039, "wall_s": 7.5, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 10, "role_filter": "user,assistant", "profile": "work"}], "final": "We decided to use **HashiCorp Vault** for platform secrets management, with:\n\n- **90-day secret rotation**\n- **Dynamic database credentials** for services", "rep": 0}
{"task": "t5_profile", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 1553, "total_tokens": 5387, "wall_s": 6.4, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 5, "role_filter": "", "profile": "work"}], "final": "We decided on **HashiCorp Vault** for platform secrets management, with:\n\n- **90-day secret rotation**\n- **Dynamic database credentials** for services", "rep": 1}
{"task": "t5_profile", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 739, "total_tokens": 3018, "wall_s": 6.4, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 10, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 10, "role_filter": "user,assistant", "profile": "work"}], "final": "You decided on **HashiCorp Vault** for platform secrets management, with:\n\n- **90-day secret rotation**\n- **Dynamic database credentials** for services", "rep": 1}
{"task": "t5_profile", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1553, "total_tokens": 7346, "wall_s": 10.5, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 5, "role_filter": "user,assistant", "profile": "work"}], "final": "We decided on **HashiCorp Vault** for platform secrets management, with:\n\n- **90-day secret rotation**\n- **Dynamic database credentials** for services", "rep": 2}
{"task": "t5_profile", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 739, "total_tokens": 3010, "wall_s": 9.7, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 5, "role_filter": "", "profile": "work"}], "final": "You decided on **HashiCorp Vault** for secrets management, with:\n\n- **90-day secret rotation**\n- **Dynamic database credentials** for services", "rep": 2}
{"task": "t6_browse", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1549, "total_tokens": 3986, "wall_s": 4.0, "calls": [{"query": "", "limit": 8, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Quick rundown of your recent work:\n\n- **GPU server:** Adjusting the idle fan curve to reduce noise.\n- **Taxes:** Starting an LLC tax-preparation checklist.\n- **Reef aquarium:** Planning a 90-gallon reef aquarium build.\n- **Apiary:** Setting up monitoring for your backyard beehive sensors.\n- **Production incident:** Troubleshooting checkout API timeouts causing 504 errors.\n- **Database infrastructure:** Planning a PostgreSQL 16 migration for the analytics cluster.", "rep": 0}
{"task": "t6_browse", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 735, "total_tokens": 2435, "wall_s": 5.9, "calls": [{"query": "", "limit": 8, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Here’s the quick rundown of your recent work:\n\n- **GPU server:** Tuning the fan curve to reduce idle noise.\n- **Taxes:** Starting an LLC tax-preparation checklist.\n- **Reef aquarium:** Planning a 90-gallon reef aquarium build.\n- **Apiary:** Setting up monitoring for your outdoor beehive sensors.\n- **Production incident:** Troubleshooting checkout API timeouts causing 504 errors.\n- **Database infrastructure:** Planning a PostgreSQL 16 migration for the analytics cluster.", "rep": 0}
{"task": "t6_browse", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1549, "total_tokens": 4005, "wall_s": 5.1, "calls": [{"query": "", "limit": 8, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recently, you’ve been working on:\n\n- **GPU server fan curve** — reducing idle fan noise.\n- **LLC tax preparation** — starting a tax prep checklist.\n- **90-gallon reef aquarium** — planning the build.\n- **Apiary monitoring** — setting up sensors for your beehives.\n- **Checkout API incident** — investigating production 504 timeouts.\n- **Postgres 16 migration** — planning the analytics-cluster migration.\n\nThe most substantial recent thread was the checkout incident: @session:default/20260818_140200_cc22dd", "rep": 1}
{"task": "t6_browse", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 735, "total_tokens": 2422, "wall_s": 5.3, "calls": [{"query": "", "limit": 8, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Here’s the quick rundown of your recent work:\n\n- **GPU server:** Tuning the idle fan curve to reduce noise.\n- **Taxes:** Starting a tax-prep checklist for your LLC.\n- **Reef aquarium:** Planning a 90-gallon reef aquarium build.\n- **Apiary:** Setting up monitoring for your outdoor beehive sensors.\n- **Production incident:** Investigating checkout API timeouts and 504 errors.\n- **Database migration:** Planning a PostgreSQL 16 migration for the analytics cluster.", "rep": 1}
{"task": "t6_browse", "arm": "main", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1549, "total_tokens": 3982, "wall_s": 5.2, "calls": [{"query": "", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recently, you’ve been working on:\n\n- **GPU server fan curve** — reducing idle fan noise.\n- **LLC tax prep** — building a tax preparation checklist.\n- **90-gallon reef aquarium** — planning the build and equipment.\n- **Apiary monitoring** — setting up monitoring for beehive sensors.\n- **Checkout API incident** — troubleshooting production 504 timeouts.\n- **Postgres 16 migration** — planning the analytics cluster migration.", "rep": 2}
{"task": "t6_browse", "arm": "pr", "model": "openai/gpt-5.6-luna", "ok": true, "n_tool_calls": 7, "bad_calls": 0, "first_prompt_tokens": 735, "total_tokens": 5001, "wall_s": 14.0, "calls": [{"query": "", "limit": 8, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260825_200000_ffeedd", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260824_090000_d5e5f6", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260822_183000_a4b4c4", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260820_101000_ee33ff", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260810_091500_aa11bb", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recently, you’ve been working on:\n\n- Tuning the **GPU server’s idle fan curve** to reduce noise: @session:default/20260825_200000_ffeedd\n- Starting an **LLC tax-prep checklist**: @session:default/20260824_090000_d5e5f6\n- Planning a **90-gallon reef aquarium build**: @session:default/20260822_183000_a4b4c4\n- Setting up **monitoring for your backyard beehive sensors**: @session:default/20260820_101000_ee33ff\n- Troubleshooting a **production checkout API timeout/504 incident**: @session:default/20260818_140200_cc22dd\n- Planning a **Postgres 16 migration** for the analytics cluster: @session:default/20260810_091500_aa11bb", "rep": 2}
evals/session_search_schema/results/pr95570/openai_gpt-5.6-terra.jsonl
+36
View File
@@ -0,0 +1,36 @@
{"task": "t1_discover", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1551, "total_tokens": 3799, "wall_s": 4.3, "calls": [{"query": "postgres migration analytics cluster", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a near-zero-downtime migration from PostgreSQL 13 to 16 using **pglogical**:\n\n- Configure the existing pg13 primary as publisher and the new pg16 cluster as subscriber.\n- Let replication catch up, then cut over during the **Sunday 02:00 maintenance window**.\n- Avoided pg_dump/restore because downtime was a priority.\n\nThe publisher/subscriber setup was already configured and syncing in @session:default/20260810_091500_aa11bb.", "rep": 0}
{"task": "t1_discover", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 737, "total_tokens": 4240, "wall_s": 7.7, "calls": [{"query": "postgres migration analytics cluster approach settled", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "analytics postgres", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "migration cluster", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "PostgreSQL OR pg migration", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a **near-zero-downtime logical-replication migration** rather than dump/restore:\n\n- Upgrade the analytics cluster from **Postgres 13 to Postgres 16**\n- Use **pglogical**: the existing pg13 primary as publisher, a new pg16 node as subscriber\n- Let replication catch up, then perform cutover in the **Sunday 02:00 maintenance window**\n\nThe publisher/subscriber sync had already been configured. See @session:default/20260810_091500_aa11bb.", "rep": 0}
{"task": "t1_discover", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1551, "total_tokens": 3776, "wall_s": 7.3, "calls": [{"query": "postgres migration analytics cluster", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We chose a near-zero-downtime migration to PostgreSQL 16 using **pglogical**:\n\n- Configure the existing PostgreSQL 13 primary as the publisher.\n- Bring up the new PostgreSQL 16 instance as a subscriber and let it sync.\n- Perform the final cutover during the **Sunday 02:00 maintenance window**.\n\nThis was set up in @session:default/20260810_091500_aa11bb.", "rep": 1}
{"task": "t1_discover", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 737, "total_tokens": 4829, "wall_s": 10.6, "calls": [{"query": "postgres migration analytics cluster approach settled", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "analytics postgres", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "migration", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "PostgreSQL", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a **near-zero-downtime logical replication migration** rather than dump/restore:\n\n- Upgrade the analytics cluster from **Postgres 13 to Postgres 16**\n- Use **pglogical**: old pg13 primary as publisher, new pg16 instance as subscriber\n- Let it fully sync, then perform the cutover in a **Sunday 02:00 maintenance window**\n\nThe publisher/subscriber replication setup was configured. See @session:default/20260810_091500_aa11bb.", "rep": 1}
{"task": "t1_discover", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1551, "total_tokens": 3785, "wall_s": 5.8, "calls": [{"query": "postgres migration analytics cluster", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a near-zero-downtime migration to PostgreSQL 16 using **pglogical**:\n\n- Configure the existing PostgreSQL 13 primary as the publisher.\n- Bring up the new PostgreSQL 16 cluster as a subscriber and let it synchronize.\n- Cut over during the **Sunday 02:00** maintenance window once replication is caught up.\n\nThe publisher/subscriber setup was already noted as configured in @session:default/20260810_091500_aa11bb.", "rep": 2}
{"task": "t1_discover", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 737, "total_tokens": 2246, "wall_s": 3.6, "calls": [{"query": "postgres migration analytics cluster", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "We settled on a **near-zero-downtime logical replication migration**:\n\n- Upgrade the analytics cluster from **Postgres 13 to Postgres 16** using **pglogical**.\n- Configure the existing pg13 primary as publisher and the new pg16 instance as subscriber, then let it fully sync.\n- Perform the final cutover during a **Sunday 02:00 maintenance window**.\n\nThe publisher/subscriber setup was already configured in the prior discussion: @session:default/20260810_091500_aa11bb", "rep": 2}
{"task": "t2_scroll", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 1565, "total_tokens": 11709, "wall_s": 10.0, "calls": [{"query": "checkout 504 final fix shipped precise setting value", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "504 incident", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout timeout", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 21, "window": 15, "role_filter": "user,assistant", "profile": ""}], "final": "The shipped setting was **`statement_timeout = 45s`** on the **payments DB pool**. We also re-created the problematic index using **`CREATE INDEX CONCURRENTLY`** during an off-peak window.\n\nThis is recorded in @session:default/20260818_140200_cc22dd.", "rep": 0}
{"task": "t2_scroll", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 751, "total_tokens": 9289, "wall_s": 8.0, "calls": [{"query": "checkout 504 incident FINAL fix shipped precise setting value", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "504", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "gateway timeout", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 25, "window": 10, "role_filter": "user,assistant", "profile": ""}], "final": "The final shipped setting was:\n\n- **`statement_timeout = 45s`** on the **payments DB pool**\n\nThe accompanying remediation was to re-create the index **`CONCURRENTLY`** during off-peak hours. Recorded in @session:default/20260818_140200_cc22dd.", "rep": 0}
{"task": "t2_scroll", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 1565, "total_tokens": 13087, "wall_s": 7.5, "calls": [{"query": "checkout 504 incident FINAL fix setting value", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "504", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout timeout", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 25, "window": 12, "role_filter": "user,assistant", "profile": ""}], "final": "The shipped setting change was:\n\n- **`statement_timeout = 45s`** on the **payments DB pool**\n\nThe final remediation also re-created the index using **`CREATE INDEX CONCURRENTLY`** off-peak. This is recorded in @session:default/20260818_140200_cc22dd.", "rep": 1}
{"task": "t2_scroll", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 751, "total_tokens": 10365, "wall_s": 10.2, "calls": [{"query": "checkout 504 incident FINAL fix shipped precise setting value", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "504", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout timeout", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 25, "window": 10, "role_filter": "user,assistant", "profile": ""}], "final": "The final shipped setting was **`statement_timeout = 45s` on the payments DB pool**. We also re-created the index using **`CONCURRENTLY` off-peak**. Recorded in @session:default/20260818_140200_cc22dd.", "rep": 1}
{"task": "t2_scroll", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 1565, "total_tokens": 11609, "wall_s": 7.9, "calls": [{"query": "checkout 504 incident final fix shipped precise setting value", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout 504", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "504", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout incident", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 24, "window": 15, "role_filter": "user,assistant", "profile": ""}], "final": "The shipped timeout setting was **`statement_timeout = 45s`** on the **payments DB pool**.\n\nThe final remediation also re-created the index using **`CONCURRENTLY`** during an off-peak window. This is in @session:default/20260818_140200_cc22dd.", "rep": 2}
{"task": "t2_scroll", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 751, "total_tokens": 11157, "wall_s": 11.7, "calls": [{"query": "checkout 504 incident FINAL fix shipped precise setting value", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout OR 504", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "\"504\"", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "checkout timeout", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260818_140200_cc22dd", "around_message_id": 25, "window": 12, "role_filter": "user,assistant", "profile": ""}], "final": "The final shipped setting was **`statement_timeout = 45s` on the payments DB pool**.\n\nThey also re-created the index using **`CONCURRENTLY` off-peak**. Source: @session:default/20260818_140200_cc22dd", "rep": 2}
{"task": "t3_broaden", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 1556, "total_tokens": 6712, "wall_s": 6.1, "calls": [{"query": "grafana beehive sensors", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "grafana", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "sensor*", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana at:\n\n`http://<garden-pi-hostname-or-IP>:3000`\n\nThe admin credentials are in your password manager. We set it up in @session:default/20260820_101000_ee33ff.", "rep": 0}
{"task": "t3_broaden", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 742, "total_tokens": 4629, "wall_s": 12.4, "calls": [{"query": "beehive sensors Grafana graphs access URL dashboard", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "grafana", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "sensors", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana at **port 3000 on the garden Pi** — e.g. `http://<garden-pi-hostname-or-IP>:3000`. The admin credentials are in your password manager.\n\nWe set it up in @session:default/20260820_101000_ee33ff.", "rep": 0}
{"task": "t3_broaden", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 1556, "total_tokens": 6771, "wall_s": 9.8, "calls": [{"query": "beehive sensors grafana graphs", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "grafana", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive OR hive OR sensors", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "influx OR prometheus OR dashboard", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana on port **3000** of the garden Pi: `http://<garden-pi-address>:3000`.\n\nThe admin credentials are in your password manager. This was from @session:default/20260820_101000_ee33ff.", "rep": 1}
{"task": "t3_broaden", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 742, "total_tokens": 4636, "wall_s": 5.7, "calls": [{"query": "beehive sensors Grafana graphs access URL dashboard", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive OR bees OR hive", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "Grafana", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "sensor OR sensors", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana on **port 3000 of the garden Pi**—for example, `http://<garden-pi-hostname-or-IP>:3000`. The admin credentials are in your password manager.\n\nWe set it up in @session:default/20260820_101000_ee33ff.", "rep": 1}
{"task": "t3_broaden", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 1556, "total_tokens": 6712, "wall_s": 14.7, "calls": [{"query": "Grafana beehive sensors graphs", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "grafana", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "sensor", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana at:\n\n`http://<garden-pi-hostname-or-IP>:3000`\n\nThe admin credentials are in your password manager. We set it up in @session:default/20260820_101000_ee33ff.", "rep": 2}
{"task": "t3_broaden", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 742, "total_tokens": 4642, "wall_s": 6.5, "calls": [{"query": "beehive sensors Grafana graphs access URL", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "grafana", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "beehive OR hive OR bees", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "sensors", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Access Grafana at **`http://<garden-pi>:3000`** (or its IP address on your local network). The admin credentials are in your password manager.\n\nWe set it up in the **Apiary Monitoring Setup** session: @session:default/20260820_101000_ee33ff", "rep": 2}
{"task": "t4_link", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 1552, "total_tokens": 5508, "wall_s": 6.7, "calls": [{"query": "reef aquarium build planned", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "reef OR aquarium OR tank", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "That’s @session:default/20260822_183000_a4b4c4 — the 90-gallon reef aquarium build plan (30g sump, AI Hydra lighting, DIY stand, and cycling timeline).", "rep": 0}
{"task": "t4_link", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 738, "total_tokens": 3250, "wall_s": 5.3, "calls": [{"query": "reef aquarium build planned planning", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "reef OR aquarium OR saltwater OR aquascape", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Found it: the **“Reef Aquarium Build Plan”** session from August 22, covering the 90g display, 30g sump, AI Hydra lighting, DIY stand, and cycling timeline. Reopen it here: @session:default/20260822_183000_a4b4c4", "rep": 0}
{"task": "t4_link", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 1552, "total_tokens": 5509, "wall_s": 5.9, "calls": [{"query": "reef aquarium build planned", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "reef OR aquarium OR tank", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "The reef plan is @session:default/20260822_183000_a4b4c4 — it covers the 90-gallon display, 30-gallon sump, AI Hydra lights, DIY stand, and cycling timeline.", "rep": 1}
{"task": "t4_link", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 738, "total_tokens": 3466, "wall_s": 6.9, "calls": [{"query": "reef aquarium build planned", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "reef OR aquarium OR saltwater", "limit": 10, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "tank build", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Found it: **Reef Aquarium Build Plan** (Aug 22), covering the 90g display, 30g sump, AI Hydra lights, DIY stand, and cycling timeline. Reopen it here: @session:default/20260822_183000_a4b4c4", "rep": 1}
{"task": "t4_link", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1552, "total_tokens": 5700, "wall_s": 11.6, "calls": [{"query": "reef aquarium build planned", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "reef OR aquarium OR saltwater", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "tank build", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "That’s @session:default/20260822_183000_a4b4c4 — the 90-gallon reef aquarium build plan (30g sump, AI Hydra lighting, DIY stand, and cycling timeline).", "rep": 2}
{"task": "t4_link", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 738, "total_tokens": 3225, "wall_s": 5.6, "calls": [{"query": "reef aquarium build planned", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}, {"query": "reef OR aquarium OR aquascape OR tank", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "The reef build planning session is **“Reef Aquarium Build Plan”** from August 22: @session:default/20260822_183000_a4b4c4", "rep": 2}
{"task": "t5_profile", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": false, "n_tool_calls": 8, "bad_calls": 0, "first_prompt_tokens": 1553, "total_tokens": 16052, "wall_s": 23.7, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}], "final": "", "rep": 0}
{"task": "t5_profile", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 2, "bad_calls": 0, "first_prompt_tokens": 739, "total_tokens": 3431, "wall_s": 7.3, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 1, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 2, "window": 20, "role_filter": "user,assistant", "profile": "work"}], "final": "We decided on **HashiCorp Vault** for the platform team, with:\n\n- a **90-day secret rotation policy**\n- **dynamic database credentials** for services\n\nSession: @session:work/20260815_110000_beef01", "rep": 0}
{"task": "t5_profile", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": false, "n_tool_calls": 8, "bad_calls": 0, "first_prompt_tokens": 1553, "total_tokens": 16074, "wall_s": 30.2, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}], "final": "", "rep": 1}
{"task": "t5_profile", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": false, "n_tool_calls": 8, "bad_calls": 0, "first_prompt_tokens": 739, "total_tokens": 9510, "wall_s": 22.7, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}], "final": "", "rep": 1}
{"task": "t5_profile", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": false, "n_tool_calls": 8, "bad_calls": 0, "first_prompt_tokens": 1553, "total_tokens": 16143, "wall_s": 24.8, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": -1, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}], "final": "", "rep": 2}
{"task": "t5_profile", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 739, "total_tokens": 4194, "wall_s": 14.4, "calls": [{"query": "", "limit": 3, "sort": "newest", "detail": "adaptive", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": "work"}, {"query": "", "limit": 3, "sort": "newest", "detail": "full", "session_id": "20260815_110000_beef01", "around_message_id": 1, "window": 20, "role_filter": "user,assistant", "profile": "work"}], "final": "We decided on **HashiCorp Vault** for the platform team, with:\n\n- a **90-day secret rotation** policy\n- **dynamic database credentials** for services\n\nSession: @session:work/20260815_110000_beef01", "rep": 2}
{"task": "t6_browse", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1549, "total_tokens": 3985, "wall_s": 3.8, "calls": [{"query": "", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recent work has included:\n\n- Tuning your **GPU server fan curve** to reduce idle noise.\n- Starting an **LLC tax-prep checklist**.\n- Planning a **90-gallon reef aquarium build**.\n- Setting up **yard beehive/apiary sensor monitoring**.\n- Investigating a **production checkout API 504 timeout incident**.\n- Planning a **Postgres 16 migration** for the analytics cluster.\n\nThe most active technical thread was the @session:default/20260818_140200_cc22dd.", "rep": 0}
{"task": "t6_browse", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 735, "total_tokens": 2453, "wall_s": 11.5, "calls": [{"query": "", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recent work has spanned a mix of infrastructure, operations, and personal projects:\n\n- **GPU server noise:** tuning the idle fan curve. @session:default/20260825_200000_ffeedd \n- **LLC taxes:** starting a tax-prep checklist. @session:default/20260824_090000_d5e5f6 \n- **Reef aquarium:** planning a 90-gallon build. @session:default/20260822_183000_a4b4c4 \n- **Apiary:** setting up yard beehive sensor monitoring. @session:default/20260820_101000_ee33ff \n- **Production incident:** investigating checkout API timeouts/504s. @session:default/20260818_140200_cc22dd \n- **Data platform:** planning a Postgres 16 migration for analytics. @session:default/20260810_091500_aa11bb", "rep": 0}
{"task": "t6_browse", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1549, "total_tokens": 3969, "wall_s": 3.9, "calls": [{"query": "", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recent work includes:\n\n- Tuning the **GPU server fan curve** to reduce idle noise.\n- Starting an **LLC tax-prep checklist**.\n- Planning a **90-gallon reef aquarium** build.\n- Setting up **yard beehive/apiary sensor monitoring**.\n- Investigating a production **checkout API 504 timeout incident**.\n- Planning a **Postgres 16 migration** for the analytics cluster.\n\nWant to resume any of these?", "rep": 1}
{"task": "t6_browse", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 735, "total_tokens": 2227, "wall_s": 2.9, "calls": [{"query": "", "limit": 5, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recent work includes:\n\n- **GPU server:** tuning the idle fan curve to reduce noise. \n- **LLC taxes:** starting a tax-prep checklist. \n- **90-gallon reef aquarium:** planning the build. \n- **Apiary:** setting up monitoring for yard beehive sensors. \n- **Production checkout incident:** investigating checkout API timeouts / user-facing 504s.", "rep": 1}
{"task": "t6_browse", "arm": "main", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1549, "total_tokens": 3984, "wall_s": 4.7, "calls": [{"query": "", "limit": 10, "sort": "newest", "detail": "adaptive", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recent work has covered a mix of infrastructure, operations, and personal projects:\n\n- Tuning a **GPU server fan curve** to reduce idle noise.\n- Starting an **LLC tax-prep checklist**.\n- Planning a **90-gallon reef aquarium** build.\n- Setting up **yard beehive/apiary sensor monitoring**.\n- Investigating and addressing a **production checkout API 504 incident**.\n- Planning a **Postgres 16 migration** for the analytics cluster.", "rep": 2}
{"task": "t6_browse", "arm": "pr", "model": "openai/gpt-5.6-terra", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 735, "total_tokens": 2323, "wall_s": 4.8, "calls": [{"query": "", "limit": 5, "sort": "newest", "detail": "full", "session_id": "", "around_message_id": 0, "window": 5, "role_filter": "user,assistant", "profile": ""}], "final": "Recent work, roughly newest first:\n\n- **GPU server fan curve:** reducing idle noise while keeping safe thermals. @session:default/20260825_200000_ffeedd\n- **LLC tax prep:** starting a checklist and organizing the needed materials. @session:default/20260824_090000_d5e5f6\n- **90-gallon reef aquarium:** planning out the build. @session:default/20260822_183000_a4b4c4\n- **Apiary monitoring:** setting up sensor monitoring for the yard beehives. @session:default/20260820_101000_ee33ff\n- **Checkout outage:** investigating production checkout API timeouts/504s. @session:default/20260818_140200_cc22dd", "rep": 2}
evals/session_search_schema/results/pr95570/qwen_qwen3-coder-30b-a3b-instruct.jsonl
+36
View File
@@ -0,0 +1,36 @@
{"task": "t1_discover", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1990, "total_tokens": 4755, "wall_s": 3.1, "calls": [{"query": "postgres migration analytics cluster"}], "final": "Based on our conversation, we settled on using pglogical for the PostgreSQL migration of the analytics cluster. The approach involves:\n\n1. Using pglogical for logical replication to achieve near-zero downtime\n2. Configuring pglogical publisher on the pg13 primary server\n3. Setting up pglogical subscriber on the new pg16 box\n4. Scheduling the cutover during a Sunday at 02:00 window\n\nThis approach allows us to migrate from PostgreSQL 13 to 16 with minimal disruption to services. The replication is already configured with the publisher on the primary and subscriber on the new server.", "rep": 0}
{"task": "t1_discover", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1142, "total_tokens": 3137, "wall_s": 3.0, "calls": [{"query": "postgres migration analytics cluster"}], "final": "Based on our conversation, we settled on using pglogical for the PostgreSQL migration of the analytics cluster. The approach was to migrate to PostgreSQL 16 using pglogical replication, with a cutover scheduled during a Sunday at 02:00 window for minimal downtime. The pglogical publisher was configured on the PostgreSQL 13 primary, while the subscriber was set up on the new PostgreSQL 16 box to handle the synchronization.\n\nYou can review the full details in our session: @session:default/20260810_091500_aa11bb", "rep": 0}
{"task": "t1_discover", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1990, "total_tokens": 4785, "wall_s": 3.2, "calls": [{"query": "postgres migration analytics cluster"}], "final": "Based on our conversation, we settled on using pglogical for the PostgreSQL migration of the analytics cluster. The approach was to:\n\n1. Use pglogical for logical replication to achieve near-zero downtime\n2. Configure pglogical publisher on the pg13 primary \n3. Set up pglogical subscriber on the new pg16 box to sync data\n4. Perform the cutover during a scheduled Sunday 02:00 window\n\nThis approach was chosen because downtime mattered, and we needed to minimize disruption to the analytics cluster during the migration process. The migration was already underway with the pglogical setup configured.\n\nThe session where we made this decision is @session:default/20260810_091500_aa11bb.", "rep": 1}
{"task": "t1_discover", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1142, "total_tokens": 3124, "wall_s": 2.7, "calls": [{"query": "postgres migration analytics cluster"}], "final": "Based on our conversation, we settled on using pglogical for the PostgreSQL migration of the analytics cluster. The approach involved:\n\n1. Setting up pglogical publisher on the pg13 primary\n2. Configuring a subscriber on the new pg16 box to sync data\n3. Planning a cutover during a Sunday 02:00 window for minimal downtime\n\nYou can review the full details in our session: @session:default/20260810_091500_aa11bb", "rep": 1}
{"task": "t1_discover", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1990, "total_tokens": 4794, "wall_s": 3.2, "calls": [{"query": "postgres migration analytics cluster"}], "final": "Based on our conversation, we settled on using pglogical for the PostgreSQL migration of the analytics cluster. The approach was to:\n\n1. Use pglogical for logical replication to achieve near-zero downtime\n2. Configure pglogical publisher on the pg13 primary \n3. Set up pglogical subscriber on the new pg16 box to sync data\n4. Perform the cutover during a scheduled Sunday 02:00 window\n\nThis approach was chosen because downtime mattered, so we needed a method that would minimize disruption to the analytics cluster during the migration process. The migration was planned to move from PostgreSQL 13 to PostgreSQL 16.\n\nThat's the session where we made this decision: @session:default/20260810_091500_aa11bb", "rep": 2}
{"task": "t1_discover", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1142, "total_tokens": 3133, "wall_s": 2.8, "calls": [{"query": "postgres migration analytics cluster"}], "final": "Based on our conversation, we settled on using pglogical for the PostgreSQL migration of the analytics cluster. The approach involves:\n\n1. Using pglogical for logical replication to achieve near-zero downtime\n2. Configuring pglogical publisher on the pg13 primary\n3. Setting up pglogical subscriber on the new pg16 box for syncing\n4. Planning a cutover during a Sunday 02:00 window\n\nThe session where we discussed this is available at @session:default/20260810_091500_aa11bb.", "rep": 2}
{"task": "t2_scroll", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": false, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 2006, "total_tokens": 12455, "wall_s": 6.9, "calls": [{"query": "checkout 504 incident final fix"}, {"query": "checkout 504 incident"}, {"query": "504 error checkout"}, {"query": "checkout timeout"}], "final": "", "rep": 0}
{"task": "t2_scroll", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 7, "bad_calls": 0, "first_prompt_tokens": 1158, "total_tokens": 27165, "wall_s": 10.8, "calls": [{"query": "checkout 504 incident final fix"}, {"query": "checkout 504"}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 7}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 12}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 17}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 22}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 27}], "final": "Based on my search of the session history, I found the final fix for the checkout 504 incident. \n\nThe precise setting and value that was shipped as the final fix was:\n\n**statement_timeout set to 45 seconds** on the payments DB pool.\n\nThis was implemented alongside re-creating the index CONCURRENTLY during off-peak hours. The incident was caused by a nightly migration job that ran CREATE INDEX without CONCURRENTLY, which was competing with checkout writes and causing the timeouts. The team first killed the problematic index build, then implemented this timeout increase to prevent future occurrences.", "rep": 0}
{"task": "t2_scroll", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 2006, "total_tokens": 17099, "wall_s": 7.1, "calls": [{"query": "checkout 504 incident final fix"}, {"query": "checkout 504 incident"}, {"query": "504 error checkout"}, {"query": "504 error"}, {"session_id": "20260818_140200_cc22dd"}], "final": "Based on our session history, I found the final fix for the checkout 504 incident. \n\nThe precise setting and value that was changed was:\n\n**statement_timeout increased to 45 seconds** on the payments DB pool.\n\nThis was the final fix implemented after the incident where a nightly migration job had created a non-concurrent index build that was blocking checkout operations. The team first killed the problematic index build, but then needed to adjust the statement timeout to prevent the 504 errors during retry operations. They also re-created the index CONCURRENTLY during off-peak hours to prevent future occurrences.\n\nThe specific change was made to the payments database connection pool configuration to increase the statement timeout from its previous value to 45 seconds.", "rep": 1}
{"task": "t2_scroll", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 7, "bad_calls": 0, "first_prompt_tokens": 1158, "total_tokens": 27166, "wall_s": 10.7, "calls": [{"query": "checkout 504 incident final fix"}, {"query": "checkout 504"}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 7}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 12}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 17}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 22}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 27}], "final": "Based on my search of the session history, I found the details about the checkout 504 incident and the final fix that was shipped.\n\nThe final fix was to raise the statement_timeout to 45 seconds on the payments DB pool. This was the precise setting and value that was changed to resolve the checkout 504 errors.\n\nThe complete solution involved two parts:\n1. Raising statement_timeout to 45s on the payments DB pool\n2. Re-creating the index CONCURRENTLY during off-peak hours\n\nThis fix was deployed everywhere after successful testing on the canary environment. The incident was caused by a nightly migration job that ran CREATE INDEX without CONCURRENTLY, which was competing with checkout writes and causing the timeouts.", "rep": 1}
{"task": "t2_scroll", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": false, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 2006, "total_tokens": 17766, "wall_s": 9.4, "calls": [{"query": "checkout 504 incident final fix"}, {"query": "checkout 504 incident"}, {"query": "504 error checkout"}, {"query": "checkout"}, {"session_id": "20260818_140200_cc22dd"}], "final": "", "rep": 2}
{"task": "t2_scroll", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 1158, "total_tokens": 13835, "wall_s": 7.7, "calls": [{"query": "checkout 504 incident final fix"}, {"query": "checkout 504"}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 7, "window": 20}, {"session_id": "20260818_140200_cc22dd", "around_message_id": 27, "window": 20}], "final": "Based on my search of the session history, I found the details about the checkout 504 incident and the final fix that was shipped.\n\nThe final fix was to raise the statement_timeout to 45 seconds on the payments DB pool. This was the precise setting and value that resolved the issue.\n\nThe incident occurred when a nightly migration job ran CREATE INDEX without CONCURRENTLY, which was competing with checkout writes and causing timeouts. Although they killed the index build and rolled back the deployment, the issue persisted due to statement timeouts firing on retries. The final solution involved increasing the statement_timeout to 45 seconds on the payments database connection pool, along with re-creating the index CONCURRENTLY during off-peak hours.", "rep": 2}
{"task": "t3_broaden", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1995, "total_tokens": 9400, "wall_s": 5.3, "calls": [{"query": "grafana beehive sensors"}, {"query": "Grafana beehive"}, {"query": "Grafana"}], "final": "I found our conversation about setting up Grafana for the beehive sensors. According to our discussion from August 20, 2026, the Grafana instance is running on port 3000 of the garden pi. You can access it by visiting http://garden-pi:3000 and using the admin login credentials that are stored in your password manager.\n\nThe relevant exchange was:\n- You asked \"I want graphs\"\n- I responded \"I set up a grafana instance for the sensor graphs.\"\n- You asked \"where do I see it\"\n- I replied \"The dashboard is on port 3000 of the garden pi, admin login in your password manager.\"\n\nThis session is titled \"Apiary Monitoring Setup\" and can be referenced as @session:default/20260820_101000_ee33ff.", "rep": 0}
{"task": "t3_broaden", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1147, "total_tokens": 6217, "wall_s": 5.2, "calls": [{"query": "grafana beehive sensors"}, {"query": "grafana beehive"}, {"query": "sensor grafana"}], "final": "I found our conversation about setting up Grafana for the beehive sensors. According to our discussion, you set up a Grafana instance for the sensor graphs. The dashboard is accessible on port 3000 of the garden pi, with the admin login details stored in your password manager.\n\nYou can access the graphs by navigating to http://garden-pi:3000 in your browser, where you'll need to log in using the admin credentials from your password manager.\n\nFor more details, you can review our full conversation: @session:default/20260820_101000_ee33ff", "rep": 0}
{"task": "t3_broaden", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 5, "bad_calls": 0, "first_prompt_tokens": 1995, "total_tokens": 14454, "wall_s": 8.1, "calls": [{"query": "grafana beehive sensors"}, {"query": "Grafana beehive"}, {"query": "sensor dashboard grafana"}, {"query": "IoT sensors monitoring"}, {"query": "beehive"}], "final": "I found our conversation about the beehive monitoring setup. Based on our discussion, you can access the Grafana graphs for the beehive sensors through the following:\n\nThe dashboard is on port 3000 of the garden pi, with admin login credentials stored in your password manager.\n\nThe specific session where we discussed this is titled \"Apiary Monitoring Setup\" (@session:default/20260820_101000_ee33ff). In that conversation, we confirmed that:\n1. The hive telemetry (temperature, humidity, weight) is already publishing to MQTT\n2. A Grafana instance was set up for the sensor graphs\n3. The dashboard is accessible via port 3000 on the garden pi\n4. Admin login details are in your password manager\n\nTo access the graphs, you would typically navigate to http://garden-pi:3000 in your browser and log in using the admin credentials from your password manager.", "rep": 1}
{"task": "t3_broaden", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1147, "total_tokens": 6183, "wall_s": 4.4, "calls": [{"query": "grafana beehive sensors"}, {"query": "grafana beehive"}, {"query": "sensor grafana"}], "final": "Based on our previous conversation, you can access the Grafana graphs for the beehive sensors through the dashboard on port 3000 of the garden pi. The admin login credentials are stored in your password manager.\n\nYou can visit the dashboard at: http://garden-pi:3000\n\nThe session where we set this up is available for reference: @session:default/20260820_101000_ee33ff", "rep": 1}
{"task": "t3_broaden", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1995, "total_tokens": 9371, "wall_s": 5.2, "calls": [{"query": "grafana beehive sensors"}, {"query": "Grafana beehive"}, {"query": "Grafana"}], "final": "I found our previous conversation about setting up Grafana for the beehive sensors. According to our discussion, you set up a Grafana instance for the sensor graphs. The dashboard is accessible on port 3000 of the garden pi, and the admin login credentials are stored in your password manager.\n\nThe relevant session is titled \"Apiary Monitoring Setup\" and you can access it at: @session:default/20260820_101000_ee33ff\n\nTo access the graphs, you would go to the garden pi's IP address on port 3000 (likely something like http://[garden-pi-ip]:3000) and log in using the admin credentials from your password manager.", "rep": 2}
{"task": "t3_broaden", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 4, "bad_calls": 0, "first_prompt_tokens": 1147, "total_tokens": 7947, "wall_s": 5.6, "calls": [{"query": "grafana beehive sensors"}, {"query": "grafana beehive"}, {"query": "sensor data grafana"}, {"query": "beehive"}], "final": "I found our conversation about the beehive monitoring system. According to our discussion, the Grafana instance for the sensor graphs is running on port 3000 of the garden pi. You can access it at:\n\n@session:default/20260820_101000_ee33ff\n\nThe dashboard should be accessible via http://garden-pi:3000 (assuming garden-pi is your hostname) or the IP address of that device. The admin login credentials are stored in your password manager.", "rep": 2}
{"task": "t4_link", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1991, "total_tokens": 4653, "wall_s": 3.1, "calls": [{"query": "reef aquarium build plan"}], "final": "I found the session where we planned the reef aquarium build. It's titled \"Reef Aquarium Build Plan\" and you can reopen it using this link: @session:default/20260822_183000_a4b4c4\n\nThe session started with you asking to plan the 90 gallon reef aquarium build, and we discussed a build sketch including a 90g display, 30g sump, AI Hydra lighting, and a DIY stand. We also touched on the cycling timeline, suggesting a 6-8 week fishless cycle with ammonia dosing followed by introducing clean-up crew members first.", "rep": 0}
{"task": "t4_link", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1143, "total_tokens": 3023, "wall_s": 2.6, "calls": [{"query": "reef aquarium build plan"}], "final": "I found the session where you planned the reef aquarium build. Here's the link to reopen it: @session:default/20260822_183000_a4b4c4\n\nThe session was titled \"Reef Aquarium Build Plan\" and started with your message: \"Help me plan the 90 gallon reef aquarium build.\" The discussion covered aspects like the 90-gallon display tank, 30-gallon sump, AI Hydra lighting, and a DIY stand, along with a fishless cycling timeline of 6-8 weeks.", "rep": 0}
{"task": "t4_link", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1991, "total_tokens": 4631, "wall_s": 2.6, "calls": [{"query": "reef aquarium build plan"}], "final": "I found the session where we planned the reef aquarium build. It's titled \"Reef Aquarium Build Plan\" and is ready for you to reopen.\n\n@session:default/20260822_183000_a4b4c4\n\nThe session contains our initial planning discussion about the 90-gallon reef aquarium build, including details about the display tank, sump, lighting, and DIY stand. We also discussed the cycling timeline for the aquarium.\n\nWould you like me to summarize what we discussed in that session?", "rep": 1}
{"task": "t4_link", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1143, "total_tokens": 3023, "wall_s": 2.9, "calls": [{"query": "reef aquarium build plan"}], "final": "I found the session where you planned the reef aquarium build. Here's the link to reopen it: @session:default/20260822_183000_a4b4c4\n\nThe session was titled \"Reef Aquarium Build Plan\" and started with your message: \"Help me plan the 90 gallon reef aquarium build.\" The discussion covered aspects like the 90-gallon display tank, 30-gallon sump, AI Hydra lighting, and a DIY stand, along with a fishless cycling timeline of 6-8 weeks.", "rep": 1}
{"task": "t4_link", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1991, "total_tokens": 4648, "wall_s": 2.7, "calls": [{"query": "reef aquarium build plan"}], "final": "I found the session where we planned the reef aquarium build. It's titled \"Reef Aquarium Build Plan\" and you can reopen it using this link: @session:default/20260822_183000_a4b4c4\n\nThe session started with your message: \"Help me plan the 90 gallon reef aquarium build.\" and we discussed a build including a 90g display, 30g sump, AI Hydra lighting, and a DIY stand. We also touched on the cycling timeline, planning for a 6-8 week fishless cycle with ammonia dosing.", "rep": 2}
{"task": "t4_link", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1143, "total_tokens": 3023, "wall_s": 2.7, "calls": [{"query": "reef aquarium build plan"}], "final": "I found the session where you planned the reef aquarium build. Here's the link to reopen it: @session:default/20260822_183000_a4b4c4\n\nThis session was titled \"Reef Aquarium Build Plan\" and started with your message: \"Help me plan the 90 gallon reef aquarium build.\" The conversation included discussions about the build specifications (90g display, 30g sump, AI Hydra lighting, DIY stand) and a cycle timeline (6-8 weeks fishless cycle with ammonia dosing).", "rep": 2}
{"task": "t5_profile", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 2002, "total_tokens": 4548, "wall_s": 2.5, "calls": [{"session_id": "20260815_110000_beef01"}], "final": "In that session, we decided to go with **HashiCorp Vault** for secrets management, implementing a **90-day rotation policy** and using **dynamic database credentials** for services. \n\nThe full conversation was:\n- User: \"We need to pick a secrets management approach for the platform team.\"\n- Assistant: \"Candidates: AWS Secrets Manager, Vault, SOPS in git.\"\n- User: \"what did we land on?\"\n- Assistant: \"Decision: HashiCorp Vault with 90-day rotation policy, dynamic DB creds for services.\"", "rep": 0}
{"task": "t5_profile", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1154, "total_tokens": 2925, "wall_s": 3.0, "calls": [{"session_id": "20260815_110000_beef01"}], "final": "Looking at our session from August 15, 2026, we decided on using **HashiCorp Vault** for secrets management with a 90-day rotation policy and dynamic database credentials for services.\n\nThe full conversation was:\n1. You asked us to pick a secrets management approach for the platform team\n2. I listed the candidates: AWS Secrets Manager, Vault, and SOPS in git\n3. You asked what we landed on\n4. I confirmed our decision was HashiCorp Vault with the 90-day rotation policy and dynamic DB creds for services\n\nThis decision was made in our session titled \"Secrets Management Decision\" on August 15, 2026.", "rep": 0}
{"task": "t5_profile", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 2002, "total_tokens": 4468, "wall_s": 2.4, "calls": [{"session_id": "20260815_110000_beef01"}], "final": "In that session, we decided to go with HashiCorp Vault for secrets management, with a 90-day rotation policy and dynamic database credentials for services.", "rep": 1}
{"task": "t5_profile", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1154, "total_tokens": 2926, "wall_s": 3.0, "calls": [{"profile": "work", "session_id": "20260815_110000_beef01"}], "final": "Looking at your session from August 15, 2026, titled \"Secrets Management Decision\", here's what you decided:\n\nYou chose **HashiCorp Vault** as your secrets management approach, with:\n- A 90-day rotation policy\n- Dynamic database credentials for services\n\nThe discussion considered three candidates: AWS Secrets Manager, Vault, and SOPS in git, but you ultimately went with HashiCorp Vault for your platform team's needs.\n\nYou can reference this decision at any time using the link: @session:work/20260815_110000_beef01", "rep": 1}
{"task": "t5_profile", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 2002, "total_tokens": 4547, "wall_s": 2.9, "calls": [{"session_id": "20260815_110000_beef01"}], "final": "In that session, we decided to go with **HashiCorp Vault** for secrets management, with a **90-day rotation policy** and **dynamic database credentials** for services. \n\nThe full conversation was:\n- User: \"We need to pick a secrets management approach for the platform team.\"\n- Assistant: \"Candidates: AWS Secrets Manager, Vault, SOPS in git.\"\n- User: \"what did we land on?\"\n- Assistant: \"Decision: HashiCorp Vault with 90-day rotation policy, dynamic DB creds for services.\"", "rep": 2}
{"task": "t5_profile", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1154, "total_tokens": 2926, "wall_s": 3.2, "calls": [{"profile": "work", "session_id": "20260815_110000_beef01"}], "final": "Looking at your session from August 15, 2026, titled \"Secrets Management Decision\", here's what you decided:\n\nYou chose **HashiCorp Vault** as your secrets management approach, with:\n- A 90-day rotation policy\n- Dynamic database credentials for services\n\nThe discussion considered three candidates: AWS Secrets Manager, Vault, and SOPS in git, but you ultimately went with HashiCorp Vault for your platform team's needs.\n\nYou can reference this decision at any time using the link: @session:work/20260815_110000_beef01", "rep": 2}
{"task": "t6_browse", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1988, "total_tokens": 4631, "wall_s": 3.4, "calls": [{}], "final": "Here's a quick rundown of your recent sessions:\n\n1. **@session:default/20260825_200000_ffeedd** - You were addressing an issue with the GPU server being too loud at idle and needed to adjust the fan curve.\n2. **@session:default/20260824_090000_d5e5f6** - You started working on a tax prep checklist for your LLC.\n3. **@session:default/20260822_183000_a4b4c4** - You were planning a 90-gallon reef aquarium build.\n\nLet me know if you'd like to dive into any of these topics further!", "rep": 0}
{"task": "t6_browse", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1140, "total_tokens": 2950, "wall_s": 4.1, "calls": [{}], "final": "Here's a quick rundown of your recent sessions:\n\n1. **[GPU Server Fan Curve](@session:default/20260825_200000_ffeedd)**: You're looking to address the issue of the GPU server being too loud at idle by adjusting the fan curve.\n\n2. **[Tax Prep Checklist](@session:default/20260824_090000_d5e5f6)**: You started working on a tax prep checklist for your LLC.\n\n3. **[Reef Aquarium Build Plan](@session:default/20260822_183000_a4b4c4)**: You're planning a 90-gallon reef aquarium build.\n\nLet me know if you'd like details on any of these!", "rep": 0}
{"task": "t6_browse", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1988, "total_tokens": 4630, "wall_s": 3.3, "calls": [{}], "final": "Here's a quick rundown of your recent sessions:\n\n1. **@session:default/20260825_200000_ffeedd** - You were addressing an issue with the GPU server being too loud at idle and needed to adjust the fan curve.\n2. **@session:default/20260824_090000_d5e5f6** - You started working on a tax prep checklist for your LLC.\n3. **@session:default/20260822_183000_a4b4c4** - You began planning a 90-gallon reef aquarium build.\n\nLet me know if you'd like to dive into any of these sessions!", "rep": 1}
{"task": "t6_browse", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 3, "bad_calls": 0, "first_prompt_tokens": 1140, "total_tokens": 6675, "wall_s": 8.2, "calls": [{"query": "recent work", "limit": 5, "sort": "newest"}, {"query": "work", "limit": 5, "sort": "newest"}, {"limit": 5, "sort": "newest"}], "final": "Here's a quick rundown of your recent sessions:\n\n1. **GPU Server Fan Curve** (@session:default/20260825_200000_ffeedd) - You were addressing an issue with the GPU server being too loud at idle and needed to adjust the fan curve.\n2. **Tax Prep Checklist** (@session:default/20260824_090000_d5e5f6) - You started working on a tax preparation checklist for an LLC.\n3. **Reef Aquarium Build Plan** (@session:default/20260822_183000_a4b4c4) - You were planning a 90-gallon reef aquarium build.\n4. **Apiary Monitoring Setup** (@session:default/20260820_101000_ee33ff) - You were setting up monitoring for beehive sensors in the yard.\n5. **Checkout 504 Incident** (@session:default/20260818_140200_cc22dd) - You were troubleshooting a checkout API timeout issue in production, which involved users seeing 504 errors.\n\nThese sessions cover a mix of technical issues, personal projects, and business-related tasks.", "rep": 1}
{"task": "t6_browse", "arm": "main", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1988, "total_tokens": 4630, "wall_s": 2.6, "calls": [{}], "final": "Here's a quick rundown of your recent sessions:\n\n1. **@session:default/20260825_200000_ffeedd** - You were addressing an issue with the GPU server being too loud at idle and needed to adjust the fan curve.\n2. **@session:default/20260824_090000_d5e5f6** - You started working on a tax prep checklist for your LLC.\n3. **@session:default/20260822_183000_a4b4c4** - You were planning a 90-gallon reef aquarium build.\n\nLet me know if you'd like to dive into any of these sessions!", "rep": 2}
{"task": "t6_browse", "arm": "pr", "model": "qwen/qwen3-coder-30b-a3b-instruct", "ok": true, "n_tool_calls": 1, "bad_calls": 0, "first_prompt_tokens": 1140, "total_tokens": 2950, "wall_s": 2.9, "calls": [{}], "final": "Here's a quick rundown of your recent sessions:\n\n1. **[GPU Server Fan Curve](@session:default/20260825_200000_ffeedd)**: You're looking to address the issue of the GPU server being too loud at idle by adjusting the fan curve.\n2. **[Tax Prep Checklist](@session:default/20260824_090000_d5e5f6)**: You started working on a tax prep checklist for your LLC.\n3. **[Reef Aquarium Build Plan](@session:default/20260822_183000_a4b4c4)**: You're planning a 90-gallon reef aquarium build.\n\nLet me know if you'd like details on any specific session!", "rep": 2}
evals/session_search_schema/runner.py
+286
View File
@@ -0,0 +1,286 @@
"""Live A/B runner: session_search schema variants, extracted from git refs.
For each arm, ``tools/session_search_tool.py`` is extracted from a git ref
(``git show <ref>:tools/session_search_tool.py``) and imported as its own
module. A minimal agent loop (OpenRouter, tools API) then runs the shared
task battery against a freshly seeded temp session DB. The ONLY variable
between arms is that module — schema text, response hints, tool behavior.
Usage:
python3 evals/session_search_schema/runner.py \
--base origin/main --cand HEAD \
--model qwen/qwen3-coder-30b-a3b-instruct --reps 3
# limit to one task
... --tasks t2_scroll
Results append to results/<label>/<model-slug>.jsonl (resume-safe: completed
(task, arm, rep) cells are skipped on re-run). Summarize with report.py.
"""
from __future__ import annotations
import argparse
import importlib.util
import json
import os
import re
import subprocess
import sys
import tempfile
import time
import traceback
from pathlib import Path
EVAL_DIR = Path(__file__).resolve().parent
REPO_ROOT = EVAL_DIR.parent.parent
sys.path.insert(0, str(EVAL_DIR))
sys.path.insert(0, str(REPO_ROOT))
from tasks import SYSTEM, TASKS # noqa: E402
ALLOWED_KEYS = {
"query", "role_filter", "limit", "session_id", "around_message_id",
"window", "sort", "profile", "detail",
}
def _load_api_key() -> str:
key = os.environ.get("OPENROUTER_API_KEY", "").strip()
if key:
return key
env_path = Path.home() / ".hermes" / ".env"
if env_path.exists():
for line in env_path.read_text().splitlines():
if line.startswith("OPENROUTER_API_KEY="):
return line.split("=", 1)[1].strip().strip('"').strip("'")
raise SystemExit("OPENROUTER_API_KEY not found (env or ~/.hermes/.env)")
def extract_arm(ref: str, workdir: Path, name: str) -> Path:
"""Extract tools/session_search_tool.py from a git ref."""
out = subprocess.run(
["git", "show", f"{ref}:tools/session_search_tool.py"],
cwd=REPO_ROOT, capture_output=True, text=True,
)
if out.returncode != 0:
raise SystemExit(f"git show {ref}: {out.stderr.strip()}")
path = workdir / f"ss_arm_{name}.py"
path.write_text(out.stdout)
return path
def load_arm(path: Path, name: str, work_db_path: Path):
"""Import an arm module and make profile resolution hermetic."""
from hermes_state import SessionDB
spec = importlib.util.spec_from_file_location(f"ss_arm_{name}", path)
mod = importlib.util.module_from_spec(spec)
sys.modules[f"ss_arm_{name}"] = mod
spec.loader.exec_module(mod)
def _fake_resolve_profile_db(profile):
if profile is None or not str(profile).strip():
return None
if str(profile).strip().lower() == "work":
return SessionDB(db_path=work_db_path, read_only=True)
raise ValueError(f"profile '{profile}' does not exist")
def _fake_locate_session_db(session_id):
try:
db = SessionDB(db_path=work_db_path, read_only=True)
row = db._conn.execute(
"SELECT 1 FROM sessions WHERE id = ?", (session_id,)
).fetchone()
if row:
return db, "work"
db.close()
except Exception:
pass
return None, None
mod._resolve_profile_db = _fake_resolve_profile_db
mod._locate_session_db = _fake_locate_session_db
return mod
def build_tools(arm_mod):
s = arm_mod.SESSION_SEARCH_SCHEMA
return [{
"type": "function",
"function": {
"name": s["name"],
"description": s["description"],
"parameters": s["parameters"],
},
}]
def exec_tool(arm_mod, args, main_db_path: Path):
from hermes_state import SessionDB
db = SessionDB(db_path=main_db_path)
try:
kwargs, bad = {}, []
for k, v in args.items():
if k in ALLOWED_KEYS:
kwargs[k] = v
else:
bad.append(k)
if bad:
return json.dumps({
"success": False,
"error": f"unexpected parameter(s): {', '.join(bad)}",
}), True
return arm_mod.session_search(db=db, **kwargs), False
except Exception as e: # noqa: BLE001 — tool errors go back to the model
return json.dumps({
"success": False, "error": f"{type(e).__name__}: {e}",
}), True
finally:
try:
db.close()
except Exception:
pass
def run_one(client, model, arm_name, arm_mod, task_id, prompt, oracle,
main_db_path: Path, max_iters: int = 8):
tools = build_tools(arm_mod)
messages = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": prompt}]
calls, bad_calls = [], 0
first_prompt_tokens, total_tokens = None, 0
final = ""
t0 = time.time()
for _ in range(max_iters):
resp = client.chat.completions.create(
model=model, messages=messages, tools=tools,
temperature=0.2, max_tokens=2000,
)
u = getattr(resp, "usage", None)
if u:
if first_prompt_tokens is None:
first_prompt_tokens = u.prompt_tokens
total_tokens += (u.total_tokens or 0)
msg = resp.choices[0].message
tcs = msg.tool_calls or []
if not tcs:
final = msg.content or ""
break
messages.append({
"role": "assistant",
"content": msg.content or "",
"tool_calls": [
{"id": tc.id, "type": "function",
"function": {"name": tc.function.name,
"arguments": tc.function.arguments}}
for tc in tcs
],
})
for tc in tcs:
try:
args = json.loads(tc.function.arguments or "{}")
except Exception:
args, bad_calls = {}, bad_calls + 1
calls.append(args)
if tc.function.name != "session_search":
out, was_err = json.dumps(
{"success": False, "error": "unknown tool"}), True
else:
out, was_err = exec_tool(arm_mod, args, main_db_path)
if was_err:
bad_calls += 1
if len(out) > 30000:
out = out[:30000] + "...[truncated]"
messages.append(
{"role": "tool", "tool_call_id": tc.id, "content": out})
return {
"task": task_id, "arm": arm_name, "model": model,
"ok": bool(oracle(final)) if final else False,
"n_tool_calls": len(calls), "bad_calls": bad_calls,
"first_prompt_tokens": first_prompt_tokens,
"total_tokens": total_tokens,
"wall_s": round(time.time() - t0, 1),
"calls": calls, "final": final[:2000],
}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--base", required=True, help="git ref for the baseline arm")
ap.add_argument("--cand", required=True, help="git ref for the candidate arm")
ap.add_argument("--model", required=True)
ap.add_argument("--reps", type=int, default=3)
ap.add_argument("--tasks", nargs="*", default=None)
ap.add_argument("--label", default="ab")
args = ap.parse_args()
from openai import OpenAI
client = OpenAI(base_url="https://openrouter.ai/api/v1",
api_key=_load_api_key())
with tempfile.TemporaryDirectory(prefix="ss_abeval_") as td:
tdir = Path(td)
from fixtures import seed
dbdir = tdir / "dbs"
seed(dbdir)
main_db = dbdir / "state.db"
work_db = dbdir / "state_work.db"
arms = {
"base": load_arm(extract_arm(args.base, tdir, "base"), "base", work_db),
"cand": load_arm(extract_arm(args.cand, tdir, "cand"), "cand", work_db),
}
outdir = EVAL_DIR / "results" / args.label
outdir.mkdir(parents=True, exist_ok=True)
outpath = outdir / (re.sub(r"[^\w.-]", "_", args.model) + ".jsonl")
done = set()
if outpath.exists():
for line in outpath.read_text().splitlines():
try:
r = json.loads(line)
done.add((r["task"], r["arm"], r["rep"]))
except Exception:
pass
with open(outpath, "a", encoding="utf-8") as f:
for task_id, (prompt, oracle, _note) in TASKS.items():
if args.tasks and task_id not in args.tasks:
continue
for rep in range(args.reps):
for arm_name, arm_mod in arms.items():
if (task_id, arm_name, rep) in done:
continue
for attempt in range(3):
try:
r = run_one(client, args.model, arm_name,
arm_mod, task_id, prompt, oracle,
main_db)
# Provider noise: zero tool calls AND empty
# final → one retry, identical on both arms.
if (not r["final"].strip()
and r["n_tool_calls"] == 0
and attempt < 2):
print(f"NOISE-RETRY {task_id} {arm_name} "
f"rep{rep}")
continue
r["rep"] = rep
r["base_ref"] = args.base
r["cand_ref"] = args.cand
f.write(json.dumps(r, ensure_ascii=False) + "\n")
f.flush()
print(f"{task_id} {arm_name} rep{rep}: "
f"ok={r['ok']} calls={r['n_tool_calls']} "
f"bad={r['bad_calls']} "
f"ptok={r['first_prompt_tokens']}")
break
except Exception as e: # noqa: BLE001
print(f"RETRY {task_id} {arm_name} rep{rep}: {e}")
traceback.print_exc()
time.sleep(5 * (attempt + 1))
print("done ->", outpath)
if __name__ == "__main__":
main()
evals/session_search_schema/tasks.py
+65
View File
@@ -0,0 +1,65 @@
"""Task prompts + programmatic oracles for the session_search schema A/B eval.
Every oracle is a pure function of the model's final answer string — no LLM
judging. Tasks map 1:1 to the sessions seeded by fixtures.py.
"""
from __future__ import annotations
import re
# task_id -> (prompt, oracle_fn, note)
TASKS = {
"t1_discover": (
"What approach did we settle on for the postgres migration of the "
"analytics cluster? Check our past conversations.",
lambda a: "pglogical" in a.lower(),
"discovery basics",
),
"t2_scroll": (
"A while back we had that checkout 504 incident. What exactly was the "
"FINAL fix we shipped (the precise setting and value)? Look it up in "
"our session history.",
lambda a: (
("statement_timeout" in a.lower() or "statement timeout" in a.lower())
and "45" in a
),
"requires forward scroll past the ±5 window (and past bookends)",
),
"t3_broaden": (
"Where do I access the grafana graphs for the beehive sensors? We set "
"this up together — search our history.",
lambda a: "3000" in a,
"AND-query misses; needs broadening (OR / fewer terms)",
),
"t4_link": (
"Find the session where we planned the reef aquarium build and point "
"me to it so I can reopen it.",
lambda a: (
bool(re.search(r"(?<!`)@session:[\w./-]*20260822_183000_a4b4c4(?!`)", a))
and "`@session" not in a
and not re.search(r"\]\(@session", a)
),
"must emit link value verbatim, not backticked/markdown",
),
"t5_profile": (
"@session:work/20260815_110000_beef01 — what did we decide in there?",
lambda a: "vault" in a.lower() and "90" in a,
"resolve profile-qualified link (read shape + profile)",
),
"t6_browse": (
"What have I been working on in my recent sessions? Just give me a "
"quick rundown.",
lambda a: sum(
k in a.lower()
for k in ("fan", "tax", "aquarium", "beehive", "apiary", "504", "checkout")
) >= 3,
"browse shape",
),
}
SYSTEM = (
"You are Hermes, a personal AI agent with persistent memory across "
"sessions. You have a session_search tool over the user's past "
"conversation history. Answer the user's question accurately and "
"concisely. Today is 2026-08-26."
)