73 lines
3.2 KiB
Markdown
73 lines
3.2 KiB
Markdown
# session_search Schema A/B Eval
|
||
|
||
Live tool-use A/B harness measuring whether changes to the `session_search`
|
||
tool schema (description/param diets, response hints) affect a model's
|
||
ability to actually use the tool. Built for PR #95570 (schema diet
|
||
1,570 → 695 tok/call), where the question was: "does moving the teaching
|
||
essay out of the schema and into response hints confuse models?"
|
||
|
||
Unlike the readtool/browser evals, this one does not run the full AIAgent —
|
||
it runs a minimal agent loop where the ONLY variable between arms is
|
||
`tools/session_search_tool.py` extracted from two git refs. Everything else
|
||
(seeded DB, tasks, oracles, system prompt, temperature) is held constant.
|
||
|
||
## What it measures
|
||
|
||
Six tasks against a deterministic seeded session DB (plus a second
|
||
"work"-profile DB), each with a programmatic oracle — no LLM judging:
|
||
|
||
| task | shape exercised | oracle |
|
||
|---|---|---|
|
||
| `t1_discover` | discovery | answer contains `pglogical` |
|
||
| `t2_scroll` | forced forward scroll — fact planted OUTSIDE the ±5 window and outside bookends | `statement_timeout` + `45` |
|
||
| `t3_broaden` | AND-query miss → must broaden (OR / fewer terms); the two query nouns never co-occur in one message | port `3000` |
|
||
| `t4_link` | verbatim `@session:` link emission | link present, NOT backticked/markdown |
|
||
| `t5_profile` | `@session:work/<id>` profile link resolution (read shape) | `vault` + `90` |
|
||
| `t6_browse` | browse shape | ≥3 recent-session topics named |
|
||
|
||
Metrics per run: oracle pass, tool-call count, malformed/errored calls,
|
||
first-call prompt tokens (measures the schema itself), total tokens, wall.
|
||
|
||
## Running
|
||
|
||
```bash
|
||
# Arms are git refs; the runner extracts tools/session_search_tool.py
|
||
# from each and imports them side by side.
|
||
python3 evals/session_search_schema/runner.py \
|
||
--base origin/main --cand HEAD \
|
||
--model qwen/qwen3-coder-30b-a3b-instruct --reps 3
|
||
|
||
python3 evals/session_search_schema/report.py results/<model>.jsonl
|
||
```
|
||
|
||
Requires `OPENROUTER_API_KEY` in `~/.hermes/.env` (or env). The seeded DB is
|
||
rebuilt fresh in a temp dir per invocation; nothing touches your real
|
||
`state.db`.
|
||
|
||
Rules of engagement (hermesbench discipline):
|
||
|
||
- 3 reps minimum; n=1 cell differences are noise — pull the transcript
|
||
(`calls` + `final` in the JSONL) before diagnosing any miss.
|
||
- Provider noise (zero tool calls AND empty final) gets one retry, applied
|
||
identically to both arms; retries are logged.
|
||
- Report per-task x/N for BOTH arms with the same denominators. Never
|
||
exclude runs from one arm only.
|
||
- Weak/mid models are the signal; frontier models mask schema ergonomics.
|
||
|
||
## Reference results (PR #95570, 2026-08-26)
|
||
|
||
108 runs, 3 models × 6 tasks × 3 reps × 2 arms
|
||
(base `2b8b4542e` = pre-diet main, cand `d8a78a4dc` = diet):
|
||
|
||
| model | base | diet | avg tok/task |
|
||
|---|---|---|---|
|
||
| qwen3-coder-30b | 16/18 | 18/18 | 11.1k → 7.1k |
|
||
| gpt-5.6-luna | 18/18 | 17/18 | 5.4k → 3.7k |
|
||
| gpt-5.6-terra | 15/18 | 17/18 | 8.0k → 5.0k |
|
||
| **total** | 49/54 | **52/54** | 7.0k → 5.3k |
|
||
|
||
Findings: diet arm held/gained accuracy; scroll `hint` measurably helped the
|
||
paging task; one 1/9 luna markdown-link miss on the diet arm; both arms
|
||
surfaced the pre-existing `around_message_id=0` falsy-sentinel bug
|
||
(issue #94792 / PR #79118).
|