# Compaction v2 — 4-transcript scorecard (2026-08-15, anchor-index build)
Four real 500K-token lineage transcripts from state.db (sweep campaign, GUI
desktop work, PR-merge campaign, ACP/PR review), 15-question recall exam
each. "recovery" = one session_search round-trip (FTS5+BM25 sim) against the
archived region. Lean build includes: 25K clamped tail, tail tool demotion,
chunked digests (noise-filtered, pristine tool contents), mechanical anchor
index, verbatim user messages, recovery footer, upgraded summarizer prompt.
> **Historical note (2026-08-30):** the "chunked digests" arm described here
> was later replaced — the detailed session log is now produced by the SAME
> single summary request (lean compaction makes exactly one auxiliary LLM
> call per attempt; no per-chunk digest calls). See #96603.
## Results (recall % @ retained tokens)
policy sweep gui prmerge acp AVG
uncompacted 93.3 @ 500K 96.7 @ 500K 96.7 @ 500K 100.0 @ 500K 96.7
current 93.3*@ 176K 26.7*@ 156K 33.3 @ 155K 30.0 @ 160K 45.8 @ 162K
lean 40.0 @ 62K 60.0 @ 41K 23.3 @ 44K 36.7 @ 50K 40.0 @ 49K
lean+recovery 70.0 @ 62K 80.0 @ 41K 43.3 @ 45K 80.0 @ 50K 68.3 @ 49K
* sweep/gui current scores are from the previous question banks (same
transcripts; banks regenerated in the 4-way run). prmerge/acp are clean
same-bank comparisons across all arms.
## Findings
1. LEAN+RECOVERY BEATS CURRENT BY +22.5pts ON AVERAGE (68.3 vs 45.8) AT 3.3x
FEWER TOKENS (49K vs 162K). It wins on 3 of 4 transcripts and loses only
sweep — the one transcript where current's fat tail got lucky with
restated facts (93.3 is bank-inflated luck; see finding 3 of the previous
scorecard).
2. THE ANCHOR INDEX FIXED THE NEEDLE-FACT CLASS. GUI closed-book went
23.3 -> 60.0 and GUI+recovery 46.7 -> 80.0 after mechanically indexing
exact identifiers (SHAs, ids, paths, error strings) instead of trusting
the summarizer with them. ACP+recovery hit 80.0.
3. TWO FRESH TRANSCRIPTS CONFIRM CURRENT IS WEAK, NOT STRONG: 33.3 and 30.0
at ~157K retained. The original sweep 93.3 was restatement luck, not
policy quality. Current's average is 45.8% for 162K tokens — lean+recovery
is 22 points better for less than a third of the spend.
4. prmerge IS THE HARD CASE for everyone (96.7 ceiling, best policy 43.3):
1.1M-token lineage truncated at 500K, dense multi-PR state. Recovery
misses there are mostly query formulation. Headroom, not a blocker.
5. Goal check (Teknium): tail = max(10K, 2.5%) ✓; summaries scoped to the
compacted region only ✓ (sentinel tripwire test); session_search pointer ✓
(+20-43pts measured); better accuracy AND more savings than current ✓
(+22.5pts at 0.30x tokens).
## Codex CLI head-to-head (same transcripts, same exams, same judge)
Real OpenAI Codex CLI (v0.147.0, gpt-5.6-sol, 258K window) run end-to-end on
the identical four transcripts: chunk files read via `codex exec` until its
REAL auto-compaction fired (verified `compacted` event in the rollout jsonl;
peak context 455-483K), then quizzed post-compaction from memory with the
same 15-question banks and scored by the same judge.
policy sweep gui prmerge acp AVG retained state
codex (real, post-cmp) 26.7% 40.0% 43.3% 36.7% 36.7% ~4.5K (opaque blob + user msgs)
hermes current 93.3%* 26.7%* 33.3% 30.0% 45.8% ~162K
hermes lean closed-book 40.0% 60.0% 23.3% 36.7% 40.0% ~49K
hermes lean+recovery 70.0% 80.0% 43.3% 80.0% 68.3% ~49K
Notes:
- codex answers from its own post-compaction session — the honest analog of
our closed-book arms. It has NO session_search equivalent (its rollout is
on disk but the agent cannot search it at runtime), so recovery has no
codex counterpart; that gap is exactly the differentiator lean leans on.
- Apples-to-apples closed-book: lean 40.0% vs codex 36.7% — parity-plus at
10x codex's retained state but 0.30x current's. With recovery: +31.6pts
over codex.
- codex ties lean+recovery on prmerge (43.3%) — the dense multi-PR campaign
is the hardest transcript for every policy and the clearest iteration
target.
- Methodology caveats: codex ingested transcripts as FILE READS (tool
outputs), not native conversation — this matches how its compaction
treats tool output (drops it all into the server-side summary) but is not
byte-identical to a native session. Its model (gpt-5.6-sol) also differs
from the answering model in our arms; scores compare COMPACTION PIPELINES
end-to-end, not models in isolation. One codex quiz reply was also
capped short (~1K chars for 15 answers), which its terse post-compaction
style invites.
## Recommendation
Ship lean as opt-in (compression.tail_mode: lean, legacy default), harness as
the permanent gate. Iterate prmerge-class recall behind the flag (query
mining, per-epoch anchor windows) before default flip.
## Appendix: full per-transcript detail
### Transcript: sweep
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| lean | 40.0% | 499,625 | 61,567 | 114.9 |
| lean+recovery | 70.0% | 499,625 | 61,792 | 114.8 |
15 exam questions (questions-30b95351c7.json)
1. **What is the reason given for never using 'git checkout pr-branch -- ' on stale branches?**
gold: `the stale file version silently deletes newer main code`
2. **According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?**
gold: `~60GB`
3. **Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?**
gold: `backgroundThrottling`
4. **What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?**
gold: `Auto merge is not allowed for this repository (enablePullRequestAutoMerge)`
5. **What is the specified 'Rule 0' that must be included in a subagent brief?**
gold: `load the skill first`
6. **In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?**
gold: `add config gate for title generation`
7. **Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?**
gold: `tests/test_packaging_metadata.py`
8. **What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?**
gold: `completed normally (exit code 0)`
9. **What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?**
gold: `relative exclude-newer makes the committed lock stale every day`
10. **How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?**
gold: `13`
11. **In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?**
gold: `-1`
12. **What is the full title block text for the technical schematic infographic generated for the Gateway Drain?**
gold: `GATEWAY DRAIN × CRON — SHUTDOWN CONTRACT`
13. **Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?**
gold: `82980`
14. **What is the specific Gist ID created for the PR infographic host in the cron cluster?**
gold: `ee33edd5804689243f974536ef7aecb9`
15. **What was the final merge SHA for Cluster D's Trigger-now PR #70638?**
gold: `f9d64b9a9d8b306f64851c1a13869d96ad5d7869`
15 exam questions (questions-5be475cde0.json)
1. **What exact command did the agent use to search for open issues related to a specific topic during Phase 1 of the cluster-sweep salvage?**
gold: `gh issue list --search "" --state open --limit 100 --json number,title`
2. **According to Teknium's design intent, what is the status of 'platform toolsets' in the codebase?**
gold: `platform toolsets are vestigial, never exposed`
3. **During the July sweep, which specific issue's config bridge was found to already exist at the exact line it was claimed to be missing?**
gold: `#32263`
4. **In the Aug 2026 cron-summarizer cluster sweep, which two PR numbers were discovered post-merge as the true first submitters?**
gold: `#60593, #61969`
5. **What is the recommended Git command to find when a specific symbol fix landed on the main branch?**
gold: `git log -S ""`
6. **Why did the #39719 salvage silently delete 236 lines of code from cli-config.yaml.example?**
gold: `the stale file version silently deletes newer main code`
7. **What is the rule for salvaging commits with placeholder identities like 'pwn@example.com'?**
gold: `do NOT cherry-pick. Surgical reapply as maintainer-authored commit, Co-authored-by the GitHub PR author`
8. **How should an agent handle a 'gh pr merge' 502 error?**
gold: `retry the same command once after the "Merge already in progress" settles (~45s); check PR state between attempts`
9. **Which two properties shape almost every design decision in Hermes according to the Development Guide?**
gold: `Per-conversation prompt caching is sacred and The core is a narrow waist; capability lives at the edges.`
10. **What error message does the live-checkout git guard display when blocking a history-rewriting command?**
gold: `Blocked: `git ` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module `
11. **What happened to the Desktop cluster's 'npm ci' command that resulted in an error writing to /tmp/ccH06T4r.s?**
gold: `No space left on device`
12. **What was the GraphQL API rate limit remaining for the user when the 'API rate limit already exceeded' error first occurred?**
gold: `0`
13. **Which PR was identified as the salvage of HexLab98's #85283 to fix hung inline API calls?**
gold: `#86645`
14. **Why did PR #79268 fix invisible overlays in the TUI?**
gold: `renderNodeToOutput skips boxes Yoga squeezes to height 0`
15. **What was the specific ModuleNotFoundError message caused by the wheel subpackage discovery trap in #34701?**
gold: `ModuleNotFoundError: No module named 'hermes_cli.dashboard_auth'`
15 exam questions (questions-36d3d87e0b.json)
1. **What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?**
gold: `#86617`
2. **According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?**
gold: `.env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.`
3. **What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?**
gold: `tests/hermes_cli/test_session_recovery_lost_and_found.py:327`
4. **What was the root cause of issue #73793 regarding mid-turn message rendering?**
gold: `redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row`
5. **Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?**
gold: `#84287`
6. **In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?**
gold: `pane sash grab band made asymmetric 1px/7px`
7. **Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?**
gold: `602028@ky-tech.com.cn`
8. **What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?**
gold: `Blocked: `git ` would rewrite Hermes's live source checkout`
9. **What is the core design principle regarding 'Narrow Waist' in Hermes development?**
gold: `The core is a narrow waist; capability lives at the edges.`
10. **What was the result of the rebase-merge attempt for PR #86589?**
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
11. **In the infographic style picker, what vibe is associated with the 'designers-republic' style?**
gold: `The Designers Republic: flat orange+violet vector schematic on pewter grey`
12. **Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?**
gold: `conflicts with main in 4 files and introduces a second competing display-dedupe scheme`
13. **What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?**
gold: `May 23 2026`
14. **What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?**
gold: `Property 'onToggleUnread' is missing in type`
15. **According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?**
gold: `Electron`
15 exam questions (questions-9c55c707b6.json)
1. **What two PR numbers are associated with the 'sidebar-nav-rows-and-overlay-panels.md' and 'hud-mode-internals.md' references in the initial tool content?**
gold: `#85162 and #82285`
2. **According to AGENTS.md, what is the 'one exception' to the rule that nothing should rebuild the system prompt mid-conversation?**
gold: `context compression`
3. **In the Contribution Rubric, what are the three allowed reasons for an automated triage sweeper to close a PR?**
gold: `implemented_on_main, cannot_reproduce, incoherent`
4. **Which contributor is credited with adding the 'Brazilian Portuguese localization' in PR #86292?**
gold: `@gui8515`
5. **What specific error message is reported in issue #83562 regarding the Windows Desktop update?**
gold: `Hermes backend exited (0)`
6. **What is the 'core problem' identified in the parallel-subagent-salvage-orchestration.md reference?**
gold: `subagents share the parent's worktree + main checkout`
7. **Why was the 'nix (macos-latest)' build failing in the salvage batches according to the orchestration reference?**
gold: `Nix build failed due to stale npm lockfile hash`
8. **Which subagent ID was assigned the goal of salvaging the 'inflight-journal duplicate-answer cluster'?**
gold: `sa-2-7318d0ba`
9. **In PR #86595, why was PR #80707 by upperagent excluded from the salvage?**
gold: `violating this PR's UI-read-only invariant`
10. **What was the root cause of the failure in Python tests slice 4/12 for PR #86597?**
gold: `AssertionError: assert 't2' == 't1'`
11. **What did the fix for issue #79157 in PR #86589 involve?**
gold: `pane sash grab band made asymmetric 1px/7px`
12. **According to the root cause analysis for #73793, which two files spliced the mid-turn user message at streamIndex?**
gold: `use-prompt-actions/index.ts and session-tile-actions.ts`
13. **What was the head SHA for the 'salvage/desktop-busy-state' branch in PR #86604?**
gold: `bddadfe9e21e24b3d52e2b15f138c42474dede42`
14. **Why was the merge of PR #86589 aborted during the 'Merge all' command?**
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
15. **What specific file was modified to fix the 'artifacts page timestamps render 1970' issue via PR #86749?**
gold: `apps/desktop/src/app/session/hooks/use-session-actions/utils.ts`
15 exam questions (questions-703ae2774a.json)
1. **Which PR number added the public subagent lifecycle API?**
gold: `#63359`
2. **What is the name of the typed service added to PluginContext for launching and monitoring child sessions?**
gold: `subagent_lifecycle`
3. **How many contract and security tests were included with the subagent lifecycle API PR?**
gold: `42`
4. **What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?**
gold: `cannot currently trigger a turn in an existing gateway session`
5. **Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?**
gold: `#64436`
6. **What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?**
gold: `must compile, must start with ≥2 literal characters`
7. **Which contributor authorized sustained help for the Phase 0–1 expansion track?**
gold: `Daniel`
8. **What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?**
gold: `#64204`
9. **What configuration setting is required to opt-in to reasoning deltas in streaming output?**
gold: `plugins.stream_reasoning_deltas: true`
10. **How many additions and across how many files were made in PR #63359?**
gold: `650 additions across 4 files`
11. **What is the name of the reference plugin shipped with the redaction pattern registry?**
gold: `nvapi-redaction`
12. **List the four observer-only streaming output plugin hooks added in PR #64317.**
gold: `on_stream_start, on_stream_delta, on_stream_end, on_interim_message`
13. **What was addressed in the update to PR #58541 regarding lifecycle hooks?**
gold: `created-hook timing and added kanban_task_promoted`
14. **Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?**
gold: `#64230`
15. **What was the Round 3 review's outcome for PR #63359 and @asimons81?**
gold: `sub-issue #65447`
15 exam questions (questions-f45358df19.json)
1. **What was the specific reason Teknium gave for reverting PR #30179 in July 2026?**
gold: `WTF??? REVERT! DAMMIT`
2. **On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?**
gold: `PR #6391`
3. **Which file path should be checked for the canonical list of provider models?**
gold: `hermes_cli/models.py`
4. **What was the identified bug in PR #2314 regarding provider names?**
gold: `checking for "alibaba-coding-plan"`
5. **What is the mandatory line limit for PR reviews requested by Teknium?**
gold: `<= 15 lines`
6. **What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?**
gold: `Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix mod`
7. **Why was PR #74658 necessary to fix Slack 'broken on main'?**
gold: `SlackResponse isn't a dict subclass, so every gate is always False.`
8. **What was the final merge commit SHA for the Slack SDK response fix on main?**
gold: `24ba86627515ad5fda69a39ef338c365713448bc`
9. **In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?**
gold: `Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1`
10. **What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?**
gold: `UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'`
11. **Which test failed in CI slice 5/12 for the kanban delivery modes PR?**
gold: `tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post`
12. **According to the transcript, why is squash merging banned as of July 2026?**
gold: `DevOps policy`
13. **Which contributor authored the first fix for issue #73030 in July?**
gold: `@Tranquil-Flow`
14. **What was the 'Superman-style' shield error in the first generation of the Kanban infographic?**
gold: `red "S" inside the diamond shield`
15. **What specific file was modified to add the 'scope_id_for_chat' method for Slack?**
gold: `plugins/platforms/slack/adapter.py`