358 lines
22 KiB
Markdown
358 lines
22 KiB
Markdown
# Compaction v2 — 4-transcript scorecard (2026-08-15, anchor-index build)
|
||
|
||
Four real 500K-token lineage transcripts from state.db (sweep campaign, GUI
|
||
desktop work, PR-merge campaign, ACP/PR review), 15-question recall exam
|
||
each. "recovery" = one session_search round-trip (FTS5+BM25 sim) against the
|
||
archived region. Lean build includes: 25K clamped tail, tail tool demotion,
|
||
chunked digests (noise-filtered, pristine tool contents), mechanical anchor
|
||
index, verbatim user messages, recovery footer, upgraded summarizer prompt.
|
||
|
||
> **Historical note (2026-08-30):** the "chunked digests" arm described here
|
||
> was later replaced — the detailed session log is now produced by the SAME
|
||
> single summary request (lean compaction makes exactly one auxiliary LLM
|
||
> call per attempt; no per-chunk digest calls). See #96603.
|
||
|
||
## Results (recall % @ retained tokens)
|
||
|
||
policy sweep gui prmerge acp AVG
|
||
uncompacted 93.3 @ 500K 96.7 @ 500K 96.7 @ 500K 100.0 @ 500K 96.7
|
||
current 93.3*@ 176K 26.7*@ 156K 33.3 @ 155K 30.0 @ 160K 45.8 @ 162K
|
||
lean 40.0 @ 62K 60.0 @ 41K 23.3 @ 44K 36.7 @ 50K 40.0 @ 49K
|
||
lean+recovery 70.0 @ 62K 80.0 @ 41K 43.3 @ 45K 80.0 @ 50K 68.3 @ 49K
|
||
|
||
* sweep/gui current scores are from the previous question banks (same
|
||
transcripts; banks regenerated in the 4-way run). prmerge/acp are clean
|
||
same-bank comparisons across all arms.
|
||
|
||
## Findings
|
||
|
||
1. LEAN+RECOVERY BEATS CURRENT BY +22.5pts ON AVERAGE (68.3 vs 45.8) AT 3.3x
|
||
FEWER TOKENS (49K vs 162K). It wins on 3 of 4 transcripts and loses only
|
||
sweep — the one transcript where current's fat tail got lucky with
|
||
restated facts (93.3 is bank-inflated luck; see finding 3 of the previous
|
||
scorecard).
|
||
|
||
2. THE ANCHOR INDEX FIXED THE NEEDLE-FACT CLASS. GUI closed-book went
|
||
23.3 -> 60.0 and GUI+recovery 46.7 -> 80.0 after mechanically indexing
|
||
exact identifiers (SHAs, ids, paths, error strings) instead of trusting
|
||
the summarizer with them. ACP+recovery hit 80.0.
|
||
|
||
3. TWO FRESH TRANSCRIPTS CONFIRM CURRENT IS WEAK, NOT STRONG: 33.3 and 30.0
|
||
at ~157K retained. The original sweep 93.3 was restatement luck, not
|
||
policy quality. Current's average is 45.8% for 162K tokens — lean+recovery
|
||
is 22 points better for less than a third of the spend.
|
||
|
||
4. prmerge IS THE HARD CASE for everyone (96.7 ceiling, best policy 43.3):
|
||
1.1M-token lineage truncated at 500K, dense multi-PR state. Recovery
|
||
misses there are mostly query formulation. Headroom, not a blocker.
|
||
|
||
5. Goal check (Teknium): tail = max(10K, 2.5%) ✓; summaries scoped to the
|
||
compacted region only ✓ (sentinel tripwire test); session_search pointer ✓
|
||
(+20-43pts measured); better accuracy AND more savings than current ✓
|
||
(+22.5pts at 0.30x tokens).
|
||
|
||
|
||
## Codex CLI head-to-head (same transcripts, same exams, same judge)
|
||
|
||
Real OpenAI Codex CLI (v0.147.0, gpt-5.6-sol, 258K window) run end-to-end on
|
||
the identical four transcripts: chunk files read via `codex exec` until its
|
||
REAL auto-compaction fired (verified `compacted` event in the rollout jsonl;
|
||
peak context 455-483K), then quizzed post-compaction from memory with the
|
||
same 15-question banks and scored by the same judge.
|
||
|
||
policy sweep gui prmerge acp AVG retained state
|
||
codex (real, post-cmp) 26.7% 40.0% 43.3% 36.7% 36.7% ~4.5K (opaque blob + user msgs)
|
||
hermes current 93.3%* 26.7%* 33.3% 30.0% 45.8% ~162K
|
||
hermes lean closed-book 40.0% 60.0% 23.3% 36.7% 40.0% ~49K
|
||
hermes lean+recovery 70.0% 80.0% 43.3% 80.0% 68.3% ~49K
|
||
|
||
Notes:
|
||
- codex answers from its own post-compaction session — the honest analog of
|
||
our closed-book arms. It has NO session_search equivalent (its rollout is
|
||
on disk but the agent cannot search it at runtime), so recovery has no
|
||
codex counterpart; that gap is exactly the differentiator lean leans on.
|
||
- Apples-to-apples closed-book: lean 40.0% vs codex 36.7% — parity-plus at
|
||
10x codex's retained state but 0.30x current's. With recovery: +31.6pts
|
||
over codex.
|
||
- codex ties lean+recovery on prmerge (43.3%) — the dense multi-PR campaign
|
||
is the hardest transcript for every policy and the clearest iteration
|
||
target.
|
||
- Methodology caveats: codex ingested transcripts as FILE READS (tool
|
||
outputs), not native conversation — this matches how its compaction
|
||
treats tool output (drops it all into the server-side summary) but is not
|
||
byte-identical to a native session. Its model (gpt-5.6-sol) also differs
|
||
from the answering model in our arms; scores compare COMPACTION PIPELINES
|
||
end-to-end, not models in isolation. One codex quiz reply was also
|
||
capped short (~1K chars for 15 answers), which its terse post-compaction
|
||
style invites.
|
||
|
||
## Recommendation
|
||
|
||
Ship lean as opt-in (compression.tail_mode: lean, legacy default), harness as
|
||
the permanent gate. Iterate prmerge-class recall behind the flag (query
|
||
mining, per-epoch anchor windows) before default flip.
|
||
|
||
|
||
## Appendix: full per-transcript detail
|
||
|
||
### Transcript: sweep
|
||
|
||
| policy | recall | tokens before | tokens after | compress s |
|
||
|---|---|---|---|---|
|
||
| lean | 40.0% | 499,625 | 61,567 | 114.9 |
|
||
| lean+recovery | 70.0% | 499,625 | 61,792 | 114.8 |
|
||
|
||
<details><summary>15 exam questions (questions-30b95351c7.json)</summary>
|
||
|
||
1. **What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?**
|
||
gold: `the stale file version silently deletes newer main code`
|
||
2. **According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?**
|
||
gold: `~60GB`
|
||
3. **Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?**
|
||
gold: `backgroundThrottling`
|
||
4. **What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?**
|
||
gold: `Auto merge is not allowed for this repository (enablePullRequestAutoMerge)`
|
||
5. **What is the specified 'Rule 0' that must be included in a subagent brief?**
|
||
gold: `load the skill first`
|
||
6. **In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?**
|
||
gold: `add config gate for title generation`
|
||
7. **Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?**
|
||
gold: `tests/test_packaging_metadata.py`
|
||
8. **What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?**
|
||
gold: `completed normally (exit code 0)`
|
||
9. **What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?**
|
||
gold: `relative exclude-newer makes the committed lock stale every day`
|
||
10. **How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?**
|
||
gold: `13`
|
||
11. **In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?**
|
||
gold: `-1`
|
||
12. **What is the full title block text for the technical schematic infographic generated for the Gateway Drain?**
|
||
gold: `GATEWAY DRAIN × CRON — SHUTDOWN CONTRACT`
|
||
13. **Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?**
|
||
gold: `82980`
|
||
14. **What is the specific Gist ID created for the PR infographic host in the cron cluster?**
|
||
gold: `ee33edd5804689243f974536ef7aecb9`
|
||
15. **What was the final merge SHA for Cluster D's Trigger-now PR #70638?**
|
||
gold: `f9d64b9a9d8b306f64851c1a13869d96ad5d7869`
|
||
|
||
</details>
|
||
|
||
<details><summary>15 exam questions (questions-5be475cde0.json)</summary>
|
||
|
||
1. **What exact command did the agent use to search for open issues related to a specific topic during Phase 1 of the cluster-sweep salvage?**
|
||
gold: `gh issue list --search "<topic>" --state open --limit 100 --json number,title`
|
||
2. **According to Teknium's design intent, what is the status of 'platform toolsets' in the codebase?**
|
||
gold: `platform toolsets are vestigial, never exposed`
|
||
3. **During the July sweep, which specific issue's config bridge was found to already exist at the exact line it was claimed to be missing?**
|
||
gold: `#32263`
|
||
4. **In the Aug 2026 cron-summarizer cluster sweep, which two PR numbers were discovered post-merge as the true first submitters?**
|
||
gold: `#60593, #61969`
|
||
5. **What is the recommended Git command to find when a specific symbol fix landed on the main branch?**
|
||
gold: `git log -S "<symbol>"`
|
||
6. **Why did the #39719 salvage silently delete 236 lines of code from cli-config.yaml.example?**
|
||
gold: `the stale file version silently deletes newer main code`
|
||
7. **What is the rule for salvaging commits with placeholder identities like 'pwn@example.com'?**
|
||
gold: `do NOT cherry-pick. Surgical reapply as maintainer-authored commit, Co-authored-by the GitHub PR author`
|
||
8. **How should an agent handle a 'gh pr merge' 502 error?**
|
||
gold: `retry the same command once after the "Merge already in progress" settles (~45s); check PR state between attempts`
|
||
9. **Which two properties shape almost every design decision in Hermes according to the Development Guide?**
|
||
gold: `Per-conversation prompt caching is sacred and The core is a narrow waist; capability lives at the edges.`
|
||
10. **What error message does the live-checkout git guard display when blocking a history-rewriting command?**
|
||
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module `
|
||
11. **What happened to the Desktop cluster's 'npm ci' command that resulted in an error writing to /tmp/ccH06T4r.s?**
|
||
gold: `No space left on device`
|
||
12. **What was the GraphQL API rate limit remaining for the user when the 'API rate limit already exceeded' error first occurred?**
|
||
gold: `0`
|
||
13. **Which PR was identified as the salvage of HexLab98's #85283 to fix hung inline API calls?**
|
||
gold: `#86645`
|
||
14. **Why did PR #79268 fix invisible overlays in the TUI?**
|
||
gold: `renderNodeToOutput skips boxes Yoga squeezes to height 0`
|
||
15. **What was the specific ModuleNotFoundError message caused by the wheel subpackage discovery trap in #34701?**
|
||
gold: `ModuleNotFoundError: No module named 'hermes_cli.dashboard_auth'`
|
||
|
||
</details>
|
||
|
||
### Transcript: gui
|
||
|
||
| policy | recall | tokens before | tokens after | compress s |
|
||
|---|---|---|---|---|
|
||
| lean | 60.0% | 499,818 | 41,232 | 118.1 |
|
||
| lean+recovery | 80.0% | 499,818 | 41,306 | 115.2 |
|
||
|
||
<details><summary>15 exam questions (questions-36d3d87e0b.json)</summary>
|
||
|
||
1. **What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?**
|
||
gold: `#86617`
|
||
2. **According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?**
|
||
gold: `.env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.`
|
||
3. **What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?**
|
||
gold: `tests/hermes_cli/test_session_recovery_lost_and_found.py:327`
|
||
4. **What was the root cause of issue #73793 regarding mid-turn message rendering?**
|
||
gold: `redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row`
|
||
5. **Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?**
|
||
gold: `#84287`
|
||
6. **In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?**
|
||
gold: `pane sash grab band made asymmetric 1px/7px`
|
||
7. **Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?**
|
||
gold: `602028@ky-tech.com.cn`
|
||
8. **What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?**
|
||
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout`
|
||
9. **What is the core design principle regarding 'Narrow Waist' in Hermes development?**
|
||
gold: `The core is a narrow waist; capability lives at the edges.`
|
||
10. **What was the result of the rebase-merge attempt for PR #86589?**
|
||
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
|
||
11. **In the infographic style picker, what vibe is associated with the 'designers-republic' style?**
|
||
gold: `The Designers Republic: flat orange+violet vector schematic on pewter grey`
|
||
12. **Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?**
|
||
gold: `conflicts with main in 4 files and introduces a second competing display-dedupe scheme`
|
||
13. **What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?**
|
||
gold: `May 23 2026`
|
||
14. **What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?**
|
||
gold: `Property 'onToggleUnread' is missing in type`
|
||
15. **According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?**
|
||
gold: `Electron`
|
||
|
||
</details>
|
||
|
||
<details><summary>15 exam questions (questions-9c55c707b6.json)</summary>
|
||
|
||
1. **What two PR numbers are associated with the 'sidebar-nav-rows-and-overlay-panels.md' and 'hud-mode-internals.md' references in the initial tool content?**
|
||
gold: `#85162 and #82285`
|
||
2. **According to AGENTS.md, what is the 'one exception' to the rule that nothing should rebuild the system prompt mid-conversation?**
|
||
gold: `context compression`
|
||
3. **In the Contribution Rubric, what are the three allowed reasons for an automated triage sweeper to close a PR?**
|
||
gold: `implemented_on_main, cannot_reproduce, incoherent`
|
||
4. **Which contributor is credited with adding the 'Brazilian Portuguese localization' in PR #86292?**
|
||
gold: `@gui8515`
|
||
5. **What specific error message is reported in issue #83562 regarding the Windows Desktop update?**
|
||
gold: `Hermes backend exited (0)`
|
||
6. **What is the 'core problem' identified in the parallel-subagent-salvage-orchestration.md reference?**
|
||
gold: `subagents share the parent's worktree + main checkout`
|
||
7. **Why was the 'nix (macos-latest)' build failing in the salvage batches according to the orchestration reference?**
|
||
gold: `Nix build failed due to stale npm lockfile hash`
|
||
8. **Which subagent ID was assigned the goal of salvaging the 'inflight-journal duplicate-answer cluster'?**
|
||
gold: `sa-2-7318d0ba`
|
||
9. **In PR #86595, why was PR #80707 by upperagent excluded from the salvage?**
|
||
gold: `violating this PR's UI-read-only invariant`
|
||
10. **What was the root cause of the failure in Python tests slice 4/12 for PR #86597?**
|
||
gold: `AssertionError: assert 't2' == 't1'`
|
||
11. **What did the fix for issue #79157 in PR #86589 involve?**
|
||
gold: `pane sash grab band made asymmetric 1px/7px`
|
||
12. **According to the root cause analysis for #73793, which two files spliced the mid-turn user message at streamIndex?**
|
||
gold: `use-prompt-actions/index.ts and session-tile-actions.ts`
|
||
13. **What was the head SHA for the 'salvage/desktop-busy-state' branch in PR #86604?**
|
||
gold: `bddadfe9e21e24b3d52e2b15f138c42474dede42`
|
||
14. **Why was the merge of PR #86589 aborted during the 'Merge all' command?**
|
||
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
|
||
15. **What specific file was modified to fix the 'artifacts page timestamps render 1970' issue via PR #86749?**
|
||
gold: `apps/desktop/src/app/session/hooks/use-session-actions/utils.ts`
|
||
|
||
</details>
|
||
|
||
### Transcript: prmerge
|
||
|
||
| policy | recall | tokens before | tokens after | compress s |
|
||
|---|---|---|---|---|
|
||
| uncompacted_control | 96.7% | 499,663 | 499,663 | — |
|
||
| current | 33.3% | 499,663 | 155,399 | 14.9 |
|
||
| lean | 23.3% | 499,663 | 44,419 | 105.4 |
|
||
| lean+recovery | 43.3% | 499,663 | 44,977 | 95.8 |
|
||
|
||
<details><summary>15 exam questions (questions-703ae2774a.json)</summary>
|
||
|
||
1. **Which PR number added the public subagent lifecycle API?**
|
||
gold: `#63359`
|
||
2. **What is the name of the typed service added to PluginContext for launching and monitoring child sessions?**
|
||
gold: `subagent_lifecycle`
|
||
3. **How many contract and security tests were included with the subagent lifecycle API PR?**
|
||
gold: `42`
|
||
4. **What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?**
|
||
gold: `cannot currently trigger a turn in an existing gateway session`
|
||
5. **Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?**
|
||
gold: `#64436`
|
||
6. **What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?**
|
||
gold: `must compile, must start with ≥2 literal characters`
|
||
7. **Which contributor authorized sustained help for the Phase 0–1 expansion track?**
|
||
gold: `Daniel`
|
||
8. **What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?**
|
||
gold: `#64204`
|
||
9. **What configuration setting is required to opt-in to reasoning deltas in streaming output?**
|
||
gold: `plugins.stream_reasoning_deltas: true`
|
||
10. **How many additions and across how many files were made in PR #63359?**
|
||
gold: `650 additions across 4 files`
|
||
11. **What is the name of the reference plugin shipped with the redaction pattern registry?**
|
||
gold: `nvapi-redaction`
|
||
12. **List the four observer-only streaming output plugin hooks added in PR #64317.**
|
||
gold: `on_stream_start, on_stream_delta, on_stream_end, on_interim_message`
|
||
13. **What was addressed in the update to PR #58541 regarding lifecycle hooks?**
|
||
gold: `created-hook timing and added kanban_task_promoted`
|
||
14. **Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?**
|
||
gold: `#64230`
|
||
15. **What was the Round 3 review's outcome for PR #63359 and @asimons81?**
|
||
gold: `sub-issue #65447`
|
||
|
||
</details>
|
||
|
||
### Transcript: acp
|
||
|
||
| policy | recall | tokens before | tokens after | compress s |
|
||
|---|---|---|---|---|
|
||
| uncompacted_control | 100.0% | 498,906 | 498,906 | — |
|
||
| current | 30.0% | 498,906 | 160,223 | 15.8 |
|
||
| lean | 36.7% | 498,906 | 49,523 | 143.3 |
|
||
| lean+recovery | 80.0% | 498,906 | 49,721 | 135.6 |
|
||
|
||
<details><summary>15 exam questions (questions-f45358df19.json)</summary>
|
||
|
||
1. **What was the specific reason Teknium gave for reverting PR #30179 in July 2026?**
|
||
gold: `WTF??? REVERT! DAMMIT`
|
||
2. **On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?**
|
||
gold: `PR #6391`
|
||
3. **Which file path should be checked for the canonical list of provider models?**
|
||
gold: `hermes_cli/models.py`
|
||
4. **What was the identified bug in PR #2314 regarding provider names?**
|
||
gold: `checking for "alibaba-coding-plan"`
|
||
5. **What is the mandatory line limit for PR reviews requested by Teknium?**
|
||
gold: `<= 15 lines`
|
||
6. **What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?**
|
||
gold: `Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix mod`
|
||
7. **Why was PR #74658 necessary to fix Slack 'broken on main'?**
|
||
gold: `SlackResponse isn't a dict subclass, so every gate is always False.`
|
||
8. **What was the final merge commit SHA for the Slack SDK response fix on main?**
|
||
gold: `24ba86627515ad5fda69a39ef338c365713448bc`
|
||
9. **In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?**
|
||
gold: `Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1`
|
||
10. **What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?**
|
||
gold: `UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'`
|
||
11. **Which test failed in CI slice 5/12 for the kanban delivery modes PR?**
|
||
gold: `tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post`
|
||
12. **According to the transcript, why is squash merging banned as of July 2026?**
|
||
gold: `DevOps policy`
|
||
13. **Which contributor authored the first fix for issue #73030 in July?**
|
||
gold: `@Tranquil-Flow`
|
||
14. **What was the 'Superman-style' shield error in the first generation of the Kanban infographic?**
|
||
gold: `red "S" inside the diamond shield`
|
||
15. **What specific file was modified to add the 'scope_id_for_chat' method for Slack?**
|
||
gold: `plugins/platforms/slack/adapter.py`
|
||
|
||
</details>
|
||
|
||
## Methodology notes
|
||
|
||
- Transcripts: 4 real session lineages reconstructed from a state.db copy
|
||
(sweep campaign 42 rotations / GUI desktop 34 / PR-merge 17 / ACP review
|
||
17), chronological 500K-token prefix, tool-group aligned.
|
||
- Question generation: main model, from the region the CURRENT policy would
|
||
summarize (most conservative boundary), cached per transcript so every
|
||
policy answers the identical exam.
|
||
- Answering: fresh LLM sees ONLY the post-compaction context (closed-book) or
|
||
context + one FTS5+BM25 search round-trip over the archived region
|
||
(+recovery). Judge sees gold; answerer never does. Scoring 2/1/0.
|
||
- Known caveats: 15 questions/transcript => +-1 question ~ 3.3pts noise;
|
||
sweep/gui current-policy rows predate a question-bank regeneration
|
||
(prmerge/acp are same-bank across all arms); the recovery sim conservatively
|
||
approximates production session_search (same engine, no windowing).
|
||
- Cost shape: lean compaction = ~25 aux-model digest calls (~2min, one-time
|
||
per compaction) vs 1 call today; every post-compaction turn is ~110K input
|
||
tokens cheaper. Break-even ~1 turn.
|