yilsem/aiturk-hermes-ide
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license

Browse Source
This commit is contained in:
yilsem
2026-09-05 13:26:46 +03:00
commit 03634b1ca3
11340 changed files with 3442369 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evals/compaction/README.md
+82
View File
@@ -0,0 +1,82 @@
# Compaction Eval Harness
Measures what context compaction actually costs in *recall*, not just tokens.
## What it does
1. Takes a real long transcript (JSON: `{"messages": [...]}`, chat format).
2. Generates a bank of factual recall questions from the region that
compaction will summarize away (cached per transcript for reproducibility).
3. Runs the transcript through `ContextCompressor.compress()` under each
policy in the matrix (current default, aggressive tail, codex-style, ...).
4. For each policy, asks a fresh LLM the recall questions with ONLY the
post-compaction context, and judges answers against gold.
5. Emits a scorecard: recall accuracy vs tokens retained, per policy.
## Usage
```bash
# from repo root, venv active
python evals/compaction/runner.py \
--transcript /path/to/lineage.json \
--policies current,aggressive,floor10k \
--questions 15 \
--out evals/compaction/results/run1
python evals/compaction/report.py evals/compaction/results/run1
```
Transcripts are NOT committed (they contain real session data). Point
`--transcript` at a local file. See `fixtures.py` for the expected shape and
a synthetic-transcript generator used by CI smoke tests.
## Building transcripts from real sessions (`scripts/`)
Compaction rotations mean a single active session rarely exceeds ~300K
tokens, but the *lineage* (parent→children chain) carries the full
uncompacted history. The scripts reconstruct those into eval transcripts:
```bash
# 1. ALWAYS copy the DB first — never point at the live state.db
cp ~/.hermes/state.db /tmp/state_copy.db
# 2. Find big lineages (sessions with parent_session_id form chains), then:
python evals/compaction/scripts/reconstruct_lineage.py \
/tmp/state_copy.db <root_session_id> /tmp/lineage.json
# 3. (optional) Replay a 500K prefix through one checkout's compressor and
# dump before/after for the HTML viewer:
python evals/compaction/scripts/replay_lineage.py <checkout> /tmp/lineage.json out.json 500000
python evals/compaction/scripts/build_html_report.py <runs_dir> report.html
```
`reconstruct_lineage.py` walks the whole descendant tree chronologically,
dedupes rotation-copied rows by content hash, strips synthetic compaction
artifacts (summaries, todo snapshots), and resolves the system prompt through
the `system_prompts` dedup table (sessions only carry a hash). The HTML
report renders before/after transcripts side by side with compaction
artifacts color-coded.
## Region-scoping tripwire
`test_region_scoping.py` plants sentinels in head/middle/tail and asserts the
summarizer's serialized-turns input carries ONLY the middle (compacted)
region in both legacy and lean modes. Run it directly or via pytest.
## Policies
Defined in `policies.py`. Each policy maps to `ContextCompressor` constructor
kwargs plus optional attribute overrides applied post-construction (e.g.
`tail_token_budget`). Add new policies there — the runner picks them up by
name.
## Notes
- Question generation and judging use `agent.auxiliary_client.call_llm`
(same transport the compressor uses), so the harness needs a configured
provider. Costs real tokens: ~(policies x questions) answer calls plus
one generation and one judge pass.
- Accuracy is judged 2/1/0 (correct / partial / wrong); the scorecard
reports normalized percent. The judge sees gold answers, the answerer
does not.
- `--also-uncompacted` adds a control arm that answers from the full
original transcript — the recall ceiling.
evals/compaction/fixtures.py
+79
View File
@@ -0,0 +1,79 @@
"""Transcript fixtures for the compaction eval harness.
Real transcripts are supplied by path (never committed). This module loads
them, estimates tokens the same way the harness scores them, and can generate
a small synthetic transcript so CI smoke tests run without real data.
"""
from __future__ import annotations
import json
import random
from typing import Any, Dict, List
def estimate_tokens(msg: Dict[str, Any]) -> int:
"""Chars/4 estimate, matching the harness's scoring convention."""
total = len(msg.get("content") or "") if isinstance(msg.get("content"), str) else 0
tc = msg.get("tool_calls")
if tc:
total += len(json.dumps(tc, default=str))
return total // 4
def total_tokens(messages: List[Dict[str, Any]]) -> int:
return sum(estimate_tokens(m) for m in messages)
def load_transcript(path: str, cap_tokens: int | None = None) -> List[Dict[str, Any]]:
"""Load a transcript JSON ({"messages": [...]}) and optionally cap it.
The cap takes the chronological prefix, then drops trailing assistant
tool_calls whose results were cut off so the input is well-formed.
"""
data = json.load(open(path, encoding="utf-8"))
msgs = data["messages"] if isinstance(data, dict) else data
if cap_tokens is None:
return msgs
prefix: List[Dict[str, Any]] = []
running = 0
for m in msgs:
t = estimate_tokens(m)
if running + t > cap_tokens and len(prefix) > 10:
break
prefix.append(m)
running += t
while prefix and prefix[-1].get("tool_calls"):
prefix.pop()
return prefix
def synthetic_transcript(n_turns: int = 60, seed: int = 7) -> List[Dict[str, Any]]:
"""Deterministic fake transcript with plantable facts for smoke tests.
Every 10th turn plants a distinctive fact ("The deploy code for region
N is XYZ") so smoke tests can assert recall mechanics without an LLM.
"""
rng = random.Random(seed)
msgs: List[Dict[str, Any]] = [
{"role": "system", "content": "You are a test agent."},
{"role": "user", "content": "Work through the checklist and remember the codes."},
]
for i in range(n_turns):
fact = ""
if i % 10 == 0:
fact = f" The deploy code for region {i // 10} is Z{rng.randint(1000, 9999)}."
msgs.append({
"role": "assistant",
"content": f"Working on step {i}.{fact}",
"tool_calls": [{
"id": f"c{i}",
"function": {"name": "terminal", "arguments": json.dumps({"command": f"echo step {i}"})},
}],
})
msgs.append({
"role": "tool",
"tool_call_id": f"c{i}",
"content": ("step output " * 200) + f"result-{i}",
})
msgs.append({"role": "assistant", "content": "Checklist complete."})
return msgs
evals/compaction/policies.py
+54
View File
@@ -0,0 +1,54 @@
"""Compaction policy matrix.
Each policy is a name -> spec mapping. A spec has:
ctor: extra kwargs for ContextCompressor(...)
attrs: attribute overrides applied after construction (lets us pin
tail_token_budget and other derived values without touching the
class)
The runner constructs one compressor per policy and calls
compress(force=True) with the transcript's estimated tokens.
"""
from __future__ import annotations
from typing import Any, Dict
# Window we evaluate against (fable-5 class model).
EVAL_MODEL = "anthropic/claude-fable-5"
EVAL_WINDOW = 1_000_000
POLICIES: Dict[str, Dict[str, Any]] = {
# Shipping behavior, untouched.
"current": {
"ctor": {},
"attrs": {},
},
# Proposed: tail = max(10K, 0.025% ... interpreted as 2.5% of window)
# capped hard at 25K on a 1M model. protect_last_n stays for message-count
# floor semantics.
"tail25k": {
"ctor": {},
"attrs": {"tail_token_budget": 25_000},
},
# Hard floor variant: minimum viable tail.
"tail10k": {
"ctor": {},
"attrs": {"tail_token_budget": 10_000},
},
# Codex posture: nearly no tail; summary carries everything.
"codex_style": {
"ctor": {"protect_last_n": 3},
"attrs": {"tail_token_budget": 2_000},
},
# Compaction-v2 lean mode: clamped 2.5% tail + tail tool demotion +
# verbatim user messages in summary + session_search recovery pointers.
"lean": {
"ctor": {"tail_mode": "lean"},
"attrs": {"_session_id": "eval-session"},
},
}
def apply_policy(compressor, spec: Dict[str, Any]):
for key, value in (spec.get("attrs") or {}).items():
setattr(compressor, key, value)
return compressor
evals/compaction/report.py
+43
View File
@@ -0,0 +1,43 @@
"""Render a compaction-eval scorecard as a terminal table + markdown.
Usage: python evals/compaction/report.py <results_dir>
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
def main():
out_dir = Path(sys.argv[1])
card = json.loads((out_dir / "scorecard.json").read_text(encoding="utf-8"))
card.sort(key=lambda s: -s["recall_pct"])
rows = []
for s in card:
before = s.get("before_tokens", 0)
after = s.get("after_tokens", 0)
kept = f"{100 * after / before:.1f}%" if before else "?"
rows.append((
s["policy"], f"{s['recall_pct']}%", f"{before:,}", f"{after:,}", kept,
str(s.get("compress_seconds", "-")),
))
headers = ("policy", "recall", "tokens before", "tokens after", "kept", "sec")
widths = [max(len(headers[i]), *(len(r[i]) for r in rows)) for i in range(len(headers))]
line = " ".join(h.ljust(widths[i]) for i, h in enumerate(headers))
print(line)
print("-" * len(line))
for r in rows:
print(" ".join(str(r[i]).ljust(widths[i]) for i in range(len(headers))))
md = ["| " + " | ".join(headers) + " |", "|" + "|".join("---" for _ in headers) + "|"]
for r in rows:
md.append("| " + " | ".join(r) + " |")
(out_dir / "scorecard.md").write_text("\n".join(md) + "\n", encoding="utf-8")
print(f"\nmarkdown -> {out_dir}/scorecard.md")
if __name__ == "__main__":
main()
evals/compaction/results/SCORECARD-2026-08-15.md
+357
View File
@@ -0,0 +1,357 @@
# Compaction v2 — 4-transcript scorecard (2026-08-15, anchor-index build)
Four real 500K-token lineage transcripts from state.db (sweep campaign, GUI
desktop work, PR-merge campaign, ACP/PR review), 15-question recall exam
each. "recovery" = one session_search round-trip (FTS5+BM25 sim) against the
archived region. Lean build includes: 25K clamped tail, tail tool demotion,
chunked digests (noise-filtered, pristine tool contents), mechanical anchor
index, verbatim user messages, recovery footer, upgraded summarizer prompt.
> **Historical note (2026-08-30):** the "chunked digests" arm described here
> was later replaced — the detailed session log is now produced by the SAME
> single summary request (lean compaction makes exactly one auxiliary LLM
> call per attempt; no per-chunk digest calls). See #96603.
## Results (recall % @ retained tokens)
policy sweep gui prmerge acp AVG
uncompacted 93.3 @ 500K 96.7 @ 500K 96.7 @ 500K 100.0 @ 500K 96.7
current 93.3*@ 176K 26.7*@ 156K 33.3 @ 155K 30.0 @ 160K 45.8 @ 162K
lean 40.0 @ 62K 60.0 @ 41K 23.3 @ 44K 36.7 @ 50K 40.0 @ 49K
lean+recovery 70.0 @ 62K 80.0 @ 41K 43.3 @ 45K 80.0 @ 50K 68.3 @ 49K
* sweep/gui current scores are from the previous question banks (same
transcripts; banks regenerated in the 4-way run). prmerge/acp are clean
same-bank comparisons across all arms.
## Findings
1. LEAN+RECOVERY BEATS CURRENT BY +22.5pts ON AVERAGE (68.3 vs 45.8) AT 3.3x
FEWER TOKENS (49K vs 162K). It wins on 3 of 4 transcripts and loses only
sweep — the one transcript where current's fat tail got lucky with
restated facts (93.3 is bank-inflated luck; see finding 3 of the previous
scorecard).
2. THE ANCHOR INDEX FIXED THE NEEDLE-FACT CLASS. GUI closed-book went
23.3 -> 60.0 and GUI+recovery 46.7 -> 80.0 after mechanically indexing
exact identifiers (SHAs, ids, paths, error strings) instead of trusting
the summarizer with them. ACP+recovery hit 80.0.
3. TWO FRESH TRANSCRIPTS CONFIRM CURRENT IS WEAK, NOT STRONG: 33.3 and 30.0
at ~157K retained. The original sweep 93.3 was restatement luck, not
policy quality. Current's average is 45.8% for 162K tokens — lean+recovery
is 22 points better for less than a third of the spend.
4. prmerge IS THE HARD CASE for everyone (96.7 ceiling, best policy 43.3):
1.1M-token lineage truncated at 500K, dense multi-PR state. Recovery
misses there are mostly query formulation. Headroom, not a blocker.
5. Goal check (Teknium): tail = max(10K, 2.5%) ✓; summaries scoped to the
compacted region only ✓ (sentinel tripwire test); session_search pointer ✓
(+20-43pts measured); better accuracy AND more savings than current ✓
(+22.5pts at 0.30x tokens).
## Codex CLI head-to-head (same transcripts, same exams, same judge)
Real OpenAI Codex CLI (v0.147.0, gpt-5.6-sol, 258K window) run end-to-end on
the identical four transcripts: chunk files read via `codex exec` until its
REAL auto-compaction fired (verified `compacted` event in the rollout jsonl;
peak context 455-483K), then quizzed post-compaction from memory with the
same 15-question banks and scored by the same judge.
policy sweep gui prmerge acp AVG retained state
codex (real, post-cmp) 26.7% 40.0% 43.3% 36.7% 36.7% ~4.5K (opaque blob + user msgs)
hermes current 93.3%* 26.7%* 33.3% 30.0% 45.8% ~162K
hermes lean closed-book 40.0% 60.0% 23.3% 36.7% 40.0% ~49K
hermes lean+recovery 70.0% 80.0% 43.3% 80.0% 68.3% ~49K
Notes:
- codex answers from its own post-compaction session — the honest analog of
our closed-book arms. It has NO session_search equivalent (its rollout is
on disk but the agent cannot search it at runtime), so recovery has no
codex counterpart; that gap is exactly the differentiator lean leans on.
- Apples-to-apples closed-book: lean 40.0% vs codex 36.7% — parity-plus at
10x codex's retained state but 0.30x current's. With recovery: +31.6pts
over codex.
- codex ties lean+recovery on prmerge (43.3%) — the dense multi-PR campaign
is the hardest transcript for every policy and the clearest iteration
target.
- Methodology caveats: codex ingested transcripts as FILE READS (tool
outputs), not native conversation — this matches how its compaction
treats tool output (drops it all into the server-side summary) but is not
byte-identical to a native session. Its model (gpt-5.6-sol) also differs
from the answering model in our arms; scores compare COMPACTION PIPELINES
end-to-end, not models in isolation. One codex quiz reply was also
capped short (~1K chars for 15 answers), which its terse post-compaction
style invites.
## Recommendation
Ship lean as opt-in (compression.tail_mode: lean, legacy default), harness as
the permanent gate. Iterate prmerge-class recall behind the flag (query
mining, per-epoch anchor windows) before default flip.
## Appendix: full per-transcript detail
### Transcript: sweep
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| lean | 40.0% | 499,625 | 61,567 | 114.9 |
| lean+recovery | 70.0% | 499,625 | 61,792 | 114.8 |
<details><summary>15 exam questions (questions-30b95351c7.json)</summary>
1. **What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?**
gold: `the stale file version silently deletes newer main code`
2. **According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?**
gold: `~60GB`
3. **Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?**
gold: `backgroundThrottling`
4. **What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?**
gold: `Auto merge is not allowed for this repository (enablePullRequestAutoMerge)`
5. **What is the specified 'Rule 0' that must be included in a subagent brief?**
gold: `load the skill first`
6. **In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?**
gold: `add config gate for title generation`
7. **Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?**
gold: `tests/test_packaging_metadata.py`
8. **What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?**
gold: `completed normally (exit code 0)`
9. **What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?**
gold: `relative exclude-newer makes the committed lock stale every day`
10. **How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?**
gold: `13`
11. **In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?**
gold: `-1`
12. **What is the full title block text for the technical schematic infographic generated for the Gateway Drain?**
gold: `GATEWAY DRAIN × CRON — SHUTDOWN CONTRACT`
13. **Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?**
gold: `82980`
14. **What is the specific Gist ID created for the PR infographic host in the cron cluster?**
gold: `ee33edd5804689243f974536ef7aecb9`
15. **What was the final merge SHA for Cluster D's Trigger-now PR #70638?**
gold: `f9d64b9a9d8b306f64851c1a13869d96ad5d7869`
</details>
<details><summary>15 exam questions (questions-5be475cde0.json)</summary>
1. **What exact command did the agent use to search for open issues related to a specific topic during Phase 1 of the cluster-sweep salvage?**
gold: `gh issue list --search "<topic>" --state open --limit 100 --json number,title`
2. **According to Teknium's design intent, what is the status of 'platform toolsets' in the codebase?**
gold: `platform toolsets are vestigial, never exposed`
3. **During the July sweep, which specific issue's config bridge was found to already exist at the exact line it was claimed to be missing?**
gold: `#32263`
4. **In the Aug 2026 cron-summarizer cluster sweep, which two PR numbers were discovered post-merge as the true first submitters?**
gold: `#60593, #61969`
5. **What is the recommended Git command to find when a specific symbol fix landed on the main branch?**
gold: `git log -S "<symbol>"`
6. **Why did the #39719 salvage silently delete 236 lines of code from cli-config.yaml.example?**
gold: `the stale file version silently deletes newer main code`
7. **What is the rule for salvaging commits with placeholder identities like 'pwn@example.com'?**
gold: `do NOT cherry-pick. Surgical reapply as maintainer-authored commit, Co-authored-by the GitHub PR author`
8. **How should an agent handle a 'gh pr merge' 502 error?**
gold: `retry the same command once after the "Merge already in progress" settles (~45s); check PR state between attempts`
9. **Which two properties shape almost every design decision in Hermes according to the Development Guide?**
gold: `Per-conversation prompt caching is sacred and The core is a narrow waist; capability lives at the edges.`
10. **What error message does the live-checkout git guard display when blocking a history-rewriting command?**
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module `
11. **What happened to the Desktop cluster's 'npm ci' command that resulted in an error writing to /tmp/ccH06T4r.s?**
gold: `No space left on device`
12. **What was the GraphQL API rate limit remaining for the user when the 'API rate limit already exceeded' error first occurred?**
gold: `0`
13. **Which PR was identified as the salvage of HexLab98's #85283 to fix hung inline API calls?**
gold: `#86645`
14. **Why did PR #79268 fix invisible overlays in the TUI?**
gold: `renderNodeToOutput skips boxes Yoga squeezes to height 0`
15. **What was the specific ModuleNotFoundError message caused by the wheel subpackage discovery trap in #34701?**
gold: `ModuleNotFoundError: No module named 'hermes_cli.dashboard_auth'`
</details>
### Transcript: gui
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| lean | 60.0% | 499,818 | 41,232 | 118.1 |
| lean+recovery | 80.0% | 499,818 | 41,306 | 115.2 |
<details><summary>15 exam questions (questions-36d3d87e0b.json)</summary>
1. **What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?**
gold: `#86617`
2. **According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?**
gold: `.env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.`
3. **What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?**
gold: `tests/hermes_cli/test_session_recovery_lost_and_found.py:327`
4. **What was the root cause of issue #73793 regarding mid-turn message rendering?**
gold: `redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row`
5. **Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?**
gold: `#84287`
6. **In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?**
gold: `pane sash grab band made asymmetric 1px/7px`
7. **Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?**
gold: `602028@ky-tech.com.cn`
8. **What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?**
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout`
9. **What is the core design principle regarding 'Narrow Waist' in Hermes development?**
gold: `The core is a narrow waist; capability lives at the edges.`
10. **What was the result of the rebase-merge attempt for PR #86589?**
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
11. **In the infographic style picker, what vibe is associated with the 'designers-republic' style?**
gold: `The Designers Republic: flat orange+violet vector schematic on pewter grey`
12. **Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?**
gold: `conflicts with main in 4 files and introduces a second competing display-dedupe scheme`
13. **What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?**
gold: `May 23 2026`
14. **What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?**
gold: `Property 'onToggleUnread' is missing in type`
15. **According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?**
gold: `Electron`
</details>
<details><summary>15 exam questions (questions-9c55c707b6.json)</summary>
1. **What two PR numbers are associated with the 'sidebar-nav-rows-and-overlay-panels.md' and 'hud-mode-internals.md' references in the initial tool content?**
gold: `#85162 and #82285`
2. **According to AGENTS.md, what is the 'one exception' to the rule that nothing should rebuild the system prompt mid-conversation?**
gold: `context compression`
3. **In the Contribution Rubric, what are the three allowed reasons for an automated triage sweeper to close a PR?**
gold: `implemented_on_main, cannot_reproduce, incoherent`
4. **Which contributor is credited with adding the 'Brazilian Portuguese localization' in PR #86292?**
gold: `@gui8515`
5. **What specific error message is reported in issue #83562 regarding the Windows Desktop update?**
gold: `Hermes backend exited (0)`
6. **What is the 'core problem' identified in the parallel-subagent-salvage-orchestration.md reference?**
gold: `subagents share the parent's worktree + main checkout`
7. **Why was the 'nix (macos-latest)' build failing in the salvage batches according to the orchestration reference?**
gold: `Nix build failed due to stale npm lockfile hash`
8. **Which subagent ID was assigned the goal of salvaging the 'inflight-journal duplicate-answer cluster'?**
gold: `sa-2-7318d0ba`
9. **In PR #86595, why was PR #80707 by upperagent excluded from the salvage?**
gold: `violating this PR's UI-read-only invariant`
10. **What was the root cause of the failure in Python tests slice 4/12 for PR #86597?**
gold: `AssertionError: assert 't2' == 't1'`
11. **What did the fix for issue #79157 in PR #86589 involve?**
gold: `pane sash grab band made asymmetric 1px/7px`
12. **According to the root cause analysis for #73793, which two files spliced the mid-turn user message at streamIndex?**
gold: `use-prompt-actions/index.ts and session-tile-actions.ts`
13. **What was the head SHA for the 'salvage/desktop-busy-state' branch in PR #86604?**
gold: `bddadfe9e21e24b3d52e2b15f138c42474dede42`
14. **Why was the merge of PR #86589 aborted during the 'Merge all' command?**
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
15. **What specific file was modified to fix the 'artifacts page timestamps render 1970' issue via PR #86749?**
gold: `apps/desktop/src/app/session/hooks/use-session-actions/utils.ts`
</details>
### Transcript: prmerge
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| uncompacted_control | 96.7% | 499,663 | 499,663 | — |
| current | 33.3% | 499,663 | 155,399 | 14.9 |
| lean | 23.3% | 499,663 | 44,419 | 105.4 |
| lean+recovery | 43.3% | 499,663 | 44,977 | 95.8 |
<details><summary>15 exam questions (questions-703ae2774a.json)</summary>
1. **Which PR number added the public subagent lifecycle API?**
gold: `#63359`
2. **What is the name of the typed service added to PluginContext for launching and monitoring child sessions?**
gold: `subagent_lifecycle`
3. **How many contract and security tests were included with the subagent lifecycle API PR?**
gold: `42`
4. **What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?**
gold: `cannot currently trigger a turn in an existing gateway session`
5. **Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?**
gold: `#64436`
6. **What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?**
gold: `must compile, must start with ≥2 literal characters`
7. **Which contributor authorized sustained help for the Phase 0–1 expansion track?**
gold: `Daniel`
8. **What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?**
gold: `#64204`
9. **What configuration setting is required to opt-in to reasoning deltas in streaming output?**
gold: `plugins.stream_reasoning_deltas: true`
10. **How many additions and across how many files were made in PR #63359?**
gold: `650 additions across 4 files`
11. **What is the name of the reference plugin shipped with the redaction pattern registry?**
gold: `nvapi-redaction`
12. **List the four observer-only streaming output plugin hooks added in PR #64317.**
gold: `on_stream_start, on_stream_delta, on_stream_end, on_interim_message`
13. **What was addressed in the update to PR #58541 regarding lifecycle hooks?**
gold: `created-hook timing and added kanban_task_promoted`
14. **Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?**
gold: `#64230`
15. **What was the Round 3 review's outcome for PR #63359 and @asimons81?**
gold: `sub-issue #65447`
</details>
### Transcript: acp
| policy | recall | tokens before | tokens after | compress s |
|---|---|---|---|---|
| uncompacted_control | 100.0% | 498,906 | 498,906 | — |
| current | 30.0% | 498,906 | 160,223 | 15.8 |
| lean | 36.7% | 498,906 | 49,523 | 143.3 |
| lean+recovery | 80.0% | 498,906 | 49,721 | 135.6 |
<details><summary>15 exam questions (questions-f45358df19.json)</summary>
1. **What was the specific reason Teknium gave for reverting PR #30179 in July 2026?**
gold: `WTF??? REVERT! DAMMIT`
2. **On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?**
gold: `PR #6391`
3. **Which file path should be checked for the canonical list of provider models?**
gold: `hermes_cli/models.py`
4. **What was the identified bug in PR #2314 regarding provider names?**
gold: `checking for "alibaba-coding-plan"`
5. **What is the mandatory line limit for PR reviews requested by Teknium?**
gold: `<= 15 lines`
6. **What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?**
gold: `Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix mod`
7. **Why was PR #74658 necessary to fix Slack 'broken on main'?**
gold: `SlackResponse isn't a dict subclass, so every gate is always False.`
8. **What was the final merge commit SHA for the Slack SDK response fix on main?**
gold: `24ba86627515ad5fda69a39ef338c365713448bc`
9. **In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?**
gold: `Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1`
10. **What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?**
gold: `UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'`
11. **Which test failed in CI slice 5/12 for the kanban delivery modes PR?**
gold: `tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post`
12. **According to the transcript, why is squash merging banned as of July 2026?**
gold: `DevOps policy`
13. **Which contributor authored the first fix for issue #73030 in July?**
gold: `@Tranquil-Flow`
14. **What was the 'Superman-style' shield error in the first generation of the Kanban infographic?**
gold: `red "S" inside the diamond shield`
15. **What specific file was modified to add the 'scope_id_for_chat' method for Slack?**
gold: `plugins/platforms/slack/adapter.py`
</details>
## Methodology notes
- Transcripts: 4 real session lineages reconstructed from a state.db copy
(sweep campaign 42 rotations / GUI desktop 34 / PR-merge 17 / ACP review
17), chronological 500K-token prefix, tool-group aligned.
- Question generation: main model, from the region the CURRENT policy would
summarize (most conservative boundary), cached per transcript so every
policy answers the identical exam.
- Answering: fresh LLM sees ONLY the post-compaction context (closed-book) or
context + one FTS5+BM25 search round-trip over the archived region
(+recovery). Judge sees gold; answerer never does. Scoring 2/1/0.
- Known caveats: 15 questions/transcript => +-1 question ~ 3.3pts noise;
sweep/gui current-policy rows predate a question-bank regeneration
(prmerge/acp are same-bank across all arms); the recovery sim conservatively
approximates production session_search (same engine, no windowing).
- Cost shape: lean compaction = ~25 aux-model digest calls (~2min, one-time
per compaction) vs 1 call today; every post-compaction turn is ~110K input
tokens cheaper. Break-even ~1 turn.
evals/compaction/results/codex-arm-2026-08-15/acp.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 36.7,
"scores": [
0,
0,
2,
1,
0,
0,
2,
0,
0,
2,
0,
0,
0,
2,
2
],
"read_seconds": 229,
"compaction_events": 1,
"peak_context_tokens": 454931,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-35-51-01a00835-fef4-7c31-944d-9d0d8ba933be.jsonl"
},
"results": [
{
"q": "What was the specific reason Teknium gave for reverting PR #30179 in July 2026?",
"gold": "WTF??? REVERT! DAMMIT",
"answer": "UNKNOWN \u2014 best guess: it caused a regression on current `main` and had to be reverted until a safer implementation was available.",
"score": 0,
"why": "The answer provides a logical guess about a regression, but it fails to capture the specific, idiosyncratic reason/quote provided in the gold answer ('WTF??? REVERT! DAMMIT')."
},
{
"q": "On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?",
"gold": "PR #6391",
"answer": "UNKNOWN \u2014 best guess: PR #37865.",
"score": 0,
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold answer (PR #37865 vs PR #6391)."
},
{
"q": "Which file path should be checked for the canonical list of provider models?",
"gold": "hermes_cli/models.py",
"answer": "UNKNOWN \u2014 best guess: `hermes_cli/models.py`.",
"score": 2,
"why": "The answer provides the correct file path as its best guess, which matches the gold answer exactly."
},
{
"q": "What was the identified bug in PR #2314 regarding provider names?",
"gold": "checking for \"alibaba-coding-plan\"",
"answer": "UNKNOWN \u2014 best guess: it used inconsistent/noncanonical provider names, so provider matching and routing failed.",
"score": 1,
"why": "The answer identifies the core issue (provider name mismatch/inconsistency) which led to the specific bug, but it does not mention the specific string 'alibaba-coding-plan' identified in the gold answer."
},
{
"q": "What is the mandatory line limit for PR reviews requested by Teknium?",
"gold": "<= 15 lines",
"answer": "Maximum 5 lines.",
"score": 0,
"why": "The answer states a limit of 5 lines, which contradicts the gold answer of 15 lines."
},
{
"q": "What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?",
"gold": "Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module versions in this running process.",
"answer": "UNKNOWN \u2014 best guess: `fatal: '<branch>' is already checked out at '<worktree path>'`.",
"score": 0,
"why": "The answer provides a standard Git error message as a guess, which does not match the specific custom error message provided in the gold answer."
},
{
"q": "Why was PR #74658 necessary to fix Slack 'broken on main'?",
"gold": "SlackResponse isn't a dict subclass, so every gate is always False.",
"answer": "Real Slack SDK calls return `SlackResponse` objects, but the main code treated them like dictionaries and called dictionary-style accessors.",
"score": 2,
"why": "The answer correctly identifies that SlackResponse objects were being treated as dictionaries when they are not, which matches the core issue described in the gold answer that their lack of dictionary subclassing caused attribute/gate checks to fail."
},
{
"q": "What was the final merge commit SHA for the Slack SDK response fix on main?",
"gold": "24ba86627515ad5fda69a39ef338c365713448bc",
"answer": "UNKNOWN.",
"score": 0,
"why": "The gold answer provides a specific commit SHA, while the answer states 'UNKNOWN', failing to provide the requested factual information."
},
{
"q": "In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?",
"gold": "Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1",
"answer": "UNKNOWN.",
"score": 0,
"why": "The answer 'UNKNOWN' is a refusal to provide the information required by the gold answer."
},
{
"q": "What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?",
"gold": "UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'",
"answer": "`UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'` \u2014 applied only during first migration/addition of the column so existing choices were not overwritten.",
"score": 2,
"why": "The answer provides the exact SQL statement specified in the gold answer and correctly contextualizes its purpose."
},
{
"q": "Which test failed in CI slice 5/12 for the kanban delivery modes PR?",
"gold": "tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post",
"answer": "UNKNOWN \u2014 best guess: a relay-metrics concurrency test, unrelated to the delivery-mode change.",
"score": 0,
"why": "The answer identifies an unrelated test (relay-metrics) rather than the specific kanban notifier test mentioned in the gold answer."
},
{
"q": "According to the transcript, why is squash merging banned as of July 2026?",
"gold": "DevOps policy",
"answer": "Squash merging destroys the contributor-authored commit history and therefore loses reliable authorship/credit.",
"score": 0,
"why": "The answer explains a general rationale for why someone might dislike squash merging, but it fails to provide the specific reason cited in the text ('DevOps policy')."
},
{
"q": "Which contributor authored the first fix for issue #73030 in July?",
"gold": "@Tranquil-Flow",
"answer": "UNKNOWN.",
"score": 0,
"why": "The answer states 'UNKNOWN' while the gold answer provides a specific contributor name (@Tranquil-Flow)."
},
{
"q": "What was the 'Superman-style' shield error in the first generation of the Kanban infographic?",
"gold": "red \"S\" inside the diamond shield",
"answer": "The generated shield contained an emblem resembling Superman\u2019s \u201cS,\u201d rather than a neutral Kanban/routing symbol.",
"score": 2,
"why": "The answer accurately identifies that the shield contained a symbol resembling Superman's 'S', which matches the core fact of the gold answer."
},
{
"q": "What specific file was modified to add the 'scope_id_for_chat' method for Slack?",
"gold": "plugins/platforms/slack/adapter.py",
"answer": "`plugins/platforms/slack/adapter.py`.",
"score": 2,
"why": "The answer matches the gold answer exactly, including the full file path."
}
]
}
evals/compaction/results/codex-arm-2026-08-15/gui.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 40.0,
"scores": [
2,
2,
0,
2,
0,
0,
0,
2,
2,
2,
0,
0,
0,
0,
0
],
"read_seconds": 207,
"compaction_events": 1,
"peak_context_tokens": 482525,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-28-36-01a0082f-5b69-7ff0-977d-998523a4b3fc.jsonl"
},
"results": [
{
"q": "What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?",
"gold": "#86617",
"answer": "PR **#86617**.",
"score": 2,
"why": "The answer provides the exact PR number specified in the gold answer."
},
{
"q": "According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?",
"gold": ".env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.",
"answer": "`.env` is for secrets and deployment-specific values; `config.yaml` is for non-secret user-facing runtime configuration.",
"score": 2,
"why": "The answer correctly identifies that .env is for secrets and config.yaml is for non-secret/behavioral settings, which matches the core distinction in the gold answer."
},
{
"q": "What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?",
"gold": "tests/hermes_cli/test_session_recovery_lost_and_found.py:327",
"answer": "**UNKNOWN** \u2014 best guess: a schema-width assertion in a Python schema/config test, expecting 55 fields but receiving 56 after `git_metadata_generation` was added.",
"score": 0,
"why": "The answer fails to provide the specific file and line number requested, which are present in the gold answer. It instead provides a guess about the context of the error without identifying the location."
},
{
"q": "What was the root cause of issue #73793 regarding mid-turn message rendering?",
"gold": "redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row",
"answer": "Two live paths handled mid-turn corrections independently, allowing corrections and assistant stream output to be inserted in the wrong causal order.",
"score": 2,
"why": "The answer identifies the same root cause as the gold answer: the independent handling of paths caused the user bubble (correction) and assistant stream to be rendered in the wrong causal order."
},
{
"q": "Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?",
"gold": "#84287",
"answer": "**PR #86299**.",
"score": 0,
"why": "The answer identifies PR #86299, but the gold answer is PR #84287. The two are different pull requests."
},
{
"q": "In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?",
"gold": "pane sash grab band made asymmetric 1px/7px",
"answer": "**UNKNOWN** \u2014 best guess: make the scrollbar gutter/overlay accept pointer events instead of letting the virtualized content layer cover it.",
"score": 0,
"why": "The answer provides a guess that contradicts the gold answer; the fix was making the pane sash grab band asymmetric (1px/7px), not changing pointer events on the gutter/overlay."
},
{
"q": "Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?",
"gold": "602028@ky-tech.com.cn",
"answer": "**UNKNOWN** \u2014 best guess: the email associated with contributor `hkfiberlaser-svg`.",
"score": 0,
"why": "The answer states the answer is unknown and provides a guess that does not match the gold answer."
},
{
"q": "What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?",
"gold": "Blocked: `git <op>` would rewrite Hermes's live source checkout",
"answer": "**UNKNOWN** \u2014 best guess: an error stating that the git command was blocked because it would rewrite the live source checkout.",
"score": 2,
"why": "The answer correctly identifies the core substance of the error message (blocked git command because it would rewrite the live source checkout) which matches the gold answer."
},
{
"q": "What is the core design principle regarding 'Narrow Waist' in Hermes development?",
"gold": "The core is a narrow waist; capability lives at the edges.",
"answer": "Keep a small, stable core interface\u2014the \u201cnarrow waist\u201d\u2014through which different clients, providers, and integrations communicate, avoiding duplicated business logic across surfaces.",
"score": 2,
"why": "The answer correctly identifies the core principle of a 'narrow waist' and explains that the logic/interfaces are centralized there while clients and providers interact at the edges, matching the substance of the gold answer."
},
{
"q": "What was the result of the rebase-merge attempt for PR #86589?",
"gold": "GraphQL: Pull Request has merge conflicts (mergePullRequest)",
"answer": "**UNKNOWN** \u2014 best guess: the rebase merge could not proceed because the PR had conflicts and was not mergeable.",
"score": 2,
"why": "The answer correctly identifies that the rebase-merge could not proceed due to merge conflicts, which matches the factual content of the gold answer."
},
{
"q": "In the infographic style picker, what vibe is associated with the 'designers-republic' style?",
"gold": "The Designers Republic: flat orange+violet vector schematic on pewter grey",
"answer": "A bold, experimental, anti-corporate graphic-design vibe\u2014brutalist/techno typography inspired by The Designers Republic.",
"score": 0,
"why": "The answer describes the real-world aesthetic of 'The Designers Republic' studio, but fails to identify the specific visual elements (flat orange+violet vector schematic on pewter grey) defined for this style within the infographic picker context."
},
{
"q": "Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?",
"gold": "conflicts with main in 4 files and introduces a second competing display-dedupe scheme",
"answer": "It did not address transcript visibility after compaction/compression; its changes concerned a different part of the compaction flow.",
"score": 0,
"why": "The answer provides a completely different reason than the gold answer, which specifies merge conflicts and a competing deduplication scheme as the reasons for exclusion."
},
{
"q": "What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?",
"gold": "May 23 2026",
"answer": "**UNKNOWN** \u2014 best guess: **2026-08-12**.",
"score": 0,
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold date (2026-08-12 vs. May 23 2026)."
},
{
"q": "What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?",
"gold": "Property 'onToggleUnread' is missing in type",
"answer": "**UNKNOWN** \u2014 best guess: a TypeScript unresolved-name/import error introduced by the rebase.",
"score": 0,
"why": "The answer identifies as UNKNOWN and provides an incorrect guess. The gold answer specifies a 'property is missing' error, while the provided answer guesses an 'unresolved-name/import' error."
},
{
"q": "According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?",
"gold": "Electron",
"answer": "The **Rust/Tauri backend** is authoritative for process lifecycle and the native filesystem.",
"score": 0,
"why": "The answer identifies the Rust/Tauri backend as the authority, which directly contradicts the gold answer stating Electron is the authority."
}
]
}
evals/compaction/results/codex-arm-2026-08-15/prmerge.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 43.3,
"scores": [
2,
2,
0,
2,
2,
0,
0,
2,
0,
0,
0,
2,
1,
0,
0
],
"read_seconds": 154,
"compaction_events": 1,
"peak_context_tokens": 465131,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-32-45-01a00833-2968-7453-8646-d5bb7f5e8400.jsonl"
},
"results": [
{
"q": "Which PR number added the public subagent lifecycle API?",
"gold": "#63359",
"answer": "PR **#63359**.",
"score": 2,
"why": "The answer matches the gold PR number exactly."
},
{
"q": "What is the name of the typed service added to PluginContext for launching and monitoring child sessions?",
"gold": "subagent_lifecycle",
"answer": "`SubagentService` (`ctx.subagents`).",
"score": 2,
"why": "The answer identifies the correct property on the context (ctx.subagents), which is the typed service used for the task, even though it uses a slightly different name for the service itself."
},
{
"q": "How many contract and security tests were included with the subagent lifecycle API PR?",
"gold": "42",
"answer": "**UNKNOWN** \u2014 best guess: **24 tests**.",
"score": 0,
"why": "The answer provides an incorrect guess (24) compared to the gold answer (42) and frames the response as unknown."
},
{
"q": "What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?",
"gold": "cannot currently trigger a turn in an existing gateway session",
"answer": "`ctx.inject_message()` could not safely target an active gateway session; it lacked a profile/session-scoped `session_key`.",
"score": 2,
"why": "The answer identifies the core limitation (targeting active gateway sessions) and provides technical detail on why a turn cannot be triggered (lack of session-scoped key)."
},
{
"q": "Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?",
"gold": "#64436",
"answer": "PR **#64436**.",
"score": 2,
"why": "The answer matches the gold answer exactly."
},
{
"q": "What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?",
"gold": "must compile, must start with \u22652 literal characters",
"answer": "Patterns must be **bounded in length** and **validated against unsafe/catastrophic regex behavior** before registration.",
"score": 0,
"why": "The answer discusses general regex safety (length and performance), whereas the gold answer specifies two precise technical constraints: compilation and a minimum of two literal characters at the start."
},
{
"q": "Which contributor authorized sustained help for the Phase 0\u20131 expansion track?",
"gold": "Daniel",
"answer": "**@asimons81**.",
"score": 0,
"why": "The answer identifies @asimons81 as the contributor, whereas the gold answer states it was Daniel. These do not match."
},
{
"q": "What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?",
"gold": "#64204",
"answer": "Issue **#64204**.",
"score": 2,
"why": "The answer matches the gold issue number exactly."
},
{
"q": "What configuration setting is required to opt-in to reasoning deltas in streaming output?",
"gold": "plugins.stream_reasoning_deltas: true",
"answer": "**UNKNOWN** \u2014 best guess: `display.show_reasoning: true`.",
"score": 0,
"why": "The answer identifies itself as a guess and provides the incorrect configuration setting ('display.show_reasoning' vs the gold 'plugins.stream_reasoning_deltas')."
},
{
"q": "How many additions and across how many files were made in PR #63359?",
"gold": "650 additions across 4 files",
"answer": "**UNKNOWN** \u2014 best guess: roughly **1,000 additions across 12 files**.",
"score": 0,
"why": "The answer states the information is unknown and provides a guess that is factually incorrect regarding both the number of additions and the number of files."
},
{
"q": "What is the name of the reference plugin shipped with the redaction pattern registry?",
"gold": "nvapi-redaction",
"answer": "**UNKNOWN** \u2014 best guess: `redaction-patterns`.",
"score": 0,
"why": "The answer identifies itself as unknown and provides a guess ('redaction-patterns') that is incorrect compared to the gold answer ('nvapi-redaction')."
},
{
"q": "List the four observer-only streaming output plugin hooks added in PR #64317.",
"gold": "on_stream_start, on_stream_delta, on_stream_end, on_interim_message",
"answer": "`on_stream_start`, `on_stream_delta`, `on_stream_end`, and `on_interim_message`.",
"score": 2,
"why": "The answer lists all four hooks correctly as specified in the gold answer."
},
{
"q": "What was addressed in the update to PR #58541 regarding lifecycle hooks?",
"gold": "created-hook timing and added kanban_task_promoted",
"answer": "It updated the lifecycle hooks to match the current registry contract, including correct hook timing, payloads, and failure isolation.",
"score": 1,
"why": "The answer correctly identifies 'hook timing' as part of the update, but it fails to mention the specific addition of 'kanban_task_promoted' and instead provides a general description of other improvements like payloads and isolation not mentioned in the gold answer."
},
{
"q": "Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?",
"gold": "#64230",
"answer": "**UNKNOWN** \u2014 best guess: **#64179**.",
"score": 0,
"why": "The answer provides an incorrect issue number (#64179) while the gold answer is #64230."
},
{
"q": "What was the Round 3 review's outcome for PR #63359 and @asimons81?",
"gold": "sub-issue #65447",
"answer": "Round 3 **approved PR #63359 and authorized @asimons81 to continue sustained Phase 0\u20131 work**.",
"score": 0,
"why": "The answer provides a general description of approval and authorization, but fails to mention the specific outcome required by the gold answer (the creation or reference of sub-issue #65447)."
}
]
}
evals/compaction/results/codex-arm-2026-08-15/sweep.json
+134
View File
@@ -0,0 +1,134 @@
{
"summary": {
"policy": "codex_real",
"recall_pct": 26.7,
"scores": [
2,
0,
2,
2,
0,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0
],
"read_seconds": 420,
"compaction_events": 1,
"peak_context_tokens": 482248,
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-20-54-01a00828-4e75-70f0-9e6d-31c7aac8d71c.jsonl"
},
"results": [
{
"q": "What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?",
"gold": "the stale file version silently deletes newer main code",
"answer": "It can silently replace current files with stale branch versions, discarding newer fixes and resurrecting deleted code; salvage must use commits and resolve conflicts explicitly.",
"score": 2,
"why": "The answer accurately reflects the gold answer's point that stale branch versions will overwrite newer code, while adding specific detail about resurrecting deleted code."
},
{
"q": "According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?",
"gold": "~60GB",
"answer": "UNKNOWN \u2014 best guess: approximately 4 GB RSS.",
"score": 0,
"why": "The answer provides a 'best guess' of 4 GB, which is factually incorrect and contradicts the gold answer of 60 GB."
},
{
"q": "Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?",
"gold": "backgroundThrottling",
"answer": "UNKNOWN \u2014 best guess: Electron\u2019s `backgroundThrottling: false`.",
"score": 2,
"why": "The answer correctly identifies the specific setting 'backgroundThrottling' as the suspected cause, despite the initial 'UNKNOWN' disclaimer."
},
{
"q": "What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?",
"gold": "Auto merge is not allowed for this repository (enablePullRequestAutoMerge)",
"answer": "`GraphQL: Pull request Auto merge is not allowed for this repository`",
"score": 2,
"why": "The answer matches the core message of the gold answer. The 'GraphQL:' prefix accurately reflects the technical origin of the error message when using the GitHub CLI."
},
{
"q": "What is the specified 'Rule 0' that must be included in a subagent brief?",
"gold": "load the skill first",
"answer": "Rule 0: do not merge anything; only the primary agent/user is authorized to merge.",
"score": 0,
"why": "The answer identifies a different 'Rule 0' regarding merging permissions, which does not match the gold answer's requirement to 'load the skill first'."
},
{
"q": "In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?",
"gold": "add config gate for title generation",
"answer": "UNKNOWN \u2014 best guess: \u201cfix: prevent duplicate message submission.\u201d",
"score": 0,
"why": "The answer provides an incorrect guess ('fix: prevent duplicate message submission.') which does not match the gold answer ('add config gate for title generation')."
},
{
"q": "Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?",
"gold": "tests/test_packaging_metadata.py",
"answer": "UNKNOWN \u2014 best guess: `tests/test_plugin_packaging.py`.",
"score": 0,
"why": "The answer identifies the wrong file (test_plugin_packaging.py vs the correct test_packaging_metadata.py)."
},
{
"q": "What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?",
"gold": "completed normally (exit code 0)",
"answer": "`npm ci` completed successfully with exit code 0.",
"score": 2,
"why": "The answer matches the gold answer perfectly, confirming the successful completion and the exit code of 0."
},
{
"q": "What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?",
"gold": "relative exclude-newer makes the committed lock stale every day",
"answer": "Root Cause A: the `all` extra referenced platform-specific/nonexistent dependency groups that were absent or inconsistent in the locked resolution.",
"score": 0,
"why": "The answer identifies an issue with dependency groups, whereas the gold answer states the root cause was the use of a relative 'exclude-newer' value that caused the lockfile to become stale daily."
},
{
"q": "How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?",
"gold": "13",
"answer": "UNKNOWN \u2014 best guess: 7 tasks done.",
"score": 0,
"why": "The answer guesses 7 tasks, while the gold answer is 13. The answer is factually incorrect."
},
{
"q": "In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?",
"gold": "-1",
"answer": "Exit code 2.",
"score": 0,
"why": "The answer provides a specific exit code (2) that contradicts the gold answer (-1)."
},
{
"q": "What is the full title block text for the technical schematic infographic generated for the Gateway Drain?",
"gold": "GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT",
"answer": "UNKNOWN \u2014 best guess: `GATEWAY DRAIN` / `INTERRUPT \u2192 DELIVER \u2192 DISCONNECT` / `GRACEFUL SHUTDOWN WITHOUT MESSAGE LOSS`.",
"score": 0,
"why": "The answer fails to provide the correct title block text specified in the gold answer. It incorrectly guesses metadata or slogans instead of the specific title 'GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT'."
},
{
"q": "Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?",
"gold": "82980",
"answer": "UNKNOWN \u2014 best guess: PR #86594.",
"score": 0,
"why": "The answer identifies the wrong PR number (86594 instead of 82980)."
},
{
"q": "What is the specific Gist ID created for the PR infographic host in the cron cluster?",
"gold": "ee33edd5804689243f974536ef7aecb9",
"answer": "UNKNOWN \u2014 I do not retain the Gist ID.",
"score": 0,
"why": "The answer states it does not know the information, whereas the gold answer provides a specific Gist ID."
},
{
"q": "What was the final merge SHA for Cluster D's Trigger-now PR #70638?",
"gold": "f9d64b9a9d8b306f64851c1a13869d96ad5d7869",
"answer": "UNKNOWN \u2014 I do not retain the final merge SHA for PR #70638.",
"score": 0,
"why": "The answer claims it does not know the information, while the gold answer provides the specific SHA requested."
}
]
}
evals/compaction/runner.py
+356
View File
@@ -0,0 +1,356 @@
"""Compaction eval runner.
Pipeline per transcript:
1. Load + cap the transcript.
2. Generate (or load cached) recall questions from the region that will be
summarized away under the CURRENT policy (the most conservative boundary:
anything the current policy summarizes is fair game for every policy).
3. For each policy: compress, then answer each question with ONLY the
compressed context, using a single LLM call per question.
4. Judge answers against gold with an LLM judge (sees gold; answerer
does not).
5. Write per-policy results JSON for report.py.
Run from repo root with the project venv (needs a configured provider).
"""
from __future__ import annotations
import argparse
import copy
import hashlib
import json
import re
import sys
import time
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
from evals.compaction.fixtures import ( # noqa: E402
estimate_tokens,
load_transcript,
total_tokens,
)
from evals.compaction.policies import EVAL_MODEL, POLICIES, apply_policy # noqa: E402
QUESTION_PROMPT = """You are building a factual recall exam from an AI-agent work session transcript.
Write {n} questions that test SPECIFIC, VERIFIABLE facts from the transcript below: identifiers (PR numbers, file paths, error messages, commit subjects), decisions and their reasons, user instructions, and outcomes. Rules:
- Every answer must appear literally in the transcript.
- No questions about the system prompt or generic behavior.
- Spread questions across the WHOLE span (early, middle, late).
- Prefer facts that matter for continuing the work (what was decided, what failed, what the user asked for).
Return STRICT JSON: a list of {{"q": "...", "gold": "...", "where": "<short quote locating the answer>"}}.
TRANSCRIPT:
{transcript}
"""
ANSWER_PROMPT = """You are an AI agent resuming a work session. Below is your CURRENT conversation context (it may include a compaction summary of earlier work). Answer the question using ONLY this context. If the context does not contain the answer, say exactly "NOT IN CONTEXT" and give your best guess after a semicolon.
CONTEXT:
{context}
QUESTION: {question}
Answer in one or two sentences."""
JUDGE_PROMPT = """Score this answer against the gold answer. Reply with STRICT JSON: {{"score": 2|1|0, "why": "..."}}.
2 = factually matches gold (wording may differ)
1 = partially correct or hedged-but-right ("NOT IN CONTEXT; guess X" where X is right scores 1)
0 = wrong, or "NOT IN CONTEXT" with a wrong/no guess
QUESTION: {question}
GOLD: {gold}
ANSWER: {answer}"""
SEARCH_QUERY_PROMPT = """You are an AI agent resuming a work session. Your context (below) includes a compaction summary noting that the full pre-compaction history is recoverable via session_search. You need to answer a question and the answer may not be in your current context.
Write the best search query (3-8 keywords, no boolean syntax) to find the answer in the archived session history. Reply with ONLY the query string.
CONTEXT (may be relevant):
{context_hint}
QUESTION: {question}"""
ANSWER_WITH_RECOVERY_PROMPT = """You are an AI agent resuming a work session. Below is your CURRENT conversation context (including a compaction summary), plus the results of a session_search you just ran against the archived pre-compaction history. Answer the question using both. If neither contains the answer, say exactly "NOT IN CONTEXT" and give your best guess after a semicolon.
CONTEXT:
{context}
SESSION_SEARCH RESULTS:
{search_results}
QUESTION: {question}
Answer in one or two sentences."""
def keyword_search(archive: list, query: str, top_k: int = 4, excerpt_chars: int = 2500) -> str:
"""Simulate session_search over the archived (compacted-away) region.
Uses an in-memory SQLite FTS5 index with BM25 ranking — the same engine
production session_search runs on — so the sim's retrieval quality
matches what a live agent gets. Falls back to term-frequency scoring if
FTS5 is unavailable.
"""
import sqlite3 as _sq
terms = [t.lower() for t in re.findall(r"[A-Za-z0-9_#./-]{3,}", query)]
if not terms:
return "(no results)"
rows = [
(i, m.get("role") or "", m["content"])
for i, m in enumerate(archive)
if isinstance(m.get("content"), str) and len(m["content"]) >= 20
]
hits = []
try:
db = _sq.connect(":memory:")
db.execute("CREATE VIRTUAL TABLE arch USING fts5(content, role UNINDEXED, idx UNINDEXED)")
db.executemany(
"INSERT INTO arch (content, role, idx) VALUES (?, ?, ?)",
[(c, r, i) for i, r, c in rows],
)
fts_query = " OR ".join(
'"' + t.replace('"', "") + '"' for t in terms
)
cur = db.execute(
"SELECT idx, role, content, bm25(arch) AS rank, "
"snippet(arch, 0, '', '', ' … ', 40) AS snip "
"FROM arch WHERE arch MATCH ? ORDER BY rank LIMIT ?",
(fts_query, top_k),
)
for idx, role, content, rank, snip in cur.fetchall():
lc = content.lower()
first = min((lc.find(t) for t in terms if lc.find(t) >= 0), default=0)
start = max(0, first - excerpt_chars // 4)
hits.append(
f"--- result (message #{idx}, role={role}) ---\n"
f"[match: {snip[:200]}]\n"
+ content[start:start + excerpt_chars]
)
db.close()
except _sq.OperationalError:
# FTS5 unavailable — degrade to term-frequency scoring.
scored = []
for i, r, c in rows:
lc = c.lower()
score = sum(lc.count(t) for t in terms) / (1 + len(c) / 4000)
if score > 0:
scored.append((score, i, r, c))
scored.sort(key=lambda x: -x[0])
for score, i, r, c in scored[:top_k]:
lc = c.lower()
first = min((lc.find(t) for t in terms if lc.find(t) >= 0), default=0)
start = max(0, first - excerpt_chars // 4)
hits.append(
f"--- result (message #{i}, role={r}) ---\n"
+ c[start:start + excerpt_chars]
)
return "\n\n".join(hits) if hits else "(no results)"
def _call(prompt: str, max_tokens: int = 2000) -> str:
from agent.auxiliary_client import call_llm
resp = call_llm(
messages=[{"role": "user", "content": prompt}],
task="compression",
max_tokens=max_tokens,
)
if hasattr(resp, "choices"):
return resp.choices[0].message.content or ""
return str(resp)
def _extract_json(text: str):
m = re.search(r"```(?:json)?\s*(.*?)```", text, re.S)
if m:
text = m.group(1)
start = min([i for i in (text.find("["), text.find("{")) if i >= 0], default=0)
return json.loads(text[start:])
def serialize_for_exam(messages, char_cap: int = 600_000) -> str:
parts = []
for m in messages:
role = m.get("role")
c = m.get("content")
if not isinstance(c, str) or not c:
continue
if role == "system":
continue
parts.append(f"[{role}] {c}")
text = "\n\n".join(parts)
if len(text) > char_cap:
half = char_cap // 2
text = text[:half] + "\n\n...[middle elided for exam generation]...\n\n" + text[-half:]
return text
def summarized_region(compressor_module, messages):
"""The middle region the current policy would summarize: everything
between the protected head and the tail cut. Questions come from here."""
from agent.context_compressor import ContextCompressor
comp = ContextCompressor(model=EVAL_MODEL, quiet_mode=True)
head_end = comp.protect_first_n
tail_start = comp._find_tail_cut_by_tokens(messages, head_end)
return messages[head_end:tail_start]
def generate_questions(messages, n: int, cache_path: Path) -> list:
if cache_path.exists():
return json.loads(cache_path.read_text(encoding="utf-8"))
import agent.context_compressor as cc
region = summarized_region(cc, messages)
text = serialize_for_exam(region)
raw = _call(QUESTION_PROMPT.format(n=n, transcript=text), max_tokens=4000)
questions = _extract_json(raw)[:n]
cache_path.parent.mkdir(parents=True, exist_ok=True)
cache_path.write_text(json.dumps(questions, indent=1), encoding="utf-8")
return questions
def run_policy(name: str, spec: dict, messages, questions, out_dir: Path,
with_recovery: bool = False) -> dict:
from agent.context_compressor import ContextCompressor
before = copy.deepcopy(messages)
comp = apply_policy(ContextCompressor(model=EVAL_MODEL, quiet_mode=True), spec)
for key, value in (spec.get("ctor") or {}).items():
setattr(comp, key, value)
t0 = time.time()
compressed = comp.compress(copy.deepcopy(messages), current_tokens=total_tokens(messages), force=True)
elapsed = time.time() - t0
# The archived region = original messages that did not survive verbatim.
surviving = set()
for m in compressed:
c = m.get("content")
if isinstance(c, str) and c:
surviving.add(c[:200])
archive = [
m for m in before
if isinstance(m.get("content"), str) and (m.get("content") or "")[:200] not in surviving
]
context_text = serialize_for_exam(compressed, char_cap=700_000)
results = []
for qa in questions:
if with_recovery:
# The summary (session log, verbatim user msgs, recovery footer) sits
# near the FRONT of the serialized context; give the query writer
# that portion plus the recent tail so it can mine anchor
# identifiers (PR numbers, paths, error strings) for the query.
hint = context_text[:60_000] + "\n...\n" + context_text[-8_000:]
query = _call(
SEARCH_QUERY_PROMPT.format(
context_hint=hint, question=qa["q"],
),
max_tokens=100,
).strip().strip('"')
search_results = keyword_search(archive, query)
answer = _call(
ANSWER_WITH_RECOVERY_PROMPT.format(
context=context_text,
search_results=search_results,
question=qa["q"],
),
max_tokens=400,
)
else:
query = None
answer = _call(ANSWER_PROMPT.format(context=context_text, question=qa["q"]), max_tokens=400)
verdict_raw = _call(JUDGE_PROMPT.format(question=qa["q"], gold=qa["gold"], answer=answer), max_tokens=300)
try:
verdict = _extract_json(verdict_raw)
except Exception:
verdict = {"score": 0, "why": f"judge parse failure: {verdict_raw[:100]}"}
entry = {"q": qa["q"], "gold": qa["gold"], "answer": answer, **verdict}
if query is not None:
entry["search_query"] = query
results.append(entry)
scored = [r["score"] for r in results]
label = f"{name}+recovery" if with_recovery else name
summary = {
"policy": label,
"before_tokens": total_tokens(before),
"after_tokens": total_tokens(compressed),
"after_msgs": len(compressed),
"compress_seconds": round(elapsed, 1),
"recall_pct": round(100 * sum(scored) / (2 * len(scored)), 1) if scored else 0.0,
"scores": scored,
"summary_error": getattr(comp, "_last_summary_error", None),
}
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / f"{label.replace('+', '_')}.json").write_text(json.dumps({"summary": summary, "results": results}, indent=1), encoding="utf-8")
return summary
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--transcript", required=True)
ap.add_argument("--cap-tokens", type=int, default=500_000)
ap.add_argument("--policies", default="current,tail25k,codex_style")
ap.add_argument("--questions", type=int, default=15)
ap.add_argument("--out", required=True)
ap.add_argument("--also-uncompacted", action="store_true")
args = ap.parse_args()
messages = load_transcript(args.transcript, cap_tokens=args.cap_tokens)
out_dir = Path(args.out)
tid = hashlib.md5(args.transcript.encode()).hexdigest()[:10]
qcache = out_dir / f"questions-{tid}.json"
questions = generate_questions(messages, args.questions, qcache)
print(f"{len(questions)} questions ready ({qcache})")
summaries = []
if args.also_uncompacted:
spec = {"ctor": {}, "attrs": {"tail_token_budget": 10**9}}
# control: no compression at all — answer from the full transcript
context_text = serialize_for_exam(messages, char_cap=900_000)
results = []
for qa in questions:
answer = _call(ANSWER_PROMPT.format(context=context_text, question=qa["q"]), max_tokens=400)
verdict_raw = _call(JUDGE_PROMPT.format(question=qa["q"], gold=qa["gold"], answer=answer), max_tokens=300)
try:
verdict = _extract_json(verdict_raw)
except Exception:
verdict = {"score": 0, "why": "judge parse failure"}
results.append({"q": qa["q"], **verdict, "answer": answer})
scored = [r["score"] for r in results]
ctl = {
"policy": "uncompacted_control",
"before_tokens": total_tokens(messages),
"after_tokens": total_tokens(messages),
"recall_pct": round(100 * sum(scored) / (2 * len(scored)), 1),
"scores": scored,
}
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "uncompacted_control.json").write_text(json.dumps({"summary": ctl, "results": results}, indent=1), encoding="utf-8")
summaries.append(ctl)
print(json.dumps(ctl, indent=1))
for name in args.policies.split(","):
name = name.strip()
with_recovery = name.endswith("+recovery")
base = name[:-len("+recovery")] if with_recovery else name
if base not in POLICIES:
print(f"unknown policy {base}, skipping"); continue
s = run_policy(base, POLICIES[base], messages, questions, out_dir,
with_recovery=with_recovery)
summaries.append(s)
print(json.dumps(s, indent=1))
(out_dir / "scorecard.json").write_text(json.dumps(summaries, indent=1), encoding="utf-8")
print(f"\nscorecard -> {out_dir}/scorecard.json")
if __name__ == "__main__":
main()
evals/compaction/scripts/build_html_report.py
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env python3
"""Build a self-contained HTML report comparing compaction runs.
Usage: build_report.py <runs_dir> <out_html>
Expects runs/<checkout>_<session>.json pairs from run_compaction.py.
"""
import html
import json
import sys
from pathlib import Path
RUNS = Path(sys.argv[1])
OUT = sys.argv[2]
pairs = {}
for f in sorted(RUNS.glob("*.json")):
co, sid = f.stem.split("_", 1)
if co == "main":
co, sid = "main-co", f.stem[len("main-co_"):]
elif co == "pr":
co, sid = "pr-co", f.stem[len("pr-co_"):]
data = json.loads(f.read_text(encoding="utf-8"))
pairs.setdefault(sid, {})[co] = data
E = html.escape
def msg_class(m):
role = m.get("role", "?")
c = m.get("content") or ""
if isinstance(c, str):
if "[CONTEXT COMPACTION" in c or "[CONTEXT SUMMARY" in c:
return "summary"
if "SKILL_PRUNED" in c:
return "skillpruned"
if "SKILL POLICY DIGEST" in c or "SKILL_POLICY_DIGEST" in c:
return "digest"
if "preserved across context compression" in c:
return "todosnap"
return role
def render_msg(m, idx):
role = m.get("role", "?")
c = m.get("content")
if not isinstance(c, str):
c = json.dumps(c, default=str)[:2000]
tool = m.get("tool_name") or ""
tcs = m.get("tool_calls") or []
tc_names = ", ".join(
(t.get("function", {}) or {}).get("name", "?") for t in tcs if isinstance(t, dict)
)
cls = msg_class(m)
nchars = len(c)
label = role
if tool:
label += f" · {tool}"
if tc_names:
label += f" → {tc_names}"
preview = c[:180].replace("\n", " ")
full = c if nchars <= 20000 else c[:20000] + f"\n…[{nchars-20000:,} more chars]"
return (
f'<details class="msg {cls}"><summary><span class="idx">#{idx}</span>'
f'<span class="role">{E(label)}</span>'
f'<span class="chars">{nchars:,}ch</span>'
f'<span class="preview">{E(preview)}</span></summary>'
f"<pre>{E(full)}</pre></details>"
)
def render_column(title, data, key):
meta = data["meta"]
msgs = data[key]
body = "".join(render_msg(m, i) for i, m in enumerate(msgs))
return (
f'<div class="col"><div class="colhead"><h3>{E(title)}</h3>'
f'<div class="stats">{meta[key.replace("before","before_msgs").replace("after","after_msgs")] if False else len(msgs)} msgs · '
f'~{(meta["before_tokens_est"] if key=="before" else meta["after_tokens_est"]):,} tok</div></div>'
f'<div class="msgs">{body}</div></div>'
)
def survival_stats(before, after):
after_texts = set()
for m in after:
c = m.get("content")
if isinstance(c, str) and c:
after_texts.add(c[:400])
kept = sum(1 for m in before if isinstance(m.get("content"), str) and (m.get("content") or "")[:400] in after_texts)
return kept
sections = []
toc = []
for sid, versions in pairs.items():
if "main-co" not in versions or "pr-co" not in versions:
continue
main_d, pr_d = versions["main-co"], versions["pr-co"]
title = main_d["meta"].get("title") or sid
mm, pm = main_d["meta"], pr_d["meta"]
def count_markers(msgs, needle):
return sum((m.get("content") or "").count(needle) for m in msgs if isinstance(m.get("content"), str))
rows = []
def stat(name, mv, pv):
cls = "diff" if mv != pv else ""
rows.append(f"<tr class='{cls}'><td>{E(name)}</td><td>{E(str(mv))}</td><td>{E(str(pv))}</td></tr>")
stat("Messages after", mm["after_msgs"], pm["after_msgs"])
stat("Est. tokens after", f"{mm['after_tokens_est']:,}", f"{pm['after_tokens_est']:,}")
stat("Reduction", f"{100-100*mm['after_tokens_est']//max(1,mm['before_tokens_est'])}%", f"{100-100*pm['after_tokens_est']//max(1,pm['before_tokens_est'])}%")
stat("Compress time", f"{mm['elapsed_s']}s", f"{pm['elapsed_s']}s")
stat("SKILL_PRUNED markers", count_markers(main_d["after"], "SKILL_PRUNED"), count_markers(pr_d["after"], "SKILL_PRUNED"))
stat("Policy digest blocks", count_markers(main_d["after"], "SKILL POLICY DIGEST") + count_markers(main_d["after"], "SKILL_POLICY_DIGEST"), count_markers(pr_d["after"], "SKILL POLICY DIGEST") + count_markers(pr_d["after"], "SKILL_POLICY_DIGEST"))
stat("Todo snapshot present", "yes" if count_markers(main_d["after"], "preserved across context compression") else "no", "yes" if count_markers(pr_d["after"], "preserved across context compression") else "no")
stat("Kept-verbatim msgs", survival_stats(main_d["before"], main_d["after"]), survival_stats(pr_d["before"], pr_d["after"]))
stat("Summary error", mm.get("summary_error") or "—", pm.get("summary_error") or "—")
todo_html = ""
for label, d in (("main", mm), ("PR #87090", pm)):
tb = d.get("todo_injection_block")
if tb:
todo_html += f"<h4>Todo injection block — {E(label)}</h4><pre class='todoblock'>{E(tb)}</pre>"
anchor = f"s-{sid}"
toc.append(f'<a href="#{anchor}">{E(title)} <span class="dim">({sid})</span></a>')
sections.append(f"""
<section id="{anchor}">
<h2>{E(title)} <span class="dim">{sid}</span></h2>
<table class="stats-table"><tr><th></th><th>main</th><th>PR #87090</th></tr>{"".join(rows)}</table>
{todo_html}
<div class="cols">
{render_column("BEFORE (original transcript)", main_d, "before")}
{render_column("AFTER — main", main_d, "after")}
{render_column("AFTER — PR #87090", pr_d, "after")}
</div>
</section>""")
page = f"""<!DOCTYPE html>
<html><head><meta charset="utf-8"><title>Compaction comparison — main vs PR #87090</title>
<style>
:root {{ color-scheme: dark; }}
body {{ background:#0d1117; color:#c9d1d9; font:14px/1.45 -apple-system,Segoe UI,sans-serif; margin:0; padding:24px; }}
h1 {{ font-size:22px; }} h2 {{ font-size:18px; border-bottom:1px solid #30363d; padding-bottom:6px; margin-top:48px; }}
.dim {{ color:#8b949e; font-weight:normal; font-size:12px; }}
nav a {{ display:block; color:#58a6ff; margin:2px 0; text-decoration:none; }}
.legend span {{ display:inline-block; padding:2px 10px; margin-right:8px; border-radius:4px; font-size:12px; }}
.stats-table {{ border-collapse:collapse; margin:12px 0; }}
.stats-table td, .stats-table th {{ border:1px solid #30363d; padding:4px 12px; text-align:left; font-size:13px; }}
.stats-table tr.diff td {{ background:#1c2a1c; }}
.cols {{ display:grid; grid-template-columns:1fr 1fr 1fr; gap:10px; }}
.col {{ min-width:0; }}
.colhead {{ position:sticky; top:0; background:#161b22; padding:8px; border:1px solid #30363d; border-radius:6px 6px 0 0; z-index:2; }}
.colhead h3 {{ margin:0; font-size:13px; }} .colhead .stats {{ color:#8b949e; font-size:12px; }}
.msgs {{ border:1px solid #30363d; border-top:none; max-height:80vh; overflow-y:auto; }}
.msg {{ border-bottom:1px solid #21262d; }}
.msg summary {{ cursor:pointer; padding:3px 6px; display:flex; gap:6px; align-items:baseline; white-space:nowrap; overflow:hidden; }}
.msg summary::-webkit-details-marker {{ display:none; }}
.idx {{ color:#484f58; font-size:11px; min-width:34px; }}
.role {{ font-size:11px; font-weight:600; min-width:110px; overflow:hidden; text-overflow:ellipsis; }}
.chars {{ color:#8b949e; font-size:11px; min-width:52px; }}
.preview {{ color:#8b949e; font-size:11px; overflow:hidden; text-overflow:ellipsis; flex:1; }}
.msg pre {{ white-space:pre-wrap; word-break:break-word; font-size:11px; background:#161b22; margin:0; padding:8px; max-height:400px; overflow-y:auto; }}
.msg.user summary {{ background:#0d2137; }} .msg.user .role {{ color:#58a6ff; }}
.msg.assistant .role {{ color:#d2a8ff; }}
.msg.tool .role {{ color:#7ee787; }}
.msg.system summary {{ background:#21262d; }} .msg.system .role {{ color:#8b949e; }}
.msg.summary summary {{ background:#3d2e00; }} .msg.summary .role {{ color:#e3b341; }}
.msg.skillpruned summary {{ background:#3d1418; }} .msg.skillpruned .role {{ color:#ff7b72; }}
.msg.digest summary {{ background:#1b3d2e; }} .msg.digest .role {{ color:#56d364; }}
.msg.todosnap summary {{ background:#2d1b3d; }} .msg.todosnap .role {{ color:#d2a8ff; }}
.todoblock {{ background:#1b1230; border:1px solid #6e40c9; padding:10px; white-space:pre-wrap; font-size:12px; }}
</style></head><body>
<h1>Compaction comparison — current main (7619564fb) vs PR #87090 (41fd511f6)</h1>
<p class="dim">Real sessions from state.db (copy), replayed through each checkout's ContextCompressor with force=True. Real LLM summaries. Click any row to expand the full message.</p>
<div class="legend">
<span style="background:#3d2e00;color:#e3b341">compaction summary</span>
<span style="background:#3d1418;color:#ff7b72">SKILL_PRUNED marker</span>
<span style="background:#1b3d2e;color:#56d364">policy digest</span>
<span style="background:#2d1b3d;color:#d2a8ff">todo snapshot</span>
<span style="background:#0d2137;color:#58a6ff">user</span>
</div>
<nav>{"".join(toc)}</nav>
{"".join(sections)}
</body></html>"""
Path(OUT).write_text(page, encoding="utf-8")
print(f"wrote {OUT} ({len(page):,} bytes, {len(sections)} sessions)")
evals/compaction/scripts/codex_arm.py
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""Run the codex CLI as an eval arm on the same transcripts + question banks.
Per transcript:
1. Split the 500K-token prefix into ~150KB chunk files in a work dir.
2. `codex exec` reads every file (2-3 sentence summary each) — the read
volume exceeds codex's 258K window, so its auto-compaction fires
naturally (verified via token_count drops / compacted events in the
rollout jsonl).
3. `codex exec resume --last` asks the SAME 15 exam questions; answers are
judged by the same LLM judge against the same golds.
Usage: codex_arm.py <lineage_json> <questions_json> <workdir> <out_json>
"""
import glob
import json
import os
import re
import subprocess
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[0] / "main-co"))
LINEAGE = sys.argv[1]
QUESTIONS = sys.argv[2]
WORKDIR = Path(sys.argv[3])
OUT = sys.argv[4]
JUDGE_PROMPT = """Score this answer against the gold answer. Reply with STRICT JSON: {{"score": 2|1|0, "why": "..."}}.
2 = factually matches gold (wording may differ)
1 = partially correct or hedged-but-right
0 = wrong, or refuses/says it doesn't know with a wrong/no guess
QUESTION: {question}
GOLD: {gold}
ANSWER: {answer}"""
def prepare_chunks() -> int:
from evals.compaction.fixtures import load_transcript
WORKDIR.mkdir(parents=True, exist_ok=True)
msgs = load_transcript(LINEAGE, cap_tokens=500_000)
chunk, size, idx = [], 0, 0
for m in msgs:
c = m.get("content") or ""
if not isinstance(c, str) or not c:
continue
chunk.append(f"--- {m['role']} ---\n{c}\n")
size += len(c)
if size > 150_000:
(WORKDIR / f"transcript_{idx:02d}.txt").write_text(
"\n".join(chunk), encoding="utf-8")
chunk, size = [], 0
idx += 1
if chunk:
(WORKDIR / f"transcript_{idx:02d}.txt").write_text(
"\n".join(chunk), encoding="utf-8")
idx += 1
return idx
def newest_rollout() -> str:
files = sorted(
glob.glob(os.path.expanduser("~/.codex/sessions/*/*/*/rollout-*.jsonl")),
key=os.path.getmtime,
)
return files[-1] if files else ""
def rollout_session_id(path: str) -> str:
for line in open(path, encoding="utf-8", errors="replace"):
try:
d = json.loads(line)
except Exception:
continue
if d.get("type") == "session_meta":
return d.get("payload", {}).get("session_id", "")
return ""
def last_agent_message(path: str) -> str:
msgs = []
for line in open(path, encoding="utf-8", errors="replace"):
try:
d = json.loads(line)
except Exception:
continue
p = d.get("payload", {})
if p.get("type") == "agent_message":
msgs.append(p.get("message", ""))
return msgs[-1] if msgs else ""
def rollout_stats(path: str) -> dict:
compacted = 0
peak = 0
for line in open(path, encoding="utf-8", errors="replace"):
try:
d = json.loads(line)
except Exception:
continue
p = d.get("payload", {})
if d.get("type") == "compacted" or p.get("type") == "compacted":
compacted += 1
if p.get("type") == "token_count" and p.get("info"):
last = p["info"].get("last_token_usage") or {}
ctx = last.get("input_tokens", 0) + last.get("cached_input_tokens", 0)
peak = max(peak, ctx)
return {"compaction_events": compacted, "peak_context_tokens": peak}
def codex(args: list, prompt: str, timeout: int = 3600) -> str:
proc = subprocess.run(
["codex", "exec", *args, "--skip-git-repo-check", prompt],
cwd=str(WORKDIR), capture_output=True, text=True, timeout=timeout,
)
return proc.stdout + proc.stderr
def judge(question: str, gold: str, answer: str) -> dict:
from agent.auxiliary_client import call_llm
resp = call_llm(
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
question=question, gold=gold, answer=answer)}],
task="compression", max_tokens=300,
)
text = resp.choices[0].message.content if hasattr(resp, "choices") else str(resp)
m = re.search(r"\{.*\}", text, re.S)
try:
return json.loads(m.group(0))
except Exception:
return {"score": 0, "why": f"judge parse failure: {text[:80]}"}
def main():
n = prepare_chunks()
print(f"[codex-arm] {WORKDIR.name}: {n} chunk files", flush=True)
t0 = time.time()
codex(
["-s", "read-only"],
f"This directory contains transcript_00.txt through transcript_{n-1:02d}.txt. "
"Read EVERY file COMPLETELY one at a time using 'cat transcript_NN.txt' "
"(full file, do not use head/tail/grep). After each file, write a 2-3 "
"sentence summary of what happened in that portion. Do not skip any file.",
)
rollout = newest_rollout()
session_id = rollout_session_id(rollout)
stats = rollout_stats(rollout)
# Codex auto-compacts at ~90% of its 258K window. If one read pass didn't
# trigger it, re-read files in the SAME session until it does (max 3
# extra passes) — the comparison requires post-compaction state.
passes = 0
while stats["compaction_events"] == 0 and passes < 3:
passes += 1
print(f"[codex-arm] no compaction yet (peak={stats['peak_context_tokens']:,}) — re-read pass {passes}", flush=True)
codex(
["resume", session_id],
"Re-read ALL transcript files again completely with 'cat', one at a "
"time, and refine each of your per-file summaries with any details "
"you missed. Do not skip any file.",
)
stats = rollout_stats(rollout)
read_s = time.time() - t0
print(f"[codex-arm] read phase {read_s:.0f}s, {stats}", flush=True)
if stats["compaction_events"] == 0:
print("[codex-arm] WARNING: compaction never fired — arm invalid", flush=True)
questions = json.loads(Path(QUESTIONS).read_text(encoding="utf-8"))
qlist = "\n".join(f"{i+1}. {q['q']}" for i, q in enumerate(questions))
codex(
["resume", session_id],
"Based on everything you learned from the transcript files earlier in "
"this session, answer the following questions from memory. Do NOT "
"re-read any files — answer only from what you currently retain in "
"context. If you don't know, say 'UNKNOWN' and give your best guess. "
"Reply with a numbered list, one concise answer per question.\n\n" + qlist,
)
quiz_text = last_agent_message(rollout)
print(f"[codex-arm] quiz reply: {len(quiz_text)} chars", flush=True)
answers = {}
for m in re.finditer(r"(?m)^\s*\**(\d{1,2})[.)]\**\s+(.+?)(?=^\s*\**\d{1,2}[.)]\**\s|\Z)",
quiz_text, re.S):
answers[int(m.group(1))] = m.group(2).strip()[:600]
results = []
for i, q in enumerate(questions):
ans = answers.get(i + 1, "(no answer parsed)")
verdict = judge(q["q"], q["gold"], ans)
results.append({"q": q["q"], "gold": q["gold"], "answer": ans, **verdict})
print(f" Q{i+1}: {verdict['score']}", flush=True)
scored = [r["score"] for r in results]
summary = {
"policy": "codex_real",
"recall_pct": round(100 * sum(scored) / (2 * len(scored)), 1),
"scores": scored,
"read_seconds": round(read_s),
**stats,
"rollout": rollout,
}
Path(OUT).write_text(json.dumps({"summary": summary, "results": results}, indent=1),
encoding="utf-8")
print(json.dumps(summary, indent=1), flush=True)
if __name__ == "__main__":
main()
evals/compaction/scripts/reconstruct_lineage.py
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""Reconstruct the full uncompacted transcript of a session LINEAGE.
Rotation children start with a copy of the compressed parent (head + summary +
tail). To rebuild the real history: walk the chain root->leaf, append messages
not seen before (hash of role+content+tool_calls), and skip synthetic
compaction summaries / todo snapshots so we get the organic transcript.
Usage: reconstruct_lineage.py <state_db_copy> <root_session_id> <out_json>
ALWAYS run against a COPY of state.db, never the live file.
"""
import hashlib
import json
import sqlite3
import sys
DB = sys.argv[1]
ROOT = sys.argv[2]
OUT = sys.argv[3]
db = sqlite3.connect(DB)
db.row_factory = sqlite3.Row
# collect the whole descendant tree, chronological by started_at
import collections
children = collections.defaultdict(list)
for r in db.execute(
"SELECT id, parent_session_id FROM sessions WHERE parent_session_id IS NOT NULL"
):
children[r["parent_session_id"]].append(r["id"])
chain = []
frontier = [ROOT]
while frontier:
sid = frontier.pop(0)
chain.append(sid)
frontier.extend(children.get(sid, []))
starts = {r["id"]: r["started_at"] or "" for r in db.execute(
f"SELECT id, started_at FROM sessions WHERE id IN ({','.join('?'*len(chain))})", chain)}
chain.sort(key=lambda s: starts.get(s, ""))
print(f"chain: {len(chain)} sessions")
SYNTH_MARKERS = (
"[CONTEXT COMPACTION", "[CONTEXT SUMMARY", "[PRIOR CONTEXT",
"preserved across context compression",
)
seen = set()
out = []
sysprompt = None
for sid in chain:
if sysprompt is None:
row = db.execute(
"SELECT s.system_prompt, sp.prompt AS dedup_prompt FROM sessions s "
"LEFT JOIN system_prompts sp ON sp.hash = s.system_prompt_hash "
"WHERE s.id=?", (sid,)).fetchone()
if row:
sysprompt = row["system_prompt"] or row["dedup_prompt"] or None
for r in db.execute(
"SELECT * FROM messages WHERE session_id=? ORDER BY id", (sid,)
):
c = r["content"] or ""
if any(m in c for m in SYNTH_MARKERS):
continue # synthetic compaction artifact, not organic history
h = hashlib.md5(
(r["role"] + "\x00" + c + "\x00" + (r["tool_calls"] or "")).encode(
"utf-8", "replace")
).hexdigest()
if h in seen:
continue
seen.add(h)
m = {"role": r["role"], "content": c}
if r["tool_calls"]:
try:
m["tool_calls"] = json.loads(r["tool_calls"])
except Exception:
pass
if r["tool_call_id"]:
m["tool_call_id"] = r["tool_call_id"]
if r["tool_name"]:
m["tool_name"] = r["tool_name"]
out.append(m)
msgs = [{"role": "system", "content": sysprompt or ""}] + out
chars = sum(len(m.get("content") or "") + len(json.dumps(m.get("tool_calls", ""), default=str)) for m in msgs)
print(f"reconstructed: {len(msgs)} msgs, {chars:,} chars (~{chars//4:,} tok)")
json.dump({"root": ROOT, "chain": chain, "messages": msgs}, open(OUT, "w", encoding="utf-8"), default=str)
print(f"wrote {OUT}")
evals/compaction/scripts/replay_lineage.py
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""Replay a ~500K-token prefix of a reconstructed lineage through compaction.
Usage: run_lineage_compaction.py <checkout> <lineage_json> <out_json> [cap_tokens]
Takes the chronological prefix of the lineage at the token cap (default 500K =
the 50% trigger on a 1M-context model), aligned to a tool-group boundary, and
runs ContextCompressor.compress() exactly as the live trigger would.
"""
import copy
import json
import sys
import time
from pathlib import Path
CHECKOUT = sys.argv[1]
LINEAGE = sys.argv[2]
OUT = sys.argv[3]
CAP = int(sys.argv[4]) if len(sys.argv) > 4 else 500_000
sys.path.insert(0, CHECKOUT)
data = json.load(open(LINEAGE, encoding="utf-8"))
msgs = data["messages"]
def tok(m):
t = len(m.get("content") or "") // 4
tc = m.get("tool_calls")
if tc:
t += len(json.dumps(tc, default=str)) // 4
return t
# chronological prefix up to CAP tokens
prefix = []
total = 0
for m in msgs:
t = tok(m)
if total + t > CAP and len(prefix) > 10:
break
prefix.append(m)
total += t
# align the end: never end on an assistant msg with tool_calls whose results
# were cut off; drop trailing orphans
while prefix and prefix[-1].get("tool_calls"):
prefix.pop()
# also drop trailing tool results with no preceding assistant tool_calls kept
# (compress()'s _sanitize_tool_pairs would handle it, but keep input clean)
before_tokens = sum(tok(m) for m in prefix)
print(f"[{Path(CHECKOUT).name}] {Path(LINEAGE).stem}: prefix {len(prefix)} msgs ~{before_tokens:,} tok (cap {CAP:,})")
from agent.context_compressor import ContextCompressor # noqa: E402
model = "anthropic/claude-fable-5"
comp = ContextCompressor(model=model, quiet_mode=True)
before = copy.deepcopy(prefix)
t0 = time.time()
compressed = comp.compress(prefix, current_tokens=before_tokens, force=True)
dt = time.time() - t0
after_tokens = sum(tok(m) for m in compressed)
print(f" -> {len(compressed)} msgs ~{after_tokens:,} tok in {dt:.1f}s (err={getattr(comp,'_last_summary_error',None)})")
json.dump({
"meta": {
"checkout": Path(CHECKOUT).name,
"session_id": data["root"],
"title": f"lineage {data['root']} ({len(data['chain'])} rotations)",
"model": model,
"elapsed_s": round(dt, 1),
"before_msgs": len(before),
"after_msgs": len(compressed),
"before_tokens_est": before_tokens,
"after_tokens_est": after_tokens,
"summary_error": getattr(comp, "_last_summary_error", None),
"todo_injection_block": None,
},
"before": before,
"after": compressed,
}, open(OUT, "w", encoding="utf-8"), default=str)
print(f" wrote {OUT}")
evals/compaction/test_region_scoping.py
+98
View File
@@ -0,0 +1,98 @@
"""Region-scoping tripwire: the summarizer must only see the compacted region.
Builds a transcript with sentinel strings planted in (a) the protected head,
(b) the middle (to-be-compacted) region, and (c) the tail, mocks call_llm to
capture the prompt, and asserts head/tail sentinels never reach the
summarizer while the middle sentinel does. Runs for both legacy and lean
modes, and asserts the lean deterministic sections (anchors, verbatim users)
also carry only middle-region content.
"""
import json
import sys
from pathlib import Path
from unittest.mock import MagicMock, patch
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
from agent.context_compressor import ContextCompressor # noqa: E402
HEAD_SENTINEL = "HEADSENTINEL_zq81"
MID_SENTINEL = "MIDSENTINEL_kv93"
TAIL_SENTINEL = "TAILSENTINEL_pw27"
def _mk_transcript():
msgs = [
{"role": "system", "content": "system prompt"},
{"role": "user", "content": f"first user message {HEAD_SENTINEL}"},
{"role": "assistant", "content": "ack"},
]
for i in range(40):
marker = f" {MID_SENTINEL}-{i}" if i % 5 == 0 else ""
msgs.append({
"role": "assistant", "content": f"mid step {i}{marker}",
"tool_calls": [{"id": f"m{i}", "function": {"name": "terminal", "arguments": "{}"}}],
})
msgs.append({"role": "tool", "tool_call_id": f"m{i}",
"content": (f"mid tool output {i} " * 300) + marker})
for i in range(6):
msgs.append({"role": "assistant", "content": f"tail step {i} {TAIL_SENTINEL}-{i}",
"tool_calls": [{"id": f"t{i}", "function": {"name": "terminal", "arguments": "{}"}}]})
msgs.append({"role": "tool", "tool_call_id": f"t{i}", "content": f"tail output {i} {TAIL_SENTINEL}-{i}"})
msgs.append({"role": "user", "content": f"latest user question {TAIL_SENTINEL}-u"})
msgs.append({"role": "assistant", "content": "final answer in tail"})
return msgs
def run_mode(tail_mode: str):
captured = []
def fake_call_llm(messages=None, **kw):
captured.append(messages[0]["content"] if messages else "")
resp = MagicMock()
resp.choices[0].message.content = "## Active Task\nsummarized"
return resp
comp = ContextCompressor(model="anthropic/claude-fable-5", quiet_mode=True,
tail_mode=tail_mode)
comp.tail_token_budget = 3_000 # force a real middle on the small fixture
comp._session_id = "scope-test"
msgs = _mk_transcript()
with patch("agent.context_compressor.call_llm", side_effect=fake_call_llm), \
patch("agent.auxiliary_client.call_llm", side_effect=fake_call_llm):
out = comp.compress(msgs, current_tokens=200_000, force=True)
all_prompts = "\n".join(captured)
assert captured, f"[{tail_mode}] summarizer never called"
assert MID_SENTINEL in all_prompts, f"[{tail_mode}] middle region missing from summarizer input"
# Head/tail user messages MAY appear inside the FOCUS TOPIC steering block
# (intentional: tells the summarizer what the user currently cares about).
# They must NOT appear in the serialized TURNS body being summarized.
for p in captured:
body = p.split("FOCUS TOPIC:")[0]
assert TAIL_SENTINEL not in body, f"[{tail_mode}] TAIL leaked into summarized turns"
assert HEAD_SENTINEL not in body, f"[{tail_mode}] protected HEAD leaked into summarized turns"
# The tail must survive verbatim; the head user message must survive.
out_text = "\n".join(str(m.get("content")) for m in out)
assert f"{TAIL_SENTINEL}-u" in out_text, f"[{tail_mode}] latest user message lost"
assert HEAD_SENTINEL in out_text, f"[{tail_mode}] head lost"
if tail_mode == "lean":
summary_msg = next(
(str(m.get("content")) for m in out
if isinstance(m.get("content"), str) and "Anchor Index" in m["content"]),
"",
)
if summary_msg:
assert TAIL_SENTINEL not in summary_msg.split("END OF CONTEXT SUMMARY")[0], \
"[lean] tail content leaked into summary sections"
print(f" {tail_mode}: OK ({len(captured)} summarizer call(s), "
f"{len(out)} msgs out)")
if __name__ == "__main__":
for mode in ("legacy", "lean"):
run_mode(mode)
print("scoping tripwire: ALL PASS")