Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license
This commit is contained in:
@@ -0,0 +1,357 @@
|
||||
# Compaction v2 — 4-transcript scorecard (2026-08-15, anchor-index build)
|
||||
|
||||
Four real 500K-token lineage transcripts from state.db (sweep campaign, GUI
|
||||
desktop work, PR-merge campaign, ACP/PR review), 15-question recall exam
|
||||
each. "recovery" = one session_search round-trip (FTS5+BM25 sim) against the
|
||||
archived region. Lean build includes: 25K clamped tail, tail tool demotion,
|
||||
chunked digests (noise-filtered, pristine tool contents), mechanical anchor
|
||||
index, verbatim user messages, recovery footer, upgraded summarizer prompt.
|
||||
|
||||
> **Historical note (2026-08-30):** the "chunked digests" arm described here
|
||||
> was later replaced — the detailed session log is now produced by the SAME
|
||||
> single summary request (lean compaction makes exactly one auxiliary LLM
|
||||
> call per attempt; no per-chunk digest calls). See #96603.
|
||||
|
||||
## Results (recall % @ retained tokens)
|
||||
|
||||
policy sweep gui prmerge acp AVG
|
||||
uncompacted 93.3 @ 500K 96.7 @ 500K 96.7 @ 500K 100.0 @ 500K 96.7
|
||||
current 93.3*@ 176K 26.7*@ 156K 33.3 @ 155K 30.0 @ 160K 45.8 @ 162K
|
||||
lean 40.0 @ 62K 60.0 @ 41K 23.3 @ 44K 36.7 @ 50K 40.0 @ 49K
|
||||
lean+recovery 70.0 @ 62K 80.0 @ 41K 43.3 @ 45K 80.0 @ 50K 68.3 @ 49K
|
||||
|
||||
* sweep/gui current scores are from the previous question banks (same
|
||||
transcripts; banks regenerated in the 4-way run). prmerge/acp are clean
|
||||
same-bank comparisons across all arms.
|
||||
|
||||
## Findings
|
||||
|
||||
1. LEAN+RECOVERY BEATS CURRENT BY +22.5pts ON AVERAGE (68.3 vs 45.8) AT 3.3x
|
||||
FEWER TOKENS (49K vs 162K). It wins on 3 of 4 transcripts and loses only
|
||||
sweep — the one transcript where current's fat tail got lucky with
|
||||
restated facts (93.3 is bank-inflated luck; see finding 3 of the previous
|
||||
scorecard).
|
||||
|
||||
2. THE ANCHOR INDEX FIXED THE NEEDLE-FACT CLASS. GUI closed-book went
|
||||
23.3 -> 60.0 and GUI+recovery 46.7 -> 80.0 after mechanically indexing
|
||||
exact identifiers (SHAs, ids, paths, error strings) instead of trusting
|
||||
the summarizer with them. ACP+recovery hit 80.0.
|
||||
|
||||
3. TWO FRESH TRANSCRIPTS CONFIRM CURRENT IS WEAK, NOT STRONG: 33.3 and 30.0
|
||||
at ~157K retained. The original sweep 93.3 was restatement luck, not
|
||||
policy quality. Current's average is 45.8% for 162K tokens — lean+recovery
|
||||
is 22 points better for less than a third of the spend.
|
||||
|
||||
4. prmerge IS THE HARD CASE for everyone (96.7 ceiling, best policy 43.3):
|
||||
1.1M-token lineage truncated at 500K, dense multi-PR state. Recovery
|
||||
misses there are mostly query formulation. Headroom, not a blocker.
|
||||
|
||||
5. Goal check (Teknium): tail = max(10K, 2.5%) ✓; summaries scoped to the
|
||||
compacted region only ✓ (sentinel tripwire test); session_search pointer ✓
|
||||
(+20-43pts measured); better accuracy AND more savings than current ✓
|
||||
(+22.5pts at 0.30x tokens).
|
||||
|
||||
|
||||
## Codex CLI head-to-head (same transcripts, same exams, same judge)
|
||||
|
||||
Real OpenAI Codex CLI (v0.147.0, gpt-5.6-sol, 258K window) run end-to-end on
|
||||
the identical four transcripts: chunk files read via `codex exec` until its
|
||||
REAL auto-compaction fired (verified `compacted` event in the rollout jsonl;
|
||||
peak context 455-483K), then quizzed post-compaction from memory with the
|
||||
same 15-question banks and scored by the same judge.
|
||||
|
||||
policy sweep gui prmerge acp AVG retained state
|
||||
codex (real, post-cmp) 26.7% 40.0% 43.3% 36.7% 36.7% ~4.5K (opaque blob + user msgs)
|
||||
hermes current 93.3%* 26.7%* 33.3% 30.0% 45.8% ~162K
|
||||
hermes lean closed-book 40.0% 60.0% 23.3% 36.7% 40.0% ~49K
|
||||
hermes lean+recovery 70.0% 80.0% 43.3% 80.0% 68.3% ~49K
|
||||
|
||||
Notes:
|
||||
- codex answers from its own post-compaction session — the honest analog of
|
||||
our closed-book arms. It has NO session_search equivalent (its rollout is
|
||||
on disk but the agent cannot search it at runtime), so recovery has no
|
||||
codex counterpart; that gap is exactly the differentiator lean leans on.
|
||||
- Apples-to-apples closed-book: lean 40.0% vs codex 36.7% — parity-plus at
|
||||
10x codex's retained state but 0.30x current's. With recovery: +31.6pts
|
||||
over codex.
|
||||
- codex ties lean+recovery on prmerge (43.3%) — the dense multi-PR campaign
|
||||
is the hardest transcript for every policy and the clearest iteration
|
||||
target.
|
||||
- Methodology caveats: codex ingested transcripts as FILE READS (tool
|
||||
outputs), not native conversation — this matches how its compaction
|
||||
treats tool output (drops it all into the server-side summary) but is not
|
||||
byte-identical to a native session. Its model (gpt-5.6-sol) also differs
|
||||
from the answering model in our arms; scores compare COMPACTION PIPELINES
|
||||
end-to-end, not models in isolation. One codex quiz reply was also
|
||||
capped short (~1K chars for 15 answers), which its terse post-compaction
|
||||
style invites.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Ship lean as opt-in (compression.tail_mode: lean, legacy default), harness as
|
||||
the permanent gate. Iterate prmerge-class recall behind the flag (query
|
||||
mining, per-epoch anchor windows) before default flip.
|
||||
|
||||
|
||||
## Appendix: full per-transcript detail
|
||||
|
||||
### Transcript: sweep
|
||||
|
||||
| policy | recall | tokens before | tokens after | compress s |
|
||||
|---|---|---|---|---|
|
||||
| lean | 40.0% | 499,625 | 61,567 | 114.9 |
|
||||
| lean+recovery | 70.0% | 499,625 | 61,792 | 114.8 |
|
||||
|
||||
<details><summary>15 exam questions (questions-30b95351c7.json)</summary>
|
||||
|
||||
1. **What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?**
|
||||
gold: `the stale file version silently deletes newer main code`
|
||||
2. **According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?**
|
||||
gold: `~60GB`
|
||||
3. **Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?**
|
||||
gold: `backgroundThrottling`
|
||||
4. **What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?**
|
||||
gold: `Auto merge is not allowed for this repository (enablePullRequestAutoMerge)`
|
||||
5. **What is the specified 'Rule 0' that must be included in a subagent brief?**
|
||||
gold: `load the skill first`
|
||||
6. **In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?**
|
||||
gold: `add config gate for title generation`
|
||||
7. **Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?**
|
||||
gold: `tests/test_packaging_metadata.py`
|
||||
8. **What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?**
|
||||
gold: `completed normally (exit code 0)`
|
||||
9. **What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?**
|
||||
gold: `relative exclude-newer makes the committed lock stale every day`
|
||||
10. **How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?**
|
||||
gold: `13`
|
||||
11. **In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?**
|
||||
gold: `-1`
|
||||
12. **What is the full title block text for the technical schematic infographic generated for the Gateway Drain?**
|
||||
gold: `GATEWAY DRAIN × CRON — SHUTDOWN CONTRACT`
|
||||
13. **Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?**
|
||||
gold: `82980`
|
||||
14. **What is the specific Gist ID created for the PR infographic host in the cron cluster?**
|
||||
gold: `ee33edd5804689243f974536ef7aecb9`
|
||||
15. **What was the final merge SHA for Cluster D's Trigger-now PR #70638?**
|
||||
gold: `f9d64b9a9d8b306f64851c1a13869d96ad5d7869`
|
||||
|
||||
</details>
|
||||
|
||||
<details><summary>15 exam questions (questions-5be475cde0.json)</summary>
|
||||
|
||||
1. **What exact command did the agent use to search for open issues related to a specific topic during Phase 1 of the cluster-sweep salvage?**
|
||||
gold: `gh issue list --search "<topic>" --state open --limit 100 --json number,title`
|
||||
2. **According to Teknium's design intent, what is the status of 'platform toolsets' in the codebase?**
|
||||
gold: `platform toolsets are vestigial, never exposed`
|
||||
3. **During the July sweep, which specific issue's config bridge was found to already exist at the exact line it was claimed to be missing?**
|
||||
gold: `#32263`
|
||||
4. **In the Aug 2026 cron-summarizer cluster sweep, which two PR numbers were discovered post-merge as the true first submitters?**
|
||||
gold: `#60593, #61969`
|
||||
5. **What is the recommended Git command to find when a specific symbol fix landed on the main branch?**
|
||||
gold: `git log -S "<symbol>"`
|
||||
6. **Why did the #39719 salvage silently delete 236 lines of code from cli-config.yaml.example?**
|
||||
gold: `the stale file version silently deletes newer main code`
|
||||
7. **What is the rule for salvaging commits with placeholder identities like 'pwn@example.com'?**
|
||||
gold: `do NOT cherry-pick. Surgical reapply as maintainer-authored commit, Co-authored-by the GitHub PR author`
|
||||
8. **How should an agent handle a 'gh pr merge' 502 error?**
|
||||
gold: `retry the same command once after the "Merge already in progress" settles (~45s); check PR state between attempts`
|
||||
9. **Which two properties shape almost every design decision in Hermes according to the Development Guide?**
|
||||
gold: `Per-conversation prompt caching is sacred and The core is a narrow waist; capability lives at the edges.`
|
||||
10. **What error message does the live-checkout git guard display when blocking a history-rewriting command?**
|
||||
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module `
|
||||
11. **What happened to the Desktop cluster's 'npm ci' command that resulted in an error writing to /tmp/ccH06T4r.s?**
|
||||
gold: `No space left on device`
|
||||
12. **What was the GraphQL API rate limit remaining for the user when the 'API rate limit already exceeded' error first occurred?**
|
||||
gold: `0`
|
||||
13. **Which PR was identified as the salvage of HexLab98's #85283 to fix hung inline API calls?**
|
||||
gold: `#86645`
|
||||
14. **Why did PR #79268 fix invisible overlays in the TUI?**
|
||||
gold: `renderNodeToOutput skips boxes Yoga squeezes to height 0`
|
||||
15. **What was the specific ModuleNotFoundError message caused by the wheel subpackage discovery trap in #34701?**
|
||||
gold: `ModuleNotFoundError: No module named 'hermes_cli.dashboard_auth'`
|
||||
|
||||
</details>
|
||||
|
||||
### Transcript: gui
|
||||
|
||||
| policy | recall | tokens before | tokens after | compress s |
|
||||
|---|---|---|---|---|
|
||||
| lean | 60.0% | 499,818 | 41,232 | 118.1 |
|
||||
| lean+recovery | 80.0% | 499,818 | 41,306 | 115.2 |
|
||||
|
||||
<details><summary>15 exam questions (questions-36d3d87e0b.json)</summary>
|
||||
|
||||
1. **What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?**
|
||||
gold: `#86617`
|
||||
2. **According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?**
|
||||
gold: `.env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.`
|
||||
3. **What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?**
|
||||
gold: `tests/hermes_cli/test_session_recovery_lost_and_found.py:327`
|
||||
4. **What was the root cause of issue #73793 regarding mid-turn message rendering?**
|
||||
gold: `redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row`
|
||||
5. **Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?**
|
||||
gold: `#84287`
|
||||
6. **In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?**
|
||||
gold: `pane sash grab band made asymmetric 1px/7px`
|
||||
7. **Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?**
|
||||
gold: `602028@ky-tech.com.cn`
|
||||
8. **What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?**
|
||||
gold: `Blocked: `git <op>` would rewrite Hermes's live source checkout`
|
||||
9. **What is the core design principle regarding 'Narrow Waist' in Hermes development?**
|
||||
gold: `The core is a narrow waist; capability lives at the edges.`
|
||||
10. **What was the result of the rebase-merge attempt for PR #86589?**
|
||||
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
|
||||
11. **In the infographic style picker, what vibe is associated with the 'designers-republic' style?**
|
||||
gold: `The Designers Republic: flat orange+violet vector schematic on pewter grey`
|
||||
12. **Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?**
|
||||
gold: `conflicts with main in 4 files and introduces a second competing display-dedupe scheme`
|
||||
13. **What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?**
|
||||
gold: `May 23 2026`
|
||||
14. **What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?**
|
||||
gold: `Property 'onToggleUnread' is missing in type`
|
||||
15. **According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?**
|
||||
gold: `Electron`
|
||||
|
||||
</details>
|
||||
|
||||
<details><summary>15 exam questions (questions-9c55c707b6.json)</summary>
|
||||
|
||||
1. **What two PR numbers are associated with the 'sidebar-nav-rows-and-overlay-panels.md' and 'hud-mode-internals.md' references in the initial tool content?**
|
||||
gold: `#85162 and #82285`
|
||||
2. **According to AGENTS.md, what is the 'one exception' to the rule that nothing should rebuild the system prompt mid-conversation?**
|
||||
gold: `context compression`
|
||||
3. **In the Contribution Rubric, what are the three allowed reasons for an automated triage sweeper to close a PR?**
|
||||
gold: `implemented_on_main, cannot_reproduce, incoherent`
|
||||
4. **Which contributor is credited with adding the 'Brazilian Portuguese localization' in PR #86292?**
|
||||
gold: `@gui8515`
|
||||
5. **What specific error message is reported in issue #83562 regarding the Windows Desktop update?**
|
||||
gold: `Hermes backend exited (0)`
|
||||
6. **What is the 'core problem' identified in the parallel-subagent-salvage-orchestration.md reference?**
|
||||
gold: `subagents share the parent's worktree + main checkout`
|
||||
7. **Why was the 'nix (macos-latest)' build failing in the salvage batches according to the orchestration reference?**
|
||||
gold: `Nix build failed due to stale npm lockfile hash`
|
||||
8. **Which subagent ID was assigned the goal of salvaging the 'inflight-journal duplicate-answer cluster'?**
|
||||
gold: `sa-2-7318d0ba`
|
||||
9. **In PR #86595, why was PR #80707 by upperagent excluded from the salvage?**
|
||||
gold: `violating this PR's UI-read-only invariant`
|
||||
10. **What was the root cause of the failure in Python tests slice 4/12 for PR #86597?**
|
||||
gold: `AssertionError: assert 't2' == 't1'`
|
||||
11. **What did the fix for issue #79157 in PR #86589 involve?**
|
||||
gold: `pane sash grab band made asymmetric 1px/7px`
|
||||
12. **According to the root cause analysis for #73793, which two files spliced the mid-turn user message at streamIndex?**
|
||||
gold: `use-prompt-actions/index.ts and session-tile-actions.ts`
|
||||
13. **What was the head SHA for the 'salvage/desktop-busy-state' branch in PR #86604?**
|
||||
gold: `bddadfe9e21e24b3d52e2b15f138c42474dede42`
|
||||
14. **Why was the merge of PR #86589 aborted during the 'Merge all' command?**
|
||||
gold: `GraphQL: Pull Request has merge conflicts (mergePullRequest)`
|
||||
15. **What specific file was modified to fix the 'artifacts page timestamps render 1970' issue via PR #86749?**
|
||||
gold: `apps/desktop/src/app/session/hooks/use-session-actions/utils.ts`
|
||||
|
||||
</details>
|
||||
|
||||
### Transcript: prmerge
|
||||
|
||||
| policy | recall | tokens before | tokens after | compress s |
|
||||
|---|---|---|---|---|
|
||||
| uncompacted_control | 96.7% | 499,663 | 499,663 | — |
|
||||
| current | 33.3% | 499,663 | 155,399 | 14.9 |
|
||||
| lean | 23.3% | 499,663 | 44,419 | 105.4 |
|
||||
| lean+recovery | 43.3% | 499,663 | 44,977 | 95.8 |
|
||||
|
||||
<details><summary>15 exam questions (questions-703ae2774a.json)</summary>
|
||||
|
||||
1. **Which PR number added the public subagent lifecycle API?**
|
||||
gold: `#63359`
|
||||
2. **What is the name of the typed service added to PluginContext for launching and monitoring child sessions?**
|
||||
gold: `subagent_lifecycle`
|
||||
3. **How many contract and security tests were included with the subagent lifecycle API PR?**
|
||||
gold: `42`
|
||||
4. **What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?**
|
||||
gold: `cannot currently trigger a turn in an existing gateway session`
|
||||
5. **Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?**
|
||||
gold: `#64436`
|
||||
6. **What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?**
|
||||
gold: `must compile, must start with ≥2 literal characters`
|
||||
7. **Which contributor authorized sustained help for the Phase 0–1 expansion track?**
|
||||
gold: `Daniel`
|
||||
8. **What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?**
|
||||
gold: `#64204`
|
||||
9. **What configuration setting is required to opt-in to reasoning deltas in streaming output?**
|
||||
gold: `plugins.stream_reasoning_deltas: true`
|
||||
10. **How many additions and across how many files were made in PR #63359?**
|
||||
gold: `650 additions across 4 files`
|
||||
11. **What is the name of the reference plugin shipped with the redaction pattern registry?**
|
||||
gold: `nvapi-redaction`
|
||||
12. **List the four observer-only streaming output plugin hooks added in PR #64317.**
|
||||
gold: `on_stream_start, on_stream_delta, on_stream_end, on_interim_message`
|
||||
13. **What was addressed in the update to PR #58541 regarding lifecycle hooks?**
|
||||
gold: `created-hook timing and added kanban_task_promoted`
|
||||
14. **Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?**
|
||||
gold: `#64230`
|
||||
15. **What was the Round 3 review's outcome for PR #63359 and @asimons81?**
|
||||
gold: `sub-issue #65447`
|
||||
|
||||
</details>
|
||||
|
||||
### Transcript: acp
|
||||
|
||||
| policy | recall | tokens before | tokens after | compress s |
|
||||
|---|---|---|---|---|
|
||||
| uncompacted_control | 100.0% | 498,906 | 498,906 | — |
|
||||
| current | 30.0% | 498,906 | 160,223 | 15.8 |
|
||||
| lean | 36.7% | 498,906 | 49,523 | 143.3 |
|
||||
| lean+recovery | 80.0% | 498,906 | 49,721 | 135.6 |
|
||||
|
||||
<details><summary>15 exam questions (questions-f45358df19.json)</summary>
|
||||
|
||||
1. **What was the specific reason Teknium gave for reverting PR #30179 in July 2026?**
|
||||
gold: `WTF??? REVERT! DAMMIT`
|
||||
2. **On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?**
|
||||
gold: `PR #6391`
|
||||
3. **Which file path should be checked for the canonical list of provider models?**
|
||||
gold: `hermes_cli/models.py`
|
||||
4. **What was the identified bug in PR #2314 regarding provider names?**
|
||||
gold: `checking for "alibaba-coding-plan"`
|
||||
5. **What is the mandatory line limit for PR reviews requested by Teknium?**
|
||||
gold: `<= 15 lines`
|
||||
6. **What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?**
|
||||
gold: `Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix mod`
|
||||
7. **Why was PR #74658 necessary to fix Slack 'broken on main'?**
|
||||
gold: `SlackResponse isn't a dict subclass, so every gate is always False.`
|
||||
8. **What was the final merge commit SHA for the Slack SDK response fix on main?**
|
||||
gold: `24ba86627515ad5fda69a39ef338c365713448bc`
|
||||
9. **In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?**
|
||||
gold: `Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1`
|
||||
10. **What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?**
|
||||
gold: `UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'`
|
||||
11. **Which test failed in CI slice 5/12 for the kanban delivery modes PR?**
|
||||
gold: `tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post`
|
||||
12. **According to the transcript, why is squash merging banned as of July 2026?**
|
||||
gold: `DevOps policy`
|
||||
13. **Which contributor authored the first fix for issue #73030 in July?**
|
||||
gold: `@Tranquil-Flow`
|
||||
14. **What was the 'Superman-style' shield error in the first generation of the Kanban infographic?**
|
||||
gold: `red "S" inside the diamond shield`
|
||||
15. **What specific file was modified to add the 'scope_id_for_chat' method for Slack?**
|
||||
gold: `plugins/platforms/slack/adapter.py`
|
||||
|
||||
</details>
|
||||
|
||||
## Methodology notes
|
||||
|
||||
- Transcripts: 4 real session lineages reconstructed from a state.db copy
|
||||
(sweep campaign 42 rotations / GUI desktop 34 / PR-merge 17 / ACP review
|
||||
17), chronological 500K-token prefix, tool-group aligned.
|
||||
- Question generation: main model, from the region the CURRENT policy would
|
||||
summarize (most conservative boundary), cached per transcript so every
|
||||
policy answers the identical exam.
|
||||
- Answering: fresh LLM sees ONLY the post-compaction context (closed-book) or
|
||||
context + one FTS5+BM25 search round-trip over the archived region
|
||||
(+recovery). Judge sees gold; answerer never does. Scoring 2/1/0.
|
||||
- Known caveats: 15 questions/transcript => +-1 question ~ 3.3pts noise;
|
||||
sweep/gui current-policy rows predate a question-bank regeneration
|
||||
(prmerge/acp are same-bank across all arms); the recovery sim conservatively
|
||||
approximates production session_search (same engine, no windowing).
|
||||
- Cost shape: lean compaction = ~25 aux-model digest calls (~2min, one-time
|
||||
per compaction) vs 1 call today; every post-compaction turn is ~110K input
|
||||
tokens cheaper. Break-even ~1 turn.
|
||||
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 36.7,
|
||||
"scores": [
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"read_seconds": 229,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 454931,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-35-51-01a00835-fef4-7c31-944d-9d0d8ba933be.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "What was the specific reason Teknium gave for reverting PR #30179 in July 2026?",
|
||||
"gold": "WTF??? REVERT! DAMMIT",
|
||||
"answer": "UNKNOWN \u2014 best guess: it caused a regression on current `main` and had to be reverted until a safer implementation was available.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a logical guess about a regression, but it fails to capture the specific, idiosyncratic reason/quote provided in the gold answer ('WTF??? REVERT! DAMMIT')."
|
||||
},
|
||||
{
|
||||
"q": "On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?",
|
||||
"gold": "PR #6391",
|
||||
"answer": "UNKNOWN \u2014 best guess: PR #37865.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold answer (PR #37865 vs PR #6391)."
|
||||
},
|
||||
{
|
||||
"q": "Which file path should be checked for the canonical list of provider models?",
|
||||
"gold": "hermes_cli/models.py",
|
||||
"answer": "UNKNOWN \u2014 best guess: `hermes_cli/models.py`.",
|
||||
"score": 2,
|
||||
"why": "The answer provides the correct file path as its best guess, which matches the gold answer exactly."
|
||||
},
|
||||
{
|
||||
"q": "What was the identified bug in PR #2314 regarding provider names?",
|
||||
"gold": "checking for \"alibaba-coding-plan\"",
|
||||
"answer": "UNKNOWN \u2014 best guess: it used inconsistent/noncanonical provider names, so provider matching and routing failed.",
|
||||
"score": 1,
|
||||
"why": "The answer identifies the core issue (provider name mismatch/inconsistency) which led to the specific bug, but it does not mention the specific string 'alibaba-coding-plan' identified in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What is the mandatory line limit for PR reviews requested by Teknium?",
|
||||
"gold": "<= 15 lines",
|
||||
"answer": "Maximum 5 lines.",
|
||||
"score": 0,
|
||||
"why": "The answer states a limit of 5 lines, which contradicts the gold answer of 15 lines."
|
||||
},
|
||||
{
|
||||
"q": "What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?",
|
||||
"gold": "Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module versions in this running process.",
|
||||
"answer": "UNKNOWN \u2014 best guess: `fatal: '<branch>' is already checked out at '<worktree path>'`.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a standard Git error message as a guess, which does not match the specific custom error message provided in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "Why was PR #74658 necessary to fix Slack 'broken on main'?",
|
||||
"gold": "SlackResponse isn't a dict subclass, so every gate is always False.",
|
||||
"answer": "Real Slack SDK calls return `SlackResponse` objects, but the main code treated them like dictionaries and called dictionary-style accessors.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies that SlackResponse objects were being treated as dictionaries when they are not, which matches the core issue described in the gold answer that their lack of dictionary subclassing caused attribute/gate checks to fail."
|
||||
},
|
||||
{
|
||||
"q": "What was the final merge commit SHA for the Slack SDK response fix on main?",
|
||||
"gold": "24ba86627515ad5fda69a39ef338c365713448bc",
|
||||
"answer": "UNKNOWN.",
|
||||
"score": 0,
|
||||
"why": "The gold answer provides a specific commit SHA, while the answer states 'UNKNOWN', failing to provide the requested factual information."
|
||||
},
|
||||
{
|
||||
"q": "In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?",
|
||||
"gold": "Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1",
|
||||
"answer": "UNKNOWN.",
|
||||
"score": 0,
|
||||
"why": "The answer 'UNKNOWN' is a refusal to provide the information required by the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?",
|
||||
"gold": "UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'",
|
||||
"answer": "`UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'` \u2014 applied only during first migration/addition of the column so existing choices were not overwritten.",
|
||||
"score": 2,
|
||||
"why": "The answer provides the exact SQL statement specified in the gold answer and correctly contextualizes its purpose."
|
||||
},
|
||||
{
|
||||
"q": "Which test failed in CI slice 5/12 for the kanban delivery modes PR?",
|
||||
"gold": "tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post",
|
||||
"answer": "UNKNOWN \u2014 best guess: a relay-metrics concurrency test, unrelated to the delivery-mode change.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies an unrelated test (relay-metrics) rather than the specific kanban notifier test mentioned in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "According to the transcript, why is squash merging banned as of July 2026?",
|
||||
"gold": "DevOps policy",
|
||||
"answer": "Squash merging destroys the contributor-authored commit history and therefore loses reliable authorship/credit.",
|
||||
"score": 0,
|
||||
"why": "The answer explains a general rationale for why someone might dislike squash merging, but it fails to provide the specific reason cited in the text ('DevOps policy')."
|
||||
},
|
||||
{
|
||||
"q": "Which contributor authored the first fix for issue #73030 in July?",
|
||||
"gold": "@Tranquil-Flow",
|
||||
"answer": "UNKNOWN.",
|
||||
"score": 0,
|
||||
"why": "The answer states 'UNKNOWN' while the gold answer provides a specific contributor name (@Tranquil-Flow)."
|
||||
},
|
||||
{
|
||||
"q": "What was the 'Superman-style' shield error in the first generation of the Kanban infographic?",
|
||||
"gold": "red \"S\" inside the diamond shield",
|
||||
"answer": "The generated shield contained an emblem resembling Superman\u2019s \u201cS,\u201d rather than a neutral Kanban/routing symbol.",
|
||||
"score": 2,
|
||||
"why": "The answer accurately identifies that the shield contained a symbol resembling Superman's 'S', which matches the core fact of the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What specific file was modified to add the 'scope_id_for_chat' method for Slack?",
|
||||
"gold": "plugins/platforms/slack/adapter.py",
|
||||
"answer": "`plugins/platforms/slack/adapter.py`.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold answer exactly, including the full file path."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 40.0,
|
||||
"scores": [
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"read_seconds": 207,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 482525,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-28-36-01a0082f-5b69-7ff0-977d-998523a4b3fc.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?",
|
||||
"gold": "#86617",
|
||||
"answer": "PR **#86617**.",
|
||||
"score": 2,
|
||||
"why": "The answer provides the exact PR number specified in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?",
|
||||
"gold": ".env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.",
|
||||
"answer": "`.env` is for secrets and deployment-specific values; `config.yaml` is for non-secret user-facing runtime configuration.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies that .env is for secrets and config.yaml is for non-secret/behavioral settings, which matches the core distinction in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?",
|
||||
"gold": "tests/hermes_cli/test_session_recovery_lost_and_found.py:327",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: a schema-width assertion in a Python schema/config test, expecting 55 fields but receiving 56 after `git_metadata_generation` was added.",
|
||||
"score": 0,
|
||||
"why": "The answer fails to provide the specific file and line number requested, which are present in the gold answer. It instead provides a guess about the context of the error without identifying the location."
|
||||
},
|
||||
{
|
||||
"q": "What was the root cause of issue #73793 regarding mid-turn message rendering?",
|
||||
"gold": "redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row",
|
||||
"answer": "Two live paths handled mid-turn corrections independently, allowing corrections and assistant stream output to be inserted in the wrong causal order.",
|
||||
"score": 2,
|
||||
"why": "The answer identifies the same root cause as the gold answer: the independent handling of paths caused the user bubble (correction) and assistant stream to be rendered in the wrong causal order."
|
||||
},
|
||||
{
|
||||
"q": "Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?",
|
||||
"gold": "#84287",
|
||||
"answer": "**PR #86299**.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies PR #86299, but the gold answer is PR #84287. The two are different pull requests."
|
||||
},
|
||||
{
|
||||
"q": "In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?",
|
||||
"gold": "pane sash grab band made asymmetric 1px/7px",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: make the scrollbar gutter/overlay accept pointer events instead of letting the virtualized content layer cover it.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a guess that contradicts the gold answer; the fix was making the pane sash grab band asymmetric (1px/7px), not changing pointer events on the gutter/overlay."
|
||||
},
|
||||
{
|
||||
"q": "Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?",
|
||||
"gold": "602028@ky-tech.com.cn",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: the email associated with contributor `hkfiberlaser-svg`.",
|
||||
"score": 0,
|
||||
"why": "The answer states the answer is unknown and provides a guess that does not match the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?",
|
||||
"gold": "Blocked: `git <op>` would rewrite Hermes's live source checkout",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: an error stating that the git command was blocked because it would rewrite the live source checkout.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies the core substance of the error message (blocked git command because it would rewrite the live source checkout) which matches the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What is the core design principle regarding 'Narrow Waist' in Hermes development?",
|
||||
"gold": "The core is a narrow waist; capability lives at the edges.",
|
||||
"answer": "Keep a small, stable core interface\u2014the \u201cnarrow waist\u201d\u2014through which different clients, providers, and integrations communicate, avoiding duplicated business logic across surfaces.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies the core principle of a 'narrow waist' and explains that the logic/interfaces are centralized there while clients and providers interact at the edges, matching the substance of the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What was the result of the rebase-merge attempt for PR #86589?",
|
||||
"gold": "GraphQL: Pull Request has merge conflicts (mergePullRequest)",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: the rebase merge could not proceed because the PR had conflicts and was not mergeable.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies that the rebase-merge could not proceed due to merge conflicts, which matches the factual content of the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "In the infographic style picker, what vibe is associated with the 'designers-republic' style?",
|
||||
"gold": "The Designers Republic: flat orange+violet vector schematic on pewter grey",
|
||||
"answer": "A bold, experimental, anti-corporate graphic-design vibe\u2014brutalist/techno typography inspired by The Designers Republic.",
|
||||
"score": 0,
|
||||
"why": "The answer describes the real-world aesthetic of 'The Designers Republic' studio, but fails to identify the specific visual elements (flat orange+violet vector schematic on pewter grey) defined for this style within the infographic picker context."
|
||||
},
|
||||
{
|
||||
"q": "Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?",
|
||||
"gold": "conflicts with main in 4 files and introduces a second competing display-dedupe scheme",
|
||||
"answer": "It did not address transcript visibility after compaction/compression; its changes concerned a different part of the compaction flow.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a completely different reason than the gold answer, which specifies merge conflicts and a competing deduplication scheme as the reasons for exclusion."
|
||||
},
|
||||
{
|
||||
"q": "What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?",
|
||||
"gold": "May 23 2026",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: **2026-08-12**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold date (2026-08-12 vs. May 23 2026)."
|
||||
},
|
||||
{
|
||||
"q": "What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?",
|
||||
"gold": "Property 'onToggleUnread' is missing in type",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: a TypeScript unresolved-name/import error introduced by the rebase.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies as UNKNOWN and provides an incorrect guess. The gold answer specifies a 'property is missing' error, while the provided answer guesses an 'unresolved-name/import' error."
|
||||
},
|
||||
{
|
||||
"q": "According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?",
|
||||
"gold": "Electron",
|
||||
"answer": "The **Rust/Tauri backend** is authoritative for process lifecycle and the native filesystem.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies the Rust/Tauri backend as the authority, which directly contradicts the gold answer stating Electron is the authority."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 43.3,
|
||||
"scores": [
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"read_seconds": 154,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 465131,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-32-45-01a00833-2968-7453-8646-d5bb7f5e8400.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "Which PR number added the public subagent lifecycle API?",
|
||||
"gold": "#63359",
|
||||
"answer": "PR **#63359**.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold PR number exactly."
|
||||
},
|
||||
{
|
||||
"q": "What is the name of the typed service added to PluginContext for launching and monitoring child sessions?",
|
||||
"gold": "subagent_lifecycle",
|
||||
"answer": "`SubagentService` (`ctx.subagents`).",
|
||||
"score": 2,
|
||||
"why": "The answer identifies the correct property on the context (ctx.subagents), which is the typed service used for the task, even though it uses a slightly different name for the service itself."
|
||||
},
|
||||
{
|
||||
"q": "How many contract and security tests were included with the subagent lifecycle API PR?",
|
||||
"gold": "42",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: **24 tests**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides an incorrect guess (24) compared to the gold answer (42) and frames the response as unknown."
|
||||
},
|
||||
{
|
||||
"q": "What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?",
|
||||
"gold": "cannot currently trigger a turn in an existing gateway session",
|
||||
"answer": "`ctx.inject_message()` could not safely target an active gateway session; it lacked a profile/session-scoped `session_key`.",
|
||||
"score": 2,
|
||||
"why": "The answer identifies the core limitation (targeting active gateway sessions) and provides technical detail on why a turn cannot be triggered (lack of session-scoped key)."
|
||||
},
|
||||
{
|
||||
"q": "Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?",
|
||||
"gold": "#64436",
|
||||
"answer": "PR **#64436**.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold answer exactly."
|
||||
},
|
||||
{
|
||||
"q": "What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?",
|
||||
"gold": "must compile, must start with \u22652 literal characters",
|
||||
"answer": "Patterns must be **bounded in length** and **validated against unsafe/catastrophic regex behavior** before registration.",
|
||||
"score": 0,
|
||||
"why": "The answer discusses general regex safety (length and performance), whereas the gold answer specifies two precise technical constraints: compilation and a minimum of two literal characters at the start."
|
||||
},
|
||||
{
|
||||
"q": "Which contributor authorized sustained help for the Phase 0\u20131 expansion track?",
|
||||
"gold": "Daniel",
|
||||
"answer": "**@asimons81**.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies @asimons81 as the contributor, whereas the gold answer states it was Daniel. These do not match."
|
||||
},
|
||||
{
|
||||
"q": "What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?",
|
||||
"gold": "#64204",
|
||||
"answer": "Issue **#64204**.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold issue number exactly."
|
||||
},
|
||||
{
|
||||
"q": "What configuration setting is required to opt-in to reasoning deltas in streaming output?",
|
||||
"gold": "plugins.stream_reasoning_deltas: true",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: `display.show_reasoning: true`.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies itself as a guess and provides the incorrect configuration setting ('display.show_reasoning' vs the gold 'plugins.stream_reasoning_deltas')."
|
||||
},
|
||||
{
|
||||
"q": "How many additions and across how many files were made in PR #63359?",
|
||||
"gold": "650 additions across 4 files",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: roughly **1,000 additions across 12 files**.",
|
||||
"score": 0,
|
||||
"why": "The answer states the information is unknown and provides a guess that is factually incorrect regarding both the number of additions and the number of files."
|
||||
},
|
||||
{
|
||||
"q": "What is the name of the reference plugin shipped with the redaction pattern registry?",
|
||||
"gold": "nvapi-redaction",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: `redaction-patterns`.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies itself as unknown and provides a guess ('redaction-patterns') that is incorrect compared to the gold answer ('nvapi-redaction')."
|
||||
},
|
||||
{
|
||||
"q": "List the four observer-only streaming output plugin hooks added in PR #64317.",
|
||||
"gold": "on_stream_start, on_stream_delta, on_stream_end, on_interim_message",
|
||||
"answer": "`on_stream_start`, `on_stream_delta`, `on_stream_end`, and `on_interim_message`.",
|
||||
"score": 2,
|
||||
"why": "The answer lists all four hooks correctly as specified in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What was addressed in the update to PR #58541 regarding lifecycle hooks?",
|
||||
"gold": "created-hook timing and added kanban_task_promoted",
|
||||
"answer": "It updated the lifecycle hooks to match the current registry contract, including correct hook timing, payloads, and failure isolation.",
|
||||
"score": 1,
|
||||
"why": "The answer correctly identifies 'hook timing' as part of the update, but it fails to mention the specific addition of 'kanban_task_promoted' and instead provides a general description of other improvements like payloads and isolation not mentioned in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?",
|
||||
"gold": "#64230",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: **#64179**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides an incorrect issue number (#64179) while the gold answer is #64230."
|
||||
},
|
||||
{
|
||||
"q": "What was the Round 3 review's outcome for PR #63359 and @asimons81?",
|
||||
"gold": "sub-issue #65447",
|
||||
"answer": "Round 3 **approved PR #63359 and authorized @asimons81 to continue sustained Phase 0\u20131 work**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a general description of approval and authorization, but fails to mention the specific outcome required by the gold answer (the creation or reference of sub-issue #65447)."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 26.7,
|
||||
"scores": [
|
||||
2,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"read_seconds": 420,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 482248,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-20-54-01a00828-4e75-70f0-9e6d-31c7aac8d71c.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?",
|
||||
"gold": "the stale file version silently deletes newer main code",
|
||||
"answer": "It can silently replace current files with stale branch versions, discarding newer fixes and resurrecting deleted code; salvage must use commits and resolve conflicts explicitly.",
|
||||
"score": 2,
|
||||
"why": "The answer accurately reflects the gold answer's point that stale branch versions will overwrite newer code, while adding specific detail about resurrecting deleted code."
|
||||
},
|
||||
{
|
||||
"q": "According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?",
|
||||
"gold": "~60GB",
|
||||
"answer": "UNKNOWN \u2014 best guess: approximately 4 GB RSS.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a 'best guess' of 4 GB, which is factually incorrect and contradicts the gold answer of 60 GB."
|
||||
},
|
||||
{
|
||||
"q": "Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?",
|
||||
"gold": "backgroundThrottling",
|
||||
"answer": "UNKNOWN \u2014 best guess: Electron\u2019s `backgroundThrottling: false`.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies the specific setting 'backgroundThrottling' as the suspected cause, despite the initial 'UNKNOWN' disclaimer."
|
||||
},
|
||||
{
|
||||
"q": "What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?",
|
||||
"gold": "Auto merge is not allowed for this repository (enablePullRequestAutoMerge)",
|
||||
"answer": "`GraphQL: Pull request Auto merge is not allowed for this repository`",
|
||||
"score": 2,
|
||||
"why": "The answer matches the core message of the gold answer. The 'GraphQL:' prefix accurately reflects the technical origin of the error message when using the GitHub CLI."
|
||||
},
|
||||
{
|
||||
"q": "What is the specified 'Rule 0' that must be included in a subagent brief?",
|
||||
"gold": "load the skill first",
|
||||
"answer": "Rule 0: do not merge anything; only the primary agent/user is authorized to merge.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies a different 'Rule 0' regarding merging permissions, which does not match the gold answer's requirement to 'load the skill first'."
|
||||
},
|
||||
{
|
||||
"q": "In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?",
|
||||
"gold": "add config gate for title generation",
|
||||
"answer": "UNKNOWN \u2014 best guess: \u201cfix: prevent duplicate message submission.\u201d",
|
||||
"score": 0,
|
||||
"why": "The answer provides an incorrect guess ('fix: prevent duplicate message submission.') which does not match the gold answer ('add config gate for title generation')."
|
||||
},
|
||||
{
|
||||
"q": "Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?",
|
||||
"gold": "tests/test_packaging_metadata.py",
|
||||
"answer": "UNKNOWN \u2014 best guess: `tests/test_plugin_packaging.py`.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies the wrong file (test_plugin_packaging.py vs the correct test_packaging_metadata.py)."
|
||||
},
|
||||
{
|
||||
"q": "What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?",
|
||||
"gold": "completed normally (exit code 0)",
|
||||
"answer": "`npm ci` completed successfully with exit code 0.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold answer perfectly, confirming the successful completion and the exit code of 0."
|
||||
},
|
||||
{
|
||||
"q": "What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?",
|
||||
"gold": "relative exclude-newer makes the committed lock stale every day",
|
||||
"answer": "Root Cause A: the `all` extra referenced platform-specific/nonexistent dependency groups that were absent or inconsistent in the locked resolution.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies an issue with dependency groups, whereas the gold answer states the root cause was the use of a relative 'exclude-newer' value that caused the lockfile to become stale daily."
|
||||
},
|
||||
{
|
||||
"q": "How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?",
|
||||
"gold": "13",
|
||||
"answer": "UNKNOWN \u2014 best guess: 7 tasks done.",
|
||||
"score": 0,
|
||||
"why": "The answer guesses 7 tasks, while the gold answer is 13. The answer is factually incorrect."
|
||||
},
|
||||
{
|
||||
"q": "In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?",
|
||||
"gold": "-1",
|
||||
"answer": "Exit code 2.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a specific exit code (2) that contradicts the gold answer (-1)."
|
||||
},
|
||||
{
|
||||
"q": "What is the full title block text for the technical schematic infographic generated for the Gateway Drain?",
|
||||
"gold": "GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT",
|
||||
"answer": "UNKNOWN \u2014 best guess: `GATEWAY DRAIN` / `INTERRUPT \u2192 DELIVER \u2192 DISCONNECT` / `GRACEFUL SHUTDOWN WITHOUT MESSAGE LOSS`.",
|
||||
"score": 0,
|
||||
"why": "The answer fails to provide the correct title block text specified in the gold answer. It incorrectly guesses metadata or slogans instead of the specific title 'GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT'."
|
||||
},
|
||||
{
|
||||
"q": "Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?",
|
||||
"gold": "82980",
|
||||
"answer": "UNKNOWN \u2014 best guess: PR #86594.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies the wrong PR number (86594 instead of 82980)."
|
||||
},
|
||||
{
|
||||
"q": "What is the specific Gist ID created for the PR infographic host in the cron cluster?",
|
||||
"gold": "ee33edd5804689243f974536ef7aecb9",
|
||||
"answer": "UNKNOWN \u2014 I do not retain the Gist ID.",
|
||||
"score": 0,
|
||||
"why": "The answer states it does not know the information, whereas the gold answer provides a specific Gist ID."
|
||||
},
|
||||
{
|
||||
"q": "What was the final merge SHA for Cluster D's Trigger-now PR #70638?",
|
||||
"gold": "f9d64b9a9d8b306f64851c1a13869d96ad5d7869",
|
||||
"answer": "UNKNOWN \u2014 I do not retain the final merge SHA for PR #70638.",
|
||||
"score": 0,
|
||||
"why": "The answer claims it does not know the information, while the gold answer provides the specific SHA requested."
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user