yilsem/aiturk-hermes-ide
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 6 Wiki Activity

Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license

Browse Source
This commit is contained in:
yilsem
2026-09-05 13:26:46 +03:00
commit 03634b1ca3
11340 changed files with 3442369 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evals/browser_use/README.md
+114
View File
@@ -0,0 +1,114 @@
# Browser Use Mode Benchmark
The A/B battery behind PR [#81958](https://github.com/NousResearch/hermes-agent/pull/81958)
(Browser Use CLI 3.0 mode, salvage of #66476 by @laithrw): built-in
`browser_*` toolset vs the single `browser_exec` driver, measured as total
task tokens / tool calls / wall clock at accuracy parity on live multi-step
web tasks.
## Design
- **Arms differ only by tree + config.** `base` runs the built-in twelve
`browser_*` tools from a merge-base checkout; `pr` runs `browser_exec`
(`browser.backend: browser-use`) from the branch checkout; `prns` is `pr`
with the schema's helpers digest stripped to the header (isolates the
digest's value). Each cell gets a throwaway `HERMES_HOME`; web-fetch
credentials are stripped so every arm must actually drive the browser.
- **Tasks are oracle-checked.** toscrape-family sites (stable content, no
anti-bot), regex oracles over the final answer. `tasks/easy.json` (5 tasks:
price lookup, category extract, count/aggregate, login, pagination) and
`tasks/hard.json` (6 tasks: full-category multi-page crawls, five-star
rating aggregation, JS/delayed render, login chain, cross-category
compare).
- **Resume-safe.** Completed cells in `results/*.jsonl` are skipped on rerun
(same pattern as `scripts/toolperf_abeval`).
- **Backend matrix.** `orchestrate.py` drives a local headless-Chrome CDP;
`orchestrate_cloud.py --backend nous-cloud|browserbase` provisions a real
cloud browser per cell through the same provider plumbing the product uses.
## Run
```bash
# arms are pinned checkouts — e.g. merge-base worktree vs your branch
export BUBENCH_BASE_TREE=/path/to/merge-base-tree
export BUBENCH_PR_TREE=/path/to/branch-tree
```
Note: since #81958 merged (and #85170 made Browser Use the default driver),
a current-main checkout resolves to `browser_exec` in BOTH arms. The `base`
arm only measures the built-in `browser_*` toolset when `BUBENCH_BASE_TREE`
is pinned to a pre-#81958 tree (the original run used the PR's merge-base
worktree). For future A/Bs of new browser changes, pin `base` to the
merge-base of the change under test — the arms are generic.
```bash
google-chrome --headless=new --remote-debugging-port=9333 \
--user-data-dir=/tmp/bubench-chrome --no-first-run --disable-gpu about:blank &
python3 orchestrate.py --tasks tasks/hard.json --reps 3 # 108 cells @ 2 models x 3 arms
python3 report.py results/results.jsonl
```
## Baseline scorecard (Aug 8-10 2026, the #81958 run — 204 cells total)
**Hard-task battery, local Chrome CDP** (6 tasks x 3 reps per cell; final
corrected-oracle readout, nothing excluded):
```
model arm ok tok_mean tok_med calls wall_s vs base tok
opus4.8 base 18/18 64594 63776 4.1 25.2 —
opus4.8 pr 18/18 25934 25030 2.0 17.5 -60%
opus4.8 prns 18/18 25578 27934 3.2 23.7 -60%
kimi-k3 base 18/18 56464 53276 5.3 50.0 —
kimi-k3 pr 18/18 19230 16710 2.4 33.3 -66%
kimi-k3 prns 18/18 23099 21160 4.1 50.5 -59%
```
Digest ablation: pr (with helpers digest) 36/36 ok, mean 22,582 tok; prns
(header-only) 36/36 ok, mean 24,339 tok — the pinned 3.4KB digest costs
nothing and saves a little; the full 11KB live skill dump adds nothing.
**Backend matrix** (pr arm, same tasks):
```
model backend ok tok_mean calls wall
opus4.8 local-cdp 17/18 25934 2.0 17.5
opus4.8 nous-cloud 12/12 33330 2.8 33.8
opus4.8 browserbase 6/6 26712 2.2 23.2
kimi-k3 local-cdp 18/18 19230 2.4 33.3
kimi-k3 nous-cloud 12/12 22050 2.9 41.4
kimi-k3 browserbase 6/6 22121 2.8 35.2
```
**Easy battery, round 1** (5 tasks x 3 reps, sonnet-5 + qwen3-coder-30b;
after excluding provider-noise runs — raw chat-template XML, 0 tool calls):
```
model arm ok prompt compl total calls wall_s
claude-sonnet-5 base 15/15 39771 324 40095 2.7 16.5
claude-sonnet-5 pr 15/15 27482 509 27991 2.4 14.3
qwen3-coder-30b base 13/14 59509 559 60068 5.7 21.5
qwen3-coder-30b pr 10/11 57146 1616 58763 6.8 26.3
```
sonnet-5: −30% tokens at parity. qwen3-30b: a wash — weak coders burn the
savings retrying exec code. The token win concentrates on multi-step tasks
and grows with task hardness; strong models also finish in fewer tool calls.
Compatibility probes from the same run: Firecrawl cloud browsers attach fine
(CDP websocket); Camofox has no CDP surface — structurally incompatible,
hence the automatic fallback to the built-in toolset in #81958.
Caveats: toscrape-family sites (no anti-bot, no heavy SPA); n<=3 per cell;
success-rate deltas at this n are noise — audit sub-100% cells run-by-run
before calling a regression.
## Provenance
The original per-run `results*.jsonl` files lived in `/tmp/bu-bench/` (tmpfs)
and were lost in a host reboot on Aug 12 2026. The harness, task definitions,
and aggregate readouts in this directory were recovered verbatim from the
session transcripts of the benchmark run (session `20260808_050008_5f615e`
tool-call history); `single_run.py`/`orchestrate*.py` are the recovered
scripts with the hardcoded `/tmp/bu-bench` paths parameterized. Rerunning the
battery reproduces fresh per-run data.
evals/browser_use/orchestrate.py
+129
View File
@@ -0,0 +1,129 @@
"""Local-CDP battery orchestrator: tasks x arms x models x reps.
Resume-safe: completed cells in results.jsonl are skipped, so a killed
battery continues where it left off (same pattern as scripts/toolperf_abeval).
Usage:
# start a headless Chrome first:
# google-chrome --headless=new --remote-debugging-port=9333 \
# --user-data-dir=/tmp/bubench-chrome --no-first-run --disable-gpu about:blank
BUBENCH_BASE_TREE=... BUBENCH_PR_TREE=... BENCH_CDP_URL=http://127.0.0.1:9333 \
python3 orchestrate.py [--tasks tasks/hard.json] [--models m1,m2] \
[--arms base,pr,prns] [--reps 3]
"""
import argparse
import itertools
import json
import os
import subprocess
import sys
import time
ROOT = os.environ.get("BUBENCH_ROOT", os.path.dirname(os.path.abspath(__file__)))
PY = sys.executable
ENV = {**os.environ}
ENV["PATH"] = os.path.expanduser("~/.local/bin") + os.pathsep + ENV.get("PATH", "")
parser = argparse.ArgumentParser()
parser.add_argument("--tasks", default=os.path.join(ROOT, "tasks", "hard.json"))
parser.add_argument("--models", default="anthropic/claude-opus-4.8,moonshotai/kimi-k3")
parser.add_argument("--arms", default="base,pr,prns")
parser.add_argument("--reps", type=int, default=3)
parser.add_argument("--results", default=os.path.join(ROOT, "results", "results.jsonl"))
parser.add_argument("--run-timeout", type=int, default=1200)
args = parser.parse_args()
os.makedirs(os.path.dirname(args.results), exist_ok=True)
ARMS = args.arms.split(",")
MODELS = args.models.split(",")
TASKS = list(json.load(open(args.tasks, encoding="utf-8")).keys())
REPS = list(range(1, args.reps + 1))
done = set()
if os.path.exists(args.results):
for line in open(args.results, encoding="utf-8"):
try:
r = json.loads(line)
done.add((r["arm"], r["task"], r["model"], r["rep"]))
except Exception:
pass
def reset_browser_state():
"""Kill lingering drivers and clear cookies between cells."""
if sys.platform == "win32":
subprocess.run(
["taskkill", "/F", "/IM", "agent-browser.exe", "/T"],
capture_output=True,
)
else:
subprocess.run(["pkill", "-f", "agent-browser"], capture_output=True)
code = "cdp('Network.clearBrowserCookies')\nprint('cleared')\n"
try:
subprocess.run(
["browser-use"],
input=code,
text=True,
capture_output=True,
timeout=120,
env=ENV,
)
except Exception:
pass
cells = [
(arm, task, model, rep)
for model, task, rep, arm in itertools.product(MODELS, TASKS, REPS, ARMS)
]
total = len(cells)
n = 0
for arm, task, model, rep in cells:
n += 1
if (arm, task, model, rep) in done:
continue
print(f"[{n}/{total}] {arm} {task} {model} rep{rep}", flush=True)
reset_browser_state()
t0 = time.time()
try:
proc = subprocess.run(
[PY, os.path.join(ROOT, "single_run.py"), arm, task, model, str(rep)],
capture_output=True,
text=True,
timeout=args.run_timeout,
env={**ENV, "BUBENCH_TASKS": args.tasks},
)
rec = None
for line in (proc.stdout or "").splitlines():
if line.startswith("RESULT_JSON:"):
rec = json.loads(line[len("RESULT_JSON:") :])
if rec is None:
rec = {
"arm": arm,
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": "no-result",
"stderr_tail": (proc.stderr or "")[-800:],
"stdout_tail": (proc.stdout or "")[-400:],
}
except subprocess.TimeoutExpired:
rec = {
"arm": arm,
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": f"orchestrator-timeout-{args.run_timeout}s",
}
rec["cell_wall_s"] = round(time.time() - t0, 1)
with open(args.results, "a", encoding="utf-8") as f:
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
print(
f" -> ok={rec.get('ok')} err={rec.get('error')} wall={rec.get('cell_wall_s')}s",
flush=True,
)
print("BATTERY COMPLETE", flush=True)
evals/browser_use/orchestrate_cloud.py
+184
View File
@@ -0,0 +1,184 @@
"""Cloud-backend battery: pr arm attached to a provisioned cloud browser per cell.
Backends:
nous-cloud - Nous Portal-provisioned Browser Use cloud browser
(plugins.browser.browser_use provider; needs gateway access)
browserbase - fresh Browserbase session per cell
(needs BROWSERBASE_API_KEY + BROWSERBASE_PROJECT_ID)
Usage:
BUBENCH_BASE_TREE=... BUBENCH_PR_TREE=... \
python3 orchestrate_cloud.py --backend nous-cloud [--reps 2]
python3 orchestrate_cloud.py --backend browserbase [--reps 1]
Per cell: provision a session, export its CDP endpoint via BENCH_CDP_URL /
BU_CDP_WS, run single_run.py (pr arm), close the session. Resume-safe.
"""
import argparse
import itertools
import json
import os
import subprocess
import sys
import time
import urllib.request
ROOT = os.environ.get("BUBENCH_ROOT", os.path.dirname(os.path.abspath(__file__)))
PY = sys.executable
ENV_BASE = {**os.environ}
ENV_BASE["PATH"] = (
os.path.expanduser("~/.local/bin") + os.pathsep + ENV_BASE.get("PATH", "")
)
parser = argparse.ArgumentParser()
parser.add_argument("--backend", required=True, choices=["nous-cloud", "browserbase"])
parser.add_argument("--tasks", default=os.path.join(ROOT, "tasks", "hard.json"))
parser.add_argument("--models", default="anthropic/claude-opus-4.8,moonshotai/kimi-k3")
parser.add_argument("--reps", type=int, default=2)
parser.add_argument("--results", default=None)
parser.add_argument("--run-timeout", type=int, default=1200)
args = parser.parse_args()
RESULTS = args.results or os.path.join(ROOT, "results", f"results_{args.backend}.jsonl")
os.makedirs(os.path.dirname(RESULTS), exist_ok=True)
MODELS = args.models.split(",")
TASKS = list(json.load(open(args.tasks, encoding="utf-8")).keys())
REPS = list(range(1, args.reps + 1))
done = set()
if os.path.exists(RESULTS):
for line in open(RESULTS, encoding="utf-8"):
try:
r = json.loads(line)
done.add((r["task"], r["model"], r["rep"]))
except Exception:
pass
class NousCloud:
def __init__(self):
sys.path.insert(0, os.environ["BUBENCH_PR_TREE"])
import importlib
self._mod = importlib.import_module("plugins.browser.browser_use.provider")
def create(self, name):
self._prov = self._mod.BrowserUseBrowserProvider()
sess = self._prov.create_session(name)
return sess, {"BENCH_CDP_URL": sess["cdp_url"]}
def close(self, sess):
self._prov.close_session(
sess.get("bb_session_id") or sess.get("session_name", "")
)
class Browserbase:
def create(self, name):
req = urllib.request.Request(
"https://api.browserbase.com/v1/sessions",
data=json.dumps({
"projectId": os.environ["BROWSERBASE_PROJECT_ID"]
}).encode(),
headers={
"x-bb-api-key": os.environ["BROWSERBASE_API_KEY"],
"Content-Type": "application/json",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=30) as resp:
sess = json.load(resp)
return sess, {
"BU_CDP_WS": sess["connectUrl"],
"BENCH_CDP_URL": sess["connectUrl"],
}
def close(self, sess):
req = urllib.request.Request(
f"https://api.browserbase.com/v1/sessions/{sess['id']}",
data=json.dumps({
"projectId": os.environ["BROWSERBASE_PROJECT_ID"],
"status": "REQUEST_RELEASE",
}).encode(),
headers={
"x-bb-api-key": os.environ["BROWSERBASE_API_KEY"],
"Content-Type": "application/json",
},
method="POST",
)
urllib.request.urlopen(req, timeout=30)
provider = NousCloud() if args.backend == "nous-cloud" else Browserbase()
cells = [(t, m, rep) for m, t, rep in itertools.product(MODELS, TASKS, REPS)]
total = len(cells)
n = 0
for task, model, rep in cells:
n += 1
if (task, model, rep) in done:
continue
print(f"[{n}/{total}] {args.backend} pr {task} {model} rep{rep}", flush=True)
sess = None
t0 = time.time()
try:
sess, extra_env = provider.create(f"bubench-{task}-{rep}")
except Exception as e: # noqa: BLE001
rec = {
"arm": f"pr-{args.backend}",
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": f"session-create: {e}",
}
with open(RESULTS, "a", encoding="utf-8") as f:
f.write(json.dumps(rec) + "\n")
continue
env = {**ENV_BASE, **extra_env, "BUBENCH_TASKS": args.tasks}
subprocess.run(["pkill", "-f", "browser_harness"], capture_output=True)
try:
proc = subprocess.run(
[PY, os.path.join(ROOT, "single_run.py"), "pr", task, model, str(rep)],
capture_output=True,
text=True,
timeout=args.run_timeout,
env=env,
)
rec = None
for line in (proc.stdout or "").splitlines():
if line.startswith("RESULT_JSON:"):
rec = json.loads(line[len("RESULT_JSON:") :])
if rec is None:
rec = {
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": "no-result",
"stderr_tail": (proc.stderr or "")[-600:],
}
except subprocess.TimeoutExpired:
rec = {
"task": task,
"model": model,
"rep": rep,
"ok": False,
"error": f"timeout-{args.run_timeout}s",
}
finally:
try:
provider.close(sess)
except Exception as e: # noqa: BLE001
print(f" close warning: {e}", flush=True)
rec["arm"] = f"pr-{args.backend}"
rec["cell_wall_s"] = round(time.time() - t0, 1)
with open(RESULTS, "a", encoding="utf-8") as f:
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
print(
f" -> ok={rec.get('ok')} err={rec.get('error')} wall={rec.get('cell_wall_s')}s",
flush=True,
)
print("CLOUD BATTERY COMPLETE", flush=True)
evals/browser_use/report.py
+70
View File
@@ -0,0 +1,70 @@
"""Aggregate results.jsonl into the scorecard tables.
Usage:
python3 report.py [results/results.jsonl ...]
Groups by (model, arm): ok-rate, token mean/median, tool calls, wall clock,
and token delta vs the ``base`` arm of the same model when present.
"""
import json
import statistics
import sys
from collections import defaultdict
def main(paths):
rows = []
for p in paths:
for line in open(p, encoding="utf-8"):
try:
rows.append(json.loads(line))
except Exception:
pass
if not rows:
print("no rows")
return
cells = defaultdict(list)
for r in rows:
cells[(r.get("model", "?"), r.get("arm", "?"))].append(r)
base_tok = {}
for (model, arm), rs in cells.items():
if arm == "base":
oks = [r for r in rs if r.get("ok")]
if oks:
base_tok[model] = statistics.mean(r.get("total_tokens", 0) for r in oks)
hdr = f"{'model':<34} {'arm':<16} {'ok':>7} {'tok_mean':>9} {'tok_med':>8} {'calls':>6} {'wall_s':>7} {'vs base':>8}"
print(hdr)
print("-" * len(hdr))
for model, arm in sorted(cells):
rs = cells[(model, arm)]
oks = [r for r in rs if r.get("ok")]
n_ok, n = len(oks), len(rs)
toks = [r.get("total_tokens", 0) for r in oks]
calls = [r.get("tool_calls", 0) for r in oks]
walls = [r.get("wall_s", 0) for r in oks]
tok_mean = statistics.mean(toks) if toks else 0
delta = ""
if arm != "base" and model in base_tok and tok_mean:
delta = f"{(tok_mean - base_tok[model]) / base_tok[model] * 100:+.0f}%"
print(
f"{model:<34} {arm:<16} {n_ok:>3}/{n:<3} {tok_mean:>9.0f} "
f"{statistics.median(toks) if toks else 0:>8.0f} "
f"{statistics.mean(calls) if calls else 0:>6.1f} "
f"{statistics.mean(walls) if walls else 0:>7.1f} {delta:>8}"
)
errs = [r for r in rows if r.get("error")]
if errs:
print(f"\nerrors: {len(errs)}")
for r in errs[:10]:
print(
f" {r.get('model')}/{r.get('arm')}/{r.get('task')}/rep{r.get('rep')}: {r['error'][:120]}"
)
if __name__ == "__main__":
main(sys.argv[1:] or ["results/results.jsonl"])
evals/browser_use/results/.gitignore
+1
View File
@@ -0,0 +1 @@
*.jsonl
evals/browser_use/single_run.py
+198
View File
@@ -0,0 +1,198 @@
"""One benchmark cell: task x arm x model x rep.
Usage:
python3 single_run.py <arm> <task_key> <model> <rep>
Arms:
base - built-in ``browser_*`` toolset (twelve tools), pinned tree $BUBENCH_BASE_TREE
pr - Browser Use CLI mode (single ``browser_exec`` tool), pinned tree $BUBENCH_PR_TREE
prns - same as pr but with the schema description stripped to the header only
(isolates the value of the helpers digest in the tool description)
Environment:
BUBENCH_ROOT workspace dir (default: dir containing this script)
BUBENCH_BASE_TREE checkout used for the ``base`` arm (e.g. a merge-base worktree)
BUBENCH_PR_TREE checkout used for the ``pr``/``prns`` arms
BUBENCH_TASKS tasks json (default: $BUBENCH_ROOT/tasks/hard.json)
BENCH_CDP_URL CDP endpoint both arms drive (default http://127.0.0.1:9333)
OPENROUTER_API_KEY provider credential for the runs
The run gets a throwaway HERMES_HOME so no local config leaks in, and the
web-fetch credential env vars are stripped so every arm must actually drive
the browser (no web_extract shortcuts).
Prints one line: ``RESULT_JSON:{...}`` consumed by orchestrate.py.
"""
import json
import os
import re
import sys
import tempfile
import time
ARM, TASK_KEY, MODEL, REP = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
ROOT = os.environ.get("BUBENCH_ROOT", os.path.dirname(os.path.abspath(__file__)))
BASE_TREE = os.environ["BUBENCH_BASE_TREE"]
PR_TREE = os.environ["BUBENCH_PR_TREE"]
WT = {"base": BASE_TREE, "pr": PR_TREE, "prns": PR_TREE}[ARM]
TASKS_PATH = os.environ.get("BUBENCH_TASKS", os.path.join(ROOT, "tasks", "hard.json"))
TASKS = json.load(open(TASKS_PATH, encoding="utf-8"))
task = TASKS[TASK_KEY]
home = tempfile.mkdtemp(prefix=f"buhome-{ARM}-")
hh = os.path.join(home, ".hermes")
os.makedirs(os.path.join(hh, "logs"), exist_ok=True)
cdp = os.environ.get("BENCH_CDP_URL", "http://127.0.0.1:9333")
browser_cfg = (
{"cloud_provider": "local", "cdp_url": cdp}
if ARM == "base"
else {"backend": "browser-use"}
)
cfg = {
"model": {"provider": "openrouter", "default": MODEL},
"browser": browser_cfg,
"display": {"quiet": True},
}
import yaml
with open(os.path.join(hh, "config.yaml"), "w", encoding="utf-8") as f:
yaml.safe_dump(cfg, f)
os.environ["HERMES_HOME"] = hh
# Strip web-fetch shortcuts: every arm must drive the browser.
os.environ.pop("BROWSER_USE_API_KEY", None)
for k in ("FIRECRAWL_API_KEY", "NOUS_API_KEY", "TAVILY_API_KEY", "SERPER_API_KEY"):
os.environ.pop(k, None)
os.environ["BU_CDP_URL"] = cdp
os.environ["PATH"] = (
os.path.expanduser("~/.local/bin") + os.pathsep + os.environ.get("PATH", "")
)
sys.path.insert(0, WT)
import logging
logging.disable(logging.CRITICAL)
import run_agent # noqa: E402
_loaded = os.path.normcase(os.path.normpath(run_agent.__file__))
_want = os.path.normcase(os.path.normpath(WT))
assert _loaded.startswith(_want), f"wrong tree: {run_agent.__file__}"
if ARM == "prns":
# Strip the helpers digest from the schema: header-only description.
import tools.browser_use_cli as bu # noqa: E402
bu._skill_text_fetched = True
bu._skill_text_cache = None
bu.BROWSER_EXEC_SCHEMA["description"] = bu._description_header()
from run_agent import AIAgent # noqa: E402
# Provider resolution: default openrouter (original battery), but allow the
# Nous-subscription path on boxes without an OpenRouter key. Credentials are
# resolved through the product's own auth state, never printed.
_or_key = os.environ.get("OPENROUTER_API_KEY", "").strip()
if _or_key:
_agent_auth = dict(
base_url="https://openrouter.ai/api/v1",
api_key=_or_key,
provider="openrouter",
)
else:
# Resolved by the orchestrator BEFORE HERMES_HOME is redirected to the
# throwaway home (auth state lives in the real profile). Never printed.
_tok = os.environ.get("BUBENCH_NOUS_TOKEN", "").strip()
if not _tok:
raise SystemExit("no OPENROUTER_API_KEY and no Nous auth available")
_agent_auth = dict(
base_url=os.environ.get("BUBENCH_NOUS_BASE_URL", "https://inference-api.nousresearch.com/v1"),
api_key=_tok,
provider="nous",
)
agent = AIAgent(
**_agent_auth,
model=MODEL,
max_iterations=30,
quiet_mode=True,
skip_context_files=True,
skip_memory=True,
# NB: "terminal" must be present for the pr arms — since #81958's terminal
# gate, browser_exec is stripped from sessions whose toolsets exclude
# terminal. Both arms get the same toolsets for parity; audit
# tool_call_names in the results for terminal-tool bypasses (curl etc.).
enabled_toolsets=["browser", "terminal"],
save_trajectories=False,
)
schema_desc_len = 0
try:
from model_tools import get_tool_definitions
for t in get_tool_definitions(agent.enabled_toolsets):
if t["function"]["name"].startswith("browser"):
schema_desc_len += len(json.dumps(t["function"]))
except Exception:
pass
t0 = time.time()
error = None
final = ""
messages = []
try:
result = agent.run_conversation(task["prompt"])
final = (
(result.get("final_response") or "")
if isinstance(result, dict)
else str(result)
)
messages = result.get("messages", []) if isinstance(result, dict) else []
except Exception as e: # noqa: BLE001
error = f"{type(e).__name__}: {e}"
messages = getattr(agent, "messages", []) or []
wall = time.time() - t0
tool_calls = []
for m in messages:
if isinstance(m, dict) and m.get("role") == "assistant":
for tc in m.get("tool_calls") or []:
fn = (
(tc.get("function") or {}).get("name") if isinstance(tc, dict) else None
)
if fn:
tool_calls.append(fn)
def _ok(text: str) -> bool:
if task.get("oracle_all"):
return all(
re.search(re.escape(x), text, re.IGNORECASE) for x in task["oracle_all"]
)
return any(
re.search(re.escape(x), text, re.IGNORECASE) for x in task.get("oracle_any", [])
)
out = {
"arm": ARM,
"task": TASK_KEY,
"model": MODEL,
"rep": int(REP),
"ok": bool(final) and _ok(final) and error is None,
"wall_s": round(wall, 1),
"prompt_tokens": getattr(agent, "session_prompt_tokens", 0),
"completion_tokens": getattr(agent, "session_completion_tokens", 0),
"total_tokens": getattr(agent, "session_total_tokens", 0),
"api_calls": len([
m for m in messages if isinstance(m, dict) and m.get("role") == "assistant"
]),
"tool_calls": len(tool_calls),
"tool_call_names": tool_calls,
"browser_schema_chars": schema_desc_len,
"error": error,
"final_snippet": (final or "")[-400:],
}
print("RESULT_JSON:" + json.dumps(out, ensure_ascii=False))
evals/browser_use/tasks/easy.json
+22
View File
@@ -0,0 +1,22 @@
{
"simple_price": {
"prompt": "Using your browser tools, open https://books.toscrape.com , find the book 'Sharp Objects' and tell me its listed price. Give the exact price in your final answer.",
"oracle_any": ["47.82"]
},
"multi_extract": {
"prompt": "Using your browser tools, go to https://books.toscrape.com , open the 'Travel' category, and list EVERY book in that category with its price. In your final answer state the total count of books and the title and price of the cheapest one.",
"oracle_all": ["11", "23.21"]
},
"count_aggregate": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com and determine which author appears most often on page 1 and how many quotes they have there. Final answer must name the author and the count.",
"oracle_all": ["Einstein", "3"]
},
"form_login": {
"prompt": "Using your browser tools, go to https://quotes.toscrape.com/login , log in with username 'admin' and password 'admin', and tell me the text of the link that replaced 'Login' in the top navigation after logging in.",
"oracle_any": ["Logout", "logout"]
},
"pagination_nav": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com , navigate to page 2, and tell me the author of the FIRST quote on page 2.",
"oracle_any": ["Marilyn Monroe"]
}
}
evals/browser_use/tasks/hard.json
+26
View File
@@ -0,0 +1,26 @@
{
"multi_page_filter": {
"prompt": "Using your browser tools, open https://books.toscrape.com , go to the 'Mystery' category, and go through ALL of its pages. Count how many Mystery books cost less than \u00a330.00, and identify the single cheapest Mystery book. Final answer must state the count and the cheapest book's exact title and price.",
"oracle_all": ["19", "Tastes Like Fear", "10.69"]
},
"rating_aggregate": {
"prompt": "Using your browser tools, open https://books.toscrape.com , browse the ENTIRE 'Mystery' category (all pages). Find every book with a five-star rating, then compute the average price of those five-star books. Final answer must state how many five-star Mystery books exist and the average price rounded to 2 decimals.",
"oracle_all": ["5", "32.27"]
},
"js_rendered": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com/js/ (the content on this page is rendered by JavaScript). Tell me the author of the LAST quote on page 1. Final answer must name the author.",
"oracle_any": ["Steve Martin"]
},
"delayed_render": {
"prompt": "Using your browser tools, open https://quotes.toscrape.com/js-delayed/ (content appears only after a delayed JavaScript render). Tell me the author of the FIRST quote on the page. Final answer must name the author.",
"oracle_any": ["Albert Einstein"]
},
"login_then_extract": {
"prompt": "Using your browser tools: (1) log in at https://quotes.toscrape.com/login with username 'admin' and password 'admin' and confirm the login worked, (2) then navigate to the tag page for the tag 'books' and report how many quotes are shown there and which authors appear more than once on that page. Final answer must state the quote count and name every author who appears twice.",
"oracle_all": ["10", "Mark Twain", "Jane Austen"]
},
"cross_category_compare": {
"prompt": "Using your browser tools, open https://books.toscrape.com and compare the 'Mystery' and 'Historical Fiction' categories across ALL their pages: which category contains the single most expensive book overall? Final answer must name that category, the book's title, and its exact price.",
"oracle_all": ["Historical Fiction", "Sara de Vos", "55.55"]
}
}