Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license
This commit is contained in:
+180
@@ -0,0 +1,180 @@
|
||||
---
|
||||
title: "Har Derived Api Client — Record a site's XHR into a HAR, derive an HTTP client"
|
||||
sidebar_label: "Har Derived Api Client"
|
||||
description: "Record a site's XHR into a HAR, derive an HTTP client"
|
||||
---
|
||||
|
||||
{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
|
||||
|
||||
# Har Derived Api Client
|
||||
|
||||
Record a site's XHR into a HAR, derive an HTTP client.
|
||||
|
||||
## Skill metadata
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Source | Optional — install with `hermes skills install official/web-development/har-derived-api-client` |
|
||||
| Path | `optional-skills/web-development\har-derived-api-client` |
|
||||
| Version | `0.1.0` |
|
||||
| Author | Hermes Agent |
|
||||
| License | MIT |
|
||||
| Platforms | linux, macos, windows |
|
||||
| Tags | `Browser`, `HAR`, `API`, `Reverse-Engineering`, `Playwright` |
|
||||
|
||||
## Reference: full SKILL.md
|
||||
|
||||
:::info
|
||||
The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
|
||||
:::
|
||||
|
||||
# HAR-Derived API Client
|
||||
|
||||
Drive a website once with a real browser while recording its network traffic
|
||||
to a HAR file, then distill that HAR into the site's private JSON API so you
|
||||
can call it directly with plain HTTP — far cheaper and faster than
|
||||
browser-controlling the page on every request. Credit: trick by Jared Longster,
|
||||
popularized by Dax (thdxr). This captures and replays; it does NOT bypass
|
||||
auth, solve CAPTCHAs, or defeat bot-detection — if the site needs a logged-in
|
||||
session, you carry its headers/cookies forward, you don't forge them.
|
||||
|
||||
The scripts are stdlib-plus-Playwright: capture needs Playwright, derivation
|
||||
is pure stdlib, replay needs only `requests`/`httpx` (or `curl`).
|
||||
|
||||
Covers **every Hermes browser pathway**: the default local `browser_navigate`
|
||||
backend, plus the cloud/remote backends (Browserbase, Browser-Use, Firecrawl)
|
||||
and any `/browser connect` CDP endpoint. There are two capture scripts — one
|
||||
for a browser you launch, one for a browser you attach to over CDP — because
|
||||
HAR recording works differently in each case (see How to Run).
|
||||
|
||||
## When to Use
|
||||
|
||||
- "Build a CLI/client for <website>" — derive its API instead of scripting clicks.
|
||||
- "This site has no public API but the page clearly fetches JSON."
|
||||
- You're about to loop `browser_navigate` for the same query repeatedly — stop and derive the endpoint once.
|
||||
- Reverse-engineering an autocomplete, search, feed, or checkout XHR.
|
||||
- You captured a session on a cloud backend (Browserbase / Browser-Use / Firecrawl) or via `/browser connect` and want the API without re-renting the browser.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Playwright + a browser binary (capture step only):
|
||||
- `pip install playwright` then `playwright install chromium`
|
||||
- (If a system Playwright already has browsers under `~/.cache/ms-playwright`, reuse it.)
|
||||
- `requests` or `httpx` for the replay step (stdlib `urllib` also works).
|
||||
- No API keys. Any keys/tokens the client needs are the ones the HAR captured.
|
||||
- For the CDP path (`har_capture_cdp.py`): a reachable CDP endpoint. On Hermes,
|
||||
run `/browser connect` to print the active endpoint, or read `BROWSER_CDP_URL`
|
||||
/ `browser.cdp_url` in config. Cloud backends expose it as `cdpUrl`/`connectUrl`.
|
||||
|
||||
## How to Run
|
||||
|
||||
Scripts under this skill's `scripts/`, invoked through the `terminal` tool.
|
||||
**Pick the capturer by pathway** — this is the part that trips people up:
|
||||
|
||||
| Browser pathway | How Hermes reaches it | Capturer |
|
||||
|---|---|---|
|
||||
| Local `browser_navigate` (default, agent-browser/Playwright) | launched locally | `har_capture.py` |
|
||||
| Camofox (`CAMOFOX_URL` set) | local REST/CDP | `har_capture_cdp.py` if it exposes CDP, else drive it yourself |
|
||||
| Browserbase / Browser-Use / Firecrawl (cloud) | **CDP** (`cdpUrl`) | `har_capture_cdp.py` |
|
||||
| `/browser connect <url>` / `BROWSER_CDP_URL` | **CDP** | `har_capture_cdp.py` |
|
||||
|
||||
Rule of thumb: **if Hermes *launched* the browser, use `har_capture.py`; if it
|
||||
*connected to* one over CDP, use `har_capture_cdp.py`.** `har_capture.py` uses
|
||||
Playwright's `record_har_path`, which only works on a locally-owned context.
|
||||
`har_capture_cdp.py` attaches with `connect_over_cdp()` and assembles the HAR
|
||||
from `page.on("request"/"response")` events, because `record_har_path` is
|
||||
unavailable on a connected browser.
|
||||
|
||||
Then, for either path:
|
||||
|
||||
- `har_to_client.py` — filters the HAR to XHR/fetch/JSON, groups by endpoint, and prints params, headers, bodies, and replay hints (User-Agent / cookie / auth).
|
||||
|
||||
Resolve paths against this skill's directory. Canonical loop:
|
||||
|
||||
```bash
|
||||
# 1a. Capture, LOCAL browser (Hermes launched it)
|
||||
python3 scripts/har_capture.py "https://SITE/" out.har \
|
||||
--action "fill:input[name=search]:my query" --action "sleep:3" --wait 2
|
||||
|
||||
# 1b. Capture, CDP browser (cloud backend or /browser connect)
|
||||
# get the endpoint from /browser connect or BROWSER_CDP_URL
|
||||
python3 scripts/har_capture_cdp.py "ws://HOST/devtools/browser/..." out.har \
|
||||
--goto "https://SITE/" --action "fill:input[name=search]:my query" \
|
||||
--action "sleep:3" --wait 2
|
||||
|
||||
# 2. Derive — read the endpoints out of the HAR
|
||||
python3 scripts/har_to_client.py out.har --host SITE --max-body 400
|
||||
|
||||
# 3. Replay — write a tiny client from the printed endpoint (see Procedure)
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```
|
||||
har_capture.py <url> <out.har> [--wait S] [--headed] [--action SPEC ...]
|
||||
action SPEC: fill:SELECTOR:TEXT | press:SELECTOR:KEY | click:SELECTOR
|
||||
goto:URL | sleep:SECONDS (run in order after page load)
|
||||
use when Hermes LAUNCHED the browser (local browser_navigate default)
|
||||
|
||||
har_capture_cdp.py <cdp_url> <out.har> [--goto URL] [--wait S] [--action SPEC ...]
|
||||
same action SPEC; attaches to an existing CDP browser and does NOT close it
|
||||
use for cloud backends (Browserbase/Browser-Use/Firecrawl) & /browser connect
|
||||
|
||||
har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
|
||||
default: keeps only XHR/fetch/JSON; --host narrows to one domain
|
||||
prints per endpoint: query params, non-boring req headers, req body sample,
|
||||
response status/content-type + body sample
|
||||
prints "### Replay hints": the browser User-Agent, cookie/auth presence
|
||||
```
|
||||
|
||||
## Procedure
|
||||
|
||||
0. **Pick the capturer by pathway** (see How to Run table). Launched-locally → `har_capture.py`; reached over CDP → `har_capture_cdp.py`. On Hermes, `/browser connect` tells you the CDP endpoint when a cloud/remote backend is active.
|
||||
1. **Find the interaction.** Open the site with `browser_navigate` (or `--headed` capture) to see which selector to type into / click, and confirm a JSON XHR fires in devtools/network.
|
||||
2. **Capture the HAR** via the `terminal` tool. Order `--action` to reach the request: `fill` the box, then `sleep` long enough for the debounced XHR, and always leave `--wait` at the end so late responses flush. Both capturers embed response bodies, so the derived client sees real payload shapes.
|
||||
3. **Derive** with `har_to_client.py --host <domain>`. Read off: the method, the URL/path template (numeric/UUID segments collapse to `{id}`), query params, request-body JSON, and the `### Replay hints` block.
|
||||
4. **Write the client.** Recreate the request exactly — same method, path, query params, body. Send the headers the site actually needs: at minimum copy the **User-Agent** from the replay hints. If hints report cookies or an auth/token header, resend those too.
|
||||
5. **Test browserless.** Run the client with the `terminal` tool and confirm it returns the same data the browser saw. This is the payoff: no browser in the loop.
|
||||
6. **(Optional) Wrap as a CLI** — a small `argparse` script over the derived call, e.g. `search.py "frank herbert"`.
|
||||
|
||||
Worked example (Wikipedia search-title, derived + replayed live):
|
||||
|
||||
```python
|
||||
import requests
|
||||
r = requests.get(
|
||||
"https://en.wikipedia.org/w/rest.php/v1/search/title",
|
||||
params={"q": "frank herbert", "limit": 5},
|
||||
headers={"accept": "application/json",
|
||||
"User-Agent": "Mozilla/5.0 ... Chrome/131 Safari/537.36"}, # from HAR
|
||||
timeout=15,
|
||||
)
|
||||
for p in r.json()["pages"]:
|
||||
print(p["title"], "-", p.get("description"))
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Default library User-Agent gets 403.** Many sites (Wikipedia, Cloudflare-fronted APIs) reject `python-requests/x.y`. Always send the browser UA from the replay hints. This is the #1 reason a derived client fails when the browser succeeded.
|
||||
- **A failed `--action` aborts before the HAR flushes** — you get no file. If capture errors on a selector, the run produced nothing; fix the selector (use `--headed` to watch) and rerun. Don't debug a missing HAR.
|
||||
- **Server-rendered pages have no XHR** to derive — `har_to_client.py` prints "No API-looking entries". The data came in the HTML; scrape it or find the interaction that does fetch JSON.
|
||||
- **Debounced/typeahead XHRs need a real pause.** Add `--action "sleep:3"` after `fill`; typing alone won't have fired the request when the HAR closes.
|
||||
- **Auth/session endpoints** need the captured `Cookie`/`Authorization` header, and those expire. The derived client is only as durable as the credential; re-capture when it 401s. HARs contain live secrets — treat `out.har` as sensitive and delete it after deriving.
|
||||
- **`record_har_content="embed"` makes big HARs.** Use `--max-body` to cap what's printed; the file itself can be large for media-heavy pages.
|
||||
- **Endpoints shift.** Sites change private APIs without notice. Re-run the capture→derive loop when a client breaks rather than patching URLs by hand.
|
||||
- **Wrong capturer = empty/no HAR.** `har_capture.py` on a cloud/CDP backend records nothing (it launches its own local browser instead of the one you meant). `har_capture_cdp.py` needs the endpoint; on Hermes get it from `/browser connect` or `BROWSER_CDP_URL`. Match the capturer to the pathway (How to Run table).
|
||||
- **Headless-Chrome UA is a weak tell.** Local/agent-browser capture yields a `HeadlessChrome/...` User-Agent; some sites sniff the "Headless" token. Cloud backends (Browserbase/Browser-Use) send a real desktop-Chrome UA, so a client derived from a cloud capture replays more reliably. If a headless-derived client 403s where the browser didn't, swap the "Headless" UA for a normal Chrome UA string before assuming the endpoint changed.
|
||||
- **CDP capture doesn't close the browser.** `har_capture_cdp.py` attaches to a browser it doesn't own and leaves it running — correct for cloud/remote sessions Hermes manages. Don't add a close; let the owning backend tear it down.
|
||||
|
||||
## Verification
|
||||
|
||||
End-to-end proof against a live site with no API key:
|
||||
|
||||
```bash
|
||||
python3 scripts/har_capture.py "https://en.wikipedia.org/wiki/Main_Page" /tmp/wiki.har \
|
||||
--action "fill:input[name=search]:dune messiah" --action "sleep:3" --wait 2
|
||||
python3 scripts/har_to_client.py /tmp/wiki.har --host wikipedia.org --max-body 200
|
||||
```
|
||||
|
||||
Expect the derivation to print `GET https://en.wikipedia.org/w/rest.php/v1/search/title`
|
||||
with `q` and `limit` params and a JSON `pages` response — then replay it with the
|
||||
Procedure snippet and confirm matching titles come back over plain HTTP.
|
||||
Reference in New Issue
Block a user