Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license
This commit is contained in:
@@ -0,0 +1,5 @@
|
||||
# Web Development
|
||||
|
||||
Optional skills for client-side web development workflows — embedding agents, copilots, and AI-native UX patterns into user-facing web apps.
|
||||
|
||||
These are distinct from Hermes' own browser automation (Browserbase, Camofox), which operate *on* websites from outside. Web-development skills here help users build *into* their own websites.
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
name: cloudflare-temporary-deploy
|
||||
description: Deploy a Worker live, no account, via wrangler --temporary.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [cloudflare, workers, wrangler, deploy, temporary, agent, serverless, web-development]
|
||||
category: web-development
|
||||
---
|
||||
|
||||
# Cloudflare Temporary Deploy Skill
|
||||
|
||||
Deploy a Cloudflare Worker to a live `workers.dev` URL with zero account setup, using `wrangler deploy --temporary`. Cloudflare provisions a throwaway account, deploys, and prints a claim URL valid for 60 minutes; unclaimed accounts auto-delete. This gives an agent a tight write → deploy → verify loop without any OAuth, signup, or token copy-paste.
|
||||
|
||||
This skill does NOT cover production deploys (use `wrangler login` + a permanent account for those), nor non-Worker Cloudflare products beyond the temporary-account limits below.
|
||||
|
||||
## When to Use
|
||||
|
||||
Load this skill when the user wants to:
|
||||
|
||||
- **Ship agent-written code to a live URL** without first creating a Cloudflare account — "deploy this and give me a link"
|
||||
- **Iterate in a background/autonomous session** where a browser OAuth step would be a hard stop
|
||||
- **Prototype or evaluate Workers** quickly with a throwaway, claimable target
|
||||
- **Build a self-verifying deploy loop** — deploy, `curl` the live URL, confirm output matches the code, redeploy
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- **Production or CI/CD** → use a permanent account (`wrangler login` or `CLOUDFLARE_API_TOKEN`). `--temporary` errors out if any credential is present.
|
||||
- **Wrangler is already authenticated** → `--temporary` returns an error by design. Run `wrangler logout` first only if the user explicitly wants a throwaway deploy.
|
||||
- **Long-lived hosting** → temporary deployments are deleted after 60 minutes unless claimed.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Wrangler 4.102.0 or later.** This is the version that introduced `--temporary`. Earlier versions do not have it. Verify with `npx wrangler@latest --version`.
|
||||
- **Node 18+ / npm** (or `npx`, `yarn`, `pnpm`). No global install needed — `npx wrangler@latest` works.
|
||||
- **No Cloudflare credentials present.** `--temporary` only works when Wrangler is unauthenticated: no OAuth login, no `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_API_KEY` env var, no `~/.wrangler` / `~/.config/.wrangler` cached OAuth. Use the `terminal` tool's environment as-is; do not set those vars.
|
||||
- Network egress to `cloudflare.com` and `workers.dev`.
|
||||
- Using `--temporary` accepts Cloudflare's Terms of Service and Privacy Policy.
|
||||
|
||||
## How to Run
|
||||
|
||||
Use the `terminal` tool for every step. Always pin the version (`wrangler@latest` or `wrangler@4.102.0` or newer) so you don't accidentally run an old global wrangler that lacks the flag.
|
||||
|
||||
1. **Scaffold a minimal Worker** (skip if the project already exists). A Worker needs a `wrangler.toml` (or `wrangler.jsonc`) and an entry script. Minimal TypeScript example — write these with `write_file`:
|
||||
|
||||
`wrangler.jsonc`:
|
||||
```jsonc
|
||||
{
|
||||
"name": "hello-agent",
|
||||
"main": "src/index.ts",
|
||||
"compatibility_date": "2025-01-01"
|
||||
}
|
||||
```
|
||||
|
||||
`src/index.ts`:
|
||||
```typescript
|
||||
export default {
|
||||
async fetch(): Promise<Response> {
|
||||
return new Response("hello cloudflare");
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
2. **Deploy with `--temporary`** from the project directory:
|
||||
```
|
||||
npx wrangler@latest deploy --temporary
|
||||
```
|
||||
The proof-of-work check adds a short automatic delay. On success Wrangler prints an `Account: <name> (created)` (or `(reused)`) line, a `Claim URL`, and the live `https://<worker>.<account>.workers.dev` URL.
|
||||
|
||||
3. **Parse the URLs** from that output. Run the helper to extract them reliably instead of eyeballing:
|
||||
```
|
||||
npx wrangler@latest deploy --temporary 2>&1 | python scripts/parse_deploy_output.py
|
||||
```
|
||||
(Resolve `scripts/parse_deploy_output.py` to this skill's absolute path.) It prints JSON: `{"live_url", "claim_url", "account", "account_state", "expires_minutes", "deployed"}`.
|
||||
|
||||
4. **Verify the deploy is actually live** — do not trust the deploy log alone. `curl` the live URL and confirm the body matches what the code returns:
|
||||
```
|
||||
curl -sS <live_url>
|
||||
```
|
||||
|
||||
5. **Iterate.** Edit the code, redeploy with the same `npx wrangler@latest deploy --temporary`. Within the 60-minute window Wrangler reuses the cached temporary account (`Account: <name> (reused)`), so the URL stays stable. `curl` again to confirm the change.
|
||||
|
||||
6. **Hand the claim URL to the user.** Tell them: open it within 60 minutes to keep the deployment and any resources; if they don't claim it, everything auto-deletes. Treat the claim URL as a secret — it grants ownership of the account.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Step | Command |
|
||||
|---|---|
|
||||
| Check version (need 4.102.0+) | `npx wrangler@latest --version` |
|
||||
| Deploy (no account) | `npx wrangler@latest deploy --temporary` |
|
||||
| Deploy + parse URLs | `npx wrangler@latest deploy --temporary 2>&1 \| python scripts/parse_deploy_output.py` |
|
||||
| Verify live | `curl -sS <live_url>` |
|
||||
| Clear cached temp account | `npx wrangler@latest logout` |
|
||||
|
||||
### Temporary account product limits
|
||||
|
||||
| Product | Limit on a temporary account |
|
||||
|---|---|
|
||||
| Workers | Deploys to `workers.dev` |
|
||||
| Static Assets | Up to 1,000 files, 5 MiB each |
|
||||
| KV | Allowed |
|
||||
| D1 | 1 database, 100 MB per DB / 100 MB total |
|
||||
| Durable Objects | Allowed |
|
||||
| Hyperdrive | 2 configs, 10 connections |
|
||||
| Queues | Up to 10 |
|
||||
| SSL/TLS certs | Allowed |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`--temporary` is not in `wrangler deploy --help` and is not a global flag.** It is intentionally hidden and surfaced dynamically: when an unauthenticated `wrangler deploy` fails, Wrangler prints "rerun with `--temporary`". Don't conclude the flag is missing just because `--help` omits it — check the version instead.
|
||||
- **Old global wrangler.** A stale globally-installed `wrangler` (`< 4.102.0`) silently lacks the flag. Always invoke `npx wrangler@latest` (or a pinned `>=4.102.0`) so you control the version.
|
||||
- **Auth present → hard error.** If `wrangler login` was ever run, or `CLOUDFLARE_API_TOKEN`/`CLOUDFLARE_API_KEY` is set, `--temporary` errors. Either unset the var for this shell or `wrangler logout`. Never strip a user's real credentials without telling them.
|
||||
- **Rate limiting.** Creating temporary accounts too fast fails. Reuse the cached account (just redeploy) within the 60-minute window instead of forcing a new one; if rate-limited, wait or use a permanent account.
|
||||
- **60-minute hard expiry, not extendable.** If the deploy must outlive an hour, the user must claim it. Surface this clearly.
|
||||
- **`curl` may briefly serve the old body after a redeploy.** `workers.dev` has a short edge cache; the `(reused)` line plus a new `Current Version ID` confirm the deploy succeeded even if `curl` shows stale content for a few seconds. Re-curl, or add a cache-busting query string, before concluding a redeploy failed.
|
||||
- **Don't log the claim URL into shared transcripts as "just a link."** It is credential-equivalent.
|
||||
|
||||
## Verification
|
||||
|
||||
- `npx wrangler@latest --version` returns `>= 4.102.0`.
|
||||
- `npx wrangler@latest deploy --temporary` prints a `workers.dev` live URL and a `claim-preview?claimToken=` claim URL.
|
||||
- `curl -sS <live_url>` returns the exact body the Worker code produces.
|
||||
- A second deploy reports `Account: <name> (reused)` and the live URL is unchanged.
|
||||
- The parser script's self-test passes: `python scripts/parse_deploy_output.py --selftest`.
|
||||
+122
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Parse `wrangler deploy --temporary` output into structured JSON.
|
||||
|
||||
Reads wrangler's stdout/stderr from STDIN and extracts the live workers.dev
|
||||
URL, the claim URL, the temporary account name/state, the claim window, and
|
||||
whether a deploy actually happened. Stdlib only — no dependencies.
|
||||
|
||||
Usage:
|
||||
npx wrangler@latest deploy --temporary 2>&1 | python3 parse_deploy_output.py
|
||||
python3 parse_deploy_output.py --selftest
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
|
||||
# Match the live workers.dev URL (subdomain.subdomain.workers.dev).
|
||||
_LIVE_URL = re.compile(r"https://[A-Za-z0-9._-]+\.workers\.dev\S*")
|
||||
# Match the claim URL. Cloudflare uses dash.cloudflare.com/claim-preview?claimToken=...
|
||||
# Keep it broad enough to survive minor path changes while still requiring a claim token.
|
||||
_CLAIM_URL = re.compile(r"https://\S*claim\S*claimToken=\S+", re.IGNORECASE)
|
||||
# "Account: Serene Temple (created)" / "Account: example-name (reused)"
|
||||
# Account names can contain spaces (e.g. "Serene Temple"), so capture everything
|
||||
# up to the trailing "(state)" marker rather than a single token.
|
||||
_ACCOUNT = re.compile(
|
||||
r"Account:\s*(?P<name>.+?)\s*\((?P<state>created|reused)\)", re.IGNORECASE
|
||||
)
|
||||
# "Claim within: 60 minutes"
|
||||
_CLAIM_WITHIN = re.compile(r"Claim within:\s*(?P<minutes>\d+)\s*minutes?", re.IGNORECASE)
|
||||
# A successful deploy prints a "Deployed" / "Uploaded" line.
|
||||
_DEPLOYED = re.compile(r"^\s*(Deployed|Uploaded)\b", re.IGNORECASE | re.MULTILINE)
|
||||
|
||||
|
||||
def _first(pattern: re.Pattern, text: str) -> str | None:
|
||||
m = pattern.search(text)
|
||||
if not m:
|
||||
return None
|
||||
# Strip trailing punctuation that often clings to a URL in log lines.
|
||||
return m.group(0).rstrip(".,);]")
|
||||
|
||||
|
||||
def parse(text: str) -> dict:
|
||||
"""Extract deploy facts from wrangler output text."""
|
||||
account = _ACCOUNT.search(text)
|
||||
claim_within = _CLAIM_WITHIN.search(text)
|
||||
return {
|
||||
"live_url": _first(_LIVE_URL, text),
|
||||
"claim_url": _first(_CLAIM_URL, text),
|
||||
"account": account.group("name") if account else None,
|
||||
"account_state": account.group("state").lower() if account else None,
|
||||
"expires_minutes": int(claim_within.group("minutes")) if claim_within else None,
|
||||
"deployed": bool(_DEPLOYED.search(text)),
|
||||
}
|
||||
|
||||
|
||||
_SAMPLE = """\
|
||||
Continuing means you accept Cloudflare's Terms of Service and Privacy Policy.
|
||||
|
||||
Temporary account ready:
|
||||
Account: example-name (created)
|
||||
Claim within: 60 minutes
|
||||
Claim URL: https://dash.cloudflare.com/claim-preview?claimToken=abc123XYZ
|
||||
|
||||
Uploaded example-worker
|
||||
Deployed example-worker triggers
|
||||
https://example-worker.example-name.workers.dev
|
||||
"""
|
||||
|
||||
_SAMPLE_REUSED = """\
|
||||
Temporary account ready:
|
||||
Account: example-name (reused)
|
||||
Claim within: 42 minutes
|
||||
Claim URL: https://dash.cloudflare.com/claim-preview?claimToken=def456
|
||||
Deployed example-worker triggers
|
||||
https://example-worker.example-name.workers.dev
|
||||
"""
|
||||
|
||||
_SAMPLE_NO_TEMP = """\
|
||||
✘ [ERROR] You are not logged in.
|
||||
|
||||
To continue without logging in, rerun this command with `--temporary`.
|
||||
"""
|
||||
|
||||
|
||||
def _selftest() -> int:
|
||||
r = parse(_SAMPLE)
|
||||
assert r["live_url"] == "https://example-worker.example-name.workers.dev", r
|
||||
assert r["claim_url"] == "https://dash.cloudflare.com/claim-preview?claimToken=abc123XYZ", r
|
||||
assert r["account"] == "example-name", r
|
||||
assert r["account_state"] == "created", r
|
||||
assert r["expires_minutes"] == 60, r
|
||||
assert r["deployed"] is True, r
|
||||
|
||||
r2 = parse(_SAMPLE_REUSED)
|
||||
assert r2["account_state"] == "reused", r2
|
||||
assert r2["expires_minutes"] == 42, r2
|
||||
assert r2["deployed"] is True, r2
|
||||
|
||||
r3 = parse(_SAMPLE_NO_TEMP)
|
||||
assert r3["live_url"] is None, r3
|
||||
assert r3["claim_url"] is None, r3
|
||||
assert r3["account"] is None, r3
|
||||
assert r3["deployed"] is False, r3
|
||||
|
||||
print("selftest: OK")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
if "--selftest" in argv:
|
||||
return _selftest()
|
||||
text = sys.stdin.read()
|
||||
result = parse(text)
|
||||
print(json.dumps(result, indent=2))
|
||||
# Non-zero exit if no live URL was found, so callers can branch on it.
|
||||
return 0 if result["live_url"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,163 @@
|
||||
---
|
||||
name: har-derived-api-client
|
||||
description: Record a site's XHR into a HAR, derive an HTTP client.
|
||||
version: 0.1.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [Browser, HAR, API, Reverse-Engineering, Playwright]
|
||||
category: web-development
|
||||
---
|
||||
|
||||
# HAR-Derived API Client
|
||||
|
||||
Drive a website once with a real browser while recording its network traffic
|
||||
to a HAR file, then distill that HAR into the site's private JSON API so you
|
||||
can call it directly with plain HTTP — far cheaper and faster than
|
||||
browser-controlling the page on every request. Credit: trick by Jared Longster,
|
||||
popularized by Dax (thdxr). This captures and replays; it does NOT bypass
|
||||
auth, solve CAPTCHAs, or defeat bot-detection — if the site needs a logged-in
|
||||
session, you carry its headers/cookies forward, you don't forge them.
|
||||
|
||||
The scripts are stdlib-plus-Playwright: capture needs Playwright, derivation
|
||||
is pure stdlib, replay needs only `requests`/`httpx` (or `curl`).
|
||||
|
||||
Covers **every Hermes browser pathway**: the default local `browser_navigate`
|
||||
backend, plus the cloud/remote backends (Browserbase, Browser-Use, Firecrawl)
|
||||
and any `/browser connect` CDP endpoint. There are two capture scripts — one
|
||||
for a browser you launch, one for a browser you attach to over CDP — because
|
||||
HAR recording works differently in each case (see How to Run).
|
||||
|
||||
## When to Use
|
||||
|
||||
- "Build a CLI/client for <website>" — derive its API instead of scripting clicks.
|
||||
- "This site has no public API but the page clearly fetches JSON."
|
||||
- You're about to loop `browser_navigate` for the same query repeatedly — stop and derive the endpoint once.
|
||||
- Reverse-engineering an autocomplete, search, feed, or checkout XHR.
|
||||
- You captured a session on a cloud backend (Browserbase / Browser-Use / Firecrawl) or via `/browser connect` and want the API without re-renting the browser.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Playwright + a browser binary (capture step only):
|
||||
- `pip install playwright` then `playwright install chromium`
|
||||
- (If a system Playwright already has browsers under `~/.cache/ms-playwright`, reuse it.)
|
||||
- `requests` or `httpx` for the replay step (stdlib `urllib` also works).
|
||||
- No API keys. Any keys/tokens the client needs are the ones the HAR captured.
|
||||
- For the CDP path (`har_capture_cdp.py`): a reachable CDP endpoint. On Hermes,
|
||||
run `/browser connect` to print the active endpoint, or read `BROWSER_CDP_URL`
|
||||
/ `browser.cdp_url` in config. Cloud backends expose it as `cdpUrl`/`connectUrl`.
|
||||
|
||||
## How to Run
|
||||
|
||||
Scripts under this skill's `scripts/`, invoked through the `terminal` tool.
|
||||
**Pick the capturer by pathway** — this is the part that trips people up:
|
||||
|
||||
| Browser pathway | How Hermes reaches it | Capturer |
|
||||
|---|---|---|
|
||||
| Local `browser_navigate` (default, agent-browser/Playwright) | launched locally | `har_capture.py` |
|
||||
| Camofox (`CAMOFOX_URL` set) | local REST/CDP | `har_capture_cdp.py` if it exposes CDP, else drive it yourself |
|
||||
| Browserbase / Browser-Use / Firecrawl (cloud) | **CDP** (`cdpUrl`) | `har_capture_cdp.py` |
|
||||
| `/browser connect <url>` / `BROWSER_CDP_URL` | **CDP** | `har_capture_cdp.py` |
|
||||
|
||||
Rule of thumb: **if Hermes *launched* the browser, use `har_capture.py`; if it
|
||||
*connected to* one over CDP, use `har_capture_cdp.py`.** `har_capture.py` uses
|
||||
Playwright's `record_har_path`, which only works on a locally-owned context.
|
||||
`har_capture_cdp.py` attaches with `connect_over_cdp()` and assembles the HAR
|
||||
from `page.on("request"/"response")` events, because `record_har_path` is
|
||||
unavailable on a connected browser.
|
||||
|
||||
Then, for either path:
|
||||
|
||||
- `har_to_client.py` — filters the HAR to XHR/fetch/JSON, groups by endpoint, and prints params, headers, bodies, and replay hints (User-Agent / cookie / auth).
|
||||
|
||||
Resolve paths against this skill's directory. Canonical loop:
|
||||
|
||||
```bash
|
||||
# 1a. Capture, LOCAL browser (Hermes launched it)
|
||||
python3 scripts/har_capture.py "https://SITE/" out.har \
|
||||
--action "fill:input[name=search]:my query" --action "sleep:3" --wait 2
|
||||
|
||||
# 1b. Capture, CDP browser (cloud backend or /browser connect)
|
||||
# get the endpoint from /browser connect or BROWSER_CDP_URL
|
||||
python3 scripts/har_capture_cdp.py "ws://HOST/devtools/browser/..." out.har \
|
||||
--goto "https://SITE/" --action "fill:input[name=search]:my query" \
|
||||
--action "sleep:3" --wait 2
|
||||
|
||||
# 2. Derive — read the endpoints out of the HAR
|
||||
python3 scripts/har_to_client.py out.har --host SITE --max-body 400
|
||||
|
||||
# 3. Replay — write a tiny client from the printed endpoint (see Procedure)
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```
|
||||
har_capture.py <url> <out.har> [--wait S] [--headed] [--action SPEC ...]
|
||||
action SPEC: fill:SELECTOR:TEXT | press:SELECTOR:KEY | click:SELECTOR
|
||||
goto:URL | sleep:SECONDS (run in order after page load)
|
||||
use when Hermes LAUNCHED the browser (local browser_navigate default)
|
||||
|
||||
har_capture_cdp.py <cdp_url> <out.har> [--goto URL] [--wait S] [--action SPEC ...]
|
||||
same action SPEC; attaches to an existing CDP browser and does NOT close it
|
||||
use for cloud backends (Browserbase/Browser-Use/Firecrawl) & /browser connect
|
||||
|
||||
har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
|
||||
default: keeps only XHR/fetch/JSON; --host narrows to one domain
|
||||
prints per endpoint: query params, non-boring req headers, req body sample,
|
||||
response status/content-type + body sample
|
||||
prints "### Replay hints": the browser User-Agent, cookie/auth presence
|
||||
```
|
||||
|
||||
## Procedure
|
||||
|
||||
0. **Pick the capturer by pathway** (see How to Run table). Launched-locally → `har_capture.py`; reached over CDP → `har_capture_cdp.py`. On Hermes, `/browser connect` tells you the CDP endpoint when a cloud/remote backend is active.
|
||||
1. **Find the interaction.** Open the site with `browser_navigate` (or `--headed` capture) to see which selector to type into / click, and confirm a JSON XHR fires in devtools/network.
|
||||
2. **Capture the HAR** via the `terminal` tool. Order `--action` to reach the request: `fill` the box, then `sleep` long enough for the debounced XHR, and always leave `--wait` at the end so late responses flush. Both capturers embed response bodies, so the derived client sees real payload shapes.
|
||||
3. **Derive** with `har_to_client.py --host <domain>`. Read off: the method, the URL/path template (numeric/UUID segments collapse to `{id}`), query params, request-body JSON, and the `### Replay hints` block.
|
||||
4. **Write the client.** Recreate the request exactly — same method, path, query params, body. Send the headers the site actually needs: at minimum copy the **User-Agent** from the replay hints. If hints report cookies or an auth/token header, resend those too.
|
||||
5. **Test browserless.** Run the client with the `terminal` tool and confirm it returns the same data the browser saw. This is the payoff: no browser in the loop.
|
||||
6. **(Optional) Wrap as a CLI** — a small `argparse` script over the derived call, e.g. `search.py "frank herbert"`.
|
||||
|
||||
Worked example (Wikipedia search-title, derived + replayed live):
|
||||
|
||||
```python
|
||||
import requests
|
||||
r = requests.get(
|
||||
"https://en.wikipedia.org/w/rest.php/v1/search/title",
|
||||
params={"q": "frank herbert", "limit": 5},
|
||||
headers={"accept": "application/json",
|
||||
"User-Agent": "Mozilla/5.0 ... Chrome/131 Safari/537.36"}, # from HAR
|
||||
timeout=15,
|
||||
)
|
||||
for p in r.json()["pages"]:
|
||||
print(p["title"], "-", p.get("description"))
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Default library User-Agent gets 403.** Many sites (Wikipedia, Cloudflare-fronted APIs) reject `python-requests/x.y`. Always send the browser UA from the replay hints. This is the #1 reason a derived client fails when the browser succeeded.
|
||||
- **A failed `--action` aborts before the HAR flushes** — you get no file. If capture errors on a selector, the run produced nothing; fix the selector (use `--headed` to watch) and rerun. Don't debug a missing HAR.
|
||||
- **Server-rendered pages have no XHR** to derive — `har_to_client.py` prints "No API-looking entries". The data came in the HTML; scrape it or find the interaction that does fetch JSON.
|
||||
- **Debounced/typeahead XHRs need a real pause.** Add `--action "sleep:3"` after `fill`; typing alone won't have fired the request when the HAR closes.
|
||||
- **Auth/session endpoints** need the captured `Cookie`/`Authorization` header, and those expire. The derived client is only as durable as the credential; re-capture when it 401s. HARs contain live secrets — treat `out.har` as sensitive and delete it after deriving.
|
||||
- **`record_har_content="embed"` makes big HARs.** Use `--max-body` to cap what's printed; the file itself can be large for media-heavy pages.
|
||||
- **Endpoints shift.** Sites change private APIs without notice. Re-run the capture→derive loop when a client breaks rather than patching URLs by hand.
|
||||
- **Wrong capturer = empty/no HAR.** `har_capture.py` on a cloud/CDP backend records nothing (it launches its own local browser instead of the one you meant). `har_capture_cdp.py` needs the endpoint; on Hermes get it from `/browser connect` or `BROWSER_CDP_URL`. Match the capturer to the pathway (How to Run table).
|
||||
- **Headless-Chrome UA is a weak tell.** Local/agent-browser capture yields a `HeadlessChrome/...` User-Agent; some sites sniff the "Headless" token. Cloud backends (Browserbase/Browser-Use) send a real desktop-Chrome UA, so a client derived from a cloud capture replays more reliably. If a headless-derived client 403s where the browser didn't, swap the "Headless" UA for a normal Chrome UA string before assuming the endpoint changed.
|
||||
- **CDP capture doesn't close the browser.** `har_capture_cdp.py` attaches to a browser it doesn't own and leaves it running — correct for cloud/remote sessions Hermes manages. Don't add a close; let the owning backend tear it down.
|
||||
|
||||
## Verification
|
||||
|
||||
End-to-end proof against a live site with no API key:
|
||||
|
||||
```bash
|
||||
python3 scripts/har_capture.py "https://en.wikipedia.org/wiki/Main_Page" /tmp/wiki.har \
|
||||
--action "fill:input[name=search]:dune messiah" --action "sleep:3" --wait 2
|
||||
python3 scripts/har_to_client.py /tmp/wiki.har --host wikipedia.org --max-body 200
|
||||
```
|
||||
|
||||
Expect the derivation to print `GET https://en.wikipedia.org/w/rest.php/v1/search/title`
|
||||
with `q` and `limit` params and a JSON `pages` response — then replay it with the
|
||||
Procedure snippet and confirm matching titles come back over plain HTTP.
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Record a HAR file while driving a website with Playwright.
|
||||
|
||||
Usage:
|
||||
python3 har_capture.py <url> <output.har> [--wait SECONDS] \
|
||||
[--action "fill:SELECTOR:TEXT"] [--action "press:SELECTOR:KEY"] \
|
||||
[--action "click:SELECTOR"] [--action "goto:URL"] [--action "sleep:SECONDS"]
|
||||
|
||||
Actions run in order after page load. The HAR embeds request/response bodies
|
||||
(record_har_content='embed') so derived clients can see payload shapes.
|
||||
|
||||
NOTE: a failing action raises before the HAR is flushed -- you get no file.
|
||||
Fix the selector (try --headed to watch) and rerun.
|
||||
"""
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
|
||||
def run_action(page, spec: str) -> None:
|
||||
parts = spec.split(":", 2)
|
||||
kind = parts[0]
|
||||
if kind == "fill":
|
||||
page.fill(parts[1], parts[2])
|
||||
elif kind == "press":
|
||||
page.press(parts[1], parts[2])
|
||||
elif kind == "click":
|
||||
page.click(parts[1])
|
||||
elif kind == "goto":
|
||||
page.goto(parts[1] + (":" + parts[2] if len(parts) > 2 else ""))
|
||||
elif kind == "sleep":
|
||||
time.sleep(float(parts[1]))
|
||||
else:
|
||||
raise ValueError(f"unknown action: {spec}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("url")
|
||||
ap.add_argument("har_path")
|
||||
ap.add_argument("--wait", type=float, default=3.0,
|
||||
help="seconds to idle at the end so late XHRs land in the HAR")
|
||||
ap.add_argument("--action", action="append", default=[],
|
||||
help="fill:SEL:TEXT | press:SEL:KEY | click:SEL | goto:URL | sleep:SECS")
|
||||
ap.add_argument("--headed", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=not args.headed)
|
||||
context = browser.new_context(
|
||||
record_har_path=args.har_path,
|
||||
record_har_content="embed", # keep response bodies in the HAR
|
||||
)
|
||||
page = context.new_page()
|
||||
page.goto(args.url, wait_until="domcontentloaded")
|
||||
for spec in args.action:
|
||||
run_action(page, spec)
|
||||
try:
|
||||
page.wait_for_load_state("networkidle", timeout=15000)
|
||||
except Exception:
|
||||
pass # some pages never fully idle; the trailing --wait covers it
|
||||
time.sleep(args.wait)
|
||||
context.close() # flushes the HAR
|
||||
browser.close()
|
||||
print(f"HAR written: {args.har_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Capture a HAR from a browser you connect to over CDP (not one you launch).
|
||||
|
||||
Use this when the browser is owned by someone else and only reachable over the
|
||||
Chrome DevTools Protocol: Hermes cloud backends (Browserbase, Browser-Use,
|
||||
Firecrawl), a Camofox session exposing CDP, or anything wired via
|
||||
`/browser connect <url>` / BROWSER_CDP_URL / browser.cdp_url in config.
|
||||
|
||||
Why this exists: Playwright's record_har_path only works on a context you
|
||||
launched locally. connect_over_cdp() attaches to an existing browser, so
|
||||
record_har is unavailable — we assemble the HAR from CDP Network.* events
|
||||
ourselves via page.on("request"/"response").
|
||||
|
||||
Usage:
|
||||
python3 har_capture_cdp.py <cdp_url> <output.har> [--wait S] \
|
||||
[--goto URL] [--action "fill:SEL:TEXT"] [--action "click:SEL"] ...
|
||||
|
||||
<cdp_url> is the ws:// or http:// CDP endpoint. For Hermes: run
|
||||
`/browser connect` to see the active endpoint, or read BROWSER_CDP_URL.
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
|
||||
def run_action(page, spec: str) -> None:
|
||||
parts = spec.split(":", 2)
|
||||
kind = parts[0]
|
||||
if kind == "fill":
|
||||
page.fill(parts[1], parts[2])
|
||||
elif kind == "press":
|
||||
page.press(parts[1], parts[2])
|
||||
elif kind == "click":
|
||||
page.click(parts[1])
|
||||
elif kind == "goto":
|
||||
page.goto(parts[1] + (":" + parts[2] if len(parts) > 2 else ""))
|
||||
elif kind == "sleep":
|
||||
time.sleep(float(parts[1]))
|
||||
else:
|
||||
raise ValueError(f"unknown action: {spec}")
|
||||
|
||||
|
||||
def _har_entry(req, resp):
|
||||
"""Build a minimal HAR entry from a Playwright request/response pair."""
|
||||
body_text, encoding = "", ""
|
||||
if resp is not None:
|
||||
try:
|
||||
raw = resp.body()
|
||||
try:
|
||||
body_text = raw.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
body_text = base64.b64encode(raw).decode("ascii")
|
||||
encoding = "base64"
|
||||
except Exception:
|
||||
pass
|
||||
post = req.post_data
|
||||
return {
|
||||
"_resourceType": req.resource_type,
|
||||
"request": {
|
||||
"method": req.method,
|
||||
"url": req.url,
|
||||
"headers": [{"name": k, "value": v} for k, v in req.headers.items()],
|
||||
"queryString": [], # har_to_client.py re-parses the URL, so leave empty
|
||||
"postData": {"mimeType": req.headers.get("content-type", ""),
|
||||
"text": post} if post else {},
|
||||
},
|
||||
"response": {
|
||||
"status": resp.status if resp else 0,
|
||||
"headers": [{"name": k, "value": v} for k, v in (resp.headers.items() if resp else [])],
|
||||
"content": {
|
||||
"mimeType": (resp.headers.get("content-type", "") if resp else ""),
|
||||
"text": body_text,
|
||||
**({"encoding": encoding} if encoding else {}),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("cdp_url")
|
||||
ap.add_argument("har_path")
|
||||
ap.add_argument("--goto", default=None, help="URL to navigate to after attaching")
|
||||
ap.add_argument("--wait", type=float, default=3.0)
|
||||
ap.add_argument("--action", action="append", default=[])
|
||||
args = ap.parse_args()
|
||||
|
||||
entries = []
|
||||
pending = {} # id(request) -> request
|
||||
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.connect_over_cdp(args.cdp_url)
|
||||
context = browser.contexts[0] if browser.contexts else browser.new_context()
|
||||
page = context.pages[0] if context.pages else context.new_page()
|
||||
|
||||
def on_request(req):
|
||||
pending[id(req)] = req
|
||||
|
||||
def on_response(resp):
|
||||
req = resp.request
|
||||
pending.pop(id(req), None)
|
||||
entries.append(_har_entry(req, resp))
|
||||
|
||||
page.on("request", on_request)
|
||||
page.on("response", on_response)
|
||||
|
||||
if args.goto:
|
||||
page.goto(args.goto, wait_until="domcontentloaded")
|
||||
for spec in args.action:
|
||||
run_action(page, spec)
|
||||
try:
|
||||
page.wait_for_load_state("networkidle", timeout=15000)
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(args.wait)
|
||||
|
||||
page.remove_listener("request", on_request)
|
||||
page.remove_listener("response", on_response)
|
||||
# Do NOT close: we connected to someone else's browser.
|
||||
|
||||
har = {"log": {"version": "1.2",
|
||||
"creator": {"name": "har_capture_cdp", "version": "0.1"},
|
||||
"entries": entries}}
|
||||
with open(args.har_path, "w", encoding="utf-8") as f:
|
||||
json.dump(har, f)
|
||||
print(f"HAR written: {args.har_path} ({len(entries)} entries)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Distill a HAR file into an API summary an agent can turn into a client.
|
||||
|
||||
Usage:
|
||||
python3 har_to_client.py <input.har> [--include-static] [--host SUBSTRING] [--max-body 600]
|
||||
|
||||
Filters to XHR/fetch/JSON traffic by default, groups by (method, host, path
|
||||
template), and prints per-endpoint: query params, interesting request headers,
|
||||
request body sample, response content-type/status, and a response body sample.
|
||||
Numeric/UUID-ish path segments are collapsed to {id} so repeated calls group.
|
||||
Also prints "### Replay hints": the browser User-Agent plus whether cookies or
|
||||
auth/token headers were present -- send those in the derived client or you may
|
||||
get a 403/401.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import OrderedDict
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
BORING_HEADERS = {
|
||||
"accept-encoding", "accept-language", "connection", "content-length",
|
||||
"host", "origin", "referer", "sec-ch-ua", "sec-ch-ua-mobile",
|
||||
"sec-ch-ua-platform", "sec-fetch-dest", "sec-fetch-mode", "sec-fetch-site",
|
||||
"user-agent", "pragma", "cache-control", "priority", "te",
|
||||
"upgrade-insecure-requests", "cookie",
|
||||
}
|
||||
ID_SEG = re.compile(r"^(\d+|[0-9a-f]{8}-[0-9a-f-]{27,}|[0-9a-f]{16,})$", re.I)
|
||||
STATIC_EXT = re.compile(r"\.(js|css|png|jpe?g|gif|svg|webp|ico|woff2?|ttf|mp4|map)$", re.I)
|
||||
|
||||
|
||||
def path_template(path: str) -> str:
|
||||
segs = path.split("/")
|
||||
return "/".join("{id}" if ID_SEG.match(s) else s for s in segs)
|
||||
|
||||
|
||||
def is_api_entry(entry: dict) -> bool:
|
||||
req = entry["request"]
|
||||
resp = entry.get("response", {})
|
||||
rtype = (entry.get("_resourceType") or "").lower()
|
||||
mime = (resp.get("content", {}).get("mimeType") or "").lower()
|
||||
if rtype in ("xhr", "fetch"):
|
||||
return True
|
||||
if "json" in mime:
|
||||
return True
|
||||
if req["method"] not in ("GET", "HEAD") and not STATIC_EXT.search(urlsplit(req["url"]).path):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def trunc(text, n: int) -> str:
|
||||
text = text if isinstance(text, str) else str(text)
|
||||
return text if len(text) <= n else text[:n] + f"... [{len(text)} chars total]"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("har")
|
||||
ap.add_argument("--include-static", action="store_true")
|
||||
ap.add_argument("--host", default=None, help="only endpoints whose host contains this")
|
||||
ap.add_argument("--max-body", type=int, default=600)
|
||||
args = ap.parse_args()
|
||||
|
||||
with open(args.har, encoding="utf-8") as f:
|
||||
har = json.load(f)
|
||||
|
||||
groups = OrderedDict()
|
||||
for entry in har["log"]["entries"]:
|
||||
req = entry["request"]
|
||||
url = urlsplit(req["url"])
|
||||
if url.scheme not in ("http", "https"):
|
||||
continue
|
||||
if args.host and args.host not in url.netloc:
|
||||
continue
|
||||
if not args.include_static:
|
||||
if STATIC_EXT.search(url.path) or not is_api_entry(entry):
|
||||
continue
|
||||
key = (req["method"], url.netloc, path_template(url.path))
|
||||
g = groups.setdefault(key, {"count": 0, "queries": set(), "headers": {},
|
||||
"req_body": None, "resp": None})
|
||||
g["count"] += 1
|
||||
for q in req.get("queryString", []):
|
||||
g["queries"].add((q["name"], trunc(q["value"], 80)))
|
||||
for h in req.get("headers", []):
|
||||
name = h["name"].lower().lstrip(":")
|
||||
if name in BORING_HEADERS or name in ("method", "path", "scheme", "authority"):
|
||||
continue
|
||||
g["headers"][name] = trunc(h["value"], 120)
|
||||
post = req.get("postData", {})
|
||||
if post.get("text") and g["req_body"] is None:
|
||||
g["req_body"] = (post.get("mimeType", ""), trunc(post["text"], args.max_body))
|
||||
resp = entry.get("response", {})
|
||||
if g["resp"] is None and resp:
|
||||
content = resp.get("content", {})
|
||||
g["resp"] = (resp.get("status"), content.get("mimeType", ""),
|
||||
trunc(content.get("text") or "", args.max_body))
|
||||
|
||||
if not groups:
|
||||
print("No API-looking entries found. Re-run with --include-static to see everything.")
|
||||
return 1
|
||||
|
||||
# Surface the browser identity so the replay client can match it (many
|
||||
# sites 403 a default library User-Agent).
|
||||
ua = None
|
||||
saw_cookie = saw_auth = False
|
||||
for entry in har["log"]["entries"]:
|
||||
for h in entry["request"].get("headers", []):
|
||||
n = h["name"].lower()
|
||||
if n == "user-agent" and ua is None:
|
||||
ua = h["value"]
|
||||
if n == "cookie":
|
||||
saw_cookie = True
|
||||
if n in ("authorization", "x-api-key") or "token" in n:
|
||||
saw_auth = True
|
||||
print("### Replay hints")
|
||||
if ua:
|
||||
print(f" User-Agent (send this): {ua}")
|
||||
if saw_cookie:
|
||||
print(" Cookies present -> session may be auth-gated; capture & resend the Cookie header.")
|
||||
if saw_auth:
|
||||
print(" Authorization/token header present -> extract and resend it.")
|
||||
|
||||
for (method, host, path), g in groups.items():
|
||||
print(f"\n=== {method} https://{host}{path} (x{g['count']})")
|
||||
if g["queries"]:
|
||||
print(" query params:")
|
||||
for name, val in sorted(g["queries"]):
|
||||
print(f" {name} = {val}")
|
||||
if g["headers"]:
|
||||
print(" request headers (non-boring):")
|
||||
for name, val in sorted(g["headers"].items()):
|
||||
print(f" {name}: {val}")
|
||||
if g["req_body"]:
|
||||
print(f" request body ({g['req_body'][0]}):")
|
||||
print(f" {g['req_body'][1]}")
|
||||
if g["resp"]:
|
||||
status, mime, body = g["resp"]
|
||||
print(f" response: {status} {mime}")
|
||||
if body:
|
||||
print(f" {body}")
|
||||
print(f"\n{len(groups)} distinct endpoints.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,190 @@
|
||||
---
|
||||
name: page-agent
|
||||
description: Embed an in-page natural-language GUI copilot in web apps.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [web, javascript, agent, browser, gui, alibaba, embed, copilot, saas]
|
||||
category: web-development
|
||||
---
|
||||
|
||||
# page-agent
|
||||
|
||||
alibaba/page-agent (https://github.com/alibaba/page-agent, 17k+ stars, MIT) is an in-page GUI agent written in TypeScript. It lives inside a webpage, reads the DOM as text (no screenshots, no multi-modal LLM), and executes natural-language instructions like "click the login button, then fill username as John" against the current page. Pure client-side — the host site just includes a script and passes an OpenAI-compatible LLM endpoint.
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Load this skill when a user wants to:
|
||||
|
||||
- **Ship an AI copilot inside their own web app** (SaaS, admin panel, B2B tool, ERP, CRM) — "users on my dashboard should be able to type 'create invoice for Acme Corp and email it' instead of clicking through five screens"
|
||||
- **Modernize a legacy web app** without rewriting the frontend — page-agent drops on top of existing DOM
|
||||
- **Add accessibility via natural language** — voice / screen-reader users drive the UI by describing what they want
|
||||
- **Demo or evaluate page-agent** against a local (Ollama) or hosted (Qwen, OpenAI, OpenRouter) LLM
|
||||
- **Build interactive training / product demos** — let an AI walk a user through "how to submit an expense report" live in the real UI
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- User wants **Hermes itself to drive a browser** → use Hermes' built-in browser tool (Browserbase / Camofox). page-agent is the *opposite* direction.
|
||||
- User wants **cross-tab automation without embedding** → use Playwright, browser-use, or the page-agent Chrome extension
|
||||
- User needs **visual grounding / screenshots** → page-agent is text-DOM only; use a multimodal browser agent instead
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node 22.13+ or 24+, npm 10+ (docs claim 11+ but 10.9 works fine)
|
||||
- An OpenAI-compatible LLM endpoint: Qwen (DashScope), OpenAI, Ollama, OpenRouter, or anything speaking `/v1/chat/completions`
|
||||
- Browser with devtools (for debugging)
|
||||
|
||||
## Path 1 — 30-second demo via CDN (no install)
|
||||
|
||||
Fastest way to see it work. Uses alibaba's free testing LLM proxy — **for evaluation only**, subject to their terms.
|
||||
|
||||
Add to any HTML page (or paste into the devtools console as a bookmarklet):
|
||||
|
||||
```html
|
||||
<script src="https://cdn.jsdelivr.net/npm/page-agent@1.8.0/dist/iife/page-agent.demo.js" crossorigin="true"></script>
|
||||
```
|
||||
|
||||
A panel appears. Type an instruction. Done.
|
||||
|
||||
Bookmarklet form (drop into bookmarks bar, click on any page):
|
||||
|
||||
```javascript
|
||||
javascript:(function(){var s=document.createElement('script');s.src='https://cdn.jsdelivr.net/npm/page-agent@1.8.0/dist/iife/page-agent.demo.js';document.head.appendChild(s);})();
|
||||
```
|
||||
|
||||
## Path 2 — npm install into your own web app (production use)
|
||||
|
||||
Inside an existing web project (React / Vue / Svelte / plain):
|
||||
|
||||
```bash
|
||||
npm install page-agent
|
||||
```
|
||||
|
||||
Wire it up with your own LLM endpoint — **never ship the demo CDN to real users**:
|
||||
|
||||
```javascript
|
||||
import { PageAgent } from 'page-agent'
|
||||
|
||||
const agent = new PageAgent({
|
||||
model: 'qwen3.5-plus',
|
||||
baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1',
|
||||
apiKey: process.env.LLM_API_KEY, // never hardcode
|
||||
language: 'en-US',
|
||||
})
|
||||
|
||||
// Show the panel for end users:
|
||||
agent.panel.show()
|
||||
|
||||
// Or drive it programmatically:
|
||||
await agent.execute('Click submit button, then fill username as John')
|
||||
```
|
||||
|
||||
Provider examples (any OpenAI-compatible endpoint works):
|
||||
|
||||
| Provider | `baseURL` | `model` |
|
||||
|----------|-----------|---------|
|
||||
| Qwen / DashScope | `https://dashscope.aliyuncs.com/compatible-mode/v1` | `qwen3.5-plus` |
|
||||
| OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` |
|
||||
| Ollama (local) | `http://localhost:11434/v1` | `qwen3:14b` |
|
||||
| OpenRouter | `https://openrouter.ai/api/v1` | `anthropic/claude-sonnet-4.6` |
|
||||
|
||||
**Key config fields** (passed to `new PageAgent({...})`):
|
||||
|
||||
- `model`, `baseURL`, `apiKey` — LLM connection
|
||||
- `language` — UI language (`en-US`, `zh-CN`, etc.)
|
||||
- Allowlist and data-masking hooks exist for locking down what the agent can touch — see https://alibaba.github.io/page-agent/ for the full option list
|
||||
|
||||
**Security.** Don't put your `apiKey` in client-side code for a real deployment — proxy LLM calls through your backend and point `baseURL` at your proxy. The demo CDN exists because alibaba runs that proxy for evaluation.
|
||||
|
||||
## Path 3 — clone the source repo (contributing, or hacking on it)
|
||||
|
||||
Use this when the user wants to modify page-agent itself, test it against arbitrary sites via a local IIFE bundle, or develop the browser extension.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/alibaba/page-agent.git
|
||||
cd page-agent
|
||||
npm ci # exact lockfile install (or `npm i` to allow updates)
|
||||
```
|
||||
|
||||
Create `.env` in the repo root with an LLM endpoint. Example:
|
||||
|
||||
```
|
||||
LLM_MODEL_NAME=gpt-4o-mini
|
||||
LLM_API_KEY=sk-...
|
||||
LLM_BASE_URL=https://api.openai.com/v1
|
||||
```
|
||||
|
||||
Ollama flavor:
|
||||
|
||||
```
|
||||
LLM_BASE_URL=http://localhost:11434/v1
|
||||
LLM_API_KEY=NA
|
||||
LLM_MODEL_NAME=qwen3:14b
|
||||
```
|
||||
|
||||
Common commands:
|
||||
|
||||
```bash
|
||||
npm start # docs/website dev server
|
||||
npm run build # build every package
|
||||
npm run dev:demo # serve IIFE bundle at http://localhost:5174/page-agent.demo.js
|
||||
npm run dev:ext # develop the browser extension (WXT + React)
|
||||
npm run build:ext # build the extension
|
||||
```
|
||||
|
||||
**Test on any website** using the local IIFE bundle. Add this bookmarklet:
|
||||
|
||||
```javascript
|
||||
javascript:(function(){var s=document.createElement('script');s.src=`http://localhost:5174/page-agent.demo.js?t=${Math.random()}`;s.onload=()=>console.log('PageAgent ready!');document.head.appendChild(s);})();
|
||||
```
|
||||
|
||||
Then: `npm run dev:demo`, click the bookmarklet on any page, and the local build injects. Auto-rebuilds on save.
|
||||
|
||||
**Warning:** your `.env` `LLM_API_KEY` is inlined into the IIFE bundle during dev builds. Don't share the bundle. Don't commit it. Don't paste the URL into Slack. (Verified: grepping the public dev bundle returns the literal values from `.env`.)
|
||||
|
||||
## Repo layout (Path 3)
|
||||
|
||||
Monorepo with npm workspaces. Key packages:
|
||||
|
||||
| Package | Path | Purpose |
|
||||
|---------|------|---------|
|
||||
| `page-agent` | `packages/page-agent/` | Main entry with UI panel |
|
||||
| `@page-agent/core` | `packages/core/` | Core agent logic, no UI |
|
||||
| `@page-agent/mcp` | `packages/mcp/` | MCP server (beta) |
|
||||
| — | `packages/llms/` | LLM client |
|
||||
| — | `packages/page-controller/` | DOM ops + visual feedback |
|
||||
| — | `packages/ui/` | Panel + i18n |
|
||||
| — | `packages/extension/` | Chrome/Firefox extension |
|
||||
| — | `packages/website/` | Docs + landing site |
|
||||
|
||||
## Verifying it works
|
||||
|
||||
After Path 1 or Path 2:
|
||||
1. Open the page in a browser with devtools open
|
||||
2. You should see a floating panel. If not, check the console for errors (most common: CORS on the LLM endpoint, wrong `baseURL`, or a bad API key)
|
||||
3. Type a simple instruction matching something visible on the page ("click the Login link")
|
||||
4. Watch the Network tab — you should see a request to your `baseURL`
|
||||
|
||||
After Path 3:
|
||||
1. `npm run dev:demo` prints `Accepting connections at http://localhost:5174`
|
||||
2. `curl -I http://localhost:5174/page-agent.demo.js` returns `HTTP/1.1 200 OK` with `Content-Type: application/javascript`
|
||||
3. Click the bookmarklet on any site; panel appears
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Demo CDN in production** — don't. It's rate-limited, uses alibaba's free proxy, and their terms forbid production use.
|
||||
- **API key exposure** — any key passed to `new PageAgent({apiKey: ...})` ships in your JS bundle. Always proxy through your own backend for real deployments.
|
||||
- **Non-OpenAI-compatible endpoints** fail silently or with cryptic errors. If your provider needs native Anthropic/Gemini formatting, use an OpenAI-compatibility proxy (LiteLLM, OpenRouter) in front.
|
||||
- **CSP blocks** — sites with strict Content-Security-Policy may refuse to load the CDN script or disallow inline eval. In that case, self-host from your origin.
|
||||
- **Restart dev server** after editing `.env` in Path 3 — Vite only reads env at startup.
|
||||
- **Node version** — the repo declares `^22.13.0 || >=24`. Node 20 will fail `npm ci` with engine errors.
|
||||
- **npm 10 vs 11** — docs say npm 11+; npm 10.9 actually works fine.
|
||||
|
||||
## Reference
|
||||
|
||||
- Repo: https://github.com/alibaba/page-agent
|
||||
- Docs: https://alibaba.github.io/page-agent/
|
||||
- License: MIT (built on browser-use's DOM processing internals, Copyright 2024 Gregor Zunic)
|
||||
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: publish-site
|
||||
description: Versioned site deploys to GitHub/Cloudflare/Netlify Pages.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent (Nous Research)
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [publish, deploy, hosting, github-pages, cloudflare-pages, netlify, static-site, versioning, rollback, web-development]
|
||||
category: web-development
|
||||
---
|
||||
|
||||
# Publish Site
|
||||
|
||||
Take a website, dashboard, or web app the user built (or you built for them) and put it online on infrastructure the user owns — GitHub Pages by default, Cloudflare Pages or Netlify when they need more. The discipline: preview locally for sign-off, version every deploy with a git tag, deploy through a provider ladder, verify the live URL with a real HTTP check, and keep rollback one command away.
|
||||
|
||||
This skill covers static sites and SPA build output (plain HTML/CSS/JS, or the `dist/`/`build/` folder from Vite/Next-export/Astro/etc.). It does not cover server-side runtimes — for throwaway serverless deploys with zero account setup, use the `cloudflare-temporary-deploy` optional skill instead.
|
||||
|
||||
## When to Use
|
||||
|
||||
Load this skill when the user asks to:
|
||||
|
||||
- **Put a site online** — "publish this", "host this somewhere", "give me a link I can share"
|
||||
- **Deploy a dashboard, report, portfolio, docs site, or prototype** you just generated
|
||||
- **Update an already-published site** with new content (redeploy = new version)
|
||||
- **Roll back** a bad deploy to the previous version
|
||||
- **Pick a host** — they don't care where, they just want a URL
|
||||
|
||||
## Prerequisites
|
||||
|
||||
At least ONE authenticated provider CLI (check in this order):
|
||||
|
||||
- **GitHub Pages (default):** `gh auth status` succeeds. Needs `git` too.
|
||||
- **Cloudflare Pages:** `wrangler whoami` succeeds (or `CLOUDFLARE_API_TOKEN` is set). Install: `npm i -g wrangler` or use `npx wrangler@latest`.
|
||||
- **Netlify (fallback):** `netlify status` succeeds. Install: `npm i -g netlify-cli`.
|
||||
|
||||
Plus:
|
||||
|
||||
- A directory of static output to publish (site root or a `dist/`/`build/` folder). If the project needs a build step, run it first and publish the output directory, never the source.
|
||||
- For local preview sharing: `cloudflared` (optional — `python3 -m http.server` covers local-only preview).
|
||||
|
||||
## How to Run
|
||||
|
||||
All commands below run via the `terminal` tool from the site's project directory. The pipeline is always the same five moves:
|
||||
|
||||
1. Build → 2. Preview for sign-off → 3. Commit + tag (version-before-deploy) → 4. Deploy via the provider ladder → 5. Verify the live URL with `curl` and report it.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Step | Command |
|
||||
|---|---|
|
||||
| Local preview | `python3 -m http.server 8080 --directory dist` |
|
||||
| Shareable preview | `cloudflared tunnel --url http://localhost:8080` |
|
||||
| Version a deploy | `git add -A && git commit -m "deploy: <what>" && git tag deploy-YYYYMMDD-HHMM` |
|
||||
| GitHub Pages (branch mode) | `git subtree push --prefix dist origin gh-pages` |
|
||||
| Enable Pages on repo | `gh api repos/{owner}/{repo}/pages -X POST -f 'source[branch]=gh-pages' -f 'source[path]=/'` |
|
||||
| Cloudflare Pages | `npx wrangler@latest pages deploy dist --project-name <name>` |
|
||||
| Netlify | `netlify deploy --prod --dir dist` |
|
||||
| Rollback | `git checkout <previous-tag> -- . && redeploy` (or provider dashboard) |
|
||||
| Verify live | `curl -sS -o /dev/null -w '%{http_code}' <url>` → expect `200` |
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Build and preview locally
|
||||
|
||||
Build if needed (`npm run build`, etc.) and identify the output directory. Serve it:
|
||||
|
||||
```bash
|
||||
python3 -m http.server 8080 --directory dist
|
||||
```
|
||||
|
||||
For a shareable preview link (user on another machine, or you want their sign-off before going live), open a quick tunnel in a background `terminal` session:
|
||||
|
||||
```bash
|
||||
cloudflared tunnel --url http://localhost:8080
|
||||
```
|
||||
|
||||
Give the user the `https://*.trycloudflare.com` URL and get sign-off before deploying. Kill the tunnel afterwards.
|
||||
|
||||
### 2. Version before deploy — no exceptions
|
||||
|
||||
Every deploy must come from a git commit, so every deploy is reproducible and rollback is trivial.
|
||||
|
||||
```bash
|
||||
git init 2>/dev/null; git add -A
|
||||
git commit -m "deploy: <short description>"
|
||||
git tag "deploy-$(date +%Y%m%d-%H%M)"
|
||||
```
|
||||
|
||||
If the project already has a repo, just commit + tag. Never deploy uncommitted files.
|
||||
|
||||
### 3. Deploy — provider ladder
|
||||
|
||||
**Rung 1 — GitHub Pages (default: free, zero extra accounts if `gh` is authed):**
|
||||
|
||||
```bash
|
||||
gh repo create <name> --public --source . --push # skip if repo exists
|
||||
git subtree push --prefix dist origin gh-pages # publish build output
|
||||
gh api "repos/{owner}/<name>/pages" -X POST \
|
||||
-f 'source[branch]=gh-pages' -f 'source[path]=/' # first time only
|
||||
```
|
||||
|
||||
Site appears at `https://<owner>.github.io/<name>/`. If the site is the repo root (no build dir), push `main` and set Pages source to `main` instead of using subtree. For build-step projects that will redeploy often, prefer the official `actions/deploy-pages` workflow so pushes auto-publish.
|
||||
|
||||
**Rung 2 — Cloudflare Pages (when the user wants a custom domain, redirects/headers, or Functions):**
|
||||
|
||||
```bash
|
||||
npx wrangler@latest pages deploy dist --project-name <name>
|
||||
```
|
||||
|
||||
First run creates the project and prints the `https://<name>.pages.dev` URL. Custom domains attach via the Cloudflare dashboard (Pages → project → Custom domains).
|
||||
|
||||
**Rung 3 — Netlify (fallback, or when the user already lives there):**
|
||||
|
||||
```bash
|
||||
netlify deploy --prod --dir dist
|
||||
```
|
||||
|
||||
`netlify deploy --dir dist` (no `--prod`) gives a draft URL — useful as a second preview stage.
|
||||
|
||||
### 4. Rollback
|
||||
|
||||
Rollback = redeploy a previous tag. Never hand-edit live output.
|
||||
|
||||
```bash
|
||||
git checkout deploy-<previous> -- . # or: git checkout deploy-<previous>; rebuild
|
||||
# then rerun the same deploy command from step 3
|
||||
```
|
||||
|
||||
Cloudflare Pages and Netlify also keep per-deploy history in their dashboards ("Rollback to this deploy"), which is faster when the CLI isn't handy.
|
||||
|
||||
### 5. Secrets and environment variables
|
||||
|
||||
- **NEVER commit secrets, API keys, or `.env` files** — they'd be public on Pages hosting. Check with `git status` before the first commit and keep `.env*` in `.gitignore`.
|
||||
- Runtime env vars belong in the provider's dashboard: Cloudflare Pages → Settings → Environment variables; Netlify → Site settings → Environment variables. GitHub Pages is static-only — no server env; anything embedded in the bundle is public by definition. Warn the user if their build inlines a key.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **SPA routes 404 on GitHub Pages.** Pages has no rewrite rules. Copy `index.html` to `404.html` in the output dir (`cp dist/index.html dist/404.html`) so client-side routing recovers. Cloudflare Pages and Netlify handle SPAs via `_redirects` (`/* /index.html 200`).
|
||||
- **GitHub Pages build lag.** The site can take 1–10 minutes to appear after the first enable, and ~1 minute per subsequent push. Don't declare failure on the first 404 — poll `curl` a few times before investigating.
|
||||
- **Case-sensitive paths.** Pages hosts are case-sensitive Linux; a site that worked on macOS/Windows can 404 on assets referenced as `Logo.PNG` but committed as `logo.png`. Grep the HTML for mismatched casing when an asset 404s.
|
||||
- **Project-page base path.** `https://<owner>.github.io/<name>/` serves under `/<name>/` — absolute asset URLs like `/app.js` break. Use relative paths or set the build tool's base (`vite build --base=/<name>/`).
|
||||
- **`wrangler` auth flow needs a browser.** `wrangler login` opens OAuth; in a headless session prefer `CLOUDFLARE_API_TOKEN` (user creates it at dash.cloudflare.com → API Tokens) and never echo the token into logs.
|
||||
- **DNS propagation on custom domains.** New CNAMEs can take minutes to hours. Verify against the provider's default URL (`*.pages.dev`, `*.netlify.app`, `*.github.io`) first, then check the custom domain separately — don't conflate the two failures.
|
||||
- **Deploying source instead of build output.** Publishing the repo root when the real site lives in `dist/` yields a directory listing or raw JSX. Always confirm the output dir contains an `index.html`.
|
||||
|
||||
## Verification
|
||||
|
||||
Do NOT report success from the deploy log alone. Before telling the user anything:
|
||||
|
||||
1. `curl -sS -o /dev/null -w '%{http_code}' <live-url>` returns `200` (retry over ~2 minutes for a first GitHub Pages deploy).
|
||||
2. `curl -sS <live-url> | head -30` shows the expected `index.html` content — optionally confirm markup with `web_extract` on the live URL.
|
||||
3. For SPAs, also curl one deep route (e.g. `/about`) and confirm it returns `200`, not `404`.
|
||||
4. `git tag --list 'deploy-*'` shows the tag for this deploy.
|
||||
|
||||
Then report the live URL to the user, along with the deploy tag they can roll back to.
|
||||
Reference in New Issue
Block a user