Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license
This commit is contained in:
@@ -0,0 +1,999 @@
|
||||
/**
|
||||
* E2E at-rest contract for the remote-gateway session token (issue #77486).
|
||||
*
|
||||
* The reported bug: configuring a remote gateway persisted the dashboard
|
||||
* session token as PLAINTEXT into `connection.json` under the app's userData
|
||||
* dir (macOS `~/Library/Application Support/Hermes/connection.json`, Windows
|
||||
* `AppData\Roaming\Hermes\connection.json`). Anything that can read the file
|
||||
* — a backup, a sync client, another local process, a support bundle — got a
|
||||
* live gateway credential.
|
||||
*
|
||||
* The contract these tests encode is deliberately stated WITHOUT naming a
|
||||
* storage strategy:
|
||||
*
|
||||
* 1. ABSENT FROM DISK. After the app has been configured with a remote
|
||||
* gateway token, the token's plaintext value must not appear anywhere in
|
||||
* `connection.json`, in any sibling file the app writes under userData,
|
||||
* or in HERMES_HOME (logs included).
|
||||
* 2. STILL FUNCTIONAL. After a restart, the app must still be able to USE
|
||||
* that credential — it decrypts the stored blob and puts the exact
|
||||
* original token on the wire.
|
||||
* 3. UNREADABLE BY OTHER LOCAL ACCOUNTS. `connection.json` must not be
|
||||
* group/other-accessible, whether the app just wrote it or inherited it
|
||||
* from an older install.
|
||||
*
|
||||
* All three matter and none is sufficient alone. (1) alone is trivially
|
||||
* satisfied by a "fix" that drops the token on the floor; (2) alone is
|
||||
* satisfied by the bug itself. So (2) is verified through the app's own
|
||||
* connection test against a fake gateway that records the
|
||||
* `X-Hermes-Session-Token` header it receives — a dropped or mangled token
|
||||
* cannot produce that header.
|
||||
*
|
||||
* (3) is orthogonal to (1) and invisible to it: safeStorage keeps the token
|
||||
* opaque no matter what the file's mode is, so a 0644 `connection.json` passes
|
||||
* the raw-bytes scan every time while still exposing the ciphertext blob, the
|
||||
* gateway URL and the SSH host/user/keyPath to any other local account. It is
|
||||
* asserted explicitly (see `expectOwnerOnlyMode`) because no amount of
|
||||
* encryption evidence implies it.
|
||||
*
|
||||
* We deliberately do NOT assert `encoding === 'safeStorage'` or any other
|
||||
* shape of the stored blob. That would be a change-detector: a fix that moved
|
||||
* to the OS keychain proper, to an async safeStorage provider, or to a
|
||||
* separate credential file would break the test while being *more* correct.
|
||||
* The load-bearing assertion is the raw-bytes absence of the secret.
|
||||
*
|
||||
* Four at-rest paths, hence four tests — three enforced, one a documented gap:
|
||||
*
|
||||
* 1. A NEWLY configured token (ACTIVE). The app's own write path routes
|
||||
* through the strict `encryptDesktopSecret`; this test holds it there
|
||||
* against regression, and pins the mode of the file it actually wrote.
|
||||
* 2. An EXISTING `connection.json` at the old 0644 (ACTIVE). Covers the
|
||||
* read-side tighten, and ONLY the mode — its token is already ciphertext,
|
||||
* which is what keeps it independent of the migration test 4 defers.
|
||||
* 3. A CORRUPT `connection.json` at 0644 (ACTIVE). The tighten must not be
|
||||
* gated on the parse succeeding: a truncated file still holds the token
|
||||
* bytes, and the parse failure is swallowed, so nothing would ever come
|
||||
* back for it.
|
||||
* 4. An EXISTING plaintext `connection.json` (`test.fixme`). Legacy payloads
|
||||
* are deliberately NOT migrated yet. The test is kept, disabled, with a
|
||||
* precise reason — see the block comment above it.
|
||||
*
|
||||
* ── Correcting the record on test 4 ─────────────────────────────────────
|
||||
*
|
||||
* An earlier revision of this file asserted that migration and justified it by
|
||||
* claiming the first implementation (`d3d177283`) fell back to
|
||||
* `{ encoding: 'plain', value }` when `isEncryptionAvailable()` was false.
|
||||
* That citation is FALSE for this codebase. What is actually true:
|
||||
*
|
||||
* git merge-base --is-ancestor d3d1772837a7b0552940b55455ae734c72e0a8f1 HEAD -> 1 (NOT an ancestor)
|
||||
* git merge-base --is-ancestor 51c68d4ab1a9e3c62fb1048fccb84144c409f0e7 HEAD -> 0 (IS an ancestor)
|
||||
* git log -S 'Fall through to plaintext' upstream/main -- apps/desktop -> (no commits)
|
||||
*
|
||||
* `d3d177283` exists only on `upstream/bb/gui-mainmerge-tmp`,
|
||||
* `brooklyn/gui-installer-prereqs`, and the `desktop-pr20059-installers`
|
||||
* pre-release tag. Mainline NEVER shipped a code path that wrote a plaintext
|
||||
* gateway token: `51c68d4ab` ("Add Hermes desktop app (#20059)"), the commit
|
||||
* that brought the desktop app to mainline, already contained the strict
|
||||
* throw ("Secure token storage is unavailable, …") in `hardening.cjs`.
|
||||
*
|
||||
* One `{ encoding: 'plain', value }` literal does remain on mainline
|
||||
* (`electron/main.ts`, in `coerceDesktopConnectionConfig`), but it is
|
||||
* unreachable as an at-rest write: it is gated on `persistToken === false`,
|
||||
* whose only caller is the connection-TEST handler, which never calls
|
||||
* `writeDesktopConnectionConfig`. That token stays in memory for the duration
|
||||
* of one probe.
|
||||
*
|
||||
* So the affected population is not "anyone who configured a gateway before
|
||||
* the fix". It is narrow and non-mainline: pre-release `bb/gui` installs
|
||||
* (including the `desktop-pr20059-installers` build) plus hand-edited or
|
||||
* hand-migrated `connection.json` files. Those files DO still work, because
|
||||
* `decryptDesktopSecret` returns any non-safeStorage `value` verbatim on read
|
||||
* — the read path is intentionally unchanged, so nobody is signed out. That
|
||||
* read-path acceptance, not a mainline writer, is what makes the fixme'd
|
||||
* fixture realistic.
|
||||
*
|
||||
* Migration is DEFERRED, not forgotten. An adversarial review of the
|
||||
* migration that briefly lived here returned DO NOT SHIP, having reproduced
|
||||
* two token-loss scenarios: it silently reverts and then destroys the opt-in
|
||||
* plaintext choice that open upstream PR #62319 deliberately adds; and it
|
||||
* converts a portable credential into a keychain-bound one with no consent,
|
||||
* destroying the only recoverable copy while not actually remediating the
|
||||
* exposure (the plaintext is already in backups, so the real remedy is
|
||||
* ROTATION). The prerequisites are enumerated above test 2.
|
||||
*
|
||||
* Environment limits are encoded rather than papered over. Electron's
|
||||
* safeStorage is unavailable on Linux with no keyring, which is the shape of
|
||||
* this suite's CI runner (ubuntu-latest, see .github/workflows/e2e-desktop.yml).
|
||||
* The absence assertion is unconditional there — it is the security
|
||||
* requirement, and it must hold in every environment. Only the *other* half is
|
||||
* conditional: with secure storage the save must succeed, and without it the
|
||||
* save must fail loudly (which is what the current strict `encryptDesktopSecret`
|
||||
* does) instead of quietly writing plaintext. See the branch comments in each
|
||||
* test for the reasoning, including the one case this spec refuses to invent a
|
||||
* policy for.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs'
|
||||
import * as http from 'node:http'
|
||||
import type { AddressInfo } from 'node:net'
|
||||
import * as path from 'node:path'
|
||||
|
||||
import { buildAppEnv, createSandbox, launchDesktop, type Sandbox } from './fixtures'
|
||||
import { allowErrorBanners, type ElectronApplication, expect, type Page, test } from './test'
|
||||
|
||||
/**
|
||||
* The secret under test. Long, random-looking, and unique to this spec so a
|
||||
* raw-bytes scan cannot produce a false negative by colliding with ordinary
|
||||
* config content. Kept to `[A-Za-z0-9-]` on purpose: encodeURIComponent() is
|
||||
* the identity function over this alphabet, so the raw-bytes needle also
|
||||
* covers the URL-encoded form the WS dialer builds (`?token=…`).
|
||||
*/
|
||||
const SENTINEL_TOKEN = 'hermes-e2e-at-rest-sentinel-Zq7Z4hV9nX2pL8sK3tB6wR1yM5jD0fG'
|
||||
|
||||
/** Skip absurdly large files during the leak scan (Chromium caches). */
|
||||
const MAX_SCAN_BYTES = 16 * 1024 * 1024
|
||||
|
||||
/**
|
||||
* One fixed Electron app name for this spec, instead of the timestamped one
|
||||
* `buildAppEnv` generates. On macOS the safeStorage keychain item is derived
|
||||
* from the app name, so a per-launch name would (a) make the post-restart
|
||||
* decrypt fail for the wrong reason and (b) leave a fresh keychain entry on
|
||||
* the developer's login keychain on every run. Safe because the suite runs
|
||||
* one worker at a time and both launches here are sequential; the
|
||||
* single-instance lock keys off userData, which is per-sandbox.
|
||||
*/
|
||||
const STABLE_APP_NAME = 'HermesE2EAtRestStorage'
|
||||
|
||||
// ─── Fake gateway ───────────────────────────────────────────────────────
|
||||
|
||||
interface FakeGateway {
|
||||
url: string
|
||||
/** Every `X-Hermes-Session-Token` value the app has sent us. */
|
||||
sessionTokens: string[]
|
||||
close: () => Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* A minimal stand-in for a remote Hermes gateway. It serves the public
|
||||
* `/api/status` probe (which the desktop connection test hits first, with the
|
||||
* session token in a header) and refuses the WebSocket upgrade immediately so
|
||||
* the second leg of the connection test fails fast instead of burning the
|
||||
* probe's 10s connect timeout. We only care about the header it captured.
|
||||
*
|
||||
* The e2e mock-server is an OpenAI-compatible *inference* mock, not a gateway,
|
||||
* so it cannot answer /api/status — hence this small local server.
|
||||
*/
|
||||
async function startFakeGateway(): Promise<FakeGateway> {
|
||||
const sessionTokens: string[] = []
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
const token = req.headers['x-hermes-session-token']
|
||||
|
||||
if (typeof token === 'string' && token) {
|
||||
sessionTokens.push(token)
|
||||
}
|
||||
|
||||
if (req.url?.startsWith('/api/status')) {
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' })
|
||||
res.end(JSON.stringify({ auth_required: false, ok: true, version: '0.0.0-e2e-fake' }))
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
res.writeHead(404, { 'Content-Type': 'application/json' })
|
||||
res.end(JSON.stringify({ detail: 'not found' }))
|
||||
})
|
||||
|
||||
// Refuse the WS leg at once: the connection test's WS probe should return a
|
||||
// fast failure rather than hang. The status header is already captured.
|
||||
server.on('upgrade', (req, socket) => {
|
||||
const token = new URL(req.url ?? '/', 'http://127.0.0.1').searchParams.get('token')
|
||||
|
||||
if (token) {
|
||||
sessionTokens.push(token)
|
||||
}
|
||||
|
||||
socket.destroy()
|
||||
})
|
||||
|
||||
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
|
||||
|
||||
const { port } = server.address() as AddressInfo
|
||||
|
||||
return {
|
||||
close: () =>
|
||||
new Promise<void>(resolve => {
|
||||
server.closeAllConnections?.()
|
||||
server.close(() => resolve())
|
||||
}),
|
||||
sessionTokens,
|
||||
url: `http://127.0.0.1:${port}`,
|
||||
}
|
||||
}
|
||||
|
||||
// ─── On-disk leak scanning ──────────────────────────────────────────────
|
||||
|
||||
interface Needle {
|
||||
bytes: Buffer
|
||||
label: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The forms a leak could take. Raw bytes, not JSON.parse + field inspection:
|
||||
* the point is that the secret is nowhere in the file — including inside a
|
||||
* nested field, a cached WS URL, or a field name nobody thought to check.
|
||||
*
|
||||
* The base64 needle catches the cheapest wrong "fix": base64 is an encoding,
|
||||
* not encryption, so a token that is merely base64'd is still plaintext at
|
||||
* rest. A real ciphertext will contain neither needle.
|
||||
*/
|
||||
function secretNeedles(secret: string): Needle[] {
|
||||
return [
|
||||
{ bytes: Buffer.from(secret, 'utf8'), label: 'plaintext' },
|
||||
{ bytes: Buffer.from(Buffer.from(secret, 'utf8').toString('base64'), 'utf8'), label: 'base64' },
|
||||
]
|
||||
}
|
||||
|
||||
/** Relative paths of every file under `root` whose bytes contain a needle. */
|
||||
function scanTreeForSecret(root: string, needles: Needle[]): string[] {
|
||||
const hits: string[] = []
|
||||
|
||||
const walk = (dir: string): void => {
|
||||
let entries: fs.Dirent[]
|
||||
|
||||
try {
|
||||
entries = fs.readdirSync(dir, { withFileTypes: true })
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
const full = path.join(dir, entry.name)
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
walk(full)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if (!entry.isFile()) {
|
||||
continue
|
||||
}
|
||||
|
||||
try {
|
||||
if (fs.statSync(full).size > MAX_SCAN_BYTES) {
|
||||
continue
|
||||
}
|
||||
} catch {
|
||||
continue
|
||||
}
|
||||
|
||||
let buf: Buffer
|
||||
|
||||
try {
|
||||
buf = fs.readFileSync(full)
|
||||
} catch {
|
||||
continue
|
||||
}
|
||||
|
||||
for (const needle of needles) {
|
||||
if (buf.includes(needle.bytes)) {
|
||||
hits.push(`${path.relative(root, full)} [${needle.label}]`)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
walk(root)
|
||||
|
||||
return hits
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a file's bytes, or an empty buffer when it does not exist. A correct
|
||||
* fix is allowed to delete/replace `connection.json` rather than rewrite it,
|
||||
* and the refusal path may never create it at all — neither should crash the
|
||||
* scan before its assertion runs.
|
||||
*/
|
||||
function readIfExists(filePath: string): Buffer {
|
||||
try {
|
||||
return fs.readFileSync(filePath)
|
||||
} catch {
|
||||
return Buffer.alloc(0)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The stored token's `encoding` tag, for diagnostics only — never its value.
|
||||
* Reported on failure so a red run says *why* (e.g. still `plain`) instead of
|
||||
* only that a scan matched. Deliberately NOT an assertion: which encoding a
|
||||
* correct fix chooses is its own business.
|
||||
*/
|
||||
function storedTokenEncoding(connectionFile: string): string {
|
||||
try {
|
||||
const parsed = JSON.parse(readIfExists(connectionFile).toString('utf8'))
|
||||
|
||||
return String(parsed?.remote?.token?.encoding ?? '<none>')
|
||||
} catch {
|
||||
return '<unparsable>'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert a credential file is not readable or writable by group/other.
|
||||
*
|
||||
* This is the one contract the raw-bytes scan above structurally cannot see:
|
||||
* safeStorage keeps the token opaque regardless of the file's mode, so a
|
||||
* world-readable `connection.json` passes every absence assertion in this file
|
||||
* while still handing the URL, the SSH host/user/keyPath, and the ciphertext
|
||||
* blob to any other local account. Encryption and permissions are independent
|
||||
* halves of "at rest", and only one of them was covered here.
|
||||
*
|
||||
* Asserted as `mode & 0o077 === 0` rather than `=== 0o600`: the requirement is
|
||||
* that nobody else can reach the file, and pinning the exact bits would make
|
||||
* this a change-detector against a future 0400 or a setgid-dir umask.
|
||||
*
|
||||
* POSIX only. `tightenSecretFileMode` no-ops on Windows deliberately (Node maps
|
||||
* chmod to the read-only bit there, and userData is already ACL'd to the user
|
||||
* profile — see the docstring in electron/hardening.ts, and PR #77527 for the
|
||||
* one place ACLs are being handled). Mode bits are advisory on Windows, so
|
||||
* asserting them would go red for behaviour the fix never claimed. The suite
|
||||
* runs ubuntu-latest today (.github/workflows/e2e-desktop.yml); nothing else in
|
||||
* this spec is platform-specific, and this assertion should not be what
|
||||
* changes that.
|
||||
*/
|
||||
function expectOwnerOnlyMode(filePath: string, why: string): void {
|
||||
if (process.platform === 'win32') {
|
||||
return
|
||||
}
|
||||
|
||||
const mode = fs.statSync(filePath).mode & 0o777
|
||||
|
||||
expect(mode & 0o077, `${why} (mode ${mode.toString(8)})`).toBe(0)
|
||||
}
|
||||
|
||||
// ─── App helpers ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Launch the desktop app against `sandbox` with a fake boot failure injected.
|
||||
*
|
||||
* The credential path we are testing is entirely main-process (IPC handler →
|
||||
* coerce → safeStorage → userData write) and does not need a live agent
|
||||
* backend, so we skip spawning `hermes serve` (no Python needed, ~3s launch,
|
||||
* hermetic). This is also a real user situation rather than an artificial one:
|
||||
* the boot-failure overlay's own recovery affordance is "Connection settings",
|
||||
* i.e. pointing the app at a remote gateway is exactly what a user does from
|
||||
* this state. BOOT_FAKE_ERROR short-circuits startHermes() *before* remote
|
||||
* resolution, so no launch ever dials the fake gateway on its own.
|
||||
*/
|
||||
async function launchAgainst(sandbox: Sandbox): Promise<{ app: ElectronApplication; page: Page }> {
|
||||
const env = buildAppEnv(sandbox, {
|
||||
HERMES_DESKTOP_APP_NAME: STABLE_APP_NAME,
|
||||
HERMES_DESKTOP_BOOT_FAKE_ERROR: 'E2E at-rest storage spec: local backend intentionally not started',
|
||||
})
|
||||
|
||||
const { app, page } = await launchDesktop(env)
|
||||
|
||||
// The capability bridge is what we drive; it lands with the preload, well
|
||||
// before the app would be "ready" in the boot sense.
|
||||
await page.waitForFunction(
|
||||
() => Boolean((window as unknown as { hermesDesktop?: Record<string, unknown> }).hermesDesktop?.saveConnectionConfig),
|
||||
undefined,
|
||||
{ timeout: 60_000 },
|
||||
)
|
||||
|
||||
return { app, page }
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask the running app where userData actually is, the same way the app does
|
||||
* (`app.getPath('userData')`). The fixtures point userData at a temp sandbox,
|
||||
* so a home-relative hardcoded path would test the wrong file — or no file.
|
||||
*/
|
||||
async function resolveUserDataDir(app: ElectronApplication): Promise<string> {
|
||||
return app.evaluate(({ app: electronApp }) => electronApp.getPath('userData'))
|
||||
}
|
||||
|
||||
interface SafeStorageCapability {
|
||||
available: boolean
|
||||
backend: string
|
||||
}
|
||||
|
||||
/**
|
||||
* What secure storage is actually capable of on THIS host, asked after ready
|
||||
* (on Linux the answer is meaningless before then).
|
||||
*
|
||||
* `backend` matters for the honest reading of a green run: on Linux with no
|
||||
* keyring, Electron can still report encryption as available while selecting
|
||||
* the `basic_text` backend, which encrypts with a hardcoded password — the
|
||||
* bytes on disk are not the plaintext, but they are not meaningfully
|
||||
* protected either. We record it rather than assert on it, because which
|
||||
* posture Hermes should take there (refuse to save vs. accept basic_text) is
|
||||
* a product decision, not something this test should silently ratify.
|
||||
*/
|
||||
async function readSafeStorageCapability(app: ElectronApplication): Promise<SafeStorageCapability> {
|
||||
return app.evaluate(async ({ app: electronApp, safeStorage }) => {
|
||||
await electronApp.whenReady()
|
||||
|
||||
let available = false
|
||||
let backend = 'unavailable'
|
||||
|
||||
try {
|
||||
available = safeStorage.isEncryptionAvailable()
|
||||
} catch {
|
||||
available = false
|
||||
}
|
||||
|
||||
try {
|
||||
// Linux-oriented API; other platforms may not implement it.
|
||||
backend = safeStorage.getSelectedStorageBackend?.() ?? 'n/a'
|
||||
} catch {
|
||||
backend = 'n/a'
|
||||
}
|
||||
|
||||
return { available, backend }
|
||||
})
|
||||
}
|
||||
|
||||
interface SaveOutcome {
|
||||
config: { remoteTokenPreview?: null | string; remoteTokenSet?: boolean; remoteUrl?: string } | null
|
||||
error: null | string
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive the app's REAL save surface: the same `saveConnectionConfig` payload
|
||||
* Settings → Gateway sends (see src/app/settings/gateway-settings.tsx). We use
|
||||
* save rather than apply so the app persists the credential without trying to
|
||||
* re-home onto the fake gateway.
|
||||
*/
|
||||
async function saveRemoteToken(page: Page, remoteUrl: string, remoteToken?: string): Promise<SaveOutcome> {
|
||||
return page.evaluate(
|
||||
async ([url, token]) => {
|
||||
const desktop = (window as unknown as { hermesDesktop: any }).hermesDesktop
|
||||
|
||||
try {
|
||||
const config = await desktop.saveConnectionConfig({
|
||||
mode: 'remote',
|
||||
remoteAuthMode: 'token',
|
||||
...(token ? { remoteToken: token } : {}),
|
||||
remoteUrl: url,
|
||||
})
|
||||
|
||||
return { config, error: null }
|
||||
} catch (error) {
|
||||
return { config: null, error: error instanceof Error ? error.message : String(error) }
|
||||
}
|
||||
},
|
||||
[remoteUrl, remoteToken ?? ''] as const,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Make the app USE the stored credential. No token in the payload, so the main
|
||||
* process must read `connection.json`, decrypt what it stored, and put the
|
||||
* plaintext on the wire itself. `buildRemoteBlock` throws "Remote gateway
|
||||
* session token is required." when the stored blob no longer decrypts, so a
|
||||
* fix that dropped the token fails here instead of quietly passing the
|
||||
* absence assertion.
|
||||
*/
|
||||
async function exerciseStoredToken(page: Page, remoteUrl: string): Promise<{ error: null | string }> {
|
||||
return page.evaluate(async url => {
|
||||
const desktop = (window as unknown as { hermesDesktop: any }).hermesDesktop
|
||||
|
||||
try {
|
||||
await desktop.testConnectionConfig({ mode: 'remote', remoteUrl: url })
|
||||
|
||||
return { error: null }
|
||||
} catch (error) {
|
||||
// A failing WS leg is expected (the fake gateway refuses the upgrade).
|
||||
// The assertion is on what the gateway received, not on this result.
|
||||
return { error: error instanceof Error ? error.message : String(error) }
|
||||
}
|
||||
}, remoteUrl)
|
||||
}
|
||||
|
||||
// ─── Tests ──────────────────────────────────────────────────────────────
|
||||
|
||||
let gateway: FakeGateway | null = null
|
||||
let sandbox: Sandbox | null = null
|
||||
let app: ElectronApplication | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
gateway = await startFakeGateway()
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await gateway?.close()
|
||||
gateway = null
|
||||
})
|
||||
|
||||
test.beforeEach(() => {
|
||||
// Boot is intentionally failed in this spec (see launchAgainst), so the
|
||||
// boot-failure overlay's error banner is expected, not a failure.
|
||||
allowErrorBanners()
|
||||
})
|
||||
|
||||
test.afterEach(async () => {
|
||||
await app?.close().catch(() => undefined)
|
||||
app = null
|
||||
sandbox?.cleanup()
|
||||
sandbox = null
|
||||
})
|
||||
|
||||
test.describe('remote gateway session token at rest', () => {
|
||||
test('with keychain encryption opted IN, a newly configured token is never written to userData in plaintext, and still works after restart', async () => {
|
||||
const fake = gateway!
|
||||
sandbox = createSandbox('at-rest-fresh')
|
||||
|
||||
// Keychain-backed encryption is opt-in (default OFF — see
|
||||
// electron/secret-storage-policy.ts). This test covers the opted-IN
|
||||
// posture, so seed the policy the way the Settings toggle writes it.
|
||||
fs.writeFileSync(
|
||||
path.join(sandbox.userDataDir, 'secure-token-storage.json'),
|
||||
JSON.stringify({ migrated: true, on: true }),
|
||||
'utf8',
|
||||
)
|
||||
|
||||
const first = await launchAgainst(sandbox)
|
||||
app = first.app
|
||||
|
||||
const capability = await readSafeStorageCapability(app)
|
||||
const userDataDir = await resolveUserDataDir(app)
|
||||
const connectionFile = path.join(userDataDir, 'connection.json')
|
||||
|
||||
test.info().annotations.push({
|
||||
description: `isEncryptionAvailable=${capability.available} backend=${capability.backend}`,
|
||||
type: 'safeStorage',
|
||||
})
|
||||
|
||||
const saved = await saveRemoteToken(first.page, fake.url, SENTINEL_TOKEN)
|
||||
|
||||
// Defined degradation, not a silent plaintext write. Where secure storage
|
||||
// works, the save must succeed. Where it genuinely does not (headless
|
||||
// Linux with no keyring, per Electron's safeStorage docs), refusing the
|
||||
// save with a loud error is an acceptable outcome — what is NEVER
|
||||
// acceptable is reporting success while leaving the secret readable on
|
||||
// disk. The absence assertion below runs in both branches.
|
||||
if (capability.available) {
|
||||
expect(
|
||||
saved.error,
|
||||
'secure storage is available on this host, so saving a remote gateway token must succeed',
|
||||
).toBeNull()
|
||||
expect(saved.config?.remoteTokenSet).toBe(true)
|
||||
} else {
|
||||
expect(
|
||||
saved.error,
|
||||
'secure storage is unavailable, so the save must fail loudly rather than persist a plaintext token',
|
||||
).not.toBeNull()
|
||||
}
|
||||
|
||||
// Guard against a vacuous pass: when the save succeeded, the artifact must
|
||||
// exist and must be the file the app really wrote for THIS connection.
|
||||
// Without this, "no plaintext on disk" would also be true if nothing had
|
||||
// been saved at all. Only asserted on the success branch — a refused save
|
||||
// legitimately leaves no file behind.
|
||||
const rawConnection = readIfExists(connectionFile)
|
||||
|
||||
if (capability.available) {
|
||||
expect(fs.existsSync(connectionFile), `expected the app to write ${connectionFile}`).toBe(true)
|
||||
expect(
|
||||
rawConnection.includes(Buffer.from(fake.url, 'utf8')),
|
||||
'connection.json should record the configured gateway URL (proves this is the real artifact)',
|
||||
).toBe(true)
|
||||
|
||||
// The write path's OTHER half of at-rest: opaque bytes AND owner-only
|
||||
// permissions. Deliberately here, on the file this test just proved the
|
||||
// app really wrote, rather than in a unit test — nothing in the repo
|
||||
// imports electron/main.ts (it imports electron), so this is the only
|
||||
// place that can witness the app's own write actually going out at 0600
|
||||
// instead of the 0644 umask default.
|
||||
expectOwnerOnlyMode(
|
||||
connectionFile,
|
||||
'connection.json is group/other-accessible, so the encrypted token blob, gateway URL and SSH fields are readable by other local accounts',
|
||||
)
|
||||
}
|
||||
|
||||
// ── The load-bearing assertion ─────────────────────────────────────
|
||||
const needles = secretNeedles(SENTINEL_TOKEN)
|
||||
|
||||
const connectionHits = needles.filter(needle => rawConnection.includes(needle.bytes)).map(needle => needle.label)
|
||||
expect(
|
||||
connectionHits,
|
||||
`the gateway session token must not be recoverable from ${connectionFile} ` +
|
||||
`(stored token encoding is "${storedTokenEncoding(connectionFile)}")`,
|
||||
).toEqual([])
|
||||
|
||||
// …and not in any sibling file the app writes alongside it, nor in
|
||||
// HERMES_HOME (desktop.log lives there).
|
||||
expect(
|
||||
scanTreeForSecret(userDataDir, needles),
|
||||
'the gateway session token leaked into a userData file',
|
||||
).toEqual([])
|
||||
expect(
|
||||
scanTreeForSecret(sandbox.hermesHome, needles),
|
||||
'the gateway session token leaked into a HERMES_HOME file (logs included)',
|
||||
).toEqual([])
|
||||
|
||||
if (!capability.available) {
|
||||
// Nothing was stored, so there is no round trip to verify. The refusal
|
||||
// itself was already asserted above.
|
||||
return
|
||||
}
|
||||
|
||||
// ── Secondary: the credential must still be USABLE ─────────────────
|
||||
// Restart against the same userData so the token comes off disk, not out
|
||||
// of a live process's memory.
|
||||
await app.close().catch(() => undefined)
|
||||
app = null
|
||||
|
||||
const second = await launchAgainst(sandbox)
|
||||
app = second.app
|
||||
|
||||
expect(
|
||||
await resolveUserDataDir(app),
|
||||
'the restarted app must resolve the same userData dir, or this is not a round trip',
|
||||
).toBe(userDataDir)
|
||||
|
||||
const reread = await second.page.evaluate(async () => {
|
||||
const desktop = (window as unknown as { hermesDesktop: any }).hermesDesktop
|
||||
|
||||
return desktop.getConnectionConfig()
|
||||
})
|
||||
|
||||
expect(reread.remoteTokenSet, 'the stored token must survive a restart').toBe(true)
|
||||
expect(reread.remoteUrl).toBe(fake.url)
|
||||
|
||||
const before = fake.sessionTokens.length
|
||||
await exerciseStoredToken(second.page, fake.url)
|
||||
|
||||
// The gateway is the witness: the app decrypted its stored blob and put
|
||||
// the original secret on the wire. A dropped, truncated, or re-encoded
|
||||
// token cannot produce this.
|
||||
expect(
|
||||
fake.sessionTokens.slice(before),
|
||||
'the app must send the exact stored token to the gateway after a restart',
|
||||
).toContain(SENTINEL_TOKEN)
|
||||
})
|
||||
|
||||
/**
|
||||
* The DEFAULT posture: keychain encryption opted out (no policy file at
|
||||
* all). Saving a token must (a) succeed without ever touching safeStorage
|
||||
* — this is the whole point of the opt-in: no macOS Keychain dialog on
|
||||
* machines with a broken login keychain — (b) store the token with a
|
||||
* non-safeStorage encoding at 0600, and (c) round-trip it across a
|
||||
* restart. The plaintext-on-disk trade-off is the user's chosen (default)
|
||||
* mode; owner-only file bits remain the at-rest boundary.
|
||||
*/
|
||||
test('with the default policy (no keychain), a token saves without secure storage, is owner-only on disk, and survives a restart', async () => {
|
||||
const fake = gateway!
|
||||
sandbox = createSandbox('at-rest-default')
|
||||
|
||||
const first = await launchAgainst(sandbox)
|
||||
app = first.app
|
||||
|
||||
const userDataDir = await resolveUserDataDir(app)
|
||||
const connectionFile = path.join(userDataDir, 'connection.json')
|
||||
|
||||
// Must succeed regardless of host keyring state — the default policy
|
||||
// never consults safeStorage, so "no keyring" cannot refuse the save.
|
||||
const saved = await saveRemoteToken(first.page, fake.url, SENTINEL_TOKEN)
|
||||
|
||||
expect(saved.error, 'the default (opted-out) policy must save without secure storage').toBeNull()
|
||||
expect(saved.config?.remoteTokenSet).toBe(true)
|
||||
|
||||
// Not a safeStorage blob, and owner-only on disk.
|
||||
expect(storedTokenEncoding(connectionFile)).not.toBe('safeStorage')
|
||||
expectOwnerOnlyMode(
|
||||
connectionFile,
|
||||
'connection.json is group/other-accessible; owner-only bits are the at-rest boundary for opted-out storage',
|
||||
)
|
||||
|
||||
// Round trip across a restart, same witness as the opted-in test.
|
||||
await app.close().catch(() => undefined)
|
||||
app = null
|
||||
|
||||
const second = await launchAgainst(sandbox)
|
||||
app = second.app
|
||||
|
||||
const reread = await second.page.evaluate(async () => {
|
||||
const desktop = (window as unknown as { hermesDesktop: any }).hermesDesktop
|
||||
|
||||
return desktop.getConnectionConfig()
|
||||
})
|
||||
|
||||
expect(reread.remoteTokenSet, 'the stored token must survive a restart').toBe(true)
|
||||
|
||||
const before = fake.sessionTokens.length
|
||||
await exerciseStoredToken(second.page, fake.url)
|
||||
|
||||
expect(
|
||||
fake.sessionTokens.slice(before),
|
||||
'the app must send the exact stored token to the gateway after a restart',
|
||||
).toContain(SENTINEL_TOKEN)
|
||||
})
|
||||
|
||||
/**
|
||||
* The read side of the same contract: an install written BEFORE the file was
|
||||
* owner-only keeps its 0644 bits until something chmods it, and the write
|
||||
* path cannot fix it — `fs.writeFileSync(path, data, { mode })` applies
|
||||
* `mode` only when it CREATES the file. Waiting for the user's next Settings
|
||||
* save would leave the file group/other-readable indefinitely, which is why
|
||||
* `readDesktopConnectionConfig` tightens on a cache miss.
|
||||
*
|
||||
* Scoped to the MODE, and deliberately independent of the deferred migration
|
||||
* below. The fixture's token is already safeStorage ciphertext (the app wrote
|
||||
* it), so nothing here re-encrypts anything, touches the #62319 opt-in
|
||||
* plaintext marker, or needs rotation guidance — the three prerequisites that
|
||||
* keep the next test fixme'd. Tightening a permission bit neither performs a
|
||||
* migration nor claims to, so it can be covered now while migration stays
|
||||
* deferred.
|
||||
*
|
||||
* The fixture is produced by the app itself rather than hand-written, so the
|
||||
* only difference from a real pre-fix install is the one bit under test.
|
||||
*/
|
||||
test('an install whose connection.json predates owner-only mode is tightened on read', async () => {
|
||||
const fake = gateway!
|
||||
sandbox = createSandbox('at-rest-tighten')
|
||||
|
||||
const first = await launchAgainst(sandbox)
|
||||
app = first.app
|
||||
|
||||
const capability = await readSafeStorageCapability(app)
|
||||
|
||||
test.info().annotations.push({
|
||||
description: `isEncryptionAvailable=${capability.available} backend=${capability.backend}`,
|
||||
type: 'safeStorage',
|
||||
})
|
||||
|
||||
if (!capability.available) {
|
||||
// Without secure storage the save is refused by design, so there is no
|
||||
// app-written artifact to loosen and re-read. The refusal itself is
|
||||
// already asserted in the first test.
|
||||
test.skip(true, 'secure storage unavailable on this host — no app-written connection.json to tighten')
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
const userDataDir = await resolveUserDataDir(app)
|
||||
const connectionFile = path.join(userDataDir, 'connection.json')
|
||||
|
||||
const saved = await saveRemoteToken(first.page, fake.url, SENTINEL_TOKEN)
|
||||
expect(saved.error, 'the fixture write must succeed, or there is nothing to tighten').toBeNull()
|
||||
|
||||
await app.close().catch(() => undefined)
|
||||
app = null
|
||||
|
||||
// Regress the file to what a pre-fix install has on disk. Everything else
|
||||
// about it — including the encrypted token — is exactly what the app wrote.
|
||||
fs.chmodSync(connectionFile, 0o644)
|
||||
expect(fs.statSync(connectionFile).mode & 0o077, 'the fixture must start group/other-accessible').not.toBe(0)
|
||||
|
||||
const seededMtimeMs = fs.statSync(connectionFile).mtimeMs
|
||||
|
||||
// A fresh process starts with an empty config cache, so the first read is a
|
||||
// miss and the tighten runs. `getConnectionConfig()` forces that read
|
||||
// through the app's own IPC surface.
|
||||
const second = await launchAgainst(sandbox)
|
||||
app = second.app
|
||||
|
||||
const reread = await second.page.evaluate(async () => {
|
||||
const desktop = (window as unknown as { hermesDesktop: any }).hermesDesktop
|
||||
|
||||
return desktop.getConnectionConfig()
|
||||
})
|
||||
|
||||
expectOwnerOnlyMode(
|
||||
connectionFile,
|
||||
'a pre-existing world-readable connection.json was not tightened when the app read it',
|
||||
)
|
||||
|
||||
// The tighten must be a chmod, not a rewrite. It sits INSIDE the function
|
||||
// whose cache keys on mtimeMs, so if it ever moved mtime it would
|
||||
// invalidate that cache on every read and re-tighten forever. chmod moves
|
||||
// ctime only, which is what makes the placement safe — this pins it.
|
||||
expect(
|
||||
Math.abs(fs.statSync(connectionFile).mtimeMs - seededMtimeMs),
|
||||
'tightening must not rewrite the file: mtime is the config cache key, so moving it would invalidate the cache the tighten sits inside',
|
||||
).toBeLessThan(1)
|
||||
|
||||
// And tightening must not have cost the user their credential — the whole
|
||||
// reason this happens on read instead of by deleting the file.
|
||||
expect(reread.remoteTokenSet, 'the stored token must survive being tightened').toBe(true)
|
||||
expect(reread.remoteUrl).toBe(fake.url)
|
||||
})
|
||||
|
||||
/**
|
||||
* The tighten must not be gated on the file being valid JSON.
|
||||
*
|
||||
* A truncated `connection.json` — an interrupted write on an older build, a
|
||||
* half-finished hand edit, a partially restored backup — still contains the
|
||||
* token bytes, and `JSON.parse` throws straight into the `catch` that falls
|
||||
* back to local mode. That fallback is never written back, so nothing
|
||||
* re-tightens the file later. With the chmod sequenced AFTER the parse,
|
||||
* exactly the file that is both corrupt AND world-readable would be the one
|
||||
* file never tightened, permanently.
|
||||
*
|
||||
* This is the only test that can tell the two orderings apart: every other
|
||||
* test here uses a parseable file, where either ordering tightens. Asserting
|
||||
* `mode === 'local'` is what makes it load-bearing — it proves the parse
|
||||
* really threw, so a green mode assertion cannot be explained by anything
|
||||
* downstream of the parse.
|
||||
*
|
||||
* Needs no secure storage: it is a chmod on a file that is never decrypted,
|
||||
* so it holds on the keyring-less CI runner too.
|
||||
*/
|
||||
test('a corrupt connection.json is tightened even though it never parses', async () => {
|
||||
sandbox = createSandbox('at-rest-tighten-corrupt')
|
||||
|
||||
const connectionFile = path.join(sandbox.userDataDir, 'connection.json')
|
||||
|
||||
// Truncated mid-token: unparseable, yet the secret bytes are right there.
|
||||
fs.writeFileSync(
|
||||
connectionFile,
|
||||
`{"mode":"remote","remote":{"authMode":"token","token":{"encoding":"plain","value":"${SENTINEL_TOKEN}`,
|
||||
{ encoding: 'utf8', mode: 0o644 },
|
||||
)
|
||||
fs.chmodSync(connectionFile, 0o644)
|
||||
expect(fs.statSync(connectionFile).mode & 0o077, 'the fixture must start group/other-accessible').not.toBe(0)
|
||||
|
||||
const seededMtimeMs = fs.statSync(connectionFile).mtimeMs
|
||||
|
||||
const launched = await launchAgainst(sandbox)
|
||||
app = launched.app
|
||||
|
||||
const reread = await launched.page.evaluate(async () => {
|
||||
const desktop = (window as unknown as { hermesDesktop: any }).hermesDesktop
|
||||
|
||||
return desktop.getConnectionConfig()
|
||||
})
|
||||
|
||||
expect(
|
||||
reread.mode,
|
||||
'the fixture must be unparseable, so the app falls back to local — otherwise this test proves nothing about ordering',
|
||||
).toBe('local')
|
||||
|
||||
expectOwnerOnlyMode(
|
||||
connectionFile,
|
||||
'a corrupt world-readable connection.json still holding token bytes was left group/other-accessible',
|
||||
)
|
||||
|
||||
// Same cache invariant as above: chmod, not rewrite.
|
||||
expect(
|
||||
Math.abs(fs.statSync(connectionFile).mtimeMs - seededMtimeMs),
|
||||
'tightening must not rewrite the file: mtime is the config cache key',
|
||||
).toBeLessThan(1)
|
||||
})
|
||||
|
||||
/**
|
||||
* DEFERRED GAP — legacy plaintext payloads are not migrated.
|
||||
*
|
||||
* Held as `fixme` rather than deleted: the fixture below is the correct
|
||||
* fixture for the population that a migration must eventually cover, and
|
||||
* the harness (seed → boot-poll → authoritative re-save → raw-bytes scan →
|
||||
* wire check) is the harness such a migration needs. Keeping it typechecked
|
||||
* and listed makes the gap visible in `--list` and in every report; deleting
|
||||
* it would make the gap invisible and cost the next implementer this setup.
|
||||
*
|
||||
* It is NOT enabled because the migration it asserted was reviewed
|
||||
* DO NOT SHIP. Before this can be un-fixme'd, three prerequisites (see the
|
||||
* header, and the matching note in electron/main.ts readDesktopConnectionConfig):
|
||||
*
|
||||
* 1. Sequence with #62319's opt-in plaintext marker, so a user who
|
||||
* deliberately chose plaintext is not silently overridden. This
|
||||
* fixture has NO marker, so it stays in scope for migration — but the
|
||||
* implementation must be able to tell the two apart.
|
||||
* 2. Write through the config sanitizer, not around it.
|
||||
* 3. Surface ROTATION guidance. Re-encrypting cannot un-expose a secret
|
||||
* that is already in a backup; it only prevents future exposure.
|
||||
*
|
||||
* Un-fixme'ing this without (1) risks destroying a deliberate user choice,
|
||||
* and without (3) it reports a remediation it did not actually perform.
|
||||
*/
|
||||
test('an existing plaintext connection.json is migrated off plaintext and keeps working', async () => {
|
||||
test.fixme(
|
||||
true,
|
||||
'Deferred: legacy plaintext connection.json is intentionally NOT migrated. ' +
|
||||
'Affected population is pre-release bb/gui installs (incl. the desktop-pr20059-installers build) ' +
|
||||
'plus hand-edited configs — mainline never wrote a plaintext gateway token. ' +
|
||||
'Blocked on: (1) #62319 opt-in-marker coordination, (2) writing through the config sanitizer, ' +
|
||||
'(3) surfacing token-rotation guidance. Re-encrypting alone does not remediate an already-backed-up secret.',
|
||||
)
|
||||
|
||||
const fake = gateway!
|
||||
sandbox = createSandbox('at-rest-migrate')
|
||||
|
||||
// Seed the file an affected user has on disk. This is live, usable
|
||||
// plaintext rather than a strawman, because `decryptDesktopSecret` returns
|
||||
// `value` verbatim for any non-safeStorage encoding — the READ path
|
||||
// accepts it. Note what does NOT justify this fixture: mainline never
|
||||
// WROTE this shape to disk. `coerceDesktopConnectionConfig` does build it,
|
||||
// but only under `persistToken: false`, whose sole caller is the
|
||||
// connection-test handler, which never persists. The writers were
|
||||
// non-mainline pre-release builds and hand edits. There is deliberately no
|
||||
// opt-in marker here, so this payload is in scope for a future migration.
|
||||
fs.writeFileSync(
|
||||
path.join(sandbox.userDataDir, 'connection.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
mode: 'remote',
|
||||
profiles: {},
|
||||
remote: {
|
||||
authMode: 'token',
|
||||
token: { encoding: 'plain', value: SENTINEL_TOKEN },
|
||||
url: fake.url,
|
||||
},
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
'utf8',
|
||||
)
|
||||
|
||||
const launched = await launchAgainst(sandbox)
|
||||
app = launched.app
|
||||
|
||||
const capability = await readSafeStorageCapability(app)
|
||||
|
||||
test.info().annotations.push({
|
||||
description: `isEncryptionAvailable=${capability.available} backend=${capability.backend}`,
|
||||
type: 'safeStorage',
|
||||
})
|
||||
|
||||
if (!capability.available) {
|
||||
// With no secure storage there is nowhere to migrate the secret TO, and
|
||||
// scrubbing it would silently sign the user out of a working gateway.
|
||||
// Asserting either outcome here would be inventing policy.
|
||||
test.skip(
|
||||
true,
|
||||
'secure storage unavailable on this host — the correct migration policy for an existing plaintext file is undecided',
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
const userDataDir = await resolveUserDataDir(app)
|
||||
const connectionFile = path.join(userDataDir, 'connection.json')
|
||||
const needles = secretNeedles(SENTINEL_TOKEN)
|
||||
|
||||
// Two chances, so the test does not depend on WHERE the fix hooks the
|
||||
// migration: (a) on read at boot, (b) on the next authoritative write.
|
||||
// Poll for (a) first.
|
||||
const deadline = Date.now() + 15_000
|
||||
let stillPlaintext = true
|
||||
|
||||
while (Date.now() < deadline) {
|
||||
stillPlaintext = readIfExists(connectionFile).includes(needles[0].bytes)
|
||||
|
||||
if (!stillPlaintext) {
|
||||
break
|
||||
}
|
||||
|
||||
await launched.page.waitForTimeout(500)
|
||||
}
|
||||
|
||||
if (stillPlaintext) {
|
||||
// (b) A real save through the app's own surface, carrying no new token —
|
||||
// the stored blob is inherited. Re-persisting an inherited secret is the
|
||||
// other place plaintext must not survive.
|
||||
const resaved = await saveRemoteToken(launched.page, fake.url)
|
||||
expect(resaved.error, 'a re-save that inherits the stored token must not fail').toBeNull()
|
||||
}
|
||||
|
||||
expect(
|
||||
scanTreeForSecret(userDataDir, needles),
|
||||
'an existing plaintext gateway token must not remain readable under userData after the app has run ' +
|
||||
`(stored token encoding is still "${storedTokenEncoding(connectionFile)}")`,
|
||||
).toEqual([])
|
||||
|
||||
// And the migration must not have cost the user their credential.
|
||||
const before = fake.sessionTokens.length
|
||||
await exerciseStoredToken(launched.page, fake.url)
|
||||
|
||||
expect(
|
||||
fake.sessionTokens.slice(before),
|
||||
'the migrated token must still reach the gateway unchanged',
|
||||
).toContain(SENTINEL_TOKEN)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,86 @@
|
||||
/**
|
||||
* E2E batch clarify test — the multi-question clarify card must mount ONCE.
|
||||
*
|
||||
* Regression coverage for the duplicated-card bug: `tool.start` carries the
|
||||
* model's tool_call_id while `clarify.request` carries a gateway-generated
|
||||
* request_id. A batch payload has no top-level `question`, so the two rows
|
||||
* only merge when the correlation key comes from the question list
|
||||
* (`batchClarifyMatchValue` in lib/chat-messages/tool-parts.ts). Before that
|
||||
* fix this exact flow rendered two identical interactive cards.
|
||||
*
|
||||
* The flow runs the real chain: composer → gateway → agent → clarify tool →
|
||||
* clarify.request event → renderer, against the mock inference server.
|
||||
*/
|
||||
|
||||
import { expect, test } from './test'
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { BATCH_CLARIFY_QUESTIONS, BATCH_CLARIFY_TRIGGER } from './mock-server'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture!, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test.describe('batch clarify card', () => {
|
||||
test('renders exactly one card and completes via per-question locks', async () => {
|
||||
const page = fixture!.page
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
|
||||
await composer.click()
|
||||
await composer.type(BATCH_CLARIFY_TRIGGER, { delay: 20 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
// The live batch form marks itself with data-clarify-batch=<count>.
|
||||
const batchCard = page.locator('form[data-clarify-batch]')
|
||||
await batchCard.first().waitFor({ state: 'visible', timeout: 60_000 })
|
||||
|
||||
// THE regression assertion: one card, not two.
|
||||
await expect(batchCard).toHaveCount(1)
|
||||
await expect(batchCard).toHaveAttribute('data-clarify-batch', String(BATCH_CLARIFY_QUESTIONS.length))
|
||||
|
||||
// Both questions render inside the single card.
|
||||
for (const entry of BATCH_CLARIFY_QUESTIONS) {
|
||||
await expect(batchCard.getByText(entry.question)).toHaveCount(1)
|
||||
}
|
||||
|
||||
// Each question text also appears exactly once in the whole transcript —
|
||||
// catches a duplicate that mounts outside a form[data-clarify-batch].
|
||||
for (const entry of BATCH_CLARIFY_QUESTIONS) {
|
||||
await expect(page.getByText(entry.question)).toHaveCount(1)
|
||||
}
|
||||
|
||||
// Answer both questions: stage picks locally (no server traffic yet).
|
||||
const confirmButton = batchCard.locator('button[type="submit"]')
|
||||
await expect(confirmButton).toContainText('Confirm and continue')
|
||||
await expect(confirmButton).toBeDisabled()
|
||||
|
||||
await batchCard.getByRole('button', { name: /Coffee/ }).click()
|
||||
await expect(confirmButton).toBeDisabled()
|
||||
|
||||
await batchCard.getByRole('button', { name: /Morning/ }).click()
|
||||
await expect(confirmButton).toBeEnabled()
|
||||
|
||||
// ONE confirm submits the whole batch.
|
||||
await confirmButton.click()
|
||||
|
||||
// The settled card lists both questions with their locked answers.
|
||||
const settled = page.locator('[data-clarify-settled]')
|
||||
await settled.waitFor({ state: 'visible', timeout: 30_000 })
|
||||
await expect(settled.getByText(BATCH_CLARIFY_QUESTIONS[0].question)).toBeVisible()
|
||||
await expect(settled.getByText('Coffee', { exact: true })).toBeVisible()
|
||||
await expect(settled.getByText(BATCH_CLARIFY_QUESTIONS[1].question)).toBeVisible()
|
||||
await expect(settled.getByText('Morning', { exact: true })).toBeVisible()
|
||||
|
||||
// And still no duplicate live card lingering after settle.
|
||||
await expect(page.locator('form[data-clarify-batch]')).toHaveCount(0)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* E2E boot-failure tests — verify the app shows an error overlay when the
|
||||
* backend can't start.
|
||||
*
|
||||
* Injects a fake boot error (HERMES_DESKTOP_BOOT_FAKE_ERROR) so the backend
|
||||
* resolution fails with a controlled error message. The app should show the
|
||||
* BootFailureOverlay with retry/repair actions.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { allowErrorBanners, test } from './test'
|
||||
|
||||
import {
|
||||
type DeadBackendFixture,
|
||||
setupDeadBackend,
|
||||
waitForBootFailure,
|
||||
} from './fixtures'
|
||||
import { expectVisualSnapshot } from './visual-snapshot'
|
||||
|
||||
let fixture: DeadBackendFixture | null = null
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test.describe('boot failure with dead backend', () => {
|
||||
test.beforeEach(() => {
|
||||
// These tests deliberately trigger boot errors — error banners
|
||||
// (notifyError → [role="alert"]) are expected, not failures.
|
||||
allowErrorBanners()
|
||||
})
|
||||
|
||||
test('app shows error state', async () => {
|
||||
// Inject a fake boot error so the backend resolution "fails" with a
|
||||
// controlled error message. This is the only reliable way to trigger
|
||||
// BootFailureOverlay in dev mode.
|
||||
fixture = await setupDeadBackend({ fakeError: true })
|
||||
|
||||
await waitForBootFailure(fixture.page, 90_000)
|
||||
})
|
||||
|
||||
test('screenshot of error state', async () => {
|
||||
if (!fixture) {
|
||||
test.skip(true, 'Previous test failed — no app running')
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
await expectVisualSnapshot(fixture!.page, { name: 'boot-failure-error-state', app: fixture.app })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,82 @@
|
||||
/**
|
||||
* E2E smoke tests for the dev-mode desktop app.
|
||||
*
|
||||
* These tests launch the Electron app from the built dist/ (not the
|
||||
* packaged binary) with a real `hermes serve` backend pointed at a mock
|
||||
* inference server. The full chain is exercised:
|
||||
*
|
||||
* electron → hermes serve (python) → mock provider → renderer
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
* Run from the nix devshell:
|
||||
* npm exec playwright test e2e/boot.spec.ts --reporter=list
|
||||
*/
|
||||
import { expect, test } from './test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
import { expectVisualSnapshot } from './visual-snapshot'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test.describe('dev-mode boot with mock backend', () => {
|
||||
test('window opens with Hermes title', async () => {
|
||||
const title = await fixture!.page.title()
|
||||
expect(title).toContain('Hermes')
|
||||
})
|
||||
|
||||
test('renderer mounts and shows DOM content', async () => {
|
||||
const page = fixture!.page
|
||||
// Wait for the React root to mount. The app renders into #root
|
||||
// (see src/main.tsx), but content may arrive through portals — so
|
||||
// check the body for any interactive content instead.
|
||||
await page.waitForSelector('body', { state: 'attached' })
|
||||
// Wait for the main app shell — the composer is always present.
|
||||
await page.waitForSelector('textarea, [contenteditable="true"]', {
|
||||
state: 'attached',
|
||||
timeout: 30_000,
|
||||
})
|
||||
})
|
||||
|
||||
// A preload that throws never reaches contextBridge, so the renderer boots
|
||||
// into "Desktop IPC bridge is unavailable" and every test below it dies on a
|
||||
// 120s never-became-ready timeout instead. Checking the bridge by name makes
|
||||
// that failure legible. The sandbox lets preload require only electron,
|
||||
// events, timers and url — adding any other node builtin lands here.
|
||||
test('the preload bridge reaches the renderer', async () => {
|
||||
const bridge = await fixture!.page.evaluate(() => {
|
||||
const desktop = (window as unknown as { hermesDesktop?: Record<string, unknown> }).hermesDesktop
|
||||
|
||||
return {
|
||||
present: typeof desktop,
|
||||
glassSupported: typeof desktop?.glassSupported,
|
||||
translucencySupported: typeof desktop?.translucencySupported
|
||||
}
|
||||
})
|
||||
|
||||
expect(bridge).toEqual({ present: 'object', glassSupported: 'boolean', translucencySupported: 'boolean' })
|
||||
})
|
||||
|
||||
test('backend boots and app becomes ready', async () => {
|
||||
// This is the big one — wait for the full boot chain to complete:
|
||||
// electron starts → hermes serve is spawned → WS connects → config
|
||||
// loaded → sessions loaded → boot overlay dismissed → composer visible.
|
||||
await waitForAppReady(fixture!, 120_000)
|
||||
})
|
||||
|
||||
test('screenshot after boot', async () => {
|
||||
await expectVisualSnapshot(fixture!.page, { name: 'boot-ready', app: fixture!.app })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,158 @@
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type MockBackendFixture,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig
|
||||
} from './fixtures'
|
||||
import { MOCK_REPLY, startMockServer } from './mock-server'
|
||||
import { RealSessionBuilder } from './real-session-builder'
|
||||
import { expect, test } from './test'
|
||||
|
||||
// A bot row previews the bot's canonical Bot Chat (the gateway resolves it by
|
||||
// name on every roster poll). Clicking the row must land on THAT conversation.
|
||||
// Before this fix a plain click fronted whatever bots-workspace tile the user
|
||||
// last had open for that bot — a `+` side thread outlived every restart in
|
||||
// Local Storage and won every click forever, while the row kept previewing the
|
||||
// Bot Chat. The user saw the sidebar and the center describe two different
|
||||
// conversations ("sessions not in sync"; support thread 1544460286084391043).
|
||||
|
||||
type Page = MockBackendFixture['page']
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
async function openBots(page: Page): Promise<void> {
|
||||
const tab = page
|
||||
.getByRole('button', { name: 'Bots', exact: true })
|
||||
.or(page.getByRole('tab', { name: 'Bots', exact: true }))
|
||||
.first()
|
||||
|
||||
await tab.click()
|
||||
await expect(page.getByRole('button', { name: 'New bot or group chat' })).toBeVisible()
|
||||
}
|
||||
|
||||
async function settle(page: Page, timeout = 90_000): Promise<void> {
|
||||
await page
|
||||
.getByText(/Waking up/i)
|
||||
.first()
|
||||
.waitFor({ state: 'hidden', timeout })
|
||||
.catch(() => undefined)
|
||||
await page.waitForTimeout(500)
|
||||
}
|
||||
|
||||
async function openUntil(action: () => Promise<void>, expected: () => Promise<void>, attempts = 3): Promise<void> {
|
||||
for (let attempt = 1; ; attempt += 1) {
|
||||
await action()
|
||||
|
||||
try {
|
||||
await expected()
|
||||
|
||||
return
|
||||
} catch (error) {
|
||||
if (attempt >= attempts) {
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function seedBot(hermesHome: string, mockUrl: string, name: string): Promise<void> {
|
||||
const dir = path.join(hermesHome, 'profiles', name)
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
writeMockProviderConfig(dir, mockUrl)
|
||||
writeEnvFile(dir)
|
||||
|
||||
const builder = await RealSessionBuilder.start(dir)
|
||||
|
||||
try {
|
||||
await builder.createSession({ title: 'Bot Chat', turns: [`Hello ${name}`] })
|
||||
} finally {
|
||||
await builder.close()
|
||||
}
|
||||
}
|
||||
|
||||
test.beforeAll(async () => {
|
||||
const mock = await startMockServer()
|
||||
const sandbox = createSandbox('bots-sync')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
await seedBot(sandbox.hermesHome, mock.url, 'alpha')
|
||||
await seedBot(sandbox.hermesHome, mock.url, 'beta')
|
||||
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
|
||||
fixture = {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
}
|
||||
}
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('a bot row click lands on the Bot Chat the row previews, not a side thread', async () => {
|
||||
test.setTimeout(300_000)
|
||||
const page = fixture!.page
|
||||
|
||||
await openBots(page)
|
||||
|
||||
const alphaRow = page.getByRole('button', { name: /^alpha\b/i }).filter({ visible: true }).first()
|
||||
const betaRow = page.getByRole('button', { name: /^beta\b/i }).filter({ visible: true }).first()
|
||||
await expect(alphaRow).toBeVisible({ timeout: 30_000 })
|
||||
await expect(betaRow).toBeVisible({ timeout: 30_000 })
|
||||
const seededTurn = page.getByText('Hello alpha', { exact: true }).filter({ visible: true })
|
||||
|
||||
await openUntil(
|
||||
() => alphaRow.click(),
|
||||
() => expect(seededTurn.first()).toBeVisible({ timeout: 45_000 })
|
||||
)
|
||||
await settle(page, 15_000)
|
||||
|
||||
// A `+` side thread for alpha, with a real turn so it is a persisted tile.
|
||||
await page.keyboard.press('Control+t')
|
||||
const composer = page.locator('[data-slot="composer-root"] [contenteditable="true"]').filter({ visible: true }).first()
|
||||
await expect(composer).toBeVisible({ timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.fill('hello alpha thread')
|
||||
await page.keyboard.press('Enter')
|
||||
await expect(page.getByText(MOCK_REPLY).filter({ visible: true }).first()).toBeVisible({ timeout: 60_000 })
|
||||
|
||||
// Leave alpha on the side thread, go to beta, come back via the row.
|
||||
await betaRow.click()
|
||||
await expect(page.getByText('Hello beta', { exact: true }).filter({ visible: true }).first()).toBeVisible({
|
||||
timeout: 60_000
|
||||
})
|
||||
await settle(page)
|
||||
|
||||
await alphaRow.click()
|
||||
// The row previews the Bot Chat; the click must front it.
|
||||
await expect(seededTurn.first()).toBeVisible({ timeout: 45_000 })
|
||||
// The side thread is still open beside it (scoped to alpha), not closed.
|
||||
await expect
|
||||
.poll(
|
||||
() =>
|
||||
page.evaluate(() =>
|
||||
[...document.querySelectorAll<HTMLElement>('[data-zone-tabstrip="grp-main"] [data-tree-tab]')]
|
||||
.map(element => element.getAttribute('data-tree-tab') ?? '')
|
||||
.filter(id => id.startsWith('session-tile:')).length
|
||||
),
|
||||
{ timeout: 15_000 }
|
||||
)
|
||||
.toBe(1)
|
||||
})
|
||||
@@ -0,0 +1,137 @@
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type MockBackendFixture,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig
|
||||
} from './fixtures'
|
||||
import { MOCK_REPLY, startMockServer } from './mock-server'
|
||||
import { RealSessionBuilder } from './real-session-builder'
|
||||
import { expect, test } from './test'
|
||||
|
||||
// Every bot's canonical chat is STORED under the same title ("Bot Chat" — the
|
||||
// name the gateway resolves it by), so the main tab strip captioned every open
|
||||
// bot chat identically and two bots' tabs were indistinguishable (#99152). The
|
||||
// tab must read the bot's display name while the stored title stays canonical.
|
||||
|
||||
type Page = MockBackendFixture['page']
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
async function openBots(page: Page): Promise<void> {
|
||||
const tab = page
|
||||
.getByRole('button', { name: 'Bots', exact: true })
|
||||
.or(page.getByRole('tab', { name: 'Bots', exact: true }))
|
||||
.first()
|
||||
|
||||
await tab.click()
|
||||
await expect(page.getByRole('button', { name: 'New bot or group chat' })).toBeVisible()
|
||||
}
|
||||
|
||||
async function openUntil(action: () => Promise<void>, expected: () => Promise<void>, attempts = 3): Promise<void> {
|
||||
for (let attempt = 1; ; attempt += 1) {
|
||||
await action()
|
||||
|
||||
try {
|
||||
await expected()
|
||||
|
||||
return
|
||||
} catch (error) {
|
||||
if (attempt >= attempts) {
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function seedBot(hermesHome: string, mockUrl: string, name: string): Promise<void> {
|
||||
const dir = path.join(hermesHome, 'profiles', name)
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
writeMockProviderConfig(dir, mockUrl)
|
||||
writeEnvFile(dir)
|
||||
|
||||
const builder = await RealSessionBuilder.start(dir)
|
||||
|
||||
try {
|
||||
await builder.createSession({ title: 'Bot Chat', turns: [`Hello ${name}`] })
|
||||
} finally {
|
||||
await builder.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Every tab caption in the main strip (the main `workspace` tab + tiles). */
|
||||
function mainStripTabTitles(page: Page): Promise<string[]> {
|
||||
return page.evaluate(() =>
|
||||
[...document.querySelectorAll<HTMLElement>('[data-zone-tabstrip="grp-main"] [data-tree-tab]')].map(element =>
|
||||
(element.textContent ?? '').trim()
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
test.beforeAll(async () => {
|
||||
const mock = await startMockServer()
|
||||
const sandbox = createSandbox('bots-tabname')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
await seedBot(sandbox.hermesHome, mock.url, 'alpha')
|
||||
await seedBot(sandbox.hermesHome, mock.url, 'beta')
|
||||
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
|
||||
fixture = {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
}
|
||||
}
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test("an open Bot Chat's tab reads the bot's name, not the canonical 'Bot Chat' title", async () => {
|
||||
test.setTimeout(300_000)
|
||||
const page = fixture!.page
|
||||
|
||||
await openBots(page)
|
||||
|
||||
const alphaRow = page.getByRole('button', { name: /^alpha\b/i }).filter({ visible: true }).first()
|
||||
await expect(alphaRow).toBeVisible({ timeout: 30_000 })
|
||||
|
||||
await openUntil(
|
||||
() => alphaRow.click(),
|
||||
() =>
|
||||
expect(page.getByText('Hello alpha', { exact: true }).filter({ visible: true }).first()).toBeVisible({
|
||||
timeout: 45_000
|
||||
})
|
||||
)
|
||||
|
||||
// A `+` side thread beside the Bot Chat gives the main zone a tab strip —
|
||||
// the surface where every bot chat used to read "Bot Chat".
|
||||
await page.keyboard.press('Control+t')
|
||||
const composer = page.locator('[data-slot="composer-root"] [contenteditable="true"]').filter({ visible: true }).first()
|
||||
await expect(composer).toBeVisible({ timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.fill('hello alpha thread')
|
||||
await page.keyboard.press('Enter')
|
||||
await expect(page.getByText(MOCK_REPLY).filter({ visible: true }).first()).toBeVisible({ timeout: 60_000 })
|
||||
|
||||
await expect.poll(() => mainStripTabTitles(page), { timeout: 15_000 }).toHaveLength(2)
|
||||
const captions = await mainStripTabTitles(page)
|
||||
expect(captions.some(caption => /alpha/i.test(caption))).toBe(true)
|
||||
expect(captions.some(caption => /bot chat/i.test(caption))).toBe(false)
|
||||
})
|
||||
@@ -0,0 +1,254 @@
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type MockBackendFixture,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig
|
||||
} from './fixtures'
|
||||
import { startMockServer } from './mock-server'
|
||||
import { RealSessionBuilder } from './real-session-builder'
|
||||
import { expect, test } from './test'
|
||||
|
||||
// User-made sections in the Bots roster: a bot is filed by dragging it onto a
|
||||
// section or through its row menu, the section is renamed through the same
|
||||
// dialog shape sessions use, and deleting a section returns its bots to
|
||||
// Unassigned (with an Undo toast, no confirmation). With no sections created
|
||||
// the roster is the plain list it always was.
|
||||
|
||||
type Page = MockBackendFixture['page']
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
// BOT_SECTIONS_SCREENSHOT_DIR=<dir> saves full-window captures at the key
|
||||
// states — handy for design review; never part of the assertions.
|
||||
async function capture(page: Page, name: string): Promise<void> {
|
||||
const dir = process.env.BOT_SECTIONS_SCREENSHOT_DIR
|
||||
|
||||
if (!dir) {
|
||||
return
|
||||
}
|
||||
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
await page.screenshot({ path: path.join(dir, `${name}.png`) })
|
||||
}
|
||||
|
||||
async function seedBot(hermesHome: string, mockUrl: string, name: string): Promise<void> {
|
||||
const dir = path.join(hermesHome, 'profiles', name)
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
writeMockProviderConfig(dir, mockUrl)
|
||||
writeEnvFile(dir)
|
||||
|
||||
const builder = await RealSessionBuilder.start(dir)
|
||||
|
||||
try {
|
||||
await builder.createSession({ title: 'Bot Chat', turns: [`Hello ${name}`] })
|
||||
} finally {
|
||||
await builder.close()
|
||||
}
|
||||
}
|
||||
|
||||
const roster = (page: Page) => page.locator('[data-slot="bots-roster"]')
|
||||
const botRow = (page: Page, name: string) => roster(page).locator(`[data-roster-key="local::${name}"]`)
|
||||
|
||||
/** A section's label span — the one node whose text is exactly the name. */
|
||||
const sectionLabel = (page: Page, name: string) =>
|
||||
page.locator('span.truncate', { hasText: new RegExp(`^${name}$`, 'i') })
|
||||
|
||||
/** The heading's fold button (label + count) — the ⋯ menu trigger is a sibling with no text. */
|
||||
const sectionHeading = (page: Page, name: string) =>
|
||||
roster(page).locator('[data-slot="bots-section"] button[aria-expanded]').filter({ has: sectionLabel(page, name) })
|
||||
|
||||
const sectionBlock = (page: Page, name: string) =>
|
||||
roster(page).locator('[data-slot="bots-section"]').filter({ has: sectionLabel(page, name) })
|
||||
|
||||
/** Section name → roster keys of the rows under it (the plain list has no sections). */
|
||||
async function layout(page: Page): Promise<Array<[string, string[]]>> {
|
||||
return roster(page).locator('[data-slot="bots-section"]').evaluateAll(blocks =>
|
||||
blocks.map(block => [
|
||||
block.querySelector('button[aria-expanded] span.truncate')?.textContent?.trim() ?? '',
|
||||
[...block.querySelectorAll<HTMLElement>('[data-roster-key]')].map(row => row.dataset.rosterKey ?? '')
|
||||
])
|
||||
)
|
||||
}
|
||||
|
||||
test.beforeAll(async () => {
|
||||
const mock = await startMockServer()
|
||||
const sandbox = createSandbox('bots-sections')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
|
||||
for (const name of ['alpha', 'beta', 'gamma']) {
|
||||
await seedBot(sandbox.hermesHome, mock.url, name)
|
||||
}
|
||||
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
|
||||
fixture = {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
}
|
||||
}
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('file bots into user sections by menu and drag; rename; delete returns them to Unassigned', async () => {
|
||||
test.setTimeout(300_000)
|
||||
const page = fixture!.page
|
||||
|
||||
const tab = page
|
||||
.getByRole('button', { name: 'Bots', exact: true })
|
||||
.or(page.getByRole('tab', { name: 'Bots', exact: true }))
|
||||
.first()
|
||||
|
||||
await tab.click()
|
||||
await expect(page.getByRole('button', { name: 'New bot or group chat' })).toBeVisible()
|
||||
await expect(botRow(page, 'alpha')).toBeVisible({ timeout: 30_000 })
|
||||
await expect(botRow(page, 'beta')).toBeVisible({ timeout: 30_000 })
|
||||
|
||||
// No sections yet: the plain list, no section chrome at all.
|
||||
await expect(roster(page).locator('[data-slot="bots-section"]')).toHaveCount(0)
|
||||
await capture(page, '1-plain-roster')
|
||||
|
||||
// Right-click alpha → Move to section → New section… → name it → alpha is filed.
|
||||
await botRow(page, 'alpha').click({ button: 'right' })
|
||||
await page.getByRole('menuitem', { name: 'Move to section' }).hover()
|
||||
await expect(page.getByRole('menuitem', { name: 'New section…' })).toBeVisible()
|
||||
await capture(page, '2-row-menu-move-to-section')
|
||||
await page.getByRole('menuitem', { name: 'New section…' }).click()
|
||||
const nameField = page.getByRole('textbox', { name: 'Section name' })
|
||||
await expect(nameField).toBeVisible()
|
||||
await nameField.fill('Clients')
|
||||
await capture(page, '3-new-section-dialog')
|
||||
await page.getByRole('button', { name: 'Create' }).click()
|
||||
|
||||
await expect(sectionHeading(page, 'Clients')).toBeVisible()
|
||||
await expect(sectionBlock(page, 'Clients').locator('[data-roster-key="local::alpha"]')).toBeVisible()
|
||||
// The remainder is Unassigned, drawn last.
|
||||
await expect
|
||||
.poll(async () => (await layout(page)).map(([name, keys]) => [name, keys.length]))
|
||||
.toEqual([
|
||||
['Clients', 1],
|
||||
['Unassigned', 3]
|
||||
])
|
||||
await capture(page, '4-alpha-filed')
|
||||
|
||||
// Drag beta over the Clients block: the target highlights while over it.
|
||||
// Escape cancels — nothing moves, nothing stays highlighted or faded.
|
||||
const target = sectionBlock(page, 'Clients')
|
||||
const from = (await botRow(page, 'beta').boundingBox())!
|
||||
const to = (await sectionHeading(page, 'Clients').boundingBox())!
|
||||
|
||||
const dragBetaOverClients = async () => {
|
||||
await page.mouse.move(from.x + from.width / 2, from.y + from.height / 2)
|
||||
await page.mouse.down()
|
||||
await page.mouse.move(from.x + from.width / 2, from.y + from.height / 2 - 10, { steps: 4 })
|
||||
await page.mouse.move(to.x + to.width / 2, to.y + to.height / 2, { steps: 12 })
|
||||
await expect(target).toHaveAttribute('data-drop-over', 'true')
|
||||
}
|
||||
|
||||
await dragBetaOverClients()
|
||||
await page.keyboard.press('Escape')
|
||||
await page.mouse.up()
|
||||
await expect(target).not.toHaveAttribute('data-drop-over', 'true')
|
||||
await expect(botRow(page, 'beta')).toHaveCSS('opacity', '1')
|
||||
expect((await layout(page)).map(([name, keys]) => [name, keys.length])).toEqual([
|
||||
['Clients', 1],
|
||||
['Unassigned', 3]
|
||||
])
|
||||
|
||||
// Drop it for real: the bot is filed.
|
||||
await dragBetaOverClients()
|
||||
await capture(page, '5-drag-over-clients')
|
||||
await page.mouse.up()
|
||||
|
||||
await expect(target.locator('[data-roster-key="local::beta"]')).toBeVisible()
|
||||
await expect(target).not.toHaveAttribute('data-drop-over', 'true')
|
||||
// The moved row remounts under its new section; it must not stay faded.
|
||||
await expect(botRow(page, 'beta')).toHaveCSS('opacity', '1')
|
||||
await expect
|
||||
.poll(async () => (await layout(page)).map(([name, keys]) => [name, keys.length]))
|
||||
.toEqual([
|
||||
['Clients', 2],
|
||||
['Unassigned', 2]
|
||||
])
|
||||
await capture(page, '6-beta-dropped')
|
||||
|
||||
// Rename through the heading's context menu — the same Dialog + Input
|
||||
// + Save shape as a session rename.
|
||||
await sectionHeading(page, 'Clients').click({ button: 'right' })
|
||||
await page.getByRole('menuitem', { name: 'Rename…' }).click()
|
||||
await expect(nameField).toHaveValue('Clients')
|
||||
await nameField.fill('Customers')
|
||||
await page.getByRole('button', { name: 'Save' }).click()
|
||||
await expect(sectionHeading(page, 'Customers')).toBeVisible()
|
||||
await expect(sectionHeading(page, 'Clients')).toHaveCount(0)
|
||||
await capture(page, '7-renamed')
|
||||
|
||||
// A second, empty section from the + menu shows its drop hint; collapsing
|
||||
// a section folds its rows like the gateway headings do.
|
||||
await page.getByRole('button', { name: 'New bot or group chat' }).click()
|
||||
await page.getByRole('menuitem', { name: 'New section' }).click()
|
||||
await nameField.fill('Team')
|
||||
await page.getByRole('button', { name: 'Create' }).click()
|
||||
await expect(sectionBlock(page, 'Team').getByText('Drag bots here')).toBeVisible()
|
||||
await sectionHeading(page, 'Customers').click()
|
||||
await expect(sectionBlock(page, 'Customers').locator('[data-roster-key]')).toHaveCount(0)
|
||||
await capture(page, '8-empty-section-and-collapsed')
|
||||
await sectionHeading(page, 'Customers').click()
|
||||
await expect(sectionBlock(page, 'Customers').locator('[data-roster-key]')).toHaveCount(2)
|
||||
|
||||
// Delete Customers: no confirmation, its two bots return to Unassigned,
|
||||
// and the toast offers Undo.
|
||||
await sectionHeading(page, 'Customers').click({ button: 'right' })
|
||||
await page.getByRole('menuitem', { name: 'Delete' }).click()
|
||||
await expect(sectionHeading(page, 'Customers')).toHaveCount(0)
|
||||
const toast = page.getByRole('status').filter({ hasText: 'Deleted “Customers”' })
|
||||
await expect(toast).toBeVisible()
|
||||
await expect
|
||||
.poll(async () => (await layout(page)).map(([name, keys]) => [name, keys.length]))
|
||||
.toEqual([
|
||||
['Team', 0],
|
||||
['Unassigned', 4]
|
||||
])
|
||||
await capture(page, '9-deleted-with-undo-toast')
|
||||
|
||||
await toast.getByRole('button', { name: 'Undo' }).click()
|
||||
await expect(sectionHeading(page, 'Customers')).toBeVisible()
|
||||
await expect
|
||||
.poll(async () => (await layout(page)).map(([name, keys]) => [name, keys.length]))
|
||||
.toEqual([
|
||||
['Customers', 2],
|
||||
['Team', 0],
|
||||
['Unassigned', 2]
|
||||
])
|
||||
|
||||
// Membership rides the bot's profile ui_meta, so it follows profile sync.
|
||||
const alphaProfile = path.join(fixture!.sandbox.hermesHome, 'profiles', 'alpha', 'profile.yaml')
|
||||
await expect.poll(() => (fs.existsSync(alphaProfile) ? fs.readFileSync(alphaProfile, 'utf8') : '')).toMatch(/sectionId:\s*sec-/)
|
||||
|
||||
// Delete both sections: the roster is the plain list again.
|
||||
for (const name of ['Customers', 'Team']) {
|
||||
await sectionHeading(page, name).click({ button: 'right' })
|
||||
await page.getByRole('menuitem', { name: 'Delete' }).click()
|
||||
}
|
||||
|
||||
await expect(roster(page).locator('[data-slot="bots-section"]')).toHaveCount(0)
|
||||
await expect(botRow(page, 'alpha')).toBeVisible()
|
||||
})
|
||||
@@ -0,0 +1,139 @@
|
||||
/**
|
||||
* E2E chat tests — send a message and verify a response appears.
|
||||
*
|
||||
* Requires the full boot chain to complete (hermes serve + mock inference
|
||||
* provider). The mock server returns a canned reply, so we verify the
|
||||
* response text shows up in the chat transcript.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, test } from './test'
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { BLOCKING_CLARIFY_QUESTION, BLOCKING_CLARIFY_TRIGGER } from './mock-server'
|
||||
import { expectVisualSnapshot } from './visual-snapshot'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture!, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test.describe('chat interaction with mock backend', () => {
|
||||
test('send a message and receive a response', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
// Find the composer — it's a contenteditable textbox.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
|
||||
// Click to focus, then type the message character by character.
|
||||
// Using `type` instead of `fill` because the composer is a
|
||||
// contenteditable div with custom keydown handling that tracks
|
||||
// IME composition state — `fill` bypasses the event chain.
|
||||
await composer.click()
|
||||
await composer.type('Hello, can you hear me?', { delay: 20 })
|
||||
|
||||
// Submit with Enter — the composer's keydown handler intercepts
|
||||
// plain Enter (without Shift) and calls submitDraft().
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
// Wait for the user's message to appear in the transcript.
|
||||
// The message renders as an assistant-ui message in the chat view.
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
const body = document.body
|
||||
|
||||
if (!body) {
|
||||
return false
|
||||
}
|
||||
|
||||
return (body.textContent ?? '').includes('Hello, can you hear me?')
|
||||
},
|
||||
undefined,
|
||||
{ timeout: 15_000 }
|
||||
)
|
||||
|
||||
// Wait for the mock response to appear. The canned reply is:
|
||||
// "Hello from the mock inference server! The full boot chain is working."
|
||||
// Give it a generous timeout — the inference request goes through the
|
||||
// gateway → hermes serve → mock server → streaming SSE back.
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
const body = document.body
|
||||
|
||||
if (!body) {
|
||||
return false
|
||||
}
|
||||
|
||||
const text = body.textContent ?? ''
|
||||
|
||||
return text.includes('mock inference server') || text.includes('boot chain is working')
|
||||
},
|
||||
undefined,
|
||||
{ timeout: 60_000 }
|
||||
)
|
||||
})
|
||||
|
||||
test('screenshot of chat with messages', async () => {
|
||||
await expectVisualSnapshot(fixture!.page, { name: 'chat-with-messages', app: fixture!.app })
|
||||
})
|
||||
|
||||
test('offers stop, steer, and queue actions while busy', async ({}, testInfo) => {
|
||||
const page = fixture!.page
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
const primary = page.locator('[data-slot="composer-root"] button[type="submit"]')
|
||||
const queue = page.locator('[data-slot="composer-root"] button[aria-label="Queue message"]')
|
||||
const dictation = page.locator('[data-slot="composer-root"] button[aria-label="Voice dictation"]')
|
||||
const speakReplies = page.locator(
|
||||
'[data-slot="composer-root"] button[aria-label="Read replies aloud"], [data-slot="composer-root"] button[aria-label="Stop reading replies aloud"]'
|
||||
)
|
||||
|
||||
await composer.click()
|
||||
await composer.type(BLOCKING_CLARIFY_TRIGGER)
|
||||
await page.keyboard.press('Enter')
|
||||
await page.getByText(BLOCKING_CLARIFY_QUESTION).waitFor({ state: 'visible', timeout: 30_000 })
|
||||
|
||||
await expect(primary).toHaveAttribute('aria-label', 'Stop')
|
||||
await expect(primary.locator('span')).toHaveClass(/bg-current/)
|
||||
|
||||
await composer.click()
|
||||
await composer.type('please answer tersely')
|
||||
// Since "running is not busy" (3bc52fb9df) the primary keeps the Send
|
||||
// affordance mid-turn — steer is routed through the submit engine, not a
|
||||
// separate labeled button. Queue remains the explicit secondary action.
|
||||
await expect(primary).toHaveAttribute('aria-label', 'Send')
|
||||
await expect(dictation).toBeVisible()
|
||||
await expect(speakReplies).toBeVisible()
|
||||
await expect(queue).toBeVisible()
|
||||
await expect(queue.locator('svg.tabler-icon-layers-intersect-2')).toBeVisible()
|
||||
const controlLabels = await page
|
||||
.locator('[data-slot="composer-root"] button')
|
||||
.evaluateAll(buttons => buttons.map(button => button.getAttribute('aria-label')))
|
||||
const speakRepliesIndex = controlLabels.findIndex(
|
||||
label => label === 'Read replies aloud' || label === 'Stop reading replies aloud'
|
||||
)
|
||||
expect(controlLabels.indexOf('Voice dictation')).toBeLessThan(speakRepliesIndex)
|
||||
expect(speakRepliesIndex).toBeLessThan(controlLabels.indexOf('Queue message'))
|
||||
expect(controlLabels.indexOf('Queue message')).toBeLessThan(controlLabels.indexOf('Send'))
|
||||
await page.screenshot({ path: testInfo.outputPath('busy-composer-steer.png') })
|
||||
await expect(primary.locator('.codicon-arrow-up')).toBeVisible()
|
||||
|
||||
await queue.click()
|
||||
await expect(primary).toHaveAttribute('aria-label', 'Stop')
|
||||
await expect(queue).toHaveCount(0)
|
||||
await page.screenshot({ path: testInfo.outputPath('busy-composer-queue.png') })
|
||||
await expect(page.getByText('1 Queued')).toBeVisible()
|
||||
|
||||
await primary.click()
|
||||
await expect(page.getByText('1 Queued — paused')).toBeVisible()
|
||||
await page.screenshot({ path: testInfo.outputPath('busy-composer-queue-paused.png') })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* Context-menu edit verbs on real editables — the regressions jsdom cannot
|
||||
* catch, exercised against the real renderer (real radix focus trap, real
|
||||
* React unmount timing, real selection).
|
||||
*
|
||||
* The class under test: "Select all" from the app context menu must act on
|
||||
* the FIELD the menu was opened on, never on the surrounding transcript.
|
||||
* The first fix (focus-restore before dispatch) passed unit tests and still
|
||||
* failed live because the radix trap steals focus back; the second fix runs
|
||||
* selection renderer-side after the trap unmounts. These tests pin the
|
||||
* observable outcome, not the mechanism.
|
||||
*
|
||||
* Menu items are addressed by accessible-name PREFIX (`/^Copy/`): the name
|
||||
* includes the shortcut suffix ("Copy Ctrl+V" / "Copy ⌘V"), which is also
|
||||
* host-dependent.
|
||||
*/
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { expect, test } from './test'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('select all from the composer context menu selects the draft, not the chat', async () => {
|
||||
const page = fixture!.page
|
||||
const composer = page.locator('[data-slot="composer-rich-input"]').first()
|
||||
|
||||
// Put a message into the transcript so there is chat text a document-wide
|
||||
// select-all WOULD grab — the bug this test exists to catch. Wait for the
|
||||
// mock reply to COMPLETE: while the turn is busy the composer is in its
|
||||
// steer shape and a typed draft does not land in it.
|
||||
await composer.click()
|
||||
await composer.pressSequentially('transcript anchor message')
|
||||
await page.keyboard.press('Enter')
|
||||
await page.waitForFunction(() => (document.body.textContent ?? '').includes('mock inference server'), undefined, {
|
||||
timeout: 60_000
|
||||
})
|
||||
|
||||
// Draft text in the composer, then right-click it.
|
||||
await composer.click()
|
||||
await composer.pressSequentially('draft under selection')
|
||||
await composer.click({ button: 'right' })
|
||||
|
||||
const selectAll = page.getByRole('menuitem', { name: /^Select all/ })
|
||||
|
||||
await selectAll.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await selectAll.click()
|
||||
|
||||
// The selection must live inside the composer and cover exactly the draft.
|
||||
await expect
|
||||
.poll(
|
||||
() =>
|
||||
page.evaluate(() => {
|
||||
const selection = window.getSelection()
|
||||
const editable = document.querySelector('[data-slot="composer-rich-input"]')
|
||||
|
||||
if (!selection || selection.rangeCount === 0 || !editable) {
|
||||
return { inside: false, text: '' }
|
||||
}
|
||||
|
||||
return {
|
||||
inside: editable.contains(selection.getRangeAt(0).commonAncestorContainer),
|
||||
text: selection.toString()
|
||||
}
|
||||
}),
|
||||
{ timeout: 10_000 }
|
||||
)
|
||||
.toEqual({ inside: true, text: 'draft under selection' })
|
||||
|
||||
// Clear the draft so later tests start clean.
|
||||
await page.keyboard.press('Delete')
|
||||
})
|
||||
|
||||
test('cut, copy, and select all gray out in an empty composer', async () => {
|
||||
const page = fixture!.page
|
||||
const composer = page.locator('[data-slot="composer-rich-input"]').first()
|
||||
|
||||
await composer.click()
|
||||
await composer.click({ button: 'right' })
|
||||
|
||||
const selectAll = page.getByRole('menuitem', { name: /^Select all/ })
|
||||
|
||||
await selectAll.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
|
||||
await expect(selectAll).toHaveAttribute('data-disabled', /.*/)
|
||||
await expect(page.getByRole('menuitem', { name: /^Cut/ })).toHaveAttribute('data-disabled', /.*/)
|
||||
await expect(page.getByRole('menuitem', { name: /^Copy/ })).toHaveAttribute('data-disabled', /.*/)
|
||||
|
||||
await page.keyboard.press('Escape')
|
||||
})
|
||||
|
||||
test('paste enables when the clipboard holds text', async () => {
|
||||
const page = fixture!.page
|
||||
const composer = page.locator('[data-slot="composer-rich-input"]').first()
|
||||
|
||||
// The empty-clipboard branch stays in the unit suite: the e2e app shares
|
||||
// the SYSTEM clipboard, and writeText('') does not reliably clear it.
|
||||
await page.evaluate(() =>
|
||||
(
|
||||
window as unknown as { hermesDesktop?: { writeClipboard?: (text: string) => Promise<boolean> } }
|
||||
).hermesDesktop?.writeClipboard?.('clipboard payload')
|
||||
)
|
||||
await composer.click()
|
||||
await composer.click({ button: 'right' })
|
||||
|
||||
const paste = page.getByRole('menuitem', { name: /^Paste/ })
|
||||
|
||||
await paste.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
|
||||
// The clipboard probe is an async IPC — the item enables when it lands.
|
||||
await expect.poll(() => paste.getAttribute('data-disabled'), { timeout: 10_000 }).toBeNull()
|
||||
|
||||
await page.keyboard.press('Escape')
|
||||
})
|
||||
@@ -0,0 +1,270 @@
|
||||
/**
|
||||
* Regression coverage for a correction sent during a live response, then a
|
||||
* warm session switch away and back. The correction is an accepted user turn,
|
||||
* not an optimistic duplicate of the original prompt, and its relative place
|
||||
* in the transcript must survive the resume reconciliation.
|
||||
*/
|
||||
|
||||
import { type TestInfo } from '@playwright/test'
|
||||
|
||||
import { expect, test, type Page } from './test'
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { CORRECTION_SWITCH_TRIGGER, MOCK_REPLY } from './mock-server'
|
||||
|
||||
const OTHER_SESSION_PROMPT = 'E2E persisted session used for a warm resume.'
|
||||
const ORIGINAL_PROMPT = `${CORRECTION_SWITCH_TRIGGER}: original prompt must remain singular after a correction.`
|
||||
const CORRECTION = 'E2E correction must stay after the original prompt.'
|
||||
const TOOL_STARTED = 'Checking the long-running task before I continue.'
|
||||
const CORRECTED_REPLY = 'The corrected task finished.'
|
||||
const INFERENCE_SWITCH_TRIGGER = 'E2E_INFERENCE_SWITCH_TRIGGER'
|
||||
const INFERENCE_PROMPT = `${INFERENCE_SWITCH_TRIGGER}: original inference prompt must remain singular.`
|
||||
const INFERENCE_CORRECTION = `${INFERENCE_SWITCH_TRIGGER}: correction sent while inference is live.`
|
||||
|
||||
// Inactive tabs stay mounted under a data-pane-hidden ancestor. Match the
|
||||
// renderer's keep-alive visibility policy instead of relying on DOM order.
|
||||
const SURFACE = '[data-composer-target]:not([data-pane-hidden] [data-composer-target])'
|
||||
|
||||
function activeSurface(page: Page) {
|
||||
return page.locator(SURFACE).last()
|
||||
}
|
||||
|
||||
async function send(page: Page, text: string): Promise<void> {
|
||||
const composer = activeSurface(page).locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.type(text, { delay: 5 })
|
||||
await page.keyboard.press('Enter')
|
||||
}
|
||||
|
||||
async function steer(page: Page, text: string): Promise<void> {
|
||||
const surface = activeSurface(page)
|
||||
const composer = surface.locator('[contenteditable="true"]').first()
|
||||
const primary = surface.locator('[data-slot="composer-root"] button[type="submit"]')
|
||||
|
||||
await composer.waitFor({ state: 'visible', timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.type(text, { delay: 5 })
|
||||
// Since "running is not busy" (3bc52fb9df) the primary keeps the Send label
|
||||
// mid-turn; the submit engine still routes a text payload to steer.
|
||||
await expect(primary).toHaveAttribute('aria-label', 'Send')
|
||||
await primary.click()
|
||||
}
|
||||
|
||||
async function waitForTranscriptText(page: Page, text: string): Promise<void> {
|
||||
await page.waitForFunction(
|
||||
([expected, surfaceSelector]: [string, string]) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const active = surfaces[surfaces.length - 1]
|
||||
|
||||
return (active?.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected)
|
||||
},
|
||||
[text, SURFACE] as [string, string],
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
}
|
||||
|
||||
async function textNodeOccurrences(page: Page, text: string): Promise<number> {
|
||||
return page.evaluate(
|
||||
([expected, surfaceSelector]: [string, string]) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const viewport = surfaces[surfaces.length - 1]?.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
if (!viewport) return 0
|
||||
|
||||
const walker = document.createTreeWalker(viewport, NodeFilter.SHOW_TEXT)
|
||||
let count = 0
|
||||
while (walker.nextNode()) {
|
||||
if (walker.currentNode.textContent?.includes(expected)) {
|
||||
count += 1
|
||||
}
|
||||
}
|
||||
return count
|
||||
},
|
||||
[text, SURFACE] as [string, string],
|
||||
)
|
||||
}
|
||||
|
||||
async function transcriptTextOrder(page: Page): Promise<string[]> {
|
||||
return page.evaluate((surfaceSelector: string) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const viewport = surfaces[surfaces.length - 1]?.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
if (!viewport) return []
|
||||
|
||||
return Array.from(viewport.querySelectorAll<HTMLElement>('[data-role="message"], [data-message-id]'))
|
||||
.map(message => message.textContent?.trim() ?? '')
|
||||
.filter(Boolean)
|
||||
}, SURFACE)
|
||||
}
|
||||
|
||||
async function transcriptMessageOrder(page: Page): Promise<string[]> {
|
||||
return page.evaluate((surfaceSelector: string) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const viewport = surfaces[surfaces.length - 1]?.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
if (!viewport) return []
|
||||
|
||||
return Array.from(
|
||||
viewport.querySelectorAll<HTMLElement>('[data-role="user"], [data-role="assistant"], [data-role="system"]'),
|
||||
)
|
||||
.map(message => message.textContent?.trim() ?? '')
|
||||
.filter(Boolean)
|
||||
}, SURFACE)
|
||||
}
|
||||
|
||||
/**
|
||||
* The sidebar "+" opens a NEW TAB beside the current chat rather than
|
||||
* replacing it, so the prior session stays mounted in its own surface. Wait
|
||||
* for the newly-mounted surface to show an empty transcript instead of waiting
|
||||
* for the old text to disappear from the page (it never will).
|
||||
*/
|
||||
async function openFreshDraft(page: Page, priorSessionText: string): Promise<void> {
|
||||
await page.locator('[data-slot="sidebar"] button[aria-label="New session"]').first().click()
|
||||
await page.waitForFunction(
|
||||
([priorText, surfaceSelector]: [string, string]) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const active = surfaces[surfaces.length - 1]
|
||||
const transcript = active?.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? ''
|
||||
|
||||
return surfaces.length > 0 && !transcript.includes(priorText)
|
||||
},
|
||||
[priorSessionText, SURFACE] as [string, string],
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
}
|
||||
|
||||
async function openSidebarSession(page: Page, sidebarText: string, expectedTranscriptText: string): Promise<void> {
|
||||
const row = page.locator('[data-slot="sidebar"] button').filter({ hasText: sidebarText }).first()
|
||||
await row.waitFor({ state: 'visible', timeout: 30_000 })
|
||||
await row.click()
|
||||
await waitForTranscriptText(page, expectedTranscriptText)
|
||||
}
|
||||
|
||||
async function reopenOriginalSession(page: Page): Promise<void> {
|
||||
// A still-running tool has not generated a final title yet, so the sidebar
|
||||
// retains the source prompt as its provisional session title.
|
||||
await openSidebarSession(page, ORIGINAL_PROMPT, ORIGINAL_PROMPT)
|
||||
}
|
||||
|
||||
async function reopenInferenceSession(page: Page): Promise<void> {
|
||||
const row = page.locator('[data-slot="sidebar"] button').filter({ hasText: INFERENCE_PROMPT }).first()
|
||||
await row.waitFor({ state: 'visible', timeout: 30_000 })
|
||||
await row.click()
|
||||
await waitForTranscriptText(page, INFERENCE_PROMPT)
|
||||
}
|
||||
|
||||
function relevantOrder(messages: string[]): string[] {
|
||||
return messages.flatMap(message => {
|
||||
if (message.includes(ORIGINAL_PROMPT)) return [ORIGINAL_PROMPT]
|
||||
if (message.includes(CORRECTION)) return [CORRECTION]
|
||||
|
||||
return []
|
||||
})
|
||||
}
|
||||
|
||||
function steerTurnOrder(messages: string[]): string[] {
|
||||
return messages.flatMap(message => {
|
||||
if (message.includes(ORIGINAL_PROMPT)) return [ORIGINAL_PROMPT]
|
||||
if (message.includes(CORRECTION)) return [CORRECTION]
|
||||
if (message.includes(CORRECTED_REPLY)) return [CORRECTED_REPLY]
|
||||
|
||||
return []
|
||||
})
|
||||
}
|
||||
|
||||
test.describe('correction session switch', () => {
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeEach(async () => {
|
||||
fixture = await setupMockBackend({
|
||||
mockServer: { holdFirstStreamForPrompt: INFERENCE_SWITCH_TRIGGER },
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterEach(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('keeps a live correction in place and does not duplicate its original prompt after switching sessions', async ({}, testInfo: TestInfo) => {
|
||||
const { page } = fixture!
|
||||
|
||||
// A blank draft does not exercise session hydration. Seed a real second
|
||||
// session first, matching the observed switch between two saved chats.
|
||||
await send(page, OTHER_SESSION_PROMPT)
|
||||
await waitForTranscriptText(page, MOCK_REPLY)
|
||||
await openFreshDraft(page, OTHER_SESSION_PROMPT)
|
||||
|
||||
await send(page, ORIGINAL_PROMPT)
|
||||
await waitForTranscriptText(page, TOOL_STARTED)
|
||||
await waitForTranscriptText(page, ORIGINAL_PROMPT)
|
||||
|
||||
// The historical session redirects while a foreground terminal task is
|
||||
// running. Use the visible Steer action to cover the real composer path.
|
||||
await steer(page, CORRECTION)
|
||||
await waitForTranscriptText(page, CORRECTION)
|
||||
|
||||
const orderBeforeSwitch = relevantOrder(await transcriptTextOrder(page))
|
||||
expect(orderBeforeSwitch).toEqual([ORIGINAL_PROMPT, CORRECTION])
|
||||
expect(await textNodeOccurrences(page, ORIGINAL_PROMPT)).toBe(1)
|
||||
expect(await textNodeOccurrences(page, CORRECTION)).toBe(1)
|
||||
await page.screenshot({ path: testInfo.outputPath('correction-before-session-switch.png') })
|
||||
|
||||
// Reproduce the observed race: switch to another persisted session while
|
||||
// the foreground tool is live, then return before its redirect settles.
|
||||
// Sidebar rows title by the session's first user prompt (auto-title is
|
||||
// disabled in the e2e fixture config).
|
||||
await openSidebarSession(page, OTHER_SESSION_PROMPT, OTHER_SESSION_PROMPT)
|
||||
await reopenOriginalSession(page)
|
||||
// The warm resume first paints the persisted history and then reconciles
|
||||
// the live turn (including a steer whose persistence may lag on a loaded
|
||||
// runner) back in. Poll to the converged order instead of sampling one
|
||||
// arbitrary mid-reconcile frame; the duplicate checks then pin the
|
||||
// regression (the prompt/correction must appear exactly once).
|
||||
await expect
|
||||
.poll(async () => relevantOrder(await transcriptTextOrder(page)), {
|
||||
message: 'correction should stay in place after the warm resume',
|
||||
timeout: 30_000,
|
||||
})
|
||||
.toEqual(orderBeforeSwitch)
|
||||
await page.screenshot({ path: testInfo.outputPath('correction-after-warm-resume.png') })
|
||||
|
||||
expect(await textNodeOccurrences(page, ORIGINAL_PROMPT)).toBe(1)
|
||||
expect(await textNodeOccurrences(page, CORRECTION)).toBe(1)
|
||||
|
||||
await waitForTranscriptText(page, CORRECTED_REPLY)
|
||||
// The post-turn stored-history reconcile can momentarily repaint from a
|
||||
// snapshot in which the steer's user row hasn't been folded back in yet —
|
||||
// poll to the converged order instead of sampling one frame.
|
||||
await expect
|
||||
.poll(async () => steerTurnOrder(await transcriptMessageOrder(page)), {
|
||||
message: 'steered turn should settle as prompt → correction → corrected reply',
|
||||
timeout: 30_000,
|
||||
})
|
||||
.toEqual([ORIGINAL_PROMPT, CORRECTION, CORRECTED_REPLY])
|
||||
})
|
||||
|
||||
test('keeps an inference-time correction visible through a warm session switch', async ({}, testInfo: TestInfo) => {
|
||||
const { mock, page } = fixture!
|
||||
|
||||
await send(page, OTHER_SESSION_PROMPT)
|
||||
await waitForTranscriptText(page, MOCK_REPLY)
|
||||
await openFreshDraft(page, OTHER_SESSION_PROMPT)
|
||||
|
||||
await send(page, INFERENCE_PROMPT)
|
||||
await mock.waitForHeldStream()
|
||||
await waitForTranscriptText(page, INFERENCE_PROMPT)
|
||||
|
||||
await send(page, INFERENCE_CORRECTION)
|
||||
await waitForTranscriptText(page, INFERENCE_CORRECTION)
|
||||
|
||||
await openSidebarSession(page, OTHER_SESSION_PROMPT, OTHER_SESSION_PROMPT)
|
||||
await reopenInferenceSession(page)
|
||||
|
||||
expect(await textNodeOccurrences(page, INFERENCE_PROMPT)).toBe(1)
|
||||
expect(await textNodeOccurrences(page, INFERENCE_CORRECTION)).toBe(1)
|
||||
await page.screenshot({ path: testInfo.outputPath('inference-correction-after-warm-resume.png') })
|
||||
|
||||
mock.releaseHeldStream()
|
||||
await waitForTranscriptText(page, MOCK_REPLY)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,100 @@
|
||||
/**
|
||||
* Locating the dev Electron binary for the e2e fixtures.
|
||||
*
|
||||
* Kept in its own module so the resolution rules can be unit-tested without
|
||||
* importing the Playwright runner (fixtures.ts pulls in `_electron`, the mock
|
||||
* server and the error-banner guard).
|
||||
*
|
||||
* Three rules the previous single-path probe got wrong:
|
||||
*
|
||||
* 1. The binary is not always under the REPO ROOT. This is an npm workspaces
|
||||
* repo, and npm only hoists a dependency to the root when nothing conflicts
|
||||
* — otherwise `electron` installs into `apps/desktop/node_modules`. Both
|
||||
* layouts are normal, so both have to be searched, nearest package first.
|
||||
* 2. The binary is `electron.exe` on Windows. A bare `electron` never exists
|
||||
* there, so the probe could only ever miss.
|
||||
* 3. `which` is not a command on Windows. The PATH fallback spawned it
|
||||
* unconditionally, so on Windows the fallback failed for the wrong reason
|
||||
* and the error message blamed a missing `npm install`.
|
||||
*/
|
||||
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import * as fs from 'node:fs'
|
||||
import { createRequire } from 'node:module'
|
||||
import * as path from 'node:path'
|
||||
|
||||
/** The dist file name: `electron.exe` on Windows, `electron` elsewhere. */
|
||||
export function electronBinaryName(platform: NodeJS.Platform = process.platform): string {
|
||||
return platform === 'win32' ? 'electron.exe' : 'electron'
|
||||
}
|
||||
|
||||
/**
|
||||
* Where an npm install can leave the binary, in probe order: nearest package
|
||||
* first, so a workspace-local install wins over a stale hoisted one.
|
||||
*/
|
||||
export function electronDistCandidates(roots: string[], platform: NodeJS.Platform = process.platform): string[] {
|
||||
return roots.map((root) => path.join(root, 'node_modules', 'electron', 'dist', electronBinaryName(platform)))
|
||||
}
|
||||
|
||||
/** The PATH-lookup command for this platform. Windows has `where`, not `which`. */
|
||||
export function pathLookupCommand(platform: NodeJS.Platform = process.platform): string {
|
||||
return platform === 'win32' ? 'where' : 'which'
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask the installed `electron` package where its own binary is.
|
||||
*
|
||||
* Its main export IS the absolute executable path, resolved from `path.txt`
|
||||
* and honouring `ELECTRON_OVERRIDE_DIST_PATH`, so this covers layouts and
|
||||
* overrides a hand-built path cannot know about. Returns null when the package
|
||||
* is not resolvable from `from`, or when it does not hand back a path (the
|
||||
* export is the Electron API object, not a path, when required from inside
|
||||
* Electron itself).
|
||||
*/
|
||||
export function electronPackagePath(from: string): null | string {
|
||||
try {
|
||||
const resolved = createRequire(path.join(from, 'package.json'))('electron') as unknown
|
||||
|
||||
return typeof resolved === 'string' && resolved ? resolved : null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the Electron binary, or throw with the layouts that were searched.
|
||||
*
|
||||
* `roots` are searched in order; pass the desktop package before the repo root.
|
||||
*/
|
||||
export function resolveElectronBinary(roots: string[]): string {
|
||||
for (const root of roots) {
|
||||
const declared = electronPackagePath(root)
|
||||
|
||||
if (declared && fs.existsSync(declared)) {
|
||||
return declared
|
||||
}
|
||||
}
|
||||
|
||||
for (const candidate of electronDistCandidates(roots)) {
|
||||
if (fs.existsSync(candidate)) {
|
||||
return candidate
|
||||
}
|
||||
}
|
||||
|
||||
// Nix devshells put `electron` on PATH with no node_modules copy at all.
|
||||
const lookup = spawnSync(pathLookupCommand(), ['electron'], { encoding: 'utf8' })
|
||||
|
||||
if (lookup.status === 0 && lookup.stdout.trim()) {
|
||||
// `where` reports every match, one per line; take the first.
|
||||
const first = lookup.stdout.trim().split(/\r?\n/)[0].trim()
|
||||
|
||||
if (first) {
|
||||
return first
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`Electron binary not found. Searched ${electronDistCandidates(roots).join(', ')} and PATH. ` +
|
||||
'Run "npm install" from the repo root to install devDependencies.',
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import * as path from 'node:path'
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { electronBinaryName, electronDistCandidates, pathLookupCommand } from './electron-binary'
|
||||
|
||||
// Platform is a parameter everywhere below rather than read from
|
||||
// process.platform, so the Windows rules are pinned on the Linux CI runner too.
|
||||
// Reading the real platform would leave every Windows-only rule untested.
|
||||
|
||||
describe('electronBinaryName', () => {
|
||||
it('asks for electron.exe on Windows', () => {
|
||||
expect(electronBinaryName('win32')).toBe('electron.exe')
|
||||
})
|
||||
|
||||
it('asks for a bare electron everywhere else', () => {
|
||||
expect(electronBinaryName('linux')).toBe('electron')
|
||||
expect(electronBinaryName('darwin')).toBe('electron')
|
||||
})
|
||||
})
|
||||
|
||||
describe('electronDistCandidates', () => {
|
||||
const desktop = path.join('repo', 'apps', 'desktop')
|
||||
const repo = 'repo'
|
||||
|
||||
it('probes the workspace-local install before the hoisted one', () => {
|
||||
// npm only hoists `electron` to the repo root when nothing conflicts, so
|
||||
// apps/desktop/node_modules is an ordinary outcome of `npm install`, not a
|
||||
// broken tree. Probing only the repo root is what makes the suite refuse to
|
||||
// start with "run npm install" on a tree that has electron installed.
|
||||
expect(electronDistCandidates([desktop, repo], 'linux')).toEqual([
|
||||
path.join(desktop, 'node_modules', 'electron', 'dist', 'electron'),
|
||||
path.join(repo, 'node_modules', 'electron', 'dist', 'electron'),
|
||||
])
|
||||
})
|
||||
|
||||
it('carries the platform binary name into every candidate', () => {
|
||||
// A bare `electron` file never exists in a Windows dist, so a probe built
|
||||
// from a hardcoded name cannot match there no matter which root it walks.
|
||||
for (const candidate of electronDistCandidates([desktop, repo], 'win32')) {
|
||||
expect(path.basename(candidate)).toBe('electron.exe')
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('pathLookupCommand', () => {
|
||||
it('uses where on Windows and which elsewhere', () => {
|
||||
// `which` is not a command on Windows; spawning it unconditionally made the
|
||||
// PATH fallback fail for a reason unrelated to whether electron is on PATH.
|
||||
expect(pathLookupCommand('win32')).toBe('where')
|
||||
expect(pathLookupCommand('linux')).toBe('which')
|
||||
expect(pathLookupCommand('darwin')).toBe('which')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,72 @@
|
||||
/**
|
||||
* Monkey-patch: playwright's test runner never calls tracing.start() on
|
||||
* Electron's internal BrowserContext because:
|
||||
* 1. Playwright._allContexts() only returns [chromium, firefox, webkit]
|
||||
* contexts — Electron's context is excluded.
|
||||
* 2. ArtifactsRecorder.didCreateBrowserContext runs in willStartTest, before
|
||||
* beforeAll launches the electron app.
|
||||
* 3. The runAfterCreateBrowserContext hook doesn't exist on the Electron
|
||||
* class (only on BrowserType).
|
||||
*
|
||||
* As a result, trace screenshots (screencast) and DOM snapshots are never
|
||||
* captured for electron tests.
|
||||
*
|
||||
* This patch:
|
||||
* 1. Patches _allContexts() to include electron contexts, so the test
|
||||
* runner's didFinishTest() cleanup calls _stopTracing() → stopChunk()
|
||||
* on the electron context (saving the trace chunk + merging it into
|
||||
* the final trace.zip).
|
||||
* 2. Manually calls tracing.start() + startChunk() after launch.
|
||||
* 3. Wraps tracing.start to become startChunk after the first call,
|
||||
* so the test runner's willStartTest doesn't throw "already started".
|
||||
*
|
||||
* Imported from playwright.config.ts so it runs before any test.
|
||||
*
|
||||
* Pinned dependency: this file reaches into Playwright internals (_playwright,
|
||||
* _allContexts, _context) that have no public contract. @playwright/test is
|
||||
* pinned exact (=1.58.2 in package.json) so a bump can't silently break the
|
||||
* monkeypatch. When bumping, re-verify these private symbols still exist on
|
||||
* the Electron / PlaywrightInternal classes and that tracing still merges.
|
||||
*/
|
||||
|
||||
import { _electron as electron, type BrowserContext } from '@playwright/test'
|
||||
import * as crypto from 'node:crypto'
|
||||
|
||||
const electronContexts = new Set<BrowserContext>()
|
||||
const originalLaunch = electron.launch.bind(electron)
|
||||
|
||||
electron.launch = async (options: any) => {
|
||||
const app = await originalLaunch(options)
|
||||
const ctx = (app as any)._context as BrowserContext
|
||||
electronContexts.add(ctx)
|
||||
ctx.once('close', () => electronContexts.delete(ctx))
|
||||
|
||||
// Patch _allContexts so the test runner sees the electron context
|
||||
// (didFinishTest cleanup → _stopTracing → stopChunk → merge into trace.zip).
|
||||
const pw = (electron as any)._playwright as any
|
||||
if (pw && !pw.__electronTracingPatched) {
|
||||
pw.__electronTracingPatched = true
|
||||
const original = pw._allContexts.bind(pw)
|
||||
pw._allContexts = () => [...original(), ...electronContexts]
|
||||
}
|
||||
|
||||
// Start tracing — mirrors ArtifactsRecorder.didCreateBrowserContext.
|
||||
const traceName = crypto.randomUUID()
|
||||
await ctx.tracing.start({
|
||||
screenshots: true,
|
||||
snapshots: true,
|
||||
sources: true,
|
||||
}).catch(() => {})
|
||||
await ctx.tracing.startChunk({ title: 'electron', name: traceName }).catch(() => {})
|
||||
|
||||
// Wrap tracing.start to redirect to startChunk after the first call.
|
||||
// The test runner's willStartTest calls tracing.start() on all contexts
|
||||
// in _allContexts(). Since we already started, redirect to startChunk
|
||||
// to avoid "Tracing has been already started" errors.
|
||||
const tracing = ctx.tracing as any
|
||||
tracing.start = async (opts: any) => {
|
||||
return tracing.startChunk(opts)
|
||||
}
|
||||
|
||||
return app
|
||||
}
|
||||
@@ -0,0 +1,746 @@
|
||||
/**
|
||||
* Shared E2E fixtures for the Hermes desktop Playwright suite.
|
||||
*
|
||||
* Two fixture modes:
|
||||
*
|
||||
* 1. `mockBackend` — starts a mock inference server, writes a config.yaml
|
||||
* that points at it, and launches the desktop app so the full chain
|
||||
* (electron → hermes serve → provider → inference → renderer) is
|
||||
* exercised with a real backend but a fake LLM.
|
||||
*
|
||||
* 2. `noProvider` — launches the app with an empty config (no provider
|
||||
* configured). The onboarding overlay should appear. Used to test the
|
||||
* first-run flow without real credentials.
|
||||
*
|
||||
* Both modes launch the *dev* Electron app (`electron .` against the built
|
||||
* `dist/`), not the packaged binary. This avoids the multi-minute
|
||||
* `electron-builder --dir` step and matches `hermes desktop --source`. The
|
||||
* packaged-binary path is already covered by `launch.spec.ts`.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so that `dist/` exists.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs'
|
||||
import * as os from 'node:os'
|
||||
import * as path from 'node:path'
|
||||
|
||||
import { _electron, type ElectronApplication, type Page } from '@playwright/test'
|
||||
|
||||
import { resolveElectronBinary } from './electron-binary'
|
||||
import { startMockServer, type MockServerOptions } from './mock-server'
|
||||
import { installErrorBannerGuard } from './test'
|
||||
|
||||
const DESKTOP_ROOT = path.resolve(import.meta.dirname, '..')
|
||||
const REPO_ROOT = path.resolve(DESKTOP_ROOT, '..', '..')
|
||||
const RELEASE_ROOT = path.join(DESKTOP_ROOT, 'release')
|
||||
|
||||
// ─── Credential stripping (matches launch.spec.ts) ──────────────────────
|
||||
|
||||
const CREDENTIAL_SUFFIXES: string[] = [
|
||||
'_API_KEY',
|
||||
'_TOKEN',
|
||||
'_SECRET',
|
||||
'_PASSWORD',
|
||||
'_CREDENTIALS',
|
||||
'_ACCESS_KEY',
|
||||
'_PRIVATE_KEY',
|
||||
'_OAUTH_TOKEN',
|
||||
]
|
||||
|
||||
const CREDENTIAL_NAMES = new Set([
|
||||
'ANTHROPIC_BASE_URL',
|
||||
'ANTHROPIC_TOKEN',
|
||||
'AWS_ACCESS_KEY_ID',
|
||||
'AWS_SECRET_ACCESS_KEY',
|
||||
'AWS_SESSION_TOKEN',
|
||||
'CUSTOM_API_KEY',
|
||||
'GEMINI_BASE_URL',
|
||||
'OPENAI_BASE_URL',
|
||||
'OPENROUTER_BASE_URL',
|
||||
'OLLAMA_BASE_URL',
|
||||
'GROQ_BASE_URL',
|
||||
'XAI_BASE_URL',
|
||||
])
|
||||
|
||||
function isCredentialEnvVar(name: string): boolean {
|
||||
if (CREDENTIAL_NAMES.has(name)) {
|
||||
return true
|
||||
}
|
||||
|
||||
return CREDENTIAL_SUFFIXES.some((suffix) => name.endsWith(suffix))
|
||||
}
|
||||
|
||||
function stripCredentials(env: Record<string, string | undefined>): Record<string, string> {
|
||||
const clean: Record<string, string> = {}
|
||||
|
||||
for (const [key, value] of Object.entries(env)) {
|
||||
if (!value) {
|
||||
continue
|
||||
}
|
||||
|
||||
if (isCredentialEnvVar(key)) {
|
||||
continue
|
||||
}
|
||||
|
||||
clean[key] = value
|
||||
}
|
||||
|
||||
return clean
|
||||
}
|
||||
|
||||
// ─── Sandbox creation ──────────────────────────────────────────────────
|
||||
|
||||
export interface Sandbox {
|
||||
root: string
|
||||
hermesHome: string
|
||||
userDataDir: string
|
||||
cleanup: () => void
|
||||
}
|
||||
|
||||
export function createSandbox(prefix: string): Sandbox {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), `hermes-e2e-${prefix}-${Math.random()}`))
|
||||
const hermesHome = path.join(root, 'hermes-home')
|
||||
const userDataDir = path.join(root, 'electron-user-data')
|
||||
|
||||
fs.mkdirSync(hermesHome, { recursive: true })
|
||||
fs.mkdirSync(userDataDir, { recursive: true })
|
||||
|
||||
// Write a fixed window-state.json so the Electron window opens at a
|
||||
// consistent size — helps with visual regression screenshots. The
|
||||
// exact size is also enforced right before each screenshot (see
|
||||
// expectVisualSnapshot in visual-snapshot.ts) because window managers
|
||||
// may resize after launch.
|
||||
fs.writeFileSync(
|
||||
path.join(userDataDir, 'window-state.json'),
|
||||
JSON.stringify(
|
||||
{ x: 0, y: 0, width: 1220, height: 800, isMaximized: false },
|
||||
null,
|
||||
2,
|
||||
),
|
||||
'utf8',
|
||||
)
|
||||
|
||||
// Pin Chromium actual-size zoom (level 0) for the suite. Fresh installs
|
||||
// ship DEFAULT_ZOOM_LEVEL at the Appearance 90% preset, but Playwright
|
||||
// click hit-testing and the committed visual baselines were calibrated at
|
||||
// 100%. Without this file every sandbox would inherit the product default
|
||||
// and fail pointer interception + snapshot diffs.
|
||||
fs.writeFileSync(
|
||||
path.join(userDataDir, 'zoom-state.json'),
|
||||
JSON.stringify({ zoomLevel: 0 }, null, 2),
|
||||
'utf8',
|
||||
)
|
||||
|
||||
return {
|
||||
root,
|
||||
hermesHome,
|
||||
userDataDir,
|
||||
cleanup: () => {
|
||||
try {
|
||||
fs.rmSync(root, { recursive: true, force: true })
|
||||
} catch {
|
||||
// best-effort
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Config writing ─────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Write a config.yaml that pre-configures a mock provider pointing at the
|
||||
* mock inference server. The provider is set as the active model provider so
|
||||
* the desktop app skips onboarding and boots straight to the chat UI.
|
||||
*
|
||||
* @param extraDisplayConfig optional YAML lines appended to the `display:`
|
||||
* section, used by the interim-message e2e test.
|
||||
* @param extraConfig optional top-level YAML sections for a test scenario.
|
||||
* @param modelContextLength optional primary-model context limit.
|
||||
*/
|
||||
export function writeMockProviderConfig(
|
||||
hermesHome: string,
|
||||
mockUrl: string,
|
||||
extraDisplayConfig?: string,
|
||||
extraConfig?: string,
|
||||
modelContextLength?: number,
|
||||
): void {
|
||||
const configPath = path.join(hermesHome, 'config.yaml')
|
||||
|
||||
const displaySection = extraDisplayConfig
|
||||
? `\ndisplay:\n${extraDisplayConfig}\n`
|
||||
: ''
|
||||
|
||||
// Title generation rides the MAIN model since 87af576e60 (#83636), so every
|
||||
// completed turn fires an extra background /v1/chat/completions at the mock.
|
||||
// That request contains the whole conversation — trigger keywords included —
|
||||
// which advances the mock's scripted-turn indices and trips hold-for-prompt
|
||||
// matchers from a request no spec ever sent. Disable it by default (no e2e
|
||||
// spec asserts on session titles); a test that passes its own `auxiliary:`
|
||||
// section via extraConfig owns the whole section instead.
|
||||
const autoTitleDefault = extraConfig?.includes('auxiliary:')
|
||||
? ''
|
||||
: 'auxiliary:\n title_generation:\n enabled: false\n'
|
||||
|
||||
// The scripted turns run REAL terminal commands, and anything the guard
|
||||
// classifies as dangerous (e.g. the sidebar sentinel-wait loop) parks the
|
||||
// turn behind a Run/Reject approval card. The default 'smart' mode then
|
||||
// fires an aux LLM approval call at the SAME mock provider — consuming a
|
||||
// scripted-turn index and never resolving — so the turn stalls until the
|
||||
// spec times out (the CI failure mode for the sidebar-dot family). No e2e
|
||||
// spec asserts on the approval flow, so run gate-free by default; a test
|
||||
// that passes its own `approvals:` section via extraConfig owns it.
|
||||
const approvalsDefault = extraConfig?.includes('approvals:')
|
||||
? ''
|
||||
: 'approvals:\n mode: "off"\n'
|
||||
|
||||
const config = `# Auto-generated by E2E test fixtures
|
||||
model:
|
||||
default: mock-model
|
||||
provider: mock
|
||||
${modelContextLength ? ` context_length: ${modelContextLength}\n` : ''}providers:
|
||||
mock:
|
||||
api: ${mockUrl}/v1
|
||||
name: Mock
|
||||
api_mode: chat_completions
|
||||
key_env: MOCK_API_KEY
|
||||
models:
|
||||
mock-model: {}
|
||||
context_length: 4096
|
||||
${autoTitleDefault}${approvalsDefault}${displaySection}${extraConfig ? `\n${extraConfig.trim()}\n` : ''}`
|
||||
|
||||
fs.writeFileSync(configPath, config, 'utf8')
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a minimal .env with the mock API key. The key_env in config.yaml
|
||||
* references MOCK_API_KEY, so the backend resolves credentials from here.
|
||||
*/
|
||||
export function writeEnvFile(hermesHome: string, apiKey = 'e2e-mock-key'): void {
|
||||
const envPath = path.join(hermesHome, '.env')
|
||||
fs.writeFileSync(envPath, `MOCK_API_KEY=${apiKey}\n`, 'utf8')
|
||||
}
|
||||
|
||||
/**
|
||||
* Write an empty config (no providers). The desktop app should show the
|
||||
* onboarding overlay because no inference provider is configured.
|
||||
*/
|
||||
function writeEmptyConfig(hermesHome: string): void {
|
||||
const configPath = path.join(hermesHome, 'config.yaml')
|
||||
fs.writeFileSync(configPath, '# Auto-generated by E2E test fixtures — no providers configured\n', 'utf8')
|
||||
}
|
||||
|
||||
// ─── Env building ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Build the environment for the Electron app process.
|
||||
*
|
||||
* Key env vars:
|
||||
* - HERMES_HOME → sandbox hermes-home (isolated config/sessions)
|
||||
* - HERMES_DESKTOP_USER_DATA_DIR → sandbox electron-user-data
|
||||
* - HERMES_DESKTOP_IGNORE_EXISTING=1 → don't pick up `hermes` from PATH
|
||||
* (we want the dev checkout at REPO_ROOT)
|
||||
* - HERMES_DESKTOP_HERMES_ROOT → REPO_ROOT (dev checkout resolution)
|
||||
* - HERMES_DESKTOP_APP_NAME → unique-ish per test (avoids single-instance lock)
|
||||
* - XDG_RUNTIME_DIR → ensure Electron has a writable runtime dir on Linux
|
||||
*/
|
||||
export function buildAppEnv(sandbox: Sandbox, extra: Record<string, string> = {}): Record<string, string> {
|
||||
const clean = stripCredentials(process.env)
|
||||
|
||||
// XDG_RUNTIME_DIR is needed for Electron on Linux when running in a
|
||||
// headless/CI context — without it the zygote may fail to initialize.
|
||||
if (!clean.XDG_RUNTIME_DIR && process.env.XDG_RUNTIME_DIR) {
|
||||
clean.XDG_RUNTIME_DIR = process.env.XDG_RUNTIME_DIR
|
||||
}
|
||||
|
||||
// DISPLAY — needed for Electron to open a window.
|
||||
if (!clean.DISPLAY && process.env.DISPLAY) {
|
||||
clean.DISPLAY = process.env.DISPLAY
|
||||
}
|
||||
|
||||
return {
|
||||
...clean,
|
||||
HERMES_HOME: sandbox.hermesHome,
|
||||
HERMES_DESKTOP_USER_DATA_DIR: sandbox.userDataDir,
|
||||
HERMES_DESKTOP_IGNORE_EXISTING: '1',
|
||||
HERMES_DESKTOP_HERMES_ROOT: REPO_ROOT,
|
||||
HERMES_DESKTOP_APP_NAME: `HermesE2E-${Date.now()}`,
|
||||
// `app.close()` in teardown must exit even when a spec leaves a turn
|
||||
// mid-flight — otherwise the quit confirmation waits on a click that no
|
||||
// one is there to make, and the worker dies on a teardown timeout.
|
||||
HERMES_DESKTOP_SKIP_QUIT_CONFIRM: '1',
|
||||
// Clear dev-server override — we want the built dist/, not a vite server.
|
||||
// The dev-server check in main.ts looks for this env var; if it's set,
|
||||
// it loads from the vite URL instead of the local file.
|
||||
...extra,
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Electron launch ────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Verify that the desktop app has been built (dist/ exists). Playwright
|
||||
* tests can't run without it — the Electron main process loads
|
||||
* dist/electron-main.mjs and the renderer loads dist/index.html.
|
||||
*/
|
||||
function assertDistBuilt(): void {
|
||||
const distDir = path.join(DESKTOP_ROOT, 'dist')
|
||||
const electronMain = path.join(distDir, 'electron-main.mjs')
|
||||
const indexHtml = path.join(distDir, 'index.html')
|
||||
|
||||
if (!fs.existsSync(electronMain)) {
|
||||
throw new Error(
|
||||
`Desktop dist not built. Run 'cd apps/desktop && npm run build' first.\n` +
|
||||
`Missing: ${electronMain}`,
|
||||
)
|
||||
}
|
||||
|
||||
if (!fs.existsSync(indexHtml)) {
|
||||
throw new Error(
|
||||
`Desktop dist/index.html not found. Run 'cd apps/desktop && npm run build' first.\n` +
|
||||
`Missing: ${indexHtml}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the Electron binary. In the nix devshell, `electron` is on PATH.
|
||||
* As a fallback, use the node_modules/electron install from either package.
|
||||
*/
|
||||
export function findElectron(): string {
|
||||
// In dev mode, we use the `electron` binary directly (not the packaged app).
|
||||
// The dev:electron script in package.json does exactly this: `electron .`
|
||||
// after building. We replicate that here.
|
||||
//
|
||||
// The desktop package is searched first: npm workspaces only hoist
|
||||
// `electron` to the repo root when nothing conflicts, so a workspace-local
|
||||
// install is just as ordinary an outcome as a hoisted one. The rules live in
|
||||
// ./electron-binary so they can be unit-tested per platform.
|
||||
return resolveElectronBinary([DESKTOP_ROOT, REPO_ROOT])
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch the desktop app in dev mode.
|
||||
*
|
||||
* @param sandbox - isolated HERMES_HOME + userData
|
||||
* @param env - the process environment (already has HERMES_HOME etc.)
|
||||
* @returns the ElectronApplication + first Page
|
||||
*/
|
||||
export async function launchDesktop(
|
||||
env: Record<string, string>,
|
||||
): Promise<{ app: ElectronApplication; page: Page }> {
|
||||
assertDistBuilt()
|
||||
|
||||
const electronBin = findElectron()
|
||||
|
||||
// `electron .` loads from the package.json `main` field
|
||||
// (dist/electron-main.mjs after build).
|
||||
const app = await _electron.launch({
|
||||
executablePath: electronBin,
|
||||
args: [
|
||||
DESKTOP_ROOT, // `electron .` — the `.` is the desktop package dir
|
||||
'--disable-gpu',
|
||||
'--no-sandbox',
|
||||
],
|
||||
env,
|
||||
cwd: DESKTOP_ROOT,
|
||||
})
|
||||
|
||||
const page = await app.firstWindow()
|
||||
|
||||
// Install the error-banner guard so any [role="alert"] that appears
|
||||
// during a test is collected and surfaced in afterEach.
|
||||
installErrorBannerGuard(page)
|
||||
|
||||
return { app, page }
|
||||
}
|
||||
|
||||
// ─── Public fixtures ────────────────────────────────────────────────────
|
||||
|
||||
export interface MockBackendFixture {
|
||||
app: ElectronApplication
|
||||
page: Page
|
||||
mock: Awaited<ReturnType<typeof startMockServer>>
|
||||
mockUrl: string
|
||||
sandbox: Sandbox
|
||||
cleanup: () => Promise<void>
|
||||
}
|
||||
|
||||
export interface MockBackendOptions {
|
||||
/**
|
||||
* Optional YAML lines to inject under the `display:` section of the
|
||||
* generated config.yaml. Used by the interim-message e2e test to toggle
|
||||
* `display.interim_assistant_messages`.
|
||||
*/
|
||||
extraDisplayConfig?: string
|
||||
/** Additional top-level config.yaml sections for an E2E scenario. */
|
||||
extraConfig?: string
|
||||
/** Override the mock model's context window for compression scenarios. */
|
||||
modelContextLength?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Set up a full mock-backend E2E environment:
|
||||
* 1. Start the mock inference server
|
||||
* 2. Create a sandbox with config.yaml pointing at it
|
||||
* 3. Launch the desktop app
|
||||
* 4. Return handles for test interaction
|
||||
*/
|
||||
export interface MockBackendOptions {
|
||||
mockServer?: MockServerOptions
|
||||
}
|
||||
|
||||
export async function setupMockBackend(options: MockBackendOptions = {}): Promise<MockBackendFixture> {
|
||||
// 1. Start mock server
|
||||
const mock = await startMockServer(options.mockServer)
|
||||
|
||||
// 2. Create sandbox + write config
|
||||
const sandbox = createSandbox('mock')
|
||||
writeMockProviderConfig(
|
||||
sandbox.hermesHome,
|
||||
mock.url,
|
||||
options.extraDisplayConfig,
|
||||
options.extraConfig,
|
||||
options.modelContextLength,
|
||||
)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
|
||||
// 3. Build env + launch
|
||||
const env = buildAppEnv(sandbox)
|
||||
const { app, page } = await launchDesktop(env)
|
||||
|
||||
return {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
export interface NoProviderFixture {
|
||||
app: ElectronApplication
|
||||
page: Page
|
||||
sandbox: Sandbox
|
||||
cleanup: () => Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch the app with no provider configured. The onboarding overlay should
|
||||
* appear because there's no inference provider in config.yaml.
|
||||
*/
|
||||
export async function setupNoProvider(): Promise<NoProviderFixture> {
|
||||
const sandbox = createSandbox('noprovider')
|
||||
writeEmptyConfig(sandbox.hermesHome)
|
||||
|
||||
const env = buildAppEnv(sandbox)
|
||||
const { app, page } = await launchDesktop(env)
|
||||
|
||||
return {
|
||||
app,
|
||||
page,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
export interface DeadBackendFixture {
|
||||
app: ElectronApplication
|
||||
page: Page
|
||||
sandbox: Sandbox
|
||||
cleanup: () => Promise<void>
|
||||
}
|
||||
|
||||
export interface DeadBackendOptions {
|
||||
/**
|
||||
* When true, inject a fake boot error via HERMES_DESKTOP_BOOT_FAKE_ERROR
|
||||
* so the backend resolution itself "fails" with a controlled error message.
|
||||
* This is the only reliable way to trigger BootFailureOverlay in dev mode
|
||||
* (the real backend always resolves via SOURCE_REPO_ROOT).
|
||||
*/
|
||||
fakeError?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch the app with a provider pointing at a dead endpoint (port 1, which
|
||||
* nothing listens on). By default the backend still boots (`hermes serve`
|
||||
* starts fine — the dead endpoint only matters at chat time). Pass
|
||||
* `{ fakeError: true }` to inject a fake boot failure, triggering the
|
||||
* BootFailureOverlay.
|
||||
*/
|
||||
export async function setupDeadBackend(options: DeadBackendOptions = {}): Promise<DeadBackendFixture> {
|
||||
const sandbox = createSandbox('dead')
|
||||
const configPath = path.join(sandbox.hermesHome, 'config.yaml')
|
||||
fs.writeFileSync(
|
||||
configPath,
|
||||
`# Auto-generated by E2E test fixtures — dead provider
|
||||
model:
|
||||
default: mock-model
|
||||
provider: mock
|
||||
providers:
|
||||
mock:
|
||||
api: http://127.0.0.1:1/v1
|
||||
name: Mock
|
||||
api_mode: chat_completions
|
||||
key_env: MOCK_API_KEY
|
||||
models:
|
||||
mock-model: {}
|
||||
context_length: 4096
|
||||
`,
|
||||
'utf8',
|
||||
)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
|
||||
const env = buildAppEnv(sandbox, options.fakeError ? { HERMES_DESKTOP_BOOT_FAKE_ERROR: 'Failed to connect to Hermes backend: connection refused' } : {})
|
||||
const { app, page } = await launchDesktop(env)
|
||||
|
||||
return {
|
||||
app,
|
||||
page,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Packaged-binary fixture ───────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Resolve the packaged Electron binary path, per-platform, matching
|
||||
* electron-builder's output layout under release/.
|
||||
*/
|
||||
function resolvePackagedBinaryPath(): string {
|
||||
if (process.platform === 'win32') {
|
||||
return path.join(RELEASE_ROOT, 'win-unpacked', 'Hermes.exe')
|
||||
}
|
||||
|
||||
if (process.platform === 'darwin') {
|
||||
const arch = process.arch === 'arm64' ? 'arm64' : 'x64'
|
||||
|
||||
return path.join(RELEASE_ROOT, `mac-${arch}`, 'Hermes.app', 'Contents', 'MacOS', 'Hermes')
|
||||
}
|
||||
|
||||
return path.join(RELEASE_ROOT, 'linux-unpacked', 'hermes')
|
||||
}
|
||||
|
||||
export const PACKAGED_BINARY_PATH = resolvePackagedBinaryPath()
|
||||
|
||||
export function packagedBinaryExists(): boolean {
|
||||
return fs.existsSync(PACKAGED_BINARY_PATH)
|
||||
}
|
||||
|
||||
export interface PackagedAppFixture {
|
||||
app: ElectronApplication
|
||||
page: Page
|
||||
sandbox: Sandbox
|
||||
cleanup: () => Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch the *packaged* Electron binary (from `npm run pack` →
|
||||
* `electron-builder --dir`) with `BOOT_FAKE=1` so it simulates boot
|
||||
* progress without spawning a real Hermes backend.
|
||||
*
|
||||
* Uses the same sandbox isolation (credential stripping, isolated
|
||||
* HERMES_HOME + userData, unique app name) as the dev-mode fixtures.
|
||||
*
|
||||
* Skips if the packaged binary doesn't exist — run `npm run pack` first.
|
||||
*/
|
||||
export async function setupPackagedApp(): Promise<PackagedAppFixture> {
|
||||
if (!packagedBinaryExists()) {
|
||||
throw new Error(
|
||||
`Built app binary not found: ${PACKAGED_BINARY_PATH}. Run 'npm run pack' first.`,
|
||||
)
|
||||
}
|
||||
|
||||
const sandbox = createSandbox('packaged')
|
||||
|
||||
// Build the sandbox env using the shared helpers, then add the
|
||||
// packaged-binary-specific overrides.
|
||||
const env = buildAppEnv(sandbox, {
|
||||
// Fake boot: simulates progress steps without spawning the real backend.
|
||||
HERMES_DESKTOP_BOOT_FAKE: '1',
|
||||
HERMES_DESKTOP_BOOT_FAKE_STEP_MS: '120',
|
||||
})
|
||||
|
||||
// Clear dev-server + hermes-root overrides — the packaged binary
|
||||
// should use its own bundled renderer, not the dev checkout.
|
||||
delete (env as Record<string, string | undefined>).HERMES_DESKTOP_DEV_SERVER
|
||||
delete (env as Record<string, string | undefined>).HERMES_DESKTOP_HERMES
|
||||
delete (env as Record<string, string | undefined>).HERMES_DESKTOP_HERMES_ROOT
|
||||
|
||||
const app = await _electron.launch({
|
||||
executablePath: PACKAGED_BINARY_PATH,
|
||||
args: ['--disable-gpu', '--no-sandbox'],
|
||||
env,
|
||||
})
|
||||
|
||||
const page = await app.firstWindow()
|
||||
installErrorBannerGuard(page)
|
||||
|
||||
return {
|
||||
app,
|
||||
page,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Wait helpers ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Wait for the desktop app to finish booting and show the main chat UI.
|
||||
*
|
||||
* The boot overlay disappears when `completeDesktopBoot()` fires in the
|
||||
* renderer — at that point the gateway is open, config is loaded, and
|
||||
* sessions are loaded. We detect this by waiting for the boot/connecting
|
||||
* overlay to become invisible and the main app shell to be present.
|
||||
*
|
||||
* Two things must both be true before we return:
|
||||
* 1. The composer (chat input) is visible — it's disabled until the
|
||||
* gateway is open.
|
||||
* 2. No full-screen overlay (onboarding Preparing, connecting overlay,
|
||||
* boot-failure) covers the viewport center. The composer can be
|
||||
* "visible" in Playwright's eyes (non-zero bounding box, not
|
||||
* display:none) even when a z-1300+ overlay is painted on top of it,
|
||||
* so checking the composer alone catches the app mid-boot at ~92%
|
||||
* with the loading bar still showing.
|
||||
*/
|
||||
export async function waitForAppReady(fixture: MockBackendFixture | NoProviderFixture | DeadBackendFixture, timeoutMs = 60_000): Promise<void> {
|
||||
const { page, app } = fixture
|
||||
|
||||
// Wait for the composer to exist in the DOM (not necessarily interactive yet).
|
||||
await page.waitForSelector('textarea, [contenteditable="true"]', {
|
||||
state: 'attached',
|
||||
timeout: timeoutMs,
|
||||
})
|
||||
|
||||
// Now poll until no full-screen overlay covers the viewport center.
|
||||
// elementFromPoint returns the topmost element at a point — if it's part
|
||||
// of a fixed inset-0 overlay (onboarding/connecting/boot-failure), the
|
||||
// app isn't ready yet.
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
const el = document.elementFromPoint(window.innerWidth / 2, window.innerHeight / 2)
|
||||
|
||||
if (!el) {
|
||||
return false
|
||||
}
|
||||
|
||||
// Walk up to the nearest positioned ancestor — overlays are
|
||||
// `position: fixed; inset: 0`. If the hit element or an ancestor
|
||||
// is a full-viewport fixed overlay, we're still covered.
|
||||
let node: Element | null = el
|
||||
while (node) {
|
||||
const cs = window.getComputedStyle(node)
|
||||
|
||||
if (cs.position === 'fixed') {
|
||||
const rect = node.getBoundingClientRect()
|
||||
|
||||
if (rect.left <= 0 && rect.top <= 0 && rect.right >= window.innerWidth && rect.bottom >= window.innerHeight) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
node = node.parentElement
|
||||
}
|
||||
|
||||
return true
|
||||
},
|
||||
undefined,
|
||||
{ timeout: timeoutMs },
|
||||
)
|
||||
|
||||
// On Electron 40.x, ready-to-show may never fire (electron/electron#51972)
|
||||
// and the window stays hidden even though the DOM is rendered. The main
|
||||
// process reveals it anyway — immediately under TEST_WORKER_INDEX, and via
|
||||
// wireWindowReveal's post-load fallback in production — but the DOM can be
|
||||
// ready before that lands. Poll until the window is actually visible so
|
||||
// interactions (click, screenshot) don't hit a hidden surface.
|
||||
if (app) {
|
||||
const deadline = Date.now() + timeoutMs
|
||||
|
||||
while (Date.now() < deadline) {
|
||||
const visible = await app.evaluate(({ BrowserWindow }) => {
|
||||
const w = BrowserWindow.getAllWindows()[0]
|
||||
|
||||
return w ? w.isVisible() : false
|
||||
}).catch(() => false)
|
||||
|
||||
if (visible) {break}
|
||||
await page.waitForTimeout(500)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for the onboarding overlay to appear (no provider configured).
|
||||
*/
|
||||
export async function waitForOnboarding(page: Page, timeoutMs = 60_000): Promise<void> {
|
||||
// The onboarding overlay contains a heading with "Choose your provider"
|
||||
// or similar text. We look for any text that indicates the picker.
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
const root = document.getElementById('root')
|
||||
|
||||
if (!root) {
|
||||
return false
|
||||
}
|
||||
|
||||
const text = root.textContent ?? ''
|
||||
|
||||
return (
|
||||
text.includes('provider') ||
|
||||
text.includes('Provider') ||
|
||||
text.includes('Choose') ||
|
||||
text.includes('API key') ||
|
||||
text.includes('Sign in')
|
||||
)
|
||||
},
|
||||
undefined,
|
||||
{ timeout: timeoutMs },
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for the boot failure overlay to appear.
|
||||
*/
|
||||
export async function waitForBootFailure(page: Page, timeoutMs = 60_000): Promise<void> {
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
// Boot failure is terminal: the backend gave up. The renderer shows
|
||||
// either BootFailureOverlay (z-1400, with Retry/Repair buttons) or
|
||||
// falls back to the onboarding picker (z-1300) as a recovery path.
|
||||
// We wait for the failure dialog itself — the Preparing component may
|
||||
// still paint its progress bar (recolored red) underneath the overlay,
|
||||
// which is harmless.
|
||||
const text = document.body.textContent ?? ''
|
||||
|
||||
// BootFailureOverlay buttons.
|
||||
const hasFailureUI =
|
||||
text.includes('Retry') ||
|
||||
text.includes('Repair') ||
|
||||
text.includes('Use local gateway') ||
|
||||
text.includes('Connection settings')
|
||||
|
||||
// The error toast / notification that fires on failDesktopBoot().
|
||||
const hasErrorToast = text.includes('Desktop boot failed')
|
||||
|
||||
return hasFailureUI || hasErrorToast
|
||||
},
|
||||
undefined,
|
||||
{ timeout: timeoutMs },
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
/**
|
||||
* E2E: the fleet profile rail with two registered gateways.
|
||||
*
|
||||
* "This device" is the Electron-managed local backend (mock inference). The
|
||||
* second gateway, "Homelab", is a REAL second `hermes serve` this spec spawns
|
||||
* with its own HERMES_HOME, profiles and session token, registered in the v2
|
||||
* connections.json as a remote URL connection. A click on an at-rest square
|
||||
* therefore performs the same dial → commit → re-home the statusbar switcher
|
||||
* does, against a real backend — not a stub.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists, and the
|
||||
* repo's Python venv (`.venv`) must exist for both backends.
|
||||
*/
|
||||
|
||||
import { type ChildProcess, spawn, spawnSync } from 'node:child_process'
|
||||
import * as fs from 'node:fs'
|
||||
import * as net from 'node:net'
|
||||
import * as path from 'node:path'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type MockBackendFixture,
|
||||
type Sandbox,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig,
|
||||
} from './fixtures'
|
||||
import { startMockServer } from './mock-server'
|
||||
import { type ElectronApplication, expect, type Page, test } from './test'
|
||||
|
||||
const DESKTOP_ROOT = path.resolve(import.meta.dirname, '..')
|
||||
const REPO_ROOT = path.resolve(DESKTOP_ROOT, '..', '..')
|
||||
|
||||
const REMOTE_LABEL = 'Homelab'
|
||||
const REMOTE_ID = 'homelab'
|
||||
const REMOTE_TOKEN = 'e2e-fleet-homelab-token'
|
||||
|
||||
interface RemoteGateway {
|
||||
url: string
|
||||
home: string
|
||||
close: () => Promise<void>
|
||||
}
|
||||
|
||||
function findHermesBinary(): string {
|
||||
const venv = path.join(REPO_ROOT, '.venv', 'bin', 'hermes')
|
||||
|
||||
if (fs.existsSync(venv)) {
|
||||
return venv
|
||||
}
|
||||
|
||||
const result = spawnSync('which', ['hermes'], { encoding: 'utf8' })
|
||||
|
||||
if (result.status === 0 && result.stdout.trim()) {
|
||||
return result.stdout.trim()
|
||||
}
|
||||
|
||||
throw new Error('hermes binary not found: create the repo venv (uv sync) or put hermes on PATH')
|
||||
}
|
||||
|
||||
async function freePort(): Promise<number> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const server = net.createServer()
|
||||
server.unref()
|
||||
server.on('error', reject)
|
||||
server.listen(0, '127.0.0.1', () => {
|
||||
const { port } = server.address() as net.AddressInfo
|
||||
server.close(() => resolve(port))
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
/** Seed `<home>/profiles/<name>/` so the backend's /api/profiles lists it. */
|
||||
function seedProfiles(home: string, names: string[]): void {
|
||||
for (const name of names) {
|
||||
const dir = path.join(home, 'profiles', name)
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
fs.writeFileSync(path.join(dir, 'config.yaml'), '', 'utf8')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Spawn a second, fully real `hermes serve` as the remote gateway. Its
|
||||
* session token is pinned through HERMES_DASHBOARD_SESSION_TOKEN so the
|
||||
* registry entry can carry a plaintext token envelope.
|
||||
*/
|
||||
async function startRemoteGateway(root: string, mockUrl: string, profiles: string[]): Promise<RemoteGateway> {
|
||||
const home = path.join(root, 'homelab-home')
|
||||
fs.mkdirSync(home, { recursive: true })
|
||||
writeMockProviderConfig(home, mockUrl)
|
||||
writeEnvFile(home)
|
||||
seedProfiles(home, profiles)
|
||||
|
||||
const port = await freePort()
|
||||
const url = `http://127.0.0.1:${port}`
|
||||
|
||||
const child: ChildProcess = spawn(
|
||||
findHermesBinary(),
|
||||
['serve', '--host', '127.0.0.1', '--port', String(port), '--skip-build'],
|
||||
{
|
||||
cwd: REPO_ROOT,
|
||||
detached: true,
|
||||
env: {
|
||||
...process.env,
|
||||
HERMES_HOME: home,
|
||||
HERMES_DASHBOARD_SESSION_TOKEN: REMOTE_TOKEN,
|
||||
},
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
},
|
||||
)
|
||||
|
||||
let log = ''
|
||||
child.stdout?.on('data', (chunk: Buffer) => {
|
||||
log += chunk.toString()
|
||||
})
|
||||
child.stderr?.on('data', (chunk: Buffer) => {
|
||||
log += chunk.toString()
|
||||
})
|
||||
|
||||
const deadline = Date.now() + 90_000
|
||||
|
||||
while (Date.now() < deadline) {
|
||||
if (child.exitCode !== null) {
|
||||
throw new Error(`remote hermes serve exited early (${child.exitCode}):\n${log}`)
|
||||
}
|
||||
|
||||
try {
|
||||
const response = await fetch(`${url}/api/status`, {
|
||||
headers: { 'X-Hermes-Session-Token': REMOTE_TOKEN },
|
||||
})
|
||||
|
||||
if (response.ok) {
|
||||
break
|
||||
}
|
||||
} catch {
|
||||
// not up yet
|
||||
}
|
||||
|
||||
await new Promise(resolve => setTimeout(resolve, 500))
|
||||
}
|
||||
|
||||
if (Date.now() >= deadline) {
|
||||
throw new Error(`remote hermes serve never became ready:\n${log}`)
|
||||
}
|
||||
|
||||
return {
|
||||
url,
|
||||
home,
|
||||
close: async () => {
|
||||
if (child.pid && child.exitCode === null) {
|
||||
try {
|
||||
process.kill(-child.pid, 'SIGTERM')
|
||||
} catch {
|
||||
child.kill('SIGTERM')
|
||||
}
|
||||
}
|
||||
|
||||
await new Promise(resolve => setTimeout(resolve, 500))
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function writeConnectionsRegistry(sandbox: Sandbox, remoteUrl: string): void {
|
||||
fs.writeFileSync(
|
||||
path.join(sandbox.userDataDir, 'connections.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
version: 2,
|
||||
primary: 'local',
|
||||
launchMode: 'primary',
|
||||
lastUsed: 'local',
|
||||
connections: [
|
||||
{ id: 'local', kind: 'local', label: 'This device' },
|
||||
{
|
||||
id: REMOTE_ID,
|
||||
kind: 'remote',
|
||||
label: REMOTE_LABEL,
|
||||
url: remoteUrl,
|
||||
authMode: 'token',
|
||||
token: { encoding: 'plain', value: REMOTE_TOKEN },
|
||||
},
|
||||
],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
{ encoding: 'utf8', mode: 0o600 },
|
||||
)
|
||||
}
|
||||
|
||||
// FLEET_RAIL_SCREENSHOT_DIR=<dir> saves full-window captures at the key
|
||||
// states — handy for design review; never part of the assertions.
|
||||
async function capture(page: Page, name: string): Promise<void> {
|
||||
const dir = process.env.FLEET_RAIL_SCREENSHOT_DIR
|
||||
|
||||
if (!dir) {
|
||||
return
|
||||
}
|
||||
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
await page.screenshot({ path: path.join(dir, `${name}.png`) })
|
||||
}
|
||||
|
||||
const rail = (page: Page) => page.locator('[data-slot="profile-rail"]')
|
||||
const gatewayGroup = (page: Page, id: string) => rail(page).locator(`[data-slot="profile-rail-gateway"][data-connection-id="${id}"]`)
|
||||
const activeGatewayLabel = (page: Page) => page.getByRole('button', { name: /^Registered gateways: / })
|
||||
|
||||
async function groupOrder(page: Page): Promise<Array<[string, boolean]>> {
|
||||
return rail(page).locator('[data-slot="profile-rail-gateway"]').evaluateAll(nodes =>
|
||||
nodes.map(node => [node.getAttribute('data-connection-id') ?? '', node.getAttribute('data-active') === 'true'] as [string, boolean]),
|
||||
)
|
||||
}
|
||||
|
||||
test.describe('fleet profile rail — two registered gateways', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let mock: Awaited<ReturnType<typeof startMockServer>>
|
||||
let sandbox: Sandbox
|
||||
let remote: RemoteGateway
|
||||
let app: ElectronApplication
|
||||
let page: Page
|
||||
|
||||
test.beforeAll(async () => {
|
||||
test.setTimeout(240_000)
|
||||
mock = await startMockServer()
|
||||
sandbox = createSandbox('fleet')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
// A named profile on This device too, so the active group has a square
|
||||
// beside its home pill. "research" exists on BOTH gateways on purpose: the
|
||||
// rail must keep the two apart by gateway, never by name alone.
|
||||
seedProfiles(sandbox.hermesHome, ['research'])
|
||||
|
||||
remote = await startRemoteGateway(sandbox.root, mock.url, ['inbox', 'research'])
|
||||
writeConnectionsRegistry(sandbox, remote.url)
|
||||
|
||||
;({ app, page } = await launchDesktop(buildAppEnv(sandbox)))
|
||||
await waitForAppReady({ app, page } as MockBackendFixture, 120_000)
|
||||
// Let boot settle fully (the gateway health item reports "ready" once the
|
||||
// primary socket is open) so the boot-time launch-mode restore has run
|
||||
// before any click — the rail must then hold whatever the user picks.
|
||||
await expect(page.locator('[data-slot="statusbar"]').getByText('ready', { exact: true })).toBeVisible({ timeout: 120_000 })
|
||||
await page.waitForTimeout(2_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await app?.close().catch(() => undefined)
|
||||
await remote?.close()
|
||||
await mock?.close()
|
||||
sandbox?.cleanup()
|
||||
})
|
||||
|
||||
test('lays both gateways on one strip, active gateway in its registry slot', async () => {
|
||||
// The statusbar readout names the gateway the workspace is on.
|
||||
await expect(activeGatewayLabel(page)).toHaveAttribute('aria-label', 'Registered gateways: This device', { timeout: 60_000 })
|
||||
|
||||
// The remote gateway's group appears once the roster has enumerated it.
|
||||
const homelab = gatewayGroup(page, REMOTE_ID)
|
||||
await expect(homelab).toBeVisible({ timeout: 60_000 })
|
||||
await expect(homelab.getByRole('button', { name: `default · ${REMOTE_LABEL}` })).toBeVisible()
|
||||
await expect(homelab.getByRole('button', { name: `inbox · ${REMOTE_LABEL}` })).toBeVisible()
|
||||
await expect(homelab.getByRole('button', { name: `research · ${REMOTE_LABEL}` })).toBeVisible()
|
||||
await expect(homelab).toHaveAttribute('data-reachable', 'true')
|
||||
|
||||
// Its marker carries the remote (network) glyph.
|
||||
await expect(
|
||||
rail(page).locator(`[data-slot="profile-rail-divider"][data-connection-id="${REMOTE_ID}"] [data-connection-kind="remote"]`),
|
||||
).toBeVisible()
|
||||
|
||||
// This device is the active group: its squares are unqualified, as before.
|
||||
const local = gatewayGroup(page, 'local')
|
||||
await expect(local).toHaveAttribute('data-active', 'true')
|
||||
await expect(local.getByRole('button', { name: 'research', exact: true })).toBeVisible()
|
||||
|
||||
// Registry order: This device first, Homelab second.
|
||||
expect(await groupOrder(page)).toEqual([
|
||||
['local', true],
|
||||
[REMOTE_ID, false],
|
||||
])
|
||||
|
||||
// Fleet pill replaces the default↔all toggle; the single-gateway plug is gone.
|
||||
await expect(rail(page).getByRole('button', { name: 'All profiles on this gateway' })).toBeVisible()
|
||||
await expect(rail(page).getByRole('button', { name: 'Manage gateways…' })).toHaveCount(0)
|
||||
|
||||
await gatewayGroup(page, REMOTE_ID).getByRole('button', { name: `inbox · ${REMOTE_LABEL}` }).hover()
|
||||
await capture(page, '1-on-this-device-hover-inbox-homelab')
|
||||
})
|
||||
|
||||
test('clicking an at-rest square re-homes onto that exact gateway and profile', async () => {
|
||||
test.setTimeout(180_000)
|
||||
await gatewayGroup(page, REMOTE_ID).getByRole('button', { name: `inbox · ${REMOTE_LABEL}` }).click()
|
||||
|
||||
// The workspace follows the agent: statusbar readout flips to Homelab…
|
||||
await expect(activeGatewayLabel(page)).toHaveAttribute('aria-label', `Registered gateways: ${REMOTE_LABEL}`, { timeout: 120_000 })
|
||||
|
||||
// …Homelab's group is now the active one, on the clicked profile…
|
||||
const homelab = gatewayGroup(page, REMOTE_ID)
|
||||
await expect(homelab).toHaveAttribute('data-active', 'true', { timeout: 30_000 })
|
||||
await expect(homelab.getByRole('button', { name: 'inbox', exact: true })).toHaveAttribute('aria-pressed', 'true', { timeout: 30_000 })
|
||||
|
||||
// …This device is at rest with qualified squares…
|
||||
const local = gatewayGroup(page, 'local')
|
||||
await expect(local).toHaveAttribute('data-active', 'false')
|
||||
await expect(local.getByRole('button', { name: 'research · This device' })).toBeVisible()
|
||||
|
||||
// …and nothing moved: the order is still This device, then Homelab.
|
||||
expect(await groupOrder(page)).toEqual([
|
||||
['local', false],
|
||||
[REMOTE_ID, true],
|
||||
])
|
||||
|
||||
await capture(page, '2-re-homed-on-homelab-inbox')
|
||||
})
|
||||
|
||||
test('an at-rest square offers gateway-scoped actions, never the legacy remote override', async () => {
|
||||
const square = gatewayGroup(page, 'local').getByRole('button', { name: 'research · This device' })
|
||||
await square.click({ button: 'right' })
|
||||
|
||||
const menu = page.getByRole('menu', { name: 'Actions' })
|
||||
await expect(menu).toBeVisible()
|
||||
await expect(menu.getByRole('menuitem', { name: 'Switch to research on This device' })).toBeVisible()
|
||||
await expect(menu.getByRole('menuitem', { name: 'Rename…' })).toBeVisible()
|
||||
await expect(menu.getByRole('menuitem', { name: 'Edit SOUL.md…' })).toBeVisible()
|
||||
await expect(menu.getByRole('menuitem', { name: 'Delete' })).toBeVisible()
|
||||
await expect(menu.getByRole('menuitem', { name: 'Connect to a remote host…' })).toHaveCount(0)
|
||||
|
||||
await capture(page, '3-at-rest-square-context-menu')
|
||||
await page.keyboard.press('Escape')
|
||||
await expect(menu).toBeHidden()
|
||||
})
|
||||
|
||||
test('editing SOUL.md on an at-rest square reads the owning gateway, not the foreground one', async () => {
|
||||
const square = gatewayGroup(page, 'local').getByRole('button', { name: 'research · This device' })
|
||||
await square.click({ button: 'right' })
|
||||
await page.getByRole('menu', { name: 'Actions' }).getByRole('menuitem', { name: 'Edit SOUL.md…' }).click()
|
||||
|
||||
const dialog = page.getByRole('dialog')
|
||||
await expect(dialog).toBeVisible()
|
||||
await expect(dialog.getByText('research · This device · SOUL.md')).toBeVisible()
|
||||
await page.keyboard.press('Escape')
|
||||
await expect(dialog).toBeHidden()
|
||||
})
|
||||
|
||||
test('switching back lands on the clicked profile of This device and keeps the order', async () => {
|
||||
test.setTimeout(180_000)
|
||||
await gatewayGroup(page, 'local').getByRole('button', { name: 'research · This device' }).click()
|
||||
|
||||
await expect(activeGatewayLabel(page)).toHaveAttribute('aria-label', 'Registered gateways: This device', { timeout: 120_000 })
|
||||
const local = gatewayGroup(page, 'local')
|
||||
await expect(local).toHaveAttribute('data-active', 'true', { timeout: 30_000 })
|
||||
await expect(local.getByRole('button', { name: 'research', exact: true })).toHaveAttribute('aria-pressed', 'true', { timeout: 30_000 })
|
||||
await expect(gatewayGroup(page, REMOTE_ID).getByRole('button', { name: `inbox · ${REMOTE_LABEL}` })).toBeVisible()
|
||||
|
||||
expect(await groupOrder(page)).toEqual([
|
||||
['local', true],
|
||||
[REMOTE_ID, false],
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,229 @@
|
||||
/**
|
||||
* E2E contract for the compositor-only GlyphSpinner.
|
||||
*
|
||||
* The spinner's whole reason for existing in this shape is a CSS animation:
|
||||
* every frame is in the DOM from mount and a `transform` keyframes animation
|
||||
* scrolls between them, so there is no JS timer and no per-tick DOM mutation
|
||||
* scheduling document-scale style recalculation.
|
||||
*
|
||||
* None of that is observable in jsdom — it has no animation engine, no
|
||||
* cascade resolution for `steps()`, and no `Element.getAnimations()`. The
|
||||
* jsdom suite (src/components/ui/glyph-spinner.test.tsx) therefore pins the
|
||||
* DATA and WIRING, and this spec pins the RENDERED BEHAVIOUR in a real
|
||||
* browser, which is the only place the stylesheet actually runs.
|
||||
*
|
||||
* This replaces three tests that asserted on the TEXT of the stylesheet.
|
||||
* Reading source in a test is banned outright (AGENTS.md) and those tests
|
||||
* proved the point: a var()-fallback edit that changed no rendered pixel
|
||||
* broke one of them, while none of them had ever executed the CSS.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, type Page, test } from '@playwright/test'
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
|
||||
/* Scope to a spinner that is actually RUNNING. Turns from earlier tests in
|
||||
* this file leave parked spinners mounted (kept-alive panes, swap overlays
|
||||
* hold them with data-paused='true'), and document.querySelector returns the
|
||||
* FIRST strip in the DOM — a stale parked one once two turns have run. */
|
||||
const STRIP = '.glyph-spinner:not([data-paused="true"]) .glyph-spinner__strip'
|
||||
|
||||
/** Prompt the mock server holds open so the spinner runs for the whole file. */
|
||||
const SPINNER_PROMPT = 'E2E_GLYPH_SPINNER_HOLD'
|
||||
|
||||
/**
|
||||
* Get a RUNNING frame strip into the DOM deterministically.
|
||||
*
|
||||
* A turn is sent so the app is genuinely busy (the mock server holds the
|
||||
* stream open), but which surface mounts a spinner mid-turn is app policy
|
||||
* that has changed before and will again — the transcript, status stack and
|
||||
* swap overlay all park/unmount theirs at different moments, which made this
|
||||
* spec racy. The contract under test is the STYLESHEET (steps() animation,
|
||||
* layer promotion, the data-paused and global pause gates), and that CSS is
|
||||
* driven entirely by the `data-paused` attribute — the same attribute the
|
||||
* parked assertions below already toggle. So: wait for any mounted spinner
|
||||
* (the ChatSwapOverlay keeps one mounted, parked, after boot), then unpark it
|
||||
* and assert against the running animation.
|
||||
*/
|
||||
async function mountSpinner(page: Page): Promise<void> {
|
||||
if (await page.locator(STRIP).count()) {
|
||||
return
|
||||
}
|
||||
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await composer.click()
|
||||
await composer.type(SPINNER_PROMPT, { delay: 10 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
await page.waitForSelector('.glyph-spinner__strip', { state: 'attached', timeout: 20_000 })
|
||||
await page.evaluate(() => {
|
||||
for (const el of document.querySelectorAll('.glyph-spinner[data-paused]')) {
|
||||
el.removeAttribute('data-paused')
|
||||
}
|
||||
})
|
||||
await page.waitForSelector(STRIP, { state: 'attached', timeout: 20_000 })
|
||||
}
|
||||
|
||||
test.describe('GlyphSpinner (compositor animation)', () => {
|
||||
let fixture: MockBackendFixture
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend({
|
||||
mockServer: { holdFirstStreamForPrompt: SPINNER_PROMPT },
|
||||
})
|
||||
await waitForAppReady(fixture)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
fixture?.mock.releaseHeldStream()
|
||||
await fixture?.cleanup()
|
||||
})
|
||||
|
||||
test('animates with a steps() transform keyframes animation, one step per frame', async () => {
|
||||
const { page } = fixture
|
||||
await mountSpinner(page)
|
||||
|
||||
const observed = await page.evaluate(strip => {
|
||||
const el = document.querySelector<HTMLElement>(strip)
|
||||
|
||||
if (!el) {
|
||||
throw new Error('no frame strip in the DOM')
|
||||
}
|
||||
|
||||
const style = getComputedStyle(el)
|
||||
const animations = el.getAnimations()
|
||||
|
||||
return {
|
||||
frameCount: el.querySelectorAll('.glyph-spinner__frame').length,
|
||||
timingFunction: style.animationTimingFunction,
|
||||
iterationCount: style.animationIterationCount,
|
||||
durationMs: animations[0]?.effect?.getTiming().duration ?? null,
|
||||
names: animations.map(a => (a as CSSAnimation).animationName),
|
||||
// A percentage translate makes the animation layout-dependent, which
|
||||
// Chromium refuses to composite. Read the engine's own keyframes: a
|
||||
// revert to translateY(-100%) shows up here, while the computed
|
||||
// `style.transform` always serializes to a matrix and can't tell.
|
||||
travel: ((animations[0]?.effect as KeyframeEffect | undefined)?.getKeyframes() ?? [])
|
||||
.map(k => String((k as Keyframe & { transform?: string }).transform ?? ''))
|
||||
.join(' | ')
|
||||
}
|
||||
}, STRIP)
|
||||
|
||||
// The strip carries every frame; `steps(N)` parks on each one in turn.
|
||||
expect(observed.frameCount).toBeGreaterThan(1)
|
||||
// Chromium has serialized jump-end as both `steps(N)` and `steps(N, end)`.
|
||||
expect(observed.timingFunction).toMatch(new RegExp(`^steps\\(${observed.frameCount}\\b`))
|
||||
expect(observed.iterationCount).toBe('infinite')
|
||||
expect(observed.names).toContain('glyph-spinner-advance')
|
||||
// One full cycle is frames x interval, so the duration must be a positive
|
||||
// multiple of the frame count — not the single-frame interval.
|
||||
expect(observed.durationMs).toBeGreaterThan(0)
|
||||
// Length-typed travel, never a percentage: `translateY(-100%)` would keep
|
||||
// the animation off the compositor. Chromium has serialized the resolved
|
||||
// keyframe both as the authored `calc(...)` and as an absolute `...px`
|
||||
// length depending on version — accept any length, reject percentages.
|
||||
expect(observed.travel).toMatch(/calc\(|px\)/)
|
||||
expect(observed.travel).not.toContain('%')
|
||||
})
|
||||
|
||||
test('is promoted to a layer while running, and neither animates nor holds a layer when parked', async () => {
|
||||
const { page } = fixture
|
||||
await mountSpinner(page)
|
||||
|
||||
const running = await page.evaluate(strip => {
|
||||
const el = document.querySelector<HTMLElement>(strip)!
|
||||
|
||||
return {
|
||||
playState: getComputedStyle(el).animationPlayState,
|
||||
willChange: getComputedStyle(el).willChange
|
||||
}
|
||||
}, STRIP)
|
||||
|
||||
expect(running.playState).toBe('running')
|
||||
// Scoped to active spinners — a permanently promoted layer per parked
|
||||
// spinner is pure memory at fan-out breadth.
|
||||
expect(running.willChange).toBe('transform')
|
||||
|
||||
// 1. The per-spinner gate: a kept-alive but inactive pane, or an explicit
|
||||
// `paused` prop (ChatSwapOverlay's fade-out).
|
||||
const parked = await page.evaluate(strip => {
|
||||
const el = document.querySelector<HTMLElement>(strip)!
|
||||
const viewport = el.closest<HTMLElement>('.glyph-spinner')!
|
||||
const previous = viewport.getAttribute('data-paused')
|
||||
|
||||
viewport.setAttribute('data-paused', 'true')
|
||||
|
||||
const state = {
|
||||
playState: getComputedStyle(el).animationPlayState,
|
||||
willChange: getComputedStyle(el).willChange
|
||||
}
|
||||
|
||||
if (previous === null) {
|
||||
viewport.removeAttribute('data-paused')
|
||||
} else {
|
||||
viewport.setAttribute('data-paused', previous)
|
||||
}
|
||||
|
||||
return state
|
||||
}, STRIP)
|
||||
|
||||
expect(parked.playState).toBe('paused')
|
||||
expect(parked.willChange).toBe('auto')
|
||||
|
||||
// 2. The global gate: window blur / minimize / document-hidden, which
|
||||
// main.tsx drives by arming this attribute on the root. The strip must
|
||||
// be named in that rule, or every spinner keeps animating behind an
|
||||
// inactive window — the CPU burn the original ticker's pause
|
||||
// controller existed to avoid.
|
||||
const globallyPaused = await page.evaluate(strip => {
|
||||
const root = document.documentElement
|
||||
const had = root.hasAttribute('data-renderer-animations-paused')
|
||||
|
||||
root.setAttribute('data-renderer-animations-paused', '')
|
||||
const playState = getComputedStyle(document.querySelector<HTMLElement>(strip)!).animationPlayState
|
||||
|
||||
if (!had) {
|
||||
root.removeAttribute('data-renderer-animations-paused')
|
||||
}
|
||||
|
||||
return playState
|
||||
}, STRIP)
|
||||
|
||||
expect(globallyPaused).toBe('paused')
|
||||
})
|
||||
|
||||
test('advances in discrete frames and creates no timer-driven DOM churn', async () => {
|
||||
const { page } = fixture
|
||||
await mountSpinner(page)
|
||||
|
||||
// Sample the resolved transform across one full cycle. A steps() animation
|
||||
// holds each value for a whole interval and jumps between them, so the
|
||||
// distinct values it visits must be bounded by the frame count — a linear
|
||||
// animation would produce a new value on every sample.
|
||||
const sampled = await page.evaluate(async strip => {
|
||||
const el = document.querySelector<HTMLElement>(strip)!
|
||||
const frames = el.querySelectorAll('.glyph-spinner__frame').length
|
||||
const duration = Number(el.getAnimations()[0]?.effect?.getTiming().duration ?? 0)
|
||||
const seen = new Set<string>()
|
||||
const textAtStart = el.textContent
|
||||
|
||||
const deadline = performance.now() + duration
|
||||
|
||||
while (performance.now() < deadline) {
|
||||
seen.add(getComputedStyle(el).transform)
|
||||
await new Promise(resolve => requestAnimationFrame(() => resolve(null)))
|
||||
}
|
||||
|
||||
return { distinct: seen.size, frames, textUnchanged: el.textContent === textAtStart }
|
||||
}, STRIP)
|
||||
|
||||
expect(sampled.distinct).toBeGreaterThan(1)
|
||||
expect(sampled.distinct).toBeLessThanOrEqual(sampled.frames + 1)
|
||||
// The old implementation rewrote textContent ~12x/second. Nothing may
|
||||
// mutate the DOM as this animates — that mutation is the whole incident.
|
||||
expect(sampled.textUnchanged).toBe(true)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,80 @@
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { expect, test } from './test'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
async function openBots(page: MockBackendFixture['page']): Promise<void> {
|
||||
const tab = page.getByRole('button', { name: 'Bots', exact: true }).or(page.getByRole('tab', { name: 'Bots', exact: true })).first()
|
||||
await tab.click()
|
||||
await expect(page.getByRole('button', { name: 'New bot or group chat' })).toBeVisible()
|
||||
}
|
||||
|
||||
async function createAgent(page: MockBackendFixture['page'], name: string, title: string): Promise<void> {
|
||||
await page.getByRole('button', { name: 'New bot or group chat' }).click()
|
||||
await page.getByRole('menuitem', { name: 'New Bot' }).click()
|
||||
|
||||
const dialog = page.getByRole('dialog', { name: 'New Bot' })
|
||||
await dialog.getByPlaceholder('inbox-triage').fill(name)
|
||||
await dialog.getByPlaceholder('Inbox Triage').fill(title)
|
||||
await dialog.getByRole('button', { name: 'Create Bot' }).click()
|
||||
await expect(dialog).toBeHidden({ timeout: 30_000 })
|
||||
await expect(page.getByRole('button', { name: new RegExp(`^${title}\\b`) }).first()).toBeVisible({ timeout: 30_000 })
|
||||
}
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('local bot replaces an open group main workspace', async () => {
|
||||
test.setTimeout(240_000)
|
||||
const page = fixture!.page
|
||||
|
||||
await openBots(page)
|
||||
await createAgent(page, 'programmer', 'Programmer')
|
||||
await createAgent(page, 'reviewer', 'Reviewer')
|
||||
|
||||
await page.getByRole('button', { name: 'New bot or group chat' }).click()
|
||||
await page.getByRole('menuitem', { name: 'New Group Chat' }).click()
|
||||
|
||||
const dialog = page.getByRole('dialog', { name: 'New Group Chat' })
|
||||
|
||||
for (const title of ['Programmer', 'Reviewer']) {
|
||||
await dialog.getByText(title, { exact: true }).locator('xpath=ancestor::label').getByRole('checkbox').click()
|
||||
}
|
||||
|
||||
await dialog.getByRole('textbox', { name: 'Group name' }).fill('Programmer, Reviewer')
|
||||
await dialog.getByRole('button', { name: 'Create Group (2)' }).click()
|
||||
|
||||
const groupTab = page.getByRole('tab', { name: /Programmer, Reviewer Close/ })
|
||||
const groupComposer = page.getByRole('textbox', { name: 'Message Programmer, Reviewer' }).filter({ visible: true })
|
||||
await expect(groupTab).toBeVisible({ timeout: 20_000 })
|
||||
await expect(groupTab).toHaveAttribute('aria-selected', 'true')
|
||||
await expect(groupComposer).toBeVisible()
|
||||
|
||||
const programmer = page.getByRole('button', { name: /^Programmer\b/ }).filter({ visible: true }).first()
|
||||
await programmer.click()
|
||||
|
||||
// The bot's canonical chat opens INTO the main workspace pane (post
|
||||
// design-system rework); as the lone pane in the zone it renders chromeless
|
||||
// — no "Bot Chat" tab exists until a second pane joins the strip. The
|
||||
// handoff is observed by the group surfaces leaving and the bot's chat
|
||||
// (here a fresh one: its empty-state splash asks for a first message)
|
||||
// taking the main workspace. The first open also spawns the bot's own
|
||||
// backend, so give the "Loading session" phase a real chance to clear.
|
||||
await expect(page.getByText('Say something to get started.').filter({ visible: true })).toBeVisible({
|
||||
timeout: 120_000
|
||||
})
|
||||
await expect(groupTab).toHaveCount(0)
|
||||
await expect(groupComposer).toHaveCount(0)
|
||||
// No "Waking up…" assertion: the mock backend can keep a bot's wake notice
|
||||
// around indefinitely (see bot-mode-row-click-mirrors-registry's settle()),
|
||||
// so its presence no longer distinguishes a stranded handoff. The splash
|
||||
// and composer above are the proof the bot's chat took the workspace.
|
||||
await expect(page.locator('[data-slot="composer-root"] [contenteditable="true"]').filter({ visible: true }).first()).toBeVisible()
|
||||
})
|
||||
@@ -0,0 +1,160 @@
|
||||
/**
|
||||
* E2E regression: desktop resume must hide agent-only transcript rows.
|
||||
*
|
||||
* Compaction handoffs are active user rows because the model needs them for
|
||||
* context continuity. They are not authored chat content, so the desktop
|
||||
* transcript must never display them after a real compressor-generated resume.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs'
|
||||
import * as path from 'node:path'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type MockBackendFixture,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig,
|
||||
} from './fixtures'
|
||||
import {
|
||||
MOCK_REPLY,
|
||||
startMockServer,
|
||||
VERIFICATION_STOP_TEXT,
|
||||
VERIFICATION_STOP_TRIGGER,
|
||||
} from './mock-server'
|
||||
import { RealSessionBuilder } from './real-session-builder'
|
||||
import { expect, test } from './test'
|
||||
|
||||
const SESSION_TITLE = 'E2E Hidden History Messages'
|
||||
const VISIBLE_USER_TEXT = 'E2E_VISIBLE_USER_HISTORY'
|
||||
const VISIBLE_POST_COMPACTION_TEXT = 'E2E_VISIBLE_POST_COMPACTION_HISTORY'
|
||||
const COMPACTION_TRIGGER_PADDING = ' force real context compression'.repeat(600)
|
||||
|
||||
async function setupSeededMockBackend(): Promise<MockBackendFixture> {
|
||||
const mock = await startMockServer()
|
||||
const sandbox = createSandbox('hidden-history')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
fs.appendFileSync(
|
||||
path.join(sandbox.hermesHome, 'config.yaml'),
|
||||
'\ncompression:\n threshold_tokens: 1\n',
|
||||
'utf8',
|
||||
)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
const builder = await RealSessionBuilder.start(sandbox.hermesHome)
|
||||
|
||||
try {
|
||||
await builder.createSession({
|
||||
title: SESSION_TITLE,
|
||||
turns: [
|
||||
`${VISIBLE_USER_TEXT}${COMPACTION_TRIGGER_PADDING}`,
|
||||
VISIBLE_POST_COMPACTION_TEXT,
|
||||
],
|
||||
})
|
||||
} finally {
|
||||
await builder.close()
|
||||
}
|
||||
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
|
||||
return {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
test('resume hides real context-compaction handoffs', async ({}, testInfo) => {
|
||||
const fixture = await setupSeededMockBackend()
|
||||
|
||||
try {
|
||||
const { page } = fixture
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
|
||||
const sessionRow = page
|
||||
.locator('[data-slot="sidebar"] button')
|
||||
.filter({ hasText: SESSION_TITLE })
|
||||
.first()
|
||||
|
||||
await sessionRow.click()
|
||||
|
||||
const transcript = page.locator('[data-slot="aui_thread-viewport"]')
|
||||
await expect(transcript).toContainText(VISIBLE_USER_TEXT)
|
||||
await expect(transcript).toContainText(VISIBLE_POST_COMPACTION_TEXT)
|
||||
await expect(transcript).toContainText(MOCK_REPLY)
|
||||
await expect(transcript).not.toContainText('[CONTEXT COMPACTION — REFERENCE ONLY]')
|
||||
await page.screenshot({ path: testInfo.outputPath('hidden-history-resume.png') })
|
||||
} finally {
|
||||
await fixture.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('live verify-on-stop continuations stay out of the transcript', async ({}, testInfo) => {
|
||||
const sandbox = createSandbox('live-verification-nudge')
|
||||
const projectRoot = path.join(sandbox.root, 'project')
|
||||
const changedFile = path.join(projectRoot, 'e2e-verification-target.py')
|
||||
fs.mkdirSync(projectRoot)
|
||||
fs.writeFileSync(
|
||||
path.join(projectRoot, 'pyproject.toml'),
|
||||
'[project]\nname = "e2e-verification-project"\nversion = "0.0.0"\n',
|
||||
'utf8',
|
||||
)
|
||||
|
||||
const mock = await startMockServer({ verificationWritePath: changedFile })
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
fs.appendFileSync(path.join(sandbox.hermesHome, 'config.yaml'), '\nagent:\n verify_on_stop: true\n', 'utf8')
|
||||
// Auto session titling (feat f726090d48) fires an auxiliary title_generation
|
||||
// LLM call whose user snippet CONTAINS the trigger keyword, so the mock's
|
||||
// isVerificationStopTrigger matches it and the title call steals a scripted
|
||||
// verify-on-stop turn (the transcript then ends on 'The code edit is
|
||||
// complete.' instead of the exhausted-verifier final). Disable the
|
||||
// model-backed title upgrade so script indices track real chat turns.
|
||||
fs.appendFileSync(
|
||||
path.join(sandbox.hermesHome, 'config.yaml'),
|
||||
'\nauxiliary:\n title_generation:\n enabled: false\n',
|
||||
'utf8',
|
||||
)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
|
||||
const fixture: MockBackendFixture = {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
|
||||
try {
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.click()
|
||||
await composer.type(VERIFICATION_STOP_TRIGGER)
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
const transcript = page.locator('[data-slot="aui_thread-viewport"]')
|
||||
await expect(transcript).toContainText(VERIFICATION_STOP_TEXT, { timeout: 60_000 })
|
||||
await expect.poll(
|
||||
() => mock.receivedPrompts.some(prompt => prompt.includes('[System: You edited code in this turn')),
|
||||
{ timeout: 30_000 },
|
||||
).toBe(true)
|
||||
expect(fs.existsSync(changedFile), 'The scripted write_file call should edit only the sandbox project').toBe(true)
|
||||
await expect(transcript).not.toContainText('[System: You edited code in this turn')
|
||||
await page.screenshot({ path: testInfo.outputPath('live-verification-nudge.png') })
|
||||
} finally {
|
||||
await fixture.cleanup()
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,199 @@
|
||||
/**
|
||||
* Regression coverage for an attached image in a durable session. The gateway
|
||||
* persists the turn, the builder exits, and desktop renders it from SessionDB
|
||||
* for the first time — the "quit and relaunch" case, where the transcript used
|
||||
* to come back as vision-enrichment prose instead of a thumbnail.
|
||||
*
|
||||
* The fixture pins `image_input_mode: native` because that is the majority
|
||||
* routing path (any vision-capable model) and the one where a text-only
|
||||
* persist override is silently dropped. The image also sits behind directory
|
||||
* and file names containing spaces, mirroring the macOS composer's
|
||||
* `~/Library/Application Support/...` staging path.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs'
|
||||
import * as path from 'node:path'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type Sandbox,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig,
|
||||
} from './fixtures'
|
||||
import { type MockServer, startMockServer } from './mock-server'
|
||||
import { RealSessionBuilder } from './real-session-builder'
|
||||
import { type ElectronApplication, expect, type Page, test } from './test'
|
||||
|
||||
// The builder-provided title now labels the sidebar row directly (seeded
|
||||
// sessions no longer fall back to the first-user-message preview).
|
||||
const SESSION_TITLE = 'E2E attached image session'
|
||||
const CAPTION = 'E2E attached image must survive a relaunch'
|
||||
const IMAGE_DIR = 'Application Support/e2e shots'
|
||||
const IMAGE_NAME = 'e2e capture.png'
|
||||
const NATIVE_IMAGE_CONFIG = 'agent:\n image_input_mode: native'
|
||||
|
||||
/** A 160x100 framed magenta block — small, but visible in the screenshots. */
|
||||
const PNG_BASE64 =
|
||||
'iVBORw0KGgoAAAANSUhEUgAAAKAAAABkCAIAAACO1KzYAAAA30lEQVR42u3dwQ2AIBAAQTAWAx1iBXYI7diCuWhEMvP2dZsj+CL30hLr2oxAYARGYARGYARGYIERGIH53n7nozpOk5rQqIcNdkQjMAIjMNPeomP3N54V+5exwY5oBEZgBEZgBEZggREYgREYgREYgQVGYARGYARGYARGYIERGIERGIERGIEFRmAERmAERmAEFhiBERiBERiBERiBBUZgBEZgBEZgBBYYgREYgREYgRFYYCMQGIERmPSj94Njb9ligxEYgRFYYJaQe2mmYIMRGIERGIERGIEFRmAERmDedAFtjAtAGWDnoAAAAABJRU5ErkJggg=='
|
||||
|
||||
interface SeededFixture {
|
||||
app: ElectronApplication
|
||||
mock: MockServer
|
||||
page: Page
|
||||
sandbox: Sandbox
|
||||
cleanup: () => Promise<void>
|
||||
}
|
||||
|
||||
function writeImage(sandbox: Sandbox): string {
|
||||
const dir = path.join(sandbox.root, IMAGE_DIR)
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
|
||||
const imagePath = path.join(dir, IMAGE_NAME)
|
||||
fs.writeFileSync(imagePath, Buffer.from(PNG_BASE64, 'base64'))
|
||||
|
||||
return imagePath
|
||||
}
|
||||
|
||||
async function setupSeededDesktop(): Promise<SeededFixture> {
|
||||
const mock = await startMockServer()
|
||||
const sandbox = createSandbox('image-attachment')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url, undefined, NATIVE_IMAGE_CONFIG)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
|
||||
const builder = await RealSessionBuilder.start(sandbox.hermesHome)
|
||||
|
||||
try {
|
||||
await builder.createSession({
|
||||
title: SESSION_TITLE,
|
||||
turns: [{ images: [writeImage(sandbox)], text: CAPTION }],
|
||||
})
|
||||
} finally {
|
||||
await builder.close()
|
||||
}
|
||||
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
|
||||
return {
|
||||
app,
|
||||
mock,
|
||||
page,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function sessionRow(page: Page) {
|
||||
return page.locator('[data-slot="sidebar"] button').filter({ hasText: SESSION_TITLE }).first()
|
||||
}
|
||||
|
||||
// Inactive tabs stay mounted under a data-pane-hidden ancestor. Match the
|
||||
// renderer's keep-alive visibility policy instead of relying on DOM order.
|
||||
const SURFACE = '[data-composer-target]:not([data-pane-hidden] [data-composer-target])'
|
||||
|
||||
function activeViewportText(surfaceSelector: string): string {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
|
||||
return surfaces[surfaces.length - 1]?.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? ''
|
||||
}
|
||||
|
||||
async function openSeededSession(page: Page): Promise<void> {
|
||||
const row = sessionRow(page)
|
||||
await row.waitFor({ state: 'visible', timeout: 60_000 })
|
||||
await row.click()
|
||||
await page.waitForFunction(
|
||||
([expected, surfaceSelector]: [string, string]) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const text = surfaces[surfaces.length - 1]?.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? ''
|
||||
|
||||
return text.includes(expected)
|
||||
},
|
||||
[CAPTION, SURFACE] as [string, string],
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* The sidebar "+" opens a NEW TAB beside the current chat instead of replacing
|
||||
* it, so the seeded session stays mounted in its own surface. Assert the new
|
||||
* surface is empty rather than waiting for the old caption to leave the page.
|
||||
*/
|
||||
async function openNewSession(page: Page): Promise<void> {
|
||||
await page.locator('[data-slot="sidebar"] button[aria-label="New session"]').first().click()
|
||||
await page.waitForFunction(
|
||||
([expected, surfaceSelector]: [string, string]) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const text = surfaces[surfaces.length - 1]?.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? ''
|
||||
|
||||
return surfaces.length > 0 && !text.includes(expected)
|
||||
},
|
||||
[CAPTION, SURFACE] as [string, string],
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
}
|
||||
|
||||
async function transcriptText(page: Page): Promise<string> {
|
||||
return page.evaluate(activeViewportText, SURFACE)
|
||||
}
|
||||
|
||||
async function assertRendersThumbnail(page: Page, label: string): Promise<void> {
|
||||
const thumbnail = page.locator('[data-slot="aui_directive-image"] img')
|
||||
await expect(thumbnail, `${label}: the attachment should render as an image`).toHaveCount(1)
|
||||
await expect(thumbnail, `${label}: the thumbnail should resolve off disk`).toHaveAttribute('src', /^data:image\//)
|
||||
|
||||
const text = await transcriptText(page)
|
||||
expect(text, `${label}: the caption should survive alongside the image`).toContain(CAPTION)
|
||||
// A broken ref falls back to a chip whose label leaks the path, and a
|
||||
// flattened multimodal turn leaves the agent's placeholder behind.
|
||||
expect(text, `${label}: the raw image path should not leak into the transcript`).not.toContain(IMAGE_NAME)
|
||||
expect(text, `${label}: the image directive should not render literally`).not.toContain('@image:')
|
||||
expect(text, `${label}: the flattening placeholder should not render`).not.toContain('[screenshot]')
|
||||
}
|
||||
|
||||
test.describe('attached image resume', () => {
|
||||
let fixture: SeededFixture | null = null
|
||||
|
||||
test.afterEach(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('renders a persisted attachment as a thumbnail on first open and after a cold reload', async ({}, testInfo) => {
|
||||
// Seeding through the real gateway plus two full app boots does not fit the
|
||||
// default per-test budget on a cold runner.
|
||||
test.slow()
|
||||
|
||||
fixture = await setupSeededDesktop()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
|
||||
// The sidebar labels a seeded session by its title. Whatever the label
|
||||
// source, an attachment directive must never leak into it as a file path.
|
||||
const row = sessionRow(fixture.page)
|
||||
await row.waitFor({ state: 'visible', timeout: 60_000 })
|
||||
|
||||
const label = (await row.textContent())?.trim() ?? ''
|
||||
expect(label.startsWith(SESSION_TITLE), `sidebar label should open with the title: ${label}`).toBe(true)
|
||||
expect(label, `sidebar label should not leak the image path: ${label}`).not.toContain(IMAGE_NAME)
|
||||
expect(label, `sidebar label should not render the directive: ${label}`).not.toContain('@image:')
|
||||
|
||||
await openSeededSession(fixture.page)
|
||||
await assertRendersThumbnail(fixture.page, 'first open')
|
||||
await fixture.page.screenshot({ path: testInfo.outputPath('attachment-first-open.png') })
|
||||
|
||||
// A reload drops every cached attachment ref, so the transcript has to come
|
||||
// back from the persisted turn alone.
|
||||
await fixture.page.reload()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
await openNewSession(fixture.page)
|
||||
|
||||
await openSeededSession(fixture.page)
|
||||
await assertRendersThumbnail(fixture.page, 'cold reload')
|
||||
await fixture.page.screenshot({ path: testInfo.outputPath('attachment-cold-reload.png') })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,275 @@
|
||||
/**
|
||||
* E2E test for the interim-assistant-message preservation fix (#65919).
|
||||
*
|
||||
* Reproduces the bug across all three layers (agent core → tui_gateway →
|
||||
* desktop renderer): when the agent emits assistant text alongside a tool
|
||||
* call, then completes the turn with a *different* final answer, the
|
||||
* interim text must survive in the transcript — not be wiped when
|
||||
* message.complete replaces the streaming bubble.
|
||||
*
|
||||
* The mock server walks through a multi-turn script when it sees the
|
||||
* trigger keyword:
|
||||
*
|
||||
* Turn 1: "Let me start by planning the approach." + todo tool_call
|
||||
* Turn 2: "Now checking the details before answering." + todo tool_call
|
||||
* Turn 3: (no text) + todo tool_call → NO interim (no visible text)
|
||||
* Turn 4: "Found something interesting worth noting." + todo tool_call
|
||||
* Turn 5: "All done! Here is the complete summary..." (final, stop)
|
||||
*
|
||||
* Two describe blocks exercise the config flag both ways:
|
||||
*
|
||||
* display.interim_assistant_messages: true (default)
|
||||
* → ALL interim texts AND the final text must be visible in the
|
||||
* settled transcript.
|
||||
*
|
||||
* display.interim_assistant_messages: false
|
||||
* → no message.interim events are emitted, so no sealed interim bubbles
|
||||
* are created while streaming. Since the post-turn stored-history
|
||||
* reconcile (sessions.changed → reconcileActiveTranscript, commit
|
||||
* 1a2b0ca8cb) converges the visible transcript to the persisted
|
||||
* transcript — which has ALWAYS contained the mid-turn commentary as
|
||||
* real assistant rows (that is what a resume shows, flag or no flag) —
|
||||
* the settled DOM shows the whole turn as ONE assistant message
|
||||
* containing commentary + final. The flag governs live sealing only.
|
||||
* The test pins that converged single-message shape: every text
|
||||
* appears exactly once, inside a single assistant message root.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, type Page, test } from '@playwright/test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
import { INTERIM_TEXTS, restartMockServer } from './mock-server'
|
||||
|
||||
// ─── Helpers ──────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Auto session titling (feat f726090d48, 2026-08-08) issues an auxiliary
|
||||
* `title_generation` LLM call against the SAME provider as the chat turn.
|
||||
* The mock server counts every completion request as a script turn, so the
|
||||
* title call races the chat turn and steals a scripted interim turn (the
|
||||
* stolen turn's text then never streams to the transcript). Disable the
|
||||
* model-backed title upgrade — the instant derived title needs no LLM call —
|
||||
* so the mock's script indices line up with real chat turns again.
|
||||
*/
|
||||
const DISABLE_AUTO_TITLE = 'auxiliary:\n title_generation:\n enabled: false'
|
||||
|
||||
/** Unique trigger keyword the mock server detects to switch to the script. */
|
||||
const TRIGGER = 'E2E_INTERIM_TRIGGER'
|
||||
|
||||
/**
|
||||
* Send a message and wait for BOTH the user's message and the agent's
|
||||
* final response to appear in the transcript. Returns when the final text
|
||||
* is visible, which means message.complete has fired and the transcript
|
||||
* has settled.
|
||||
*/
|
||||
async function sendInterimMessage(page: Page): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await composer.click()
|
||||
await composer.type(TRIGGER, { delay: 20 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
// Wait for the user's trigger message to appear.
|
||||
await page.waitForFunction(
|
||||
() => (document.body.textContent ?? '').includes('E2E_INTERIM_TRIGGER'),
|
||||
undefined,
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
|
||||
// Wait for the agent's FINAL response (last turn). This means
|
||||
// message.complete has fired and the transcript is settled.
|
||||
await page.waitForFunction(
|
||||
(finalText) => (document.body.textContent ?? '').includes(finalText),
|
||||
INTERIM_TEXTS.finalText,
|
||||
{ timeout: 90_000 },
|
||||
)
|
||||
|
||||
// Give the renderer a moment to settle any final state updates
|
||||
// (hydration, stored-history reconcile, session refresh) before asserting.
|
||||
await page.waitForTimeout(2000)
|
||||
}
|
||||
|
||||
/**
|
||||
* Count how many times `text` appears as distinct text in the chat transcript
|
||||
* (excluding the session sidebar, whose session-preview label shows the
|
||||
* first streamed text as a title).
|
||||
*
|
||||
* The desktop app renders the transcript inside a
|
||||
* `[data-slot="aui_thread-viewport"]` container (from @assistant-ui/react).
|
||||
* The session sidebar's preview labels live outside that container, so
|
||||
* scoping the DOM walk to the viewport cleanly excludes them.
|
||||
*/
|
||||
async function countTranscriptMessagesContaining(page: Page, text: string): Promise<number> {
|
||||
return page.evaluate(
|
||||
(search) => {
|
||||
const viewport = document.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
|
||||
if (!viewport) {
|
||||
return 0
|
||||
}
|
||||
|
||||
let count = 0
|
||||
|
||||
const walker = document.createTreeWalker(
|
||||
viewport,
|
||||
NodeFilter.SHOW_ELEMENT,
|
||||
{
|
||||
acceptNode: (node) => {
|
||||
const el = node as HTMLElement
|
||||
const directText = el.textContent ?? ''
|
||||
|
||||
if (!directText.includes(search)) {
|
||||
return NodeFilter.FILTER_SKIP
|
||||
}
|
||||
|
||||
// Only count leaf-ish elements to avoid double-counting.
|
||||
const hasChildWithText = Array.from(el.children).some(
|
||||
(child) => (child.textContent ?? '').includes(search),
|
||||
)
|
||||
|
||||
if (hasChildWithText) {
|
||||
return NodeFilter.FILTER_SKIP
|
||||
}
|
||||
|
||||
return NodeFilter.FILTER_ACCEPT
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
while (walker.nextNode()) {
|
||||
count++
|
||||
}
|
||||
|
||||
return count
|
||||
},
|
||||
text,
|
||||
)
|
||||
}
|
||||
|
||||
/** Count assistant message roots in the settled transcript. */
|
||||
async function countAssistantMessageRoots(page: Page): Promise<number> {
|
||||
return page.evaluate(() => {
|
||||
const viewport = document.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
|
||||
return viewport
|
||||
? viewport.querySelectorAll('[data-slot="aui_assistant-message-root"]').length
|
||||
: 0
|
||||
})
|
||||
}
|
||||
|
||||
// ─── Flag ON: interim_assistant_messages = true (default) ─────────────
|
||||
|
||||
test.describe('interim assistant messages — flag ON (default)', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({ extraConfig: DISABLE_AUTO_TITLE })
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
})
|
||||
|
||||
test('all interim texts survive alongside the final response', async () => {
|
||||
const page = fixture.page
|
||||
await sendInterimMessage(page)
|
||||
|
||||
// Every interim text (turns with visible text + tool calls) must be
|
||||
// present in the settled transcript — NOT wiped by message.complete.
|
||||
// (Live, each seals as its own bubble; the post-turn stored-history
|
||||
// reconcile then converges the turn into one assistant message that
|
||||
// still carries all of them.)
|
||||
for (const interimText of INTERIM_TEXTS.interims) {
|
||||
await expect
|
||||
.poll(
|
||||
() => countTranscriptMessagesContaining(page, interimText),
|
||||
{ timeout: 15_000, message: `interim text "${interimText}" should be visible` },
|
||||
)
|
||||
.toBeGreaterThanOrEqual(1)
|
||||
}
|
||||
|
||||
// The final text must also be visible.
|
||||
await expect
|
||||
.poll(
|
||||
() => countTranscriptMessagesContaining(page, INTERIM_TEXTS.finalText),
|
||||
{ timeout: 15_000, message: 'final text should be visible' },
|
||||
)
|
||||
.toBeGreaterThanOrEqual(1)
|
||||
|
||||
// No duplicates: the reconcile must CONVERGE (replace the sealed live
|
||||
// bubbles), never render a stored copy alongside a live one.
|
||||
for (const text of [...INTERIM_TEXTS.interims, INTERIM_TEXTS.finalText]) {
|
||||
const count = await countTranscriptMessagesContaining(page, text)
|
||||
expect(count, `"${text}" must not be duplicated after reconcile`).toBe(1)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// ─── Flag OFF: interim_assistant_messages = false ────────────────────
|
||||
|
||||
test.describe('interim assistant messages — flag OFF', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({
|
||||
extraDisplayConfig: ' interim_assistant_messages: false',
|
||||
extraConfig: DISABLE_AUTO_TITLE,
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
})
|
||||
|
||||
test('settled transcript converges to stored history as a single turn message', async () => {
|
||||
const page = fixture.page
|
||||
await sendInterimMessage(page)
|
||||
|
||||
// The final text must be visible.
|
||||
await expect
|
||||
.poll(
|
||||
() => countTranscriptMessagesContaining(page, INTERIM_TEXTS.finalText),
|
||||
{ timeout: 15_000, message: 'final text should be visible' },
|
||||
)
|
||||
.toBeGreaterThanOrEqual(1)
|
||||
|
||||
// With the flag off, the tui_gateway never installs
|
||||
// interim_assistant_callback, so no message.interim events fire and no
|
||||
// sealed interim bubbles are created while streaming. After
|
||||
// message.complete, the stored-history reconcile (sessions.changed →
|
||||
// reconcileActiveTranscript) converges the view to the persisted
|
||||
// transcript, which contains the mid-turn commentary as real assistant
|
||||
// rows — exactly what a resume of this session would show. Pin that
|
||||
// converged shape: ONE assistant message root for the whole turn…
|
||||
await expect
|
||||
.poll(
|
||||
() => countAssistantMessageRoots(page),
|
||||
{ timeout: 15_000, message: 'the settled turn should render as one assistant message' },
|
||||
)
|
||||
.toBe(1)
|
||||
|
||||
// …containing every commentary text and the final text exactly once.
|
||||
for (const text of [...INTERIM_TEXTS.interims, INTERIM_TEXTS.finalText]) {
|
||||
await expect
|
||||
.poll(
|
||||
() => countTranscriptMessagesContaining(page, text),
|
||||
{ timeout: 15_000, message: `"${text}" should appear exactly once in the converged turn` },
|
||||
)
|
||||
.toBe(1)
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,255 @@
|
||||
import * as path from 'node:path'
|
||||
|
||||
import { type TestInfo } from '@playwright/test'
|
||||
|
||||
import { expect, test, type ElectronApplication, type Page } from './test'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type Sandbox,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig,
|
||||
} from './fixtures'
|
||||
import { MOCK_REPLY, startMockServer, type MockServer, type MockServerOptions } from './mock-server'
|
||||
import { RealSessionBuilder } from './real-session-builder'
|
||||
|
||||
const DESKTOP_ROOT = path.resolve(import.meta.dirname, '..')
|
||||
const SESSION_TITLE = 'E2E large persisted session'
|
||||
const EXPECTED_TEXT = 'E2E persisted user message 52'
|
||||
// The oldest seeded turn (HISTORY_TURNS[0]). The transcript first paints only
|
||||
// the newest turns (FIRST_PAINT_BUDGET) and backfills the rest in a rAF; a
|
||||
// baseline count taken before that backfill sees a clipped transcript and
|
||||
// falsely reports duplicates once the full list mounts. Waiting for this
|
||||
// oldest row means the baseline reflects the fully-mounted transcript.
|
||||
const OLDEST_SEEDED_TEXT = 'E2E persisted user message 0: audit the compatibility matrix'
|
||||
const BACKGROUND_PROMPT = 'E2E background inference must remain attached across resume'
|
||||
const HISTORY_TURNS = Array.from(
|
||||
{ length: 27 },
|
||||
(_, index) => `E2E persisted user message ${index * 2}: audit the compatibility matrix`,
|
||||
)
|
||||
|
||||
interface SeededFixture {
|
||||
app: ElectronApplication
|
||||
mock: MockServer
|
||||
mockUrl: string
|
||||
page: Page
|
||||
sandbox: Sandbox
|
||||
cleanup: () => Promise<void>
|
||||
}
|
||||
|
||||
interface PaintState {
|
||||
bursts: number
|
||||
timeline: Array<{ mutations: number; time: number }>
|
||||
}
|
||||
|
||||
async function setupSeededDesktop(mockServer?: MockServerOptions): Promise<SeededFixture> {
|
||||
const mock = await startMockServer(mockServer)
|
||||
const sandbox = createSandbox('large-session')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
|
||||
const builder = await RealSessionBuilder.start(sandbox.hermesHome)
|
||||
try {
|
||||
await builder.createSession({ title: SESSION_TITLE, turns: HISTORY_TURNS })
|
||||
} finally {
|
||||
await builder.close()
|
||||
}
|
||||
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
|
||||
return {
|
||||
app,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
page,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function sessionRow(page: Page) {
|
||||
return page.locator('[data-slot="sidebar"] button').filter({ hasText: SESSION_TITLE }).first()
|
||||
}
|
||||
|
||||
async function openSeededSession(page: Page): Promise<void> {
|
||||
const row = sessionRow(page)
|
||||
await row.waitFor({ state: 'visible', timeout: 60_000 })
|
||||
await row.click()
|
||||
await page.waitForFunction(
|
||||
expected => (document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
EXPECTED_TEXT,
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
}
|
||||
|
||||
async function openNewSession(page: Page): Promise<void> {
|
||||
const button = page.locator('[data-slot="sidebar"] button').filter({ hasText: 'New session' }).first()
|
||||
await button.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await button.click()
|
||||
await page.waitForFunction(
|
||||
expected => !(document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
EXPECTED_TEXT,
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
}
|
||||
|
||||
async function submitPrompt(page: Page, prompt: string): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.type(prompt, { delay: 2 })
|
||||
await page.keyboard.press('Enter')
|
||||
await page.waitForFunction(
|
||||
expected => (document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
prompt,
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
}
|
||||
|
||||
async function startPaintObserver(page: Page): Promise<void> {
|
||||
await page.evaluate(() => {
|
||||
const viewport = document.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
const state = { bursts: 0, timeline: [] as Array<{ mutations: number; time: number }> }
|
||||
;(window as Window & { __largeSessionPaints?: typeof state }).__largeSessionPaints = state
|
||||
if (!viewport) return
|
||||
|
||||
let additions = 0
|
||||
let flushTimer: ReturnType<typeof setTimeout> | undefined
|
||||
new MutationObserver(records => {
|
||||
additions += records.reduce(
|
||||
(count, record) => count + (record.type === 'childList' && record.addedNodes.length > 0 ? 1 : 0),
|
||||
0,
|
||||
)
|
||||
if (additions === 0) return
|
||||
if (flushTimer) clearTimeout(flushTimer)
|
||||
flushTimer = setTimeout(() => {
|
||||
state.bursts += 1
|
||||
state.timeline.push({ mutations: additions, time: Date.now() })
|
||||
additions = 0
|
||||
}, 30)
|
||||
}).observe(viewport, { childList: true, subtree: true })
|
||||
})
|
||||
}
|
||||
|
||||
async function paintState(page: Page): Promise<PaintState> {
|
||||
const state = await page.evaluate(() => (window as Window & { __largeSessionPaints?: PaintState }).__largeSessionPaints)
|
||||
expect(state, 'paint observer should attach to the thread viewport').toBeDefined()
|
||||
return state!
|
||||
}
|
||||
|
||||
async function textNodeOccurrences(page: Page, expected: string): Promise<number> {
|
||||
return page.evaluate(text => {
|
||||
const viewport = document.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
if (!viewport) return 0
|
||||
|
||||
const walker = document.createTreeWalker(viewport, NodeFilter.SHOW_TEXT)
|
||||
let count = 0
|
||||
while (walker.nextNode()) {
|
||||
if (walker.currentNode.textContent?.includes(text)) {
|
||||
count += 1
|
||||
}
|
||||
}
|
||||
return count
|
||||
}, expected)
|
||||
}
|
||||
|
||||
async function reloadIntoColdRenderer(fixture: SeededFixture): Promise<void> {
|
||||
await fixture.page.reload()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
await openNewSession(fixture.page)
|
||||
}
|
||||
|
||||
async function assertUnchangedResume(page: Page, testInfo: TestInfo): Promise<void> {
|
||||
await openSeededSession(page)
|
||||
await page.waitForTimeout(1_000)
|
||||
await page.screenshot({ path: testInfo.outputPath('unchanged-session-resume.png'), fullPage: false })
|
||||
|
||||
const paints = await paintState(page)
|
||||
expect(await textNodeOccurrences(page, EXPECTED_TEXT), 'the resumed user message should appear once').toBe(1)
|
||||
// A warm session first restores its retained view, then reconciles it with the
|
||||
// authoritative transcript. That is bounded at two builds; a third paint was
|
||||
// the old eager-prefetch + runtime-rebuild regression. A cold restore has one.
|
||||
expect(paints.bursts, `unexpected transcript paint count: ${JSON.stringify(paints.timeline)}`).toBeLessThanOrEqual(2)
|
||||
}
|
||||
|
||||
test.describe('large session resume', () => {
|
||||
let fixture: SeededFixture | null = null
|
||||
|
||||
test.afterEach(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('cold resume of an unchanged session has one user row and bounded transcript paints', async ({}, testInfo) => {
|
||||
fixture = await setupSeededDesktop()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
|
||||
await startPaintObserver(fixture.page)
|
||||
await assertUnchangedResume(fixture.page, testInfo)
|
||||
})
|
||||
|
||||
test('fast resume of an unchanged session has one user row and bounded transcript paints', async ({}, testInfo) => {
|
||||
// Known RED: a rapid warm resume rebuilds the transcript three times
|
||||
// (28 → 53 → 53 DOM additions) instead of the two-paint budget. Keep the
|
||||
// regression visible without making unrelated desktop work fail CI.
|
||||
test.fixme(true, 'Fast warm resume has an unresolved third transcript rebuild')
|
||||
|
||||
fixture = await setupSeededDesktop()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
|
||||
await openSeededSession(fixture.page)
|
||||
await openNewSession(fixture.page)
|
||||
await startPaintObserver(fixture.page)
|
||||
await assertUnchangedResume(fixture.page, testInfo)
|
||||
})
|
||||
|
||||
for (const resumeKind of ['fast', 'cold'] as const) {
|
||||
test(`${resumeKind} resume keeps background inference attached without duplicate messages`, async ({}, testInfo) => {
|
||||
fixture = await setupSeededDesktop({ holdFirstStreamForPrompt: BACKGROUND_PROMPT })
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
|
||||
await openSeededSession(fixture.page)
|
||||
// The transcript first paints only the newest turns (FIRST_PAINT_BUDGET)
|
||||
// and backfills older turns in a rAF. Wait for the oldest seeded row to
|
||||
// mount before taking the baseline so it reflects the full transcript —
|
||||
// otherwise a clipped baseline makes the backfilled rows look like
|
||||
// duplicates of the completed reply.
|
||||
await fixture.page.waitForFunction(
|
||||
expected => (document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
OLDEST_SEEDED_TEXT,
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
const initialMockReplyCount = await textNodeOccurrences(fixture.page, MOCK_REPLY)
|
||||
await submitPrompt(fixture.page, BACKGROUND_PROMPT)
|
||||
await fixture.mock.waitForHeldStream()
|
||||
await openNewSession(fixture.page)
|
||||
|
||||
if (resumeKind === 'cold') {
|
||||
await reloadIntoColdRenderer(fixture)
|
||||
}
|
||||
|
||||
await openSeededSession(fixture.page)
|
||||
fixture.mock.releaseHeldStream()
|
||||
await fixture.page.waitForFunction(
|
||||
expected => (document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
MOCK_REPLY,
|
||||
{ timeout: 60_000 },
|
||||
)
|
||||
await fixture.page.waitForTimeout(300)
|
||||
await fixture.page.screenshot({ path: testInfo.outputPath(`${resumeKind}-background-inference-resume.png`), fullPage: false })
|
||||
|
||||
expect(await textNodeOccurrences(fixture.page, BACKGROUND_PROMPT), 'the running user prompt should appear once').toBe(1)
|
||||
expect(
|
||||
await textNodeOccurrences(fixture.page, MOCK_REPLY),
|
||||
'the completed assistant reply should add exactly one transcript row',
|
||||
).toBe(initialMockReplyCount + 1)
|
||||
})
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,177 @@
|
||||
import { expect, test } from './test'
|
||||
|
||||
import {
|
||||
PACKAGED_BINARY_PATH,
|
||||
type PackagedAppFixture,
|
||||
packagedBinaryExists,
|
||||
setupPackagedApp,
|
||||
} from './fixtures'
|
||||
import { expectVisualSnapshot } from './visual-snapshot'
|
||||
|
||||
/**
|
||||
* E2E smoke tests for the packaged Hermes desktop app.
|
||||
*
|
||||
* Launches the real packaged Electron binary (produced by `npm run pack` →
|
||||
* `electron-builder --dir`) with BOOT_FAKE=1 and full sandbox isolation
|
||||
* (credential stripping, isolated HERMES_HOME + userData, unique app name).
|
||||
*
|
||||
* Skips if the packaged binary doesn't exist — run `npm run pack` first.
|
||||
*/
|
||||
|
||||
let fixture: PackagedAppFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
test.skip(
|
||||
!packagedBinaryExists(),
|
||||
`Built app binary not found: ${PACKAGED_BINARY_PATH}. Run 'npm run pack' first.`,
|
||||
)
|
||||
|
||||
fixture = await setupPackagedApp()
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('window opens with the Hermes title', async () => {
|
||||
const title = await fixture!.page.title()
|
||||
expect(title).toContain('Hermes')
|
||||
})
|
||||
|
||||
test('renderer loads and shows DOM content', async () => {
|
||||
const page = fixture!.page
|
||||
await page.waitForSelector('#root', { state: 'attached', timeout: 30_000 })
|
||||
const childCount = await page.locator('#root > *').count()
|
||||
expect(childCount).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
test('boots to the app UI, not the QueryClient error boundary (#95560)', async () => {
|
||||
const page = fixture!.page
|
||||
await page.waitForSelector('#root', { state: 'attached', timeout: 30_000 })
|
||||
|
||||
// Wait until the root has real content (boot overlay fades, app paints) —
|
||||
// the error boundary also paints, so assert on its absence explicitly.
|
||||
await page.waitForFunction(
|
||||
() => (document.getElementById('root')?.textContent ?? '').trim().length > 0,
|
||||
undefined,
|
||||
{ timeout: 60_000 },
|
||||
)
|
||||
|
||||
const text = await page.locator('#root').textContent()
|
||||
// The #95560 crash: a duplicate @tanstack/react-query runtime made the
|
||||
// QueryClientProvider's context invisible to useQuery, so the app hit the
|
||||
// error boundary at launch. Neither the boundary headline nor the throw
|
||||
// message may appear on a healthy boot.
|
||||
expect(text).not.toContain('No QueryClient set')
|
||||
expect(text).not.toContain('Something broke in the interface')
|
||||
})
|
||||
|
||||
test('HUD composer remains fully inside the transparent window', async () => {
|
||||
const hudPagePromise = fixture!.app.waitForEvent('window')
|
||||
|
||||
await fixture!.page.evaluate(() =>
|
||||
(window as typeof window & {
|
||||
hermesDesktop?: { hud?: { open: (options: { sessionId: null }) => Promise<void> } }
|
||||
}).hermesDesktop?.hud?.open({ sessionId: null })
|
||||
)
|
||||
|
||||
const hudPage = await hudPagePromise
|
||||
await hudPage.waitForSelector('[data-slot="composer-rich-input"]', { state: 'visible' })
|
||||
|
||||
const geometry = await hudPage.evaluate(() => {
|
||||
const dock = document.querySelector<HTMLElement>('[data-slot="composer-dock"]')
|
||||
const input = document.querySelector<HTMLElement>('[data-slot="composer-rich-input"]')
|
||||
|
||||
if (!dock || !input) {
|
||||
throw new Error('HUD composer did not render')
|
||||
}
|
||||
|
||||
const dockRect = dock.getBoundingClientRect()
|
||||
const inputRect = input.getBoundingClientRect()
|
||||
|
||||
return {
|
||||
viewportWidth: window.innerWidth,
|
||||
viewportHeight: window.innerHeight,
|
||||
dockLeft: dockRect.left,
|
||||
dockRight: dockRect.right,
|
||||
dockTop: dockRect.top,
|
||||
dockBottom: dockRect.bottom,
|
||||
inputLeft: inputRect.left,
|
||||
inputRight: inputRect.right,
|
||||
inputTop: inputRect.top,
|
||||
inputBottom: inputRect.bottom,
|
||||
// The bug class this guards: a build-time CSS optimization folding the
|
||||
// dock's identity `translate` override into `transform`, leaving
|
||||
// Tailwind's standalone `translate: -50%` live and shifting the dock
|
||||
// half a window off-screen. Surface the computed value so a failure
|
||||
// says WHY the dock moved, not just that it did.
|
||||
dockTranslate: getComputedStyle(dock).translate,
|
||||
}
|
||||
})
|
||||
|
||||
// Horizontal containment — the composer shifted half a window left when the
|
||||
// standalone `translate: -50%` survived optimization (#82214, #82233).
|
||||
expect(geometry.dockLeft).toBeGreaterThanOrEqual(0)
|
||||
expect(geometry.inputLeft).toBeGreaterThanOrEqual(0)
|
||||
expect(geometry.dockRight).toBeLessThanOrEqual(geometry.viewportWidth)
|
||||
expect(geometry.inputRight).toBeLessThanOrEqual(geometry.viewportWidth)
|
||||
|
||||
// Vertical containment — the toolbar/transcript clipping reported on
|
||||
// Windows (#82203) and macOS (#82214) is the same "composer escapes the
|
||||
// window" class on the other axis.
|
||||
expect(geometry.dockTop).toBeGreaterThanOrEqual(0)
|
||||
expect(geometry.inputTop).toBeGreaterThanOrEqual(0)
|
||||
expect(geometry.dockBottom).toBeLessThanOrEqual(geometry.viewportHeight)
|
||||
expect(geometry.inputBottom).toBeLessThanOrEqual(geometry.viewportHeight)
|
||||
|
||||
// The dock's centering translate must be fully neutralized. Any live
|
||||
// percentage translate means the HUD override lost to the app's centering.
|
||||
// (Computed `translate` keeps percentages as-is, so this is assertable;
|
||||
// computed `transform` resolves to a matrix and is covered by the
|
||||
// geometric containment checks above.)
|
||||
expect(geometry.dockTranslate ?? 'none').not.toContain('%')
|
||||
|
||||
await hudPage.close()
|
||||
})
|
||||
|
||||
test('boot progress overlay fades out or shows error state', async () => {
|
||||
const page = fixture!.page
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
const root = document.getElementById('root')
|
||||
|
||||
if (!root) {
|
||||
return false
|
||||
}
|
||||
|
||||
const text = root.textContent ?? ''
|
||||
|
||||
// Error path: boot failure overlay renders an error message.
|
||||
if (text.includes('error') || text.includes('Error') || text.includes('failed')) {
|
||||
return true
|
||||
}
|
||||
|
||||
// Success path: overlay disappears and the app renders. If there's
|
||||
// no "boot" / "starting" / "installing" text visible, boot has
|
||||
// completed (either to the main UI or to onboarding).
|
||||
const bootIndicators = ['starting', 'resolving', 'spawning', 'waiting', 'installing']
|
||||
const lower = text.toLowerCase()
|
||||
|
||||
return !bootIndicators.some((word) => lower.includes(word))
|
||||
},
|
||||
undefined,
|
||||
{ timeout: 60_000 },
|
||||
)
|
||||
})
|
||||
|
||||
test('can capture a screenshot for the CI artifact', async () => {
|
||||
if (!fixture) {
|
||||
test.skip(true, 'Previous test failed — no app running')
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Visual snapshot — won't fail on diff, just logs + generates diff image
|
||||
await expectVisualSnapshot(fixture!.page, { name: 'packaged-app-booted', timeout: 10_000, app: fixture!.app })
|
||||
})
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* E2E tests asserting the mock backend gets the app past the setup/onboarding
|
||||
* screen.
|
||||
*
|
||||
* The mock backend fixture writes a config.yaml with a pre-configured mock
|
||||
* provider pointing at a mock inference server. When the app boots, the
|
||||
* runtime readiness check should detect the working provider and dismiss the
|
||||
* onboarding overlay — landing straight on the chat UI without ever showing
|
||||
* the "Let's get you setup with Hermes Agent" screen.
|
||||
*
|
||||
* If these tests fail, the mock backend config isn't getting the app past
|
||||
* onboarding — the chat interaction tests (chat.spec.ts) will also fail
|
||||
* because the composer is blocked by the setup overlay.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, test } from './test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
import { expectVisualSnapshot } from './visual-snapshot'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture!, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test.describe('mock backend gets past setup screen', () => {
|
||||
test('onboarding overlay is not shown', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
// The onboarding overlay renders "Let's get you setup with Hermes Agent"
|
||||
// when the runtime check fails to find a working provider. With the mock
|
||||
// backend configured, the runtime check should pass and the overlay
|
||||
// returns null — this text should NOT be present in the DOM.
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
const text = document.body.textContent ?? ''
|
||||
|
||||
return !text.includes("Let's get you setup")
|
||||
},
|
||||
undefined,
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
})
|
||||
|
||||
test('chat composer is visible', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
// The composer (contenteditable div) should be visible and not blocked
|
||||
// by the onboarding overlay. If the first test passed, the overlay is
|
||||
// gone and the composer is the primary interactive surface.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await expect(composer).toBeVisible()
|
||||
})
|
||||
|
||||
test('can type into the composer', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
// If the setup overlay is truly gone, the composer accepts input.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.click()
|
||||
await composer.type('hello mock backend', { delay: 20 })
|
||||
|
||||
// Verify the typed text appears in the DOM.
|
||||
await page.waitForFunction(
|
||||
() => (document.body.textContent ?? '').includes('hello mock backend'),
|
||||
undefined,
|
||||
{ timeout: 10_000 },
|
||||
)
|
||||
})
|
||||
|
||||
test('screenshot shows chat UI without setup screen', async () => {
|
||||
await expectVisualSnapshot(fixture!.page, { name: 'mock-backend-chat-ready', app: fixture!.app })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,984 @@
|
||||
/**
|
||||
* Minimal OpenAI-compatible mock inference server for E2E tests.
|
||||
*
|
||||
* Implements just enough of the /v1/* surface for `hermes serve` to resolve a
|
||||
* provider, list models, and stream a canned chat completion back to the
|
||||
* desktop app — without any real LLM.
|
||||
*
|
||||
* Endpoints:
|
||||
* GET /v1/models → { data: [{ id, ... }] }
|
||||
* POST /v1/chat/completions → streaming (SSE) or non-streaming response
|
||||
*
|
||||
* The canned response is a short, deterministic assistant message. Tool-call
|
||||
* requests are not simulated — the E2E tests only need the chat surface to
|
||||
* prove the full boot → gateway → inference → renderer chain works.
|
||||
*/
|
||||
|
||||
import fs from 'node:fs'
|
||||
import http from 'node:http'
|
||||
import type { ServerResponse } from 'node:http'
|
||||
import os from 'node:os'
|
||||
import nodePath from 'node:path'
|
||||
|
||||
/** A canned assistant reply used for every chat completion request. */
|
||||
export const MOCK_REPLY = 'Hello from the mock inference server! The full boot chain is working.'
|
||||
|
||||
export interface MockServerOptions {
|
||||
/** Pause the matching stream after its first token for session-switch E2E coverage. */
|
||||
holdFirstStreamForPrompt?: string
|
||||
/** Pause the first completion whose request JSON contains this text. */
|
||||
holdFirstCompletionContaining?: string
|
||||
/** Absolute sandbox path written by the verify-on-stop scripted tool call. */
|
||||
verificationWritePath?: string
|
||||
/**
|
||||
* Sentinel path that ends the E2E_SIDEBAR_CROSS background process.
|
||||
*
|
||||
* Without it that process is a bare `sleep 5`, which races the agent turn and
|
||||
* the 4s auto-dismiss linger — see `createBackgroundReleaseHandle`. Pass a
|
||||
* handle's `path` to let the test decide when the process exits.
|
||||
*/
|
||||
backgroundReleasePath?: string
|
||||
}
|
||||
|
||||
export interface MockServer {
|
||||
port: number
|
||||
url: string
|
||||
receivedPrompts: string[]
|
||||
waitForHeldStream: () => Promise<void>
|
||||
waitForHeldCompletion: () => Promise<void>
|
||||
releaseHeldStream: () => void
|
||||
heldCompletionCount: () => number
|
||||
close: () => Promise<void>
|
||||
}
|
||||
|
||||
// ─── Multi-turn interim script ─────────────────────────────────────────
|
||||
//
|
||||
// When the user's message contains the trigger keyword, the mock server
|
||||
// walks through a scripted sequence of responses that exercise the
|
||||
// interim-assistant-message fix (#65919) across several patterns:
|
||||
//
|
||||
// 1. text + single tool_call → should produce an interim message
|
||||
// 2. text + single tool_call → another interim message
|
||||
// 3. no text + tool_call → NO interim (no visible text alongside tools)
|
||||
// 4. text + single tool_call → another interim message
|
||||
// 5. final answer (stop) → message.complete, different from all interims
|
||||
//
|
||||
// Each "turn" is one API call. The agent executes the tool after each
|
||||
// tool_calls response, then re-calls the API, advancing to the next turn.
|
||||
|
||||
export interface ScriptedTurn {
|
||||
/** Assistant text content to stream. Empty string = no visible text. */
|
||||
text: string
|
||||
/** Tool calls to emit. Empty array = final turn (finish_reason: stop). */
|
||||
toolCalls?: Array<{
|
||||
name: string
|
||||
args: Record<string, unknown>
|
||||
}>
|
||||
}
|
||||
|
||||
const INTERIM_SCRIPT: ScriptedTurn[] = [
|
||||
{
|
||||
text: 'Let me start by planning the approach.',
|
||||
toolCalls: [{ name: 'todo', args: { todos: [{ id: '1', content: 'Plan', status: 'in_progress' }] } }],
|
||||
},
|
||||
{
|
||||
text: 'Now checking the details before answering.',
|
||||
toolCalls: [{ name: 'todo', args: { todos: [{ id: '2', content: 'Check details', status: 'in_progress' }] } }],
|
||||
},
|
||||
{
|
||||
// No visible text alongside this tool call — should NOT produce an
|
||||
// interim message. The agent fires _emit_interim_assistant_message
|
||||
// but _interim_assistant_visible_text returns "" so it's a no-op.
|
||||
text: '',
|
||||
toolCalls: [{ name: 'todo', args: { todos: [{ id: '3', content: 'Silent step', status: 'completed' }] } }],
|
||||
},
|
||||
{
|
||||
text: 'Found something interesting worth noting.',
|
||||
toolCalls: [{ name: 'todo', args: { todos: [{ id: '4', content: 'Note finding', status: 'completed' }] } }],
|
||||
},
|
||||
{
|
||||
// Final answer — different from all interim texts.
|
||||
text: 'All done! Here is the complete summary of what I found.',
|
||||
},
|
||||
]
|
||||
|
||||
/** Per-server request counter so we can walk through the script turns. */
|
||||
let _scriptIndex = 0
|
||||
|
||||
/** Per-server counter for the sidebar-states script (independent from _scriptIndex). */
|
||||
let _sidebarScriptIndex = 0
|
||||
|
||||
/** Per-server counter for the cross-session sidebar script. */
|
||||
let _sidebarCrossIndex = 0
|
||||
|
||||
/** Per-server counter for the queue-stop script. */
|
||||
let _queueStopIndex = 0
|
||||
|
||||
/** Per-server counter for the correction/session-switch script. */
|
||||
let _correctionSwitchIndex = 0
|
||||
|
||||
/** Per-server counter for the verify-on-stop script. */
|
||||
let _verificationStopIndex = 0
|
||||
|
||||
/** Per-server counter for the task-panel warm-resume script. */
|
||||
let _taskPanelResumeIndex = 0
|
||||
|
||||
/** User messages received by the mock, for E2E assertions on real submits. */
|
||||
const _receivedUserTexts: string[] = []
|
||||
|
||||
/** Reset the script indices (called between tests via restartMockServer). */
|
||||
function resetScriptIndex(): void {
|
||||
_scriptIndex = 0
|
||||
_sidebarScriptIndex = 0
|
||||
_sidebarCrossIndex = 0
|
||||
_queueStopIndex = 0
|
||||
_correctionSwitchIndex = 0
|
||||
_verificationStopIndex = 0
|
||||
_taskPanelResumeIndex = 0
|
||||
_receivedUserTexts.length = 0
|
||||
}
|
||||
|
||||
/** Return the user prompts the real backend submitted to this mock server. */
|
||||
export function receivedUserTexts(): readonly string[] {
|
||||
return _receivedUserTexts
|
||||
}
|
||||
|
||||
// ─── Sidebar-states script ─────────────────────────────────────────────
|
||||
//
|
||||
// A separate trigger (E2E_SIDEBAR_TRIGGER) exercises the desktop sidebar's
|
||||
// background-process and subagent states. The mock returns tool_calls that
|
||||
// the agent executes for real — `terminal(background=true)` spawns a real
|
||||
// (but trivial) background process, and `delegate_task` spawns a real
|
||||
// subagent that calls the mock server and gets the canned reply.
|
||||
//
|
||||
// Turn 1: text + terminal(bg=true) + delegate_task → tools execute
|
||||
// Turn 2: final answer → message.complete, dot transitions
|
||||
|
||||
const SIDEBAR_SCRIPT: ScriptedTurn[] = [
|
||||
{
|
||||
text: 'Let me run a background task and delegate some work.',
|
||||
toolCalls: [
|
||||
{
|
||||
name: 'terminal',
|
||||
args: {
|
||||
command: 'echo "background process output" && sleep 1 && echo "done"',
|
||||
background: true,
|
||||
notify_on_complete: true,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'delegate_task',
|
||||
args: {
|
||||
goal: 'Summarize the test results',
|
||||
context: 'This is a test subagent for the sidebar states E2E test.',
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
text: 'All tasks complete. The background process finished and the subagent returned its summary.',
|
||||
},
|
||||
]
|
||||
|
||||
// ─── Sidebar cross-session script ──────────────────────────────────────
|
||||
//
|
||||
// E2E_SIDEBAR_CROSS starts a long background process plus a subagent so the
|
||||
// tests can:
|
||||
// 1. See the background dot while the subagent runs.
|
||||
// 2. Open a different session and see session A's dot transition to
|
||||
// "finished unread" when the background process completes.
|
||||
//
|
||||
// The background process must outlive the agent turn — the whole point is a
|
||||
// dot that is still "running" after the final answer lands. A fixed `sleep`
|
||||
// cannot guarantee that: on a loaded CI runner the turn (two model round
|
||||
// trips + a real subagent delegation) can take longer than the sleep, the
|
||||
// process exits early, the 4s success linger elapses, and the dot is gone
|
||||
// before the test looks. That is a wall-clock race between three independent
|
||||
// timers, and it made this the flakiest spec in the suite.
|
||||
//
|
||||
// When `backgroundReleasePath` is set the process instead blocks until the
|
||||
// test creates that sentinel file, so the test — not the clock — decides when
|
||||
// the dot clears. `sleep 5` remains the fallback for callers that don't pass
|
||||
// a handle.
|
||||
function sidebarCrossBgCommand(releasePath?: string): string {
|
||||
if (!releasePath) {
|
||||
return 'echo "long bg output" && sleep 5 && echo "finished"'
|
||||
}
|
||||
// Bounded wait (60s): if a test forgets to release (or crashes mid-way),
|
||||
// the process still exits instead of hanging the worker until the suite
|
||||
// times out.
|
||||
const quoted = JSON.stringify(releasePath)
|
||||
return [
|
||||
'echo "long bg output"',
|
||||
`for _ in $(seq 1 600); do [ -e ${quoted} ] && break; sleep 0.1; done`,
|
||||
'echo "finished"',
|
||||
].join(' && ')
|
||||
}
|
||||
|
||||
function sidebarCrossScript(releasePath?: string): ScriptedTurn[] {
|
||||
return [
|
||||
{
|
||||
text: 'Starting a long background task and delegating work.',
|
||||
toolCalls: [
|
||||
{
|
||||
name: 'terminal',
|
||||
args: {
|
||||
command: sidebarCrossBgCommand(releasePath),
|
||||
background: true,
|
||||
notify_on_complete: true,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'delegate_task',
|
||||
args: {
|
||||
goal: 'Analyze cross-session state',
|
||||
context: 'Testing that the background dot updates across sessions.',
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
text: 'Both tasks are running in the background now.',
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
const SIDEBAR_CROSS_SCRIPT: ScriptedTurn[] = sidebarCrossScript()
|
||||
|
||||
const QUEUE_STOP_SCRIPT: ScriptedTurn[] = [
|
||||
{
|
||||
text: 'Starting a task that will keep this turn active.',
|
||||
toolCalls: [{ name: 'clarify', args: { question: 'Keep working?', choices: ['Yes', 'No'] } }],
|
||||
},
|
||||
{ text: 'The paused task completed.' },
|
||||
]
|
||||
|
||||
// The reported correction arrived while a foreground tool was still running.
|
||||
// Keep that boundary open long enough for the renderer to redirect the turn,
|
||||
// then let the next model request complete normally.
|
||||
const CORRECTION_SWITCH_SCRIPT: ScriptedTurn[] = [
|
||||
{
|
||||
text: 'Checking the long-running task before I continue.',
|
||||
toolCalls: [{ name: 'terminal', args: { command: 'sleep 5' } }],
|
||||
},
|
||||
{ text: 'The corrected task finished.' },
|
||||
]
|
||||
|
||||
export const CORRECTION_SWITCH_TRIGGER = 'E2E_CORRECTION_SWITCH_TRIGGER'
|
||||
|
||||
/**
|
||||
* Drives a real code edit followed by two finish attempts. Hermes should add
|
||||
* its synthetic verify-on-stop continuation after each finish attempt until
|
||||
* the bounded verifier gives up. The mock's request capture proves the nudge
|
||||
* reached the model; desktop must never render it as chat content.
|
||||
*/
|
||||
function verificationStopScript(writePath: string): ScriptedTurn[] {
|
||||
return [
|
||||
{
|
||||
text: 'I will make the requested code change.',
|
||||
toolCalls: [{
|
||||
name: 'write_file',
|
||||
args: {
|
||||
path: writePath,
|
||||
content: 'def changed_by_e2e():\n return "changed"\n',
|
||||
},
|
||||
}],
|
||||
},
|
||||
{ text: 'The code edit is complete.' },
|
||||
{ text: 'I cannot provide fresh verification evidence for that edit.' },
|
||||
]
|
||||
}
|
||||
|
||||
export const VERIFICATION_STOP_TRIGGER = 'E2E_VERIFY_ON_STOP_TRIGGER'
|
||||
export const VERIFICATION_STOP_TEXT = 'I cannot provide fresh verification evidence for that edit.'
|
||||
|
||||
/**
|
||||
* A marker that makes the mock emit a real blocking clarify tool call. Tests
|
||||
* use it to hold a turn open while exercising busy-composer interactions.
|
||||
*/
|
||||
export const BLOCKING_CLARIFY_TRIGGER = 'E2E_BLOCKING_CLARIFY_TRIGGER'
|
||||
export const BLOCKING_CLARIFY_QUESTION = 'Keep this test turn running?'
|
||||
|
||||
/**
|
||||
* A long live response with a five-row todo card, held open by a foreground tool.
|
||||
* The transcript is deliberately taller than the viewport so warm-session
|
||||
* tests can detect when re-opening the session leaves it above the true bottom.
|
||||
*/
|
||||
export const TASK_PANEL_RESUME_TRIGGER = 'E2E_TASK_PANEL_RESUME_TRIGGER'
|
||||
export const TASK_PANEL_RESUME_TEXT = Array.from(
|
||||
{ length: 24 },
|
||||
(_, index) => `Task-panel clearance line ${index + 1}: inspect the restored working session geometry.`,
|
||||
).join('\n\n')
|
||||
|
||||
const TASK_PANEL_RESUME_SCRIPT: ScriptedTurn[] = [
|
||||
{
|
||||
text: TASK_PANEL_RESUME_TEXT,
|
||||
toolCalls: [
|
||||
{
|
||||
name: 'todo',
|
||||
args: {
|
||||
todos: [
|
||||
{ id: 'design', content: 'Design the restored layout', status: 'completed' },
|
||||
{ id: 'implement', content: 'Implement the measured clearance', status: 'in_progress' },
|
||||
{ id: 'verify', content: 'Verify the latest message stays visible', status: 'pending' },
|
||||
{ id: 'review', content: 'Review the visual regression', status: 'pending' },
|
||||
{ id: 'ship', content: 'Ship the focused fix', status: 'pending' },
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'terminal',
|
||||
args: { command: 'sleep 60' },
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
const BLOCKING_CLARIFY_TURN: ScriptedTurn = {
|
||||
text: '',
|
||||
toolCalls: [{ name: 'clarify', args: { question: BLOCKING_CLARIFY_QUESTION, choices: ['Yes', 'No'] } }],
|
||||
}
|
||||
|
||||
/**
|
||||
* A marker that makes the mock emit a blocking BATCH clarify tool call
|
||||
* (multi-question form). Regression coverage for the duplicated-card bug:
|
||||
* the tool.start row and the clarify.request row carry different ids and a
|
||||
* batch payload has no top-level question, so the correlation key must come
|
||||
* from the question list or the card mounts twice.
|
||||
*/
|
||||
export const BATCH_CLARIFY_TRIGGER = 'E2E_BATCH_CLARIFY_TRIGGER'
|
||||
export const BATCH_CLARIFY_QUESTIONS = [
|
||||
{ question: 'Pick a batch drink?', choices: ['Coffee', 'Tea'] },
|
||||
{ question: 'Pick a batch time?', choices: ['Morning', 'Night'] },
|
||||
]
|
||||
|
||||
const BATCH_CLARIFY_TURN: ScriptedTurn = {
|
||||
text: '',
|
||||
toolCalls: [{ name: 'clarify', args: { questions: BATCH_CLARIFY_QUESTIONS } }],
|
||||
}
|
||||
|
||||
function includesBatchClarifyTrigger(value: unknown): boolean {
|
||||
if (typeof value === 'string') {
|
||||
return value.includes(BATCH_CLARIFY_TRIGGER)
|
||||
}
|
||||
|
||||
if (Array.isArray(value)) {
|
||||
return value.some(includesBatchClarifyTrigger)
|
||||
}
|
||||
|
||||
if (value && typeof value === 'object') {
|
||||
return Object.values(value).some(includesBatchClarifyTrigger)
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
function includesBlockingClarifyTrigger(value: unknown): boolean {
|
||||
if (typeof value === 'string') {
|
||||
return value.includes(BLOCKING_CLARIFY_TRIGGER)
|
||||
}
|
||||
|
||||
if (Array.isArray(value)) {
|
||||
return value.some(includesBlockingClarifyTrigger)
|
||||
}
|
||||
|
||||
if (value && typeof value === 'object') {
|
||||
return Object.values(value).some(includesBlockingClarifyTrigger)
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Start the mock server on an ephemeral port.
|
||||
*
|
||||
* @returns a handle with `port`, `url`, received user prompts, and `close()`.
|
||||
*/
|
||||
export function startMockServer(options: MockServerOptions = {}): Promise<MockServer> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const receivedPrompts: string[] = []
|
||||
let resolveHeldStreamStarted: (() => void) | null = null
|
||||
let releaseHeldStream: (() => void) | null = null
|
||||
let heldCompletionCount = 0
|
||||
const heldStreamStarted = new Promise<void>(resolveHeld => {
|
||||
resolveHeldStreamStarted = resolveHeld
|
||||
})
|
||||
const heldStreamReleased = new Promise<void>(resolveRelease => {
|
||||
releaseHeldStream = resolveRelease
|
||||
})
|
||||
const server = http.createServer((req, res) => {
|
||||
// CORS headers — the Electron renderer doesn't need them, but they
|
||||
// don't hurt and make the server usable from a browser context too.
|
||||
res.setHeader('Access-Control-Allow-Origin', '*')
|
||||
res.setHeader('Access-Control-Allow-Headers', '*')
|
||||
res.setHeader('Access-Control-Allow-Methods', 'GET, POST, OPTIONS')
|
||||
|
||||
if (req.method === 'OPTIONS') {
|
||||
res.writeHead(204)
|
||||
res.end()
|
||||
return
|
||||
}
|
||||
|
||||
// GET /v1/models — return a single fake model.
|
||||
if (req.method === 'GET' && req.url === '/v1/models') {
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' })
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
object: 'list',
|
||||
data: [
|
||||
{
|
||||
id: 'mock-model',
|
||||
object: 'model',
|
||||
created: 0,
|
||||
owned_by: 'mock',
|
||||
},
|
||||
],
|
||||
}),
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
// POST /v1/chat/completions — return a canned response.
|
||||
if (req.method === 'POST' && req.url?.startsWith('/v1/chat/completions')) {
|
||||
let body = ''
|
||||
|
||||
req.on('data', (chunk: Buffer) => {
|
||||
body += chunk.toString()
|
||||
})
|
||||
|
||||
req.on('end', () => {
|
||||
let parsed: any = {}
|
||||
|
||||
try {
|
||||
parsed = JSON.parse(body)
|
||||
} catch {
|
||||
// malformed JSON — treat as non-streaming with defaults
|
||||
}
|
||||
|
||||
const lastUserMessage = [...(parsed.messages ?? [])]
|
||||
.reverse()
|
||||
.find((message: { role?: unknown }) => message?.role === 'user')
|
||||
|
||||
if (typeof lastUserMessage?.content === 'string') {
|
||||
receivedPrompts.push(lastUserMessage.content)
|
||||
}
|
||||
|
||||
const stream = parsed.stream === true
|
||||
const model = parsed.model || 'mock-model'
|
||||
const holdThisCompletion = Boolean(
|
||||
options.holdFirstCompletionContaining &&
|
||||
heldCompletionCount === 0 &&
|
||||
JSON.stringify(parsed).includes(options.holdFirstCompletionContaining),
|
||||
)
|
||||
|
||||
// Detect the interim-message test trigger: the user's message
|
||||
// contains a specific keyword. The mock walks through the
|
||||
// INTERIM_SCRIPT turns in sequence.
|
||||
//
|
||||
// The trigger keyword is chosen so normal chat tests (which send
|
||||
// "Hello, can you hear me?" etc.) never hit this path.
|
||||
const messages: any[] = Array.isArray(parsed.messages) ? parsed.messages : []
|
||||
const lastUserMsg = [...messages].reverse().find(m => m?.role === 'user')
|
||||
const userText = typeof lastUserMsg?.content === 'string' ? lastUserMsg.content : ''
|
||||
if (userText) {
|
||||
_receivedUserTexts.push(userText)
|
||||
}
|
||||
const isInterimTrigger = userText.includes('E2E_INTERIM_TRIGGER')
|
||||
const isSidebarTrigger = userText.includes('E2E_SIDEBAR_TRIGGER')
|
||||
const isSidebarCrossTrigger = userText.includes('E2E_SIDEBAR_CROSS')
|
||||
const isQueueStopTrigger = userText.includes('E2E_QUEUE_STOP_TRIGGER')
|
||||
const isTaskPanelResumeTrigger = userText.includes(TASK_PANEL_RESUME_TRIGGER)
|
||||
const isVerificationStopTrigger = messages.some(
|
||||
message => typeof message?.content === 'string' && message.content.includes(VERIFICATION_STOP_TRIGGER),
|
||||
)
|
||||
const isCorrectionSwitchTrigger = messages.some(
|
||||
message => typeof message?.content === 'string' && message.content.includes(CORRECTION_SWITCH_TRIGGER),
|
||||
)
|
||||
|
||||
if (isTaskPanelResumeTrigger) {
|
||||
const turn =
|
||||
TASK_PANEL_RESUME_SCRIPT[_taskPanelResumeIndex] ??
|
||||
TASK_PANEL_RESUME_SCRIPT[TASK_PANEL_RESUME_SCRIPT.length - 1]
|
||||
_taskPanelResumeIndex++
|
||||
const respond = () => {
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, turn)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, turn)
|
||||
}
|
||||
}
|
||||
|
||||
if (holdThisCompletion) {
|
||||
heldCompletionCount++
|
||||
resolveHeldStreamStarted?.()
|
||||
void heldStreamReleased.then(respond)
|
||||
} else {
|
||||
respond()
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (includesBatchClarifyTrigger(parsed.messages)) {
|
||||
// Only the FIRST completion of the conversation scripts the batch
|
||||
// clarify. The trigger text stays in message history, so once the
|
||||
// answered tool result is present the turn falls through to the
|
||||
// canned reply — otherwise the mock loops the quiz forever.
|
||||
const hasToolResult = Array.isArray(parsed.messages)
|
||||
&& parsed.messages.some((message: { role?: string }) => message?.role === 'tool')
|
||||
|
||||
if (!hasToolResult) {
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, BATCH_CLARIFY_TURN)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, BATCH_CLARIFY_TURN)
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if (includesBlockingClarifyTrigger(parsed.messages)) {
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, BLOCKING_CLARIFY_TURN)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, BLOCKING_CLARIFY_TURN)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (isQueueStopTrigger) {
|
||||
const turn = QUEUE_STOP_SCRIPT[_queueStopIndex] ?? QUEUE_STOP_SCRIPT[QUEUE_STOP_SCRIPT.length - 1]
|
||||
_queueStopIndex++
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, turn)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, turn)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (isVerificationStopTrigger) {
|
||||
const script = verificationStopScript(options.verificationWritePath ?? 'e2e-verification-target.py')
|
||||
const turn = script[_verificationStopIndex] ?? script[script.length - 1]
|
||||
_verificationStopIndex++
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, turn)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, turn)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (isCorrectionSwitchTrigger) {
|
||||
const turn = CORRECTION_SWITCH_SCRIPT[_correctionSwitchIndex] ?? CORRECTION_SWITCH_SCRIPT[CORRECTION_SWITCH_SCRIPT.length - 1]
|
||||
_correctionSwitchIndex++
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, turn)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, turn)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (isSidebarCrossTrigger) {
|
||||
const script = sidebarCrossScript(options.backgroundReleasePath)
|
||||
const turn = script[_sidebarCrossIndex] ?? script[script.length - 1]
|
||||
_sidebarCrossIndex++
|
||||
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, turn)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, turn)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (isSidebarTrigger) {
|
||||
const turn = SIDEBAR_SCRIPT[_sidebarScriptIndex] ?? SIDEBAR_SCRIPT[SIDEBAR_SCRIPT.length - 1]
|
||||
_sidebarScriptIndex++
|
||||
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, turn)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, turn)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (isInterimTrigger) {
|
||||
const turn = INTERIM_SCRIPT[_scriptIndex] ?? INTERIM_SCRIPT[INTERIM_SCRIPT.length - 1]
|
||||
_scriptIndex++
|
||||
if (stream) {
|
||||
streamScriptedTurn(res, model, turn)
|
||||
} else {
|
||||
nonStreamingScriptedTurn(res, model, turn)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (stream) {
|
||||
const holdThisStream = Boolean(
|
||||
options.holdFirstStreamForPrompt && typeof lastUserMessage?.content === 'string' &&
|
||||
lastUserMessage.content.includes(options.holdFirstStreamForPrompt),
|
||||
)
|
||||
streamTextResponse(res, model, MOCK_REPLY, holdThisStream || holdThisCompletion ? () => {
|
||||
if (holdThisCompletion) {
|
||||
heldCompletionCount++
|
||||
}
|
||||
resolveHeldStreamStarted?.()
|
||||
return heldStreamReleased
|
||||
} : undefined)
|
||||
} else {
|
||||
if (holdThisCompletion) {
|
||||
heldCompletionCount++
|
||||
resolveHeldStreamStarted?.()
|
||||
void heldStreamReleased.then(() => nonStreamingTextResponse(res, model, MOCK_REPLY))
|
||||
} else {
|
||||
nonStreamingTextResponse(res, model, MOCK_REPLY)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
req.on('error', () => {
|
||||
res.writeHead(400)
|
||||
res.end('Bad request')
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Fallback — 404 for anything else
|
||||
res.writeHead(404, { 'Content-Type': 'application/json' })
|
||||
res.end(JSON.stringify({ error: 'Not found' }))
|
||||
})
|
||||
|
||||
server.on('error', reject)
|
||||
|
||||
server.listen(0, '127.0.0.1', () => {
|
||||
const addr = server.address()
|
||||
if (addr === null || typeof addr === 'string') {
|
||||
reject(new Error('Failed to get server address'))
|
||||
return
|
||||
}
|
||||
|
||||
const port = addr.port
|
||||
const url = `http://127.0.0.1:${port}`
|
||||
|
||||
resolve({
|
||||
port,
|
||||
url,
|
||||
receivedPrompts,
|
||||
waitForHeldStream: () => heldStreamStarted,
|
||||
waitForHeldCompletion: () => heldStreamStarted,
|
||||
releaseHeldStream: () => releaseHeldStream?.(),
|
||||
heldCompletionCount: () => heldCompletionCount,
|
||||
close: () =>
|
||||
new Promise((resolveClose, rejectClose) => {
|
||||
server.close((err) => {
|
||||
if (err) {
|
||||
rejectClose(err)
|
||||
} else {
|
||||
resolveClose()
|
||||
}
|
||||
})
|
||||
}),
|
||||
})
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
// ─── Response helpers ──────────────────────────────────────────────────
|
||||
|
||||
/** SSE chunk shape for a streaming chat completion. */
|
||||
function sseChunk(model: string, delta: Record<string, unknown>, finishReason: string | null = null): string {
|
||||
return `data: ${JSON.stringify({
|
||||
id: 'mock-completion',
|
||||
object: 'chat.completion.chunk',
|
||||
created: 0,
|
||||
model,
|
||||
choices: [{ index: 0, delta, finish_reason: finishReason }],
|
||||
})}\n\n`
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream a plain text response (no tool calls) as SSE, finishing with
|
||||
* `finish_reason: "stop"`. This is the default canned-reply path.
|
||||
*/
|
||||
function streamTextResponse(
|
||||
res: ServerResponse,
|
||||
model: string,
|
||||
text: string,
|
||||
waitForRelease?: () => Promise<void>,
|
||||
): void {
|
||||
res.writeHead(200, {
|
||||
'Content-Type': 'text/event-stream',
|
||||
'Cache-Control': 'no-cache',
|
||||
Connection: 'keep-alive',
|
||||
})
|
||||
|
||||
const words = text.split(' ')
|
||||
let i = 0
|
||||
|
||||
const sendChunk = (): void => {
|
||||
if (i >= words.length) {
|
||||
res.write(sseChunk(model, {}, 'stop'))
|
||||
res.write('data: [DONE]\n\n')
|
||||
res.end()
|
||||
return
|
||||
}
|
||||
|
||||
const word = i === 0 ? words[i] : ' ' + words[i]
|
||||
res.write(sseChunk(model, { content: word }))
|
||||
i++
|
||||
if (waitForRelease && i === 1) {
|
||||
waitForRelease().then(() => setTimeout(sendChunk, 20))
|
||||
return
|
||||
}
|
||||
setTimeout(sendChunk, 20)
|
||||
}
|
||||
|
||||
sendChunk()
|
||||
}
|
||||
|
||||
/** Non-streaming plain text response. */
|
||||
function nonStreamingTextResponse(res: ServerResponse, model: string, text: string): void {
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' })
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
id: 'mock-completion',
|
||||
object: 'chat.completion',
|
||||
created: 0,
|
||||
model,
|
||||
choices: [
|
||||
{
|
||||
index: 0,
|
||||
message: { role: 'assistant', content: text },
|
||||
finish_reason: 'stop',
|
||||
},
|
||||
],
|
||||
usage: { prompt_tokens: 10, completion_tokens: 20, total_tokens: 30 },
|
||||
}),
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream a single scripted turn: first the text content (word by word),
|
||||
* then a chunk carrying the tool_calls (if any), with the appropriate
|
||||
* finish_reason.
|
||||
*
|
||||
* If the turn has no text and no tool calls, it's an empty final response.
|
||||
* If it has text but no tool calls, it's a final answer (finish_reason: stop).
|
||||
* If it has tool calls (with or without text), finish_reason is "tool_calls".
|
||||
*/
|
||||
function streamScriptedTurn(
|
||||
res: ServerResponse,
|
||||
model: string,
|
||||
turn: ScriptedTurn,
|
||||
): void {
|
||||
res.writeHead(200, {
|
||||
'Content-Type': 'text/event-stream',
|
||||
'Cache-Control': 'no-cache',
|
||||
Connection: 'keep-alive',
|
||||
})
|
||||
|
||||
const hasToolCalls = turn.toolCalls && turn.toolCalls.length > 0
|
||||
const finishReason = hasToolCalls ? 'tool_calls' : 'stop'
|
||||
|
||||
// If there's no text to stream, go straight to the tool_calls / finish.
|
||||
if (!turn.text) {
|
||||
if (hasToolCalls) {
|
||||
res.write(
|
||||
sseChunk(model, {
|
||||
tool_calls: turn.toolCalls!.map((tc, idx) => ({
|
||||
index: idx,
|
||||
id: `call_e2e_${_scriptIndex}_${idx}`,
|
||||
type: 'function',
|
||||
function: { name: tc.name, arguments: JSON.stringify(tc.args) },
|
||||
})),
|
||||
}, finishReason),
|
||||
)
|
||||
} else {
|
||||
res.write(sseChunk(model, {}, finishReason))
|
||||
}
|
||||
res.write('data: [DONE]\n\n')
|
||||
res.end()
|
||||
return
|
||||
}
|
||||
|
||||
// Stream the text word by word, then emit tool_calls if present.
|
||||
const words = turn.text.split(' ')
|
||||
let i = 0
|
||||
|
||||
const sendChunk = (): void => {
|
||||
if (i >= words.length) {
|
||||
// All text streamed — emit tool_calls if present, then finish.
|
||||
if (hasToolCalls) {
|
||||
res.write(
|
||||
sseChunk(model, {
|
||||
tool_calls: turn.toolCalls!.map((tc, idx) => ({
|
||||
index: idx,
|
||||
id: `call_e2e_${_scriptIndex}_${idx}`,
|
||||
type: 'function',
|
||||
function: { name: tc.name, arguments: JSON.stringify(tc.args) },
|
||||
})),
|
||||
}, finishReason),
|
||||
)
|
||||
} else {
|
||||
res.write(sseChunk(model, {}, finishReason))
|
||||
}
|
||||
res.write('data: [DONE]\n\n')
|
||||
res.end()
|
||||
return
|
||||
}
|
||||
|
||||
const word = i === 0 ? words[i] : ' ' + words[i]
|
||||
res.write(sseChunk(model, { content: word }))
|
||||
i++
|
||||
setTimeout(sendChunk, 20)
|
||||
}
|
||||
|
||||
sendChunk()
|
||||
}
|
||||
|
||||
/** Non-streaming version of a scripted turn. */
|
||||
function nonStreamingScriptedTurn(
|
||||
res: ServerResponse,
|
||||
model: string,
|
||||
turn: ScriptedTurn,
|
||||
): void {
|
||||
const hasToolCalls = turn.toolCalls && turn.toolCalls.length > 0
|
||||
const finishReason = hasToolCalls ? 'tool_calls' : 'stop'
|
||||
|
||||
const message: Record<string, unknown> = { role: 'assistant' }
|
||||
if (turn.text) {
|
||||
message.content = turn.text
|
||||
}
|
||||
if (hasToolCalls) {
|
||||
message.tool_calls = turn.toolCalls!.map((tc, idx) => ({
|
||||
id: `call_e2e_${_scriptIndex}_${idx}`,
|
||||
type: 'function',
|
||||
function: { name: tc.name, arguments: JSON.stringify(tc.args) },
|
||||
}))
|
||||
}
|
||||
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' })
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
id: 'mock-completion',
|
||||
object: 'chat.completion',
|
||||
created: 0,
|
||||
model,
|
||||
choices: [{ index: 0, message, finish_reason: finishReason }],
|
||||
usage: { prompt_tokens: 10, completion_tokens: 20, total_tokens: 30 },
|
||||
}),
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Restart the mock server's script index so each test starts from turn 0.
|
||||
* Call this between tests that use the interim trigger.
|
||||
*/
|
||||
export function restartMockServer(): void {
|
||||
resetScriptIndex()
|
||||
}
|
||||
|
||||
/** Test-controlled lifetime for the E2E_SIDEBAR_CROSS background process. */
|
||||
export interface BackgroundReleaseHandle {
|
||||
/** Sentinel path — pass as `backgroundReleasePath` to `startMockServer`. */
|
||||
path: string
|
||||
/** End the background process now (creates the sentinel). */
|
||||
release: () => void
|
||||
/** Remove the sentinel if it still exists. Safe to call twice. */
|
||||
cleanup: () => void
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a sentinel that keeps the E2E_SIDEBAR_CROSS background process alive
|
||||
* until the test explicitly releases it.
|
||||
*
|
||||
* The cross-session sidebar tests need a background process that is still
|
||||
* RUNNING after the agent turn finishes — that is the state under test (a
|
||||
* session whose turn is done but whose background work is not). With a fixed
|
||||
* `sleep`, three independent clocks race: the sleep, the agent turn (two model
|
||||
* round trips plus a real subagent delegation), and the 4s success linger
|
||||
* before a finished task auto-dismisses. When a loaded CI runner makes the
|
||||
* turn slower than the sleep, the process is already gone and the assertion
|
||||
* samples an empty sidebar. Observed on CI 2026-07-26 across two unrelated
|
||||
* PRs: the "should appear" poll needed 7.5s to see the dot, by which point
|
||||
* `sleep 5` had exited.
|
||||
*
|
||||
* With a sentinel there is one clock and the test owns it:
|
||||
*
|
||||
* ```ts
|
||||
* const release = createBackgroundReleaseHandle()
|
||||
* const mock = await startMockServer({ backgroundReleasePath: release.path })
|
||||
* // ... assert the dot is visible; it cannot vanish on its own ...
|
||||
* release.release() // now, and only now, the process exits
|
||||
* ```
|
||||
*/
|
||||
export function createBackgroundReleaseHandle(): BackgroundReleaseHandle {
|
||||
const path = nodePath.join(
|
||||
os.tmpdir(),
|
||||
`hermes-e2e-bg-release-${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
)
|
||||
return {
|
||||
path,
|
||||
release: () => {
|
||||
try {
|
||||
fs.writeFileSync(path, 'release')
|
||||
} catch {
|
||||
// The process also has a bounded fallback wait; a failed write must
|
||||
// not crash the test before its real assertions run.
|
||||
}
|
||||
},
|
||||
cleanup: () => {
|
||||
try {
|
||||
fs.rmSync(path, { force: true })
|
||||
} catch {
|
||||
// Best-effort — the sentinel lives in the OS temp dir.
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The interim script's text constants, exported for test assertions.
|
||||
* Each entry is the visible text of one turn. Turns with empty text
|
||||
* produce no interim message and are excluded from this list.
|
||||
*/
|
||||
export const INTERIM_TEXTS = {
|
||||
/** All interim texts that should appear as sealed messages when the flag is ON. */
|
||||
interims: INTERIM_SCRIPT
|
||||
.filter((t) => t.text && t.toolCalls)
|
||||
.map((t) => t.text),
|
||||
/** The final answer text. */
|
||||
finalText: INTERIM_SCRIPT[INTERIM_SCRIPT.length - 1].text,
|
||||
/** Text that should NOT produce an interim (empty-text tool turn). */
|
||||
silentTurnIndex: INTERIM_SCRIPT.findIndex((t) => !t.text && t.toolCalls),
|
||||
} as const
|
||||
|
||||
/** The sidebar-states script's text constants, exported for test assertions. */
|
||||
export const SIDEBAR_TEXTS = {
|
||||
/** The interim text from turn 1 (alongside tool calls). */
|
||||
interimText: SIDEBAR_SCRIPT[0].text,
|
||||
/** The final answer text. */
|
||||
finalText: SIDEBAR_SCRIPT[SIDEBAR_SCRIPT.length - 1].text,
|
||||
/** The background process command (for asserting process.list entries). */
|
||||
bgCommand: 'echo "background process output" && sleep 1 && echo "done"',
|
||||
/** The subagent's goal (for asserting subagent panel state). */
|
||||
subagentGoal: 'Summarize the test results',
|
||||
} as const
|
||||
|
||||
/** The cross-session sidebar script's text constants. */
|
||||
export const SIDEBAR_CROSS_TEXTS = {
|
||||
/** The interim text from turn 1. */
|
||||
interimText: SIDEBAR_CROSS_SCRIPT[0].text,
|
||||
/** The final answer text. */
|
||||
finalText: SIDEBAR_CROSS_SCRIPT[SIDEBAR_CROSS_SCRIPT.length - 1].text,
|
||||
/**
|
||||
* The default (unheld) background process command. Tests that pass a
|
||||
* `backgroundReleasePath` get a sentinel-waiting command instead — see
|
||||
* `createBackgroundReleaseHandle`.
|
||||
*/
|
||||
bgCommand: sidebarCrossBgCommand(),
|
||||
/** The subagent's goal. */
|
||||
subagentGoal: 'Analyze cross-session state',
|
||||
} as const
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* E2E onboarding tests — verify the provider picker appears when no
|
||||
* inference provider is configured.
|
||||
*
|
||||
* Launches the app with an empty config.yaml (no providers). The renderer
|
||||
* should detect the unconfigured state and show the DesktopOnboardingOverlay
|
||||
* with provider options / API key form.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, test } from './test'
|
||||
|
||||
import {
|
||||
type NoProviderFixture,
|
||||
setupNoProvider,
|
||||
waitForOnboarding,
|
||||
} from './fixtures'
|
||||
import { expectVisualSnapshot } from './visual-snapshot'
|
||||
|
||||
let fixture: NoProviderFixture | null = null
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test.describe('onboarding with no provider configured', () => {
|
||||
test('onboarding overlay appears on first boot', async () => {
|
||||
fixture = await setupNoProvider()
|
||||
|
||||
// The app should boot (hermes serve starts fine even without a provider),
|
||||
// but the renderer should show the onboarding overlay because no
|
||||
// provider is configured.
|
||||
await waitForOnboarding(fixture.page, 90_000)
|
||||
})
|
||||
|
||||
test('onboarding shows provider options or API key form', async () => {
|
||||
if (!fixture) {
|
||||
test.skip(true, 'Previous test failed — no app running')
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
const page = fixture.page
|
||||
|
||||
// The onboarding overlay should contain provider-related text.
|
||||
// It might show OAuth providers, an API key form, or a "choose later"
|
||||
// link. Verify at least one of these is visible.
|
||||
const rootText = await page.evaluate(() => {
|
||||
const root = document.getElementById('root')
|
||||
|
||||
return root?.textContent ?? ''
|
||||
})
|
||||
|
||||
const hasProviderText =
|
||||
rootText.includes('provider') ||
|
||||
rootText.includes('Provider') ||
|
||||
rootText.includes('API key') ||
|
||||
rootText.includes('Sign in') ||
|
||||
rootText.includes('OpenRouter') ||
|
||||
rootText.includes('OpenAI')
|
||||
|
||||
expect(hasProviderText).toBe(true)
|
||||
})
|
||||
|
||||
test('screenshot of onboarding overlay', async () => {
|
||||
if (!fixture) {
|
||||
test.skip(true, 'Previous test failed — no app running')
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
await expectVisualSnapshot(fixture.page, { name: 'onboarding-overlay', app: fixture.app })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,119 @@
|
||||
/**
|
||||
* A queued prompt must remain local until the current inference turn settles.
|
||||
*
|
||||
* Hold the first streamed reply open after its first token. This gives the
|
||||
* composer a live, busy turn while the user queues a follow-up, then lets us
|
||||
* assert against the mock provider's real request log before and after the
|
||||
* held turn completes.
|
||||
*/
|
||||
|
||||
import { expect, test, type Page } from './test'
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { MOCK_REPLY } from './mock-server'
|
||||
|
||||
const ACTIVE_PROMPT = 'E2E_QUEUE_TURN_BOUNDARY_ACTIVE'
|
||||
const QUEUED_PROMPT = 'E2E_QUEUE_TURN_BOUNDARY_QUEUED'
|
||||
const STEER_PROMPT = 'E2E_STEER_TURN_BOUNDARY_CORRECTION'
|
||||
|
||||
async function send(page: Page, text: string): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.type(text, { delay: 5 })
|
||||
await page.keyboard.press('Enter')
|
||||
}
|
||||
|
||||
async function steer(page: Page, text: string): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
|
||||
await composer.click()
|
||||
await composer.type(text, { delay: 5 })
|
||||
await page.keyboard.press('Enter')
|
||||
}
|
||||
|
||||
async function queue(page: Page, text: string): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
|
||||
await composer.click()
|
||||
await composer.type(text, { delay: 5 })
|
||||
await page.keyboard.press('Control+Enter')
|
||||
}
|
||||
|
||||
async function transcriptMessageOrder(page: Page): Promise<string[]> {
|
||||
return page.evaluate(() => {
|
||||
const viewport = document.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
if (!viewport) return []
|
||||
|
||||
return Array.from(viewport.querySelectorAll<HTMLElement>('[data-role="user"], [data-role="assistant"]'))
|
||||
.map(message => message.textContent?.trim() ?? '')
|
||||
.filter(Boolean)
|
||||
})
|
||||
}
|
||||
|
||||
function steerTurnOrder(messages: string[]): string[] {
|
||||
return messages.flatMap(message => {
|
||||
if (message.includes(ACTIVE_PROMPT)) return [ACTIVE_PROMPT]
|
||||
if (message.includes(STEER_PROMPT)) return [STEER_PROMPT]
|
||||
if (message.includes(MOCK_REPLY)) return [MOCK_REPLY]
|
||||
|
||||
return []
|
||||
})
|
||||
}
|
||||
|
||||
test.describe('queued prompt turn boundary', () => {
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeEach(async () => {
|
||||
fixture = await setupMockBackend({
|
||||
mockServer: { holdFirstStreamForPrompt: ACTIVE_PROMPT }
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterEach(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('submits a queued prompt only after the active turn completes', async () => {
|
||||
const { mock, page } = fixture!
|
||||
|
||||
await send(page, ACTIVE_PROMPT)
|
||||
await mock.waitForHeldStream()
|
||||
await queue(page, QUEUED_PROMPT)
|
||||
await expect(page.getByText('1 Queued')).toBeVisible()
|
||||
|
||||
// The mock keeps the active SSE stream open, so a queued prompt has no
|
||||
// completed-turn boundary that could legitimately drain it. Wait past the
|
||||
// queue retry interval and assert the provider saw only the active turn.
|
||||
await page.waitForTimeout(1_000)
|
||||
expect(mock.receivedPrompts.filter(prompt => prompt === QUEUED_PROMPT)).toHaveLength(0)
|
||||
await expect(page.locator('[data-slot="aui_thread-viewport"]')).not.toContainText(QUEUED_PROMPT)
|
||||
|
||||
mock.releaseHeldStream()
|
||||
await page.waitForFunction(
|
||||
expected => (document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
MOCK_REPLY,
|
||||
{ timeout: 60_000 }
|
||||
)
|
||||
await expect.poll(() => mock.receivedPrompts.filter(prompt => prompt === QUEUED_PROMPT)).toHaveLength(1)
|
||||
})
|
||||
|
||||
test('places a steer prompt before the reply it redirects', async () => {
|
||||
const { mock, page } = fixture!
|
||||
|
||||
await send(page, ACTIVE_PROMPT)
|
||||
await mock.waitForHeldStream()
|
||||
await steer(page, STEER_PROMPT)
|
||||
mock.releaseHeldStream()
|
||||
|
||||
await page.waitForFunction(
|
||||
expected => (document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
MOCK_REPLY,
|
||||
{ timeout: 60_000 }
|
||||
)
|
||||
|
||||
expect(steerTurnOrder(await transcriptMessageOrder(page))).toEqual([ACTIVE_PROMPT, STEER_PROMPT, MOCK_REPLY])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,239 @@
|
||||
import { type ChildProcessWithoutNullStreams, spawn } from 'node:child_process'
|
||||
import * as path from 'node:path'
|
||||
import { createInterface } from 'node:readline'
|
||||
|
||||
const DESKTOP_ROOT = path.resolve(import.meta.dirname, '..')
|
||||
const REPO_ROOT = path.resolve(DESKTOP_ROOT, '..', '..')
|
||||
const DEFAULT_TIMEOUT_MS = 60_000
|
||||
|
||||
interface JsonRpcError {
|
||||
code?: number
|
||||
message?: string
|
||||
}
|
||||
|
||||
interface JsonRpcFrame {
|
||||
error?: JsonRpcError
|
||||
id?: number
|
||||
method?: string
|
||||
params?: {
|
||||
payload?: unknown
|
||||
session_id?: string
|
||||
type?: string
|
||||
}
|
||||
result?: unknown
|
||||
}
|
||||
|
||||
interface CreatedSession {
|
||||
session_id: string
|
||||
stored_session_id: string
|
||||
}
|
||||
|
||||
export interface RealSessionTurn {
|
||||
/** Local image paths attached before the prompt, as the composer would. */
|
||||
images?: readonly string[]
|
||||
text: string
|
||||
}
|
||||
|
||||
export interface RealSessionSpec {
|
||||
/** Session label. The durable row stores no title, so clients fall back to
|
||||
* the preview (the first 60 characters of the first user message). */
|
||||
title: string
|
||||
/** Each item becomes one real user prompt followed by the mock provider's reply. */
|
||||
turns: readonly (RealSessionTurn | string)[]
|
||||
}
|
||||
|
||||
export interface RealSession {
|
||||
/** Runtime-only TUI session id, valid only while the builder process is alive. */
|
||||
runtimeId: string
|
||||
/** Durable SessionDB id that desktop resumes after the builder exits. */
|
||||
sessionId: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates durable desktop session history through the real TUI gateway and
|
||||
* AIAgent loop, using the E2E mock provider configured in `hermesHome`.
|
||||
*
|
||||
* This intentionally uses the shipped stdio JSON-RPC transport instead of
|
||||
* importing SessionDB or launching Electron. The desktop's WebSocket backend
|
||||
* dispatches the same `tui_gateway.server` methods.
|
||||
*/
|
||||
export class RealSessionBuilder {
|
||||
private readonly child: ChildProcessWithoutNullStreams
|
||||
private nextRequestId = 0
|
||||
private readonly pending = new Map<number, { reject: (reason: Error) => void; resolve: (value: unknown) => void }>()
|
||||
private readonly events: JsonRpcFrame[] = []
|
||||
private readonly eventWaiters: Array<{
|
||||
predicate: (frame: JsonRpcFrame) => boolean
|
||||
reject: (reason: Error) => void
|
||||
resolve: (frame: JsonRpcFrame) => void
|
||||
}> = []
|
||||
private readonly stderr: string[] = []
|
||||
private closed = false
|
||||
|
||||
private constructor(hermesHome: string) {
|
||||
this.child = spawn('uv', ['run', '--active', '--no-sync', 'python', '-m', 'tui_gateway.entry'], {
|
||||
cwd: REPO_ROOT,
|
||||
env: {
|
||||
...process.env,
|
||||
HERMES_HOME: hermesHome,
|
||||
PYTHONPATH: REPO_ROOT,
|
||||
},
|
||||
stdio: 'pipe',
|
||||
})
|
||||
|
||||
createInterface({ input: this.child.stdout }).on('line', line => this.handleLine(line))
|
||||
createInterface({ input: this.child.stderr }).on('line', line => {
|
||||
this.stderr.push(line)
|
||||
if (this.stderr.length > 80) this.stderr.shift()
|
||||
})
|
||||
this.child.once('error', error => this.failAll(new Error(`real-session gateway failed to start: ${error.message}`)))
|
||||
this.child.once('exit', (code, signal) => {
|
||||
if (!this.closed) {
|
||||
this.failAll(new Error(`real-session gateway exited unexpectedly (${signal ?? code ?? 'unknown'}):\n${this.stderr.join('\n')}`))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
static async start(hermesHome: string): Promise<RealSessionBuilder> {
|
||||
const builder = new RealSessionBuilder(hermesHome)
|
||||
await builder.waitForEvent(frame => frame.params?.type === 'gateway.ready')
|
||||
return builder
|
||||
}
|
||||
|
||||
async createSession(spec: RealSessionSpec): Promise<RealSession> {
|
||||
if (spec.turns.length === 0) {
|
||||
throw new Error('RealSessionBuilder requires at least one turn so the real agent creates a durable session row')
|
||||
}
|
||||
|
||||
const created = await this.request<CreatedSession>('session.create', {
|
||||
cols: 120,
|
||||
cwd: REPO_ROOT,
|
||||
source: 'desktop',
|
||||
title: spec.title,
|
||||
})
|
||||
const runtimeId = requireString(created, 'session_id')
|
||||
const sessionId = requireString(created, 'stored_session_id')
|
||||
|
||||
for (const turn of spec.turns) {
|
||||
const { images = [], text } = typeof turn === 'string' ? { text: turn } : turn
|
||||
|
||||
for (const image of images) {
|
||||
await this.request('image.attach', { session_id: runtimeId, path: image })
|
||||
}
|
||||
|
||||
const completion = this.waitForEvent(
|
||||
frame => frame.params?.type === 'message.complete' && frame.params.session_id === runtimeId,
|
||||
)
|
||||
await this.request('prompt.submit', { session_id: runtimeId, text })
|
||||
const frame = await completion
|
||||
const status = readString(frame.params?.payload, 'status')
|
||||
if (status !== 'complete') {
|
||||
throw new Error(`real session turn failed with status ${status ?? 'unknown'}: ${JSON.stringify(frame.params?.payload)}`)
|
||||
}
|
||||
}
|
||||
|
||||
await this.request('session.close', { session_id: runtimeId })
|
||||
return { runtimeId, sessionId }
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.closed) return
|
||||
this.closed = true
|
||||
this.child.stdin.end()
|
||||
await new Promise<void>(resolve => {
|
||||
const timeout = setTimeout(() => {
|
||||
this.child.kill('SIGTERM')
|
||||
resolve()
|
||||
}, 5_000)
|
||||
this.child.once('exit', () => {
|
||||
clearTimeout(timeout)
|
||||
resolve()
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
private request<T = unknown>(method: string, params: Record<string, unknown>): Promise<T> {
|
||||
const id = ++this.nextRequestId
|
||||
return this.withTimeout(new Promise<T>((resolve, reject) => {
|
||||
this.pending.set(id, { resolve: value => resolve(value as T), reject })
|
||||
this.child.stdin.write(`${JSON.stringify({ jsonrpc: '2.0', id, method, params })}\n`, error => {
|
||||
if (error) {
|
||||
this.pending.delete(id)
|
||||
reject(error)
|
||||
}
|
||||
})
|
||||
}), `request ${method}`)
|
||||
}
|
||||
|
||||
private waitForEvent(predicate: (frame: JsonRpcFrame) => boolean): Promise<JsonRpcFrame> {
|
||||
const index = this.events.findIndex(predicate)
|
||||
if (index >= 0) {
|
||||
return Promise.resolve(this.events.splice(index, 1)[0])
|
||||
}
|
||||
return this.withTimeout(new Promise<JsonRpcFrame>((resolve, reject) => {
|
||||
this.eventWaiters.push({ predicate, resolve, reject })
|
||||
}), 'gateway event')
|
||||
}
|
||||
|
||||
private handleLine(line: string): void {
|
||||
let frame: JsonRpcFrame
|
||||
try {
|
||||
frame = JSON.parse(line) as JsonRpcFrame
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
|
||||
if (typeof frame.id === 'number') {
|
||||
const pending = this.pending.get(frame.id)
|
||||
if (!pending) return
|
||||
this.pending.delete(frame.id)
|
||||
if (frame.error) {
|
||||
pending.reject(new Error(`JSON-RPC error ${frame.error.code ?? 'unknown'}: ${frame.error.message ?? 'unknown error'}`))
|
||||
} else {
|
||||
pending.resolve(frame.result)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (frame.method !== 'event') return
|
||||
const waiter = this.eventWaiters.find(candidate => candidate.predicate(frame))
|
||||
if (!waiter) {
|
||||
this.events.push(frame)
|
||||
return
|
||||
}
|
||||
this.eventWaiters.splice(this.eventWaiters.indexOf(waiter), 1)
|
||||
waiter.resolve(frame)
|
||||
}
|
||||
|
||||
private withTimeout<T>(promise: Promise<T>, operation: string): Promise<T> {
|
||||
return new Promise<T>((resolve, reject) => {
|
||||
const timer = setTimeout(() => reject(new Error(`Timed out after ${DEFAULT_TIMEOUT_MS / 1000}s waiting for ${operation}:\n${this.stderr.join('\n')}`)), DEFAULT_TIMEOUT_MS)
|
||||
promise.then(value => {
|
||||
clearTimeout(timer)
|
||||
resolve(value)
|
||||
}, error => {
|
||||
clearTimeout(timer)
|
||||
reject(error)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
private failAll(error: Error): void {
|
||||
for (const pending of this.pending.values()) pending.reject(error)
|
||||
this.pending.clear()
|
||||
for (const waiter of this.eventWaiters) waiter.reject(error)
|
||||
this.eventWaiters.length = 0
|
||||
}
|
||||
}
|
||||
|
||||
function readString(value: unknown, key: string): string | undefined {
|
||||
if (!value || typeof value !== 'object') return undefined
|
||||
const candidate = (value as Record<string, unknown>)[key]
|
||||
return typeof candidate === 'string' ? candidate : undefined
|
||||
}
|
||||
|
||||
function requireString(value: unknown, key: string): string {
|
||||
const candidate = readString(value, key)
|
||||
if (!candidate) throw new Error(`Gateway response omitted required ${key}: ${JSON.stringify(value)}`)
|
||||
return candidate
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
import { test, expect } from './test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('persistent terminal overlay follows the pane after split dragging', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
await page.keyboard.press('Control+`')
|
||||
await page.locator('[data-terminal-slot]').waitFor({ state: 'visible', timeout: 30_000 })
|
||||
await page.locator('[data-persistent-terminal] .xterm').waitFor({ state: 'visible', timeout: 30_000 })
|
||||
|
||||
const result = await page.evaluate(async () => {
|
||||
const slot = document.querySelector('[data-terminal-slot]')
|
||||
const overlay = document.querySelector('[data-persistent-terminal]')
|
||||
|
||||
if (!slot || !overlay) {
|
||||
return { drift: -1, moved: 0, target: false }
|
||||
}
|
||||
|
||||
const before = slot.getBoundingClientRect()
|
||||
const target = [...document.querySelectorAll<HTMLElement>('[role="separator"]')]
|
||||
.map(element => {
|
||||
const box = element.getBoundingClientRect()
|
||||
const horizontal = box.width > box.height
|
||||
const center = horizontal
|
||||
? (box.top + box.bottom) / 2
|
||||
: (box.left + box.right) / 2
|
||||
const sides = horizontal
|
||||
? [before.top, before.bottom]
|
||||
: [before.left, before.right]
|
||||
|
||||
return {
|
||||
element,
|
||||
box,
|
||||
horizontal,
|
||||
score: Math.min(...sides.map(side => Math.abs(center - side))),
|
||||
}
|
||||
})
|
||||
.filter(item => item.box.width > 0 && item.box.height > 0)
|
||||
.sort((a, b) => a.score - b.score)[0]
|
||||
|
||||
if (!target) {
|
||||
return { drift: -1, moved: 0, target: false }
|
||||
}
|
||||
|
||||
const x = target.box.left + target.box.width / 2
|
||||
const y0 = target.box.top + target.box.height / 2
|
||||
const nearestSide = target.horizontal
|
||||
? Math.abs(y0 - before.top) < Math.abs(y0 - before.bottom)
|
||||
? 'top'
|
||||
: 'bottom'
|
||||
: Math.abs(x - before.left) < Math.abs(x - before.right)
|
||||
? 'left'
|
||||
: 'right'
|
||||
const deltaX = nearestSide === 'left' ? -1 : nearestSide === 'right' ? 1 : 0
|
||||
const deltaY = nearestSide === 'top' ? -1 : nearestSide === 'bottom' ? 1 : 0
|
||||
let currentX = x
|
||||
let y = y0
|
||||
const pointer = {
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
pointerId: 71,
|
||||
pointerType: 'mouse',
|
||||
isPrimary: true,
|
||||
button: 0,
|
||||
buttons: 1,
|
||||
}
|
||||
|
||||
target.element.dispatchEvent(
|
||||
new PointerEvent('pointerdown', { ...pointer, clientX: x, clientY: y }),
|
||||
)
|
||||
|
||||
for (let index = 0; index < 24; index += 1) {
|
||||
currentX += deltaX
|
||||
y += deltaY
|
||||
window.dispatchEvent(
|
||||
new PointerEvent('pointermove', {
|
||||
...pointer,
|
||||
clientX: currentX,
|
||||
clientY: y,
|
||||
}),
|
||||
)
|
||||
await new Promise<void>(resolve => requestAnimationFrame(() => resolve()))
|
||||
}
|
||||
|
||||
window.dispatchEvent(
|
||||
new PointerEvent('pointerup', {
|
||||
...pointer,
|
||||
buttons: 0,
|
||||
clientX: currentX,
|
||||
clientY: y,
|
||||
}),
|
||||
)
|
||||
await new Promise<void>(resolve => setTimeout(resolve, 350))
|
||||
await new Promise<void>(resolve =>
|
||||
requestAnimationFrame(() => requestAnimationFrame(() => resolve())),
|
||||
)
|
||||
|
||||
const next = slot.getBoundingClientRect()
|
||||
const fixed = overlay.getBoundingClientRect()
|
||||
|
||||
return {
|
||||
drift: Math.max(
|
||||
Math.abs(next.top - fixed.top),
|
||||
Math.abs(next.left - fixed.left),
|
||||
Math.abs(next.width - fixed.width),
|
||||
Math.abs(next.height - fixed.height),
|
||||
),
|
||||
moved: Math.max(
|
||||
Math.abs(next.top - before.top),
|
||||
Math.abs(next.left - before.left),
|
||||
Math.abs(next.width - before.width),
|
||||
Math.abs(next.height - before.height),
|
||||
),
|
||||
target: true,
|
||||
}
|
||||
})
|
||||
|
||||
expect(result.target).toBe(true)
|
||||
expect(result.moved).toBeGreaterThan(10)
|
||||
expect(result.drift).toBeLessThanOrEqual(1)
|
||||
})
|
||||
@@ -0,0 +1,162 @@
|
||||
/**
|
||||
* E2E coverage for session compression, which rotates a live backend session.
|
||||
*/
|
||||
|
||||
import { expect, test, type Page } from '@playwright/test'
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { MOCK_REPLY, receivedUserTexts, restartMockServer } from './mock-server'
|
||||
|
||||
async function send(page: Page, text: string, delay = 15): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.click()
|
||||
await composer.type(text, { delay })
|
||||
await page.keyboard.press('Enter')
|
||||
}
|
||||
|
||||
async function pasteAndSend(page: Page, text: string): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.click()
|
||||
await page.keyboard.insertText(text)
|
||||
await page.keyboard.press('Enter')
|
||||
}
|
||||
|
||||
async function waitForTranscript(page: Page, text: string, timeout = 90_000): Promise<void> {
|
||||
await page.waitForFunction(
|
||||
expected => document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent?.includes(expected) ?? false,
|
||||
text,
|
||||
{ timeout }
|
||||
)
|
||||
}
|
||||
|
||||
test.describe('session compression', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
})
|
||||
|
||||
test('compresses an existing session and accepts a follow-up turn on its continuation', async () => {
|
||||
const { page } = fixture
|
||||
const reply = 'Hello from the mock inference server! The full boot chain is working.'
|
||||
|
||||
// Three completed exchanges leave a compressible middle after the
|
||||
// compressor's protected head/tail boundaries.
|
||||
await send(page, 'E2E_COMPRESSION_FIRST')
|
||||
await waitForTranscript(page, reply)
|
||||
await send(page, 'E2E_COMPRESSION_SECOND')
|
||||
await expect.poll(() => receivedUserTexts().filter(text => text === 'E2E_COMPRESSION_SECOND').length).toBe(1)
|
||||
await send(page, 'E2E_COMPRESSION_THIRD')
|
||||
await expect.poll(() => receivedUserTexts().filter(text => text === 'E2E_COMPRESSION_THIRD').length).toBe(1)
|
||||
|
||||
// The mock receiving the third prompt does not mean the TURN is over —
|
||||
// /compress on a busy session errors with "session busy — /interrupt the
|
||||
// current turn before /compress". Wait for the third reply to render and
|
||||
// for the composer to leave its busy state (no Stop affordance) first.
|
||||
await page.waitForFunction(
|
||||
expected =>
|
||||
((document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').split(expected).length - 1) >= 3,
|
||||
reply,
|
||||
{ timeout: 90_000 }
|
||||
)
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator('[data-slot="composer-root"] button[aria-label="Stop"]').count(),
|
||||
{ timeout: 30_000, message: 'turn should settle before /compress' }
|
||||
)
|
||||
.toBe(0)
|
||||
|
||||
// This test covers compression and continuation, not slash completion.
|
||||
// Insert the complete command atomically and click Send so an async
|
||||
// completion response cannot consume Enter as a picker acceptance.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.click()
|
||||
await page.keyboard.insertText('/compress preserve the three test turns')
|
||||
await expect.poll(() => composer.textContent()).toContain('preserve the three test turns')
|
||||
await page.getByRole('button', { name: 'Send', exact: true }).click()
|
||||
await expect
|
||||
.poll(() => page.locator('[data-slot="aui_thread-viewport"]').textContent(), { timeout: 90_000 })
|
||||
.toMatch(/Compressed|No changes from compression/)
|
||||
|
||||
// Compression rotates the agent's live session id. A post-compression
|
||||
// ordinary turn proves the desktop's runtime binding followed that child.
|
||||
await send(page, 'E2E_COMPRESSION_FOLLOW_UP')
|
||||
await expect.poll(() => receivedUserTexts().filter(text => text === 'E2E_COMPRESSION_FOLLOW_UP').length).toBe(1)
|
||||
await waitForTranscript(page, reply)
|
||||
await page.screenshot({ path: 'test-results/session-compression-continuation.png' })
|
||||
})
|
||||
})
|
||||
|
||||
test.describe('session compression in progress', () => {
|
||||
let fixture: MockBackendFixture
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend({
|
||||
modelContextLength: 64_000,
|
||||
extraConfig: `compression:
|
||||
threshold_tokens: 22000
|
||||
protect_first_n: 0
|
||||
protect_last_n: 1
|
||||
auxiliary:
|
||||
title_generation:
|
||||
enabled: false
|
||||
compression:
|
||||
provider: custom
|
||||
model: mock-model`,
|
||||
mockServer: {
|
||||
holdFirstCompletionContaining: 'You are a summarization agent creating a context checkpoint.'
|
||||
}
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
})
|
||||
|
||||
test('queues an Enter-submitted draft instead of steering while compaction is active', async ({}, testInfo) => {
|
||||
const { page } = fixture
|
||||
const queued = 'E2E_QUEUED_DURING_COMPACTION'
|
||||
|
||||
// A normal message crosses the tiny configured context budget. The mock
|
||||
// blocks only the resulting summary request, so these assertions run
|
||||
// during automatic compaction rather than a slash-command path.
|
||||
// The payload must cross threshold_tokens (22k) on its OWN weight
|
||||
// (~12k tokens) on top of the system prompt. Do not shrink it: at
|
||||
// repeat(500) the trigger only worked because the ambient system prompt
|
||||
// (skills index + tool schemas) happened to carry it over the line, and
|
||||
// a 160-token skills-index cleanup on main broke the test for a day.
|
||||
await pasteAndSend(page, 'E2E_COMPACTION_HISTORY_ONE '.repeat(5))
|
||||
await waitForTranscript(page, MOCK_REPLY)
|
||||
await pasteAndSend(page, 'E2E_COMPACTION_HISTORY_TWO '.repeat(5))
|
||||
await waitForTranscript(page, MOCK_REPLY)
|
||||
await pasteAndSend(page, 'E2E_TRIGGER_AUTOMATIC_COMPACTION '.repeat(1500))
|
||||
await fixture.mock.waitForHeldCompletion()
|
||||
await expect(page.getByRole('status', { name: 'Summarizing thread' }).last()).toBeVisible()
|
||||
|
||||
const primary = page.locator('[data-slot="composer-root"] button[type="submit"]')
|
||||
// Since "running is not busy" (3bc52fb9df) an empty composer mid-turn
|
||||
// shows Stop — the Queue affordance appears once a payload is typed, and
|
||||
// the Enter path below still queues instead of steering while compaction
|
||||
// holds the turn.
|
||||
await expect(primary).toHaveAttribute('aria-label', 'Stop')
|
||||
|
||||
await send(page, queued)
|
||||
await expect(page.getByText('1 Queued')).toBeVisible()
|
||||
expect(fixture.mock.heldCompletionCount()).toBe(1)
|
||||
expect(receivedUserTexts()).not.toContain(queued)
|
||||
await page.screenshot({ path: testInfo.outputPath('queued-during-compaction.png') })
|
||||
|
||||
fixture.mock.releaseHeldStream()
|
||||
await expect.poll(() => receivedUserTexts().filter(text => text === queued).length).toBe(1)
|
||||
expect(fixture.mock.heldCompletionCount()).toBe(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,321 @@
|
||||
/**
|
||||
* E2E tests for desktop sidebar states — background processes, subagents,
|
||||
* and session dot transitions.
|
||||
*
|
||||
* The mock server returns scripted tool_calls that the agent executes for
|
||||
* real (trivial commands + real subagent delegations). The tests assert the
|
||||
* sidebar states driven by real gateway events.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, test, type Page } from '@playwright/test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
import {
|
||||
createBackgroundReleaseHandle,
|
||||
restartMockServer,
|
||||
SIDEBAR_CROSS_TEXTS,
|
||||
SIDEBAR_TEXTS,
|
||||
} from './mock-server'
|
||||
|
||||
/** Background-running dot aria-label (from i18n en.ts). */
|
||||
const BG_DOT_LABEL = 'Background task running'
|
||||
/** Foreground turn-running dot aria-label. */
|
||||
const SESSION_RUNNING_DOT_LABEL = 'Session running'
|
||||
/** Finished-unread dot aria-label. */
|
||||
const UNREAD_DOT_LABEL = 'Finished — unread'
|
||||
|
||||
/**
|
||||
* The auto-title auxiliary call hits the SAME mock provider as the chat turn,
|
||||
* and its request carries the user's message — trigger keyword included. The
|
||||
* mock's trigger matching is text-based, so the title call consumes a script
|
||||
* index: the real chat turn then gets turn 2 (final answer, NO tool calls),
|
||||
* the background process is never spawned, and the bg dot never appears.
|
||||
* Whether that happens depends on which request lands first — the CI flake
|
||||
* these specs had. Disable auto-title so script indices line up with real
|
||||
* chat turns (same fix as interim-messages.spec.ts).
|
||||
*/
|
||||
const DISABLE_AUTO_TITLE = 'auxiliary:\n title_generation:\n enabled: false'
|
||||
|
||||
/** Send a message and wait for the final response to appear. */
|
||||
async function sendMessageAndWait(
|
||||
page: Page,
|
||||
trigger: string,
|
||||
finalText: string,
|
||||
timeout = 90_000,
|
||||
): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await composer.click()
|
||||
await composer.type(trigger, { delay: 20 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
await page.waitForFunction(
|
||||
() => (document.body.textContent ?? '').includes('E2E_'),
|
||||
undefined,
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
|
||||
await page.waitForFunction(
|
||||
(text) => (document.body.textContent ?? '').includes(text),
|
||||
finalText,
|
||||
{ timeout },
|
||||
)
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Test 1: background process + subagent appear in sidebar during turn
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test.describe('sidebar states — background process and subagent', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({ extraConfig: DISABLE_AUTO_TITLE })
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
})
|
||||
|
||||
test('background process dot appears and disappears, subagent runs, final answer visible', async () => {
|
||||
const page = fixture.page
|
||||
|
||||
await sendMessageAndWait(page, 'E2E_SIDEBAR_TRIGGER', SIDEBAR_TEXTS.finalText)
|
||||
|
||||
// The background process (sleep 1) should have shown a "Background task
|
||||
// running" dot at some point during the turn. We try to catch it; if
|
||||
// the process was too fast, that's OK — the real assertion is that the
|
||||
// final answer appeared and the dot is gone afterward.
|
||||
try {
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 15_000, message: 'background dot should appear' },
|
||||
)
|
||||
.toBeGreaterThan(0)
|
||||
} catch {
|
||||
// sleep 1 may have finished before we polled — not a failure.
|
||||
}
|
||||
|
||||
// After the turn completes and auto-dismiss fires, the background dot
|
||||
// should be gone.
|
||||
await page.waitForTimeout(8000)
|
||||
const bgCount = await page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count()
|
||||
expect(bgCount, 'background dot should be gone after auto-dismiss').toBe(0)
|
||||
|
||||
// Evidence: capture the final state — no background dot, final answer visible.
|
||||
await page.screenshot({ path: 'test-results/bg-dot-gone-after-dismiss.png' })
|
||||
|
||||
// The final answer text must be in the transcript.
|
||||
const viewportText = await page
|
||||
.locator('[data-slot="aui_thread-viewport"]')
|
||||
.textContent()
|
||||
expect(viewportText).toContain(SIDEBAR_TEXTS.finalText)
|
||||
})
|
||||
})
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Test 2: subagent running shows background dot too (longer bg process)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test.describe('sidebar states — subagent and background dot coexist', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
// Hold the background process open until the test releases it. Without the
|
||||
// sentinel the process is a bare `sleep 5` racing the agent turn (two model
|
||||
// trips + a real subagent spawn): on a loaded runner the turn outlives the
|
||||
// sleep, the process is reaped mid-turn, and the dot never appears at all —
|
||||
// the CI flake this spec had.
|
||||
const bgRelease = createBackgroundReleaseHandle()
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({
|
||||
extraConfig: DISABLE_AUTO_TITLE,
|
||||
mockServer: { backgroundReleasePath: bgRelease.path },
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
bgRelease.release()
|
||||
await fixture?.cleanup()
|
||||
bgRelease.cleanup()
|
||||
})
|
||||
|
||||
test('background dot visible while subagent runs', async () => {
|
||||
const page = fixture.page
|
||||
|
||||
// Start the turn — a held background process plus a real subagent.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await composer.click()
|
||||
await composer.type('E2E_SIDEBAR_CROSS', { delay: 20 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
// Wait for the user's message to appear.
|
||||
await page.waitForFunction(
|
||||
() => (document.body.textContent ?? '').includes('E2E_SIDEBAR_CROSS'),
|
||||
undefined,
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
|
||||
// While the turn is busy the dot-state priority paints the session as
|
||||
// "working" ('Session running') — that claim OUTRANKS 'background', so
|
||||
// polling for the bg dot mid-turn races the turn length against the poll
|
||||
// budget. Wait for the turn to END (final text + running dot cleared),
|
||||
// then assert the background dot as a stable, sentinel-held state.
|
||||
await page.waitForFunction(
|
||||
(text) => (document.body.textContent ?? '').includes(text),
|
||||
SIDEBAR_CROSS_TEXTS.finalText,
|
||||
{ timeout: 90_000 },
|
||||
)
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${SESSION_RUNNING_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'session running dot should disappear after turn completes' },
|
||||
)
|
||||
.toBe(0)
|
||||
|
||||
// The background process is held open by the sentinel, so the bg dot is
|
||||
// a stable state — poll only to absorb the event-driven flip landing a
|
||||
// tick after the running dot clears.
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'background dot should be visible after turn completes' },
|
||||
)
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
// Evidence: the background dot is visible while the process runs.
|
||||
await page.screenshot({ path: 'test-results/bg-dot-while-subagent-runs.png' })
|
||||
|
||||
// Release the process; the dot should clear on the completion event —
|
||||
// event-driven, not a fixed sleep.
|
||||
bgRelease.release()
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'background dot should be gone after process exits' },
|
||||
)
|
||||
.toBe(0)
|
||||
})
|
||||
})
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Test 3: cross-session — dot updates when viewing a different session
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test.describe('sidebar states — cross-session dot transition', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
// Keeps the background process alive until this test releases it, so the
|
||||
// "still running after the turn finished" state can't expire on its own.
|
||||
const bgRelease = createBackgroundReleaseHandle()
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({
|
||||
extraConfig: DISABLE_AUTO_TITLE,
|
||||
mockServer: { backgroundReleasePath: bgRelease.path },
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
// Release first so the process exits even if the test failed early,
|
||||
// then drop the sentinel file.
|
||||
bgRelease.release()
|
||||
await fixture?.cleanup()
|
||||
bgRelease.cleanup()
|
||||
})
|
||||
|
||||
test('background dot transitions to finished when viewing another session', async () => {
|
||||
const page = fixture.page
|
||||
|
||||
// Start a turn whose background process runs until we release it.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await composer.click()
|
||||
await composer.type('E2E_SIDEBAR_CROSS', { delay: 20 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
// While the turn is busy the dot-state priority paints the session as
|
||||
// "working" ('Session running') — that claim OUTRANKS 'background', so
|
||||
// polling for the bg dot mid-turn races the turn length (two model trips
|
||||
// + a real subagent spawn) against the poll budget: the CI flake this
|
||||
// spec had. Wait for the turn to END first, then assert the bg dot as a
|
||||
// stable, sentinel-held state.
|
||||
//
|
||||
// The final answer text streams before message.complete, so text visibility
|
||||
// alone is not a completion barrier. Wait for the foreground-running state
|
||||
// to clear before asserting the background-process state.
|
||||
await page.waitForFunction(
|
||||
(text) => (document.body.textContent ?? '').includes(text),
|
||||
SIDEBAR_CROSS_TEXTS.finalText,
|
||||
{ timeout: 90_000 },
|
||||
)
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${SESSION_RUNNING_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'session running dot should disappear after turn completes' },
|
||||
)
|
||||
.toBe(0)
|
||||
|
||||
// The background dot must be visible now: the turn is done but the
|
||||
// process is held open by the sentinel, so this is a stable state rather
|
||||
// than a window we have to catch in time. Poll to absorb the event-driven
|
||||
// flip landing a tick after the running dot clears.
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'background dot should be visible after turn completes' },
|
||||
)
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
// Evidence: bg dot visible on session A while its turn is done but the
|
||||
// background process hasn't exited yet.
|
||||
await page.screenshot({ path: 'test-results/cross-session-bg-dot-before-switch.png' })
|
||||
|
||||
// Create a new session (click "New session" button).
|
||||
await page.locator('button:has-text("New session")').first().click()
|
||||
await page.waitForTimeout(2000)
|
||||
|
||||
// Now let the background process finish. The session A dot should
|
||||
// transition away from "background running".
|
||||
bgRelease.release()
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'background dot should disappear after process finishes' },
|
||||
)
|
||||
.toBe(0)
|
||||
|
||||
// The original session should show a "finished unread" indicator (green dot)
|
||||
// since its turn completed while we were in a different session. This is an
|
||||
// event-driven transition, so wait for it instead of sampling the DOM right
|
||||
// after the running dot disappears.
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${UNREAD_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'original session should show finished-unread dot' },
|
||||
)
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
// Evidence: the green "finished unread" dot on the original session after
|
||||
// switching to a new session — the cross-session dot transition.
|
||||
await page.screenshot({ path: 'test-results/cross-session-unread-dot-after-switch.png' })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* Regression coverage for #69578: harmless route-token churn during a send
|
||||
* must not make the desktop silently drop the prompt before prompt.submit.
|
||||
*/
|
||||
|
||||
import { test, expect } from './test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
|
||||
const PROMPT = 'E2E route token drift must still submit this prompt.'
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture!, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('submits while same-chat search tokens churn during new-session creation', async ({}, testInfo) => {
|
||||
const { page, mock } = fixture!
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
|
||||
await composer.click()
|
||||
await composer.type(PROMPT, { delay: 10 })
|
||||
|
||||
// The submit pipeline snapshots the route synchronously, then awaits session
|
||||
// creation. Keep changing only the query string of whichever chat route is
|
||||
// current. Before #69578, comparing the raw route token treated this as a
|
||||
// user chat switch and aborted before prompt.submit.
|
||||
await page.evaluate(() => {
|
||||
let revision = 0
|
||||
const interval = window.setInterval(() => {
|
||||
const pathname = window.location.hash.slice(1).split(/[?#]/, 1)[0] || '/new'
|
||||
window.location.hash = `${pathname}?e2e-route-churn=${revision++}`
|
||||
}, 1)
|
||||
|
||||
;(window as typeof window & { __e2eStopRouteChurn?: () => void }).__e2eStopRouteChurn = () => {
|
||||
window.clearInterval(interval)
|
||||
}
|
||||
})
|
||||
|
||||
try {
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
await expect
|
||||
.poll(() => mock.receivedPrompts.includes(PROMPT), { timeout: 60_000 })
|
||||
.toBe(true)
|
||||
|
||||
await page.waitForFunction(
|
||||
prompt => document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent?.includes(prompt) ?? false,
|
||||
PROMPT,
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
await page.waitForFunction(
|
||||
() => document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent?.includes('mock inference server') ?? false,
|
||||
undefined,
|
||||
{ timeout: 60_000 },
|
||||
)
|
||||
} finally {
|
||||
await page.evaluate(() => {
|
||||
;(window as typeof window & { __e2eStopRouteChurn?: () => void }).__e2eStopRouteChurn?.()
|
||||
})
|
||||
}
|
||||
|
||||
await page.screenshot({ path: testInfo.outputPath('same-chat-route-churn-submitted.png') })
|
||||
})
|
||||
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* Regression coverage for returning to a working session as its task panel
|
||||
* expands. The transcript must reconcile to the composer's full measured
|
||||
* height without needing a manual scroll to repair the position.
|
||||
*/
|
||||
|
||||
import { expect, test, type Page } from './test'
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { TASK_PANEL_RESUME_TRIGGER } from './mock-server'
|
||||
|
||||
const SURFACE = '[data-composer-target]:visible'
|
||||
const PROMPT = `${TASK_PANEL_RESUME_TRIGGER}: keep the task panel expanded while this session is reopened.`
|
||||
|
||||
function activeSurface(page: Page) {
|
||||
return page.locator(SURFACE).last()
|
||||
}
|
||||
|
||||
async function send(page: Page, text: string): Promise<void> {
|
||||
const composer = activeSurface(page).locator('[contenteditable="true"]').first()
|
||||
|
||||
await composer.waitFor({ state: 'visible', timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.type(text, { delay: 5 })
|
||||
await page.keyboard.press('Enter')
|
||||
}
|
||||
|
||||
async function openFreshDraft(page: Page): Promise<void> {
|
||||
await page.locator('[data-slot="sidebar"] button[aria-label="New session"]').first().click()
|
||||
await expect(activeSurface(page).locator('[data-slot="aui_thread-viewport"]')).not.toContainText(PROMPT)
|
||||
await page.waitForTimeout(1_000)
|
||||
}
|
||||
|
||||
async function reopenWorkingSession(page: Page): Promise<void> {
|
||||
const sidebar = page.locator('[data-slot="sidebar"]')
|
||||
const row = sidebar.getByRole('button', { name: /^(?:Session running|Needs your input|Working)\b/ }).first()
|
||||
|
||||
await row.waitFor({ state: 'visible', timeout: 30_000 })
|
||||
await row.click()
|
||||
await expect(activeSurface(page).locator('[data-slot="aui_thread-viewport"]')).toContainText(
|
||||
'Task-panel clearance line 24',
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
}
|
||||
|
||||
interface ClearanceMetrics {
|
||||
composerHeight: number
|
||||
distanceFromBottom: number
|
||||
latestMessageBottom: number
|
||||
statusPanelTop: number
|
||||
viewportHeight: number
|
||||
}
|
||||
|
||||
async function clearanceMetrics(page: Page): Promise<ClearanceMetrics> {
|
||||
return activeSurface(page).evaluate(surface => {
|
||||
const chatSurface = surface.closest<HTMLElement>('[data-chat-surface]')!
|
||||
const viewport = surface.querySelector<HTMLElement>('[data-slot="aui_thread-viewport"]')!
|
||||
const latest = Array.from(surface.querySelectorAll<HTMLElement>('[data-role="assistant"]')).at(-1)!
|
||||
const status = surface.querySelector<HTMLElement>('[data-slot="composer-status-stack"]')!
|
||||
const styles = getComputedStyle(chatSurface)
|
||||
|
||||
return {
|
||||
composerHeight: Number.parseFloat(styles.getPropertyValue('--composer-measured-height')),
|
||||
distanceFromBottom: viewport.scrollHeight - viewport.clientHeight - viewport.scrollTop,
|
||||
latestMessageBottom: latest.getBoundingClientRect().bottom,
|
||||
statusPanelTop: status.getBoundingClientRect().top,
|
||||
viewportHeight: viewport.clientHeight,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
test.describe('working-session task-panel clearance', () => {
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeEach(async () => {
|
||||
fixture = await setupMockBackend({
|
||||
mockServer: { holdFirstCompletionContaining: TASK_PANEL_RESUME_TRIGGER },
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterEach(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('window focus reanchors a working session above the expanded task panel', async ({}, testInfo) => {
|
||||
const page = fixture!.page
|
||||
|
||||
await send(page, PROMPT)
|
||||
await fixture!.mock.waitForHeldCompletion()
|
||||
await openFreshDraft(page)
|
||||
|
||||
// Re-open while the long response is still streaming. Its todo call lands
|
||||
// afterward, so the already-visible composer grows only after the initial
|
||||
// session-load scroll settle has finished.
|
||||
fixture!.mock.releaseHeldStream()
|
||||
await page.waitForTimeout(1_000)
|
||||
await reopenWorkingSession(page)
|
||||
await expect(activeSurface(page).getByText('Tasks 1/5')).toBeVisible({ timeout: 30_000 })
|
||||
|
||||
// Reproduce the stale geometry at the foreground boundary. Active turns
|
||||
// disable Chromium's background throttling, so visibility can stay `visible`
|
||||
// and window focus is the only foreground edge that can repair it.
|
||||
await page.waitForTimeout(750)
|
||||
const staleState = await activeSurface(page)
|
||||
.locator('[data-slot="aui_thread-viewport"]')
|
||||
.evaluate(viewport => {
|
||||
// Grow scrollHeight before the observed thread-content node. This
|
||||
// shifts the transcript behind the dock without resizing the observed
|
||||
// node or synthesizing a user scroll (which must escape the lock).
|
||||
const staleClearance = document.createElement('div')
|
||||
staleClearance.style.height = '160px'
|
||||
staleClearance.setAttribute('aria-hidden', 'true')
|
||||
viewport.prepend(staleClearance)
|
||||
|
||||
const distance = viewport.scrollHeight - viewport.clientHeight - viewport.scrollTop
|
||||
const surface = viewport.closest<HTMLElement>('[data-composer-target]')!
|
||||
const latest = Array.from(surface.querySelectorAll<HTMLElement>('[data-role="assistant"]')).at(-1)!
|
||||
const status = surface.querySelector<HTMLElement>('[data-slot="composer-status-stack"]')!
|
||||
|
||||
window.dispatchEvent(new Event('focus'))
|
||||
|
||||
return {
|
||||
distance,
|
||||
following: viewport.dataset.following,
|
||||
latestMessageBottom: latest.getBoundingClientRect().bottom,
|
||||
statusPanelTop: status.getBoundingClientRect().top,
|
||||
visibility: document.visibilityState,
|
||||
}
|
||||
})
|
||||
|
||||
expect(staleState.visibility, JSON.stringify(staleState)).toBe('visible')
|
||||
expect(staleState.following, JSON.stringify(staleState)).toBe('true')
|
||||
expect(staleState.distance).toBeGreaterThan(100)
|
||||
expect(staleState.latestMessageBottom, JSON.stringify(staleState)).toBeGreaterThan(staleState.statusPanelTop)
|
||||
await page.waitForTimeout(1_000)
|
||||
const metrics = await clearanceMetrics(page)
|
||||
await page.screenshot({ path: testInfo.outputPath('task-panel-after-resume.png') })
|
||||
|
||||
expect(metrics.composerHeight, JSON.stringify(metrics)).toBeGreaterThanOrEqual(190)
|
||||
expect(metrics.distanceFromBottom, JSON.stringify(metrics)).toBeLessThan(staleState.distance / 2)
|
||||
expect(metrics.latestMessageBottom, JSON.stringify(metrics)).toBeLessThanOrEqual(metrics.statusPanelTop)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,166 @@
|
||||
/**
|
||||
* Extended Playwright test fixture that auto-fails any test if an error
|
||||
* banner (notification toast with role="alert") appears in the DOM.
|
||||
*
|
||||
* The desktop app surfaces errors as `[data-slot="alert"][role="alert"]`
|
||||
* elements (see components/notifications.tsx). When one appears during a
|
||||
* test, it means something went wrong (resume failed, boot error, etc.)
|
||||
* — the test should fail with the error message, not silently pass while
|
||||
* an error toast is visible on screen.
|
||||
*
|
||||
* Usage: import { test, expect } from './test' instead of
|
||||
* '@playwright/test'. The guard is auto-installed on every page — no
|
||||
* per-spec setup needed.
|
||||
*/
|
||||
|
||||
import { test as base, expect, type Page, type ElectronApplication, _electron } from '@playwright/test'
|
||||
|
||||
// Track error messages per test so afterEach can assert + report.
|
||||
const seenErrors: string[] = []
|
||||
let activePage: Page | null = null
|
||||
// When true, the afterEach guard skips the error-banner check.
|
||||
// Set by tests that deliberately trigger error states (e.g. boot-failure).
|
||||
let errorBannersAllowed = false
|
||||
|
||||
/**
|
||||
* Opt out of the error-banner guard for the current test. Call in
|
||||
* test.beforeEach or at the top of a test body when error banners are
|
||||
* expected (e.g. boot-failure tests that deliberately trigger errors).
|
||||
*/
|
||||
export function allowErrorBanners(): void {
|
||||
errorBannersAllowed = true
|
||||
}
|
||||
|
||||
/**
|
||||
* Install the error-banner guard on a page. Watches for `[role="alert"]`
|
||||
* elements appearing in the DOM. When one is found, records its text
|
||||
* content for the afterEach assertion.
|
||||
*
|
||||
* Exported so e2e fixture functions (which create pages via _electron.launch)
|
||||
* can install the guard on their custom pages — the default Playwright `page`
|
||||
* fixture override only catches pages created by Playwright itself, not
|
||||
* pages created by the test's own Electron launch.
|
||||
*/
|
||||
export function installErrorBannerGuard(page: Page): void {
|
||||
activePage = page
|
||||
|
||||
// Clear any errors from a previous test when a new page is created.
|
||||
seenErrors.length = 0
|
||||
|
||||
// Use a MutationObserver to catch error banners as they appear.
|
||||
// We inject this via addInitScript so it runs before any app code.
|
||||
page.addInitScript(() => {
|
||||
const seen: string[] = []
|
||||
;(window as unknown as { __ERROR_BANNER_GUARD__?: string[] }).__ERROR_BANNER_GUARD__ = seen
|
||||
|
||||
const observer = new MutationObserver(() => {
|
||||
const alerts = document.querySelectorAll('[role="alert"]')
|
||||
|
||||
for (const alert of alerts) {
|
||||
const text = (alert.textContent ?? '').trim()
|
||||
|
||||
if (text && !seen.includes(text)) {
|
||||
seen.push(text)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Start observing once the DOM is ready.
|
||||
if (document.body) {
|
||||
observer.observe(document.body, { childList: true, subtree: true })
|
||||
} else {
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
observer.observe(document.body, { childList: true, subtree: true })
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
// Also poll via evaluate — MutationObserver via addInitScript can miss
|
||||
// elements that appear during the Electron renderer's initial mount
|
||||
// (before the observer is installed). A periodic poll catches those.
|
||||
page.on('console', () => {
|
||||
// Console messages are not errors — but we keep the listener to
|
||||
// ensure the page context is active for our evaluate calls.
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Check for error banners that appeared during the test. Called in
|
||||
* afterEach via the custom fixture below. Also exported so specs that
|
||||
* manage their own page lifecycle can call it directly.
|
||||
*/
|
||||
export async function collectErrorBanners(page: Page | null): Promise<string[]> {
|
||||
if (!page) {
|
||||
return []
|
||||
}
|
||||
|
||||
try {
|
||||
// Read errors collected by the MutationObserver in the page context.
|
||||
const pageErrors = await page.evaluate(() => {
|
||||
const w = window as unknown as { __ERROR_BANNER_GUARD__?: string[] }
|
||||
|
||||
return [...(w.__ERROR_BANNER_GUARD__ ?? [])]
|
||||
})
|
||||
|
||||
// Also do a final DOM scan for any alert elements still visible.
|
||||
const domAlerts = await page
|
||||
.locator('[role="alert"]')
|
||||
.allTextContents()
|
||||
.catch(() => [] as string[])
|
||||
|
||||
const all = [...new Set([...pageErrors, ...domAlerts.map(t => t.trim()).filter(Boolean)])]
|
||||
seenErrors.push(...all)
|
||||
|
||||
return [...new Set(seenErrors)]
|
||||
} catch {
|
||||
// Page might be closed — return whatever we have.
|
||||
return [...new Set(seenErrors)]
|
||||
}
|
||||
}
|
||||
|
||||
// Extended test fixture: wraps the default page with the error guard.
|
||||
export const test = base.extend({
|
||||
// Override the page fixture to auto-install the guard.
|
||||
page: async ({ page }, use) => {
|
||||
installErrorBannerGuard(page)
|
||||
await use(page)
|
||||
},
|
||||
})
|
||||
|
||||
// afterEach: fail the test if any error banners appeared.
|
||||
// Always fires — even if the test already failed for another reason.
|
||||
// An error banner often IS the root cause (e.g. "resume failed" from a
|
||||
// backend bug), and suppressing it when the test also fails on an
|
||||
// assertion hides the real problem.
|
||||
//
|
||||
// Uses `activePage` (set by installErrorBannerGuard) instead of the
|
||||
// default `page` fixture — Electron tests create their own page via
|
||||
// app.firstWindow(), so the default `page` fixture is undefined.
|
||||
base.afterEach(async ({}, testInfo) => {
|
||||
const wasAllowed = errorBannersAllowed
|
||||
// Reset for the next test.
|
||||
errorBannersAllowed = false
|
||||
|
||||
if (wasAllowed) {
|
||||
// Test opted out — clear any collected errors without asserting.
|
||||
seenErrors.length = 0
|
||||
return
|
||||
}
|
||||
|
||||
const errors = await collectErrorBanners(activePage)
|
||||
|
||||
if (errors.length > 0) {
|
||||
throw new Error(
|
||||
`Error banner(s) appeared during test "${testInfo.title}":\n` +
|
||||
errors.map(e => ` • ${e}`).join('\n'),
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
// Reset for the next test file.
|
||||
base.afterAll(async () => {
|
||||
seenErrors.length = 0
|
||||
activePage = null
|
||||
})
|
||||
|
||||
export { expect, type Page, type ElectronApplication, _electron }
|
||||
@@ -0,0 +1,274 @@
|
||||
/**
|
||||
* E2E tests for the tile-unread bug — two scenarios:
|
||||
*
|
||||
* 1. TAB (stacked, not visible) — a session opened as a tab via ⌃-click is
|
||||
* NOT visible on screen. When it finishes, the green "unread" dot IS
|
||||
* correct — the user isn't looking at it. This test PASSES.
|
||||
*
|
||||
* 2. SPLIT (side-by-side, visible) — a session dragged to the edge of the
|
||||
* workspace zone opens as a split tile, visible on screen at the same time
|
||||
* as the main session. When it finishes, it should NOT get the green
|
||||
* "unread" dot — the user is looking right at it. This test FAILS until
|
||||
* the fix in session-states.ts:174 lands (the unread check only compares
|
||||
* against $selectedStoredSessionId and ignores $sessionTiles).
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, test } from '@playwright/test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
import {
|
||||
type BackgroundReleaseHandle,
|
||||
createBackgroundReleaseHandle,
|
||||
restartMockServer,
|
||||
SIDEBAR_CROSS_TEXTS,
|
||||
} from './mock-server'
|
||||
|
||||
/** Finished-unread dot aria-label. */
|
||||
const UNREAD_DOT_LABEL = 'Finished — unread'
|
||||
/** Background-running dot aria-label. */
|
||||
const BG_DOT_LABEL = 'Background task running'
|
||||
/** Foreground turn-running dot aria-label. */
|
||||
const SESSION_RUNNING_DOT_LABEL = 'Session running'
|
||||
|
||||
/**
|
||||
* The auto-title auxiliary call hits the SAME mock provider as the chat turn,
|
||||
* and its request carries the user's message — trigger keyword included. The
|
||||
* mock's trigger matching is text-based, so the title call consumes a script
|
||||
* index: the real chat turn then gets turn 2 (final answer, NO tool calls),
|
||||
* the background process is never spawned, and the bg dot never appears.
|
||||
* Whether that happens depends on which request lands first — the CI flake
|
||||
* this spec had. Disable auto-title so script indices line up with real chat
|
||||
* turns (same fix as interim-messages.spec.ts).
|
||||
*/
|
||||
const DISABLE_AUTO_TITLE = 'auxiliary:\n title_generation:\n enabled: false'
|
||||
|
||||
/** Locate a session's sidebar row by its preview text. */
|
||||
function sessionRow(page: import('@playwright/test').Page, text: string) {
|
||||
return page.locator('[data-slot="sidebar"] button').filter({ hasText: text }).first()
|
||||
}
|
||||
|
||||
/** Common setup: start a turn with a held bg process + subagent, wait for
|
||||
* the turn to complete, then switch to a new session so the first session is
|
||||
* no longer $selectedStoredSessionId (required before opening a tile). */
|
||||
async function startTurnAndSwitchAway(page: import('@playwright/test').Page) {
|
||||
// Send E2E_SIDEBAR_CROSS — starts a turn with sleep 5 + subagent.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
await composer.click()
|
||||
await composer.type('E2E_SIDEBAR_CROSS', { delay: 20 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
// Wait for the user's message to appear.
|
||||
await page.waitForFunction(
|
||||
() => (document.body.textContent ?? '').includes('E2E_SIDEBAR_CROSS'),
|
||||
undefined,
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
|
||||
// NOTE: while the turn is busy the dot-state priority paints the session as
|
||||
// "working" ('Session running'), which OUTRANKS the background claim — the
|
||||
// 'Background task running' dot only appears once the turn completes while
|
||||
// the (sentinel-held) process is still alive. Polling for the bg dot mid-turn
|
||||
// races the turn length (two model trips + a real subagent spawn) against
|
||||
// the poll budget, which is exactly the flake this spec had on CI. So: wait
|
||||
// for the turn to END first, then assert the bg dot as a stable state.
|
||||
|
||||
// The final answer text streams before message.complete, so text visibility
|
||||
// alone is not a completion barrier. Wait for the foreground-running state
|
||||
// to clear before asserting the background-process state.
|
||||
await page.waitForFunction(
|
||||
(text) => (document.body.textContent ?? '').includes(text),
|
||||
SIDEBAR_CROSS_TEXTS.finalText,
|
||||
{ timeout: 90_000 },
|
||||
)
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${SESSION_RUNNING_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'session running dot should disappear after turn completes' },
|
||||
)
|
||||
.toBe(0)
|
||||
|
||||
// The background dot must be visible now: the turn is done but the process
|
||||
// is held open by the sentinel, so this is a stable state rather than a
|
||||
// window we have to catch in time. Poll rather than sampling once — the
|
||||
// dot flip is event-driven off the busy=false publish and can land a tick
|
||||
// after the running dot clears.
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'background dot should be visible after turn completes' },
|
||||
)
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
// Switch to a new session — session A is no longer $selectedStoredSessionId.
|
||||
// This is required: openSessionTile bails if the session is already selected.
|
||||
await page.locator('button:has-text("New session")').first().click()
|
||||
await page.waitForTimeout(2000)
|
||||
}
|
||||
|
||||
/** Release the held background process, then wait for its dot to clear. */
|
||||
async function waitForBgProcessToFinish(
|
||||
page: import('@playwright/test').Page,
|
||||
release?: BackgroundReleaseHandle,
|
||||
) {
|
||||
release?.release()
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${BG_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'background dot should disappear after process finishes' },
|
||||
)
|
||||
.toBe(0)
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Test 1: TAB (not visible) — unread dot IS correct (PASSES)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test.describe('sidebar states — tab (hidden) unread is correct', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
const bgRelease = createBackgroundReleaseHandle()
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({
|
||||
extraConfig: DISABLE_AUTO_TITLE,
|
||||
mockServer: { backgroundReleasePath: bgRelease.path },
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
bgRelease.release()
|
||||
await fixture?.cleanup()
|
||||
bgRelease.cleanup()
|
||||
})
|
||||
|
||||
test('session opened as a tab (not visible) correctly gets unread dot', async () => {
|
||||
const page = fixture.page
|
||||
|
||||
await startTurnAndSwitchAway(page)
|
||||
|
||||
// Evidence: session A is in the background (bg dot in sidebar).
|
||||
await page.screenshot({ path: 'test-results/tile-bug-tab-switched-away.png' })
|
||||
|
||||
// ⌃-click opens the session as a TAB (center dock = stacked, not visible
|
||||
// unless it's the active tab). The session is NOT on screen.
|
||||
//
|
||||
// With auto-title disabled the sidebar row is titled by the user's
|
||||
// message (the trigger keyword), not the assistant's final text.
|
||||
const row = sessionRow(page, 'E2E_SIDEBAR_CROSS')
|
||||
await row.click({ modifiers: ['Control'] })
|
||||
await page.waitForTimeout(2000)
|
||||
|
||||
// Evidence: the tab is open but the session is not visible on screen.
|
||||
await page.screenshot({ path: 'test-results/tile-bug-tab-opened.png' })
|
||||
|
||||
await waitForBgProcessToFinish(page, bgRelease)
|
||||
|
||||
// A tab that's not the active tab IS hidden — the unread dot is correct.
|
||||
// The user is NOT looking at it, so marking it "unread" is right.
|
||||
//
|
||||
// Poll rather than sampling once: "finished-unread" is an event-driven
|
||||
// transition that lands slightly after the running dot clears, and with a
|
||||
// released (rather than slowly-expiring) process there is no incidental
|
||||
// slack between the two. Same reasoning as the cross-session spec.
|
||||
await expect
|
||||
.poll(
|
||||
() => page.locator(`[aria-label="${UNREAD_DOT_LABEL}"]`).count(),
|
||||
{ timeout: 30_000, message: 'hidden tab should be marked unread' },
|
||||
)
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
await page.screenshot({ path: 'test-results/tile-bug-tab-unread-correct.png' })
|
||||
})
|
||||
})
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Test 2: SPLIT (visible) — unread dot is WRONG (FAILS until fix)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test.describe.skip('sidebar states — split (visible) unread bug (RED)', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
const bgRelease = createBackgroundReleaseHandle()
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({
|
||||
extraConfig: DISABLE_AUTO_TITLE,
|
||||
mockServer: { backgroundReleasePath: bgRelease.path },
|
||||
})
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
bgRelease.release()
|
||||
await fixture?.cleanup()
|
||||
bgRelease.cleanup()
|
||||
})
|
||||
|
||||
test('session visible in a split tile does NOT get unread dot when it finishes', async () => {
|
||||
const page = fixture.page
|
||||
|
||||
await startTurnAndSwitchAway(page)
|
||||
|
||||
// Evidence: session A is in the background (bg dot in sidebar).
|
||||
await page.screenshot({ path: 'test-results/tile-bug-split-switched-away.png' })
|
||||
|
||||
// Drag the session row from the sidebar to the right edge of the workspace
|
||||
// zone to create a SPLIT (side-by-side) tile. This triggers the real
|
||||
// startSessionDrag → onCommit → openSessionTile(id, 'right', anchor) path.
|
||||
// With auto-title disabled the sidebar row is titled by the user's message.
|
||||
const row = sessionRow(page, 'E2E_SIDEBAR_CROSS')
|
||||
const rowBox = await row.boundingBox()
|
||||
expect(rowBox, 'session row must be visible').not.toBeNull()
|
||||
|
||||
// Find the workspace zone — the main chat area. We drop on its right edge.
|
||||
const workspace = page.locator('[data-session-anchor="workspace"]')
|
||||
const wsBox = await workspace.boundingBox()
|
||||
expect(wsBox, 'workspace zone must be visible').not.toBeNull()
|
||||
|
||||
// Drag from the session row to the right edge of the workspace.
|
||||
// The drag-session's subZonePosition resolves a right-edge drop as 'right'
|
||||
// (a split dock), not 'center' (which would be a composer link).
|
||||
await page.mouse.move(rowBox!.x + rowBox!.width / 2, rowBox!.y + rowBox!.height / 2)
|
||||
await page.mouse.down()
|
||||
// Move in steps so the drag-session's pointermove handler tracks the
|
||||
// position and resolves the drop zone (a single jump can miss the
|
||||
// threshold/engage logic).
|
||||
const targetX = wsBox!.x + wsBox!.width - 20
|
||||
const targetY = wsBox!.y + wsBox!.height / 2
|
||||
const steps = 10
|
||||
for (let i = 1; i <= steps; i++) {
|
||||
const x = rowBox!.x + rowBox!.width / 2 + (targetX - (rowBox!.x + rowBox!.width / 2)) * (i / steps)
|
||||
const y = rowBox!.y + rowBox!.height / 2 + (targetY - (rowBox!.y + rowBox!.height / 2)) * (i / steps)
|
||||
await page.mouse.move(x, y)
|
||||
await page.waitForTimeout(30)
|
||||
}
|
||||
await page.mouse.up()
|
||||
await page.waitForTimeout(2000)
|
||||
|
||||
// Evidence: the split tile is now open side-by-side — both sessions visible.
|
||||
await page.screenshot({ path: 'test-results/tile-bug-split-opened.png' })
|
||||
|
||||
await waitForBgProcessToFinish(page, bgRelease)
|
||||
|
||||
// THE BUG: the session visible in the split tile should NOT have the green
|
||||
// "finished unread" dot — the user is looking right at it. This assertion
|
||||
// FAILS until the fix in session-states.ts:174 lands.
|
||||
const unreadCount = await page.locator(`[aria-label="${UNREAD_DOT_LABEL}"]`).count()
|
||||
expect(unreadCount, 'session visible in a split tile should NOT be marked unread').toBe(0)
|
||||
|
||||
// Evidence: the green dot should NOT be here — this screenshot shows the bug.
|
||||
await page.screenshot({ path: 'test-results/tile-bug-split-unread-should-not-exist.png' })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,162 @@
|
||||
/**
|
||||
* E2E test for PERSISTED unread state — the green "Finished — unread" dot
|
||||
* must survive an app restart.
|
||||
*
|
||||
* Regression coverage for the reset-on-restart bug: the unread flag used to
|
||||
* live only in the in-memory `$unreadFinishedSessionIds` atom, so closing and
|
||||
* reopening the desktop app grayed out every green dot. The persisted layer
|
||||
* (src/store/session-unread.ts) now rebuilds the dot from localStorage-backed
|
||||
* finish markers + seen-count watermarks.
|
||||
*
|
||||
* The scenario uses TWO sessions on purpose: with a single session the app
|
||||
* can reopen straight into it after a restart, which acks the session (the
|
||||
* user is looking at it) and would mask the dot. Session A finishes in the
|
||||
* background while session B is the selected one; the restart reopens into
|
||||
* B, so A's dot is observable.
|
||||
*
|
||||
* 1. Session A: start a turn, hold its stream open.
|
||||
* 2. Session B: new session, send a message, let it finish while SELECTED.
|
||||
* 3. Release A's stream → its background finish paints A's green dot.
|
||||
* 4. QUIT the app, relaunch on the SAME sandbox → A's dot must still be
|
||||
* there (previously it was lost).
|
||||
* 5. Open session A → dot clears.
|
||||
* 6. Restart once more → the cleared state must persist too (no zombie dot).
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, type Page, test } from '@playwright/test'
|
||||
import { type ElectronApplication } from '@playwright/test'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
launchDesktop,
|
||||
type MockBackendFixture,
|
||||
setupMockBackend,
|
||||
waitForAppReady,
|
||||
} from './fixtures'
|
||||
import { restartMockServer } from './mock-server'
|
||||
|
||||
/** Finished-unread dot aria-label (from i18n en.ts). */
|
||||
const UNREAD_DOT_LABEL = 'Finished — unread'
|
||||
|
||||
/** Held prompt — the mock pauses this stream until we release it, so the
|
||||
* turn is deterministically still running when we switch away. */
|
||||
const HELD_PROMPT = 'E2E unread restart: hold this stream until released.'
|
||||
/** Second session's prompt — completes normally while selected. */
|
||||
const SECOND_PROMPT = 'E2E unread restart: second session, read while open.'
|
||||
|
||||
const unreadDots = (page: Page) => page.locator(`[aria-label="${UNREAD_DOT_LABEL}"]`)
|
||||
|
||||
async function sendMessage(page: Page, text: string): Promise<void> {
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.waitFor({ state: 'visible', timeout: 15_000 })
|
||||
await composer.click()
|
||||
await composer.type(text, { delay: 5 })
|
||||
await page.keyboard.press('Enter')
|
||||
}
|
||||
|
||||
test.describe('unread dot survives app restart', () => {
|
||||
test.describe.configure({ mode: 'serial' })
|
||||
// Three full app boots in one scenario — give it more than the global 90s.
|
||||
test.setTimeout(300_000)
|
||||
|
||||
let fixture: MockBackendFixture
|
||||
let app: ElectronApplication
|
||||
let page: Page
|
||||
|
||||
test.beforeAll(async () => {
|
||||
restartMockServer()
|
||||
fixture = await setupMockBackend({
|
||||
mockServer: { holdFirstStreamForPrompt: HELD_PROMPT },
|
||||
})
|
||||
app = fixture.app
|
||||
page = fixture.page
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
// The fixture's own app handle may already be closed by the restart
|
||||
// steps — close whatever is current, then drop the sandbox + mock.
|
||||
try {
|
||||
await app?.close()
|
||||
} catch {
|
||||
// already closed
|
||||
}
|
||||
|
||||
fixture?.mock.close()
|
||||
fixture?.sandbox.cleanup()
|
||||
})
|
||||
|
||||
/** Relaunch the desktop app against the SAME sandbox (same userData →
|
||||
* same localStorage, same HERMES_HOME → same session store). */
|
||||
async function restartApp(): Promise<void> {
|
||||
await app.close()
|
||||
|
||||
const relaunched = await launchDesktop(buildAppEnv(fixture.sandbox))
|
||||
app = relaunched.app
|
||||
page = relaunched.page
|
||||
await waitForAppReady({ ...fixture, app, page }, 120_000)
|
||||
}
|
||||
|
||||
test('green dot persists across restart and its clear persists too', async () => {
|
||||
// ── 1. Session A: start a turn whose stream the mock holds open ────
|
||||
await sendMessage(page, HELD_PROMPT)
|
||||
await fixture.mock.waitForHeldStream()
|
||||
|
||||
// ── 2. Session B: complete a turn while SELECTED (stays read) ──────
|
||||
await page.locator('button:has-text("New session")').first().click()
|
||||
await sendMessage(page, SECOND_PROMPT)
|
||||
|
||||
// B's reply lands in the open transcript — this session is "read".
|
||||
await page.waitForFunction(
|
||||
() => (document.body.textContent ?? '').includes('Hello from the mock inference server'),
|
||||
undefined,
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
|
||||
// ── 3. Release A's stream → background finish paints A's dot ──────
|
||||
fixture.mock.releaseHeldStream()
|
||||
|
||||
await expect
|
||||
.poll(() => unreadDots(page).count(), {
|
||||
timeout: 30_000,
|
||||
message: 'green unread dot should appear after the background finish',
|
||||
})
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
// ── 4. Restart the app — the dot must survive ──────────────────────
|
||||
// The app reopens into session B (or a fresh draft), NOT session A, so
|
||||
// A's dot is observable rather than being acked by the route restore.
|
||||
await restartApp()
|
||||
|
||||
await expect
|
||||
.poll(() => unreadDots(page).count(), {
|
||||
timeout: 60_000,
|
||||
message: 'green unread dot should be rebuilt from persisted state after restart',
|
||||
})
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
// ── 5. Open session A — the dot clears ─────────────────────────────
|
||||
// The dot sits inside A's sidebar row button; click that row.
|
||||
await unreadDots(page)
|
||||
.first()
|
||||
.locator('xpath=ancestor::button[1]')
|
||||
.click()
|
||||
|
||||
await expect
|
||||
.poll(() => unreadDots(page).count(), {
|
||||
timeout: 15_000,
|
||||
message: 'opening the session should clear its unread dot',
|
||||
})
|
||||
.toBe(0)
|
||||
|
||||
// ── 6. Restart again — the CLEARED state must persist as well ──────
|
||||
await restartApp()
|
||||
|
||||
// Give the sidebar a moment to load rows, then assert no dot returns.
|
||||
await page.waitForSelector('[data-slot="sidebar"]', { timeout: 60_000 })
|
||||
await page.waitForTimeout(5_000)
|
||||
expect(await unreadDots(page).count(), 'acked session must stay read after restart').toBe(0)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,150 @@
|
||||
/**
|
||||
* Visual snapshot helper — wraps `toHaveScreenshot` so visual diffs are
|
||||
* reported without failing the test suite.
|
||||
*
|
||||
* On CI, the JSON reporter + post-test script parse the results and post a
|
||||
* summary to the GitHub Actions step output, and diff images are uploaded
|
||||
* as artifacts. This keeps visual regressions visible without gating PRs
|
||||
* on pixel-perfect matches.
|
||||
*
|
||||
* The actual screenshot is always written to the test output dir so CI
|
||||
* artifacts include every screenshot — not just the ones that diffed.
|
||||
* When it differs, this helper also writes expected and diff images:
|
||||
* <name>-actual.png, <name>-expected.png, <name>-diff.png
|
||||
*/
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
|
||||
import { type ElectronApplication, type Page, test } from '@playwright/test'
|
||||
|
||||
/** Fixed window dimensions for visual regression screenshots. */
|
||||
export const VISUAL_WINDOW_WIDTH = 1220
|
||||
export const VISUAL_WINDOW_HEIGHT = 800
|
||||
|
||||
export interface VisualSnapshotOptions {
|
||||
/** Snapshot name — defaults to the test title. */
|
||||
name?: string
|
||||
/** Full page screenshot vs. viewport-only (default). */
|
||||
fullPage?: boolean
|
||||
/** Timeout in ms. */
|
||||
timeout?: number
|
||||
/** The Electron app handle — used to size and decode screenshots. */
|
||||
app: ElectronApplication
|
||||
}
|
||||
|
||||
/**
|
||||
* Force the Electron window to a fixed size so screenshots are comparable
|
||||
* across runs and CI environments. Window managers (Hyprland, etc.) may
|
||||
* auto-tile or resize windows after launch; calling this right before the
|
||||
* screenshot ensures the viewport is always the expected size.
|
||||
*/
|
||||
async function forceFixedSize(app: ElectronApplication): Promise<void> {
|
||||
await app.evaluate(({ BrowserWindow }, { width, height }) => {
|
||||
const win = BrowserWindow.getAllWindows()[0]
|
||||
|
||||
if (win) {
|
||||
win.unmaximize()
|
||||
// setMinimumSize must be ≤ the target, otherwise setSize is clamped.
|
||||
win.setMinimumSize(width, height)
|
||||
win.setSize(width, height, false)
|
||||
win.setBounds({ x: 0, y: 0, width, height })
|
||||
}
|
||||
}, { width: VISUAL_WINDOW_WIDTH, height: VISUAL_WINDOW_HEIGHT })
|
||||
}
|
||||
|
||||
/**
|
||||
* Take a screenshot and compare it against the baseline.
|
||||
*
|
||||
* If the baseline doesn't exist yet (first run), Playwright creates it.
|
||||
* If it differs, the test logs a soft warning but does NOT fail — the diff
|
||||
* images are still generated for CI to surface.
|
||||
*/
|
||||
export async function expectVisualSnapshot(
|
||||
page: Page,
|
||||
options: VisualSnapshotOptions,
|
||||
): Promise<void> {
|
||||
const { name, fullPage = false, timeout = 30_000, app } = options
|
||||
|
||||
// Force the window to a fixed size right before the screenshot so it's
|
||||
// always comparable, regardless of WM resizing during the test.
|
||||
await forceFixedSize(app)
|
||||
// Give the renderer a moment to relayout after the resize.
|
||||
await page.waitForTimeout(500)
|
||||
|
||||
// Playwright appends a platform suffix (e.g. "-linux") and requires
|
||||
// a .png extension on the name argument. Auto-append it if missing.
|
||||
const snapshotName = name ? (name.endsWith('.png') ? name : `${name}.png`) : undefined
|
||||
|
||||
const info = test.info()
|
||||
const actual = await page.screenshot({ animations: 'disabled', caret: 'hide', fullPage, timeout })
|
||||
const baselinePath = info.snapshotPath(snapshotName ?? `${info.title}.png`)
|
||||
const outputName = (snapshotName ?? 'snapshot.png').replace(/\.png$/, '')
|
||||
|
||||
if (info.config.updateSnapshots === 'all' || info.config.updateSnapshots === 'changed') {
|
||||
fs.mkdirSync(path.dirname(baselinePath), { recursive: true })
|
||||
fs.writeFileSync(baselinePath, actual)
|
||||
// Also write to the output dir so CI artifacts include the screenshot.
|
||||
fs.writeFileSync(info.outputPath(`${outputName}-actual.png`), actual)
|
||||
console.log(`[visual-baseline] updated ${baselinePath}`)
|
||||
return
|
||||
}
|
||||
|
||||
if (!fs.existsSync(baselinePath)) {
|
||||
fs.writeFileSync(info.outputPath(`${outputName}-actual.png`), actual)
|
||||
console.log(`[visual-diff] ${name ?? '(unnamed)'} — no baseline available`)
|
||||
return
|
||||
}
|
||||
|
||||
const expected = fs.readFileSync(baselinePath)
|
||||
const comparison = await app.evaluate(
|
||||
({ nativeImage }, images) => {
|
||||
const actualImage = nativeImage.createFromBuffer(Buffer.from(images.actual, 'base64'))
|
||||
const expectedImage = nativeImage.createFromBuffer(Buffer.from(images.expected, 'base64'))
|
||||
const actualSize = actualImage.getSize()
|
||||
const expectedSize = expectedImage.getSize()
|
||||
|
||||
if (actualSize.width !== expectedSize.width || actualSize.height !== expectedSize.height) {
|
||||
return { mismatchRatio: 1, diff: images.actual }
|
||||
}
|
||||
|
||||
const actualPixels = actualImage.toBitmap()
|
||||
const expectedPixels = expectedImage.toBitmap()
|
||||
const diffPixels = Buffer.alloc(actualPixels.length)
|
||||
let mismatched = 0
|
||||
|
||||
for (let i = 0; i < actualPixels.length; i += 4) {
|
||||
const different =
|
||||
Math.abs(actualPixels[i] - expectedPixels[i]) > 51 ||
|
||||
Math.abs(actualPixels[i + 1] - expectedPixels[i + 1]) > 51 ||
|
||||
Math.abs(actualPixels[i + 2] - expectedPixels[i + 2]) > 51 ||
|
||||
Math.abs(actualPixels[i + 3] - expectedPixels[i + 3]) > 51
|
||||
|
||||
if (different) {
|
||||
mismatched++
|
||||
diffPixels[i + 2] = 255
|
||||
}
|
||||
diffPixels[i + 3] = 255
|
||||
}
|
||||
|
||||
return {
|
||||
mismatchRatio: mismatched / (actualPixels.length / 4),
|
||||
diff: nativeImage.createFromBitmap(diffPixels, actualSize).toPNG().toString('base64'),
|
||||
}
|
||||
},
|
||||
{ actual: actual.toString('base64'), expected: expected.toString('base64') },
|
||||
)
|
||||
|
||||
// Always write the actual screenshot to the output dir so CI artifacts
|
||||
// include every screenshot — not just the ones that diffed.
|
||||
fs.writeFileSync(info.outputPath(`${outputName}-actual.png`), actual)
|
||||
|
||||
if (comparison.mismatchRatio <= 0.01) {
|
||||
return
|
||||
}
|
||||
|
||||
fs.writeFileSync(info.outputPath(`${outputName}-expected.png`), expected)
|
||||
fs.writeFileSync(info.outputPath(`${outputName}-diff.png`), Buffer.from(comparison.diff, 'base64'))
|
||||
console.log(
|
||||
`[visual-diff] ${name ?? '(unnamed)'} — ${(comparison.mismatchRatio * 100).toFixed(2)}% of pixels differ`,
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,477 @@
|
||||
/**
|
||||
* E2E regression: warm-route resume must not re-render the transcript more
|
||||
* than once.
|
||||
*
|
||||
* When a session is already in the runtime-id cache (the "warm" path in
|
||||
* `resumeSession()`), clicking its sidebar row should paint the transcript
|
||||
* exactly once. Before the fix, the warm cache painted via
|
||||
* `syncSessionStateToView`, then the `session.activate` RPC returned a
|
||||
* reconciled message list with different message object references, causing
|
||||
* `syncSessionStateToView` to fire a second `setMessages` — a visual
|
||||
* flicker as the transcript DOM was updated.
|
||||
*
|
||||
* This test pre-seeds a session into state.db, boots the app,
|
||||
* clicks the session (cold resume — populates the warm cache), navigates
|
||||
* away to a new chat, then clicks back (warm resume). Two detectors run:
|
||||
*
|
||||
* 1. A MutationObserver counts additive DOM mutation bursts (childList
|
||||
* additions). More than 1 burst = the transcript was repainted.
|
||||
*
|
||||
* 2. A 2ms innerHTML-length poll counts "reconciles" — DOM content changes
|
||||
* that happen AFTER the initial paint, while messages are already on
|
||||
* screen. This catches the case where React reconciles by key without
|
||||
* adding/removing nodes (same keys → in-place prop update → no
|
||||
* MutationObserver burst), but `$messages` was still set twice.
|
||||
*
|
||||
* The test passes when bursts === 1 AND reconciles === 0.
|
||||
* The sidebar "+" keeps the session warm in another tab. Its reactivation
|
||||
* follows the same contract: one additive paint and zero reconciles.
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { expect, test } from './test'
|
||||
|
||||
import {
|
||||
type MockBackendFixture,
|
||||
waitForAppReady,
|
||||
createSandbox,
|
||||
writeMockProviderConfig,
|
||||
writeEnvFile,
|
||||
buildAppEnv,
|
||||
launchDesktop,
|
||||
} from './fixtures'
|
||||
import { startMockServer } from './mock-server'
|
||||
import { RealSessionBuilder } from './real-session-builder'
|
||||
|
||||
const SESSION_TITLE = 'E2E Warm Resume Jitter Test'
|
||||
|
||||
// Inactive tabs stay mounted under a data-pane-hidden ancestor. Match the
|
||||
// renderer's keep-alive visibility policy instead of relying on DOM order.
|
||||
const SURFACE = '[data-composer-target]:not([data-pane-hidden] [data-composer-target])'
|
||||
const ALL_SURFACES = '[data-composer-target]'
|
||||
/**
|
||||
* 16 messages (8 user/assistant pairs) — enough DOM churn for detection while
|
||||
* still fitting a hot-hidden pane's retention budget. A kept-alive pane keeps
|
||||
* only its live tail (HIDDEN_TRANSCRIPT_RENDER_BUDGET = 40 weight units in
|
||||
* thread/list.tsx); 16 short messages ≈ 32 units, so the whole transcript
|
||||
* survives hiding. Above the budget, reveal legitimately backfills trimmed
|
||||
* turns (additive DOM bursts) — that is paging, not the repaint bug this
|
||||
* suite hunts, and it would drown the detectors.
|
||||
*/
|
||||
const MESSAGE_COUNT = 16
|
||||
/** Seeded PRNG so the generated content is deterministic across runs. */
|
||||
const RNG_SEED = 42
|
||||
|
||||
/** Mulberry32 — tiny deterministic PRNG. */
|
||||
function mulberry32(seed: number): () => number {
|
||||
let a = seed
|
||||
return () => {
|
||||
a |= 0
|
||||
a = (a + 0x6d2b79f5) | 0
|
||||
let t = Math.imul(a ^ (a >>> 15), 1 | a)
|
||||
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296
|
||||
}
|
||||
}
|
||||
|
||||
/** Generate ~40 chars of gibberish from a seeded PRNG. */
|
||||
function gibberish(rng: () => number): string {
|
||||
const len = 30 + Math.floor(rng() * 20)
|
||||
let s = ''
|
||||
for (let i = 0; i < len; i++) {
|
||||
s += String.fromCharCode(97 + Math.floor(rng() * 26))
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
/** First user message — used as a wait target in the test. */
|
||||
const FIRST_USER_MSG = gibberish(mulberry32(RNG_SEED))
|
||||
|
||||
/**
|
||||
* Generate the user turns for a real session. The mock provider produces the
|
||||
* assistant side of each pair through the normal AIAgent persistence path.
|
||||
*/
|
||||
function generateSessionTurns(): string[] {
|
||||
const rng = mulberry32(RNG_SEED)
|
||||
const turns: string[] = []
|
||||
|
||||
for (let i = 0; i < MESSAGE_COUNT / 2; i++) {
|
||||
turns.push(gibberish(rng))
|
||||
gibberish(rng)
|
||||
}
|
||||
|
||||
return turns
|
||||
}
|
||||
|
||||
/**
|
||||
* Set up a mock-backend sandbox with a real persisted session in state.db.
|
||||
*
|
||||
* Unlike the shared `setupMockBackend()`, this variant creates the session
|
||||
* through the real stdio gateway before launching desktop so the session is
|
||||
* visible in the sidebar on first load.
|
||||
*/
|
||||
async function setupSeededMockBackend(): Promise<MockBackendFixture> {
|
||||
// 1. Start mock server
|
||||
const mock = await startMockServer()
|
||||
|
||||
// 2. Create sandbox + write config
|
||||
const sandbox = createSandbox('warm-seed')
|
||||
writeMockProviderConfig(sandbox.hermesHome, mock.url)
|
||||
writeEnvFile(sandbox.hermesHome)
|
||||
|
||||
// 3. Produce all 16 user/assistant pairs through the real TUI gateway,
|
||||
// AIAgent, mock provider, and SessionDB persistence path before desktop starts.
|
||||
const builder = await RealSessionBuilder.start(sandbox.hermesHome)
|
||||
try {
|
||||
await builder.createSession({ title: SESSION_TITLE, turns: generateSessionTurns() })
|
||||
} finally {
|
||||
await builder.close()
|
||||
}
|
||||
|
||||
// 4. Build env + launch
|
||||
const env = buildAppEnv(sandbox)
|
||||
const { app, page } = await launchDesktop(env)
|
||||
|
||||
return {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupSeededMockBackend()
|
||||
await waitForAppReady(fixture!, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
/**
|
||||
* Install a MutationObserver + text-content poll on the thread viewport
|
||||
* to detect re-renders after the initial paint. Returns nothing — call
|
||||
* `readRenderCount` to stop and collect results.
|
||||
*
|
||||
* - MutationObserver: counts additive childList bursts (5ms coalescing).
|
||||
* - Text-content poll: counts "reconciles" — first-message text changes
|
||||
* after the initial paint, catching key-based reconciles that don't
|
||||
* add/remove nodes.
|
||||
*/
|
||||
async function installRenderCounter(
|
||||
page: import('@playwright/test').Page,
|
||||
transcriptText?: string,
|
||||
): Promise<void> {
|
||||
await page.evaluate(([visibleSelector, allSelector, expected]: [string, string, string | undefined]) => {
|
||||
const surfaces = [...document.querySelectorAll(expected ? allSelector : visibleSelector)]
|
||||
const surface = expected
|
||||
? surfaces.find(candidate =>
|
||||
(candidate.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected),
|
||||
)
|
||||
: surfaces.at(-1)
|
||||
const viewport = surface?.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
if (!viewport) {
|
||||
const diag = [...document.querySelectorAll(allSelector)].map(s => ({
|
||||
hidden: Boolean(s.closest('[data-pane-hidden]')),
|
||||
target: s.getAttribute('data-composer-target'),
|
||||
hasViewport: Boolean(s.querySelector('[data-slot="aui_thread-viewport"]')),
|
||||
textLen: (s.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').length,
|
||||
head: (s.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').slice(0, 80),
|
||||
tail: (s.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').slice(-80),
|
||||
includesExpected: expected ? (s.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected) : null,
|
||||
}))
|
||||
throw new Error('Thread viewport not found before warm resume DIAG=' + JSON.stringify(diag) + ' expected=' + expected)
|
||||
}
|
||||
|
||||
const state = { bursts: 0, mutations: 0, timeline: [] as number[], stopped: false, reconciles: 0 }
|
||||
const debugWindow = window as unknown as {
|
||||
__RENDER_COUNT__: typeof state
|
||||
__RENDER_VIEWPORT__: Element
|
||||
}
|
||||
debugWindow.__RENDER_COUNT__ = state
|
||||
debugWindow.__RENDER_VIEWPORT__ = viewport
|
||||
|
||||
let currentBatch = 0
|
||||
let flushTimer: ReturnType<typeof setTimeout> | null = null
|
||||
|
||||
const flush = () => {
|
||||
flushTimer = null
|
||||
if (currentBatch > 0 && !state.stopped) {
|
||||
state.bursts += 1
|
||||
state.timeline.push(currentBatch)
|
||||
currentBatch = 0
|
||||
}
|
||||
}
|
||||
|
||||
const observer = new MutationObserver(records => {
|
||||
if (state.stopped) return
|
||||
let batchAdded = 0
|
||||
for (const record of records) {
|
||||
state.mutations += 1
|
||||
if (record.type === 'childList' && record.addedNodes.length > 0) {
|
||||
batchAdded += 1
|
||||
}
|
||||
}
|
||||
if (batchAdded > 0) {
|
||||
currentBatch += batchAdded
|
||||
if (flushTimer) clearTimeout(flushTimer)
|
||||
flushTimer = setTimeout(flush, 5)
|
||||
}
|
||||
})
|
||||
|
||||
observer.observe(viewport, {
|
||||
childList: true,
|
||||
subtree: true,
|
||||
attributes: false,
|
||||
characterData: false,
|
||||
})
|
||||
|
||||
// Poll the first message's text content every 2ms. The MutationObserver
|
||||
// only catches childList additions; React may reconcile by key without
|
||||
// adding/removing nodes (same keys → in-place prop update → no childList
|
||||
// mutation). The poll catches this by detecting text content changes in
|
||||
// the first message after the initial paint. Metadata-only changes (model
|
||||
// name, busy indicator) don't affect message text, so they don't produce
|
||||
// false positives.
|
||||
const contentEl = viewport.querySelector('[data-slot="aui_thread-content"]') ?? viewport
|
||||
let lastFirstMsgText = ''
|
||||
let hasMessages = false
|
||||
const pollInterval = setInterval(() => {
|
||||
if (state.stopped) {
|
||||
clearInterval(pollInterval)
|
||||
return
|
||||
}
|
||||
const firstMsg = contentEl.querySelector('[data-role="message"], [data-message-id]')
|
||||
const firstMsgText = firstMsg?.textContent ?? ''
|
||||
if (firstMsgText && firstMsgText !== lastFirstMsgText) {
|
||||
if (hasMessages) {
|
||||
state.reconciles = (state.reconciles ?? 0) + 1
|
||||
}
|
||||
lastFirstMsgText = firstMsgText
|
||||
hasMessages = true
|
||||
}
|
||||
}, 2)
|
||||
}, [SURFACE, ALL_SURFACES, transcriptText] as [string, string, string | undefined])
|
||||
}
|
||||
|
||||
/** Wait until the ACTIVE chat surface's transcript contains `text`. */
|
||||
async function waitForActiveTranscriptText(
|
||||
page: import('@playwright/test').Page,
|
||||
text: string,
|
||||
timeout = 30_000,
|
||||
): Promise<void> {
|
||||
await page.waitForFunction(
|
||||
([expected, surfaceSelector]: [string, string]) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const active = surfaces[surfaces.length - 1]
|
||||
|
||||
return (active?.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected)
|
||||
},
|
||||
[text, SURFACE] as [string, string],
|
||||
{ timeout },
|
||||
)
|
||||
}
|
||||
|
||||
async function waitForActiveTranscriptWithoutText(
|
||||
page: import('@playwright/test').Page,
|
||||
text: string,
|
||||
): Promise<void> {
|
||||
await page.waitForFunction(
|
||||
([expected, surfaceSelector]: [string, string]) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const active = surfaces[surfaces.length - 1]
|
||||
|
||||
return !(active?.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(expected)
|
||||
},
|
||||
[text, SURFACE] as [string, string],
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
}
|
||||
|
||||
/** Replace the primary surface with a draft while retaining its warm cache. */
|
||||
async function openFreshDraft(page: import('@playwright/test').Page, priorText: string): Promise<void> {
|
||||
await page.keyboard.press(process.platform === 'darwin' ? 'Meta+N' : 'Control+N')
|
||||
await waitForActiveTranscriptWithoutText(page, priorText)
|
||||
}
|
||||
|
||||
/** Stack an empty tab while leaving the current transcript mounted and warm. */
|
||||
async function openNewSessionTab(page: import('@playwright/test').Page, priorText: string): Promise<void> {
|
||||
await page.locator('[data-slot="sidebar"] button[aria-label="New session"]').first().click()
|
||||
await waitForActiveTranscriptWithoutText(page, priorText)
|
||||
}
|
||||
|
||||
/** Stop the render counter and return the recorded burst/reconcile counts. */
|
||||
async function readRenderCount(page: import('@playwright/test').Page): Promise<{
|
||||
bursts: number
|
||||
mutations: number
|
||||
timeline: number[]
|
||||
reconciles: number
|
||||
} | null> {
|
||||
return page.evaluate(() => {
|
||||
type RenderCount = { bursts: number; mutations: number; timeline: number[]; stopped: boolean; reconciles: number }
|
||||
const w = window as unknown as { __RENDER_COUNT__?: RenderCount }
|
||||
const rc = w.__RENDER_COUNT__
|
||||
if (rc) {
|
||||
rc.stopped = true
|
||||
}
|
||||
return rc ? { bursts: rc.bursts, mutations: rc.mutations, timeline: rc.timeline, reconciles: rc.reconciles } : null
|
||||
})
|
||||
}
|
||||
|
||||
async function observedViewportIsActive(page: import('@playwright/test').Page): Promise<boolean> {
|
||||
return page.evaluate((surfaceSelector: string) => {
|
||||
const surfaces = document.querySelectorAll(surfaceSelector)
|
||||
const activeViewport = surfaces[surfaces.length - 1]?.querySelector('[data-slot="aui_thread-viewport"]')
|
||||
const observedViewport = (window as unknown as { __RENDER_VIEWPORT__?: Element }).__RENDER_VIEWPORT__
|
||||
|
||||
return activeViewport === observedViewport
|
||||
}, SURFACE)
|
||||
}
|
||||
|
||||
/** A kept-alive tab must become visible without rebuilding its transcript. */
|
||||
function assertNoRepaint(result: { bursts: number; mutations: number; timeline: number[]; reconciles: number } | null): void {
|
||||
expect(result, 'MutationObserver should have recorded render data').toBeTruthy()
|
||||
expect(
|
||||
result!.bursts,
|
||||
`Expected no additive render bursts for a kept-alive tab, but got ${result!.bursts}. ` +
|
||||
`Mutation timeline: ${JSON.stringify(result!.timeline)}.`,
|
||||
).toBe(0)
|
||||
expect(
|
||||
result!.reconciles,
|
||||
`Expected no transcript reconciles for a kept-alive tab, but got ${result!.reconciles}.`,
|
||||
).toBe(0)
|
||||
}
|
||||
|
||||
/** Assert the render counter shows exactly one paint with no re-renders. */
|
||||
function assertNoJitter(result: { bursts: number; mutations: number; timeline: number[]; reconciles: number } | null): void {
|
||||
expect(result, 'MutationObserver should have recorded render data').toBeTruthy()
|
||||
expect(
|
||||
result!.bursts,
|
||||
`Expected 1 additive render burst (single paint), but got ${result!.bursts} bursts. ` +
|
||||
`Mutation timeline: ${JSON.stringify(result!.timeline)}.`,
|
||||
).toBe(1)
|
||||
expect(
|
||||
result!.reconciles,
|
||||
`Expected 0 reconciles (no re-render after initial paint), but got ${result!.reconciles}. ` +
|
||||
`This means the warm-route resume re-rendered the transcript after the initial paint ` +
|
||||
`— the "warm resume jitter" bug is present.`,
|
||||
).toBe(0)
|
||||
}
|
||||
|
||||
test('tab reactivation preserves the mounted transcript without repainting', async ({}, testInfo) => {
|
||||
const page = fixture!.page
|
||||
|
||||
// Wait for the sidebar to populate with our seeded session.
|
||||
const sessionRow = page
|
||||
.locator('[data-slot="sidebar"] button')
|
||||
.filter({ hasText: SESSION_TITLE })
|
||||
.first()
|
||||
await sessionRow.waitFor({ state: 'visible', timeout: 60_000 })
|
||||
|
||||
// Step 1: Cold resume — click the session row to load it.
|
||||
// This populates the warm cache (runtimeIdByStoredSessionId + sessionStateByRuntimeId).
|
||||
await sessionRow.click()
|
||||
|
||||
// Wait for the transcript to appear — the first user message text confirms
|
||||
// the cold-path prefetch painted.
|
||||
await waitForActiveTranscriptText(page, FIRST_USER_MSG)
|
||||
|
||||
// Wait for the session to fully settle (cold-path RPC + reconciliation).
|
||||
await page.waitForTimeout(2_000)
|
||||
|
||||
// Stack a new tab, then observe the seeded transcript while it is hidden.
|
||||
// Installing after the switch isolates reactivation from mutations caused
|
||||
// while the new tab was being created.
|
||||
await openNewSessionTab(page, FIRST_USER_MSG)
|
||||
await page.waitForTimeout(500)
|
||||
await installRenderCounter(page, FIRST_USER_MSG)
|
||||
|
||||
// Step 3: Click back and verify the same kept-alive viewport becomes active
|
||||
// without rebuilding or reconciling its transcript.
|
||||
await sessionRow.click()
|
||||
|
||||
await waitForActiveTranscriptText(page, FIRST_USER_MSG)
|
||||
await page.waitForTimeout(2_000)
|
||||
expect(await observedViewportIsActive(page), 'Reactivation should reveal the observed kept-alive viewport').toBe(true)
|
||||
|
||||
const result = await readRenderCount(page)
|
||||
await page.screenshot({ path: testInfo.outputPath('warm-resume-idle.png') })
|
||||
assertNoRepaint(result)
|
||||
})
|
||||
|
||||
test('warm-route resume after background inference completes (no jitter)', async ({}, testInfo) => {
|
||||
test.fixme(
|
||||
true,
|
||||
'Warm resume repaints after inference: expected one additive burst, got two ([18,1]).',
|
||||
)
|
||||
|
||||
const page = fixture!.page
|
||||
const { mock } = fixture!
|
||||
|
||||
// Wait for the sidebar to populate with our seeded session.
|
||||
const sessionRow = page
|
||||
.locator('[data-slot="sidebar"] button')
|
||||
.filter({ hasText: SESSION_TITLE })
|
||||
.first()
|
||||
await sessionRow.waitFor({ state: 'visible', timeout: 60_000 })
|
||||
|
||||
// Step 1: Cold resume — populate the warm cache.
|
||||
await sessionRow.click()
|
||||
await waitForActiveTranscriptText(page, FIRST_USER_MSG)
|
||||
await page.waitForTimeout(2_000)
|
||||
|
||||
// Step 2: Send a message — triggers inference via the mock server.
|
||||
const PROMPT = 'E2E post-inference warm resume test prompt'
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.click()
|
||||
await composer.type(PROMPT, { delay: 10 })
|
||||
await page.keyboard.press('Enter')
|
||||
|
||||
// Wait for the mock response to appear in the transcript, confirming
|
||||
// the turn completed and message.complete fired (which updates the warm
|
||||
// cache via updateSessionState).
|
||||
await waitForActiveTranscriptText(page, 'mock inference server', 60_000)
|
||||
// Extra settle for message.complete → updateSessionState → cache write.
|
||||
await page.waitForTimeout(2_000)
|
||||
|
||||
// Verify the prompt was received by the mock server.
|
||||
expect(mock.receivedPrompts).toContain(PROMPT)
|
||||
|
||||
// Step 3: Replace the primary chat; the warm cache retains the updated messages.
|
||||
await openFreshDraft(page, PROMPT)
|
||||
await page.waitForTimeout(500)
|
||||
|
||||
// Step 4: Install render counter, click back (warm resume), wait, assert.
|
||||
await installRenderCounter(page)
|
||||
await sessionRow.click()
|
||||
|
||||
// Wait for the transcript to reappear — the warm cache should already
|
||||
// have the completed turn (updated by message.complete events).
|
||||
await waitForActiveTranscriptText(page, FIRST_USER_MSG)
|
||||
|
||||
// Wait for at least 1 burst, then settle.
|
||||
await page.waitForFunction(
|
||||
() => {
|
||||
const w = window as unknown as { __RENDER_COUNT__?: { bursts: number } }
|
||||
return Boolean(w.__RENDER_COUNT__ && w.__RENDER_COUNT__.bursts > 0)
|
||||
},
|
||||
undefined,
|
||||
{ timeout: 10_000 },
|
||||
)
|
||||
await page.waitForTimeout(2_000)
|
||||
|
||||
const result = await readRenderCount(page)
|
||||
await page.screenshot({ path: testInfo.outputPath('warm-resume-post-inference.png') })
|
||||
assertNoJitter(result)
|
||||
})
|
||||
@@ -0,0 +1,243 @@
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import * as fs from 'node:fs'
|
||||
import * as path from 'node:path'
|
||||
|
||||
import {
|
||||
buildAppEnv,
|
||||
createSandbox,
|
||||
launchDesktop,
|
||||
type MockBackendFixture,
|
||||
waitForAppReady,
|
||||
writeEnvFile,
|
||||
writeMockProviderConfig,
|
||||
} from './fixtures'
|
||||
import { startMockServer } from './mock-server'
|
||||
import { expect, test } from './test'
|
||||
import { expectVisualSnapshot } from './visual-snapshot'
|
||||
|
||||
const BRANCH_NAME = 'e2e-composer-branch'
|
||||
|
||||
/**
|
||||
* Enough branches to make both the base-branch popover and the convert-branch
|
||||
* list taller than their default height. That is the condition in which the
|
||||
* dialog's own scroll box clips the popover, and that regression is what the
|
||||
* visual snapshots here guard against.
|
||||
*/
|
||||
const EXTRA_BRANCHES = [
|
||||
'feature/alpha-one',
|
||||
'feature/beta-two',
|
||||
'feature/gamma-three',
|
||||
'fix/delta-four',
|
||||
'fix/epsilon-five',
|
||||
'chore/zeta-six',
|
||||
'chore/eta-seven',
|
||||
'spike/theta-eight',
|
||||
'spike/iota-nine',
|
||||
'release/kappa-ten',
|
||||
]
|
||||
|
||||
function createGitRepo(root: string): string {
|
||||
const repo = path.join(root, 'repo')
|
||||
|
||||
fs.mkdirSync(repo, { recursive: true })
|
||||
execFileSync('git', ['init', '--initial-branch=main'], { cwd: repo })
|
||||
execFileSync('git', ['config', 'user.email', 'e2e@example.com'], { cwd: repo })
|
||||
execFileSync('git', ['config', 'user.name', 'Hermes E2E'], { cwd: repo })
|
||||
fs.writeFileSync(path.join(repo, 'README.md'), '# E2E repo\n', 'utf8')
|
||||
execFileSync('git', ['add', 'README.md'], { cwd: repo })
|
||||
execFileSync('git', ['commit', '-m', 'initial'], { cwd: repo })
|
||||
|
||||
for (const branch of EXTRA_BRANCHES) {
|
||||
execFileSync('git', ['branch', branch], { cwd: repo })
|
||||
}
|
||||
|
||||
return repo
|
||||
}
|
||||
|
||||
function configureRepoCwd(hermesHome: string, mockUrl: string, repo: string): void {
|
||||
writeMockProviderConfig(hermesHome, mockUrl)
|
||||
fs.appendFileSync(path.join(hermesHome, 'config.yaml'), `\nterminal:\n cwd: ${repo}\n`, 'utf8')
|
||||
writeEnvFile(hermesHome)
|
||||
}
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
/** A dialog renders as `[data-slot="dialog-content"]` (components/ui/dialog.tsx). */
|
||||
const DIALOG = '[data-slot="dialog-content"]'
|
||||
|
||||
/** Open the worktree dialog with the global ⌘⇧B / ctrl+shift+B hotkey. */
|
||||
async function openWorktreeDialog(): Promise<void> {
|
||||
const page = fixture!.page
|
||||
await page.keyboard.press('Control+Shift+B')
|
||||
await expect(page.locator(DIALOG)).toBeVisible()
|
||||
}
|
||||
|
||||
/** Close the open dialog and wait until it leaves the DOM. */
|
||||
async function closeDialog(): Promise<void> {
|
||||
const page = fixture!.page
|
||||
await page.keyboard.press('Escape')
|
||||
await expect(page.locator(DIALOG)).toHaveCount(0)
|
||||
}
|
||||
|
||||
test.beforeAll(async () => {
|
||||
const sandbox = createSandbox('worktree-branch-status')
|
||||
const repo = createGitRepo(sandbox.root)
|
||||
const mock = await startMockServer()
|
||||
|
||||
configureRepoCwd(sandbox.hermesHome, mock.url, repo)
|
||||
|
||||
const { app, page } = await launchDesktop(buildAppEnv(sandbox))
|
||||
fixture = {
|
||||
app,
|
||||
page,
|
||||
mock,
|
||||
mockUrl: mock.url,
|
||||
sandbox,
|
||||
cleanup: async () => {
|
||||
await app.close().catch(() => undefined)
|
||||
await mock.close()
|
||||
sandbox.cleanup()
|
||||
},
|
||||
}
|
||||
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
|
||||
// The coding rail, and thus the ⌘⇧B worktree dialog, mounts only after the
|
||||
// session resolves a cwd that holds a repo. This happens on the first turn.
|
||||
const composer = page.locator('[contenteditable="true"]').first()
|
||||
await composer.click()
|
||||
await composer.type('create a repo-backed e2e session', { delay: 2 })
|
||||
await page.keyboard.press('Enter')
|
||||
await page.waitForFunction(
|
||||
prompt => (document.querySelector('[data-slot="aui_thread-viewport"]')?.textContent ?? '').includes(prompt),
|
||||
'create a repo-backed e2e session',
|
||||
{ timeout: 15_000 },
|
||||
)
|
||||
await expect(page.locator('.coding-status-bar')).toContainText('main')
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('worktree dialog renders the base-branch picker over the dialog, not clipped by it', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
await openWorktreeDialog()
|
||||
|
||||
// Open the base-branch combobox. With 11 branches, the list is taller than
|
||||
// the space below the trigger. A popover that portals into the dialog's
|
||||
// `overflow-y-auto` box is therefore cut off. This snapshot catches that bug.
|
||||
await page.getByRole('button', { name: /branch off/i }).click()
|
||||
await expect(page.getByPlaceholder('Search branches…')).toBeVisible()
|
||||
await expect(page.getByRole('option', { name: 'feature/alpha-one' })).toBeVisible()
|
||||
|
||||
await expectVisualSnapshot(page, { name: 'worktree-dialog-base-branch-picker', app: fixture!.app })
|
||||
|
||||
// This check does not depend on pixels: the dialog's scroll box must not crop
|
||||
// the painted box of the popover. Measure the geometry, so a headless run
|
||||
// fails on this regression before a person looks at a diff image.
|
||||
const clipped = await page.evaluate(() => {
|
||||
const popover = document.querySelector('[data-slot="popover-content"]')
|
||||
const dialog = document.querySelector('[data-slot="dialog-content"]')
|
||||
|
||||
if (!popover || !dialog) {
|
||||
return { reason: 'missing', clipped: true }
|
||||
}
|
||||
|
||||
const p = popover.getBoundingClientRect()
|
||||
const d = dialog.getBoundingClientRect()
|
||||
const scrolls = window.getComputedStyle(dialog).overflowY
|
||||
|
||||
return {
|
||||
reason: 'measured',
|
||||
// Only a clipping ancestor can crop the popover. The popover is cut when
|
||||
// the dialog scrolls its overflow AND the popover goes past the box of
|
||||
// the dialog.
|
||||
clipped: (scrolls === 'auto' || scrolls === 'scroll' || scrolls === 'hidden') &&
|
||||
(p.bottom > d.bottom + 1 || p.top < d.top - 1 || p.right > d.right + 1 || p.left < d.left - 1),
|
||||
}
|
||||
})
|
||||
|
||||
expect(clipped.clipped, `base-branch popover is clipped by the dialog (${clipped.reason})`).toBe(false)
|
||||
|
||||
await page.keyboard.press('Escape')
|
||||
await closeDialog()
|
||||
})
|
||||
|
||||
test('worktree dialog convert-an-existing-branch sub-view lists the repo branches', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
await openWorktreeDialog()
|
||||
await page.getByRole('button', { name: 'Convert an existing branch' }).click()
|
||||
|
||||
await expect(page.getByPlaceholder('Search branches…')).toBeVisible()
|
||||
await expect(page.getByRole('option', { name: /feature\/alpha-one/ })).toBeVisible()
|
||||
|
||||
await expectVisualSnapshot(page, { name: 'worktree-dialog-convert-branch', app: fixture!.app })
|
||||
|
||||
await closeDialog()
|
||||
})
|
||||
|
||||
test('creating a branch with ctrl-shift-b updates the composer git-status branch and leaves no dialog behind', async ({}, testInfo) => {
|
||||
const page = fixture!.page
|
||||
const codingRow = page.locator('.coding-status-bar')
|
||||
|
||||
await openWorktreeDialog()
|
||||
// Exactly one dialog instance. A second dialog here, hidden or empty, is the
|
||||
// symptom of the double-open bug.
|
||||
await expect(page.locator(DIALOG)).toHaveCount(1)
|
||||
|
||||
const branchInput = page.locator('input[placeholder="e.g. my-feature"]').first()
|
||||
await expect(branchInput).toBeVisible()
|
||||
await branchInput.fill(BRANCH_NAME)
|
||||
// Select a base branch, so this test uses the same path as the user: open the
|
||||
// picker, select a branch, then submit. It does not use the default value.
|
||||
// The keyboard drives this step. The dialog still clips the popover, so a
|
||||
// mouse click on an option is not reliable until that bug is corrected. The
|
||||
// double-open check below is therefore independent of the clipping bug.
|
||||
await page.getByRole('button', { name: /branch off/i }).click()
|
||||
await page.getByPlaceholder('Search branches…').fill('main')
|
||||
await expect(page.getByRole('option', { name: 'main' }).first()).toBeVisible()
|
||||
await page.keyboard.press('Enter')
|
||||
await expect(page.locator('[data-slot="popover-content"]')).toHaveCount(0)
|
||||
|
||||
await page.getByRole('button', { name: 'New worktree' }).click()
|
||||
|
||||
await expect(codingRow).toContainText(BRANCH_NAME, { timeout: 15_000 })
|
||||
|
||||
// The dialog must close and stay closed. No empty second dialog can remain
|
||||
// after the new worktree session starts.
|
||||
await expect(page.locator(DIALOG)).toHaveCount(0)
|
||||
await page.waitForTimeout(2000)
|
||||
await expect(page.locator(DIALOG)).toHaveCount(0)
|
||||
|
||||
await page.screenshot({ path: testInfo.outputPath('composer-branch-after-create.png') })
|
||||
})
|
||||
|
||||
test('ctrl-shift-b opens exactly one worktree dialog when a second composer is on screen', async ({}, testInfo) => {
|
||||
const page = fixture!.page
|
||||
|
||||
// ⌘T / ctrl+T stacks a second session tile. That gives a second live composer
|
||||
// and therefore a second coding rail. Each rail mounted its own
|
||||
// WorktreeDialog, and each rail subscribed to the same global token. One
|
||||
// keypress therefore opened two stacked dialogs, and the dialog the user
|
||||
// dismissed showed an identical empty one behind it. One mount in the sidebar
|
||||
// makes that impossible by structure.
|
||||
await page.keyboard.press('Control+T')
|
||||
await expect(page.locator('.coding-status-bar')).toHaveCount(2, { timeout: 20_000 })
|
||||
|
||||
await page.keyboard.press('Control+Shift+B')
|
||||
await expect(page.locator(DIALOG).first()).toBeVisible()
|
||||
// Wait: let the effect of every subscriber flush before the count.
|
||||
await page.waitForTimeout(500)
|
||||
|
||||
const count = await page.locator(DIALOG).count()
|
||||
await page.screenshot({ path: testInfo.outputPath('worktree-dialog-two-composers.png') })
|
||||
|
||||
expect(count, 'one hotkey press must open exactly one worktree dialog').toBe(1)
|
||||
|
||||
// A dismissal then leaves nothing behind.
|
||||
await closeDialog()
|
||||
})
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* E2E regression: in-page route navigation must preserve the chosen UI scale.
|
||||
*
|
||||
* Desktop is a HashRouter over one file:// document, so every route is a
|
||||
* distinct URL to Chromium's per-URL zoom store, and a route with no record of
|
||||
* its own resolves to the host default (100%). In-page navigation fires no load
|
||||
* or window event, so nothing re-asserted the persisted level: switching
|
||||
* sessions dropped the window to 100% while Appearance kept reading the chosen
|
||||
* scale (#48658, #38854, #79863).
|
||||
*
|
||||
* Prerequisite: `npm run build` must have been run so dist/ exists.
|
||||
*/
|
||||
|
||||
import { type MockBackendFixture, setupMockBackend, waitForAppReady } from './fixtures'
|
||||
import { expect, test } from './test'
|
||||
|
||||
const SCALE = 110
|
||||
|
||||
let fixture: MockBackendFixture | null = null
|
||||
|
||||
async function readZoomPercent(): Promise<number> {
|
||||
return fixture!.page.evaluate(async () => {
|
||||
const desktop = window as unknown as {
|
||||
hermesDesktop: { zoom: { get: () => Promise<{ percent: number }> } }
|
||||
}
|
||||
|
||||
return (await desktop.hermesDesktop.zoom.get()).percent
|
||||
})
|
||||
}
|
||||
|
||||
async function setZoomPercent(percent: number): Promise<void> {
|
||||
await fixture!.page.evaluate(target => {
|
||||
const desktop = window as unknown as {
|
||||
hermesDesktop: { zoom: { setPercent: (percent: number) => void } }
|
||||
}
|
||||
|
||||
desktop.hermesDesktop.zoom.setPercent(target)
|
||||
}, percent)
|
||||
await expect.poll(readZoomPercent).toBe(percent)
|
||||
}
|
||||
|
||||
async function gotoRoute(route: string): Promise<void> {
|
||||
const page = fixture!.page
|
||||
|
||||
await page.evaluate(target => {
|
||||
window.location.hash = target
|
||||
}, route)
|
||||
await page.waitForFunction(target => window.location.hash === `#${target}`, route)
|
||||
}
|
||||
|
||||
test.beforeAll(async () => {
|
||||
fixture = await setupMockBackend()
|
||||
await waitForAppReady(fixture, 120_000)
|
||||
})
|
||||
|
||||
test.afterAll(async () => {
|
||||
await fixture?.cleanup()
|
||||
fixture = null
|
||||
})
|
||||
|
||||
test('a non-default UI scale survives navigation to never-zoomed routes', async () => {
|
||||
await setZoomPercent(SCALE)
|
||||
|
||||
// Routes Chromium has no zoom record for — what opening a new session looks
|
||||
// like to the per-URL store. Pre-fix, the first hop reports 100%.
|
||||
const fresh = `/e2e-zoom-${Date.now()}`
|
||||
|
||||
for (const route of [`${fresh}-one`, `${fresh}-two`, '/settings?tab=config%3Aappearance']) {
|
||||
await gotoRoute(route)
|
||||
await expect.poll(readZoomPercent, { message: `UI scale after navigating to ${route}` }).toBe(SCALE)
|
||||
}
|
||||
})
|
||||
|
||||
test('Cmd/Ctrl+N preserves a non-default UI scale', async () => {
|
||||
const page = fixture!.page
|
||||
|
||||
await gotoRoute('/settings')
|
||||
await setZoomPercent(SCALE)
|
||||
|
||||
await page.evaluate(() => {
|
||||
;(document.activeElement as HTMLElement | null)?.blur()
|
||||
})
|
||||
await page.keyboard.press(process.platform === 'darwin' ? 'Meta+N' : 'Control+N')
|
||||
await page.waitForFunction(() => window.location.hash === '#/')
|
||||
|
||||
await expect.poll(readZoomPercent).toBe(SCALE)
|
||||
})
|
||||
Reference in New Issue
Block a user