feat(harness): agent-drivable dev harness and CI that gates
A coding agent could develop this repo's Go packages and could not develop the application: every path to running YellowJacket ended in a blocking GTK window, so 265 bound methods, 46 events, 33 component directories and 13 stores had exactly one form of verification available — `tsc --noEmit`. The unlock is that `wails dev`'s dev server on :34115 serves the real frontend with the real generated bindings against the same Go backend a desktop window attaches to, so a plain Chromium under Xvfb gets a fully functional app. Four test tiers now exist, cheapest first: - `make ui-test` — 313 Vitest tests in a real browser in ~2 s, no app, no backend, no display. Works because `frontend/wailsjs/` is a pure passthrough to `window.go`/`window.runtime`, so faking just those two globals runs the real bindings and the real store code. - `make test` — services in-process, asserting on the payload the frontend would receive, via a new `events.Emit` wrapper. - `make dev-headless` + `playwright-cli` — the real app, driven interactively, with an event bridge on `window.__yjEvents` and a dev-only control surface at `/__test/`. - `make e2e` — 19 of those flows frozen as Playwright specs. `events.Emit(ctx, …)` replaces all 35 direct `runtime.EventsEmit` call sites: wails' `getEvents` `log.Fatalf`s on any context without its runtime, so those paths could not run under test and a background worker could take the app down. Four packages had each hand-rolled the same guard; nine more guarded on `ctx != nil`, which does not help. `TestNoDirectRuntimeEmits` fails the build on a new one. Fixtures are generated, not committed (`make testdata`), and seeds are built by *running the app* — never by hand-writing config and DB rows, which would be a second description of a valid YJ_HOME. `.gitea/workflows/ci.yml` is the first workflow here that tests anything; the other three only package, so `gitea_ci` reported only packaging jobs and misled anyone asking whether a push was healthy. Both jobs were prototyped to green in a bare ubuntu:24.04 container before the YAML was written, which immediately caught `make lint` linting three configurations that nothing builds: all three passes omitted `webkit2_41`, so wails resolved webkit2gtk-4.0 — which Arch still ships and Ubuntu 24.04 dropped. Operational instructions live in `.pi/skills/yellowjacket-dev/`, measured discoveries in `.planning/NOTES.md`, and architecture in `CLAUDE.md` — split by tense, not by topic, because a topical split gives every new fact two plausible homes. `make skill-check` fails a commit if the skill cites a make target that does not exist.
This commit is contained in:
@@ -0,0 +1,172 @@
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
|
||||
import { test as base, expect, type Page } from '@playwright/test';
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
/** The same bridge `playwright-cli` loads, so an exploratory session and
|
||||
* a committed spec see an identical page. */
|
||||
const INIT_SCRIPT = resolve(here, '../../.playwright/init-events.js');
|
||||
|
||||
/**
|
||||
* The 90-second fixture track (`cmd/gentestdata`, case `edge-lengths`).
|
||||
*
|
||||
* Every other fixture is 2–6 seconds, which is shorter than the time a
|
||||
* spec takes to click something — a "pause it" test against one of
|
||||
* those races the track finishing and fails on a UI that is correct.
|
||||
*/
|
||||
export const LONG_TRACK = 'Long Player';
|
||||
|
||||
/** Shape of the recorder installed by .playwright/init-events.js. */
|
||||
export type YjEvent = {
|
||||
seq: number;
|
||||
name: string;
|
||||
data: unknown[];
|
||||
dir: 'in' | 'out';
|
||||
t: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Await a backend event instead of a timeout.
|
||||
*
|
||||
* Half of this app is push-driven, and the events that matter
|
||||
* (scan progress, job updates, playback state) arrive whenever the
|
||||
* backend gets to them. `waitForEvent` resolves against events already
|
||||
* buffered as well as future ones, so there is no race between doing
|
||||
* the thing and starting to listen.
|
||||
*/
|
||||
export async function waitForEvent(
|
||||
page: Page,
|
||||
name: string,
|
||||
opts: { timeoutMs?: number; since?: number } = {},
|
||||
): Promise<YjEvent> {
|
||||
return page.evaluate(
|
||||
([n, o]) => window.__yjEvents.wait(n as string, o as object),
|
||||
[name, { timeoutMs: 10_000, ...opts }] as const,
|
||||
) as Promise<YjEvent>;
|
||||
}
|
||||
|
||||
/** Drop the event buffer. Never re-register a recorder: listeners
|
||||
* survive across evaluate calls and a second recorder double-counts. */
|
||||
export async function resetEvents(page: Page): Promise<void> {
|
||||
await page.evaluate(() => void window.__yjEvents.reset());
|
||||
}
|
||||
|
||||
/** name -> count, for asserting on (or debugging) what actually fired. */
|
||||
export async function eventNames(
|
||||
page: Page,
|
||||
): Promise<Record<string, number>> {
|
||||
return page.evaluate(() => window.__yjEvents.names());
|
||||
}
|
||||
|
||||
/**
|
||||
* Call a bound Go method with a timeout.
|
||||
*
|
||||
* Wrong argument types make the backend log "error parsing arguments"
|
||||
* and never fire the callback, so an unguarded call hangs until the
|
||||
* whole spec times out with no clue why. This fails in seconds and
|
||||
* says where to look.
|
||||
*/
|
||||
export async function callBinding<T = unknown>(
|
||||
page: Page,
|
||||
path: string,
|
||||
args: unknown[] = [],
|
||||
timeoutMs = 10_000,
|
||||
): Promise<T> {
|
||||
return page.evaluate(
|
||||
([p, a, t]) =>
|
||||
window.__yjEvents.call(p as string, a as unknown[], t as number),
|
||||
[path, args, timeoutMs] as const,
|
||||
) as Promise<T>;
|
||||
}
|
||||
|
||||
/** Thin client for the dev-only /__test/ surface (backend/testctl). */
|
||||
export class TestCtl {
|
||||
constructor(private readonly baseURL: string) {}
|
||||
|
||||
private async req(path: string, init?: RequestInit) {
|
||||
const res = await fetch(`${this.baseURL}${path}`, {
|
||||
signal: AbortSignal.timeout(120_000),
|
||||
...init,
|
||||
});
|
||||
const body = await res.json();
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`testctl ${path}: ${body.error ?? res.status}`);
|
||||
}
|
||||
|
||||
return body;
|
||||
}
|
||||
|
||||
health() {
|
||||
return this.req('/__test/health');
|
||||
}
|
||||
|
||||
snapshot(name: string) {
|
||||
return this.req(`/__test/db/snapshot?name=${name}`, { method: 'POST' });
|
||||
}
|
||||
|
||||
restore(name: string) {
|
||||
return this.req(`/__test/db/restore?name=${name}`, { method: 'POST' });
|
||||
}
|
||||
|
||||
emit(name: string, ...data: unknown[]) {
|
||||
return this.req('/__test/emit', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ name, data }),
|
||||
});
|
||||
}
|
||||
|
||||
sql(sql: string, args: unknown[] = []) {
|
||||
return this.req('/__test/sql', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ sql, args }),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export const test = base.extend<{ app: Page; testctl: TestCtl }>({
|
||||
/** A page with the event bridge installed and the app loaded and
|
||||
* actually talking to the backend — not merely DOM-ready, which is
|
||||
* earlier and lies. */
|
||||
app: async ({ page, baseURL }, use) => {
|
||||
await page.addInitScript({ path: INIT_SCRIPT });
|
||||
await page.goto(baseURL!);
|
||||
await page.evaluate(() => window.__yjEvents.ready(20_000));
|
||||
await use(page);
|
||||
},
|
||||
|
||||
testctl: async ({ baseURL }, use) => {
|
||||
await use(new TestCtl(baseURL!));
|
||||
},
|
||||
});
|
||||
|
||||
export { expect };
|
||||
|
||||
declare global {
|
||||
interface Window {
|
||||
__yjEvents: {
|
||||
version: number;
|
||||
seq: number;
|
||||
log: YjEvent[];
|
||||
reset(): number;
|
||||
all(name?: string): YjEvent[];
|
||||
count(name?: string): number;
|
||||
last(name?: string): YjEvent | null;
|
||||
since(seq: number): YjEvent[];
|
||||
names(): Record<string, number>;
|
||||
wait(
|
||||
name: string,
|
||||
opts?: {
|
||||
timeoutMs?: number;
|
||||
since?: number;
|
||||
match?: (data: unknown[], entry: YjEvent) => boolean;
|
||||
},
|
||||
): Promise<YjEvent>;
|
||||
ready(timeoutMs?: number): Promise<boolean>;
|
||||
call(path: string, args?: unknown[], timeoutMs?: number): Promise<any>;
|
||||
};
|
||||
go: Record<string, Record<string, Record<string, (...a: any[]) => any>>>;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
import type { FullConfig } from '@playwright/test';
|
||||
|
||||
/**
|
||||
* Fail fast, and fail legibly.
|
||||
*
|
||||
* Without this the first spec dies on a connection refused deep inside
|
||||
* a `page.goto`, which reads like a Playwright problem rather than "you
|
||||
* forgot to start the app". Checking /__test/health also verifies the
|
||||
* two things every spec below assumes: that the control surface is
|
||||
* mounted (dev build + YJ_TESTCTL=1) and that the app is pointed at the
|
||||
* seeded fixture library rather than someone's real collection.
|
||||
*/
|
||||
const HELP = `
|
||||
The app is not running, or is not a seeded dev build. Start it with:
|
||||
|
||||
make testdata # once — generates the fixtures
|
||||
make sandbox-seed NAME=default # once — builds a seed by running the app
|
||||
make dev-headless SEED=default # starts in the background and returns
|
||||
|
||||
and stop it afterwards with 'make dev-stop'.
|
||||
`;
|
||||
|
||||
export default async function globalSetup(config: FullConfig) {
|
||||
const baseURL =
|
||||
config.projects[0]?.use?.baseURL ?? 'http://localhost:34115';
|
||||
|
||||
let health: any;
|
||||
|
||||
try {
|
||||
const res = await fetch(`${baseURL}/__test/health`, {
|
||||
signal: AbortSignal.timeout(5_000),
|
||||
});
|
||||
|
||||
if (!res.ok) throw new Error(`health returned ${res.status}`);
|
||||
|
||||
health = await res.json();
|
||||
} catch (err) {
|
||||
throw new Error(
|
||||
`e2e: no healthy backend at ${baseURL} (${String(err)})\n${HELP}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (!health.libraries?.length) {
|
||||
throw new Error(
|
||||
`e2e: backend has no library configured — specs would land on ` +
|
||||
`the first-run wizard, which intercepts every pointer event.\n${HELP}`,
|
||||
);
|
||||
}
|
||||
|
||||
const tracks = health.counts?.tracks ?? 0;
|
||||
|
||||
if (tracks < 1) {
|
||||
throw new Error(
|
||||
`e2e: backend library is empty (${tracks} tracks).\n${HELP}`,
|
||||
);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`e2e: backend up — ${tracks} tracks in ` +
|
||||
`${health.libraries.map((l: any) => l.name).join(', ')}`,
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user