yonlu/yellowjacket
1
1
Fork 0
Code Issues 52 Pull Requests 4 Actions Packages Projects Releases 12 Wiki Activity

feat(harness): agent-drivable dev harness and CI that gates
Build & publish Arch package / arch-package (push) Successful in 2m8s
Details
CI / check (push) Failing after 1m56s
Details
CI / e2e (push) Skipped
Details
Search index maintenance / maintain-index (push) Successful in 13s
Details

Browse Source 
A coding agent could develop this repo's Go packages and could not
develop the application: every path to running YellowJacket ended in a
blocking GTK window, so 265 bound methods, 46 events, 33 component
directories and 13 stores had exactly one form of verification
available — `tsc --noEmit`.

The unlock is that `wails dev`'s dev server on :34115 serves the real
frontend with the real generated bindings against the same Go backend a
desktop window attaches to, so a plain Chromium under Xvfb gets a fully
functional app. Four test tiers now exist, cheapest first:

- `make ui-test` — 313 Vitest tests in a real browser in ~2 s, no app,
  no backend, no display. Works because `frontend/wailsjs/` is a pure
  passthrough to `window.go`/`window.runtime`, so faking just those two
  globals runs the real bindings and the real store code.
- `make test` — services in-process, asserting on the payload the
  frontend would receive, via a new `events.Emit` wrapper.
- `make dev-headless` + `playwright-cli` — the real app, driven
  interactively, with an event bridge on `window.__yjEvents` and a
  dev-only control surface at `/__test/`.
- `make e2e` — 19 of those flows frozen as Playwright specs.

`events.Emit(ctx, …)` replaces all 35 direct `runtime.EventsEmit` call
sites: wails' `getEvents` `log.Fatalf`s on any context without its
runtime, so those paths could not run under test and a background
worker could take the app down. Four packages had each hand-rolled the
same guard; nine more guarded on `ctx != nil`, which does not help.
`TestNoDirectRuntimeEmits` fails the build on a new one.

Fixtures are generated, not committed (`make testdata`), and seeds are
built by *running the app* — never by hand-writing config and DB rows,
which would be a second description of a valid YJ_HOME.

`.gitea/workflows/ci.yml` is the first workflow here that tests
anything; the other three only package, so `gitea_ci` reported only
packaging jobs and misled anyone asking whether a push was healthy.
Both jobs were prototyped to green in a bare ubuntu:24.04 container
before the YAML was written, which immediately caught `make lint`
linting three configurations that nothing builds: all three passes
omitted `webkit2_41`, so wails resolved webkit2gtk-4.0 — which Arch
still ships and Ubuntu 24.04 dropped.

Operational instructions live in `.pi/skills/yellowjacket-dev/`,
measured discoveries in `.planning/NOTES.md`, and architecture in
`CLAUDE.md` — split by tense, not by topic, because a topical split
gives every new fact two plausible homes. `make skill-check` fails a
commit if the skill cites a make target that does not exist.
This commit is contained in:
logan
2026-08-10 23:20:42 -04:00
parent 65333857e2
commit 5ca6cad45a
117 changed files with 14585 additions and 262 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/NOTES.md
+470
View File
@@ -168,3 +168,473 @@ built from scratch rather than restored; the old implementation gated
several component code paths in ad hoc ways. A separate
`deep_catalog_enabled` backend flag briefly existed for the same idea and
was removed earlier, when the dump importer left the app binary.
## The dev server is a real, drivable app (verified 2026-08-10)
`wails dev` binds an HTTP + WebSocket server on `localhost:34115`
(`internal/frontend/devserver/devserver.go`) that serves the frontend
with the generated bindings on `window.go` and bridges every call and
every `runtime.EventsEmit` to the **same** Go backend the desktop
window uses. A browser pointed at it is not a mock — measured:
`queue.Queue.GetState()` returned real JSON, `SetVolume(42)` produced a
real `VolumeChanged`. Multiple clients are supported by design
(`notifyExcludingSender` fans events to the other web clients *and* the
desktop frontend). See plan 005.
Four facts that cost time to find:
- **The GTK window cannot be suppressed.** `devserver.Run` ends in
`d.Frontend.Run(ctx)` with no flag to skip it, so headless needs
Xvfb. Nobody upstream has a way around this.
- **Build the dev binary directly.** `app_dev.go` parses `-devserver`,
`-assetdir`, `-loglevel` from `os.Args`, so
`go build -tags "dev webkit2_41"` plus those flags gives the same
server with no watcher, no reload broadcast and one PID.
- **A binding call with wrong argument types hangs forever.** The
backend logs `error parsing arguments` and never fires the callback,
so the caller's promise never settles. Always use a timeout; the app
log is the only place the reason shows up.
- **`dbus-run-session` does not break audio, and fixes MPRIS.** It
replaces the bus, not `/run/user/1000`, so PulseAudio still works and
`org.mpris.MediaPlayer2.yellowjacket` registers on the private bus.
## Playwright's WebKit does not run on Arch (measured 2026-08-10)
`playwright-cli install-browser webkit` downloads fine and then fails to
link: its Linux build wants Ubuntu 24.04 libraries (`libicu74`,
`libWPEWebKit-2.0.so.1`, `libflite`) that Arch does not provide, and the
dependency check emits `apt-get` advice. So `--browser=webkit` — the
cheap way to approximate the WebKit2GTK renderer we actually ship — is a
CI-only capability. Local browser work is Chromium, which is unaffected.
## The fixture library is generated, and generated by our own writers
`test_data/music_library_test/` is produced by `cmd/gentestdata`
(`make testdata`), not committed — 31 tracks across MP3, FLAC, Ogg
Vorbis and WAV, ~700 KB, ~1 s to build. Two rules keep it honest:
- **Tags are written by `backend/tagwriter`, not by ffmpeg.** ffmpeg
only encodes (with `-map_metadata -1`); every tag comes from the same
writers the app uses, so a fixture and the reader under test cannot
drift into agreeing with each other and disagreeing with reality.
`tagwriter.WriteFileTags` exists for this — it is the format switch
`WriteUntrackedFileTags` already had, lifted out so tooling with no
app to construct can call it.
- **The manifest hash covers the spec, not the bytes.** ffmpeg stamps
encoder version strings, so identical specs produce different bytes
on different ffmpeg builds. `test_data/music_library_test.manifest.json`
hashes paths, formats, durations, tags and cover identity instead,
and lives *outside* the library root so the scanner never sees it.
Fixtures are selected in tests by *case* (`testfixtures.CaseCoverDedup`,
`CaseUnicode`, `CaseDuplicates`, …) rather than by path. Deliberately
malformed files live in a sibling root, `test_data/music_library_broken/`
— the clean library's track count has to stay at exactly 31 for seeds
to be verifiable, and a zero-byte `.flac` in the scanned tree used to
get swept into `testFlacFiles` and fail the duration parser.
## WAV tags are write-only (found 2026-08-10)
`backend/tagwriter` writes WAV tags into a RIFF `id3 ` chunk, and
`backend/metadata` reads through `dhowden/tag`, which recognises MP3,
FLAC, OGG, MP4 and DSF and has **no RIFF parser at all**. So every tag
the app writes to a WAV is invisible to the app that wrote it, and WAV
tracks always scan in untitled — visible in the Artists view, where the
fixture library's WAV tracks produce no "Field Recordings" artist.
The fix is small (unwrap the `id3 ` chunk and hand the payload to
`tag.ReadFrom`) but was out of scope for plan 005.
`TestWAVTagsAreNotReadableYet` asserts the gap so that fixing the
reader turns into a failing test rather than nothing at all.
## The headless harness: how to run this app without a window
`make dev-headless [SEED=<name>]` starts the app in the background and
returns; `make dev-stop`, `make dev-logs`. It runs the dev *binary*
(`go build -tags "dev webkit2_41"`), not `wails dev`, under
`dbus-run-session -- xvfb-run`. See plan 005 and
`scripts/dev-headless.sh` for why each of those three is load-bearing.
**Seeds are built by running the app.** The first-run wizard's
dismissal condition is not a config file — it is
`GetAllLibrariesWithTrackCounts()` returning something — so
`make sandbox-seed NAME=<n>` boots a fresh `YJ_HOME`, calls the real
`AddLibrary` binding through `playwright-cli`, waits for the real scan
to reach the manifest's track count, stops the app with SIGTERM so the
shutdown hooks persist state, and tars the result. Never hand-write a
`config.toml` and DB rows: that is a second description of a valid
`YJ_HOME`, free to drift, exactly like the migration chain was.
Seeding points `YJ_CORE_INDEX_URL` at a dead address on purpose, so no
seed depends on what the explore artifact server was serving that day.
Two parsing traps, both already paid for:
- `playwright-cli` echoes the evaluated source back after the result,
so scraping its output for bare digits picks up numbers from your own
JavaScript. Return a tagged sentinel (`'YJTRACKS' + '=' + n`) and
grep for that.
- Waiting on a fixed sleep or on a scan event is worse than waiting on
the observable outcome. Polling the track count the app itself
reports also validates the fixture manifest against the real scanner.
## Restoring a database needs foreign keys *off*, not deferred
`/__test/db/restore` (`backend/testctl`) copies every ordinary table
out of an ATTACHed snapshot. The obvious implementation — one
transaction with `PRAGMA defer_foreign_keys = ON` — fails at COMMIT
with a bare `FOREIGN KEY constraint failed (787)` that names nothing.
Deferring postpones the *check*; it does not stop `ON DELETE CASCADE`
from firing. Tables are copied in name order, which is not dependency
order, so `DELETE FROM libraries` cascades away rows of a child table
that was already restored earlier in the loop, and the final state is
genuinely inconsistent.
`PRAGMA foreign_keys` is a no-op inside a transaction, so it has to be
set on the connection around it. That is safe only because the writer
is a single connection (`SetMaxOpenConns(1)`); the restore re-enables
enforcement afterwards and runs `PRAGMA foreign_key_check`, so a bad
restore is reported instead of left in place.
`TestRestoreRoundTrip` pins it.
Two related traps in the same path: FTS5 virtual tables cannot be
written with `SELECT *` and their shadow tables (`_data`, `_idx`,
`_docsize`, `_config`) must be rebuilt rather than copied — but the
prefix test that excludes them must not swallow `explore_index`, an
ordinary table whose name is a prefix of two virtual ones.
## The event bridge hooks EventsNotify, not EventsOn
`.playwright/init-events.js` records backend events by wrapping
`window.wails.EventsNotify`. That is the single choke point: wails'
`ipc_websocket.js` does `case "n": window.wails.EventsNotify(msg)` and
fans out to listeners from there, so one wrap captures all 46 events
whether or not the app subscribes to them. Wrapping `EventsOn` would
have needed 46 registrations and would have missed anything the app
does not listen for.
`window.wails` does not exist when an initScript runs, so the script
installs an accessor on `window` and wraps at assignment time (wails'
`main.js` does a plain `window.wails = {...}`), then redefines the
property as a plain value so nothing downstream can tell.
The buffer lives on `window.__yjEvents` with `wait()`, `reset()`,
`names()`, a `ready()` that resolves only when a binding actually
round-trips, and a `call()` that **times out** — a binding invoked with
wrong argument types never fires its callback, and a 2s rejection
naming `.dev/app.log` is worth more than an infinite hang.
## Small harness traps, each of which cost a cycle
- **Paths in `.playwright/cli.config.json` resolve against the config
file's directory**, not the repo root. `".playwright/init-events.js"`
becomes `.playwright/.playwright/init-events.js`.
- **`playwright-cli` and `@playwright/test` have separate browser
caches.** The CLI working is no guarantee `npx playwright test` can
launch; it needs its own `npx playwright install chromium` (the
runner wants `chrome-headless-shell`, which the CLI never fetched).
- **`getByRole('button', { name: 'Play' })` also matches "Add queue to
playlist".** Accessible-name matching is substring by default; the
transport controls need `exact: true`.
- **Every fixture track except one is 2–6 seconds.** A spec that plays
a track and then clicks pause races the track ending and fails
against a correct UI. Use the 90-second `Long Player`
(`edge-lengths`), exported as `LONG_TRACK` from `e2e/support`.
- **`e2e/` needs `"type": "module"`** or Playwright transpiles the
specs to CJS and every `import.meta` in the support code throws
"Cannot use 'import.meta' outside a module" — reported as
"No tests found".
## The component tier fakes two globals, and that is all it fakes
`frontend/wailsjs/` is a pure passthrough: every generated binding is
`window['go'][svc][Type][Method](args)` and every runtime call is
`window.runtime.X(...)`. So `frontend/test/support/wails-fake.ts`
replaces those two globals and nothing else, and the tests then run the
*real* generated bindings and the *real* store code. No module mocking,
and no second description of the Wails layer to drift from the first —
the same discipline `sql/schemas/` and the seeds get.
The dispatcher mirrors wails' own `internal/frontend/runtime/desktop/
events.js`, which matters in two places: listeners registered with
`maxCallbacks` expire and are removed mid-iteration, and `EventsEmit`
from the frontend notifies local JS listeners *before* it notifies Go
(so a frontend emit is observable in-page).
Four things that cost time:
- **Store singletons are constructed at module import**, so the fake
must be installed from `setupFiles`, and any store that reads config
in its constructor loads before a test can stub it. `test/setup.ts`
carries import-time defaults for exactly those. Without them a store
caches `undefined` where Go would have sent `[]`, and four
components then crash on `.length` — which reads as a component bug
and is not one.
- **`vitest.config.mts`, not `.ts`.** The repo's vite config is
`vite.config.mts`; a `.ts` sibling cannot import it, and `mergeConfig`
is how the `@go`/`@store`/`@components` aliases get reused rather
than restated.
- **Vitest 4 takes a provider factory, not a string.**
`provider: playwright()` from `@vitest/browser-playwright`, which is
a third package beyond `vitest` and `@vitest/browser`, and needs its
own `npx playwright install chromium` — a third browser cache after
`playwright-cli`'s and `@playwright/test`'s.
- **Pre-bundle Web Awesome or Vite reloads mid-test.** Its components
are one deep import per element; discovering them lazily makes Vite
re-optimise and reload the page underneath a running test. The glob
`@awesome.me/webawesome/dist/components/*/*.js` in `optimizeDeps.
include` settles it.
Screenshots need the app's surface, not the default white page: the
setup file imports `@store/theme-store` for its side effect (it applies
the `--yj-*` ramp to `:root`) and sets the two `index.css` declarations
that matter, or a component renders white-on-white and the baseline is
blank. And a `@lit-labs/virtualizer` list never produces two identical
frames, so `toMatchScreenshot` on `<queue-panel>` fails with "could not
capture a stable screenshot" rather than a diff — assert its rows
instead.
## Binding drift is now checked, and it is fast
`frontend/wailsjs/` is generated by `wails`, **not** by `go generate`,
so the pre-commit codegen check never covered it: a renamed Go bound
method or struct field first showed up at runtime, in a window, as a
call that never settles. `wails generate module` (v2.10.2) rebuilds it
in ~1.5 s, which is cheap enough to gate a commit on —
`scripts/bindings-check.sh`, `make bindings-check`, and a `lefthook`
pre-commit entry. Verified by renaming `queue.GetState` and watching it
fail.
One quirk: the generator rewrites the three `wailsjs/runtime/` files as
mode 755 every run. That is not drift, so the check compares with
`git -c core.fileMode=false` and restores the modes afterwards.
## Emitting an event is now one call, and it cannot kill the process
`events.Emit(ctx, name, data...)` (`backend/events/emit.go`) is the
only supported way to push a Wails event; `TestNoDirectRuntimeEmits`
fails the build on any other `runtime.EventsEmit` in the tree.
The reason is that `runtime.getEvents` (wails `runtime.go:47`)
`log.Fatalf`s — `os.Exit`, unrecoverable — whenever the context lacks
its internal `"events"` value. That is any `context.Background()`, so
in-process service tests were impossible and background workers that
outlived their context could take the app down at launch.
Four packages had independently discovered this and hand-rolled a
guard (`library.emit`, `download.emit`, `playlist.emitEvent`, and
`autotagservice.emitEvent` with a whole `ctxReady` field). Nine other
sites guarded on `ctx != nil`, **which does not help** — a non-nil
context without the runtime is exactly the fatal case. The wrapper
replicates wails' own precondition once and drops at debug level.
Three things worth knowing:
- **The test sink rides in the context**, `events.WithSink(ctx, rec)`,
not in a package global. A global cannot survive `t.Parallel()` and
would put a mutex on every production emit.
- **`events.Deliver` is `Emit` that returns `ErrNoRuntime`**, and has
exactly one caller: `/__test/emit` in `backend/testctl`. That
endpoint exists to *impersonate* a backend emit, so answering `200`
for an event that reached nobody would send you debugging the
frontend for a backend no-op. Ordinary emitters want `Emit`.
- **Enforcement is a walk of the tree, not a lint rule.**
golangci-lint runs once per build configuration, so a stray emit in
an `indexbuild`- or `dev`-tagged file is only visible to the pass
that compiles it. One text walk sees all three, plus anything tagged
out entirely. Two traps if you touch that test: the needle has to be
built at runtime or the file matches itself, and it has to be the
qualified selector (`.EventsEmit(`) or it matches the test's own
function name.
What it unblocks is a fourth test tier — services, in-process, with no
app: `backend/queue/emit_test.go`, `backend/config/emit_test.go`,
`backend/playlist/emit_test.go` assert on the payload the *frontend
receives*, which had never been covered. Two gotchas found writing
them: `config.Save` refuses to write a config that was never
`Load`ed (so a test that only calls `applyDefaults` sees its second
setter fail, not its first), and `queue.SetQueue` resolves anything
over `initialBatchSize` in a background phase, so assert with
`rec.Wait` rather than immediately after the call.
## Docs are split by tense, and the split is checkable
Three places now describe this repo, and the rule for which one a new
paragraph goes in is **grammatical, not topical** — a topical split
("architecture here, testing there") is what rots, because every new
fact has two plausible homes.
- `.planning/NOTES.md` — **past**: measured, dated, append-only.
- `CLAUDE.md` — **present**: what the system is, and why.
- `.pi/skills/yellowjacket-dev/` — **imperative**: what to run, in what
order, and what it looks like when it fails.
So the skill carries the checklist for a schema change and CLAUDE.md
carries the reasoning behind the two-file rule; the skill carries the
headless lifecycle and CLAUDE.md carries only the invariant that seeds
are produced by running the app. Phase 6 deleted about half of
CLAUDE.md's harness section on those grounds.
`make skill-check` (`scripts/skill-check.sh`, pre-commit) makes it
enforceable: every `make <target>` mentioned under `.pi/**/*.md` must
exist. That is the actual anti-drift mechanism — the Makefile is the
source of truth for *how* to invoke something and the skill only decides
*which*, so a renamed target fails a commit instead of sending an agent
confidently at a command that no longer exists. A skill that documents a
command slightly wrong is worse than no skill.
One shell trap it cost: under `set -euo pipefail`,
`x="$(make -pqRr | awk … )"` sinks the whole assignment, because
`make -q` exits non-zero whenever a target is out of date and `pipefail`
propagates that. Wrap it in `{ …; || true; }`.
## The skill was followed cold, and lost time in exactly one place
An agent that did not write `.pi/skills/yellowjacket-dev/` brought the
app up from a wiped `.dev/` and no fixture library, drove a flow the
skill does not describe (open the queue panel, toggle shuffle, assert
on `QueueModeChanged`, confirm against `queue.Queue.GetState`) and
stopped it — about a minute of wall clock, no dead ends. All four
tiers then re-ran green from that cold state: 313 ui-test, 0 issues ×
3 lint configurations, 3 test passes, 19/19 e2e.
The one expensive thing was a genuine config bug, not a doc error.
`.playwright/cli.config.json` had `outputDir: "../.playwright-cli"`,
written on the belief — which `references/harness.md` stated as a flat
rule — that every path in that file resolves against the config file's
directory. **Only `initScript` does.** `outputDir` resolves against the
shell's cwd, so every snapshot and console log was landing in
`/home/logan/Development/.playwright-cli`, one level *above* the repo:
outside `.gitignore`, outside `find`, and invisible to the obvious
`ls .playwright-cli/`. That directory still held a stale snapshot from
the previous session, so the obvious `ls -t | head -1` returned it
silently, and the transport buttons appeared to have lost their
accessible names — a fabricated regression in phase 3's work that took
a DOM walk to disprove. Reading a *stale* artifact is much worse than
reading none, because it answers.
Four smaller corrections, all now in the skill:
- `make sandbox-seed` already depends on `make testdata`, so listing
both made the fixture step look separately required. It also takes
~10 s with warm caches, not the ~30 s claimed.
- `make ui-setup` and `make e2e-setup` are once-per-clone
prerequisites and are *not* dependencies of `make ui-test` /
`make e2e`. The skill never mentioned them; on a fresh clone both
fail with a missing-browser error that reads like a broken test.
This matters for CI, which has no warm caches by definition.
- `snapshot` prints a *path*, not the tree. Not said anywhere.
- `make dev-stop` does not close the browser session;
`playwright-cli -s=yj close` is a separate step.
And one place the tooling taught the opposite of the skill:
`scripts/dev-headless.sh`'s own success banner suggested
`eval "async () => await window.go.queue.Queue.GetState()"` — a bare
`window.go` call with no timeout, which is precisely the hang the
banner's next paragraph warns about. A gotcha documented in prose and
contradicted by the copy-pasteable line three inches above it will lose
every time; the banner now prints the `__yjEvents.call` form.
## An ALSA null PCM is enough for CI audio, and it clocks
Phase 7's job 2 needs playback to actually advance, because
`e2e/specs/playback.spec.ts` asserts the elapsed clock moves — a
missing audio device fails it in a way that reads like flake, since
`app.go` joins `InitSpeaker` failure into `startupErr` and lets
everything else work.
Measured locally, with PulseAudio made unreachable
(`XDG_RUNTIME_DIR` pointed at an empty dir, `PULSE_SERVER=none`) and
`ALSA_CONFIG_PATH` pointing at four lines:
```
</usr/share/alsa/alsa.conf>
pcm.!default { type null }
ctl.!default { type null }
```
`InitSpeaker` succeeded in 36 ms and all six playback/queue specs
passed, including "the elapsed time advances". oto/v3 talks to
libasound directly, and ALSA's `null` plugin advances its pointer on a
timer rather than discarding instantly, so beep's stream is consumed at
real-time rate. **No PipeWire, no PulseAudio and no daemon of any kind
is required in the container** — one env var and a file.
Also found while setting this up: `scripts/dev-headless.sh` does *not*
set `YJ_CORE_INDEX_URL`; only `scripts/seed-sandbox.sh` does. So a
seeded run started by hand still reaches for the real explore artifact.
Harmless locally, a network dependency and a minute of wall clock in
CI — job 2 must set the dead-address override itself.
## CI was prototyped in a container before it was written, and it found a real bug
Both jobs of `.gitea/workflows/ci.yml` were built as shell scripts and
run to green in a bare `ubuntu:24.04` container (`docker run -v
repo:/src -v cache:/cache`) before a line of YAML existed, then the
YAML was transcribed back out of the workflow and re-run in the same
container to prove the transcription. That is worth the extra half
hour on a self-hosted runner: the alternative is push-and-see, and a
Gitea Actions run that never starts looks exactly like one that passed.
**`make lint` was linting three configurations nothing builds.** All
three passes omitted `webkit2_41`, so wails resolved `webkit2gtk-4.0`.
Arch still ships `webkit2gtk-4.0.pc`, so it passed locally and had
done for the life of the repo; Ubuntu 24.04 dropped 4.0, and the
**`dev` pass** fails there — wails' own `app_dev.go` is `dev`-tagged
and drags in the 4.0 assetserver, which the other two passes never
compile. The tag sets now match `make test` exactly
(`webkit2_41`, `webkit2_41 indexbuild`, `webkit2_41 dev`). Still 0
issues × 3 on Arch, and now 0 × 3 on Ubuntu too. Note what this means:
"lint passes" and "the thing lint compiled is the thing we ship" were
different claims, and only a second distro could tell them apart.
Five smaller container facts, all now comments in the workflow:
- **`libasound2-dev`, not just `libasound2t64`.** oto/v3 dies at
`pkg-config --cflags -- alsa` before a line is compiled.
- **`PLAYWRIGHT_BROWSERS_PATH` unifies the location, not the
revisions.** `@playwright/cli` bundles its own `playwright-core`
pinned to a different Chromium build than `e2e/`'s
`@playwright/test`, so both must install into the shared directory.
Installing one gives the other "Browser chromium is not installed;
expected executable at …/chromium-1237/…". The "three separate
browser caches" trap survives being pointed at one path.
- **`git config --global --add safe.directory`** or `bindings-check`
fails on a clone the container user does not own.
- **The runner already mounts and exports `GOMODCACHE`, `GOCACHE` and
`GOLANGCI_LINT_CACHE`** for every job via `container.options`, and
`valid_volumes` is a glob over the cache root
(`/home/logan/docker/gitea/data/runner/cache/**`), so new caches need
no runner-side change. Only the Node-side ones had to be declared.
- **The fixture hash is deterministic per ffmpeg, not across
versions**: `5425fbb454a2` on Arch (ffmpeg n8.1.2), `599a8dd4f152`
on Ubuntu 24.04. Nothing asserts a literal hash, so this is
harmless — but a test that pinned one would be portable only by
accident.
**Setting `YJ_CORE_INDEX_URL` for the *app* run, not just for seeding,
is worth 8x on the suite.** `scripts/dev-headless.sh` never set it —
only `seed-sandbox.sh` did — so a seeded local run still fetches the
real explore artifact, and `testctl.spec.ts`'s restore then copies
every table of a database full of catalogue: 42 s locally, versus a
whole 19-spec suite in 7.3 s in CI with the artifact stubbed out.
## Playwright's WebKit passes, so it gates
19/19, in the same container, ~11 s on top of Chromium's ~7 s. It had
never been run anywhere before — Arch cannot start it — so the honest
default would have been advisory. Running it once in a throwaway
container turned a coin flip into a decision: it is a **required**
step.
Two things make that safe rather than brave. Nothing in `e2e/`
compares pixels — every assertion is an event payload, a `data-testid`,
an attribute or backend state, and the `toMatchScreenshot` baselines
live in the Chromium-only Vitest tier — so a WebKit failure cannot be
antialiasing noise; it is an engine difference in custom-element
upgrade, a11y-tree shape or event ordering, which is exactly the
WebKit2GTK signal we otherwise have no way to get. And it is cheap
enough that the earlier plan to scope it (skip `testctl.spec.ts`,
which tests Go and has no engine content) is not worth the
complexity at 11 s.
.planning/plans/active/005-agent-development-harness.md
+692
View File
@@ -0,0 +1,692 @@
# 005 — Agent development harness
**Status:** complete — all seven phases shipped
**Branch:** —
**Created:** 2026-08-10
**Follows:** 004-wanted-list
## Progress
| Phase | State | Notes |
|---|---|---|
| 1 — Reproducible fixtures | **done** | `cmd/gentestdata`, `make testdata`, `internal/testfixtures` |
| 2 — Headless launch | **done** | `scripts/dev-headless.sh`, `dev-stop.sh`, `seed-sandbox.sh` |
| 3 — Driving and seeing | **done** | event bridge, `data-testid`/aria pass, `backend/testctl`, `e2e/` smoke suite |
| 4 — Component coverage | **done** | Vitest 4 browser mode, 313 tests, `make ui-test`; `make bindings-check` |
| 5 — `events.Emit` wrapper | **done** | `backend/events/emit.go`, `Recorder`, 35 sites converted, service tests in `queue`/`config`/`playlist` |
| 6 — pi affordances | **done** | `.pi/skills/yellowjacket-dev/`, `.pi/prompts/e2e.md`, `.pi/journal.md`, `make skill-check` |
| 7 — CI that gates | **done** | `.gitea/workflows/ci.yml`, two jobs, both prototyped in a container first |
**Verified end to end after phase 7:** both jobs were built as shell
scripts and run to green in a bare `ubuntu:24.04` container before any
YAML existed, then the steps were transcribed *back out of the
workflow* and re-run in the same container to prove the transcription —
job 1 (lint × 3, test × 3, `tsc --noEmit`, 313 ui-tests,
`bindings-check`, `skill-check`) and job 2 (fixtures, seed,
`dev-headless`, 19/19 chromium, 19/19 **webkit**). Push-and-see was
not an acceptable loop here: a Gitea Actions run that never starts
looks identical to one that passed.
It found a real bug on day one. **`make lint` was linting three
configurations that nothing builds** — all three passes omitted
`webkit2_41`, so wails resolved `webkit2gtk-4.0`, which Arch still
ships and Ubuntu 24.04 dropped. The `dev` pass is the one that breaks,
because wails' own `app_dev.go` is `dev`-tagged and drags in the 4.0
assetserver that the other two passes never compile. The tag sets now
match `make test` exactly. "Lint passes" and "lint compiled what we
ship" were different claims, and only a second distro could tell them
apart.
Two decisions were settled by measurement rather than argument. The
**audio sink** is a four-line ALSA `null` PCM, no daemon: `InitSpeaker`
succeeds in 36 ms and the elapsed clock advances, because ALSA's null
plugin advances its pointer on a timer. The **explore artifact** is
stubbed at a dead address as `seed-sandbox.sh` already does — and
setting it for the *app* run too, which `dev-headless.sh` never did,
turned out to be worth 8x on suite wall clock. **Playwright's WebKit
is a required step**, not an advisory one: it had never been run
anywhere, so one throwaway container run turned a coin flip into a
decision (19/19, +11 s), and nothing in `e2e/` compares pixels, so a
WebKit failure is an engine bug rather than baseline noise.
**Verified end to end after phase 6:** all four tiers were re-run
green *before* anything was written — `make ui-test` (313),
`make lint` (0 issues × 3 configurations), `make test` (3 passes),
`make e2e` (19/19 against a seeded `dev-headless` app) — so the skill
documents commands that were observed working, not remembered.
`make skill-check` then verified the 25 make targets the skill cites
all exist, and was itself verified to fail on a missing one. The
remaining check is the one no tooling can do: an agent following
`.pi/skills/yellowjacket-dev/` cold on a real task, which should
happen before phase 7 encodes the same commands into CI.
**That cold run has now happened.** An agent that did not write the
skill brought the app up from a wiped `.dev/` and no fixture library,
drove an undocumented flow (queue panel + shuffle, asserted on
`QueueModeChanged`, confirmed against `queue.Queue.GetState`) and
stopped it, in about a minute with no dead ends; all four tiers then
re-ran green from that cold state. It found one real config bug —
`outputDir` in `.playwright/cli.config.json` resolves against the
shell's cwd, not the config file's directory, so every snapshot was
landing one level *above* the repo where a stale copy from the
previous session answered instead — and four missing or wrong steps
(`sandbox-seed` already runs `testdata`; `make ui-setup` /
`make e2e-setup` are undocumented once-per-clone prerequisites;
`snapshot` prints a path, not a tree; `make dev-stop` leaves the
browser session open). All fixed in place, with the detail in
`.planning/NOTES.md`.
**Verified end to end after phase 5:** all 35 `runtime.EventsEmit`
call sites across 14 files now route through `events.Emit`, and
`TestNoDirectRuntimeEmits` fails the build if a new one appears. Four
packages had each hand-rolled their own guard against the same
`log.Fatalf` (`library.emit`, `download.emit`, `autotagservice.
emitEvent` with its own `ctxReady` field, `playlist.emitEvent`) and
nine more sites guarded on `ctx != nil`, which does not actually
prevent it; all of that collapsed into one place. 16 new tests assert
what the *frontend receives* — queue mode/index/delta payloads,
config theme and shortcut snapshots, playlist create/add/delete —
none of which was reachable before. `make lint` (3 configurations),
`make test` (3 passes), `make bindings-check`, `make ui-test` and
`make e2e` (19/19 against the seeded headless app) all green.
**Verified end to end after phase 4:** `make ui-test` runs 313 tests in
a real Chromium in ~2 s with no app, no backend and no display — 196
covering all 13 stores plus the keyboard shortcut service, 117 covering
components (transport, sidebar, library filter, status indicator,
track-info, now-playing, queue panel) including a smoke mount of all 46
custom elements against an empty backend. `make ui-visual` adds six
`toMatchScreenshot` baselines. `make bindings-check` regenerates
`frontend/wailsjs` in ~1.5 s and was verified to fail on a renamed
bound method. `tsc --noEmit`, `make lint` (all three configurations)
and `make e2e` (19/19) all stayed green, and the one frontend fix the
tier surfaced was confirmed in the running app by screenshot.
**Verified end to end after phase 3:** `make e2e` runs 19 Playwright
specs against the seeded app — harness self-tests, library views
(31 fixture tracks, unicode, sidebar navigation), playback (play,
pause, elapsed time, volume round-trip), queue (population, shuffle
state) and the control surface (snapshot → mutate → restore, forced
event, SQL, input validation). All 19 pass; `make lint` is at 0 issues
across all three build configurations and `make test` is green.
**Verified end to end after phase 2:** `make sandbox-seed NAME=default`
built a seed by driving the real `AddLibrary` binding and waiting for
the real scan to reach 31 tracks; `make dev-headless SEED=default`
restored it and landed *in* the app with no first-run wizard;
`playwright-cli` clicked through to Artists and screenshotted six real
artists with generated cover art, unicode names and the long-artist
truncation case; `LoadFile` + `Play` produced audible playback with the
transport bar at 00:04.
One re-sequencing against the plan below: `sandbox-seed` is described
under phase 1 but shipped at the end of phase 2, because seeding *by
running the app* makes it a consumer of the launcher.
One bug found by the fixtures, not yet fixed: **WAV tags are
write-only.** `backend/tagwriter` writes them into a RIFF `id3 ` chunk;
`backend/metadata` reads through `dhowden/tag`, which has no RIFF
parser, so every WAV scans in untitled. Pinned by
`TestWAVTagsAreNotReadableYet`.
## Problem
A coding agent can develop the Go packages of this repo competently and
cannot develop the *application* at all. It can read 66k lines of
backend, run 31k lines of tests, and lint two build configurations. It
cannot start the app, see a window, click anything, or find out whether
a change it made to a Lit component rendered.
The gap is not "we lack tests". It is that every path to running
YellowJacket ends in a blocking GTK window:
| Entry point | Behaviour |
|---|---|
| `make dev` | launches a WebKit window, blocks the terminal forever |
| `make sandbox <n>` | same, plus an interactive name argument |
| `make fresh-install` | same, and lands on the first-run wizard every time |
So 265 bound methods across 11 services, 46 backend events, 33 Lit
component directories, 13 reactive stores and a 357-line keyboard
shortcut service have exactly one form of verification available to an
agent: `tsc --noEmit`.
Three secondary facts make it worse. `test_data/music_library_test/` is
referenced by three test files, is in `.gitignore`, is not on disk, and
has no generator — so the audio path and `YELLOWJACKET_INTEGRATION=1`
are unreachable from a clean clone. No Gitea workflow runs `make test`
or `make lint`; quality gating exists only in `lefthook.yml`, which is
local and `--no-verify`-skippable. And there is no `.pi/` directory, so
none of the awkward invocations (`-tags "webkit2_41 indexbuild"`,
sandbox lifecycle, log tailing) are wrapped in anything an agent can
call.
## The unlock
`wails dev` already runs a full HTTP + WebSocket dev server on
`localhost:34115` (`internal/frontend/devserver/devserver.go`). It
serves the real frontend assets, injects the real generated bindings,
and bridges every method call and every `runtime.EventsEmit` over a
websocket to the **same running Go backend** the desktop window is
attached to.
A plain Chromium can load `http://localhost:34115` and get a fully
functional YellowJacket. Not a mock, not a stub `wailsjs` layer: the
actual application, talking to the actual `explore`, `library`,
`player` and `queue` services, receiving the actual events. The
bindings land on `window.go`, so anything reachable from the frontend
is reachable from a one-line `page.evaluate`.
This is not a trick we invented. Wails v3's documentation ships an
"End-to-End Testing" guide that is exactly this, and the v2 community
arrived at the same answer independently
(`wailsapp/wails` discussion #4205). It is the sanctioned approach.
**The one caveat:** `devserver.Run` still calls `d.Frontend.Run(ctx)`,
which opens the GTK window and blocks. No flag suppresses it, and
nobody upstream has found a way around it. The app needs a display —
a virtual one.
## Validated end to end, 2026-08-10
The premise was proven before this plan was committed, on a scratch
`YJ_HOME` under `~/.cache/yellowjacket-harness`:
```
go build -tags "dev webkit2_41" -o build/bin/yj-dev .
setsid dbus-run-session -- xvfb-run -a ./build/bin/yj-dev \
-devserver localhost:34115 -assetdir frontend/dist
playwright-cli -s=yj open http://localhost:34115
```
| Claim | Result |
|---|---|
| App boots headless under Xvfb | yes, ~1 s; `:34115` listening |
| `YJ_HOME` isolates the sandbox | yes, own `yj.db`, untouched real install |
| Chromium loads the real app | yes, console shows `wails dev / Connected to backend` |
| a11y snapshot pierces shadow DOM | yes — sidebar, queue panel, transport buttons, all with stable refs, through Lit **and** Web Awesome roots |
| `window.go` carries the bindings | yes, all 11 services |
| A bound method round-trips to Go | yes — `queue.Queue.GetState()` returned real JSON |
| Events reach the browser | yes — `SetVolume(42)` produced `VolumeChanged` with payload `42` |
| Screenshot is readable by the agent | yes — full render, correct theme, fonts and icons |
| MPRIS registers | yes — `org.mpris.MediaPlayer2.yellowjacket` on the private bus |
| Audio initialises | **yes** — see below |
Five things the run taught that were not obvious beforehand:
- **No null audio sink is needed.** `dbus-run-session` replaces the
*bus*, not the runtime dir, so `/run/user/1000/pulse` stays reachable
and `InitSpeaker` succeeded in 17 ms. The mitigation planned for
Phase 3 is unnecessary on a developer machine. A CI container with no
`/run/user` will still need one.
- **The first-run wizard blocks every interaction.** The first click
attempt failed with `<first-run-wizard> intercepts pointer events`.
Phase 1 is not a convenience; nothing downstream works without it.
- **A malformed binding call hangs forever.** `SetVolume(0.42)` against
a `player.UserVolume` (an `int`) made the backend log
`error parsing arguments` and never fire the callback, so the
in-page promise never settled. Every harness call needs a timeout,
and the app log is the only place the reason appears.
- **Playwright's WebKit does not run on Arch.** Its Linux build links
Ubuntu 24.04 libraries — `libicu74`, `libWPEWebKit-2.0.so.1`,
`libflite` — none of which Arch provides. `--browser=webkit` is a
**CI-only** capability, not a local one. Chromium is unaffected.
- **Event listeners accumulate across calls.** Hooks registered by one
`eval` survive into the next, so a naive recorder double-counts. The
`initScript` must install exactly one recorder, and tests must reset
its buffer rather than re-register.
## Tooling decisions taken up front
Three things exist that we would otherwise have built badly.
**`@playwright/cli`** (`npm i -g @playwright/cli`) is Microsoft's
CLI-plus-agent-skills front end to Playwright, built specifically
because coding agents do better with terse commands than with MCP tool
schemas. `playwright-cli install --skills` drops the skills where an
agent finds them. It gives us, for free, everything this plan was
otherwise going to hand-roll:
| Need | Command |
|---|---|
| See the page | `snapshot` — a11y tree with stable `ref=eNN` handles, pierces open shadow roots |
| Search a big page | `find <text>` / `find --regex` |
| Call a bound method | `eval "() => window.go.player.Player.Play(1)"` |
| Screenshot for the agent to read | `screenshot --filename=` |
| Frontend errors | `console` — Lit render failures are currently invisible |
| Stub the explore artifact | `route <pattern>` |
| Keep a browser across separate shell calls | `-s=<session>` |
| Watch, and take over | `show` — live dashboard, per-session screencast, click in to grab the mouse |
Plus video and trace recording when a flow needs explaining rather than
asserting. It is v0.1.x and moving; `@playwright/mcp` is the same engine
behind an MCP server and is the fallback if the CLI churns.
**Playwright's WebKit build.** The shipped binary is WebKit2GTK, so a
Chromium-only suite would validate a renderer we do not ship.
`--browser=webkit` is not byte-identical to WebKit2GTK but shares the
engine core, and it is a flag rather than a project. The X11-grab of the
real GTK window drops to an optional spot-check.
**Vitest 4 browser mode.** Stable Browser Mode plus `toMatchScreenshot`
landed in Vitest 4.0, it is the Lit ecosystem's current recommendation
over `@web/test-runner`, and it uses Playwright as its provider — the
same browsers already cached. Components render in a real browser with
real shadow DOM, and get visual regression, **with no Wails, no backend,
no seeded library and no virtual display**. This is a tier the earlier
draft of this plan did not have and is the cheapest coverage available.
So the harness is three tiers, cheapest first:
1. **Vitest browser mode** — components and stores. Seconds. No app.
2. **`playwright-cli` against `:34115`** — real flows against the real
backend, driven interactively by an agent.
3. **Playwright specs** — the same thing, frozen as a regression suite,
in CI.
Only tier 2 and 3 need the app running, and therefore Xvfb.
## Phase 1 — Reproducible fixtures *(shipped)*
Nothing can be driven end-to-end against an empty library, and no two
runs are comparable unless the library is identical. No tool provides
this; it is ours to write.
**`cmd/gentestdata`** writes `test_data/music_library_test/`
deterministically: silent/tone audio at known durations across MP3,
FLAC, OGG Vorbis and WAV, with tags written by our own `tagwriter` so
fixtures and reader cannot drift. Coverage must include the cases the
app has code for — embedded cover art shared across an album (dedup),
missing and partial tags, unicode and RTL text, multi-disc, various
artists, and a deliberate duplicate pair for
`duplicate-tracks-dialog`.
`make testdata` generates it; it stays gitignored. A manifest hash lets
a test assert it is looking at the library it thinks it is.
**Seeded sandboxes.** `make sandbox-seed NAME=<n>` builds a `YJ_HOME`
with `config.toml` already pointing at the fixture library and `yj.db`
already scanned, so a run starts *in the app* rather than in the
first-run wizard. A `--fresh` variant deliberately omits config, because
the wizard is itself a surface that needs testing. Seeds rebuild from
scratch in seconds and are never hand-edited.
Explicitly **not** seeded: the explore artifact. `artifactfetch.go`
already honours `YJ_CORE_INDEX_URL` ("overridable for testing"), so
tests point it at a local file server holding a cut-down artifact —
`cmd/indexexport` already produces that shape, so a tiny core is a
config change, not new code. This also makes the failure paths testable
(404, checksum mismatch, the `206` partial-content resume). A nightly
job can use the real artifact.
## Phase 2 — Headless launch *(shipped)*
`scripts/dev-headless.sh` wraps:
```
dbus-run-session -- xvfb-run -a \
./build/bin/yj-dev -devserver localhost:34115 -assetdir frontend/dist
```
**Run the dev binary directly, not `wails dev`.** `app_dev.go` parses
`-assetdir`, `-devserver`, `-frontenddevserverurl` and `-loglevel`
straight from `os.Args`, so `go build -tags "dev webkit2_41"` produces a
binary that serves the identical devserver with no file watcher, no
rebuild supervisor and no reload broadcast. One process, one PID,
deterministic startup. `wails dev`'s watcher is a human ergonomic; an
agent that just edited a file knows to rebuild. (`-noreload` and
`-nogorebuild` exist if the watcher is ever wanted anyway.)
**`dbus-run-session` is not incidental.** A private session bus means
`backend/mediacontrols/mpris_linux.go` actually registers, which turns
MPRIS from "untestable" into a surface assertable with `busctl` —
properties out, `Play`/`Pause`/`Next` in.
The script backgrounds the process, writes `.dev/app.pid` and
`.dev/app.log`, polls `:34115` until it answers, then exits, leaving the
app up. `make dev-headless SEED=<n>`, `make dev-stop`, `make dev-logs`.
**Kill by saved PID, never `pkill -f`.** A `pkill -f` whose pattern
appears in the invoking shell's own command line kills that shell and
silently drops the rest of the chain.
New dependency: `xorg-server-xvfb`. Everything else — Playwright and its
Chromium, ffmpeg, `import`, `dbus-run-session`, `busctl`, `pactl` — is
already present.
**Audio needs nothing locally.** Measured: `InitSpeaker` succeeds under
`dbus-run-session` + Xvfb because the PulseAudio socket in
`/run/user/1000` is untouched by a private bus. Only a CI container
without `/run/user` needs a null sink (PipeWire null sink, or an ALSA
`null` PCM via a scoped `asoundrc`), and even then `app.go:325` joins
`InitSpeaker()` failure into `startupErr` rather than aborting, so
everything except playback still runs. Sample-level correctness stays
where it already is, in `backend/player` unit tests.
## Phase 3 — Driving and seeing the app *(shipped)*
What landed, and the five things that were not obvious:
- **`.playwright/cli.config.json`** now sets `testIdAttribute`, a
1440×900 viewport, timeouts and the `initScript`. Every path in it is
resolved **relative to the config file**, not the repo root.
- **`.playwright/init-events.js`** is the event bridge, and it hooks
`window.wails.EventsNotify` rather than `EventsOn` — every backend
event enters the page at that one call (`case "n"` in wails'
`ipc_websocket.js`), so one wrap captures all 46 whether or not the
app subscribes. `window.wails` does not exist when an initScript
runs, so it is wrapped via an accessor installed on `window` that
collapses back to a data property on assignment. It also carries
`ready()` and `call()`, the latter timing out so the
"malformed binding call hangs forever" trap is paid for once.
- **No closed shadow roots** anywhere: nothing in `frontend/src`
overrides `createRenderRoot`/`shadowRootOptions` and Web Awesome's
dist never calls `attachShadow` directly. Snapshots pierce
everything.
- **The `data-testid` pass was mostly an accessibility fix.** The five
transport buttons had no accessible name at all, so they were
unnameable to a screen reader *and* to a selector; they now carry
`aria-label` plus `aria-pressed` for the shuffle/repeat toggles.
`data-testid` was added only where a selector would otherwise be
structural: `track-row`, `queue-row` (both with `data-file-path`),
`main-content` (plus a `data-active-view` attribute, since which view
is showing was previously only inferable from which cached child
lacked `.view-hidden`), the now-playing title/artist and the seek
bar's two clocks. Sidebar items got `data-testid` and `aria-current`.
- **`backend/testctl`** mounts `/__test/` on the existing asset
handler: `health`, `db/snapshot`, `db/restore`, `emit`, `sql`. Gated
twice — the implementation is behind the `dev` build tag with a no-op
`!dev` twin, and it refuses to register unless `YJ_TESTCTL=1`, which
`dev-headless.sh` sets and `make dev` does not.
- **`e2e/`** is its own npm package (`make e2e`), deliberately not
inside `frontend/`, so phase 4's Vitest browser mode does not have to
share a package with the Playwright runner.
The original plan for the phase follows.
- `playwright-cli install --skills`, and a project
`.playwright/cli.config.json` setting `testIdAttribute`, viewport, and
an `initScript`.
- **The `initScript` is the event bridge.** It runs before any app
script, so it can hook `EventsOn` and buffer all 46 backend events on
`window.__yjEvents`, read back with `eval`. Half of what this app does
is push-driven — scan progress, job updates, download progress,
`WantedListChanged` — and assertions must await an event, not a
timeout.
- Add `data-testid` where selectors would otherwise be structural. First
task of the phase is confirming nothing in the Lit / Web Awesome tree
uses a *closed* shadow root, which would defeat snapshots.
- A **dev-only control surface**. `backend/assets/handler.go` is ours and
already has `RegisterHandler(pattern, handler)`, so `/__test/...` can
be mounted on the same port with no new server: seed, snapshot and
restore the SQLite DB mid-run, force backend-internal state. Roughly
five endpoints, compiled out of non-dev builds. This is the residue of
what Playwright genuinely cannot reach — everything browser-side is
already covered by the CLI.
**Screenshots as the primary agent primitive.** `screenshot --filename=`
then reading the PNG is the feedback loop that makes UI iteration
possible at all, and it matters more than the assertion suite built on
top of it. `snapshot` is the cheaper companion for structure.
**Smoke suite**, once flows are stable, frozen as Playwright specs:
first-run wizard, library views (artists / genres / cover grid / track
list), playback and queue manipulation, playlist and smart-playlist
editing, explore search and detail pages, settings (the HTMX/templ path,
which renders differently from everything else), jobs, downloads.
**Renderer fidelity is CI-only.** Playwright's Linux WebKit is built
against Ubuntu 24.04 and will not start on Arch (missing `libicu74`,
`libWPEWebKit-2.0.so.1`, `libflite`); the download succeeds and the
binary then fails to link. So `--browser=webkit` runs in Job 2 of CI,
where the runner image is Debian-family, and local work is Chromium
only. The X11 grab of the real GTK window stays available as an
optional spot-check for views where WebKit2GTK-specific rendering
matters — and is the *only* WebKit2GTK signal obtainable on this
machine.
**Two live frontends, one backend.** The GTK window and the browser are
both websocket clients of the same backend. This is supported —
`devserver.go` keeps a client map and `notifyExcludingSender`
deliberately fans frontend-emitted events out to the other clients *and*
the desktop frontend; `-browser` exists for exactly this. The risk is
not the transport but our own singletons: 13 stores × 2 instances means
duplicate cover-art fetches on connect and two clients able to issue
`player.Play`. If that proves noisy, the fix is contained — our asset
handler can serve a blank page to the WebKitGTK user agent under
`YJ_HEADLESS=1`, making the window inert. Start without it.
## Phase 4 — Component and store coverage *(shipped)*
What landed:
- **The Wails fake is the whole trick.** Everything in
`frontend/wailsjs/` is a pure passthrough to `window.go` and
`window.runtime`, so `test/support/wails-fake.ts` replaces those two
globals and every test then runs the *real* generated bindings and
the *real* store code. No module mocking, and no second description
of the Wails layer free to drift. Its event dispatcher mirrors
`desktop/events.js`, including `maxCallbacks` expiry and the fact
that a frontend `EventsEmit` notifies local listeners before Go.
- **Stores are singletons constructed at import**, so the fake is
installed from `setupFiles`, which runs first. A handful of stores
read config in their constructor before any test can stub, so the
setup file carries import-time defaults — without them a store
caches `undefined` where Go would have sent `[]`, and every consumer
crashes on `.length` in a way that looks like a component bug.
- **`make ui-test` / `ui-watch` / `ui-visual` / `ui-visual-update`.**
Visual regression is opt-in (`YJ_VISUAL=1`) because baselines are
font-hinting and compositing sensitive; the default run asserts
behaviour only, so nobody's loop breaks over antialiasing.
- **`make bindings-check`** (`scripts/bindings-check.sh`) runs
`wails generate module` and fails on a dirty tree, ignoring the file
modes the generator churns. Now in `lefthook.yml` pre-commit; the
Vitest suite is in pre-push.
- Two frontend bugs the tier found: `ScrollManager.setupResizeObserver`
threw an unhandled rejection on an empty library (fixed, one guard),
and `themeStore.loadFromBackend`'s failure handler throws again on
the state that failed it, so it cannot recover (left alone —
reachable only if the backend returns an empty accent).
The original plan for the phase follows.
Vitest 4 browser mode with the Playwright provider, in `frontend/`.
- The 13 stores and the keyboard shortcut service. Queue mutation,
shuffle, repeat transitions, explore cache invalidation and shortcut
dispatch are near-pure TypeScript with zero tests today.
- Component rendering for the 33 component directories, with
`toMatchScreenshot` visual regression per component. Real browser,
real shadow DOM, no app, no display — this is where the bulk of UI
regression should live, leaving e2e for flows.
- **Binding drift check.** `frontend/wailsjs/` is generated by
`wails build`, *not* `go generate`, so the existing pre-commit codegen
check does not cover it. A renamed Go struct field currently surfaces
at runtime, in a window. Add a target that regenerates bindings and
fails on a dirty tree.
## Phase 5 — An `events.Emit` wrapper *(shipped)*
What landed:
- **`events.Emit(ctx, name, data...)`** drops an event that has
nowhere to go, at debug level, instead of taking the process down.
**`events.Deliver`** is the same call returning `ErrNoRuntime`, for
the one caller that must know: `/__test/emit`, whose job is to
impersonate a backend emit and which would otherwise answer `200`
for an event that reached nobody.
- **The sink is carried in the context**, not in a package-level
variable — `events.WithSink(ctx, rec)` — so parallel tests cannot
observe each other's events and production emits pay no
synchronisation cost. `events.Recorder` implements it with
`Events`/`Named`/`Names`/`Count`/`Last`/`Reset` and a `Wait` that
blocks on background emitters (scan progress, `SetQueue` phase 2).
- **Enforcement is a test, not a linter.** golangci-lint runs once per
build configuration, so a stray emit in an `indexbuild`- or
`dev`-tagged file would only be seen by the pass that compiles it;
`TestNoDirectRuntimeEmits` walks the tree and sees all of them.
- **The tier it unblocks, exercised**: `backend/queue` (7),
`backend/config` (5), `backend/playlist` (4). Playlist is the one
that matters beyond the wrapper itself — it proves the pattern on a
service whose emits interleave with SQLite writes and M3U8 file
writes, and its test reads the playlist back the way the frontend
would on receipt of the event.
Deferred out of this phase: a general `backend/playlist` CRUD suite.
The service is 2,900 lines with no CRUD coverage today, and that is
its own piece of work rather than a rider on a mechanical refactor.
The original plan for the phase follows.
34 call sites use `runtime.EventsEmit` directly. `runtime.getEvents`
(`runtime.go:47`) `log.Fatalf`s unless `ctx.Value("events")` satisfies
`frontend.Events` — an interface under `wails/v2/internal/`, which we
cannot implement. So none of those code paths can run outside a real
Wails app, and in-process service tests are impossible.
A thin `events.Emit(ctx, name, data...)` in `backend/events`,
delegating to `runtime.EventsEmit` normally and to a recorder when a
test sink is installed, unblocks that. It is mechanical, and it pays for
itself independently as the one place to log or trace all 46 events.
Sequenced after the e2e tiers because it is a refactor touching many
packages, and the tiers above deliver value without it.
## Phase 6 — pi affordances *(shipped)*
What landed, and the one decision that mattered:
- **`.pi/skills/yellowjacket-dev/`**, a directory rather than a flat
file. Only a skill's description is always in context, so `SKILL.md`
holds the tier decision table, the canonical command sequences and
the five gotchas — the last inline rather than in a reference,
because they are needed *before* the failure — and
`references/{harness,fixtures,ui-tier,schema-change}.md` hold the
per-surface depth.
- **The split from `CLAUDE.md` is grammatical, not topical.** A topical
split is what rots: every new fact has two plausible homes. Three
docs, three tenses — `NOTES.md` past (measured, dated, append-only),
`CLAUDE.md` present (what the system is), the skill imperative (what
to run). CLAUDE.md's harness section lost about half its length to
this; leaving both would have been exactly the duplicate description
this repo has a standing rule against.
- **`make skill-check`** makes the rule enforceable rather than
aspirational: every command in `.pi/**/*.md` must be a real make
target, so the Makefile stays the source of truth for *how* to invoke
something and the skill only decides *which* and *in what order*. A
pre-commit hook; instant.
- **`.pi/prompts/e2e.md`** treats promotion as a transcription with
four fixed substitutions (snapshot refs → testids, sleeps →
`waitForEvent`, raw `window.go` → `callBinding`, short fixture →
`LONG_TRACK`) and three runs — pass, pass again, pass after a DB
restore — because the characteristic failure of a promoted spec is
depending on state the hand-driving left behind.
- **`.pi/journal.md`**, per the `/handoff` convention.
The original plan for the phase follows.
With the mechanics settled, wrap them. Much less than the first draft
assumed, because `playwright-cli`'s own skills cover browser work.
`.pi/` gains:
- **`skills/yellowjacket-dev/`** — the build-tag matrix, the two-file
schema rule, seed and sandbox lifecycle, the harness commands, and
when to reach for which of the three test tiers. `CLAUDE.md` has the
architectural half; this is the operational half. It must also carry
the gotchas the live run surfaced: time out every binding call, check
`.dev/app.log` when one hangs, and never assume a click will land
while the first-run wizard is up.
- **`settings.json`** pointing at `../.claude/skills`, because
`playwright-cli install --skills` writes to `.claude/skills/` and pi
does not discover that path by default. *(Already in place.)*
- **`.pi/journal.md`**, per the `/handoff` convention.
- A `/e2e` prompt template for promoting an exploratory
`playwright-cli` session into a committed spec.
No custom extension. Browser control is a solved, actively maintained
problem and a hand-rolled version would be worse and would rot.
## Phase 7 — CI that actually gates *(shipped)*
What landed, and the decisions behind it:
- **One image for both jobs, `ubuntu:24.04`.** Not `golang:1.25`,
because job 1 runs `make ui-test` — Vitest *browser* mode — so the
"fast job needs no browser" split does not survive contact. Not the
Playwright image either, because `e2e/` pins `@playwright/test`
^1.56 and `frontend/` pins `playwright` ^1.62, so a prebuilt browser
set matches at most one of them. Ubuntu 24.04 is also what
Playwright's WebKit links against, which job 2 needs.
- **Caching needed no runner-side change.** `valid_volumes` is already
a glob over the runner's cache root, and `GOMODCACHE` / `GOCACHE` /
`GOLANGCI_LINT_CACHE` are mounted and exported for every job by
`container.options`. Only the Node-side caches (browsers, pnpm store,
the Go tarball) are declared in the workflow.
- **The repo is cloned by hand**, as the other three workflows do:
`actions/checkout` is a JS action and needs node in the container
before any step has had a chance to install it.
- **Failure output goes to the job log, not only to an artifact.**
`.dev/app.log` is tailed into the log on failure so `gitea_ci
job_logs` can reach it, with the Playwright report uploaded
alongside as `continue-on-error` so a broken upload cannot mask the
real failure.
The original plan for the phase follows.
`.gitea/workflows/ci.yml` — the repository has three workflows and none
of them test anything, so `gitea_ci` currently reports only packaging
jobs, which actively misleads an agent checking whether a push was
healthy.
- **Job 1 (fast, no display):** `make lint`, both `make test` passes,
`tsc --noEmit`, Vitest browser mode.
- **Job 2 (display):** Xvfb + `dbus-run-session` + a seeded sandbox +
`make dev-headless` + the Playwright smoke suite, with screenshots and
traces uploaded on failure.
Job 2 depends on the fixture generator and the stubbed artifact, so it
lands last.
## Order and why
1 and 2 are the hard blockers and are worth doing even if nothing else
follows — a seeded, scriptable, non-blocking launch is the difference
between an agent that can and cannot run this app. 3 is the payoff and
is now mostly configuration. 4 is the cheapest coverage per hour and can
proceed in parallel with everything else, since it depends on none of
it. 5 is a refactor that unblocks a fourth tier we do not have yet. 6 is
ergonomics and should wait until the commands stop changing. 7 is last
because it depends on all of it.
## Risks
- **`@playwright/cli` is v0.1.x.** Interface churn is likely. The
mitigation is that `@playwright/mcp` is the same engine behind a
different front end, so a switch is a config change, and the specs
written in Phase 3 are plain Playwright either way.
- **Playwright's WebKit is not WebKit2GTK, and does not run here at
all.** Closer than Chromium in CI, unavailable locally. A
GTK-specific rendering bug can still escape, and will not be caught
until CI runs — or ever, for views not in the smoke suite.
- **Xvfb is X11, and the app has a Wayland-specific NVIDIA workaround**
(`main.go`'s DMABuf disable). CI will not exercise the Wayland path at
all. Acceptable — that path is a crash workaround, not a feature — but
it should be a known blind spot rather than a surprise.
- **Seeds are a second description of a valid `YJ_HOME`.** If the
generator drifts from what the app actually writes, tests pass against
a state no real install has. Seeds must be produced by *running the
app*, not by writing config and DB rows by hand — the same discipline
`sql/schemas/` gets, for the same reason.
## Deferred
- Driving the real WebKit2GTK window directly.
`WEBKIT_INSPECTOR_SERVER=127.0.0.1:9222` exposes WebKit's remote
inspector, but the protocol is not CDP and Playwright cannot attach.
A bespoke client is the only route and is not worth it.
- Wails v3, whose e2e story is better documented and whose dev server is
the same idea on port 9245. Not a reason to migrate.
- Component testing via `playwright-ct-web`. Vitest browser mode covers
the same ground with fewer moving parts and a first-party visual
regression story.