yonlu/yellowjacket
1
1
Fork 0
Code Issues 55 Pull Requests 6 Actions Packages Projects Releases 12 Wiki Activity

make ui-test flakes: three suites fail to import setup.ts from the Vitest dev server, once in five runs #128

New Issue
Closed
opened 2026-08-19 13:14:03 +00:00 by yonlu · 3 comments
yonlu commented 2026-08-19 13:14:03 +00:00
Owner
Copy Link
Copy Source

Report

make ui-test (and the pre-push hook that runs it) failed once, then passed on four consecutive runs of the identical tree — clean worktree, same commit, no rebuild in between.

What the failing run looked like

Three suites failed to import at all:

FAIL |chromium| test/harness.test.ts [ test/harness.test.ts ]
FAIL |chromium| test/stores/player-store.test.ts
FAIL |chromium| test/utils/credit-link.test.ts
Error: Failed to import test file frontend/test/setup.ts
Caused by: TypeError: Failed to fetch dynamically imported module:
  http://127.0.0.1:63315/...frontend/test/setup.ts?import&browserv=...

and one assertion failed alongside them:

FAIL test/components/explore-track-details.test.ts >
  opens the dialog on the library track behind the row
AssertionError: expected undefined to be '/music/an-album/01.flac'

The import failures are the interesting half: setup.ts is fetched from Vitest's own dev server, so "failed to fetch" is the harness, not the code. The assertion failure is plausibly collateral from the same run — that spec passed 5/5 in isolation immediately afterwards, and the full suite passed 898/898 three times running.

Why it matters

This is the pre-push hook and a CI step, so a flake here reads as "your branch broke the frontend" and costs a re-run at best. It is worth knowing about before the next person spends a session bisecting a change that was never at fault — which is roughly what happened here.

Direction

Unreproduced, so this is a report rather than a diagnosis. Worth checking whether the browser provider is being started with enough headroom (the run that failed was the second full suite in ~40 s, behind a go test -race pass in the same hook), and whether setup.ts being fetched per-suite rather than bundled is what makes the server the shared point of failure. If it recurs, capture the Vitest server log rather than the summary.

**Report** `make ui-test` (and the `pre-push` hook that runs it) failed once, then passed on four consecutive runs of the identical tree — clean worktree, same commit, no rebuild in between. **What the failing run looked like** Three suites failed to *import* at all: ``` FAIL |chromium| test/harness.test.ts [ test/harness.test.ts ] FAIL |chromium| test/stores/player-store.test.ts FAIL |chromium| test/utils/credit-link.test.ts Error: Failed to import test file frontend/test/setup.ts Caused by: TypeError: Failed to fetch dynamically imported module: http://127.0.0.1:63315/...frontend/test/setup.ts?import&browserv=... ``` and one assertion failed alongside them: ``` FAIL test/components/explore-track-details.test.ts > opens the dialog on the library track behind the row AssertionError: expected undefined to be '/music/an-album/01.flac' ``` The import failures are the interesting half: `setup.ts` is fetched from Vitest's own dev server, so "failed to fetch" is the harness, not the code. The assertion failure is plausibly collateral from the same run — that spec passed 5/5 in isolation immediately afterwards, and the full suite passed 898/898 three times running. **Why it matters** This is the `pre-push` hook and a CI step, so a flake here reads as "your branch broke the frontend" and costs a re-run at best. It is worth knowing about *before* the next person spends a session bisecting a change that was never at fault — which is roughly what happened here. **Direction** Unreproduced, so this is a report rather than a diagnosis. Worth checking whether the browser provider is being started with enough headroom (the run that failed was the second full suite in ~40 s, behind a `go test -race` pass in the same hook), and whether `setup.ts` being fetched per-suite rather than bundled is what makes the server the shared point of failure. If it recurs, capture the Vitest server log rather than the summary.
yonlu added the Kind/TestingArea/Library-UI
Priority
Low
4
Platform/Desktop labels 2026-08-19 13:14:03 +00:00
yonlu commented 2026-08-19 13:16:28 +00:00
Author
Owner
Copy Link
Copy Source

Diagnosed. It is not the specs — it is the hook.

pre-push is parallel: true over go test -race -count=1 ./... and vitest run. The Go pass saturates every core for ~47 s; the browser tier is a real Chromium with wall-clock timeouts. So the two flakes seen so far are the same fault wearing different clothes:

run 1: 3 suites fail to import setup.ts from the Vitest dev server
       + explore-track-details assertion
run 2: folder-picker times out at 15001ms, "did not mount itself"

The tell is in the summary line: setup took 106 s inside the hook against 63 s for a standalone make ui-test, and the suite that fails changes every run. Standalone it has now passed 898/898 five times running, and explore-track-details passed 5/5 in isolation immediately after failing in the hook.

So a browser tier with real timeouts cannot share a machine with -race and stay honest, and Priority/Low is too low for something that fails a push at random.

Diagnosed. It is not the specs — it is the hook. `pre-push` is `parallel: true` over `go test -race -count=1 ./...` and `vitest run`. The Go pass saturates every core for ~47 s; the browser tier is a real Chromium with wall-clock timeouts. So the two flakes seen so far are the same fault wearing different clothes: ``` run 1: 3 suites fail to import setup.ts from the Vitest dev server + explore-track-details assertion run 2: folder-picker times out at 15001ms, "did not mount itself" ``` The tell is in the summary line: `setup` took **106 s** inside the hook against **63 s** for a standalone `make ui-test`, and the suite that fails changes every run. Standalone it has now passed 898/898 five times running, and `explore-track-details` passed 5/5 in isolation immediately after failing in the hook. So a browser tier with real timeouts cannot share a machine with `-race` and stay honest, and `Priority/Low` is too low for something that fails a push at random.
yonlu added
Priority
Medium
3
Reviewed
Confirmed
1
 and removed
Priority
Low
4
 labels 2026-08-19 13:16:29 +00:00
yonlu self-assigned this 2026-08-19 13:16:30 +00:00
yonlu added the
Status
In Progress
 label 2026-08-19 13:16:30 +00:00
yonlu commented 2026-08-19 13:16:30 +00:00
Author
Owner
Copy Link
Copy Source

Fixing on fix/player-playing-state, where it surfaced: it fails that branch every other push.

Approach: make pre-push sequential. The browser tier cannot keep wall-clock timeouts while sharing cores with -race, and the ~15s the parallelism saves is not worth a gate that fails at random.

Fixing on `fix/player-playing-state`, where it surfaced: it fails that branch every other push. Approach: make `pre-push` sequential. The browser tier cannot keep wall-clock timeouts while sharing cores with `-race`, and the ~15s the parallelism saves is not worth a gate that fails at random.
logan commented 2026-08-19 14:43:50 +00:00
Collaborator
Copy Link
Copy Source

Another sighting, and a slightly different symptom worth recording: this fired
from the pre-push hook (lefthook.yml ui-test, which is
./node_modules/.bin/vitest run with root: frontend/) rather than from
make ui-test.

The push was rejected with ui-test failing after 38s. Re-running the exact
same command directly (cd frontend && ./node_modules/.bin/vitest run) gave
80 passed / 904 passed, and the immediately following git push passed the
same hook in 17.6s with no change to the tree.

So it reproduces under the hook as well as under make, it is not specific to
the Makefile's invocation, and it is timing-shaped — the failing run took twice
as long as the passing one, which fits a setup-import race rather than a real
assertion failure. Nothing in my branch touches the suites involved.

Not blocking me (a retry pushed cleanly), but it means the pre-push gate is
currently a coin toss on a cold cache, which is worse than a flaky make:
it rejects a push and the natural reading is that you broke something.

Another sighting, and a slightly different symptom worth recording: this fired from the **pre-push** hook (`lefthook.yml` `ui-test`, which is `./node_modules/.bin/vitest run` with `root: frontend/`) rather than from `make ui-test`. The push was rejected with `ui-test` failing after 38s. Re-running the *exact* same command directly (`cd frontend && ./node_modules/.bin/vitest run`) gave `80 passed / 904 passed`, and the immediately following `git push` passed the same hook in 17.6s with no change to the tree. So it reproduces under the hook as well as under `make`, it is not specific to the Makefile's invocation, and it is timing-shaped — the failing run took twice as long as the passing one, which fits a setup-import race rather than a real assertion failure. Nothing in my branch touches the suites involved. Not blocking me (a retry pushed cleanly), but it means the pre-push gate is currently a coin toss on a cold cache, which is worse than a flaky `make`: it rejects a push and the natural reading is that you broke something.
logan closed this issue 2026-08-19 14:53:46 +00:00
gitea-actions bot removed the
Status
In Progress
 label 2026-08-19 14:55:21 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Area/Design
Area/Downloads
Area/Explore
Area/Library-UI
Area/Metadata
Area/Packaging
Area/Player
Area/Queue
Area/Settings
Area/Shell-Nav
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
 Platform/Android
Platform/Desktop
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Medium
3
The priority is medium
Priority
Low
4
The priority is low
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
In Progress
Somebody is actively working on this right now
No labels Area/Library-UI Kind/Testing Platform/Desktop
Priority
Medium
3
Reviewed
Confirmed
1
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan yonlu
No Assignees yonlu
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: yonlu/yellowjacket#128