test(download): write the yt-dlp stub under ForkLock #235

2026-08-30T11:41:19Z

logan commented

2026-08-30 11:41:19 +00:00

The issue. TestYtDlpSearchSkipsUnparseableLines failed once in a
full make test run and once in CI, both times on a tree whose diff
contained no Go at all, with
fork/exec …/yt-dlp: text file busy. #146 diagnosed the mechanism:
the kernel refuses to exec a file that is open for writing anywhere in
the process, and these tests are t.Parallel(), so a sibling's
fork duplicates stubYtDlp's write descriptor in the moment it is
open and carries it past our close.

What changed. The write in stubYtDlp is held under
syscall.ForkLock — six lines and a comment, in one test file. No
production code, nothing user-visible.

Why that and not either fix the issue named. "Close the file before
any sibling can fork" is not available: os.WriteFile has already
closed it before anything execs, and the offending descriptor is a
copy taken by a fork that was already in flight. "Write the stub once
per package into a shared dir" narrows the window without removing it,
since the first write of each script still races. ForkLock is the
lock syscall.forkExec takes across the fork itself, so holding it
over the write means no child can exist while the descriptor does, and
a child forked earlier cannot hold one for a file that did not yet
exist. It is declared on Windows and darwin as well as Unix, which
matters because this helper skips at runtime rather than behind a build
tag.

Verification.

tier	result
throwaway probe in `backend/download` (12 goroutines × 200 real `stubYtDlp` calls + exec)	189 / 176 / 178 ETXTBSY of 2400 before, 0 / 0 / 0 after
standalone harness outside the repo	271–302 of 4800 before, 0 of 4800 after
`go test -race ./backend/download/`, ×3	green
`make lint` (three build configurations)	0 issues
`make test` (all three passes), ×2	green
`GOOS=windows go vet ./backend/download/`	passes (`GOOS=darwin` fails for the documented `wails/v3/pkg/mac` cgo reason, unrelated)

The probe is not committed: it is a measurement, and a stress test
that only sometimes fails is the class of thing this issue is about.
The before/after numbers above are what it produced.

Deliberately not done.

make ui-test, make e2e, make generate, make bindings — no
frontend, no .sql, no .templ, no bound signature.
CLAUDE.md untouched: it documents no test helper at this level, and
four open PRs are each editing that file in a different place.
The original failure is still not reproducible as a failure; what
is now impossible is the mechanism. A recurrence would mean a
different cause and deserves its own issue.

No new issues filed — stubYtDlp is the only test helper in the repo
that writes an executable and then execs it (cmd/indexbuild/deps_test.go
execs go list), so there is no second site.

Closes #146

**The issue.** `TestYtDlpSearchSkipsUnparseableLines` failed once in a full `make test` run and once in CI, both times on a tree whose diff contained no Go at all, with `fork/exec …/yt-dlp: text file busy`. #146 diagnosed the mechanism: the kernel refuses to exec a file that is open for writing anywhere in the process, and these tests are `t.Parallel()`, so a **sibling's** `fork` duplicates `stubYtDlp`'s write descriptor in the moment it is open and carries it past our close. **What changed.** The write in `stubYtDlp` is held under `syscall.ForkLock` — six lines and a comment, in one test file. No production code, nothing user-visible. **Why that and not either fix the issue named.** "Close the file before any sibling can fork" is not available: `os.WriteFile` has already closed it before anything execs, and the offending descriptor is a *copy* taken by a fork that was already in flight. "Write the stub once per package into a shared dir" narrows the window without removing it, since the first write of each script still races. `ForkLock` is the lock `syscall.forkExec` takes across the fork itself, so holding it over the write means no child can exist while the descriptor does, and a child forked earlier cannot hold one for a file that did not yet exist. It is declared on Windows and darwin as well as Unix, which matters because this helper skips at runtime rather than behind a build tag. **Verification.** | tier | result | | --- | --- | | throwaway probe in `backend/download` (12 goroutines × 200 real `stubYtDlp` calls + exec) | **189 / 176 / 178** ETXTBSY of 2400 before, **0 / 0 / 0** after | | standalone harness outside the repo | 271–302 of 4800 before, 0 of 4800 after | | `go test -race ./backend/download/`, ×3 | green | | `make lint` (three build configurations) | 0 issues | | `make test` (all three passes), ×2 | green | | `GOOS=windows go vet ./backend/download/` | passes (`GOOS=darwin` fails for the documented `wails/v3/pkg/mac` cgo reason, unrelated) | The probe is **not** committed: it is a measurement, and a stress test that only sometimes fails is the class of thing this issue is about. The before/after numbers above are what it produced. **Deliberately not done.** - `make ui-test`, `make e2e`, `make generate`, `make bindings` — no frontend, no `.sql`, no `.templ`, no bound signature. - `CLAUDE.md` untouched: it documents no test helper at this level, and four open PRs are each editing that file in a different place. - The original failure is still not reproducible *as a failure*; what is now impossible is the mechanism. A recurrence would mean a different cause and deserves its own issue. No new issues filed — `stubYtDlp` is the only test helper in the repo that writes an executable and then execs it (`cmd/indexbuild/deps_test.go` execs `go list`), so there is no second site. Closes #146

logan added 1 commit 2026-08-30 11:41:19 +00:00

test(download): write the yt-dlp stub under ForkLock

CI / check (push) Skipped

CI / e2e (push) Skipped

CI / check (pull_request) Successful in 2m48s

Details

CI / e2e (pull_request) Successful in 10m21s

Details

49445ded77

The kernel refuses to exec a file that is open for writing anywhere in
the process, and these tests are parallel: a sibling's fork duplicates
stubYtDlp's write descriptor in the moment it is open and carries it
past our close, so the exec a moment later fails with ETXTBSY. That is
the flake seen once locally and once in CI, both times on a tree with
no Go in its diff.

Closing sooner is not available -- os.WriteFile has already closed the
file before anything execs it -- and O_CLOEXEC does not help, because
the window is between another goroutine's fork and its own exec.
syscall.ForkLock is the lock forkExec takes across that fork, so
holding it over the write means no child can exist while the
descriptor does.

Measured on the helper itself under 12 concurrent writers: 176-189 of
2400 execs refused before, 0 of 2400 after, three runs each.

Closes #146

logan commented

2026-08-30 11:55:19 +00:00

CI is green. Run 18482: check success, e2e success (chromium
and WebKit both ran).

Not merging — leaving it for a human, with Status/In Progress still on
#146.

**CI is green.** Run 18482: `check` success, `e2e` success (chromium and WebKit both ran). Not merging — leaving it for a human, with `Status/In Progress` still on #146.

logan referenced this pull request

2026-08-30 11:55:19 +00:00

Flaky: TestYtDlpSearchSkipsUnparseableLines failed once in a full make test run #146

All checks were successful

CI / check (push) Skipped

Required

CI / e2e (push) Skipped

Required

CI / check (pull_request) Successful in 2m48s

Required

Details

CI / e2e (pull_request) Successful in 10m21s

Required

Details

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/146-stub-etxtbsy:fix/146-stub-etxtbsy

git checkout fix/146-stub-etxtbsy

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: yonlu/yellowjacket#235