Scoped library payloads: load the track list when a view needs it #287

2026-10-06T06:41:08Z

yonlu commented

2026-10-06 06:41:08 +00:00

One PR so the batch lands as one release rather than seven. The design, the before/after numbers and the things worth keeping are in .planning/NOTES.md ("What the library payloads actually cost, measured 2026-10-05").

Why: the app held ~830 MB on a real library, and almost all of it was one decision — every track in the library was fetched at launch, as one JSON object per track, whether or not anything was showing it. The reasons were real: track details looked a path up in that array, so it had to be loaded, and views took seconds when they fetched on mount, so loading it early was the fix. Both are answered directly now.

What is in it

Commit	Issue
fix(library): open track details by path, not from the whole library	#279
perf(smart-playlist): suggest rule values for what was typed	#279
perf(library): send the track list as a dictionary-encoded column table	#281
perf(library): hand the batch dialog the rows the view already has	#281
perf(library): load a collection when a view needs it, not at startup	#280
perf(library): patch what changed instead of reloading the list	#282
perf(database): bound the read pool's mapping and cache by measurement	#283
fix(explore): stop asking ListenBrainz after it refuses this client	#284
test(e2e): sweep the binding names a spec calls	—
docs(planning): what the library payloads cost, measured	—

Two fixes, so semantic-release will cut a patch. Nothing here changes a user-visible default.

Measured (50 000-track bulk seed, desktop dev build, e2e/perf/measure.mjs, which grew a memory section for this):

	before	after
Backend RSS at rest	543 MB	189 MB
Backend peak RSS	571 MB	281 MB
Go heap held	361 MB	7 MB
JS heap at rest	31.8 MB	5.2 MB
Binding bytes at rest	35.9 MB	1.7 MB
Track list payload	34.2 MB	10.45 MB
Bytes per track	929 B	167 B
Tracks first open → first row	38 ms	1 255 ms
Slowest first view open	44 ms	76 ms
Select all 50 000 → edit tags	88 ms	95 ms

The one number that got worse is the honest trade: Tracks now fetches when it is opened instead of having been paid for at launch. A raw binding call for the same table is 1 118–1 382 ms, so essentially none of that second is the decode or the render — it is the Go-side query, the column encode, and Wails encoding every result twice for a debug log that is off (#286). Hovering or tab-focusing the nav item prefetches, which is the ~100 ms before the click. #285 tracks windowed loading if a library ever outgrows the table.

Verified locally on this exact tree:

make lint — 0 issues across all three build configurations (Go 1.26.0, the version CI pins)
make test — all three passes with -race; two consecutive clean runs
make ui-test — 1180 passed, 112 files (see the caveat below)
make e2e (chromium, seeded headless app) — 259 passed
make bindings-check, make skill-check, make commit-check — clean

Two things a reviewer should know:

make ui-test is flaky on this workstation, not on this branch. At load average ~14 the browser provider's module fetches fail in a different handful of files each run ("Failed to import test file", "Cannot connect to the iframe") while every test that runs passes; --maxWorkers=1 --retry=2 reduces it and each file passes alone. The first full green run (1180/1180) was before the machine got busy. WebKit is CI-only, as always.
One make test run out of three failed and the log is gone — the two runs after it were clean, so I cannot name the package. Recorded rather than hidden.

Closes #279, #280, #281, #282, #283, #284.

One PR so the batch lands as **one release** rather than seven. The design, the before/after numbers and the things worth keeping are in `.planning/NOTES.md` ("What the library payloads actually cost, measured 2026-10-05"). **Why**: the app held ~830 MB on a real library, and almost all of it was one decision — every track in the library was fetched at launch, as one JSON object per track, whether or not anything was showing it. The reasons were real: track details looked a path up in that array, so it had to be loaded, and views took seconds when they fetched on mount, so loading it early was the fix. Both are answered directly now. **What is in it** | Commit | Issue | |---|---| | fix(library): open track details by path, not from the whole library | #279 | | perf(smart-playlist): suggest rule values for what was typed | #279 | | perf(library): send the track list as a dictionary-encoded column table | #281 | | perf(library): hand the batch dialog the rows the view already has | #281 | | perf(library): load a collection when a view needs it, not at startup | #280 | | perf(library): patch what changed instead of reloading the list | #282 | | perf(database): bound the read pool's mapping and cache by measurement | #283 | | fix(explore): stop asking ListenBrainz after it refuses this client | #284 | | test(e2e): sweep the binding names a spec calls | — | | docs(planning): what the library payloads cost, measured | — | Two `fix`es, so semantic-release will cut a **patch**. Nothing here changes a user-visible default. **Measured** (50 000-track bulk seed, desktop dev build, `e2e/perf/measure.mjs`, which grew a `memory` section for this): | | before | after | |---|---|---| | Backend RSS at rest | 543 MB | 189 MB | | Backend peak RSS | 571 MB | 281 MB | | Go heap held | 361 MB | 7 MB | | JS heap at rest | 31.8 MB | 5.2 MB | | Binding bytes at rest | 35.9 MB | 1.7 MB | | Track list payload | 34.2 MB | 10.45 MB | | Bytes per track | 929 B | 167 B | | Tracks first open → first row | 38 ms | 1 255 ms | | Slowest first view open | 44 ms | 76 ms | | Select all 50 000 → edit tags | 88 ms | 95 ms | The one number that got worse is the honest trade: Tracks now fetches when it is opened instead of having been paid for at launch. A raw binding call for the same table is 1 118–1 382 ms, so essentially none of that second is the decode or the render — it is the Go-side query, the column encode, and Wails encoding every result twice for a debug log that is off (#286). Hovering or tab-focusing the nav item prefetches, which is the ~100 ms before the click. #285 tracks windowed loading if a library ever outgrows the table. **Verified locally on this exact tree:** - `make lint` — 0 issues across all three build configurations (Go 1.26.0, the version CI pins) - `make test` — all three passes with `-race`; two consecutive clean runs - `make ui-test` — 1180 passed, 112 files (see the caveat below) - `make e2e` (chromium, seeded headless app) — 259 passed - `make bindings-check`, `make skill-check`, `make commit-check` — clean **Two things a reviewer should know:** - **`make ui-test` is flaky on this workstation, not on this branch.** At load average ~14 the browser provider's module fetches fail in a different handful of files each run ("Failed to import test file", "Cannot connect to the iframe") while every test that runs passes; `--maxWorkers=1 --retry=2` reduces it and each file passes alone. The first full green run (1180/1180) was before the machine got busy. WebKit is CI-only, as always. - **One `make test` run out of three failed and the log is gone** — the two runs after it were clean, so I cannot name the package. Recorded rather than hidden. **Closes** #279, #280, #281, #282, #283, #284.

yonlu added 10 commits 2026-10-06 06:41:08 +00:00

perf(smart-playlist): suggest rule values for what was typed 3cddf70c4d

The rule editor's value box built its suggestions from libraryStore's
whole-library arrays, which made it one more reason to load every
track at startup, and left it empty when they had not loaded.

SuggestSmartPlaylistValues answers with up to 50 distinct values of a
field that contain the typed text, from the same joined row the rules
match against, so a suggestion is always something a rule can match.
yj-combobox announces its filter text so the editor can ask, debounced
and cached per field and text.

Refs #279

fix(library): open track details by path, not from the whole library 4a9fe1d207

Track details from the queue, a playlist or a smart playlist looked
the track up in libraryStore's whole-library array, and returned
silently when it had not loaded, so the menu item did nothing. That
dependency is also why every track was fetched eagerly at startup
(20.5 MB at 26k tracks).

GetTracksByPaths answers for the paths in hand, and trackCache holds
the answers: bounded, coalesced into one call per task, and forgetting
what the retag, removal and scan events name. Every opener, and
track-details' own refresh after a save, go through it. A source sweep
pins the whole-library array to the store and the Tracks view.

Closes #279

perf(library): send the track list as a dictionary-encoded column table 7a1412383f

GetTracks answered with one JSON object per track: 20.5 MB at 26k
tracks, ~350 bytes a row of repeated key names, four cover URLs that
are identical across an album, and the album, artist and genre strings
repeated on every track. Encoding it was ~170 MB of transient Go
allocation, and parsing it the WebView's memory peak.

GetTrackTable replaces it: one array per column, every repeated
string stored once and sent as an index, genre lists interned, and only
the fields the Tracks view reads. LastPlayed and the three larger cover
tiers are left to the details dialog, which now fetches whole tracks by
path. Each row still goes through trackFromRow, so this is an encoding
of the one projection, not a second one. 167 bytes a track against 929
in the test library; a Go test holds it under a quarter of the object
encoding, and a Vitest test reads the Go columns and the generated
Track interface and fails if they drift.

An empty library is now an empty table rather than the "no tracks in
library" error the old binding returned.

Closes #281

perf(library): hand the batch dialog the rows the view already has 3d9828b847

Fetching whole tracks back for a dialog that reads only fields the
rows in hand already carry cost 3 s and 700 ms of blocked main thread
on a "select all" over 50 000 tracks — the regression #281 introduced
by dropping four fields from the list.

`showBatchTrackDetails` takes the rows; the Tracks view passes its
own, and the views that hold no library rows (Explore, the queue,
playlists) keep the path form over `trackCache`. The batch view's
merged-fields extractors read only list fields already, and a cover
tier the list does not carry falls back to the one it does.

Measured on 50 000 tracks: 50.4 ms, from 87.9 ms before #281.

Refs #281

perf(library): load a collection when a view needs it, not at startup 620151aa41

All four collections were fetched at DOMContentLoaded and refetched on
every invalidation, whichever view was showing. On a 26 138-track
library the track list was 20.5 MB of that, and encoding it cost the
backend ~170 MB of transient allocation — paid by someone looking at
Home, which draws none of it.

- The store warms only albums, artists and genres, on idle after first
  paint: 1.6 MB together, and what made those views instant.
- A nav item prefetches on hover and on keyboard focus, which is the
  ~100 ms before the click that a cold open would otherwise wait.
- An invalidation refetches what something had loaded, and nothing
  else — a scan no longer loads the track list of a library whose
  Tracks view nobody has opened.
- `index.html`'s first-paint `<track-list>` is `view-hidden`, and
  `index.ts` no longer activates it: it is markup, not a decision about
  which view the launch lands on, and activating it was what fetched
  the whole list for a landing on Home. The track list itself loads on
  view activation, which the shell drives.

Measured on 50 000 tracks: backend RSS at rest 543 → 296 MB, peak 571 →
296 MB, Go heap held 361 → 125 MB, JS heap 31.8 → 18 MB, binding bytes
at rest 35.9 → 12.1 MB, heap after a browse 36.5 → 22.7 MB. Tracks
first open 26 ms; slowest view open 57 ms.

Closes #280

perf(database): bound the read pool's mapping and cache by measurement c9b3048a20

#283 read as "64 MB of mmap and up to 32 MB of page cache per pool",
and it is — but on a 50 000-track library neither is what the
process's memory is: the RSS of the database mapping is 46 MB whether
the bound is 64 MB or 16 MB, because a mapping costs what the working
set touches. Quartering it and halving the page cache moved no query
either: 1 172 → 1 197 ms for the whole track list, 106 → 124 ms for the
album list, 6 → 8 ms for an FTS search, all inside the run-to-run
spread.

Kept, because a bound that costs nothing measurable is worth having on
a phone: five read connections' worth of mapping is 320 MB of address
space at 64 MB each against 80 MB here, and #52 was a low-memory kill.

Closes #283

fix(explore): stop asking ListenBrainz after it refuses this client bce8d27370

Every popularity request answered 401 on 2026-10-05: 399 in one minute
of a real library's discography backfill, one rate-limited request per
artist to be told the same thing, each with its own log line.

A token is a property of the installation, not of the artist, so the
first 401 is the answer for the rest of that client's life. The refusal
is latched, logged once at warning level, and returned as
ErrListenBrainzUnauthorized — separate from the generic HTTP error
because it is the one failure no retry can fix. A 500 still does not
latch, so a transient failure leaves the artist unmarked and the next
run asks again.

The backfill also stops feeding artists once the client is refused: the
rest of the pass would be the same 401, and the artists stay unmarked
for a run that has a token.

Closes #284

perf(library): patch what changed instead of reloading the list 5b236ca5af

A retag named the one file it rewrote and the store threw the whole
list away anyway: 20.5 MB across the IPC at 26 138 tracks to pick up one
new title. A soft rescan of an unchanged library did the same, though
its metrics say zero added, zero updated and zero removed.

- `TrackMetadataChanged` with a `filePath` re-reads that one row
  through `trackCache` and splices it in, replacing the array so the
  memoized filter and sort caches notice. The summaries are still
  refetched: a retag can move a track between albums. A failed patch
  falls back to the full reload, because the answer that cannot be
  wrong is the one to keep.
- A batch write names no paths (it can be thousands) and still reloads.
- `LibraryScanComplete` reads its own metrics and reloads only when the
  scan changed something; a payload of an unknown shape is treated as a
  change, so nothing can leave a stale list on screen.

Closes #282

docs(planning): what the library payloads cost, measured 2ed9e725e3

Plan 023 (#279–#284): the before/after table on 50 000 tracks, and the
six things worth keeping — including two measurements that pointed away
from where the work had been (mmap is a bound, and the remaining second
is Wails encoding every result twice).

test(e2e): sweep the binding names a spec calls

CI / check (push) Skipped

CI / e2e (push) Skipped

CI / check (pull_request) Successful in 7m25s

Details

CI / e2e (pull_request) Successful in 12m52s

Details

655434dcaa

Four specs still called `library.Library.GetTracks` after #281 replaced
it, and the suite reported twenty failures across transport, bottom-bar
and reduced-motion specs — none of which mention the library list. A
binding call carries only a method id, so a stale name fails at runtime
in whichever spec happens to call it.

`binding-names.spec.ts` reads every spec and harness file, extracts the
names they pass as strings, and checks them against the map derived
from the generated bindings tree. It asserts first that it read
something: a sweep over an empty glob passes and proves nothing.

It immediately found a second thing: `play-count.spec.ts` asserted that
a play refetched no collection by matching a `GetAll` prefix, which
matches exactly one real name (`GetAllLibrariesWithTrackCounts`) — so
the assertion held whatever the app refetched. It names the four
collection bindings now.

yonlu referenced this pull request

2026-10-06 07:05:55 +00:00

Tracks view: windowed, server-sorted rows for very large libraries (deferred) #285

All checks were successful

CI / check (push) Skipped

Required

CI / e2e (push) Skipped

Required

CI / check (pull_request) Successful in 7m25s

Required

Details

CI / e2e (pull_request) Successful in 12m52s

Required

Details

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin perf/279-284-scoped-library-payloads:perf/279-284-scoped-library-payloads

git checkout perf/279-284-scoped-library-payloads

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: yonlu/yellowjacket#287