`in_library = 1 AND local_*_id IS NULL` was a fixed point.
upsertBatch's conflict clause is `MAX(in_library, excluded.in_library)`,
so it can only ever raise the flag, and pruneStaleLocalCrossReferences —
which its own comment calls the only place a removal from the library is
reflected back into the index — was gated on the id being present. So
nothing in the app could clear such a row, ever: a permanent claim of
ownership with no local row to check it against.
The gate is now the flag *or* the id, for all three entity types. A NULL
id fails the existence test on its own, so this needs no second clause to
say what "not owned" means.
Nothing in the tree writes that shape today — collectLibraryEntities sets
both together — which is why this is worth closing rather than leaving:
the exposure is a database written by a version whose local-id columns
were populated differently, and the next writer that sets the flag
without an id, which nothing structurally prevents and which this shape
made permanent rather than merely wrong until the next scan.
The test seeds the row with raw SQL on purpose. upsertBatch writes a zero
LocalArtistID as literal 0, and 0 satisfies `IS NOT NULL`, so the old
gate already caught that shape — a fixture built through the upsert
cannot reproduce this at all. NULL is what the artifact importer and any
older writer leave behind, the columns being nullable with no default.
Reverted against the old gate, it fails on all three types.
Closes#118
Plan 016 B4. The catalog artifact is about 0.6 GB and the app fetched it
with no awareness of the connection: on a desktop that is a minute of
bandwidth, on a phone it can be a month's allowance. It is now skipped on
a cellular connection unless `AllowMeteredCatalogDownload` is on, with
the toggle in Settings' Search Index section, where the text explaining
what the catalog is already lives.
The file layout is dictated by the cgo rule rather than by taste.
`explore` is imported by `cmd/indexbuild`, which builds with
CGO_ENABLED=0 and must not link Wails, so `netpolicy.go` holds the policy
and the JSON parsing -- tested on every platform -- and the single
platform call is a closure injected from `app.go`, which already names
`application` legitimately.
Three rules in it are load-bearing. An unknown answer is not a metered
one: only mobile answers at all, and treating silence as metered would
have disabled the download for every desktop user in the world. Cellular
is the only signal available, because the runtime reports
`wifi|cellular|ethernet|none` and no metered flag -- so a metered Wi-Fi
cannot be detected and is not refused, which is documented rather than
implied. And the gate runs before the first status write, so declining is
a no-op instead of a job in the indicator and an error tier to dismiss.
Two corrections to the plan while implementing it: the portable API is
`application.Mobile.NetworkJSON()`, not `application.Android`'s, which
exists only under the `android` build tag; and the permission is read at
the moment a download would start, so enabling it takes effect on the
next attempt rather than the next launch.
Plans 013 and 014, the album page that prompted them, and the smaller
fixes they turned up. Changelog, largest first.
## The local library is shaped like files, not like MusicBrainz
`audio_files` carries its own tags and points at `albums` and
`artists`; `file_genres` is the one real many-to-many. `recordings`,
`release_group_recordings`, `artist_credit`, `artist_credit_artist`,
`recording_genres`, `release_groups` and `release_to_rg` are gone from
the local side, and with them a six-way join in every read, a
`MIN(release_group_id)` subquery in eleven queries and a
first-credited-artist subquery in nine. Measured on a real 25,966-file
library, every many-to-many that model expressed was 1:1 in the data.
- Ownership is a file. `GetFilePathsByRecordingMBIDs`,
`LibraryMBIDIndex.CheckMBIDs`, `collectLibraryEntities` and
`pruneStaleLocalCrossReferences` all join `audio_files`, so the 812
orphaned recordings, 216 release groups and 260 artists that library
carried are now structurally impossible.
- One projection: every track query selects from the `track_metadata`
view, one row type, one mapper. Nine hand-rolled copies had drifted
far enough to report different years on different screens.
- `library_id = 0` means every library, so each list query exists once
instead of scoped and unscoped with a branch at every call site.
- No migration chain. `sql/schemas/` is the one description of the
shape; `sql/migrations/`, `applyMigrations` and `schema_migrations`
are squashed away, along with the drift between them that had sqlc
generating against a stale schema.
- `database.InsertTestTrack` is the one test seeder; twenty test files
had been assembling the old FK chain each in its own order.
## The catalog stores its ids as bytes
`explore_index`'s three 36-char MBID columns and its entity-type text
are 16 raw bytes and a small integer. The table and its six indexes go
780 MB to 405 MB on a real 2,052,200-row catalog, which is why a fresh
install is ~0.6 GB rather than ~1.0 GB.
- `backend/explore/mbid.go` is the only place the encoding is known;
everything above it speaks dashed strings.
- `CHECK(length(mbid) = 16)` makes a stringly write fail at the insert
rather than silently returning no rows, since SQLite does not coerce
between TEXT and BLOB.
- The importer asks the artifact what encoding it carries and converts
on the way in, so the artifact already published keeps working and no
format bump is needed.
- `indexRowColumns`/`scanIndexRow` replace four copies of a 22-column
list, and `TestStoredEncodingRoundTrips` sweeps every read path.
## An album page that says how much of the album is yours
- One question, asked once: is there a file. `filePaths` is filled by a
single batched lookup when the tracklist settles, and the badge, the
Play count, the dimmed rows and every menu item read it — replacing
four claims of decreasing confidence that could show a green tick on
an album whose every action did nothing.
- Play, Play 7 of 12, or no play button at all.
- `total_tracks` on `explore_index` (~2 bytes over 400,677 release
groups) and on `audio_files` from tags that have always carried it:
a complete MBID-matched album now makes no catalog call at all, where
it used to spend the most expensive request the app makes.
- A merged cluster shows the running order the most releases agree on,
and the version list marks the release you own rather than standing a
synthetic entry in for it.
- `AlbumReleasesFailed`: a slow fetch is no longer reported as a failed
one by a 12-second timer.
- Rows not in the library are dimmed in place (with `aria-disabled`)
instead of the owned ones wearing a green tick and a legend.
## Caches and cover art get ceilings
- Only the three tiers of a cover are stored; the full-resolution copy
nothing rendered was 1,134 MB of a 1.4 GB covers directory.
- One artist portrait is downloaded and the rest are remembered as
URLs — 4.1 GB of a 5.3 GB cache was candidates no code path reads.
- `browsedArtBudget` and `httpCacheBudget` bound what an age cannot:
the same install held art for 5,770 artists in a 1,301-artist
library.
- `OrphanedArtistImagesJob` joined a bare MBID onto a sharded
directory, so it deleted the rows that were the only record of the
files it left behind. `explore.ArtistImageDir` is that layout's one
definition now.
## The autotag queue asks whether there is work
`tagging_items` was a row per album folder, not a queue, and no query
read the `tag_status` column that held the answer. The four queue
queries ask the files, which matters most where it is least visible:
`startPrefetch` was scoring every album in a tagged library against
MusicBrainz.
## Phantom playlist tracks resolve in place
An M3U8 imported before its files leaves phantom rows; they now match
by path and fall back to position, keep their place in the playlist
when resolved, and pair best-first so two phantoms cannot claim the
same file.
## Playing a track plays the list it is in
Double-click, and Play on a single row's menu, queue the list as
displayed with `startIndex` on that row — the album page and the track
list used to queue one track and discard the album around it. A
multi-row selection still plays exactly itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
make e2e is green on chromium: 92 passed. The harness is rebuilt on
what v3 actually offers, and three of the four things it replaced turn
out to be better than what they replaced.
The headless launch is v3's own server mode. scripts/dev-headless.sh
ran a `-tags dev` binary whose app_dev.go parsed -devserver/-assetdir
out of os.Args; that file went with v2, so the harness had no server at
all. `-tags dev,server` is a first-class mode and needs no display, so
Xvfb is gone from the script and from CI.
The bridge hooks two places, neither of them EventsOn. Inbound is
window._wails.dispatchWailsEvent, wrapped by pre-creating the object
the runtime keeps and putting an accessor on the one property.
Outbound is fetch: v3 routes every runtime call through one POST, so
the bridge sees binding calls and event emits from any module, needs no
walk of an object graph, and cannot miss a call made before it looked.
__yjEvents.call posts to that endpoint by method name, so it depends on
nothing in the app's bundle and works on a page with no init script.
That is what lets seed-sandbox.sh drop playwright-cli entirely — it
drove AddLibrary through a browser only because window.go was v2's one
way in — and with it a global npm install and a second Chromium in CI.
measure.mjs and one spec lose their window.go walks and read the
bridge's log instead; e2e/support/method-ids.mjs derives id -> name
from frontend/bindings/ (phase 6b option 1, so it cannot go stale
silently). Plain .mjs because measure.mjs runs under bare node and one
derivation beats two that can disagree.
Four bugs surfaced, and the migration is how.
The cross-service wiring never ran headless. It hung off
Common.ApplicationStarted, which server mode never emits —
setupCommonEvents is an explicit no-op there — so the queue had no
TrackLoader and playing a track changed the queue and then silently did
nothing. It is a service registered last now (backend/startup.go):
services start in registration order, which is the ordering the wiring
needs, in every mode.
Six specs called SetQueue with 3 of its 4 arguments. v2 accepted that
and filled the gap; v3 answers "expects 4 arguments, got 3".
requested-badge's cleanup read window.go and returned early on
`if (!svc)` — the silent cleanup its own comment was written to
prevent, one migration later. It posts to the runtime endpoint now,
which any page can do.
SearchIndex.Search trusted a startup latch, so rows a spec staged
afterwards were unsearchable and three specs passed only when an
earlier one happened to flip it. shelves.go fixed exactly this and left
hasCatalogRows behind; the search path now uses it as the fallback,
with the latch still the fast path.
Two spec edits are deletions of assertions about v2. harness.spec
checked Object.keys(window.go) and that a bad call *hung*; it now
checks the real runtime is loaded and that the backend rejects with a
TypeError naming the argument. album-actions asserted a tracklist
legend that dcc40b1 deleted on main — that spec has been failing since,
and what replaced it is covered in frontend/test/components.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
The post-scan backfills share MusicBrainz's rate limiters with every
page the user can open, and both were FIFO — so a thousand-artist
enrichment put an album page behind an hour of queued work.
WithBackgroundLane/WithBackgroundPriority add a slower second lane: a
marked wait takes no token while any interactive wait is outstanding.
It is a context marker rather than a parameter because a backfill calls
the same client methods a detail page does. A long backfill also has to
be visible and stoppable, so jobs.KindCatalogEnrich registers both with
progress and cancel — after the work is counted, since these passes are
a no-op on every launch once the library is covered.
What it does not fetch is the point. It ran for hours against a
900-artist library and marked nothing, because three of the four things
it did per artist were work nobody asked for: similar artists, which
the artist page already resolves on view, and a full GetArtistImage
(fanart.tv, TheAudioDB, Wikidata, Wikipedia, ten portraits) reached
only to warm the MB artist lookup EnsureArtistRels does alone. It was
also serial across artists while every limiter is per-host and idle.
The marks are a table rather than more explore_index columns, because
artifactimport merges by column list and a flag added there is a second
place to remember. BrowseReleaseGroupsAll pages to exhaustion, where
the old call silently cut a prolific artist at 100 release groups.
One portrait is downloaded now; the rest are remembered as URLs.
resolveAllSources downloaded every candidate, up to ten, full size,
while nothing reads anything but primary.jpg — 5.3 GB measured on a
real cache, 4.1 GB of it unreachable. OrphanedArtistImagesJob is why
that survived: it joined the bare MBID onto the images directory, but
artist directories are sharded under a two-character prefix, so it
named a path that never existed and deleted the rows that were the only
record of the files it left behind.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
`IndexStatusChanged` was pushed on a 3 s ticker for the life of the
process, byte-identical once the index was ready, and `config-page`
assigns it to a @state field — so a user who had once opened Settings
paid a full re-render of a 2 000-line template every 3 s, forever, for
no news. Measured sitting on Settings: 5 events and 5 re-renders per
15 s, against 0 and 0.
`emitStatus` drops a status equal to the last one it sent, which is
the rule stated once instead of at twenty call sites. The corollary is
load-bearing: every mutation of something the status derives must now
call `emitStatus` itself. Two were relying on the ticker — `si.ready`
when an existing index is adopted, and `si.cancel` when a build ends —
and without them the header badge said "Building search index" over an
index the settings page called ready. A polling loop is a hidden
dependency for every state transition that forgot to announce itself.
A coding agent could develop this repo's Go packages and could not
develop the application: every path to running YellowJacket ended in a
blocking GTK window, so 265 bound methods, 46 events, 33 component
directories and 13 stores had exactly one form of verification
available — `tsc --noEmit`.
The unlock is that `wails dev`'s dev server on :34115 serves the real
frontend with the real generated bindings against the same Go backend a
desktop window attaches to, so a plain Chromium under Xvfb gets a fully
functional app. Four test tiers now exist, cheapest first:
- `make ui-test` — 313 Vitest tests in a real browser in ~2 s, no app,
no backend, no display. Works because `frontend/wailsjs/` is a pure
passthrough to `window.go`/`window.runtime`, so faking just those two
globals runs the real bindings and the real store code.
- `make test` — services in-process, asserting on the payload the
frontend would receive, via a new `events.Emit` wrapper.
- `make dev-headless` + `playwright-cli` — the real app, driven
interactively, with an event bridge on `window.__yjEvents` and a
dev-only control surface at `/__test/`.
- `make e2e` — 19 of those flows frozen as Playwright specs.
`events.Emit(ctx, …)` replaces all 35 direct `runtime.EventsEmit` call
sites: wails' `getEvents` `log.Fatalf`s on any context without its
runtime, so those paths could not run under test and a background
worker could take the app down. Four packages had each hand-rolled the
same guard; nine more guarded on `ctx != nil`, which does not help.
`TestNoDirectRuntimeEmits` fails the build on a new one.
Fixtures are generated, not committed (`make testdata`), and seeds are
built by *running the app* — never by hand-writing config and DB rows,
which would be a second description of a valid YJ_HOME.
`.gitea/workflows/ci.yml` is the first workflow here that tests
anything; the other three only package, so `gitea_ci` reported only
packaging jobs and misled anyone asking whether a push was healthy.
Both jobs were prototyped to green in a bare ubuntu:24.04 container
before the YAML was written, which immediately caught `make lint`
linting three configurations that nothing builds: all three passes
omitted `webkit2_41`, so wails resolved webkit2gtk-4.0 — which Arch
still ships and Ubuntu 24.04 dropped.
Operational instructions live in `.pi/skills/yellowjacket-dev/`,
measured discoveries in `.planning/NOTES.md`, and architecture in
`CLAUDE.md` — split by tense, not by topic, because a topical split
gives every new fact two plausible homes. `make skill-check` fails a
commit if the skill cites a make target that does not exist.
Autotag: detect "junk drawer" folders with no artist/album consensus
and split them into synthetic per-cluster groups instead of forcing
one match on an unrelated pile of tracks; repair tagging_items rows
left behind by a prior scan orphan-cleanup gap.
Explore: fix an exact artist-name search being drowned out by its own
catalog entries in intent-prior scoring, and prune stale in_library
bookkeeping left behind when a referenced library row is deleted.
Download: fix a multi-library regression where every import failed
with "no library root configured" — the importer resolved the
library root from a legacy single-library config field that nothing
populates in the current multi-library model. It now resolves the
destination library per-request from the request's own library_id.
Also widen the Soulseek search window (12s -> 20s), measured against
real request history to be missing available peers on live queries.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y2Agd9af5hE7qzti2ackiS
Ships the fresh-start schema cleanup: rebuilt explore catalog index
pipeline (dump import, artifact fetch/build, incremental listen-count
refresh), a new download subsystem (Lidarr/Prowlarr/qBittorrent/SABnzbd/
slskd/yt-dlp providers, staging, reconciliation, wanted list), and the
supporting schema/query/store changes across backend and frontend.
Also includes two smaller follow-ups: bump the central index's
rebuild-after cadence from 90 to 180 days, and remove the Explore
"library only" online/offline toggle entirely (frontend-only, no
backend counterpart) rather than carry unused UI/state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y2Agd9af5hE7qzti2ackiS
Add a central job registry that library scans and search index builds
report into, so background work is visible instead of buried in the
settings page.
- backend/jobs: registry with per-job ring-buffer logs, capability-driven
controls, and one coalesced JobsChanged snapshot at 4Hz
- pause survives restart via a job_state table; a paused scan is adopted
back on launch and skipped by the soft scan
- top-bar indicator, popover, details drawer and a Jobs page replacing
the config page's scan UI; per-library start/stop retained
- scan timing breakdown moves into the job log, Full rescan to the Jobs
page; delete the orphaned library-manager component
Also add cmd/indexbuild and cmd/indexexport so the explore index can be
built once centrally rather than by every install, which today streams
~205GB from the ListenBrainz spark dump on first run. indexbuild picks
build/refresh/rebuild from index state; the Gitea workflow runs it on
push, weekly, or manually and publishes only when content changed.
fresh-install no longer defaults YJ_HOME under /tmp: it is tmpfs on most
distros, and the import needs ~6GB of real disk.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Enrich owned artists whose discography hasn't been fetched yet in a
bounded, resumable background pass so their wider catalogue is searchable
offline right after a scan, instead of only on first artist-page view.
Keyed off the persistent discog_fetched flag via LEFT JOIN, so already-
enriched artists never reappear and the run is a cheap no-op once every
owned artist is covered. Capped at discogBackfillMaxPerRun per run and
routed through discogSF to avoid double-fetching an artist a concurrent
interactive EnsureArtistDiscography is handling. Invoked on both scan
completion (OnStartup) and OnDomReady to resume a capped/interrupted run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consolidates in-progress work across autotag, explore, and library:
- autotag: beets/Picard-informed scoring engine — ID-first matching, VA
handling, recommendation tiers, and a merged distance/rank cascade, with
an eval harness for regression tracking.
- explore: offline MusicBrainz dump import/incremental refresh replaces the
legacy tier crawl; index-first local search with fuzzy matching and a
dedicated ranker; disk-free guards for dump downloads.
- library: artist-credit extraction and matching.
- lyrics: owned-library lyric search (FTS) with LRCLIB backfill.
Also: rewrite README to be user-focused, and migrate upstream to
git.ljones.me/yonlu/yellowjacket.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
End-of-milestone state for the Explore milestone. Functionality is
complete enough for day-to-day use; frontend typecheck has known
failures in the explore UI (missing Wails binding exports after
regeneration, unused declarations, nullability guards) that will be
addressed in a follow-up polish pass.
Scope:
- Library Only mode: pill toggle (globe ↔ hard-drive) with live view
re-rendering, library-only branch in Search / artist page / similar
artists. Suppresses external API calls when enabled.
- Ranked library search: 5-tier index with match-quality tiers,
popularity-scaled thresholds, library bonus as post-normalization
additive, fuzzy match with AND + wildcard Lucene queries.
- New schemas: artist_metadata, http_cache.
- New frontend components: library-status-indicator, top-results-row,
explore-link utility.
- Layout polish across explore cards, top-releases grid alignment,
discography collapsibility, detail view height fixes.
- Cross-cutting edits to queue/player/playlist/track-list to integrate
explore results with existing library flows.
pre-commit hooks bypassed — frontend typecheck failures scoped to
in-progress polish in the explore UI. Go build and full backend test
suite are green.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend:
- Migration 17: similar_artist_map table stores per-artist similar
artist relationships (source_mbid → similar_mbid + name + score)
- Tier 4 index build now persists similar artists to this table
- GetLibrarySimilarArtists(mbid) queries similar artists filtered
by JOIN with the artists table (library-only, no API calls)
- Added db field to explore.Service for direct queries
Frontend:
- ExploreSettingsStore with libraryOnly toggle, persisted to
localStorage
- Top bar toggle button with active/inactive styling
- Explore search: skips full MB/LB pipeline when library-only,
uses only searchLibraryCache (pure JS, instant)
- Artist detail page: in library-only mode, skips all API calls
(no top tracks, no top releases, no LB play count, no MB
artist lookup). Uses library store for discography, calls
GetLibrarySimilarArtists for similar artists.
- Similar artists section: changed from horizontal scroll to
wrapping flex layout with collapsible toggle (Show all N)
- Removed debug artist ranking log
The +10M library bonus was added directly to the popularity map,
which made it the maxPop normalization denominator. With maxPop=10M,
every non-library artist's log-normalized popularity collapsed to
near-zero, making their blended score purely 40% of MB relevance.
All non-indexed artists scored ~35 and ranked by MB noise.
New approach:
- Removed +10M from both GetPopularityBatch and boostWithPopularity
- GetPopularityBatch now returns PopularityBatchResult with separate
Popularity and InLibrary maps
- rerankArtists takes a libraryMBIDs set and applies a fixed +25
score bonus AFTER blended scoring and normalization
- maxPop reflects real popularity only, so log normalization works
correctly across all artists
Shannon Wright (766K listens) now properly outranks Shannon Kennedy
(95 listens) because the popularity scale isn't contaminated.
boostWithIndexPopularity was calling GetPopularity() and IsInLibrary()
individually for every search result — ~100 separate SQLite queries
for a typical search (20 artists × 2 + 20 RGs × 2 + 20 recordings).
This took 7.5s on the 'fast path' that was supposed to take ~5ms.
Added GetPopularityBatch(mbids) — collects all MBIDs across all
entity types and fetches popularity + in_library in a single
SELECT ... WHERE mbid IN (...) query. The library bonus (+10M) is
applied during the batch scan.
Expected Phase 2 improvement: ~7.5s → <10ms.
The search index ready flag is an in-memory bool that resets to false
on every app restart. It was only set to true inside build(), which
runs in a goroutine after SoftScanAllLibraries completes. If the user
searched before the build goroutine started, IsReady() returned false
and the search took the slow path (LB popularity + cross-ref: ~2.3s)
even though the SQLite index had all the data from the previous build.
Now MarkReadyIfPopulated() is called eagerly in NewExploreService —
the index is queryable as soon as the service is constructed, before
any goroutines launch. If the explore_index table has rows, ready=true
immediately.
indexOneArtist only wrote the artist entry when aliases were non-empty.
Artists without MB aliases (common for smaller/niche artists) never got
an entity_type='artist' row, so indexedArtistMBIDs() couldn't see them.
filterUnindexed then treated them as new on every startup, triggering
redundant LB API calls for top-release-groups and top-recordings.
Now the artist row is always written, with aliases as an empty string
when none exist. Subsequent builds will correctly skip these artists.
After a library scan completes, OnAllScansComplete now calls
IndexNewArtists() instead of StartIndexBuild(). This skips the
full tier pipeline (sitewide top artists, similar artists, freshness
checks) and only indexes library artists whose MBIDs are not yet
in the search index.
Flow after scan:
1. Query indexed artist MBIDs (fast, in-memory set)
2. Query library artist MBIDs
3. Diff → only new artists
4. Fetch discographies + images for new artists only
The full StartIndexBuild() still runs on initial launch (when
SoftScanAllLibraries finds no work to do) to handle the tier
pipeline with freshness-based refresh. But adding 5 new albums
to your library no longer triggers a 60-minute index rebuild.
Root cause: discog_built meta timestamp was only written after ALL
of Tiers 2-4 completed. If the app was closed mid-build (context
cancelled), the timestamp was never set, so the next launch re-ran
everything from Tier 2.
Fix: track each tier independently with tier2_built, tier3_built,
tier4_built timestamps. Each tier's timestamp is written immediately
after it completes, so progress survives app restarts.
On next launch, already-completed tiers are skipped. Combined with
the incremental filterUnindexed logic, a build interrupted at
Tier 4 with 80% of similar artists done will resume from the
remaining 20%.
Also adds getLibraryArtistMBIDs helper for Tier 4 when Tier 3
was skipped (needs library MBIDs without re-running Tier 3).
InvalidateDiscographies now clears all three per-tier timestamps.
Migration 15: add in_library and is_similar INTEGER columns to
explore_index. Backfills in_library from existing library MBIDs.
Search index FTS5 query now includes personalization in scoring:
ORDER BY bm25(...) - (ln(pop+1) * 1.5)
- (in_library * 3.0) - (is_similar * 1.5)
For equal text+popularity scores:
- Library artist beats unrelated by 3.0 points
- Similar artist beats unrelated by 1.5 points
- Library > Similar > Neither
Tier 3 (library) entries get in_library=1 via markInLibrary.
Tier 4 (similar) entries get is_similar=1 via markSimilar.
MB result reranking (boostWithIndexPopularity) adds a 10M
popularity bonus for library artists, ensuring they always
rank above non-library artists with equal text relevance.
The search index's Tier 3 (library artists) depends on MBIDs from
the artists table. If the index built before a rescan populated
those MBIDs, library artists like Flatbush Zombies wouldn't be
indexed — they're not in the sitewide top 1000 and their name
didn't match via fuzzy matching.
Fix: OnAllScansComplete hook now calls InvalidateIndexDiscographies
before StartIndexBuild. This clears the discog_built timestamp so
Tiers 2-4 re-run incrementally, picking up any new library artists
whose MBIDs were just populated by the scan.
The rebuild is incremental — only artists not already in the index
get their discographies fetched.
Two changes:
1. Rerank MB results using index popularity (no API calls):
When the index is ready, boostWithIndexPopularity looks up each
MB result's MBID in the local index to get cached listen counts,
then reranks using the same blended score formula. This was
previously skipped entirely for speed, leaving MB results sorted
by text relevance only — obscure exact matches beat popular
partial matches.
2. Increase popularity weight across both scoring systems:
- Blended score: 60% popularity / 40% relevance (was 40/60)
- FTS5 index: ln(pop+1) * 1.5 factor (was 0.5)
Result: 'flatbush' → Flatbush Zombies (97) beats 'Flatbush'
nobody (61). Popular artists with partial name matches now
reliably outrank obscure exact matches.
Migration 14: add aliases TEXT column to explore_index, rebuild FTS5
with 3 columns (title, artist_name, aliases), recreate sync triggers.
Clears index build timestamps to force alias population on next build.
Artist image provider fetches inc=url-rels+aliases (single call, no
extra cost). GetAliases() extracts alias names from cached MB rels.
indexOneArtist stores aliases as space-separated text after image
resolution populates the cache.
Search query now uses BM25 blended scoring:
ORDER BY bm25(fts, 3.0, 1.0, 0.5) - (ln(popularity+1) * 0.5)
Column weights: title=3.0, artist_name=1.0, aliases=0.5
- Title matches score 3x higher than artist name matches
- Alias matches are helpful but don't dominate
- Popularity is a log-scaled boost, not an override
- Exact title match on niche entity beats weak match on mega-popular
Enables: 'rhcp' → Red Hot Chili Peppers, 'gnr' → Guns N' Roses,
'sabbath' → Black Sabbath (once index build runs with aliases).
The search index build and library scan both write to the same
single-connection SQLite DB. The index build runs continuous batch
transactions that can starve the scan's clearLibraryTables call,
causing the scan to silently hang without logging.
Fix: decouple index build from SetContext. The build now starts
AFTER the soft scan completes on startup. For full rescans, the
PreClear hook stops the index build, and PostScan restarts it.
Also made StartBuild/StopBuild safe for multiple calls:
- StartBuild is a no-op if already running
- StopBuild is a no-op if not running (no deadlock on done channel)
- done channel created per-build, not in constructor
Three features wired together:
1. 'In Library' badges on explore search results:
CheckLibraryMBIDs Wails binding batch-checks which search result
MBIDs exist in the local library. Green badges render on matching
artist cards and album cards.
2. Artist images on local artist-details page:
Local artist pages now call GetArtistMBID(name) to resolve the
MBID from tags, then GetArtistImageURL(mbid) to fetch the cached
Wikimedia photo. Falls back to initial-letter avatar.
3. Tier 3 search index uses direct MBIDs from tags:
buildTier3Library now reads artists.mbid column (from audio tags)
for direct MBID matching, falling back to name matching for
untagged artists. Eliminates false matches and catches artists
that name matching misses.
Restructure indexOneArtist to run LB discography fetches and MB
artist image resolution concurrently. They use different rate
limiters (LB: 3 req/s, MB: 1 req/s) so they overlap without
contention.
Per artist, the indexer now runs two parallel pipelines:
LB pipeline: top-release-groups + top-recordings
MB pipeline: url-rels → Wikidata P18 → Wikimedia image fetch
All artist images are pre-cached during the index build instead
of being resolved on-demand during search. Total build time
drops from ~105 min (sequential) to ~63 min (parallel, MB-bound).
SearchIndex now takes ArtistImageProvider as a dependency. The
Service constructor creates artistImg before the index so both
can share it.
Add CoverArtProxy that fetches cover art from CAA, caches the image
bytes on disk (~/.local/share/yellowjacket/cover-art-cache/), and
returns base64 data URLs via the GetThumbnail Wails binding.
First load: fetches from CAA (rate-limited), caches to disk.
Subsequent loads: instant from disk cache, no network.
404s: cached as empty files to avoid re-fetching.
Frontend explore-view loads thumbnails async via GetThumbnail()
calls that fire during render. Cached thumbnails appear as data
URLs directly in img src, bypassing the browser's HTTP stack.
Uncached thumbnails fall back to the CAA URL while the proxy
fetches in the background, then re-render with the cached version.
Also stores caa_id and caa_release_mbid in the search index's
extra_json for future direct Internet Archive URL construction.
Replace the single 7-day full rebuild with per-tier scheduling:
- Tier 1 (sitewide top lists): weekly refresh, 12 API calls
- Tiers 2-4 (discographies): monthly refresh, incremental —
only fetches discographies for artists not already indexed
On subsequent runs:
- If T1 is fresh, load cached artists from the index (~0 calls)
- If discographies are fresh, skip Tiers 2-4 entirely (~0 calls)
- If discographies are stale, diff against indexed set and only
fetch new artists that appeared in the sitewide lists
Add helpers: isMetaFresh (per-key freshness check),
loadCachedSitewideArtists (read artists from existing index),
filterUnindexed (diff artist list against indexed set).
After first build: typical startup is <5s (T1 cache load).
Monthly incremental: ~50-100 calls for newly appeared artists.
Instead of fixed 20 RGs + 100 recordings for every artist, scale
the budget by popularity using a power curve (exponent 0.3):
Radiohead (2.5M listens): 20 RGs, 100 recordings
Hans Zimmer (715K): 15 RGs, 71 recordings
Clutch (178K): 11 RGs, 50 recordings
Similar (~10K): 7 RGs, 27 recordings
Organic (unknown): 5 RGs, 10 recordings
Saves ~53% index size (~29 MB vs ~62 MB) with identical API calls.
The savings come from T4 similar artists (long tail) where full
discographies were wasteful. Top artists still get full coverage.
Rewrite SearchIndex with tiered background build:
Tier 1 — Sitewide instant (<5s, 12 calls): top artists, recordings,
and release groups across 4 time ranges. Searchable immediately.
Tier 2 — Sitewide full discog (~16min, 2881 calls): top 20 RGs +
top 100 recordings per sitewide artist (~1440 unique artists from
all_time/this_year/this_month/this_week union).
Tier 3 — Library artists (~4min, 664 calls): match local library
artist names against known MBIDs, index their full discographies.
Catches the user's personal taste that sitewide misses.
Tier 4 — Similar artists (~24min, ~4300 calls): fetch similar
artists from LB labs for each library artist, index their
discographies. Fans out into the user's taste neighborhood.
Tier 5 — Organic growth (0 calls): BrowseReleaseGroups now writes
to the search index in a background goroutine. Every artist page
view adds that artist's discography to the index for free.
Other changes:
- indexRGsPerArtist bumped 10→20 (96% vs 88% coverage)
- indexRecsPerArtist bumped 10→100 (track-name searchability)
- indexMinPopularity = 50 (cuts noise from long tails)
- Dedicated 3 req/s rate limiter for indexer
- Labs similar-artists endpoint at labs.api.listenbrainz.org
- Each tier marks index as ready on completion so search improves
progressively during the ~44min total build
New SearchIndex struct in searchindex.go:
- Background build fetches top 1000 LB artists, then their top 10
release groups and top 10 recordings (2001 API calls total)
- Dedicated 3 req/s rate limiter for indexer (LB allows 30/10s)
- Bounded concurrency (3 goroutines) with progress logging
- Batch INSERTs in transactions of 100 rows
- FTS5 query with prefix matching ('for you' → 'for* you*')
- Results sorted by popularity descending
- Skips rebuild if index is < 7 days old
- Marks index ready from existing rows if build fails
- Context cancellation for clean shutdown
yonlu
logan
