Complete rewrite of the artist image pipeline:
STORAGE:
- Migration 16: artist_images table tracking source, URL, path,
primary flag, dimensions per image (up to 10 per artist)
- Directory structure: artist-images/{mbid[:2]}/{mbid}/ with
primary.jpg + primary_sm.jpg/_md.jpg/_lg.jpg thumbnails
- Miss marker (.miss file) prevents re-fetching artists with no image
SOURCES (priority order):
1. MusicBrainz direct image relations (Wikimedia Commons)
2. Wikidata P18 property (Wikimedia Commons)
3. Wikipedia lead image (NEW — via Wikidata sitelinks → Wikipedia API)
Each source is checked, deduplicated, and the first available
image becomes the primary with sm/md/lg thumbnail generation
(100px/200px/400px, matching cover art tier sizes).
ASSET SERVING:
- /artist-images/ path registered with Wails asset handler
- Serves files via http.FileServer from the artist-images directory
- Same pattern as /covers/ for cover art
ARTIST MODEL:
- Artist struct gains ImageSmall/ImageMedium/ImageLarge fields
- resolveArtistImages does bulk MBID lookup → disk stat for each
- Populated in GetAllArtists and GetAllArtistsByLibrary
GRID VIEW:
- artists-view uses model URLs directly (no more base64 data URLs)
- Size selection based on imageSize * devicePixelRatio (like cover-grid)
- Removed batch GetArtistImages call and in-memory cache — no longer needed
Replace per-artist sequential GetArtistMBID + GetArtistImageURL
calls (2 Wails round-trips × N artists) with a single batch
GetArtistImages(names[]) call that:
1. Resolves all names → MBIDs via AllArtistMBIDs() (one DB query)
2. Checks disk cache for each MBID via GetCachedImage (no network)
3. Returns map[name]→dataURL in one Wails bridge round-trip
Only returns already-cached images from the disk cache populated
by the index build. No network fetches triggered — artists whose
images haven't been cached yet keep the initial letter fallback
until the index build resolves them in the background.
Result: all cached artist images appear simultaneously on first
render instead of loading one-by-one over several seconds.
Migration 15: add in_library and is_similar INTEGER columns to
explore_index. Backfills in_library from existing library MBIDs.
Search index FTS5 query now includes personalization in scoring:
ORDER BY bm25(...) - (ln(pop+1) * 1.5)
- (in_library * 3.0) - (is_similar * 1.5)
For equal text+popularity scores:
- Library artist beats unrelated by 3.0 points
- Similar artist beats unrelated by 1.5 points
- Library > Similar > Neither
Tier 3 (library) entries get in_library=1 via markInLibrary.
Tier 4 (similar) entries get is_similar=1 via markSimilar.
MB result reranking (boostWithIndexPopularity) adds a 10M
popularity bonus for library artists, ensuring they always
rank above non-library artists with equal text relevance.
The search index's Tier 3 (library artists) depends on MBIDs from
the artists table. If the index built before a rescan populated
those MBIDs, library artists like Flatbush Zombies wouldn't be
indexed — they're not in the sitewide top 1000 and their name
didn't match via fuzzy matching.
Fix: OnAllScansComplete hook now calls InvalidateIndexDiscographies
before StartIndexBuild. This clears the discog_built timestamp so
Tiers 2-4 re-run incrementally, picking up any new library artists
whose MBIDs were just populated by the scan.
The rebuild is incremental — only artists not already in the index
get their discographies fetched.
Two changes:
1. Rerank MB results using index popularity (no API calls):
When the index is ready, boostWithIndexPopularity looks up each
MB result's MBID in the local index to get cached listen counts,
then reranks using the same blended score formula. This was
previously skipped entirely for speed, leaving MB results sorted
by text relevance only — obscure exact matches beat popular
partial matches.
2. Increase popularity weight across both scoring systems:
- Blended score: 60% popularity / 40% relevance (was 40/60)
- FTS5 index: ln(pop+1) * 1.5 factor (was 0.5)
Result: 'flatbush' → Flatbush Zombies (97) beats 'Flatbush'
nobody (61). Popular artists with partial name matches now
reliably outrank obscure exact matches.
Migration 14: add aliases TEXT column to explore_index, rebuild FTS5
with 3 columns (title, artist_name, aliases), recreate sync triggers.
Clears index build timestamps to force alias population on next build.
Artist image provider fetches inc=url-rels+aliases (single call, no
extra cost). GetAliases() extracts alias names from cached MB rels.
indexOneArtist stores aliases as space-separated text after image
resolution populates the cache.
Search query now uses BM25 blended scoring:
ORDER BY bm25(fts, 3.0, 1.0, 0.5) - (ln(popularity+1) * 0.5)
Column weights: title=3.0, artist_name=1.0, aliases=0.5
- Title matches score 3x higher than artist name matches
- Alias matches are helpful but don't dominate
- Popularity is a log-scaled boost, not an override
- Exact title match on niche entity beats weak match on mega-popular
Enables: 'rhcp' → Red Hot Chili Peppers, 'gnr' → Guns N' Roses,
'sabbath' → Black Sabbath (once index build runs with aliases).
The search index build and library scan both write to the same
single-connection SQLite DB. The index build runs continuous batch
transactions that can starve the scan's clearLibraryTables call,
causing the scan to silently hang without logging.
Fix: decouple index build from SetContext. The build now starts
AFTER the soft scan completes on startup. For full rescans, the
PreClear hook stops the index build, and PostScan restarts it.
Also made StartBuild/StopBuild safe for multiple calls:
- StartBuild is a no-op if already running
- StopBuild is a no-op if not running (no deadlock on done channel)
- done channel created per-build, not in constructor
Three features wired together:
1. 'In Library' badges on explore search results:
CheckLibraryMBIDs Wails binding batch-checks which search result
MBIDs exist in the local library. Green badges render on matching
artist cards and album cards.
2. Artist images on local artist-details page:
Local artist pages now call GetArtistMBID(name) to resolve the
MBID from tags, then GetArtistImageURL(mbid) to fetch the cached
Wikimedia photo. Falls back to initial-letter avatar.
3. Tier 3 search index uses direct MBIDs from tags:
buildTier3Library now reads artists.mbid column (from audio tags)
for direct MBID matching, falling back to name matching for
untagged artists. Eliminates false matches and catches artists
that name matching misses.
Migration 13 adds nullable mbid TEXT columns to artists,
release_groups, and recordings with partial indexes.
Metadata extraction (tags.go) now reads MusicBrainz IDs from Raw()
tags — handles both Vorbis (musicbrainz_artistid) and ID3v2
(MusicBrainz Artist Id) key formats.
Scan pipeline (library.go) updates MBIDs after entity upsert via
raw SQL UPDATE. Only sets mbid if currently NULL (preserves existing
values on rescan).
LibraryMBIDIndex (librarymbid.go) provides:
- CheckMBIDs: batch lookup for 'In Library' badges
- GetArtistMBID: single artist name→MBID lookup
- AllArtistMBIDs: full dump for search index Tier 3
MBIDs will be populated on next library rescan. Existing files
need a rescan to backfill.
Restructure indexOneArtist to run LB discography fetches and MB
artist image resolution concurrently. They use different rate
limiters (LB: 3 req/s, MB: 1 req/s) so they overlap without
contention.
Per artist, the indexer now runs two parallel pipelines:
LB pipeline: top-release-groups + top-recordings
MB pipeline: url-rels → Wikidata P18 → Wikimedia image fetch
All artist images are pre-cached during the index build instead
of being resolved on-demand during search. Total build time
drops from ~105 min (sequential) to ~63 min (parallel, MB-bound).
SearchIndex now takes ArtistImageProvider as a dependency. The
Service constructor creates artistImg before the index so both
can share it.
Two changes:
1. Artist image disk cache: ArtistImageProvider now fetches the
actual image bytes from Wikimedia Commons and caches them on
disk (~/.local/share/yellowjacket/artist-image-cache/{mbid}.jpg).
Returns base64 data URLs, same pattern as CoverArtProxy.
First lookup: resolve URL via MB/Wikidata + fetch image (~2s).
Subsequent: instant from disk cache.
404s cached as empty files to avoid re-fetching.
2. Top results artist photos: the Top Results section now shows
artist images from the artistImageCache, same as the Artists
section. Also shows englishName in the top card display name.
Artist image resolution was hitting musicbrainz.org with 10
concurrent unthrottled requests per search — enough to trigger
MB's rate limit rejection. Two fixes:
Backend: add dedicated 1 req/s RateLimiter for MB url-rels fetches
in ArtistImageProvider. Each fetch waits on the limiter before
the HTTP call. Results are cached 30 days so repeat lookups are
instant.
Frontend: switch loadArtistImages from concurrent fire-all to
sequential await loop. Each artist image loads one at a time,
images appear progressively as they resolve instead of all
failing from rate limit rejection.
The previous implementation called LookupArtist (rate-limited MB API)
before fetching url-rels, wasting a rate limiter slot. The fetchURL
for rels also bypassed the MB rate limiter, risking 503 rejections.
Rewrite: fetch MB url-rels once (direct HTTP, cached 30 days), parse
both image and wikidata relations from the same response, resolve
Wikimedia thumb URL. No dependency on MusicBrainzClient — just the
Cache for storage and a plain http.Client.
Also cleared 34 stale cached empty results from previous failed
resolution attempts that were blocking image lookup.
Add ArtistImageProvider that resolves artist MBIDs to photo URLs:
1. MB url-rels 'image' type → extract Commons filename → thumb URL
2. MB url-rels 'wikidata' type → Wikidata P18 property → thumb URL
3. No image → falls back to initial-letter avatar
Wikimedia Commons thumb URLs constructed via MD5 hash bucketing
(standard Commons URL scheme). Results cached in explore_cache
with 30-day TTL — subsequent lookups are instant.
Frontend: search results and artist detail page show artist photos
in the circular avatar when available. Images load async and
replace the initial-letter fallback on arrival. Artist detail page
fires the image fetch alongside the other 4 parallel data loads.
Architecture supports adding more sources (fanart.tv, etc.) by
extending the resolve() method's source chain.
Replace per-card GetThumbnail calls (10 round-trips) with a single
GetThumbnails batch call that fetches all visible album thumbnails
in one Wails bridge round-trip.
Backend GetThumbnails accepts []ThumbnailRequest and returns
map[mbid]→dataURL. Each request still checks library → disk cache
→ CAA in order, but the bridge overhead is 1 call instead of 10.
Frontend fires loadThumbnails() once after search results arrive.
Album cards render immediately with CAA URL fallback, then re-render
once the batch resolves with cached/local data URLs.
CoverArtProxy now checks three sources in order:
1. Local library (instant) — matches by album+artist name against
the release_groups/cover_art tables. Albums the user already
owns show their local cover art immediately.
2. Disk cache (instant) — previously fetched CAA thumbnails.
3. Cover Art Archive (network) — fetches and caches to disk.
Library index is built once on first access (sync.Once) from a
single SQL query joining release_groups → cover_art → artists.
Keyed by lowercased 'album\x00artist' for exact name matching.
GetThumbnail now takes (mbid, albumName, artistName) so the proxy
can check the library before falling back to CAA. Frontend passes
the album title and artist credit from the search result.
The CAA proxy was caching all fetch failures (including 503s and
timeouts) as empty files, treating them as permanent 'no art' misses.
During Internet Archive outages, this meant every album got cached
as having no cover art, and the cache persisted after IA recovered.
Now only 404 responses (no cover art exists) are cached as permanent
misses. 503, timeouts, and other transient errors are not cached,
so the next request retries the fetch.
Also cleared 33 incorrectly cached 0-byte miss files from a
concurrent IA outage.
Add CoverArtProxy that fetches cover art from CAA, caches the image
bytes on disk (~/.local/share/yellowjacket/cover-art-cache/), and
returns base64 data URLs via the GetThumbnail Wails binding.
First load: fetches from CAA (rate-limited), caches to disk.
Subsequent loads: instant from disk cache, no network.
404s: cached as empty files to avoid re-fetching.
Frontend explore-view loads thumbnails async via GetThumbnail()
calls that fire during render. Cached thumbnails appear as data
URLs directly in img src, bypassing the browser's HTTP stack.
Uncached thumbnails fall back to the CAA URL while the proxy
fetches in the background, then re-render with the cached version.
Also stores caa_id and caa_release_mbid in the search index's
extra_json for future direct Internet Archive URL construction.
Phases 2 (3 LB popularity POST calls) and 3 (3 MB discography
browse calls) were adding ~3-6 seconds to every search through
rate-limited API calls. Now they only run as a fallback during
first launch before the search index is built.
Once the index is ready (after Tier 1, <5 seconds from startup):
Phase 0: local FTS5 index query (instant)
Phase 1: MB search (3 concurrent calls, ~1s)
Phase 4: merge index hits (instant)
Phase 5: filter and cap (instant)
Search drops from ~4-7s to ~1s. The index already carries
popularity data and covers discography cross-referencing,
making the live API calls redundant.
Replace the single 7-day full rebuild with per-tier scheduling:
- Tier 1 (sitewide top lists): weekly refresh, 12 API calls
- Tiers 2-4 (discographies): monthly refresh, incremental —
only fetches discographies for artists not already indexed
On subsequent runs:
- If T1 is fresh, load cached artists from the index (~0 calls)
- If discographies are fresh, skip Tiers 2-4 entirely (~0 calls)
- If discographies are stale, diff against indexed set and only
fetch new artists that appeared in the sitewide lists
Add helpers: isMetaFresh (per-key freshness check),
loadCachedSitewideArtists (read artists from existing index),
filterUnindexed (diff artist list against indexed set).
After first build: typical startup is <5s (T1 cache load).
Monthly incremental: ~50-100 calls for newly appeared artists.
Instead of fixed 20 RGs + 100 recordings for every artist, scale
the budget by popularity using a power curve (exponent 0.3):
Radiohead (2.5M listens): 20 RGs, 100 recordings
Hans Zimmer (715K): 15 RGs, 71 recordings
Clutch (178K): 11 RGs, 50 recordings
Similar (~10K): 7 RGs, 27 recordings
Organic (unknown): 5 RGs, 10 recordings
Saves ~53% index size (~29 MB vs ~62 MB) with identical API calls.
The savings come from T4 similar artists (long tail) where full
discographies were wasteful. Top artists still get full coverage.
Rewrite SearchIndex with tiered background build:
Tier 1 — Sitewide instant (<5s, 12 calls): top artists, recordings,
and release groups across 4 time ranges. Searchable immediately.
Tier 2 — Sitewide full discog (~16min, 2881 calls): top 20 RGs +
top 100 recordings per sitewide artist (~1440 unique artists from
all_time/this_year/this_month/this_week union).
Tier 3 — Library artists (~4min, 664 calls): match local library
artist names against known MBIDs, index their full discographies.
Catches the user's personal taste that sitewide misses.
Tier 4 — Similar artists (~24min, ~4300 calls): fetch similar
artists from LB labs for each library artist, index their
discographies. Fans out into the user's taste neighborhood.
Tier 5 — Organic growth (0 calls): BrowseReleaseGroups now writes
to the search index in a background goroutine. Every artist page
view adds that artist's discography to the index for free.
Other changes:
- indexRGsPerArtist bumped 10→20 (96% vs 88% coverage)
- indexRecsPerArtist bumped 10→100 (track-name searchability)
- indexMinPopularity = 50 (cuts noise from long tails)
- Dedicated 3 req/s rate limiter for indexer
- Labs similar-artists endpoint at labs.api.listenbrainz.org
- Each tier marks index as ready on completion so search improves
progressively during the ~44min total build
Create SearchIndex in NewExploreService, start background build in
SetContext (on app startup). Search() now has 6 phases:
Phase 0: query local FTS5 index (instant, no API calls)
Phase 1: concurrent MB search
Phase 2: LB popularity boost
Phase 3: cross-reference artist discographies
Phase 4: merge index hits (prepend new entries, dedup by MBID)
Phase 5: filter and cap
Index hits for release groups/recordings not already in MB results
are prepended so popular albums surface even when MB search can't
find them. scalePopularity() maps raw listen counts to 0-100 scores
via log scaling for compatibility with the blended score system.
New SearchIndex struct in searchindex.go:
- Background build fetches top 1000 LB artists, then their top 10
release groups and top 10 recordings (2001 API calls total)
- Dedicated 3 req/s rate limiter for indexer (LB allows 30/10s)
- Bounded concurrency (3 goroutines) with progress logging
- Batch INSERTs in transactions of 100 rows
- FTS5 query with prefix matching ('for you' → 'for* you*')
- Results sorted by popularity descending
- Skips rebuild if index is < 7 days old
- Marks index ready from existing rows if build fails
- Context cancellation for clean shutdown
After MB search + popularity reranking, browse the discographies of
the top 3 artists and fuzzy-match the full query against album titles.
Matching albums not already in results are injected at the front.
Fuzzy matching uses substring containment with word-level ratio
(handles 'for you tatsuro' → 'FOR YOU' at 0.667) and word overlap
as fallback. Threshold: 0.4 ratio.
Example: 'for you tatsuro' now finds FOR YOU by 山下達郎 even though
MB text search treats 'for' and 'you' as stop words and never
returns it. The album is found via Yamashita's cached discography.
Extract primary English alias from MusicBrainz artist data when the
canonical name uses non-Latin script (CJK, Cyrillic, etc.). Display
it as the primary name in search results and artist detail header,
with the native script name as a subtitle beneath.
Example: 山下達郎 now shows 'Tatsuro Yamashita' prominently with
'山下達郎' as a subtitle. Artists with Latin names are unchanged.
Drop artists and recordings with blended score < 25 after popularity
reranking. Cap each entity slice to 15 server-side. Request 20 from
MB to allow filtering headroom. Reduces payload size and noise.
After MB search returns text-relevance-scored results, fetch bulk
popularity data from ListenBrainz (POST /1/popularity/{artist,
recording,release-group}) for all result MBIDs. Blend scores:
final = 0.6 * mb_relevance + 0.4 * log10_popularity
Log-scale normalization ensures massive artists don't drown out
everything, but popular results rise above obscure exact matches.
Release groups (no MB score) sort by raw popularity.
Three LB POST calls run concurrently — each hits a different
endpoint. All are rate-limited and cached (24h TTL).
Example: searching 'tatsuro' now ranks Tatsuro Yamashita (2.5M LB
listens, score 97) above 'tatsuro' vocaloid producer (4 listens,
score 64) despite the latter being an exact name match on MB.
The ListenBrainz popularity API returns snake_case JSON fields
(recording_name, artist_name, total_listen_count, recording_mbid)
but LBTopRecording used camelCase JSON tags for Wails serialization.
All fields silently deserialized as zero values — empty strings and
zero counts — producing ~8000 blank rows in the top tracks section.
Fix: add lbTopRecordingWire with snake_case tags for API unmarshal,
convert to LBTopRecording (camelCase) for Wails. Cap results at 10
to avoid rendering thousands of rows for prolific artists.
yonlu