The post-scan backfills share MusicBrainz's rate limiters with every
page the user can open, and both were FIFO — so a thousand-artist
enrichment put an album page behind an hour of queued work.
WithBackgroundLane/WithBackgroundPriority add a slower second lane: a
marked wait takes no token while any interactive wait is outstanding.
It is a context marker rather than a parameter because a backfill calls
the same client methods a detail page does. A long backfill also has to
be visible and stoppable, so jobs.KindCatalogEnrich registers both with
progress and cancel — after the work is counted, since these passes are
a no-op on every launch once the library is covered.
What it does not fetch is the point. It ran for hours against a
900-artist library and marked nothing, because three of the four things
it did per artist were work nobody asked for: similar artists, which
the artist page already resolves on view, and a full GetArtistImage
(fanart.tv, TheAudioDB, Wikidata, Wikipedia, ten portraits) reached
only to warm the MB artist lookup EnsureArtistRels does alone. It was
also serial across artists while every limiter is per-host and idle.
The marks are a table rather than more explore_index columns, because
artifactimport merges by column list and a flag added there is a second
place to remember. BrowseReleaseGroupsAll pages to exhaustion, where
the old call silently cut a prolific artist at 100 release groups.
One portrait is downloaded now; the rest are remembered as URLs.
resolveAllSources downloaded every candidate, up to ten, full size,
while nothing reads anything but primary.jpg — 5.3 GB measured on a
real cache, 4.1 GB of it unreachable. OrphanedArtistImagesJob is why
that survived: it joined the bare MBID onto the images directory, but
artist directories are sharded under a two-character prefix, so it
named a path that never existed and deleted the rows that were the only
record of the files it left behind.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
The shared 1 req/sec MB rate limiter was serializing the 3 concurrent
search calls in Phase 1 to ~3s minimum. Interactive search needs short
bursts (3 calls at once) but not sustained throughput.
Split into two MB rate limiters:
- mbSearchLimiter: burst=3, refill=1/sec — allows one search's 3
concurrent calls to fire immediately, then rate-limits sustained use
- mbBackgroundLimiter: strict 1/sec — gates artist image resolution
in the indexer to avoid 429s during sustained background work
Added NewRateLimiterBurst(n, b) constructor for configurable burst.
Expected Phase 1 improvement: ~3.5s → ~1s (3 calls fire in parallel
instead of serializing through the limiter).
Three changes to reduce subsequent index build times:
1. Positive image/rels cache TTL: 30 days → 365 days
Artist url-rels and resolved image URLs rarely change.
Already-indexed artists make zero API calls on rebuild.
2. Negative cache (misses) stays at 30 days so new images
are discovered within a month of being added upstream.
3. MB rate limiter for background indexing: 1.0 → 1.5 req/s
url-rels lookups are lightweight; 1.5/s is well within
what MB handles (Picard and Kodi both use similar rates).
Cuts the MB-bound portion of index build by ~33%.
Also adds NewRateLimiterF for fractional rates and caches
MB rels fetch failures (30-day miss TTL) to avoid retrying
unreachable artists every build.
New SearchIndex struct in searchindex.go:
- Background build fetches top 1000 LB artists, then their top 10
release groups and top 10 recordings (2001 API calls total)
- Dedicated 3 req/s rate limiter for indexer (LB allows 30/10s)
- Bounded concurrency (3 goroutines) with progress logging
- Batch INSERTs in transactions of 100 rows
- FTS5 query with prefix matching ('for you' → 'for* you*')
- Results sorted by popularity descending
- Skips rebuild if index is < 7 days old
- Marks index ready from existing rows if build fails
- Context cancellation for clean shutdown
yonluandClaude Opus 5