perf: unified per-artist indexing — discography + image in parallel
Restructure indexOneArtist to run LB discography fetches and MB artist image resolution concurrently. They use different rate limiters (LB: 3 req/s, MB: 1 req/s) so they overlap without contention. Per artist, the indexer now runs two parallel pipelines: LB pipeline: top-release-groups + top-recordings MB pipeline: url-rels → Wikidata P18 → Wikimedia image fetch All artist images are pre-cached during the index build instead of being resolved on-demand during search. Total build time drops from ~105 min (sequential) to ~63 min (parallel, MB-bound). SearchIndex now takes ArtistImageProvider as a dependency. The Service constructor creates artistImg before the index so both can share it.
This commit is contained in:
@@ -36,11 +36,11 @@ func NewExploreService(logger *slog.Logger, db *database.DB) *Service {
|
||||
limiter := NewRateLimiter()
|
||||
mb := NewMusicBrainzClient(cache, logger.WithGroup("musicbrainz"))
|
||||
lb := NewListenBrainzClient(limiter, cache, logger.WithGroup("listenbrainz"))
|
||||
index := NewSearchIndex(db, lb, logger.WithGroup("search-index"))
|
||||
artProxy := NewCoverArtProxy(db, limiter)
|
||||
artistImg := NewArtistImageProvider(
|
||||
db, cache, NewRateLimiter(), logger.WithGroup("artist-image"),
|
||||
)
|
||||
index := NewSearchIndex(db, lb, artistImg, logger.WithGroup("search-index"))
|
||||
|
||||
logger.Info("explore service created")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user