perf(explore): make the owned-artist backfill yield, mark, and stop
The post-scan backfills share MusicBrainz's rate limiters with every page the user can open, and both were FIFO — so a thousand-artist enrichment put an album page behind an hour of queued work. WithBackgroundLane/WithBackgroundPriority add a slower second lane: a marked wait takes no token while any interactive wait is outstanding. It is a context marker rather than a parameter because a backfill calls the same client methods a detail page does. A long backfill also has to be visible and stoppable, so jobs.KindCatalogEnrich registers both with progress and cancel — after the work is counted, since these passes are a no-op on every launch once the library is covered. What it does not fetch is the point. It ran for hours against a 900-artist library and marked nothing, because three of the four things it did per artist were work nobody asked for: similar artists, which the artist page already resolves on view, and a full GetArtistImage (fanart.tv, TheAudioDB, Wikidata, Wikipedia, ten portraits) reached only to warm the MB artist lookup EnsureArtistRels does alone. It was also serial across artists while every limiter is per-host and idle. The marks are a table rather than more explore_index columns, because artifactimport merges by column list and a flag added there is a second place to remember. BrowseReleaseGroupsAll pages to exhaustion, where the old call silently cut a prolific artist at 100 release groups. One portrait is downloaded now; the rest are remembered as URLs. resolveAllSources downloaded every candidate, up to ten, full size, while nothing reads anything but primary.jpg — 5.3 GB measured on a real cache, 4.1 GB of it unreachable. OrphanedArtistImagesJob is why that survived: it joined the bare MBID onto the images directory, but artist directories are sharded under a two-character prefix, so it named a path that never existed and deleted the rows that were the only record of the files it left behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
This commit is contained in:
@@ -328,6 +328,83 @@ func (c *MusicBrainzClient) BrowseReleaseGroups(
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// browseMaxPages bounds BrowseReleaseGroupsAll. At MaxLimit per page
|
||||
// that is 1 000 release groups, which no real artist reaches — it is a
|
||||
// runaway guard for a server that stops honouring the offset, not a
|
||||
// coverage decision.
|
||||
const browseMaxPages = 10
|
||||
|
||||
// BrowseReleaseGroupsAll is BrowseReleaseGroups paged to exhaustion.
|
||||
//
|
||||
// The single-page call above asks for MaxLimit (100) and takes whatever
|
||||
// comes back, which silently truncates a prolific artist's discography
|
||||
// at 100 release groups — invisible unless you count, since a hundred
|
||||
// albums looks like a complete page. This is the call to use when the
|
||||
// answer is meant to be the whole discography rather than a page of it.
|
||||
//
|
||||
// The result is cached under the same key the single-page call reads,
|
||||
// so a later interactive browse is served the complete list.
|
||||
func (c *MusicBrainzClient) BrowseReleaseGroupsAll(
|
||||
ctx context.Context, artistMBID string,
|
||||
) ([]MBReleaseGroup, error) {
|
||||
cacheKey := "mb:browse:release-groups:" + artistMBID
|
||||
|
||||
if data, ok := c.cache.Get(cacheKey); ok {
|
||||
var out []MBReleaseGroup
|
||||
if err := json.Unmarshal(data, &out); err == nil {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
|
||||
var out []MBReleaseGroup
|
||||
|
||||
for page := range browseMaxPages {
|
||||
if err := c.limiter.Wait(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
offset := page * musicbrainzws2.MaxLimit
|
||||
|
||||
c.logger.Info("musicbrainz browse release groups",
|
||||
"artistMBID", artistMBID,
|
||||
"offset", offset,
|
||||
)
|
||||
|
||||
result, err := c.mb.BrowseReleaseGroups(ctx,
|
||||
musicbrainzws2.ReleaseGroupFilter{
|
||||
ArtistMBID: mbtypes.MBID(artistMBID),
|
||||
},
|
||||
musicbrainzws2.Paginator{
|
||||
Limit: musicbrainzws2.MaxLimit,
|
||||
Offset: offset,
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
// Pages already fetched are still worth keeping if there are
|
||||
// any: a partial discography beats none, and the caller's
|
||||
// mark is only set on a nil error, so the rest is retried.
|
||||
if len(out) > 0 {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
out = append(out, convertReleaseGroups(result.ReleaseGroups)...)
|
||||
|
||||
// A short page is the last page. MB reports the full count too,
|
||||
// but a short page is the condition that terminates correctly
|
||||
// even when the count and the pages disagree.
|
||||
if len(result.ReleaseGroups) < musicbrainzws2.MaxLimit {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
c.cacheJSON(cacheKey, out, cacheTTLEntity, artistMBID, "artist")
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// LookupRelease fetches a single release by MBID (with media +
|
||||
// recordings). Used by the autotag paste-URL escape hatch.
|
||||
// Cached for 7 days.
|
||||
|
||||
Reference in New Issue
Block a user