perf(explore): make the owned-artist backfill yield, mark, and stop

The post-scan backfills share MusicBrainz's rate limiters with every
page the user can open, and both were FIFO — so a thousand-artist
enrichment put an album page behind an hour of queued work.
WithBackgroundLane/WithBackgroundPriority add a slower second lane: a
marked wait takes no token while any interactive wait is outstanding.
It is a context marker rather than a parameter because a backfill calls
the same client methods a detail page does. A long backfill also has to
be visible and stoppable, so jobs.KindCatalogEnrich registers both with
progress and cancel — after the work is counted, since these passes are
a no-op on every launch once the library is covered.

What it does not fetch is the point. It ran for hours against a
900-artist library and marked nothing, because three of the four things
it did per artist were work nobody asked for: similar artists, which
the artist page already resolves on view, and a full GetArtistImage
(fanart.tv, TheAudioDB, Wikidata, Wikipedia, ten portraits) reached
only to warm the MB artist lookup EnsureArtistRels does alone. It was
also serial across artists while every limiter is per-host and idle.

The marks are a table rather than more explore_index columns, because
artifactimport merges by column list and a flag added there is a second
place to remember. BrowseReleaseGroupsAll pages to exhaustion, where
the old call silently cut a prolific artist at 100 release groups.

One portrait is downloaded now; the rest are remembered as URLs.
resolveAllSources downloaded every candidate, up to ten, full size,
while nothing reads anything but primary.jpg — 5.3 GB measured on a
real cache, 4.1 GB of it unreachable. OrphanedArtistImagesJob is why
that survived: it joined the bare MBID onto the images directory, but
artist directories are sharded under a two-character prefix, so it
named a path that never existed and deleted the rows that were the only
record of the files it left behind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
This commit is contained in:
2026-08-14 13:33:54 -04:00
co-authored by Claude Opus 5
parent 878cf4b561
commit 20fbf28f2a
22 changed files with 2058 additions and 236 deletions
+77
View File
@@ -328,6 +328,83 @@ func (c *MusicBrainzClient) BrowseReleaseGroups(
return out, nil
}
// browseMaxPages bounds BrowseReleaseGroupsAll. At MaxLimit per page
// that is 1 000 release groups, which no real artist reaches — it is a
// runaway guard for a server that stops honouring the offset, not a
// coverage decision.
const browseMaxPages = 10
// BrowseReleaseGroupsAll is BrowseReleaseGroups paged to exhaustion.
//
// The single-page call above asks for MaxLimit (100) and takes whatever
// comes back, which silently truncates a prolific artist's discography
// at 100 release groups — invisible unless you count, since a hundred
// albums looks like a complete page. This is the call to use when the
// answer is meant to be the whole discography rather than a page of it.
//
// The result is cached under the same key the single-page call reads,
// so a later interactive browse is served the complete list.
func (c *MusicBrainzClient) BrowseReleaseGroupsAll(
ctx context.Context, artistMBID string,
) ([]MBReleaseGroup, error) {
cacheKey := "mb:browse:release-groups:" + artistMBID
if data, ok := c.cache.Get(cacheKey); ok {
var out []MBReleaseGroup
if err := json.Unmarshal(data, &out); err == nil {
return out, nil
}
}
var out []MBReleaseGroup
for page := range browseMaxPages {
if err := c.limiter.Wait(ctx); err != nil {
return nil, err
}
offset := page * musicbrainzws2.MaxLimit
c.logger.Info("musicbrainz browse release groups",
"artistMBID", artistMBID,
"offset", offset,
)
result, err := c.mb.BrowseReleaseGroups(ctx,
musicbrainzws2.ReleaseGroupFilter{
ArtistMBID: mbtypes.MBID(artistMBID),
},
musicbrainzws2.Paginator{
Limit: musicbrainzws2.MaxLimit,
Offset: offset,
},
)
if err != nil {
// Pages already fetched are still worth keeping if there are
// any: a partial discography beats none, and the caller's
// mark is only set on a nil error, so the rest is retried.
if len(out) > 0 {
return out, nil
}
return nil, err
}
out = append(out, convertReleaseGroups(result.ReleaseGroups)...)
// A short page is the last page. MB reports the full count too,
// but a short page is the condition that terminates correctly
// even when the count and the pages disagree.
if len(result.ReleaseGroups) < musicbrainzws2.MaxLimit {
break
}
}
c.cacheJSON(cacheKey, out, cacheTTLEntity, artistMBID, "artist")
return out, nil
}
// LookupRelease fetches a single release by MBID (with media +
// recordings). Used by the autotag paste-URL escape hatch.
// Cached for 7 days.