yonluandClaude Opus 5
20fbf28f2a
The post-scan backfills share MusicBrainz's rate limiters with every page the user can open, and both were FIFO — so a thousand-artist enrichment put an album page behind an hour of queued work. WithBackgroundLane/WithBackgroundPriority add a slower second lane: a marked wait takes no token while any interactive wait is outstanding. It is a context marker rather than a parameter because a backfill calls the same client methods a detail page does. A long backfill also has to be visible and stoppable, so jobs.KindCatalogEnrich registers both with progress and cancel — after the work is counted, since these passes are a no-op on every launch once the library is covered. What it does not fetch is the point. It ran for hours against a 900-artist library and marked nothing, because three of the four things it did per artist were work nobody asked for: similar artists, which the artist page already resolves on view, and a full GetArtistImage (fanart.tv, TheAudioDB, Wikidata, Wikipedia, ten portraits) reached only to warm the MB artist lookup EnsureArtistRels does alone. It was also serial across artists while every limiter is per-host and idle. The marks are a table rather than more explore_index columns, because artifactimport merges by column list and a flag added there is a second place to remember. BrowseReleaseGroupsAll pages to exhaustion, where the old call silently cut a prolific artist at 100 release groups. One portrait is downloaded now; the rest are remembered as URLs. resolveAllSources downloaded every candidate, up to ten, full size, while nothing reads anything but primary.jpg — 5.3 GB measured on a real cache, 4.1 GB of it unreachable. OrphanedArtistImagesJob is why that survived: it joined the bare MBID onto the images directory, but artist directories are sharded under a two-character prefix, so it named a path that never existed and deleted the rows that were the only record of the files it left behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
113 lines
3.0 KiB
Go
113 lines
3.0 KiB
Go
package explore
|
|
|
|
import (
|
|
"context"
|
|
|
|
"yellowjacket/backend/jobs"
|
|
)
|
|
|
|
// discogBackfillJobID is the stable registry ID for the owned-artist
|
|
// discography backfill. Only one runs at a time (the launch and
|
|
// post-scan triggers both go through the same bounded, resumable call),
|
|
// so the ID is a constant and a re-run reuses the handle and its log.
|
|
const discogBackfillJobID = "explore:discography-backfill"
|
|
|
|
// rgMBIDBackfillJobID is the same for the release-group MBID
|
|
// resolution pass.
|
|
const rgMBIDBackfillJobID = "explore:release-group-mbid-backfill"
|
|
|
|
// lyricsBackfillJobID is the same for the LRCLIB lyrics pass.
|
|
const lyricsBackfillJobID = "explore:lyrics-backfill"
|
|
|
|
// backfillJob is the registry side of a background catalog backfill:
|
|
// a handle, and the cancel func the registry's Cancel control trips.
|
|
//
|
|
// Every method is nil-safe, because a nil *backfillJob is the ordinary
|
|
// state in tests and in any build with no registry wired — the backfill
|
|
// itself must not care whether anyone is watching.
|
|
type backfillJob struct {
|
|
h *jobs.Handle
|
|
cancel context.CancelFunc
|
|
}
|
|
|
|
// startBackfillJob registers a cancellable job and returns it alongside
|
|
// a context that the job's Cancel control cancels. A nil registry (or
|
|
// a zero total) yields a nil job and the original context, so callers
|
|
// need no branch of their own.
|
|
//
|
|
// It is deliberately called *after* the work has been counted: a run
|
|
// with nothing to do must not put a job in the indicator, and this
|
|
// backfill is a no-op on every launch once the library is covered.
|
|
func startBackfillJob(
|
|
ctx context.Context,
|
|
reg *jobs.Registry,
|
|
id, title, subtitle string,
|
|
total int,
|
|
) (*backfillJob, context.Context) {
|
|
if reg == nil || total <= 0 {
|
|
return nil, ctx
|
|
}
|
|
|
|
jobCtx, cancel := context.WithCancel(ctx)
|
|
|
|
b := &backfillJob{cancel: cancel}
|
|
b.h = reg.Start(jobs.Spec{
|
|
ID: id,
|
|
Kind: jobs.KindCatalogEnrich,
|
|
Title: title,
|
|
Subtitle: subtitle,
|
|
Total: int64(total),
|
|
State: jobs.StateRunning,
|
|
Caps: jobs.Caps{
|
|
// Not pausable: the run is bounded and resumable by
|
|
// construction — each artist is marked as it completes, so
|
|
// cancelling and re-running is exactly what a pause would
|
|
// achieve, without a second checkpoint to keep honest.
|
|
Cancellable: true,
|
|
},
|
|
Controls: jobs.Controls{Cancel: cancel},
|
|
})
|
|
|
|
return b, jobCtx
|
|
}
|
|
|
|
// progress reports how far the run has got.
|
|
func (b *backfillJob) progress(current, total int) {
|
|
if b == nil || b.h == nil {
|
|
return
|
|
}
|
|
|
|
b.h.SetProgress(int64(current), int64(total))
|
|
}
|
|
|
|
// logf appends a line to the job's log.
|
|
func (b *backfillJob) logf(level jobs.Level, message string) {
|
|
if b == nil || b.h == nil {
|
|
return
|
|
}
|
|
|
|
b.h.Logf(level, message)
|
|
}
|
|
|
|
// finish closes the job out, reporting cancellation when that is what
|
|
// stopped it, and always releases the context.
|
|
func (b *backfillJob) finish(ctx context.Context) {
|
|
if b == nil {
|
|
return
|
|
}
|
|
|
|
defer b.cancel()
|
|
|
|
if b.h == nil {
|
|
return
|
|
}
|
|
|
|
if ctx.Err() != nil {
|
|
b.h.Cancelled()
|
|
|
|
return
|
|
}
|
|
|
|
b.h.Complete()
|
|
}
|