perf(explore): make the owned-artist backfill yield, mark, and stop
The post-scan backfills share MusicBrainz's rate limiters with every page the user can open, and both were FIFO — so a thousand-artist enrichment put an album page behind an hour of queued work. WithBackgroundLane/WithBackgroundPriority add a slower second lane: a marked wait takes no token while any interactive wait is outstanding. It is a context marker rather than a parameter because a backfill calls the same client methods a detail page does. A long backfill also has to be visible and stoppable, so jobs.KindCatalogEnrich registers both with progress and cancel — after the work is counted, since these passes are a no-op on every launch once the library is covered. What it does not fetch is the point. It ran for hours against a 900-artist library and marked nothing, because three of the four things it did per artist were work nobody asked for: similar artists, which the artist page already resolves on view, and a full GetArtistImage (fanart.tv, TheAudioDB, Wikidata, Wikipedia, ten portraits) reached only to warm the MB artist lookup EnsureArtistRels does alone. It was also serial across artists while every limiter is per-host and idle. The marks are a table rather than more explore_index columns, because artifactimport merges by column list and a flag added there is a second place to remember. BrowseReleaseGroupsAll pages to exhaustion, where the old call silently cut a prolific artist at 100 release groups. One portrait is downloaded now; the rest are remembered as URLs. resolveAllSources downloaded every candidate, up to ten, full size, while nothing reads anything but primary.jpg — 5.3 GB measured on a real cache, 4.1 GB of it unreachable. OrphanedArtistImagesJob is why that survived: it joined the bare MBID onto the images directory, but artist directories are sharded under a two-character prefix, so it named a path that never existed and deleted the rows that were the only record of the files it left behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
This commit is contained in:
+17
-1
@@ -182,6 +182,16 @@ func NewYellowJacketApp(
|
||||
yjApp.library.SetJobRegistry(yjApp.jobs)
|
||||
yjApp.explore.SetJobRegistry(yjApp.jobs)
|
||||
|
||||
// Let the release prefetch skip albums the user already owns in
|
||||
// full — those open with no catalog call at all, so warming their
|
||||
// tracklists spends the most expensive request in the app on
|
||||
// nothing. Injected because neither package imports the other.
|
||||
yjApp.explore.SetAlbumComplete(func(albumID int64) bool {
|
||||
c, err := yjApp.library.GetAlbumCompleteness(albumID)
|
||||
|
||||
return err == nil && c.Known && c.Complete
|
||||
})
|
||||
|
||||
// create autotag service (depends on explore + tagWriter)
|
||||
yjApp.autotag = autotagservice.NewService(
|
||||
yjApp.logger.WithGroup("autotag"),
|
||||
@@ -697,7 +707,13 @@ func (yj *YellowJacketApp) startJanitor() {
|
||||
yj.database, coversDir, library.CoverArtFileSet,
|
||||
))
|
||||
yj.janitor.Register(maintenance.OrphanedArtistImagesJob(
|
||||
yj.database, filepath.Join(dataDir, explore.ArtistImageDirName),
|
||||
yj.database,
|
||||
filepath.Join(dataDir, explore.ArtistImageDirName),
|
||||
explore.ArtistImageDir,
|
||||
))
|
||||
yj.janitor.Register(maintenance.StrayArtistImageFilesJob(
|
||||
filepath.Join(dataDir, explore.ArtistImageDirName),
|
||||
explore.ArtistImageKeepNames(),
|
||||
))
|
||||
yj.janitor.Register(maintenance.ExpiredProxyCacheJob(
|
||||
filepath.Join(dataDir, explore.CoverArtCacheDirName),
|
||||
|
||||
Reference in New Issue
Block a user