yonluandClaude Opus 5
20fbf28f2a
The post-scan backfills share MusicBrainz's rate limiters with every page the user can open, and both were FIFO — so a thousand-artist enrichment put an album page behind an hour of queued work. WithBackgroundLane/WithBackgroundPriority add a slower second lane: a marked wait takes no token while any interactive wait is outstanding. It is a context marker rather than a parameter because a backfill calls the same client methods a detail page does. A long backfill also has to be visible and stoppable, so jobs.KindCatalogEnrich registers both with progress and cancel — after the work is counted, since these passes are a no-op on every launch once the library is covered. What it does not fetch is the point. It ran for hours against a 900-artist library and marked nothing, because three of the four things it did per artist were work nobody asked for: similar artists, which the artist page already resolves on view, and a full GetArtistImage (fanart.tv, TheAudioDB, Wikidata, Wikipedia, ten portraits) reached only to warm the MB artist lookup EnsureArtistRels does alone. It was also serial across artists while every limiter is per-host and idle. The marks are a table rather than more explore_index columns, because artifactimport merges by column list and a flag added there is a second place to remember. BrowseReleaseGroupsAll pages to exhaustion, where the old call silently cut a prolific artist at 100 release groups. One portrait is downloaded now; the rest are remembered as URLs. resolveAllSources downloaded every candidate, up to ten, full size, while nothing reads anything but primary.jpg — 5.3 GB measured on a real cache, 4.1 GB of it unreachable. OrphanedArtistImagesJob is why that survived: it joined the bare MBID onto the images directory, but artist directories are sharded under a two-character prefix, so it named a path that never existed and deleted the rows that were the only record of the files it left behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDCbcCZQepnpSQYJ6SxxZm
54 lines
1.9 KiB
Go
54 lines
1.9 KiB
Go
package explore
|
|
|
|
import "path/filepath"
|
|
|
|
// Asset directory names under the user data directory.
|
|
//
|
|
// These are exported so the maintenance janitor can sweep them without
|
|
// importing the internals of the providers that write them. Each is
|
|
// catalogued in backend/datamap as cache data: rebuildable from the
|
|
// network, but expensive enough that eviction is by age rather than
|
|
// tied to the lifetime of anything else.
|
|
const (
|
|
// ArtistImageDirName holds one subdirectory per artist MBID, each
|
|
// containing fetched photos, a primary.jpg with its thumbnails, and
|
|
// possibly a .miss marker recording that no artwork was found.
|
|
ArtistImageDirName = "artist-images"
|
|
|
|
// CoverArtCacheDirName holds Cover Art Archive thumbnails fetched
|
|
// while browsing Explore, named by release-group MBID. Nothing in
|
|
// the database references these files.
|
|
CoverArtCacheDirName = "cover-art-cache"
|
|
)
|
|
|
|
// ArtistImageDir returns the directory holding one artist's images.
|
|
//
|
|
// Artist directories are sharded under the MBID's first two characters,
|
|
// which the janitor cannot be expected to know and had got wrong — so
|
|
// this is the one definition of that layout, and ArtistImageProvider's
|
|
// own artistDir defers to it.
|
|
func ArtistImageDir(baseDir, mbid string) string {
|
|
if len(mbid) < 2 {
|
|
return filepath.Join(baseDir, "xx", mbid)
|
|
}
|
|
|
|
return filepath.Join(baseDir, mbid[:2], mbid)
|
|
}
|
|
|
|
// ArtistImageKeepNames is every file an artist's directory is meant to
|
|
// hold: the portrait, its three size tiers, and the marker recording
|
|
// that no portrait was found. Anything else is a downloaded candidate
|
|
// from a version that kept all of them (see StrayArtistImageFilesJob).
|
|
func ArtistImageKeepNames() map[string]bool {
|
|
keep := map[string]bool{
|
|
"primary.jpg": true,
|
|
artistMissFile: true,
|
|
}
|
|
|
|
for _, tier := range artistImageTiers {
|
|
keep["primary"+tier.Suffix+".jpg"] = true
|
|
}
|
|
|
|
return keep
|
|
}
|