yonlu/yellowjacket
1
1
Fork 0
Code Issues 55 Pull Requests 6 Actions Packages Projects Releases 12 Wiki Activity

Explore: warm the release cache for owned albums that are not known-complete #85

New Issue
Open
opened 2026-08-18 20:05:17 +00:00 by yonlu · 0 comments
yonlu commented 2026-08-18 20:05:17 +00:00
Owner
Copy Link
Copy Source

Opening an album detail page for an album you already own hits MusicBrainz, every time it is not in the response cache — which for most of a library is every time, because nothing warms that cache except a capped prefetch on the artist page.

Converted from .planning/plans/pending/010-owned-album-catalog-offline.md, which is deleted in favour of this issue.

What already shipped, and what it leaves

The common case is solved. GetAlbumCompleteness reads the "5/12" denominator off the files' own tags, and an album that is MBID-matched and complete now opens with no catalog call at all — identity from the MBID, tracklist from the tags, which were the two things the browse was being spent on.

So the set left to serve is not "albums you own a track of". It is:

  • albums that are genuinely incomplete — the catalog is the only way to say which tracks are missing, since tags give the count and not the names; and
  • albums whose tags never declared a total, where completeness is unknowable locally.

On a well-tagged library that is a small minority, which changes the economics considerably. Re-measure before building — the answer may now be "the prefetch is enough".

Why the discography backfill does not already cover it

BackfillLibraryDiscographies / EnsureArtistDiscography write flat explore_index rows. There is no release group → tracklist relation anywhere in the index and no release-level rows at all; caa_release_mbid and release_name name the release used for cover art, not a tracklist. "We have full discographies for library artists" means we know which albums the artist made, offline. It has never meant we know what is on any of them.

The only store of release-level catalog data is http_cache under mb:browse:releases:<rg> (90-day TTL), populated only by a live BrowseReleases with Includes: ["recordings", "media"] — the most expensive call the app makes — and warmed by exactly one thing, PrefetchReleases, capped at 8 and called only when an artist page renders.

What to build

A post-scan backfill that warms the release cache for release groups that are owned but not known-complete — bounded, resumable, shaped like BackfillLibraryDiscographies, which is the proven pattern here.

  1. A query for release groups with at least one owned track and no warm release-cache entry. Ownership is asked of audio_files, like every other ownership question in this codebase.
  2. Order by owned-track count descending, so the albums the user has most of are warmed first — same reasoning and same benefit if a run is cut short.
  3. Run through releasesSF, so it never double-fetches a release group an interactive open is already handling.
  4. Bound a run (discogBackfillMaxPerRun has a value to copy). The resume marker is the response cache itself — BrowseReleasesCached already answers "is this one done", so unlike the discography path this needs no new flag column.
  5. Register it with jobs so it has progress, pause and cancel like every other long-running operation, and mark its context with WithBackgroundPriority.

The rate-limiter half of the original plan is already built: RateLimiter.WithBackgroundLane + WithBackgroundPriority(ctx) make a marked caller yield to interactive work, and jobs.KindCatalogEnrich + startBackfillJob give the existing backfills progress and cancel. PrefetchReleases' cap of 8 is still unrevisited and should be looked at in the same change.

The alternative that was considered and rejected

Projecting release-group tracklists in the dump build and shipping them in the artifact. The data is there and it is derivable from bytes the index build already streams. It is rejected because the artifact is built centrally and is byte-identical for every user, so "albums the user owns a track of" cannot be a filter on it.

The argument that actually kills it: the popularity floor is not one number over artists but a per-artist track budget (50 tracks tier A, 25 tier B, 12 tier C). A projected tracklist would be whichever of an album's tracks survived that budget, with nothing marking the rest absent — so the album page would count owned against a truncated denominator and render "Play 7 of 9" for a twelve-track album. That is a confident lie, where the states this approach produces (complete / incomplete / unknown) are at worst silent.

Worth revisiting only if the artifact ever gains per-user tailoring.

Done when

  • Opening an owned album that has never been opened before renders its catalog tracklist with no network call, after one backfill run.
  • An interactive browse issued while the backfill is running is not delayed by it.
  • The backfill appears in the jobs indicator and can be paused and cancelled there.
  • A second run after a completed one does approximately nothing.
Opening an album detail page for an album **you already own** hits MusicBrainz, every time it is not in the response cache — which for most of a library is every time, because nothing warms that cache except a capped prefetch on the artist page. Converted from `.planning/plans/pending/010-owned-album-catalog-offline.md`, which is deleted in favour of this issue. ## What already shipped, and what it leaves The common case is solved. `GetAlbumCompleteness` reads the "5/12" denominator off the files' own tags, and an album that is **MBID-matched and complete** now opens with **no catalog call at all** — identity from the MBID, tracklist from the tags, which were the two things the browse was being spent on. So the set left to serve is not "albums you own a track of". It is: - albums that are genuinely **incomplete** — the catalog is the only way to say *which* tracks are missing, since tags give the count and not the names; and - albums whose tags **never declared a total**, where completeness is unknowable locally. On a well-tagged library that is a small minority, which changes the economics considerably. **Re-measure before building — the answer may now be "the prefetch is enough".** ## Why the discography backfill does not already cover it `BackfillLibraryDiscographies` / `EnsureArtistDiscography` write flat `explore_index` rows. There is no release group → tracklist relation anywhere in the index and no release-level rows at all; `caa_release_mbid` and `release_name` name the release used for cover art, not a tracklist. "We have full discographies for library artists" means we know *which albums the artist made*, offline. It has never meant we know what is on any of them. The only store of release-level catalog data is `http_cache` under `mb:browse:releases:<rg>` (90-day TTL), populated only by a live `BrowseReleases` with `Includes: ["recordings", "media"]` — the most expensive call the app makes — and warmed by exactly one thing, `PrefetchReleases`, capped at 8 and called only when an artist page renders. ## What to build A post-scan backfill that warms the release cache for release groups that are **owned but not known-complete** — bounded, resumable, shaped like `BackfillLibraryDiscographies`, which is the proven pattern here. 1. A query for release groups with at least one owned track and no warm release-cache entry. Ownership is asked of `audio_files`, like every other ownership question in this codebase. 2. Order by owned-track count descending, so the albums the user has most of are warmed first — same reasoning and same benefit if a run is cut short. 3. Run through `releasesSF`, so it never double-fetches a release group an interactive open is already handling. 4. Bound a run (`discogBackfillMaxPerRun` has a value to copy). The resume marker is the response cache itself — `BrowseReleasesCached` already answers "is this one done", so unlike the discography path this needs **no new flag column**. 5. Register it with `jobs` so it has progress, pause and cancel like every other long-running operation, and mark its context with `WithBackgroundPriority`. The rate-limiter half of the original plan **is already built**: `RateLimiter.WithBackgroundLane` + `WithBackgroundPriority(ctx)` make a marked caller yield to interactive work, and `jobs.KindCatalogEnrich` + `startBackfillJob` give the existing backfills progress and cancel. `PrefetchReleases`' cap of 8 is still unrevisited and should be looked at in the same change. ## The alternative that was considered and rejected **Projecting release-group tracklists in the dump build and shipping them in the artifact.** The data is there and it is derivable from bytes the index build already streams. It is rejected because the artifact is built centrally and is byte-identical for every user, so "albums the user owns a track of" cannot be a filter on it. The argument that actually kills it: the popularity floor is not one number over artists but a **per-artist track budget** (50 tracks tier A, 25 tier B, 12 tier C). A projected tracklist would be whichever of an album's tracks survived that budget, with nothing marking the rest absent — so the album page would count owned against a truncated denominator and render "Play 7 of 9" for a twelve-track album. That is a confident lie, where the states this approach produces (complete / incomplete / unknown) are at worst silent. Worth revisiting only if the artifact ever gains per-user tailoring. ## Done when - Opening an owned album that has never been opened before renders its catalog tracklist with no network call, after one backfill run. - An interactive browse issued while the backfill is running is not delayed by it. - The backfill appears in the jobs indicator and can be paused and cancelled there. - A second run after a completed one does approximately nothing.
yonlu self-assigned this 2026-08-18 20:08:34 +00:00
yonlu added the
Status
In Progress
 label 2026-08-18 20:08:35 +00:00
yonlu removed their assignment 2026-08-18 20:08:35 +00:00
yonlu removed the
Status
In Progress
 label 2026-08-18 20:08:36 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Area/Design
Area/Downloads
Area/Explore
Area/Library-UI
Area/Metadata
Area/Packaging
Area/Player
Area/Queue
Area/Settings
Area/Shell-Nav
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
 Platform/Android
Platform/Desktop
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Medium
3
The priority is medium
Priority
Low
4
The priority is low
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
In Progress
Somebody is actively working on this right now
No labels Area/Explore Kind/Enhancement Platform/Android Platform/Desktop
Priority
Medium
3
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan yonlu
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: yonlu/yellowjacket#85