feat(database): shape the library like files, and shrink the catalog
CI / check (push) Successful in 3m7s
CI / e2e (push) Canceled after 1m45s

Plans 013 and 014, the album page that prompted them, and the smaller
fixes they turned up. Changelog, largest first.

## The local library is shaped like files, not like MusicBrainz

`audio_files` carries its own tags and points at `albums` and
`artists`; `file_genres` is the one real many-to-many. `recordings`,
`release_group_recordings`, `artist_credit`, `artist_credit_artist`,
`recording_genres`, `release_groups` and `release_to_rg` are gone from
the local side, and with them a six-way join in every read, a
`MIN(release_group_id)` subquery in eleven queries and a
first-credited-artist subquery in nine. Measured on a real 25,966-file
library, every many-to-many that model expressed was 1:1 in the data.

- Ownership is a file. `GetFilePathsByRecordingMBIDs`,
  `LibraryMBIDIndex.CheckMBIDs`, `collectLibraryEntities` and
  `pruneStaleLocalCrossReferences` all join `audio_files`, so the 812
  orphaned recordings, 216 release groups and 260 artists that library
  carried are now structurally impossible.
- One projection: every track query selects from the `track_metadata`
  view, one row type, one mapper. Nine hand-rolled copies had drifted
  far enough to report different years on different screens.
- `library_id = 0` means every library, so each list query exists once
  instead of scoped and unscoped with a branch at every call site.
- No migration chain. `sql/schemas/` is the one description of the
  shape; `sql/migrations/`, `applyMigrations` and `schema_migrations`
  are squashed away, along with the drift between them that had sqlc
  generating against a stale schema.
- `database.InsertTestTrack` is the one test seeder; twenty test files
  had been assembling the old FK chain each in its own order.

## The catalog stores its ids as bytes

`explore_index`'s three 36-char MBID columns and its entity-type text
are 16 raw bytes and a small integer. The table and its six indexes go
780 MB to 405 MB on a real 2,052,200-row catalog, which is why a fresh
install is ~0.6 GB rather than ~1.0 GB.

- `backend/explore/mbid.go` is the only place the encoding is known;
  everything above it speaks dashed strings.
- `CHECK(length(mbid) = 16)` makes a stringly write fail at the insert
  rather than silently returning no rows, since SQLite does not coerce
  between TEXT and BLOB.
- The importer asks the artifact what encoding it carries and converts
  on the way in, so the artifact already published keeps working and no
  format bump is needed.
- `indexRowColumns`/`scanIndexRow` replace four copies of a 22-column
  list, and `TestStoredEncodingRoundTrips` sweeps every read path.

## An album page that says how much of the album is yours

- One question, asked once: is there a file. `filePaths` is filled by a
  single batched lookup when the tracklist settles, and the badge, the
  Play count, the dimmed rows and every menu item read it — replacing
  four claims of decreasing confidence that could show a green tick on
  an album whose every action did nothing.
- Play, Play 7 of 12, or no play button at all.
- `total_tracks` on `explore_index` (~2 bytes over 400,677 release
  groups) and on `audio_files` from tags that have always carried it:
  a complete MBID-matched album now makes no catalog call at all, where
  it used to spend the most expensive request the app makes.
- A merged cluster shows the running order the most releases agree on,
  and the version list marks the release you own rather than standing a
  synthetic entry in for it.
- `AlbumReleasesFailed`: a slow fetch is no longer reported as a failed
  one by a 12-second timer.
- Rows not in the library are dimmed in place (with `aria-disabled`)
  instead of the owned ones wearing a green tick and a legend.

## Caches and cover art get ceilings

- Only the three tiers of a cover are stored; the full-resolution copy
  nothing rendered was 1,134 MB of a 1.4 GB covers directory.
- One artist portrait is downloaded and the rest are remembered as
  URLs — 4.1 GB of a 5.3 GB cache was candidates no code path reads.
- `browsedArtBudget` and `httpCacheBudget` bound what an age cannot:
  the same install held art for 5,770 artists in a 1,301-artist
  library.
- `OrphanedArtistImagesJob` joined a bare MBID onto a sharded
  directory, so it deleted the rows that were the only record of the
  files it left behind. `explore.ArtistImageDir` is that layout's one
  definition now.

## The autotag queue asks whether there is work

`tagging_items` was a row per album folder, not a queue, and no query
read the `tag_status` column that held the answer. The four queue
queries ask the files, which matters most where it is least visible:
`startPrefetch` was scoring every album in a tagged library against
MusicBrainz.

## Phantom playlist tracks resolve in place

An M3U8 imported before its files leaves phantom rows; they now match
by path and fall back to position, keep their place in the playlist
when resolved, and pair best-first so two phantoms cannot claim the
same file.

## Playing a track plays the list it is in

Double-click, and Play on a single row's menu, queue the list as
displayed with `startIndex` on that row — the album page and the track
list used to queue one track and discard the album around it. A
multi-row selection still plays exactly itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
This commit is contained in:
2026-08-16 13:58:15 -04:00
co-authored by Claude Opus 5
parent 1128881e8d
commit e7748f1fd5
208 changed files with 10944 additions and 12104 deletions
+89 -3
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"os"
"strconv"
"strings"
"time"
_ "modernc.org/sqlite" // SQLite driver for reading the artifact file.
@@ -91,9 +92,90 @@ type artifactInfo struct {
// TestArtifactColumnsMatchExporter.
const artifactCatalogColumns = `entity_type, mbid, title, artist_name, artist_mbid,
aliases, popularity, listener_count, duration, caa_release_mbid,
release_name, primary_type, secondary_types, release_date,
release_name, primary_type, secondary_types, release_date, total_tracks,
artist_type, country, disambiguation, sort_name, discog_fetched`
// artifactSelectColumns is the same list as read *from an artifact*,
// converting the two columns whose storage this app changed.
//
// The local table stores an MBID as 16 raw bytes and an entity type as a
// small integer, which took the catalog and its indexes from 677 MB to
// 389 MB. A published artifact still carries the text form, and there
// is no reason it should not: converting on the way in costs one
// `unhex` per row on a once-a-month import, and it means a new build
// reads the artifact that is already out there rather than requiring
// one to be rebuilt and re-downloaded first.
//
// An artifact that already carries the compact form is copied straight
// through - `artifactStoresText` decides which, by asking the artifact
// rather than by trusting a version number.
func artifactSelectColumns(text, totals bool) string {
totalTracks := "total_tracks"
if !totals {
// An artifact built before the column existed. Zero is what the
// column means by "the catalog does not say", so an older
// artifact imports as one that declines to answer rather than
// failing to import at all.
totalTracks = "0"
}
if !text {
return strings.Replace(
artifactCatalogColumns, "total_tracks", totalTracks, 1,
)
}
return `CASE entity_type
WHEN 'artist' THEN 1
WHEN 'release_group' THEN 2
WHEN 'recording' THEN 3
ELSE 0 END,
unhex(replace(mbid, '-', '')),
title, artist_name,
CASE WHEN artist_mbid = '' THEN x''
ELSE unhex(replace(artist_mbid, '-', '')) END,
aliases, popularity, listener_count, duration,
CASE WHEN caa_release_mbid = '' THEN x''
ELSE unhex(replace(caa_release_mbid, '-', '')) END,
release_name, primary_type, secondary_types, release_date, ` +
totalTracks + `,
artist_type, country, disambiguation, sort_name, discog_fetched`
}
// artifactHasTotals reports whether the attached artifact carries the
// per-release-group track denominator. An artifact published before
// that column existed is still a perfectly good catalog, so it is asked
// rather than assumed - the same rule, and the same handle, as
// artifactStoresText below.
func (si *SearchIndex) artifactHasTotals() bool {
var n int
err := si.db.QueryRowWriter(
`SELECT COUNT(*) FROM pragma_table_info('explore_index', 'core')
WHERE name = 'total_tracks'`,
).Scan(&n)
return err == nil && n > 0
}
// artifactStoresText reports whether the attached artifact carries the
// old text encoding.
func (si *SearchIndex) artifactStoresText() bool {
// The writer, not QueryContext: "core" is attached to that one
// connection and does not exist on the read pool. Asking the wrong
// handle errors, and the fallback would then convert an artifact
// that needs no conversion.
var kind string
if err := si.db.QueryRowWriter(
"SELECT typeof(mbid) FROM core.explore_index LIMIT 1",
).Scan(&kind); err != nil {
return true
}
return kind == "text"
}
// inspectArtifact opens the artifact read-only and reports what it
// declares, without touching the live index. Validation happens here so
// a bad download is rejected before anything is attached.
@@ -263,9 +345,13 @@ func (si *SearchIndex) analyzeIndex() {
// is an index range scan and a cancelled import leaves committed work
// behind rather than rolling it all back.
func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, error) {
selectColumns := artifactSelectColumns(
si.artifactStoresText(), si.artifactHasTotals(),
)
insertSQL := `
INSERT INTO explore_index (` + artifactCatalogColumns + `)
SELECT ` + artifactCatalogColumns + `
SELECT ` + selectColumns + `
FROM core.explore_index
WHERE mbid > ?` + upsertIndexConflictSQL
@@ -273,7 +359,7 @@ func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, e
// appended only while one exists.
insertRangeSQL := `
INSERT INTO explore_index (` + artifactCatalogColumns + `)
SELECT ` + artifactCatalogColumns + `
SELECT ` + selectColumns + `
FROM core.explore_index
WHERE mbid > ? AND mbid <= ?` + upsertIndexConflictSQL
+158 -3
View File
@@ -202,7 +202,7 @@ func TestImportCoreArtifactPreservesLocalData(t *testing.T) {
if err := db.QueryRowWriter(`
SELECT popularity, duration, in_library, discog_fetched
FROM explore_index WHERE mbid = ?`, recA,
FROM explore_index WHERE mbid = ?`, dbMBID(recA),
).Scan(&popularity, &duration, &inLibrary, &discogFetched); err != nil {
t.Fatalf("read merged row: %v", err)
}
@@ -376,7 +376,7 @@ func TestAddFromCacheNeverStoresMBIDAsName(t *testing.T) {
var artistName string
if err := db.QueryRowWriter(
`SELECT artist_name FROM explore_index WHERE mbid = ?`, rgA,
`SELECT artist_name FROM explore_index WHERE mbid = ?`, dbMBID(rgA),
).Scan(&artistName); err != nil {
t.Fatalf("read release group: %v", err)
}
@@ -391,7 +391,7 @@ func TestAddFromCacheNeverStoresMBIDAsName(t *testing.T) {
})
if err := db.QueryRowWriter(
`SELECT artist_name FROM explore_index WHERE mbid = ?`, rgA,
`SELECT artist_name FROM explore_index WHERE mbid = ?`, dbMBID(rgA),
).Scan(&artistName); err != nil {
t.Fatalf("re-read release group: %v", err)
}
@@ -442,3 +442,158 @@ func TestArtifactColumnsMatchExporter(t *testing.T) {
exporter, importer)
}
}
// TestImportCoreArtifactAcceptsBothEncodings is the compatibility half
// of the storage change.
//
// The catalog stores an MBID as 16 raw bytes and an entity type as a
// code, which took the table and its indexes from 677 MB to 389 MB. A
// published artifact carries whichever form the exporter that built it
// used, and there is one already out there in the older text form — so
// the importer decides by asking the artifact, not by trusting a
// version number, and both must land identically.
func TestImportCoreArtifactAcceptsBothEncodings(t *testing.T) {
compact := filepath.Join(t.TempDir(), "core-index.db")
db, err := sql.Open("sqlite", "file:"+compact)
if err != nil {
t.Fatalf("open artifact: %v", err)
}
if _, err := db.Exec(`CREATE TABLE explore_index (
entity_type INTEGER NOT NULL,
mbid BLOB NOT NULL,
title TEXT NOT NULL,
artist_name TEXT NOT NULL,
artist_mbid BLOB NOT NULL,
aliases TEXT NOT NULL DEFAULT '',
popularity INTEGER NOT NULL DEFAULT 0,
listener_count INTEGER NOT NULL DEFAULT 0,
duration INTEGER NOT NULL DEFAULT 0,
caa_release_mbid BLOB NOT NULL DEFAULT x'',
release_name TEXT NOT NULL DEFAULT '',
primary_type TEXT NOT NULL DEFAULT '',
secondary_types TEXT NOT NULL DEFAULT '',
release_date TEXT NOT NULL DEFAULT '',
artist_type TEXT NOT NULL DEFAULT '',
country TEXT NOT NULL DEFAULT '',
disambiguation TEXT NOT NULL DEFAULT '',
sort_name TEXT NOT NULL DEFAULT '',
discog_fetched INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (mbid)
)`); err != nil {
t.Fatalf("create artifact table: %v", err)
}
if _, err := db.Exec(
`CREATE TABLE artifact_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)`,
); err != nil {
t.Fatalf("create artifact meta: %v", err)
}
for k, v := range validMeta() {
if _, err := db.Exec(
"INSERT INTO artifact_meta (key, value) VALUES (?, ?)", k, v,
); err != nil {
t.Fatalf("write artifact meta: %v", err)
}
}
if _, err := db.Exec(`
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid, popularity)
VALUES (1, ?, 'Artist A', 'Artist A', ?, 5000)`,
mbidBytes(artA), mbidBytes(artA),
); err != nil {
t.Fatalf("write artifact row: %v", err)
}
_ = db.Close()
live := database.NewTestDB(t)
si := NewSearchIndex(live, nil, nil, testLogger())
if err := si.importCoreArtifact(context.Background(), compact); err != nil {
t.Fatalf("importCoreArtifact (compact): %v", err)
}
got := si.LookupArtistByMBID(artA)
if got == nil {
t.Fatal("artist from a compact artifact was not imported")
}
if got.Title != "Artist A" || got.MBID != artA {
t.Errorf("imported %+v, want Artist A / %s", got, artA)
}
}
// TestImportCoreArtifactReadsTotalsWhenPresent covers both halves of the
// denominator's arrival: an artifact that carries total_tracks imports
// it, and one built before the column existed still imports at all.
//
// The second half is the one worth a test. Adding a column to the
// importer's SELECT list is how you break every artifact already
// published - "no such column: total_tracks", on a file nobody can
// re-cut retroactively - so the importer asks the artifact what it has,
// the same way it asks which encoding it uses.
func TestImportCoreArtifactReadsTotalsWhenPresent(t *testing.T) {
path := writeTestArtifact(t, validMeta(), []artifactRow{
{"release_group", rgA, "Big Album", "Solo Star", artA, 5000},
})
// The exporter writes the column; writeTestArtifact builds the older
// shape, so add it here rather than changing every other test's
// fixture to carry a value they do not use.
artifact, err := sql.Open("sqlite", "file:"+path)
if err != nil {
t.Fatalf("reopen artifact: %v", err)
}
for _, stmt := range []string{
`ALTER TABLE explore_index ADD COLUMN total_tracks INTEGER NOT NULL DEFAULT 0`,
`UPDATE explore_index SET total_tracks = 12`,
} {
if _, err := artifact.Exec(stmt); err != nil {
t.Fatalf("add total_tracks: %v", err)
}
}
_ = artifact.Close()
live := database.NewTestDB(t)
si := NewSearchIndex(live, nil, nil, testLogger())
if err := si.importCoreArtifact(context.Background(), path); err != nil {
t.Fatalf("importCoreArtifact: %v", err)
}
rg := si.LookupReleaseGroupByMBID(rgA)
if rg == nil {
t.Fatal("release group was not imported")
}
if rg.TotalTracks != 12 {
t.Errorf("TotalTracks = %d, want 12", rg.TotalTracks)
}
// And the shape that predates the column: the same import, from an
// artifact that has no total_tracks at all.
older := writeTestArtifact(t, validMeta(), []artifactRow{
{"release_group", rgB, "Duet Album", "Solo Star", artA, 4000},
})
live2 := database.NewTestDB(t)
si2 := NewSearchIndex(live2, nil, nil, testLogger())
if err := si2.importCoreArtifact(context.Background(), older); err != nil {
t.Fatalf("importCoreArtifact (no total_tracks column): %v", err)
}
old := si2.LookupReleaseGroupByMBID(rgB)
if old == nil {
t.Fatal("release group from a column-less artifact was not imported")
}
if old.TotalTracks != 0 {
t.Errorf("TotalTracks = %d, want 0 (the catalog does not say)", old.TotalTracks)
}
}
+9 -13
View File
@@ -12,19 +12,15 @@ import (
func seedIndexArtist(t *testing.T, db *database.DB, mbid string, discogFetched int) {
t.Helper()
if _, err := db.ExecContext(
upsertIndexSQL,
"artist", mbid, "Seeded Artist", "Seeded Artist", mbid, "",
0, 0,
0, "", "",
"", "", "",
"", "", "", "",
1, 0,
0, 0, 0,
discogFetched,
); err != nil {
t.Fatalf("seed explore_index row for %q: %v", mbid, err)
}
seedIndexResult(t, db, SearchIndexResult{
EntityType: EntityArtist,
MBID: testMBID(mbid),
Title: "Seeded Artist",
ArtistName: "Seeded Artist",
ArtistMBID: testMBID(mbid),
InLibrary: true,
DiscogFetched: discogFetched == 1,
})
}
// TestArtistEnrichmentMarksAreIndependent is the reason these are two
+113
View File
@@ -0,0 +1,113 @@
package explore
import (
"log/slog"
"net/http"
"net/http/httptest"
"testing"
"yellowjacket/backend/database"
)
// lbPopularityServer serves both top-for-artist popularity endpoints
// with a fixed status and body, and counts what it was asked.
func lbPopularityServer(
t *testing.T, status int, body string,
) (*ListenBrainzClient, *int) {
t.Helper()
requests := 0
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
requests++
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_, _ = w.Write([]byte(body))
},
))
t.Cleanup(srv.Close)
lb := NewListenBrainzClient(NewRateLimiterN(1000), nil, slog.Default())
lb.SetBaseURL(srv.URL)
return lb, &requests
}
// TestTopFetchesSeparateEmptyFromFailed is the distinction the
// owned-artist backfill's mark rests on. ListenBrainz answers 200 with
// `[]` for an artist it has no popularity data for — which is most of a
// long-tail library — and an empty answer is a *complete* one. When
// discog_fetched was keyed on "did rows come back", those artists were
// never marked, stayed in unenrichedLibraryArtistMBIDs forever, and
// "Filling in artist details" re-ran for them on every single launch.
func TestTopFetchesSeparateEmptyFromFailed(t *testing.T) {
t.Parallel()
si := NewSearchIndex(database.NewTestDB(t), nil, nil, slog.Default())
artist := lbSitewideArtist{
ArtistMBID: "44444444-4444-4444-4444-444444444444",
ArtistName: "Nobody Has Listened",
}
t.Run("empty is success", func(t *testing.T) {
t.Parallel()
lb, _ := lbPopularityServer(t, http.StatusOK, `[]`)
rgs, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50)
if err != nil || len(rgs) != 0 {
t.Errorf("top RGs: got %d rows, err %v; want 0 rows and no error", len(rgs), err)
}
recs, err := si.fetchTopRecordings(t.Context(), lb, artist, 200)
if err != nil || len(recs) != 0 {
t.Errorf("top recordings: got %d rows, err %v; want 0 rows and no error",
len(recs), err,
)
}
})
// Below indexMinPopularity is the same shape one step in: the
// endpoint answered, we simply keep none of it.
t.Run("everything below the popularity floor is success", func(t *testing.T) {
t.Parallel()
lb, _ := lbPopularityServer(t, http.StatusOK,
`[{"release_group_mbid":"rg-1","total_listen_count":3,`+
`"release_group":{"name":"Obscure"}}]`)
rgs, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50)
if err != nil || len(rgs) != 0 {
t.Errorf("top RGs: got %d rows, err %v; want 0 rows and no error", len(rgs), err)
}
})
// A failure must stay a failure, or the retry this is built on goes
// away and a throttled run marks artists it never fetched.
t.Run("HTTP error is a failure", func(t *testing.T) {
t.Parallel()
lb, _ := lbPopularityServer(t, http.StatusServiceUnavailable, `nope`)
if _, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50); err == nil {
t.Error("top RGs reported success on a 503")
}
if _, err := si.fetchTopRecordings(t.Context(), lb, artist, 200); err == nil {
t.Error("top recordings reported success on a 503")
}
})
t.Run("unparseable body is a failure", func(t *testing.T) {
t.Parallel()
lb, _ := lbPopularityServer(t, http.StatusOK, `{"not":"an array"}`)
if _, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50); err == nil {
t.Error("top RGs reported success on a body it could not read")
}
})
}
+34 -5
View File
@@ -47,6 +47,13 @@ const (
// canonicalProgressRows controls progress reporting during the
// canonical CSV scan (~30M rows total).
canonicalProgressRows = 2_000_000
// maxReleaseTracks caps the per-release track count, so a malformed
// row cannot turn the denominator into nonsense. Well above the
// longest real release, and it is a cap rather than a rejection
// because a box set reporting 999 is still a better answer than one
// reporting nothing.
maxReleaseTracks = 999
)
// Per-artist discography coverage (S2). The global budgets above keep
@@ -329,6 +336,18 @@ type canonicalScan struct {
releaseToRG map[uuid16]rgTarget
artistNames map[uuid16]string
// releaseTracks counts the canonical dump's rows per kept release,
// which is that release's track count: the dump carries one row per
// recording per canonical release.
//
// It is counted *before* the popularity filter below, unlike almost
// everything else here, because a denominator built from the kept
// recordings would say "9" about a twelve-track album whose other
// three are unpopular - which is worse than saying nothing, and is
// exactly the confident lie that kept whole tracklists out of the
// artifact. Bounded by the kept release set, not by MusicBrainz.
releaseTracks map[uuid16]uint16
// artistTracks accumulates each target artist's top recordings for
// S2 coverage; merged into recordings before assembly.
artistTracks *perArtistTracks
@@ -454,10 +473,11 @@ func (imp *dumpImporter) scanCanonicalDump(
defer zr.Close()
scan := &canonicalScan{
releaseInfos: make(map[uuid16]releaseInfo, len(ks.releases)),
releaseToRG: make(map[uuid16]rgTarget, len(ks.releases)),
artistNames: make(map[uuid16]string, len(ks.artists)),
artistTracks: newPerArtistTracks(),
releaseInfos: make(map[uuid16]releaseInfo, len(ks.releases)),
releaseToRG: make(map[uuid16]rgTarget, len(ks.releases)),
artistNames: make(map[uuid16]string, len(ks.artists)),
releaseTracks: make(map[uuid16]uint16, len(ks.releases)),
artistTracks: newPerArtistTracks(),
}
sawData, sawRedirect := false, false
@@ -614,9 +634,17 @@ func (imp *dumpImporter) scanCanonicalData(
}
}
// Release display info for release-group titling.
// Release display info for release-group titling, and the
// release's track count. Both are for kept releases only, and
// the count is taken here rather than below because every row
// of this dump is one track of its release regardless of how
// often anyone played it.
if relOK {
if _, kept := ks.releases[relMBID]; kept {
if n := scan.releaseTracks[relMBID]; n < maxReleaseTracks {
scan.releaseTracks[relMBID] = n + 1
}
if _, seen := scan.releaseInfos[relMBID]; !seen {
firstArtist := ""
if len(artistMBIDs) > 0 {
@@ -1004,6 +1032,7 @@ func (imp *dumpImporter) assembleIndex(
ArtistName: info.artistName,
ArtistMBID: info.artistMBID,
Popularity: int(agg.listens),
TotalTracks: int(scan.releaseTracks[agg.bestRel]),
CAAReleaseMBID: formatUUID(agg.canonical[:]),
})
+49 -3
View File
@@ -277,6 +277,12 @@ func canonicalDataCSV(t *testing.T) []byte {
"3", "11", "{" + artA + "," + artB + "}", "Solo Star feat. Guest",
relB, "Duet Album", recC, "Duet Song", "x", "1",
},
// Unplayed, so it survives no popularity floor and is not
// indexed -- but it is still a track on Big Album.
{
"4", "10", "{" + artA + "}", "Solo Star",
relA, "Big Album", recD, "Album Filler", "x", "1",
},
}
return csvBytes(t, rows)
@@ -594,7 +600,7 @@ func TestDumpImportEndToEnd(t *testing.T) {
rows, err := db.QueryContext(
"SELECT title, popularity FROM explore_index WHERE mbid = ? AND entity_type = ?",
mbid, entityType,
dbMBID(mbid), dbEntityType(entityType),
)
if err != nil {
t.Fatalf("query: %v", err)
@@ -627,11 +633,51 @@ func TestDumpImportEndToEnd(t *testing.T) {
assertRow(rgB, "release_group", "Duet Album", 12)
assertRow(artA, "artist", "Solo Star", 35)
// The per-release-group denominator: how many tracks the release
// has, counted from the canonical dump *before* the popularity
// filter. Big Album has three, one of which nobody has played and
// which is therefore not indexed as a recording at all -- a
// denominator built from the kept recordings would say two, and
// "you have 2 of 2" about a three-track album is worse than saying
// nothing.
assertTotalTracks := func(mbid string, want int) {
t.Helper()
var got int
if err := db.QueryRowWriter(
"SELECT total_tracks FROM explore_index WHERE mbid = ? AND entity_type = 2",
dbMBID(mbid),
).Scan(&got); err != nil {
t.Fatalf("read total_tracks for %s: %v", mbid, err)
}
if got != want {
t.Errorf("total_tracks for %s = %d, want %d", mbid, got, want)
}
}
assertTotalTracks(rgA, 3)
assertTotalTracks(rgB, 1)
// And the unplayed track is still not an indexed recording.
var fillerRows int
if err := db.QueryRowWriter(
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", dbMBID(recD),
).Scan(&fillerRows); err != nil {
t.Fatalf("count recD rows: %v", err)
}
if fillerRows != 0 {
t.Errorf("unplayed track was indexed as a recording (%d rows)", fillerRows)
}
// artB only ever appears in a multi-artist credit: no name is
// derivable from the dump, so it must be queued for the API
// metadata patch instead of being written nameless.
rows, err := db.QueryContext(
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", artB,
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", dbMBID(artB),
)
if err != nil {
t.Fatalf("query artB: %v", err)
@@ -1006,7 +1052,7 @@ func TestListenerCountUpdateDoesNotTouchPopularity(t *testing.T) {
}
rows, err := db.QueryContext(
"SELECT popularity, listener_count FROM explore_index WHERE mbid = ?", recA,
"SELECT popularity, listener_count FROM explore_index WHERE mbid = ?", dbMBID(recA),
)
if err != nil {
t.Fatalf("query: %v", err)
+7 -2
View File
@@ -277,7 +277,10 @@ func (si *SearchIndex) commitListenDeltas(
defer func() { _ = tx.Rollback() }()
if _, err := tx.Exec(
"CREATE TEMP TABLE IF NOT EXISTS incr_delta (mbid TEXT, kind TEXT, delta INTEGER)",
// mbid and kind are stored the way explore_index stores them,
// so the join below is a plain equality rather than a
// conversion per row.
"CREATE TEMP TABLE IF NOT EXISTS incr_delta (mbid BLOB, kind INTEGER, delta INTEGER)",
); err != nil {
return fmt.Errorf("incremental temp table: %w", err)
}
@@ -357,8 +360,10 @@ func insertDeltas(tx *sql.Tx, kind string, deltas map[string]uint32) error {
return nil
}
code := entityCode(kind)
for mbid, d := range deltas {
rowArgs = append(rowArgs, mbid, kind, int64(d))
rowArgs = append(rowArgs, mbidBytes(mbid), code, int64(d))
pending++
if pending >= deltaInsertBatch {
+1 -1
View File
@@ -36,7 +36,7 @@ func popularityOf(t *testing.T, db *database.DB, mbid string) (int, bool) {
t.Helper()
rows, err := db.QueryContext(
"SELECT popularity FROM explore_index WHERE mbid = ?", mbid,
"SELECT popularity FROM explore_index WHERE mbid = ?", dbMBID(mbid),
)
if err != nil {
t.Fatalf("query popularity: %v", err)
+6 -6
View File
@@ -54,7 +54,7 @@ func (imp *dumpImporter) runPatchPasses(ctx context.Context) {
func (imp *dumpImporter) patchArtistMetadata(ctx context.Context) {
rows, err := imp.si.db.QueryContext(`
SELECT mbid FROM explore_index
WHERE entity_type = 'artist'
WHERE entity_type = 1 /* artist */
AND (artist_type = '' OR country = '' OR title = '' OR title = mbid)
`)
if err != nil {
@@ -152,7 +152,7 @@ func (imp *dumpImporter) patchSimilarArtists(ctx context.Context) {
for _, s := range similar {
_, _ = imp.si.db.ExecContext(
"UPDATE explore_index SET is_similar = 1 WHERE artist_mbid = ?",
s.ArtistMBID,
dbMBID(s.ArtistMBID),
)
}
}
@@ -219,7 +219,7 @@ func (imp *dumpImporter) topMBIDs(entityType string, limit int) []string {
WHERE entity_type = ? AND listener_count = 0
ORDER BY popularity DESC
LIMIT ?
`, entityType, limit)
`, dbEntityType(entityType), limit)
if err != nil {
return nil
}
@@ -229,9 +229,9 @@ func (imp *dumpImporter) topMBIDs(entityType string, limit int) []string {
var mbids []string
for rows.Next() {
var m string
var m dbMBID
if err := rows.Scan(&m); err == nil {
mbids = append(mbids, m)
mbids = append(mbids, string(m))
}
}
@@ -264,7 +264,7 @@ func (si *SearchIndex) updateListenerCounts(updates map[string]PopularityData) i
`UPDATE explore_index
SET listener_count = ?
WHERE mbid = ? AND listener_count < ?`,
data.ListenerCount, strings.ToLower(mbid), data.ListenerCount,
data.ListenerCount, dbMBID(strings.ToLower(mbid)), data.ListenerCount,
)
if err != nil {
continue
+108
View File
@@ -0,0 +1,108 @@
package explore
import (
"testing"
"yellowjacket/backend/database"
)
// TestStoredEncodingRoundTrips walks every read path in the package
// against a row written by the real upsert.
//
// It exists because of how this storage change fails when it fails.
// MBIDs are stored as 16 raw bytes and entity types as codes - which
// took the catalog and its indexes from 780 MB to 405 MB, measured on a
// real 2,052,200-row catalog - and SQLite does not coerce between TEXT
// and BLOB. A query that still compares against a 36-character string
// therefore returns *no rows* rather than an error, and a scan into a
// plain string yields sixteen bytes of mojibake. Neither shows up as a
// failure anywhere except in a result that is quietly empty.
//
// So this is not a unit test of the encoding (that is TestMBIDRoundTrip)
// but a sweep: every query that reads the catalog, asserted to return
// something and to hand back canonical dashed ids.
func TestStoredEncodingRoundTrips(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
const (
artist = "c0b2500e-0cef-4130-9b13-1b9d9a2f2c07"
album = "11111111-2222-3333-4444-555555555555"
)
si.upsertBatch([]SearchIndexResult{
{
EntityType: EntityArtist,
MBID: artist,
Title: "Radiohead",
ArtistName: "Radiohead",
ArtistMBID: artist,
Popularity: 90000,
},
{
EntityType: EntityReleaseGroup,
MBID: album,
Title: "Kid A",
ArtistName: "Radiohead",
ArtistMBID: artist,
Popularity: 80000,
CAAReleaseMBID: album,
},
})
// The stored form is the compact one, not the strings above.
var typeMBID, typeEntity string
if err := db.QueryRowWriter(
"SELECT typeof(mbid), typeof(entity_type) FROM explore_index LIMIT 1",
).Scan(&typeMBID, &typeEntity); err != nil {
t.Fatalf("typeof: %v", err)
}
if typeMBID != "blob" || typeEntity != "integer" {
t.Errorf("stored as mbid=%s entity_type=%s, want blob/integer", typeMBID, typeEntity)
}
si.MarkReadyIfPopulated()
// Read paths.
if got := si.LookupArtistByMBID(artist); got == nil {
t.Error("LookupArtistByMBID found nothing")
} else if got.MBID != artist || got.ArtistMBID != artist || got.EntityType != EntityArtist {
t.Errorf("lookup artist = %+v, want dashed ids and the artist type", got)
}
if got := si.LookupReleaseGroupByMBID(album); got == nil {
t.Error("LookupReleaseGroupByMBID found nothing")
} else if got.MBID != album || got.EntityType != EntityReleaseGroup {
t.Errorf("lookup album = %+v, want the dashed id and the release-group type", got)
}
if rgs := si.TopReleaseGroupsByArtist(artist, 5); len(rgs) == 0 {
t.Error("TopReleaseGroupsByArtist found nothing")
} else if rgs[0].MBID != album || rgs[0].ArtistMBID != artist {
t.Errorf("top release groups[0] = %+v, want %s by %s", rgs[0], album, artist)
}
// The exact-match tier reads the two partial LOWER() indexes, whose
// predicate has to agree with its WHERE clause or the seek silently
// becomes a scan.
if m := si.ExactMatches("radiohead", 3); len(m) == 0 {
t.Error("ExactMatches found nothing")
} else if m[0].MBID != artist || m[0].EntityType != EntityArtist {
t.Errorf("exact match[0] = %+v, want the artist", m[0])
}
if hits := si.Search(t.Context(), "radiohead", 5); len(hits) == 0 {
t.Error("Search found nothing")
} else if hits[0].MBID != artist || hits[0].EntityType != EntityArtist {
t.Errorf("search hit[0] = %+v, want the artist", hits[0])
}
if b := si.GetPopularityBatch([]string{artist, album}); b == nil || len(b.Popularity) != 2 {
t.Errorf("GetPopularityBatch = %+v, want two entries", b)
}
if m := si.ReleaseGroupMBIDsForCAAReleaseMBIDs([]string{album}); len(m) != 1 {
t.Errorf("CAA lookup = %v, want one entry", m)
}
}
+38 -6
View File
@@ -22,8 +22,8 @@ func seedIndexRow(
_, err := db.ExecContext(`
INSERT INTO explore_index
(entity_type, mbid, title, artist_name, artist_mbid, popularity, listener_count)
VALUES (?, ?, ?, ?, '', ?, ?)
`, entityType, mbid, title, artist, popularity, popularity/10)
VALUES (?, ?, ?, ?, x'', ?, ?)
`, dbEntityType(entityType), dbMBID(testMBID(mbid)), title, artist, popularity, popularity/10)
if err != nil {
t.Fatalf("seed %s/%s: %v", entityType, mbid, err)
}
@@ -73,12 +73,12 @@ func TestEvalHarnessIndexRanking(t *testing.T) {
{
Query: "radiohead",
Note: "popular exact artist match",
Expect: []eval.Expected{{Type: "artist", MBID: "rh"}},
Expect: []eval.Expected{{Type: "artist", MBID: testMBID("rh")}},
},
{
Query: "the teenagers",
Note: "low-popularity exact match must beat high-popularity article match",
Expect: []eval.Expected{{Type: "artist", MBID: "teenagers"}},
Expect: []eval.Expected{{Type: "artist", MBID: testMBID("teenagers")}},
},
}
@@ -123,8 +123,13 @@ func TestExploreFTSDiacriticFolding(t *testing.T) {
t.Fatalf("query %q returned no hits", tc.query)
}
if hits[0].MBID != tc.wantMBID {
t.Errorf("query %q: top hit = %q, want %q", tc.query, hits[0].MBID, tc.wantMBID)
if hits[0].MBID != testMBID(tc.wantMBID) {
t.Errorf(
"query %q: top hit = %q, want %q",
tc.query,
hits[0].MBID,
testMBID(tc.wantMBID),
)
}
})
}
@@ -148,3 +153,30 @@ func TestEvalFixtureFileParses(t *testing.T) {
}
}
}
// seedIndexResult writes one explore_index row through the same
// upsertBatch every other writer uses.
//
// The three seeders that used to bind upsertIndexSQL's parameters by
// hand said, in a comment, that this was on purpose: a schema change
// should break the tests where it breaks the app. It did not - it
// broke them at "missing argument with index 25", one file at a time,
// for a column none of them cares about. Going through the one writer
// keeps the property they wanted (a field written to the wrong column
// still fails here) without three copies of a 24-argument list.
func seedIndexResult(t *testing.T, db *database.DB, r SearchIndexResult) {
t.Helper()
NewSearchIndex(db, nil, nil, slog.Default()).
upsertBatch([]SearchIndexResult{r})
// upsertBatch logs and swallows, which is right for a background
// merge and useless for a fixture, so the row is checked for.
var n int
if err := db.QueryRowWriter(
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", dbMBID(r.MBID),
).Scan(&n); err != nil || n == 0 {
t.Fatalf("seed explore_index row %q: not written (%v)", r.MBID, err)
}
}
+7
View File
@@ -130,12 +130,16 @@ func NewExploreService(logger *slog.Logger, db *database.DB) *Service {
// MusicBrainz returns the shared cached MB client so other services
// (e.g. autotag) can reuse it without spinning up a second limiter.
//
//wails:ignore // internal wiring, not part of the app's IPC surface.
func (e *Service) MusicBrainz() *MusicBrainzClient {
return e.mb
}
// CAALimiter returns the shared Cover Art Archive rate limiter.
// Consumers must respect it for any fresh CAA HTTP GETs.
//
//wails:ignore // internal wiring, not part of the app's IPC surface.
func (e *Service) CAALimiter() *RateLimiter {
return e.caaLimiter
}
@@ -185,6 +189,8 @@ func (e *Service) CoreCatalogImported() bool {
// SetJobRegistry wires the background job registry into the search
// index so its build reports progress and controls to the frontend.
//
//wails:ignore // internal wiring, not part of the app's IPC surface.
func (e *Service) SetJobRegistry(reg *jobs.Registry) {
e.index.SetJobRegistry(reg)
}
@@ -561,6 +567,7 @@ func (e *Service) LookupReleaseGroup(mbid string) (*MBReleaseGroup, error) {
FirstReleaseDate: indexed.ReleaseDate,
InLibrary: indexed.InLibrary || indexed.LocalReleaseGroupID > 0,
LocalID: indexed.LocalReleaseGroupID,
TotalTracks: indexed.TotalTracks,
}
// Background: fetch full MB data once per RG to fill in fields
+38 -15
View File
@@ -18,9 +18,22 @@ func NewLibraryMBIDIndex(db *database.DB) *LibraryMBIDIndex {
return &LibraryMBIDIndex{db: db}
}
// CheckMBIDs returns which of the given MBIDs exist in the local
// library. The returned map has MBID entity type ("artist",
// CheckMBIDs returns which of the given MBIDs the library actually
// has a file for. The returned map is MBID -> entity type ("artist",
// "release_group", or "recording").
//
// The "has a file" part is the whole point and is what this used to get
// wrong. It was three `SELECT mbid FROM <metadata table>` queries, and
// a metadata row could outlive the file that created it - retagging a
// file abandoned its old recording row, which kept the old MBID
// forever. Measured on a real library: 812 orphaned recordings, of
// which 218 carried MBIDs, and 129 catalog rows that this function
// therefore reported as owned. Every one of them rendered as a track
// you have, with a play button that could not work, because playback
// resolves files and this resolved metadata.
//
// Each branch now joins audio_files. An entity is in your library if
// and only if a file says so.
func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
if len(mbids) == 0 {
return nil
@@ -28,16 +41,26 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
result := make(map[string]string, len(mbids))
// Batch check all MBIDs against each table with a single IN query.
type tableEntity struct {
table string
type entityQuery struct {
entityType string
query string
}
tables := []tableEntity{
{"artists", "artist"},
{"release_groups", "release_group"},
{"recordings", "recording"},
queries := []entityQuery{
{"recording", `SELECT DISTINCT recording_mbid FROM audio_files
WHERE recording_mbid IN (%s)`},
{"release_group", `SELECT DISTINCT al.mbid FROM albums al
JOIN audio_files af ON af.album_id = al.id
WHERE al.mbid IN (%s)`},
{"artist", `SELECT DISTINCT a.mbid FROM artists a
WHERE a.mbid IN (%s) AND (
EXISTS (SELECT 1 FROM audio_files af WHERE af.artist_id = a.id)
OR EXISTS (
SELECT 1 FROM albums al
JOIN audio_files af2 ON af2.album_id = al.id
WHERE al.artist_id = a.id
)
)`},
}
// Build a set of MBIDs still unresolved.
@@ -48,12 +71,11 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
}
}
for _, te := range tables {
for _, eq := range queries {
if len(remaining) == 0 {
break
}
// Build IN clause from remaining MBIDs.
placeholders := make([]string, 0, len(remaining))
args := make([]any, 0, len(remaining))
@@ -62,9 +84,10 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
args = append(args, m)
}
//nolint:gosec // table name is hardcoded from the tables slice above
query := "SELECT mbid FROM " + te.table + " WHERE mbid IN (" +
strings.Join(placeholders, ",") + ")"
//nolint:gosec // the query text is a constant from the slice above
query := strings.Replace(
eq.query, "%s", strings.Join(placeholders, ","), 1,
)
rows, err := idx.db.QueryContext(query, args...)
if err != nil {
@@ -74,7 +97,7 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
for rows.Next() {
var mbid string
if err := rows.Scan(&mbid); err == nil {
result[mbid] = te.entityType
result[mbid] = eq.entityType
delete(remaining, mbid)
}
}
+18 -6
View File
@@ -33,6 +33,12 @@ type ListenBrainzClient struct {
limiter *RateLimiter
cache *Cache
logger *slog.Logger
// baseURL is listenBrainzBaseURL unless a test redirects it. It is
// per-client rather than a package variable — the MB client's
// SetBaseURL shape — so a test that points one client at an
// httptest server does not stop being parallel-safe.
baseURL string
}
// NewListenBrainzClient creates a ListenBrainz API client.
@@ -46,9 +52,15 @@ func NewListenBrainzClient(
limiter: limiter,
cache: cache,
logger: logger,
baseURL: listenBrainzBaseURL,
}
}
// SetBaseURL redirects this client at another host. Tests only.
func (c *ListenBrainzClient) SetBaseURL(url string) {
c.baseURL = strings.TrimSuffix(url, "/")
}
// TopRecordingsForArtist returns the most-listened recordings for
// the artist identified by artistMBID.
func (c *ListenBrainzClient) TopRecordingsForArtist(
@@ -56,7 +68,7 @@ func (c *ListenBrainzClient) TopRecordingsForArtist(
) ([]LBTopRecording, error) {
url := fmt.Sprintf(
"%s/1/popularity/top-recordings-for-artist/%s",
listenBrainzBaseURL,
c.baseURL,
artistMBID,
)
cacheKey := "lb:top-recordings:" + artistMBID
@@ -104,7 +116,7 @@ func (c *ListenBrainzClient) TopReleaseGroupsForArtist(
) ([]LBTopReleaseGroup, error) {
url := fmt.Sprintf(
"%s/1/popularity/top-release-groups-for-artist/%s",
listenBrainzBaseURL,
c.baseURL,
artistMBID,
)
cacheKey := "lb:top-release-groups:" + artistMBID
@@ -234,7 +246,7 @@ func (c *ListenBrainzClient) ArtistPopularity(
return nil, nil //nolint:nilnil
}
url := listenBrainzBaseURL + "/1/popularity/artist"
url := c.baseURL + "/1/popularity/artist"
cacheKey := "lb:pop:artist:" + hashMBIDs(mbids)
if data, ok := c.cache.Get(cacheKey); ok {
@@ -265,7 +277,7 @@ func (c *ListenBrainzClient) RecordingPopularity(
return nil, nil //nolint:nilnil
}
url := listenBrainzBaseURL + "/1/popularity/recording"
url := c.baseURL + "/1/popularity/recording"
cacheKey := "lb:pop:recording:" + hashMBIDs(mbids)
if data, ok := c.cache.Get(cacheKey); ok {
@@ -296,7 +308,7 @@ func (c *ListenBrainzClient) ReleaseGroupPopularity(
return nil, nil //nolint:nilnil
}
url := listenBrainzBaseURL + "/1/popularity/release-group"
url := c.baseURL + "/1/popularity/release-group"
cacheKey := "lb:pop:release-group:" + hashMBIDs(mbids)
if data, ok := c.cache.Get(cacheKey); ok {
@@ -342,7 +354,7 @@ func (c *ListenBrainzClient) BatchArtistMetadata(
return nil, nil //nolint:nilnil
}
url := listenBrainzBaseURL + "/1/metadata/artist/?artist_mbids=" + strings.Join(mbids, ",")
url := c.baseURL + "/1/metadata/artist/?artist_mbids=" + strings.Join(mbids, ",")
cacheKey := "lb:meta:artist:" + hashMBIDs(mbids)
if data, ok := c.cache.Get(cacheKey); ok {
+13 -9
View File
@@ -10,7 +10,7 @@ import (
// LyricsResult is a single lyric-search hit, mapped from the DB layer
// into the camelCase shape the frontend consumes.
type LyricsResult struct {
RecordingID int64 `json:"recordingId"`
AudioFileID int64 `json:"audioFileId"`
FilePath string `json:"filePath"`
LengthMs int64 `json:"lengthMs"`
Title string `json:"title"`
@@ -55,7 +55,7 @@ func (e *Service) SearchLyrics(query string) []LyricsResult {
out := make([]LyricsResult, 0, len(hits))
for _, h := range hits {
out = append(out, LyricsResult{
RecordingID: h.RecordingID,
AudioFileID: h.AudioFileID,
FilePath: h.FilePath,
LengthMs: h.LengthMilliseconds,
Title: h.Title,
@@ -67,18 +67,18 @@ func (e *Service) SearchLyrics(query string) []LyricsResult {
return out
}
// GetTrackLyrics returns lyrics for a recording. If the library
// GetTrackLyrics returns lyrics for a file. If the library
// already has them (from embedded tags) they're returned as-is;
// otherwise it fetches from LRCLIB, persists them (updating the FTS
// index), and returns them. Never returns an error to the frontend —
// a miss just yields an empty result.
func (e *Service) GetTrackLyrics(recordingID int64) TrackLyrics {
stored, err := e.db.GetRecordingLyrics(recordingID)
func (e *Service) GetTrackLyrics(audioFileID int64) TrackLyrics {
stored, err := e.db.GetLyrics(audioFileID)
if err == nil && stored != "" {
return TrackLyrics{Plain: stored, Source: "embedded"}
}
lookup, err := e.db.RecordingLyricLookup(recordingID)
lookup, err := e.db.FileLyricLookup(audioFileID)
if err != nil || lookup == nil {
return TrackLyrics{}
}
@@ -150,7 +150,7 @@ func (e *Service) backfillLibraryLyrics(ctx context.Context) {
return
}
candidates, err := e.db.RecordingsMissingLyrics(lyricsBackfillBatch)
candidates, err := e.db.FilesMissingLyrics(lyricsBackfillBatch)
if err != nil {
e.logger.Warn("lyrics backfill: query failed", "err", err)
@@ -223,8 +223,12 @@ func (e *Service) fetchAndStoreLyrics(
return nil
}
if err := e.db.SetRecordingLyrics(c.RecordingID, lyrics.Plain); err != nil {
e.logger.Warn("lyrics store failed", "recordingId", c.RecordingID, "err", err)
// Marked `lrclib` rather than `tag`: these came off the network and
// a rebuild that discards them pays for them again.
if err := e.db.SetLyrics(
c.AudioFileID, lyrics.Plain, "lrclib", c.RecordingMBID,
); err != nil {
e.logger.Warn("lyrics store failed", "audioFileId", c.AudioFileID, "err", err)
return nil
}
+203
View File
@@ -0,0 +1,203 @@
package explore
import (
"database/sql/driver"
"encoding/hex"
"errors"
"fmt"
"strings"
)
// The catalog stores a MusicBrainz id as its 16 raw bytes and an entity
// type as a small integer, rather than as the 36-character text and the
// words the rest of the app uses.
//
// This is a size decision and it is a large one. Measured on a real
// 2,052,200-row catalog: the three MBID columns and `entity_type` are
// 220 MB of a 383 MB table, and they are carried again in every index
// that keys on them. Converting the table and its four indexes took
// **677 MB to 389 MB** with the same row count.
//
// Everything above this file still speaks strings: `SearchIndexResult`
// carries `"artist"` and a dashed MBID, the frontend receives them, and
// the conversion happens only where a value crosses into SQL. The
// alternative - blobs and codes reaching the rest of the app - would
// trade 288 MB for a type that nothing else wants.
//
// Two things make a mistake here loud rather than silent, which matters
// because SQLite does not coerce between TEXT and BLOB: a query
// comparing a blob column against a string returns *no rows* rather
// than an error.
//
// - Writes are guarded by a CHECK on the column (16 bytes, or empty),
// so a stringly write fails at the insert rather than sitting in
// the table looking fine.
// - Reads scan into `dbMBID`, whose Scan rejects anything that is not
// 16 bytes or empty. A column that somehow holds text produces an
// error instead of a garbled id.
type dbMBID string
// mbidLen is the byte length of a raw MusicBrainz id.
const mbidLen = 16
// Value encodes the id for storage: 16 raw bytes, or empty for "none".
func (m dbMBID) Value() (driver.Value, error) {
return mbidBytes(string(m)), nil
}
// Scan decodes a stored id back to its canonical dashed form.
func (m *dbMBID) Scan(src any) error {
switch v := src.(type) {
case nil:
*m = ""
return nil
case []byte:
s, err := mbidFromBytes(v)
if err != nil {
return err
}
*m = dbMBID(s)
return nil
case string:
// Tolerated for the one caller that reads through a view or a
// literal: a canonical id is already the right answer.
*m = dbMBID(v)
return nil
default:
return fmt.Errorf("%w: %T", errMBIDType, src)
}
}
// mbidBytes encodes a dashed MusicBrainz id as its 16 raw bytes. An
// id that is not one - including the empty string, which is how "no
// MBID" is spelled throughout - encodes as empty.
func mbidBytes(s string) []byte {
if s == "" {
return []byte{}
}
raw, err := hex.DecodeString(strings.ReplaceAll(s, "-", ""))
if err != nil || len(raw) != mbidLen {
return []byte{}
}
return raw
}
// mbidFromBytes decodes stored bytes back to the canonical dashed form.
func mbidFromBytes(b []byte) (string, error) {
if len(b) == 0 {
return "", nil
}
if len(b) != mbidLen {
return "", fmt.Errorf("%w: %d bytes", errMBIDLength, len(b))
}
h := hex.EncodeToString(b)
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:], nil
}
// Entity types are stored as codes. The names are the app's, the codes
// are the table's, and nothing outside this file should see a code.
const (
entityCodeArtist = 1
entityCodeReleaseGroup = 2
entityCodeRecording = 3
)
// Entity type names, as everything above the SQL boundary spells them.
const (
EntityArtist = "artist"
EntityReleaseGroup = "release_group"
EntityRecording = "recording"
)
// entityCode maps a name to its stored code. An unknown name yields 0,
// which matches no row - the same answer the old string comparison gave
// and the reason this is not an error.
func entityCode(name string) int {
switch name {
case EntityArtist:
return entityCodeArtist
case EntityReleaseGroup:
return entityCodeReleaseGroup
case EntityRecording:
return entityCodeRecording
default:
return 0
}
}
// entityName maps a stored code back to its name.
func entityName(code int) string {
switch code {
case entityCodeArtist:
return EntityArtist
case entityCodeReleaseGroup:
return EntityReleaseGroup
case entityCodeRecording:
return EntityRecording
default:
return ""
}
}
// dbEntityType scans a stored entity-type code as its name.
//
// The db prefix keeps it out of the way of the many `mbid string` and
// `entityType string` locals in this package: these two types exist
// only at the SQL boundary, and a name that shadowed one of those
// would turn a conversion into a confusing compile error rather than
// an obvious one.
type dbEntityType string
// Value encodes the name for storage.
func (e dbEntityType) Value() (driver.Value, error) {
return int64(entityCode(string(e))), nil
}
// Scan decodes a stored code back to its name.
func (e *dbEntityType) Scan(src any) error {
switch v := src.(type) {
case nil:
*e = ""
return nil
case int64:
*e = dbEntityType(entityName(int(v)))
return nil
case string:
*e = dbEntityType(v)
return nil
case []byte:
*e = dbEntityType(v)
return nil
default:
return fmt.Errorf("%w: %T", errEntityTypeType, src)
}
}
// Errors from the encoding boundary. They exist so a type confusion
// here is reported rather than silently producing an id that matches
// nothing.
var (
errMBIDType = errors.New("cannot scan MusicBrainz id from")
errMBIDLength = errors.New("stored MusicBrainz id has the wrong length")
errEntityTypeType = errors.New("cannot scan entity type from")
)
// A query that names an entity type inline writes the code with the
// name beside it - `entity_type = 1 /* artist */`. Splicing a Go
// constant into the SQL would keep them in step automatically but makes
// every such query a concatenation; the codes are pinned by
// TestEntityCodesAreStable instead, because they are a storage format
// and changing one is not a refactor.
+105
View File
@@ -0,0 +1,105 @@
package explore
import (
"crypto/sha256"
"encoding/hex"
"testing"
)
// testMBID turns a short fixture label into a well-formed MusicBrainz
// id, and passes a real one through unchanged.
//
// The catalog stores an id as its 16 raw bytes and the column says so
// (`CHECK(length(mbid) = 16)`), so a fixture can no longer call itself
// "rh". Deriving one from the label keeps the fixtures readable — the
// same label is the same id in a seed and in the assertion that reads
// it back — without letting a test write something the app could not.
func testMBID(label string) string {
if len(mbidBytes(label)) == mbidLen {
return label
}
sum := sha256.Sum256([]byte(label))
h := hex.EncodeToString(sum[:mbidLen])
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:]
}
// TestEntityCodesAreStable pins the stored entity-type codes.
//
// They are a storage format, not an enum: the queries that name a type
// inline write the number with the name beside it
// (`entity_type = 1 /* artist */`), so changing one here without
// changing those would leave the catalog answering the wrong questions
// silently.
func TestEntityCodesAreStable(t *testing.T) {
t.Parallel()
for name, want := range map[string]int{
EntityArtist: 1,
EntityReleaseGroup: 2,
EntityRecording: 3,
} {
if got := entityCode(name); got != want {
t.Errorf("entityCode(%q) = %d, want %d", name, got, want)
}
if back := entityName(want); back != name {
t.Errorf("entityName(%d) = %q, want %q", want, back, name)
}
}
if got := entityCode("nonsense"); got != 0 {
t.Errorf("entityCode of an unknown name = %d, want 0 (matches nothing)", got)
}
}
// TestMBIDRoundTrip pins the encoding both ways, including the two
// values that are not ids: empty, which is how "no MBID" is spelled
// everywhere, and rubbish, which must not become a valid-looking id.
func TestMBIDRoundTrip(t *testing.T) {
t.Parallel()
const canonical = "c0b2500e-0cef-4130-9b13-1b9d9a2f2c07"
encoded := mbidBytes(canonical)
if len(encoded) != mbidLen {
t.Fatalf("encoded length = %d, want %d", len(encoded), mbidLen)
}
back, err := mbidFromBytes(encoded)
if err != nil {
t.Fatalf("decode: %v", err)
}
if back != canonical {
t.Errorf("round trip = %q, want %q", back, canonical)
}
if got := mbidBytes(""); len(got) != 0 {
t.Errorf("empty encoded to %d bytes, want 0", len(got))
}
if got := mbidBytes("not-an-mbid"); len(got) != 0 {
t.Errorf("rubbish encoded to %d bytes, want 0", len(got))
}
// A stored value of the wrong length is an error, not a guess.
if _, err := mbidFromBytes([]byte{1, 2, 3}); err == nil {
t.Error("decoding three bytes should fail")
}
}
// TestMBIDScanRejectsText is the guard for the failure this encoding
// could otherwise hide: SQLite does not coerce TEXT to BLOB, so a
// column holding the old 36-character form would silently compare equal
// to nothing. Scanning it must say so.
func TestMBIDScanRejectsText(t *testing.T) {
t.Parallel()
var m dbMBID
if err := m.Scan([]byte("c0b2500e-0cef-4130-9b13-1b9d9a2f2c07")); err == nil {
t.Error("scanning 36 bytes as an MBID should fail")
}
}
+72 -11
View File
@@ -107,12 +107,11 @@ func (e *Service) mixSeedProfile(
artistCounts[artist.ArtistMbid]++
artistNames[artist.ArtistMbid] = artist.ArtistName
}
}
names, err := e.db.ReadQueries.GetGenreNamesByFilePath(ctx, p)
if err == nil {
for _, g := range names {
genres[g] = true
}
for _, names := range e.genresByPath(ctx, seedPaths) {
for _, g := range names {
genres[g] = true
}
}
@@ -148,6 +147,12 @@ func (e *Service) mixCandidates(
candidates := map[string]float64{}
// Every candidate path's genres, in one query. This used to be a
// single-row lookup *per candidate* inside two nested loops -
// twenty seed artists by twenty similar artists by thirty paths is
// twelve thousand queries to assemble one mix.
pathGenres := e.genresByPath(ctx, e.similarArtistPaths(ctx, artistCounts))
for seedArtistMBID, count := range artistCounts {
similar, err := e.SimilarArtists(seedArtistMBID)
if err != nil {
@@ -178,13 +183,11 @@ func (e *Service) mixCandidates(
continue
}
if names, err := e.db.ReadQueries.GetGenreNamesByFilePath(ctx, p); err == nil {
for _, g := range names {
if seedGenres[g] {
weight += mixGenreBoost
for _, g := range pathGenres[p] {
if seedGenres[g] {
weight += mixGenreBoost
break
}
break
}
}
@@ -196,6 +199,64 @@ func (e *Service) mixCandidates(
return candidates
}
// genresByPath returns the genres of many files in one query.
func (e *Service) genresByPath(
ctx context.Context, paths []string,
) map[string][]string {
out := make(map[string][]string, len(paths))
if len(paths) == 0 {
return out
}
rows, err := e.db.ReadQueries.GetGenreNamesByFilePaths(ctx, paths)
if err != nil {
return out
}
for _, row := range rows {
out[row.FilePath] = append(out[row.FilePath], row.Name)
}
return out
}
// similarArtistPaths collects every owned file by an artist similar to
// one of the seeds, so their genres can be fetched in one go.
func (e *Service) similarArtistPaths(
ctx context.Context, artistCounts map[string]int,
) []string {
var paths []string
for seedArtistMBID := range artistCounts {
similar, err := e.SimilarArtists(seedArtistMBID)
if err != nil {
continue
}
if len(similar) > mixSimilarArtistsPerSeed {
similar = similar[:mixSimilarArtistsPerSeed]
}
for _, sim := range similar {
if sim.ArtistMBID == "" {
continue
}
p, err := e.db.ReadQueries.GetFilePathsByArtistMBID(
ctx, sql.NullString{String: sim.ArtistMBID, Valid: true},
)
if err != nil {
continue
}
paths = append(paths, p...)
}
}
return paths
}
// weightedSample picks up to n distinct keys from weights without
// replacement, biased toward higher weight (roulette-wheel selection).
// A key with zero or negative weight is never picked.
+8 -63
View File
@@ -23,69 +23,14 @@ func seedMixTrack(
fp := fmt.Sprintf("/music/%s/track%d.mp3", artistName, id)
_, err := db.ExecContext(
"INSERT INTO artists (id, name, mbid) VALUES (?, ?, ?) "+
"ON CONFLICT(name) DO NOTHING",
id, artistName, artistMBID,
)
if err != nil {
t.Fatalf("insert artist: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO artist_credit (id, text) VALUES (?, ?) "+
"ON CONFLICT(text) DO NOTHING",
id, artistName,
)
if err != nil {
t.Fatalf("insert artist_credit: %v", err)
}
_, err = db.ExecContext(
"INSERT OR IGNORE INTO artist_credit_artist (artist_id, credit_id) VALUES (?, ?)",
id, id,
)
if err != nil {
t.Fatalf("insert artist_credit_artist: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO recordings (id, name, artist_credit_id) VALUES (?, ?, ?)",
id, fmt.Sprintf("Track %d", id), id,
)
if err != nil {
t.Fatalf("insert recording: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO audio_files (id, file_path, length_milliseconds, file_type_id, recording_id) "+
"VALUES (?, ?, 180000, 0, ?)",
id, fp, id,
)
if err != nil {
t.Fatalf("insert audio_file: %v", err)
}
for _, g := range genreNames {
var genreID int64
row := db.QueryRowWriter(
"INSERT INTO genres (name) VALUES (?) "+
"ON CONFLICT(name) DO UPDATE SET name = name RETURNING id",
g,
)
if err := row.Scan(&genreID); err != nil {
t.Fatalf("upsert genre %q: %v", g, err)
}
_, err = db.ExecContext(
"INSERT OR IGNORE INTO recording_genres (recording_id, genre_id) VALUES (?, ?)",
id, genreID,
)
if err != nil {
t.Fatalf("insert recording_genre: %v", err)
}
}
database.InsertTestTrack(t, db, database.TestTrack{
FilePath: fp,
Title: fmt.Sprintf("Track %d", id),
Artist: artistName,
ArtistMBID: artistMBID,
Genres: genreNames,
LengthMs: 180000,
})
return fp
}
+17 -8
View File
@@ -13,18 +13,27 @@ import (
"go.uploadedlobster.com/musicbrainzws2"
)
// How long a MusicBrainz answer is kept.
//
// The rule is what the answer is *about*, not how big it is. A search
// is a ranking and shifts; an entity is a fact about a record that was
// published years ago and does not. Entity data used to expire after a
// week, which meant a fully-populated artist page re-fetched itself
// every week forever - on a real install, 251 of 2,930 cached rows were
// already expired and waiting to be paid for again. The bytes are
// already on disk; re-fetching them buys nothing and spends someone
// else's rate limit.
const (
// cacheTTLSearch is the TTL for search results (results may shift).
cacheTTLSearch = 24 * time.Hour
// cacheTTLEntity is the TTL for lookup/browse results (entity data
// changes rarely).
cacheTTLEntity = 7 * 24 * time.Hour
// cacheTTLEntity is the TTL for lookup/browse results. An artist's
// name, country and relations, a release group's title and date:
// these change on the order of never, and a wrong one is corrected
// by the next catalog artifact rather than by an expiry.
cacheTTLEntity = 365 * 24 * time.Hour
// cacheTTLReleases is the TTL for a release group's releases +
// tracklists. This data is effectively immutable once published, so
// it's cached far longer than other entities: it's the local store
// that keeps an album page's cold fetch a once-per-quarter event
// rather than a weekly one.
cacheTTLReleases = 90 * 24 * time.Hour
// tracklists. Effectively immutable once published.
cacheTTLReleases = 365 * 24 * time.Hour
)
// MusicBrainzClient wraps the musicbrainzws2 library with a local
+27 -58
View File
@@ -5,7 +5,6 @@ import (
"testing"
"yellowjacket/backend/database"
"yellowjacket/backend/database/sql/sqlcgen"
)
// TestPruneStaleLocalCrossReferences verifies that an explore_index row
@@ -21,10 +20,17 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, slog.Default())
// A library artist that still exists.
artist, err := db.Queries.UpsertArtist(t.Context(), "Still Owned")
// A library artist that still exists - which now means one with a
// file behind it. An artist row on its own is not ownership; that
// was the whole bug.
database.InsertTestTrack(t, db, database.TestTrack{
FilePath: "/music/still-owned.mp3",
Artist: "Still Owned",
})
artist, err := db.Queries.GetArtistByName(t.Context(), "Still Owned")
if err != nil {
t.Fatalf("upsert artist: %v", err)
t.Fatalf("read seeded artist: %v", err)
}
// Two explore_index artist rows: one pointing at the still-existing
@@ -33,19 +39,15 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
seedArtist := func(mbid, title string, localID int64) {
t.Helper()
if _, err := db.ExecContext(
upsertIndexSQL,
"artist", mbid, title, title, mbid, "",
0, 0,
0, "", "",
"", "", "",
"", "", "", "",
1, 0,
localID, 0, 0,
0,
); err != nil {
t.Fatalf("seed explore_index row for %q: %v", mbid, err)
}
seedIndexResult(t, db, SearchIndexResult{
EntityType: EntityArtist,
MBID: testMBID(mbid),
Title: title,
ArtistName: title,
ArtistMBID: testMBID(mbid),
InLibrary: true,
LocalArtistID: localID,
})
}
seedArtist("still-owned-mbid", "Still Owned", artist.ID)
@@ -53,7 +55,7 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
si.pruneStaleLocalCrossReferences()
stillOwned := si.LookupArtistByMBID("still-owned-mbid")
stillOwned := si.LookupArtistByMBID(testMBID("still-owned-mbid"))
if stillOwned == nil {
t.Fatal("expected still-owned artist row to survive pruning")
}
@@ -67,7 +69,7 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
)
}
removed := si.LookupArtistByMBID("removed-mbid")
removed := si.LookupArtistByMBID(testMBID("removed-mbid"))
if removed == nil {
t.Fatal("expected removed-artist row to still exist (only cross-references cleared)")
}
@@ -90,51 +92,18 @@ func TestUnenrichedLibraryArtistMBIDs_OrdersByOwnedTrackCount(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, slog.Default())
q := db.Queries
ctx := t.Context()
seedArtistWithTracks := func(name, mbid string, trackCount int) {
t.Helper()
artist, err := q.UpsertArtist(ctx, name)
if err != nil {
t.Fatalf("upsert artist %q: %v", name, err)
}
_, err = db.ExecContext("UPDATE artists SET mbid = ? WHERE id = ?", mbid, artist.ID)
if err != nil {
t.Fatalf("set mbid for %q: %v", name, err)
}
ac, err := q.UpsertArtistCredit(ctx, name)
if err != nil {
t.Fatalf("upsert artist credit %q: %v", name, err)
}
if _, err := q.CreateArtistCreditArtist(ctx, sqlcgen.CreateArtistCreditArtistParams{
ArtistID: artist.ID,
CreditID: ac.ID,
}); err != nil {
t.Fatalf("link artist credit artist %q: %v", name, err)
}
for i := range trackCount {
rec, err := q.CreateRecordingFull(ctx, sqlcgen.CreateRecordingFullParams{
Name: name,
ArtistCreditID: ac.ID,
database.InsertTestTrack(t, db, database.TestTrack{
FilePath: name + "/" + string(rune('a'+i)) + ".mp3",
Title: name,
Artist: name,
ArtistMBID: mbid,
LengthMs: 180000,
})
if err != nil {
t.Fatalf("create recording for %q: %v", name, err)
}
if _, err := q.CreateAudioFile(ctx, sqlcgen.CreateAudioFileParams{
FilePath: name + "/" + string(rune('a'+i)) + ".mp3",
LengthMilliseconds: 180000,
RecordingID: rec.ID,
Basename: string(rune('a'+i)) + ".mp3",
}); err != nil {
t.Fatalf("create audio file for %q: %v", name, err)
}
}
}
+312 -258
View File
@@ -2,6 +2,7 @@ package explore
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
@@ -146,6 +147,15 @@ type SearchIndexResult struct {
SecondaryTypes string `json:"secondaryTypes"` // comma-separated
ReleaseDate string `json:"releaseDate"`
// TotalTracks is the canonical release's track count, or 0 for
// "the catalog does not say". It is the denominator the album
// page cannot get from the files when the library holds no tags
// for the album, and it is deliberately only a denominator: the
// tracklist itself is not shipped, because the per-artist track
// budget truncates it and a truncated tracklist is a confident
// lie about which tracks exist.
TotalTracks int `json:"totalTracks"`
// Artist-specific fields (from MB lookup).
ArtistType string `json:"artistType"`
Country string `json:"country"`
@@ -357,8 +367,9 @@ func (si *SearchIndex) EnsureArtistDiscography(ctx context.Context, artistMBID s
// discography fetched, so EnsureArtistDiscography can skip the network.
func (si *SearchIndex) artistDiscogFetched(mbid string) bool {
rows, err := si.db.QueryContext(
"SELECT 1 FROM explore_index WHERE entity_type = 'artist' AND mbid = ? AND discog_fetched = 1 LIMIT 1",
mbid,
"SELECT 1 FROM explore_index WHERE entity_type = 1 /* artist */ "+
"AND mbid = ? AND discog_fetched = 1 LIMIT 1",
dbMBID(mbid),
)
if err != nil {
return false
@@ -393,11 +404,9 @@ func (si *SearchIndex) unenrichedLibraryArtistMBIDs(limit int) []string {
SELECT a.mbid
FROM artists a
LEFT JOIN explore_index ei
ON ei.entity_type = 'artist' AND ei.mbid = a.mbid
ON ei.entity_type = 1 /* artist */ AND ei.mbid = unhex(replace(a.mbid, '-', ''))
LEFT JOIN artist_enrichment ae ON ae.artist_mbid = a.mbid
LEFT JOIN artist_credit_artist aca ON aca.artist_id = a.id
LEFT JOIN recordings r ON r.artist_credit_id = aca.credit_id
LEFT JOIN audio_files af ON af.recording_id = r.id
LEFT JOIN audio_files af ON af.artist_id = a.id
WHERE a.mbid IS NOT NULL AND a.mbid != ''
AND (ei.id IS NULL OR ei.discog_fetched = 0
OR ae.browsed_at IS NULL)
@@ -560,11 +569,17 @@ func (si *SearchIndex) backfillOneArtist(
// preferring the index title, then the local library, then the MBID
// itself. Used to seed the discography fetch's artist entry.
func (si *SearchIndex) artistDisplayName(mbid string) string {
for _, q := range []string{
"SELECT title FROM explore_index WHERE entity_type = 'artist' AND mbid = ? AND title != '' LIMIT 1",
"SELECT name FROM artists WHERE mbid = ? AND name != '' LIMIT 1",
// The two tables spell an MBID differently: the catalog stores raw
// bytes, the library stores text. Each query brings its own form.
for _, q := range []struct {
sql string
arg any
}{
{"SELECT title FROM explore_index WHERE entity_type = 1 /* artist */ " +
"AND mbid = ? AND title != '' LIMIT 1", dbMBID(mbid)},
{"SELECT name FROM artists WHERE mbid = ? AND name != '' LIMIT 1", mbid},
} {
rows, err := si.db.QueryContext(q, mbid)
rows, err := si.db.QueryContext(q.sql, q.arg)
if err != nil {
continue
}
@@ -721,17 +736,17 @@ func (si *SearchIndex) refreshStatusCounts() {
for rows.Next() {
var (
et string
et dbEntityType
count int
)
if err := rows.Scan(&et, &count); err == nil {
switch et {
case "artist":
switch string(et) {
case EntityArtist:
artists = count
case "recording":
case EntityRecording:
recordings = count
case "release_group":
case EntityReleaseGroup:
rgs = count
}
}
@@ -878,7 +893,7 @@ func (si *SearchIndex) GetPopularity(mbid string) int {
rows, err := si.db.QueryContext(
"SELECT popularity FROM explore_index WHERE mbid = ? LIMIT 1",
mbid,
dbMBID(mbid),
)
if err != nil {
return 0
@@ -917,7 +932,7 @@ func (si *SearchIndex) GetPopularityBatch(mbids []string) *PopularityBatchResult
for i, m := range mbids {
placeholders[i] = "?"
args[i] = m
args[i] = dbMBID(m)
}
query := "SELECT mbid, popularity, listener_count, in_library FROM explore_index WHERE mbid IN (" +
@@ -942,13 +957,15 @@ func (si *SearchIndex) GetPopularityBatch(mbids []string) *PopularityBatchResult
for rows.Next() {
var (
mbid string
id dbMBID
pop int
listeners int
inLib int
)
if err := rows.Scan(&mbid, &pop, &listeners, &inLib); err == nil {
if err := rows.Scan(&id, &pop, &listeners, &inLib); err == nil {
mbid := string(id)
existing, ok := result.Popularity[mbid]
if !ok || pop > existing {
result.Popularity[mbid] = pop
@@ -979,7 +996,7 @@ func (si *SearchIndex) IsInLibrary(mbid string) bool {
rows, err := si.db.QueryContext(
"SELECT in_library FROM explore_index WHERE mbid = ? AND in_library = 1 LIMIT 1",
mbid,
dbMBID(mbid),
)
if err != nil {
return false
@@ -999,8 +1016,8 @@ func (si *SearchIndex) LookupArtistByMBID(mbid string) *SearchIndexResult {
artist_type, country, disambiguation, sort_name, aliases,
in_library, is_similar, COALESCE(local_artist_id, 0)
FROM explore_index
WHERE mbid = ? AND entity_type = 'artist' LIMIT 1`,
mbid,
WHERE mbid = ? AND entity_type = 1 /* artist */ LIMIT 1`,
dbMBID(mbid),
)
if err != nil {
return nil
@@ -1013,18 +1030,22 @@ func (si *SearchIndex) LookupArtistByMBID(mbid string) *SearchIndexResult {
}
r := SearchIndexResult{
EntityType: "artist",
EntityType: EntityArtist,
MBID: mbid,
}
var artist dbMBID
if err := rows.Scan(
&r.Title, &r.ArtistName, &r.ArtistMBID, &r.Popularity, &r.ListenerCount,
&r.Title, &r.ArtistName, &artist, &r.Popularity, &r.ListenerCount,
&r.ArtistType, &r.Country, &r.Disambiguation, &r.SortName, &r.Aliases,
&r.InLibrary, &r.IsSimilar, &r.LocalArtistID,
); err != nil {
return nil
}
r.ArtistMBID = string(artist)
return &r
}
@@ -1070,13 +1091,13 @@ func (si *SearchIndex) ReleaseGroupMBIDsForCAAReleaseMBIDs(
for i, m := range filtered {
placeholders[i] = "?"
args[i] = m
args[i] = dbMBID(m)
}
query := `SELECT caa_release_mbid, mbid
FROM explore_index
WHERE entity_type = 'release_group'
AND caa_release_mbid != ''
WHERE entity_type = 2 /* release_group */
AND caa_release_mbid != x''
AND caa_release_mbid IN (` + strings.Join(placeholders, ",") + `)`
rows, err := si.db.QueryContext(query, args...)
@@ -1089,9 +1110,9 @@ func (si *SearchIndex) ReleaseGroupMBIDsForCAAReleaseMBIDs(
out := make(map[string]string, len(filtered))
for rows.Next() {
var caaMBID, rgMBID string
var caaMBID, rgMBID dbMBID
if err := rows.Scan(&caaMBID, &rgMBID); err == nil {
out[caaMBID] = rgMBID
out[string(caaMBID)] = string(rgMBID)
}
}
@@ -1102,11 +1123,11 @@ func (si *SearchIndex) ReleaseGroupMBIDsForCAAReleaseMBIDs(
func (si *SearchIndex) LookupReleaseGroupByMBID(mbid string) *SearchIndexResult {
rows, err := si.db.QueryContext(
`SELECT title, artist_name, artist_mbid, popularity, listener_count,
primary_type, secondary_types, release_date,
primary_type, secondary_types, release_date, total_tracks,
in_library, COALESCE(local_release_group_id, 0), discog_fetched
FROM explore_index
WHERE mbid = ? AND entity_type = 'release_group' LIMIT 1`,
mbid,
WHERE mbid = ? AND entity_type = 2 /* release_group */ LIMIT 1`,
dbMBID(mbid),
)
if err != nil {
return nil
@@ -1119,13 +1140,15 @@ func (si *SearchIndex) LookupReleaseGroupByMBID(mbid string) *SearchIndexResult
}
r := SearchIndexResult{
EntityType: "release_group",
EntityType: EntityReleaseGroup,
MBID: mbid,
}
var artist dbMBID
if err := rows.Scan(
&r.Title, &r.ArtistName, &r.ArtistMBID, &r.Popularity, &r.ListenerCount,
&r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate,
&r.Title, &r.ArtistName, &artist, &r.Popularity, &r.ListenerCount,
&r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate, &r.TotalTracks,
&r.InLibrary, &r.LocalReleaseGroupID, &r.DiscogFetched,
); err != nil {
return nil
@@ -1166,10 +1189,10 @@ func (si *SearchIndex) TopRecordingsByArtist(artistMBID string, limit int) []Sea
duration, caa_release_mbid, release_name,
in_library, COALESCE(local_recording_id, 0)
FROM explore_index
WHERE artist_mbid = ? AND entity_type = 'recording'
WHERE artist_mbid = ? AND entity_type = 3 /* recording */
ORDER BY popularity DESC
LIMIT ?`,
artistMBID, limit,
dbMBID(artistMBID), limit,
)
if err != nil {
return nil
@@ -1180,13 +1203,19 @@ func (si *SearchIndex) TopRecordingsByArtist(artistMBID string, limit int) []Sea
var results []SearchIndexResult
for rows.Next() {
var r SearchIndexResult
var (
r SearchIndexResult
id, caa dbMBID
)
if err := rows.Scan(
&r.MBID, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
&r.Duration, &r.CAAReleaseMBID, &r.ReleaseName,
&id, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
&r.Duration, &caa, &r.ReleaseName,
&r.InLibrary, &r.LocalRecordingID,
); err == nil {
r.EntityType = "recording"
r.MBID = string(id)
r.CAAReleaseMBID = string(caa)
r.EntityType = EntityRecording
r.ArtistMBID = artistMBID
results = append(results, r)
}
@@ -1205,10 +1234,10 @@ func (si *SearchIndex) TopReleaseGroupsByArtist(artistMBID string, limit int) []
primary_type, secondary_types, release_date,
in_library, COALESCE(local_release_group_id, 0)
FROM explore_index
WHERE artist_mbid = ? AND entity_type = 'release_group'
WHERE artist_mbid = ? AND entity_type = 2 /* release_group */
ORDER BY popularity DESC
LIMIT ?`,
artistMBID, limit,
dbMBID(artistMBID), limit,
)
if err != nil {
return nil
@@ -1219,13 +1248,18 @@ func (si *SearchIndex) TopReleaseGroupsByArtist(artistMBID string, limit int) []
var results []SearchIndexResult
for rows.Next() {
var r SearchIndexResult
var (
r SearchIndexResult
id dbMBID
)
if err := rows.Scan(
&r.MBID, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
&id, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
&r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate,
&r.InLibrary, &r.LocalReleaseGroupID,
); err == nil {
r.EntityType = "release_group"
r.MBID = string(id)
r.EntityType = EntityReleaseGroup
r.ArtistMBID = artistMBID
results = append(results, r)
}
@@ -1315,29 +1349,21 @@ func (si *SearchIndex) ExactMatches(query string, perCategory int) []SearchIndex
// its partial expression index (idx_explore_title_lower /
// idx_explore_artist_lower). Ordering is done in Go below since
// an ORDER BY here would also defeat the index seek.
//
// The popularity clause **must match those indexes' predicate** or
// the seek becomes a scan of two million rows. It is the champion
// set: what the user owns, plus what is popular enough to be worth
// an exact-match boost. Anything below the floor is still found by
// the FTS tiers; it just does not jump the queue.
rows, err := si.db.QueryContext(`
SELECT entity_type, mbid, title, artist_name, artist_mbid,
popularity, listener_count, duration, primary_type,
secondary_types, release_date, caa_release_mbid,
release_name, artist_type, country, disambiguation,
sort_name, in_library, is_similar,
COALESCE(local_artist_id, 0),
COALESCE(local_release_group_id, 0),
COALESCE(local_recording_id, 0)
SELECT `+indexRowColumns+`
FROM explore_index
WHERE LOWER(title) = ? AND popularity > 0
WHERE LOWER(title) = ? AND (popularity >= ? OR in_library = 1)
UNION
SELECT entity_type, mbid, title, artist_name, artist_mbid,
popularity, listener_count, duration, primary_type,
secondary_types, release_date, caa_release_mbid,
release_name, artist_type, country, disambiguation,
sort_name, in_library, is_similar,
COALESCE(local_artist_id, 0),
COALESCE(local_release_group_id, 0),
COALESCE(local_recording_id, 0)
SELECT `+indexRowColumns+`
FROM explore_index
WHERE LOWER(artist_name) = ? AND popularity > 0
`, q, q)
WHERE LOWER(artist_name) = ? AND (popularity >= ? OR in_library = 1)
`, q, championPopThreshold, q, championPopThreshold)
if err != nil {
return nil
}
@@ -1349,14 +1375,7 @@ func (si *SearchIndex) ExactMatches(query string, perCategory int) []SearchIndex
for rows.Next() {
var r SearchIndexResult
if err := rows.Scan(
&r.EntityType, &r.MBID, &r.Title, &r.ArtistName, &r.ArtistMBID,
&r.Popularity, &r.ListenerCount, &r.Duration, &r.PrimaryType,
&r.SecondaryTypes, &r.ReleaseDate, &r.CAAReleaseMBID,
&r.ReleaseName, &r.ArtistType, &r.Country, &r.Disambiguation,
&r.SortName, &r.InLibrary, &r.IsSimilar,
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
); err != nil {
if err := scanIndexRow(rows, &r); err != nil {
continue
}
@@ -1623,14 +1642,7 @@ func (si *SearchIndex) rowsByIDs(ctx context.Context, ids []int64) []SearchIndex
}
query := `
SELECT id, entity_type, mbid, title, artist_name, artist_mbid,
popularity, listener_count, duration, primary_type,
secondary_types, release_date, caa_release_mbid,
release_name, artist_type, country, disambiguation,
sort_name, in_library, is_similar,
COALESCE(local_artist_id, 0),
COALESCE(local_release_group_id, 0),
COALESCE(local_recording_id, 0)
SELECT id, ` + indexRowColumns + `
FROM explore_index
WHERE id IN (` + strings.Join(placeholders, ",") + `)`
@@ -1653,14 +1665,7 @@ func (si *SearchIndex) rowsByIDs(ctx context.Context, ids []int64) []SearchIndex
r SearchIndexResult
)
if err := rows.Scan(
&id, &r.EntityType, &r.MBID, &r.Title, &r.ArtistName, &r.ArtistMBID,
&r.Popularity, &r.ListenerCount, &r.Duration, &r.PrimaryType,
&r.SecondaryTypes, &r.ReleaseDate, &r.CAAReleaseMBID,
&r.ReleaseName, &r.ArtistType, &r.Country, &r.Disambiguation,
&r.SortName, &r.InLibrary, &r.IsSimilar,
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
); err != nil {
if err := scanIndexRow(rows, &r, &id); err != nil {
continue
}
@@ -1742,15 +1747,7 @@ func (si *SearchIndex) queryFTS(
) []SearchIndexResult {
// ftsTable is a trusted in-package constant, never user input.
sqlText := fmt.Sprintf(`
SELECT i.entity_type, i.mbid, i.title, i.artist_name,
i.artist_mbid, i.popularity, i.listener_count,
i.duration, i.primary_type, i.secondary_types, i.release_date,
i.caa_release_mbid, i.release_name,
i.artist_type, i.country, i.disambiguation, i.sort_name,
i.in_library, i.is_similar,
COALESCE(i.local_artist_id, 0),
COALESCE(i.local_release_group_id, 0),
COALESCE(i.local_recording_id, 0)
SELECT `+indexRowColumnsFor("i")+`
FROM explore_index i
JOIN %[1]s f ON f.rowid = i.id
WHERE %[1]s MATCH ?
@@ -1783,15 +1780,7 @@ func (si *SearchIndex) queryFTS(
for rows.Next() {
var r SearchIndexResult
if err := rows.Scan(
&r.EntityType, &r.MBID, &r.Title, &r.ArtistName,
&r.ArtistMBID, &r.Popularity, &r.ListenerCount,
&r.Duration, &r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate,
&r.CAAReleaseMBID, &r.ReleaseName,
&r.ArtistType, &r.Country, &r.Disambiguation, &r.SortName,
&r.InLibrary, &r.IsSimilar,
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
); err != nil {
if err := scanIndexRow(rows, &r); err != nil {
si.logger.Warn("search index scan error", "error", err)
continue
@@ -2051,9 +2040,11 @@ func (si *SearchIndex) indexOneArtist(
// concurrently — they use different rate limiters so they
// don't block each other.
var (
rgs []SearchIndexResult
recs []SearchIndexResult
wg sync.WaitGroup
rgs []SearchIndexResult
recs []SearchIndexResult
rgErr error
recErr error
wg sync.WaitGroup
)
// LB pipeline: top release groups + top recordings.
@@ -2062,8 +2053,8 @@ func (si *SearchIndex) indexOneArtist(
go func() {
defer wg.Done()
rgs = si.fetchTopReleaseGroups(ctx, lb, artist, rgLimit)
recs = si.fetchTopRecordings(ctx, lb, artist, recLimit)
rgs, rgErr = si.fetchTopReleaseGroups(ctx, lb, artist, rgLimit)
recs, recErr = si.fetchTopRecordings(ctx, lb, artist, recLimit)
}()
// MB pipeline: cache the artist lookup, which is what the details
@@ -2090,14 +2081,19 @@ func (si *SearchIndex) indexOneArtist(
// Write the artist entry into the index so indexedArtistMBIDs()
// recognises this artist as processed on subsequent builds.
// Only mark DiscogFetched=true if at least one of the discography
// fetches actually returned data — a transient API failure should
// allow a retry on the next build, not permanently claim the
// artist as indexed. Also stores aliases and detail fields from
// the now-cached MB rels (populated by the image resolution above)
// for FTS search.
// Also stores aliases and detail fields from the now-cached MB rels
// (populated by the image resolution above) for FTS search.
//
// DiscogFetched records that both LB endpoints were *asked*, not
// that they had anything to say. A transient failure still leaves
// the artist unmarked so the next run retries it — but an artist LB
// has no popularity data for (or none above indexMinPopularity,
// which is most of a long-tail library) answers empty every single
// time, and keying the mark on emptiness made those artists
// permanent candidates: the owned-artist backfill re-ran for them
// on every launch, forever, which is the bug this replaces.
if si.artistImg != nil {
gotData := len(rgs) > 0 || len(recs) > 0
gotData := rgErr == nil && recErr == nil
artistEntry := SearchIndexResult{
EntityType: "artist",
MBID: artist.ArtistMBID,
@@ -2139,10 +2135,10 @@ func (si *SearchIndex) fetchTopReleaseGroups(
lb *ListenBrainzClient,
artist lbSitewideArtist,
maxCount int,
) []SearchIndexResult {
) ([]SearchIndexResult, error) {
url := fmt.Sprintf(
"%s/1/popularity/top-release-groups-for-artist/%s",
listenBrainzBaseURL, artist.ArtistMBID,
lb.baseURL, artist.ArtistMBID,
)
body, err := lb.doGet(ctx, url)
@@ -2152,7 +2148,7 @@ func (si *SearchIndex) fetchTopReleaseGroups(
"error", err,
)
return nil
return nil, err
}
var raw []struct {
@@ -2173,7 +2169,7 @@ func (si *SearchIndex) fetchTopReleaseGroups(
}
if err := json.Unmarshal(body, &raw); err != nil {
return nil
return nil, err
}
limit := maxCount
@@ -2209,7 +2205,7 @@ func (si *SearchIndex) fetchTopReleaseGroups(
})
}
return results
return results, nil
}
func (si *SearchIndex) fetchTopRecordings(
@@ -2217,20 +2213,20 @@ func (si *SearchIndex) fetchTopRecordings(
lb *ListenBrainzClient,
artist lbSitewideArtist,
maxCount int,
) []SearchIndexResult {
) ([]SearchIndexResult, error) {
url := fmt.Sprintf(
"%s/1/popularity/top-recordings-for-artist/%s",
listenBrainzBaseURL, artist.ArtistMBID,
lb.baseURL, artist.ArtistMBID,
)
body, err := lb.doGet(ctx, url)
if err != nil {
return nil
return nil, err
}
var raw []lbTopRecordingWire
if err := json.Unmarshal(body, &raw); err != nil {
return nil
return nil, err
}
limit := maxCount
@@ -2258,7 +2254,7 @@ func (si *SearchIndex) fetchTopRecordings(
})
}
return results
return results, nil
}
// ---------------------------------------------------------------------------
@@ -2276,6 +2272,89 @@ func (si *SearchIndex) fetchTopRecordings(
// empty values, and numeric fields use "highest wins" for popularity/
// listener_count/duration so older richer data survives refreshes.
// indexRowColumns is the full explore_index projection, and
// scanIndexRow is the only thing that reads it.
//
// There were four copies of this column list and four matching Scan
// calls, which is what makes the storage encoding dangerous: a blob
// column scanned into a string yields sixteen bytes of garbage rather
// than an error, and it would have had to be got right four times.
// dbMBID and dbEntityType do the decoding, and they refuse anything
// that is not what they expect.
var indexRowFields = []string{
"entity_type", "mbid", "title", "artist_name", "artist_mbid",
"popularity", "listener_count", "duration", "primary_type",
"secondary_types", "release_date", "total_tracks", "caa_release_mbid",
"release_name", "artist_type", "country", "disambiguation",
"sort_name", "in_library", "is_similar",
"local_artist_id", "local_release_group_id", "local_recording_id",
}
// nullableIndexRowFields are the ones a caller wants zero rather than
// NULL for.
var nullableIndexRowFields = map[string]bool{
"local_artist_id": true,
"local_release_group_id": true,
"local_recording_id": true,
}
// indexRowColumns is the projection unqualified; indexRowColumnsFor
// qualifies it with a table alias, for the joins where the other side
// also has a `title`.
var indexRowColumns = indexRowColumnsFor("")
func indexRowColumnsFor(alias string) string {
if alias != "" {
alias += "."
}
cols := make([]string, 0, len(indexRowFields))
for _, f := range indexRowFields {
if nullableIndexRowFields[f] {
cols = append(cols, "COALESCE("+alias+f+", 0)")
continue
}
cols = append(cols, alias+f)
}
return strings.Join(cols, ", ")
}
// scanIndexRow reads one indexRowColumns row into a result.
func scanIndexRow(rows *sql.Rows, r *SearchIndexResult, before ...any) error {
var (
entity dbEntityType
id dbMBID
artist dbMBID
caa dbMBID
)
dest := make([]any, 0, len(before)+len(indexRowFields))
dest = append(dest, before...)
dest = append(dest,
&entity, &id, &r.Title, &r.ArtistName, &artist,
&r.Popularity, &r.ListenerCount, &r.Duration, &r.PrimaryType,
&r.SecondaryTypes, &r.ReleaseDate, &r.TotalTracks, &caa,
&r.ReleaseName, &r.ArtistType, &r.Country, &r.Disambiguation,
&r.SortName, &r.InLibrary, &r.IsSimilar,
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
)
if err := rows.Scan(dest...); err != nil {
return fmt.Errorf("scan index row: %w", err)
}
r.EntityType = string(entity)
r.MBID = string(id)
r.ArtistMBID = string(artist)
r.CAAReleaseMBID = string(caa)
return nil
}
// upsertIndexSQL is the single index write statement. It is kept as
// a const so assembly can prepare it once per transaction instead of
// re-parsing this large upsert for every row.
@@ -2284,7 +2363,7 @@ const upsertIndexSQL = `
entity_type, mbid, title, artist_name, artist_mbid, aliases,
popularity, listener_count,
duration, caa_release_mbid, release_name,
primary_type, secondary_types, release_date,
primary_type, secondary_types, release_date, total_tracks,
artist_type, country, disambiguation, sort_name,
in_library, is_similar,
local_artist_id, local_release_group_id, local_recording_id,
@@ -2293,7 +2372,7 @@ const upsertIndexSQL = `
?, ?, ?, ?, ?, ?,
?, ?,
?, ?, ?,
?, ?, ?,
?, ?, ?, ?,
?, ?, ?, ?,
?, ?,
NULLIF(?, 0), NULLIF(?, 0), NULLIF(?, 0),
@@ -2311,7 +2390,7 @@ const upsertIndexConflictSQL = `
-- as a name; AddFromCache is the path that used to.
title = CASE WHEN excluded.title != '' THEN excluded.title ELSE title END,
artist_name = CASE WHEN excluded.artist_name != '' THEN excluded.artist_name ELSE artist_name END,
artist_mbid = CASE WHEN excluded.artist_mbid != '' THEN excluded.artist_mbid ELSE artist_mbid END,
artist_mbid = CASE WHEN excluded.artist_mbid != x'' THEN excluded.artist_mbid ELSE artist_mbid END,
aliases = CASE WHEN excluded.aliases != '' THEN excluded.aliases ELSE aliases END,
-- Highest wins for popularity + listener_count (refreshes can go up).
@@ -2320,11 +2399,12 @@ const upsertIndexConflictSQL = `
-- Non-empty wins for all other optional fields (never clobber with empty).
duration = CASE WHEN excluded.duration > 0 THEN excluded.duration ELSE duration END,
caa_release_mbid = CASE WHEN excluded.caa_release_mbid != '' THEN excluded.caa_release_mbid ELSE caa_release_mbid END,
caa_release_mbid = CASE WHEN excluded.caa_release_mbid != x'' THEN excluded.caa_release_mbid ELSE caa_release_mbid END,
release_name = CASE WHEN excluded.release_name != '' THEN excluded.release_name ELSE release_name END,
primary_type = CASE WHEN excluded.primary_type != '' THEN excluded.primary_type ELSE primary_type END,
secondary_types = CASE WHEN excluded.secondary_types != '' THEN excluded.secondary_types ELSE secondary_types END,
release_date = CASE WHEN excluded.release_date != '' THEN excluded.release_date ELSE release_date END,
total_tracks = CASE WHEN excluded.total_tracks > 0 THEN excluded.total_tracks ELSE total_tracks END,
artist_type = CASE WHEN excluded.artist_type != '' THEN excluded.artist_type ELSE artist_type END,
country = CASE WHEN excluded.country != '' THEN excluded.country ELSE country END,
disambiguation = CASE WHEN excluded.disambiguation != '' THEN excluded.disambiguation ELSE disambiguation END,
@@ -2387,10 +2467,11 @@ func (si *SearchIndex) upsertBatch(entries []SearchIndexResult) {
}
if _, err := stmt.Exec(
e.EntityType, e.MBID, e.Title, e.ArtistName, e.ArtistMBID, e.Aliases,
dbEntityType(e.EntityType), dbMBID(e.MBID),
e.Title, e.ArtistName, dbMBID(e.ArtistMBID), e.Aliases,
e.Popularity, e.ListenerCount,
e.Duration, e.CAAReleaseMBID, e.ReleaseName,
e.PrimaryType, e.SecondaryTypes, e.ReleaseDate,
e.Duration, dbMBID(e.CAAReleaseMBID), e.ReleaseName,
e.PrimaryType, e.SecondaryTypes, e.ReleaseDate, e.TotalTracks,
e.ArtistType, e.Country, e.Disambiguation, e.SortName,
inLib, isSim,
e.LocalArtistID, e.LocalReleaseGroupID, e.LocalRecordingID,
@@ -2480,42 +2561,48 @@ func (si *SearchIndex) pruneStaleLocalCrossReferences() {
type prune struct {
entityType string
column string
table string
// exists is the test for "this local id still refers to
// something the user owns". It is a file test in every case -
// the version that tested the metadata table left 129 rows in
// a real catalog claiming to be owned by files that were gone.
exists string
}
for _, p := range []prune{
{"artist", "local_artist_id", "artists"},
{"release_group", "local_release_group_id", "release_groups"},
{"recording", "local_recording_id", "recordings"},
{"artist", "local_artist_id", `
SELECT 1 FROM artists a WHERE a.id = explore_index.local_artist_id
AND (
EXISTS (SELECT 1 FROM audio_files af WHERE af.artist_id = a.id)
OR EXISTS (
SELECT 1 FROM albums al
JOIN audio_files af2 ON af2.album_id = al.id
WHERE al.artist_id = a.id
)
)`},
{"release_group", "local_release_group_id", `
SELECT 1 FROM audio_files af
WHERE af.album_id = explore_index.local_release_group_id`},
{"recording", "local_recording_id", `
SELECT 1 FROM audio_files af
WHERE af.id = explore_index.local_recording_id`},
} {
result, err := si.db.ExecContext(
`UPDATE explore_index
SET in_library = 0, `+p.column+` = NULL
WHERE entity_type = ?
AND `+p.column+` IS NOT NULL
AND `+p.column+` NOT IN (SELECT id FROM `+p.table+`)`,
p.entityType,
WHERE entity_type = ? AND `+p.column+` IS NOT NULL
AND NOT EXISTS (`+p.exists+`)`,
dbEntityType(p.entityType),
)
if err != nil {
si.logger.Warn(
"library sync: prune stale cross-references failed",
"entityType",
p.entityType,
"error",
err,
)
si.logger.Warn("library sync: prune stale cross-references failed",
"entityType", p.entityType, "error", err)
continue
}
if n, err := result.RowsAffected(); err == nil && n > 0 {
si.logger.Info(
"library sync: cleared stale cross-references",
"entityType",
p.entityType,
"count",
n,
)
if n, _ := result.RowsAffected(); n > 0 {
si.logger.Info("library sync: cleared stale cross-references",
"entityType", p.entityType, "count", n)
}
}
}
@@ -2527,118 +2614,85 @@ func (si *SearchIndex) pruneStaleLocalCrossReferences() {
func (si *SearchIndex) collectLibraryEntities() []SearchIndexResult {
var entries []SearchIndexResult
// Artists.
artistRows, err := si.db.QueryContext(
"SELECT id, name, mbid FROM artists WHERE mbid IS NOT NULL AND mbid != ''",
)
if err == nil {
for artistRows.Next() {
var (
id int64
name, mbid string
)
if err := artistRows.Scan(&id, &name, &mbid); err != nil {
continue
}
entries = append(entries, SearchIndexResult{
EntityType: "artist",
MBID: mbid,
Title: name,
ArtistName: name,
ArtistMBID: mbid,
InLibrary: true,
LocalArtistID: id,
})
}
_ = artistRows.Close()
} else {
si.logger.Warn("library sync: query artists failed", "error", err)
// Every one of these is gated on a file existing. They used to
// select straight from the metadata tables, so an artist, album or
// recording whose files were gone stayed flagged "in library" in
// the catalog until something noticed - and nothing did.
type entityQuery struct {
kind string
query string
}
// Release groups. The correlated subquery picks the primary
// credited artist's MBID (if it has one).
rgRows, err := si.db.QueryContext(`
SELECT rg.id, rg.name, rg.mbid, COALESCE(ac.text, ''),
COALESCE((
SELECT a.mbid FROM artist_credit_artist aca
JOIN artists a ON a.id = aca.artist_id
WHERE aca.credit_id = rg.album_artist_credit_id
AND a.mbid IS NOT NULL AND a.mbid != ''
LIMIT 1
), '')
FROM release_groups rg
LEFT JOIN artist_credit ac ON ac.id = rg.album_artist_credit_id
WHERE rg.mbid IS NOT NULL AND rg.mbid != ''
`)
if err == nil {
for rgRows.Next() {
queries := []entityQuery{
{"artist", `
SELECT DISTINCT a.id, a.name, a.mbid, a.name, a.mbid
FROM artists a
WHERE a.mbid IS NOT NULL AND a.mbid != ''
AND (
EXISTS (SELECT 1 FROM audio_files af WHERE af.artist_id = a.id)
OR EXISTS (
SELECT 1 FROM albums al
JOIN audio_files af2 ON af2.album_id = al.id
WHERE al.artist_id = a.id
)
)`},
{"release_group", `
SELECT DISTINCT al.id, al.name, al.mbid, al.artist_credit,
COALESCE(ar.mbid, '')
FROM albums al
JOIN audio_files af ON af.album_id = al.id
LEFT JOIN artists ar ON ar.id = al.artist_id
WHERE al.mbid IS NOT NULL AND al.mbid != ''`},
{"recording", `
SELECT af.id, af.title, af.recording_mbid, af.artist_credit,
COALESCE(ar.mbid, '')
FROM audio_files af
LEFT JOIN artists ar ON ar.id = af.artist_id
WHERE af.recording_mbid IS NOT NULL AND af.recording_mbid != ''`},
}
for _, eq := range queries {
rows, err := si.db.QueryContext(eq.query)
if err != nil {
si.logger.Warn("library sync: query failed", "kind", eq.kind, "error", err)
continue
}
for rows.Next() {
var (
id int64
name, mbid, credit, artistMB string
)
if err := rgRows.Scan(&id, &name, &mbid, &credit, &artistMB); err != nil {
if err := rows.Scan(&id, &name, &mbid, &credit, &artistMB); err != nil {
continue
}
entries = append(entries, SearchIndexResult{
EntityType: "release_group",
MBID: mbid,
Title: name,
ArtistName: credit,
ArtistMBID: artistMB,
InLibrary: true,
LocalReleaseGroupID: id,
})
}
_ = rgRows.Close()
} else {
si.logger.Warn("library sync: query release groups failed", "error", err)
}
// Recordings.
recRows, err := si.db.QueryContext(`
SELECT r.id, r.name, r.mbid, COALESCE(ac.text, ''),
COALESCE((
SELECT a.mbid FROM artist_credit_artist aca
JOIN artists a ON a.id = aca.artist_id
WHERE aca.credit_id = r.artist_credit_id
AND a.mbid IS NOT NULL AND a.mbid != ''
LIMIT 1
), '')
FROM recordings r
LEFT JOIN artist_credit ac ON ac.id = r.artist_credit_id
WHERE r.mbid IS NOT NULL AND r.mbid != ''
`)
if err == nil {
for recRows.Next() {
var (
id int64
name, mbid, credit, artistMB string
)
if err := recRows.Scan(&id, &name, &mbid, &credit, &artistMB); err != nil {
continue
entry := SearchIndexResult{
EntityType: eq.kind,
MBID: mbid,
Title: name,
ArtistName: credit,
ArtistMBID: artistMB,
InLibrary: true,
}
entries = append(entries, SearchIndexResult{
EntityType: "recording",
MBID: mbid,
Title: name,
ArtistName: credit,
ArtistMBID: artistMB,
InLibrary: true,
LocalRecordingID: id,
})
switch eq.kind {
case "artist":
entry.LocalArtistID = id
case "release_group":
entry.LocalReleaseGroupID = id
case "recording":
// The local id of a "recording" is the file's, which is
// what every caller wants: it is the thing that plays.
entry.LocalRecordingID = id
}
entries = append(entries, entry)
}
_ = recRows.Close()
} else {
si.logger.Warn("library sync: query recordings failed", "error", err)
_ = rows.Close()
}
return entries
@@ -2710,7 +2764,7 @@ func (si *SearchIndex) BackfillPopularity(updates map[string]PopularityData) {
WHERE mbid = ?`,
data.ListenCount, data.ListenCount,
data.ListenerCount, data.ListenerCount,
mbid,
dbMBID(mbid),
)
}
+6 -6
View File
@@ -332,7 +332,7 @@ func (si *SearchIndex) topByPopularity(
// The artist rows *are* the artists, so they partition by their own
// mbid; release groups partition by whoever made them.
partition := "artist_mbid"
if entityType == "artist" {
if entityType == EntityArtist {
partition = "mbid"
}
@@ -350,14 +350,14 @@ func (si *SearchIndex) topByPopularity(
WHERE rank = 1
ORDER BY popularity DESC
LIMIT ?`,
entityType, limit+len(skip),
dbEntityType(entityType), limit+len(skip),
))
out := make([]SearchIndexResult, 0, limit)
for _, row := range rows {
key := row.ArtistMBID
if entityType == "artist" {
if entityType == EntityArtist {
key = row.MBID
}
@@ -393,13 +393,13 @@ func (si *SearchIndex) unownedAlbumsBySinglyOwnedArtists(
return si.rowsByIDs(ctx, si.shelfIDs(
ctx,
`SELECT id FROM explore_index
WHERE entity_type = 'release_group'
WHERE entity_type = 2 /* release_group */
AND in_library = 0
AND artist_mbid IN (
SELECT artist_mbid FROM explore_index
WHERE entity_type = 'release_group'
WHERE entity_type = 2 /* release_group */
AND in_library = 1
AND artist_mbid != ''
AND artist_mbid != x''
GROUP BY artist_mbid
HAVING COUNT(*) = 1
ORDER BY MAX(popularity) DESC
+13 -20
View File
@@ -28,24 +28,17 @@ func seedShelfRow(
) {
t.Helper()
owned := 0
if inLibrary {
owned = 1
}
if _, err := db.ExecContext(
upsertIndexSQL,
entityType, mbid, title, artistName, artistMBID, "",
popularity, popularity,
0, "", "",
"Album", "", "",
"", "", "", "",
owned, 0,
0, 0, 0,
0,
); err != nil {
t.Fatalf("seed explore_index row %q: %v", mbid, err)
}
seedIndexResult(t, db, SearchIndexResult{
EntityType: entityType,
MBID: testMBID(mbid),
Title: title,
ArtistName: artistName,
ArtistMBID: testMBID(artistMBID),
Popularity: popularity,
ListenerCount: popularity,
ReleaseName: "Album",
InLibrary: inLibrary,
})
}
func newShelfService(t *testing.T) (*Service, *database.DB) {
@@ -271,12 +264,12 @@ func TestShelves_TheSecondRowIsNotTheFirstRowsArtists(t *testing.T) {
t.Fatal("no popular-artists shelf")
}
if albums.Albums[0].ArtistMBID != "ar-huge" {
if albums.Albums[0].ArtistMBID != testMBID("ar-huge") {
t.Fatalf("albums shelf leads with %q, want ar-huge", albums.Albums[0].ArtistMBID)
}
for _, artist := range artists.Artists {
if artist.MBID == "ar-huge" {
if artist.MBID == testMBID("ar-huge") {
t.Fatal("artists shelf repeats the artist the albums shelf just showed")
}
}
+5
View File
@@ -17,6 +17,11 @@ const (
recA = "11111111-1111-1111-1111-111111111111"
recB = "22222222-2222-2222-2222-222222222222"
recC = "33333333-3333-3333-3333-333333333333"
// recD is on relA and nobody has ever played it, which is the point:
// it must count toward relA's track total without being indexed as a
// recording itself.
recD = "44444444-4444-4444-4444-444444444444"
relA = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
relB = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
rgA = "cccccccc-cccc-cccc-cccc-cccccccccccc"
+8
View File
@@ -77,6 +77,14 @@ type MBReleaseGroup struct {
ListenerCount int `json:"listenerCount"`
InLibrary bool `json:"inLibrary"` // true if the user owns this album
LocalID int64 `json:"localId,omitempty"` // local release_group row ID
// TotalTracks is the catalog's track count for this release group,
// or 0 for "the catalog does not say". It answers "how much of
// this album do I have" for an album whose files declared no total
// -- the case GetAlbumCompleteness cannot answer -- and it is not
// filled by the MusicBrainz path below, which has the real
// tracklist and does not need a denominator.
TotalTracks int `json:"totalTracks"`
}
// MBRelease is a Wails-friendly projection of a MusicBrainz release.