feat(database): shape the library like files, and shrink the catalog
Plans 013 and 014, the album page that prompted them, and the smaller fixes they turned up. Changelog, largest first. ## The local library is shaped like files, not like MusicBrainz `audio_files` carries its own tags and points at `albums` and `artists`; `file_genres` is the one real many-to-many. `recordings`, `release_group_recordings`, `artist_credit`, `artist_credit_artist`, `recording_genres`, `release_groups` and `release_to_rg` are gone from the local side, and with them a six-way join in every read, a `MIN(release_group_id)` subquery in eleven queries and a first-credited-artist subquery in nine. Measured on a real 25,966-file library, every many-to-many that model expressed was 1:1 in the data. - Ownership is a file. `GetFilePathsByRecordingMBIDs`, `LibraryMBIDIndex.CheckMBIDs`, `collectLibraryEntities` and `pruneStaleLocalCrossReferences` all join `audio_files`, so the 812 orphaned recordings, 216 release groups and 260 artists that library carried are now structurally impossible. - One projection: every track query selects from the `track_metadata` view, one row type, one mapper. Nine hand-rolled copies had drifted far enough to report different years on different screens. - `library_id = 0` means every library, so each list query exists once instead of scoped and unscoped with a branch at every call site. - No migration chain. `sql/schemas/` is the one description of the shape; `sql/migrations/`, `applyMigrations` and `schema_migrations` are squashed away, along with the drift between them that had sqlc generating against a stale schema. - `database.InsertTestTrack` is the one test seeder; twenty test files had been assembling the old FK chain each in its own order. ## The catalog stores its ids as bytes `explore_index`'s three 36-char MBID columns and its entity-type text are 16 raw bytes and a small integer. The table and its six indexes go 780 MB to 405 MB on a real 2,052,200-row catalog, which is why a fresh install is ~0.6 GB rather than ~1.0 GB. - `backend/explore/mbid.go` is the only place the encoding is known; everything above it speaks dashed strings. - `CHECK(length(mbid) = 16)` makes a stringly write fail at the insert rather than silently returning no rows, since SQLite does not coerce between TEXT and BLOB. - The importer asks the artifact what encoding it carries and converts on the way in, so the artifact already published keeps working and no format bump is needed. - `indexRowColumns`/`scanIndexRow` replace four copies of a 22-column list, and `TestStoredEncodingRoundTrips` sweeps every read path. ## An album page that says how much of the album is yours - One question, asked once: is there a file. `filePaths` is filled by a single batched lookup when the tracklist settles, and the badge, the Play count, the dimmed rows and every menu item read it — replacing four claims of decreasing confidence that could show a green tick on an album whose every action did nothing. - Play, Play 7 of 12, or no play button at all. - `total_tracks` on `explore_index` (~2 bytes over 400,677 release groups) and on `audio_files` from tags that have always carried it: a complete MBID-matched album now makes no catalog call at all, where it used to spend the most expensive request the app makes. - A merged cluster shows the running order the most releases agree on, and the version list marks the release you own rather than standing a synthetic entry in for it. - `AlbumReleasesFailed`: a slow fetch is no longer reported as a failed one by a 12-second timer. - Rows not in the library are dimmed in place (with `aria-disabled`) instead of the owned ones wearing a green tick and a legend. ## Caches and cover art get ceilings - Only the three tiers of a cover are stored; the full-resolution copy nothing rendered was 1,134 MB of a 1.4 GB covers directory. - One artist portrait is downloaded and the rest are remembered as URLs — 4.1 GB of a 5.3 GB cache was candidates no code path reads. - `browsedArtBudget` and `httpCacheBudget` bound what an age cannot: the same install held art for 5,770 artists in a 1,301-artist library. - `OrphanedArtistImagesJob` joined a bare MBID onto a sharded directory, so it deleted the rows that were the only record of the files it left behind. `explore.ArtistImageDir` is that layout's one definition now. ## The autotag queue asks whether there is work `tagging_items` was a row per album folder, not a queue, and no query read the `tag_status` column that held the answer. The four queue queries ask the files, which matters most where it is least visible: `startPrefetch` was scoring every album in a tagged library against MusicBrainz. ## Phantom playlist tracks resolve in place An M3U8 imported before its files leaves phantom rows; they now match by path and fall back to position, keep their place in the playlist when resolved, and pair best-first so two phantoms cannot claim the same file. ## Playing a track plays the list it is in Double-click, and Play on a single row's menu, queue the list as displayed with `startIndex` on that row — the album page and the track list used to queue one track and discard the album around it. A multi-row selection still plays exactly itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
This commit is contained in:
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
_ "modernc.org/sqlite" // SQLite driver for reading the artifact file.
|
||||
@@ -91,9 +92,90 @@ type artifactInfo struct {
|
||||
// TestArtifactColumnsMatchExporter.
|
||||
const artifactCatalogColumns = `entity_type, mbid, title, artist_name, artist_mbid,
|
||||
aliases, popularity, listener_count, duration, caa_release_mbid,
|
||||
release_name, primary_type, secondary_types, release_date,
|
||||
release_name, primary_type, secondary_types, release_date, total_tracks,
|
||||
artist_type, country, disambiguation, sort_name, discog_fetched`
|
||||
|
||||
// artifactSelectColumns is the same list as read *from an artifact*,
|
||||
// converting the two columns whose storage this app changed.
|
||||
//
|
||||
// The local table stores an MBID as 16 raw bytes and an entity type as a
|
||||
// small integer, which took the catalog and its indexes from 677 MB to
|
||||
// 389 MB. A published artifact still carries the text form, and there
|
||||
// is no reason it should not: converting on the way in costs one
|
||||
// `unhex` per row on a once-a-month import, and it means a new build
|
||||
// reads the artifact that is already out there rather than requiring
|
||||
// one to be rebuilt and re-downloaded first.
|
||||
//
|
||||
// An artifact that already carries the compact form is copied straight
|
||||
// through - `artifactStoresText` decides which, by asking the artifact
|
||||
// rather than by trusting a version number.
|
||||
func artifactSelectColumns(text, totals bool) string {
|
||||
totalTracks := "total_tracks"
|
||||
if !totals {
|
||||
// An artifact built before the column existed. Zero is what the
|
||||
// column means by "the catalog does not say", so an older
|
||||
// artifact imports as one that declines to answer rather than
|
||||
// failing to import at all.
|
||||
totalTracks = "0"
|
||||
}
|
||||
|
||||
if !text {
|
||||
return strings.Replace(
|
||||
artifactCatalogColumns, "total_tracks", totalTracks, 1,
|
||||
)
|
||||
}
|
||||
|
||||
return `CASE entity_type
|
||||
WHEN 'artist' THEN 1
|
||||
WHEN 'release_group' THEN 2
|
||||
WHEN 'recording' THEN 3
|
||||
ELSE 0 END,
|
||||
unhex(replace(mbid, '-', '')),
|
||||
title, artist_name,
|
||||
CASE WHEN artist_mbid = '' THEN x''
|
||||
ELSE unhex(replace(artist_mbid, '-', '')) END,
|
||||
aliases, popularity, listener_count, duration,
|
||||
CASE WHEN caa_release_mbid = '' THEN x''
|
||||
ELSE unhex(replace(caa_release_mbid, '-', '')) END,
|
||||
release_name, primary_type, secondary_types, release_date, ` +
|
||||
totalTracks + `,
|
||||
artist_type, country, disambiguation, sort_name, discog_fetched`
|
||||
}
|
||||
|
||||
// artifactHasTotals reports whether the attached artifact carries the
|
||||
// per-release-group track denominator. An artifact published before
|
||||
// that column existed is still a perfectly good catalog, so it is asked
|
||||
// rather than assumed - the same rule, and the same handle, as
|
||||
// artifactStoresText below.
|
||||
func (si *SearchIndex) artifactHasTotals() bool {
|
||||
var n int
|
||||
|
||||
err := si.db.QueryRowWriter(
|
||||
`SELECT COUNT(*) FROM pragma_table_info('explore_index', 'core')
|
||||
WHERE name = 'total_tracks'`,
|
||||
).Scan(&n)
|
||||
|
||||
return err == nil && n > 0
|
||||
}
|
||||
|
||||
// artifactStoresText reports whether the attached artifact carries the
|
||||
// old text encoding.
|
||||
func (si *SearchIndex) artifactStoresText() bool {
|
||||
// The writer, not QueryContext: "core" is attached to that one
|
||||
// connection and does not exist on the read pool. Asking the wrong
|
||||
// handle errors, and the fallback would then convert an artifact
|
||||
// that needs no conversion.
|
||||
var kind string
|
||||
|
||||
if err := si.db.QueryRowWriter(
|
||||
"SELECT typeof(mbid) FROM core.explore_index LIMIT 1",
|
||||
).Scan(&kind); err != nil {
|
||||
return true
|
||||
}
|
||||
|
||||
return kind == "text"
|
||||
}
|
||||
|
||||
// inspectArtifact opens the artifact read-only and reports what it
|
||||
// declares, without touching the live index. Validation happens here so
|
||||
// a bad download is rejected before anything is attached.
|
||||
@@ -263,9 +345,13 @@ func (si *SearchIndex) analyzeIndex() {
|
||||
// is an index range scan and a cancelled import leaves committed work
|
||||
// behind rather than rolling it all back.
|
||||
func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, error) {
|
||||
selectColumns := artifactSelectColumns(
|
||||
si.artifactStoresText(), si.artifactHasTotals(),
|
||||
)
|
||||
|
||||
insertSQL := `
|
||||
INSERT INTO explore_index (` + artifactCatalogColumns + `)
|
||||
SELECT ` + artifactCatalogColumns + `
|
||||
SELECT ` + selectColumns + `
|
||||
FROM core.explore_index
|
||||
WHERE mbid > ?` + upsertIndexConflictSQL
|
||||
|
||||
@@ -273,7 +359,7 @@ func (si *SearchIndex) mergeArtifactRows(ctx context.Context, total int) (int, e
|
||||
// appended only while one exists.
|
||||
insertRangeSQL := `
|
||||
INSERT INTO explore_index (` + artifactCatalogColumns + `)
|
||||
SELECT ` + artifactCatalogColumns + `
|
||||
SELECT ` + selectColumns + `
|
||||
FROM core.explore_index
|
||||
WHERE mbid > ? AND mbid <= ?` + upsertIndexConflictSQL
|
||||
|
||||
|
||||
@@ -202,7 +202,7 @@ func TestImportCoreArtifactPreservesLocalData(t *testing.T) {
|
||||
|
||||
if err := db.QueryRowWriter(`
|
||||
SELECT popularity, duration, in_library, discog_fetched
|
||||
FROM explore_index WHERE mbid = ?`, recA,
|
||||
FROM explore_index WHERE mbid = ?`, dbMBID(recA),
|
||||
).Scan(&popularity, &duration, &inLibrary, &discogFetched); err != nil {
|
||||
t.Fatalf("read merged row: %v", err)
|
||||
}
|
||||
@@ -376,7 +376,7 @@ func TestAddFromCacheNeverStoresMBIDAsName(t *testing.T) {
|
||||
|
||||
var artistName string
|
||||
if err := db.QueryRowWriter(
|
||||
`SELECT artist_name FROM explore_index WHERE mbid = ?`, rgA,
|
||||
`SELECT artist_name FROM explore_index WHERE mbid = ?`, dbMBID(rgA),
|
||||
).Scan(&artistName); err != nil {
|
||||
t.Fatalf("read release group: %v", err)
|
||||
}
|
||||
@@ -391,7 +391,7 @@ func TestAddFromCacheNeverStoresMBIDAsName(t *testing.T) {
|
||||
})
|
||||
|
||||
if err := db.QueryRowWriter(
|
||||
`SELECT artist_name FROM explore_index WHERE mbid = ?`, rgA,
|
||||
`SELECT artist_name FROM explore_index WHERE mbid = ?`, dbMBID(rgA),
|
||||
).Scan(&artistName); err != nil {
|
||||
t.Fatalf("re-read release group: %v", err)
|
||||
}
|
||||
@@ -442,3 +442,158 @@ func TestArtifactColumnsMatchExporter(t *testing.T) {
|
||||
exporter, importer)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportCoreArtifactAcceptsBothEncodings is the compatibility half
|
||||
// of the storage change.
|
||||
//
|
||||
// The catalog stores an MBID as 16 raw bytes and an entity type as a
|
||||
// code, which took the table and its indexes from 677 MB to 389 MB. A
|
||||
// published artifact carries whichever form the exporter that built it
|
||||
// used, and there is one already out there in the older text form — so
|
||||
// the importer decides by asking the artifact, not by trusting a
|
||||
// version number, and both must land identically.
|
||||
func TestImportCoreArtifactAcceptsBothEncodings(t *testing.T) {
|
||||
compact := filepath.Join(t.TempDir(), "core-index.db")
|
||||
|
||||
db, err := sql.Open("sqlite", "file:"+compact)
|
||||
if err != nil {
|
||||
t.Fatalf("open artifact: %v", err)
|
||||
}
|
||||
|
||||
if _, err := db.Exec(`CREATE TABLE explore_index (
|
||||
entity_type INTEGER NOT NULL,
|
||||
mbid BLOB NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
artist_name TEXT NOT NULL,
|
||||
artist_mbid BLOB NOT NULL,
|
||||
aliases TEXT NOT NULL DEFAULT '',
|
||||
popularity INTEGER NOT NULL DEFAULT 0,
|
||||
listener_count INTEGER NOT NULL DEFAULT 0,
|
||||
duration INTEGER NOT NULL DEFAULT 0,
|
||||
caa_release_mbid BLOB NOT NULL DEFAULT x'',
|
||||
release_name TEXT NOT NULL DEFAULT '',
|
||||
primary_type TEXT NOT NULL DEFAULT '',
|
||||
secondary_types TEXT NOT NULL DEFAULT '',
|
||||
release_date TEXT NOT NULL DEFAULT '',
|
||||
artist_type TEXT NOT NULL DEFAULT '',
|
||||
country TEXT NOT NULL DEFAULT '',
|
||||
disambiguation TEXT NOT NULL DEFAULT '',
|
||||
sort_name TEXT NOT NULL DEFAULT '',
|
||||
discog_fetched INTEGER NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (mbid)
|
||||
)`); err != nil {
|
||||
t.Fatalf("create artifact table: %v", err)
|
||||
}
|
||||
|
||||
if _, err := db.Exec(
|
||||
`CREATE TABLE artifact_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)`,
|
||||
); err != nil {
|
||||
t.Fatalf("create artifact meta: %v", err)
|
||||
}
|
||||
|
||||
for k, v := range validMeta() {
|
||||
if _, err := db.Exec(
|
||||
"INSERT INTO artifact_meta (key, value) VALUES (?, ?)", k, v,
|
||||
); err != nil {
|
||||
t.Fatalf("write artifact meta: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := db.Exec(`
|
||||
INSERT INTO explore_index (entity_type, mbid, title, artist_name, artist_mbid, popularity)
|
||||
VALUES (1, ?, 'Artist A', 'Artist A', ?, 5000)`,
|
||||
mbidBytes(artA), mbidBytes(artA),
|
||||
); err != nil {
|
||||
t.Fatalf("write artifact row: %v", err)
|
||||
}
|
||||
|
||||
_ = db.Close()
|
||||
|
||||
live := database.NewTestDB(t)
|
||||
si := NewSearchIndex(live, nil, nil, testLogger())
|
||||
|
||||
if err := si.importCoreArtifact(context.Background(), compact); err != nil {
|
||||
t.Fatalf("importCoreArtifact (compact): %v", err)
|
||||
}
|
||||
|
||||
got := si.LookupArtistByMBID(artA)
|
||||
if got == nil {
|
||||
t.Fatal("artist from a compact artifact was not imported")
|
||||
}
|
||||
|
||||
if got.Title != "Artist A" || got.MBID != artA {
|
||||
t.Errorf("imported %+v, want Artist A / %s", got, artA)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportCoreArtifactReadsTotalsWhenPresent covers both halves of the
|
||||
// denominator's arrival: an artifact that carries total_tracks imports
|
||||
// it, and one built before the column existed still imports at all.
|
||||
//
|
||||
// The second half is the one worth a test. Adding a column to the
|
||||
// importer's SELECT list is how you break every artifact already
|
||||
// published - "no such column: total_tracks", on a file nobody can
|
||||
// re-cut retroactively - so the importer asks the artifact what it has,
|
||||
// the same way it asks which encoding it uses.
|
||||
func TestImportCoreArtifactReadsTotalsWhenPresent(t *testing.T) {
|
||||
path := writeTestArtifact(t, validMeta(), []artifactRow{
|
||||
{"release_group", rgA, "Big Album", "Solo Star", artA, 5000},
|
||||
})
|
||||
|
||||
// The exporter writes the column; writeTestArtifact builds the older
|
||||
// shape, so add it here rather than changing every other test's
|
||||
// fixture to carry a value they do not use.
|
||||
artifact, err := sql.Open("sqlite", "file:"+path)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen artifact: %v", err)
|
||||
}
|
||||
|
||||
for _, stmt := range []string{
|
||||
`ALTER TABLE explore_index ADD COLUMN total_tracks INTEGER NOT NULL DEFAULT 0`,
|
||||
`UPDATE explore_index SET total_tracks = 12`,
|
||||
} {
|
||||
if _, err := artifact.Exec(stmt); err != nil {
|
||||
t.Fatalf("add total_tracks: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
_ = artifact.Close()
|
||||
|
||||
live := database.NewTestDB(t)
|
||||
si := NewSearchIndex(live, nil, nil, testLogger())
|
||||
|
||||
if err := si.importCoreArtifact(context.Background(), path); err != nil {
|
||||
t.Fatalf("importCoreArtifact: %v", err)
|
||||
}
|
||||
|
||||
rg := si.LookupReleaseGroupByMBID(rgA)
|
||||
if rg == nil {
|
||||
t.Fatal("release group was not imported")
|
||||
}
|
||||
|
||||
if rg.TotalTracks != 12 {
|
||||
t.Errorf("TotalTracks = %d, want 12", rg.TotalTracks)
|
||||
}
|
||||
|
||||
// And the shape that predates the column: the same import, from an
|
||||
// artifact that has no total_tracks at all.
|
||||
older := writeTestArtifact(t, validMeta(), []artifactRow{
|
||||
{"release_group", rgB, "Duet Album", "Solo Star", artA, 4000},
|
||||
})
|
||||
|
||||
live2 := database.NewTestDB(t)
|
||||
si2 := NewSearchIndex(live2, nil, nil, testLogger())
|
||||
|
||||
if err := si2.importCoreArtifact(context.Background(), older); err != nil {
|
||||
t.Fatalf("importCoreArtifact (no total_tracks column): %v", err)
|
||||
}
|
||||
|
||||
old := si2.LookupReleaseGroupByMBID(rgB)
|
||||
if old == nil {
|
||||
t.Fatal("release group from a column-less artifact was not imported")
|
||||
}
|
||||
|
||||
if old.TotalTracks != 0 {
|
||||
t.Errorf("TotalTracks = %d, want 0 (the catalog does not say)", old.TotalTracks)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,19 +12,15 @@ import (
|
||||
func seedIndexArtist(t *testing.T, db *database.DB, mbid string, discogFetched int) {
|
||||
t.Helper()
|
||||
|
||||
if _, err := db.ExecContext(
|
||||
upsertIndexSQL,
|
||||
"artist", mbid, "Seeded Artist", "Seeded Artist", mbid, "",
|
||||
0, 0,
|
||||
0, "", "",
|
||||
"", "", "",
|
||||
"", "", "", "",
|
||||
1, 0,
|
||||
0, 0, 0,
|
||||
discogFetched,
|
||||
); err != nil {
|
||||
t.Fatalf("seed explore_index row for %q: %v", mbid, err)
|
||||
}
|
||||
seedIndexResult(t, db, SearchIndexResult{
|
||||
EntityType: EntityArtist,
|
||||
MBID: testMBID(mbid),
|
||||
Title: "Seeded Artist",
|
||||
ArtistName: "Seeded Artist",
|
||||
ArtistMBID: testMBID(mbid),
|
||||
InLibrary: true,
|
||||
DiscogFetched: discogFetched == 1,
|
||||
})
|
||||
}
|
||||
|
||||
// TestArtistEnrichmentMarksAreIndependent is the reason these are two
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
)
|
||||
|
||||
// lbPopularityServer serves both top-for-artist popularity endpoints
|
||||
// with a fixed status and body, and counts what it was asked.
|
||||
func lbPopularityServer(
|
||||
t *testing.T, status int, body string,
|
||||
) (*ListenBrainzClient, *int) {
|
||||
t.Helper()
|
||||
|
||||
requests := 0
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
requests++
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_, _ = w.Write([]byte(body))
|
||||
},
|
||||
))
|
||||
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
lb := NewListenBrainzClient(NewRateLimiterN(1000), nil, slog.Default())
|
||||
lb.SetBaseURL(srv.URL)
|
||||
|
||||
return lb, &requests
|
||||
}
|
||||
|
||||
// TestTopFetchesSeparateEmptyFromFailed is the distinction the
|
||||
// owned-artist backfill's mark rests on. ListenBrainz answers 200 with
|
||||
// `[]` for an artist it has no popularity data for — which is most of a
|
||||
// long-tail library — and an empty answer is a *complete* one. When
|
||||
// discog_fetched was keyed on "did rows come back", those artists were
|
||||
// never marked, stayed in unenrichedLibraryArtistMBIDs forever, and
|
||||
// "Filling in artist details" re-ran for them on every single launch.
|
||||
func TestTopFetchesSeparateEmptyFromFailed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
si := NewSearchIndex(database.NewTestDB(t), nil, nil, slog.Default())
|
||||
artist := lbSitewideArtist{
|
||||
ArtistMBID: "44444444-4444-4444-4444-444444444444",
|
||||
ArtistName: "Nobody Has Listened",
|
||||
}
|
||||
|
||||
t.Run("empty is success", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
lb, _ := lbPopularityServer(t, http.StatusOK, `[]`)
|
||||
|
||||
rgs, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50)
|
||||
if err != nil || len(rgs) != 0 {
|
||||
t.Errorf("top RGs: got %d rows, err %v; want 0 rows and no error", len(rgs), err)
|
||||
}
|
||||
|
||||
recs, err := si.fetchTopRecordings(t.Context(), lb, artist, 200)
|
||||
if err != nil || len(recs) != 0 {
|
||||
t.Errorf("top recordings: got %d rows, err %v; want 0 rows and no error",
|
||||
len(recs), err,
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
// Below indexMinPopularity is the same shape one step in: the
|
||||
// endpoint answered, we simply keep none of it.
|
||||
t.Run("everything below the popularity floor is success", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
lb, _ := lbPopularityServer(t, http.StatusOK,
|
||||
`[{"release_group_mbid":"rg-1","total_listen_count":3,`+
|
||||
`"release_group":{"name":"Obscure"}}]`)
|
||||
|
||||
rgs, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50)
|
||||
if err != nil || len(rgs) != 0 {
|
||||
t.Errorf("top RGs: got %d rows, err %v; want 0 rows and no error", len(rgs), err)
|
||||
}
|
||||
})
|
||||
|
||||
// A failure must stay a failure, or the retry this is built on goes
|
||||
// away and a throttled run marks artists it never fetched.
|
||||
t.Run("HTTP error is a failure", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
lb, _ := lbPopularityServer(t, http.StatusServiceUnavailable, `nope`)
|
||||
|
||||
if _, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50); err == nil {
|
||||
t.Error("top RGs reported success on a 503")
|
||||
}
|
||||
|
||||
if _, err := si.fetchTopRecordings(t.Context(), lb, artist, 200); err == nil {
|
||||
t.Error("top recordings reported success on a 503")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("unparseable body is a failure", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
lb, _ := lbPopularityServer(t, http.StatusOK, `{"not":"an array"}`)
|
||||
|
||||
if _, err := si.fetchTopReleaseGroups(t.Context(), lb, artist, 50); err == nil {
|
||||
t.Error("top RGs reported success on a body it could not read")
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -47,6 +47,13 @@ const (
|
||||
// canonicalProgressRows controls progress reporting during the
|
||||
// canonical CSV scan (~30M rows total).
|
||||
canonicalProgressRows = 2_000_000
|
||||
|
||||
// maxReleaseTracks caps the per-release track count, so a malformed
|
||||
// row cannot turn the denominator into nonsense. Well above the
|
||||
// longest real release, and it is a cap rather than a rejection
|
||||
// because a box set reporting 999 is still a better answer than one
|
||||
// reporting nothing.
|
||||
maxReleaseTracks = 999
|
||||
)
|
||||
|
||||
// Per-artist discography coverage (S2). The global budgets above keep
|
||||
@@ -329,6 +336,18 @@ type canonicalScan struct {
|
||||
releaseToRG map[uuid16]rgTarget
|
||||
artistNames map[uuid16]string
|
||||
|
||||
// releaseTracks counts the canonical dump's rows per kept release,
|
||||
// which is that release's track count: the dump carries one row per
|
||||
// recording per canonical release.
|
||||
//
|
||||
// It is counted *before* the popularity filter below, unlike almost
|
||||
// everything else here, because a denominator built from the kept
|
||||
// recordings would say "9" about a twelve-track album whose other
|
||||
// three are unpopular - which is worse than saying nothing, and is
|
||||
// exactly the confident lie that kept whole tracklists out of the
|
||||
// artifact. Bounded by the kept release set, not by MusicBrainz.
|
||||
releaseTracks map[uuid16]uint16
|
||||
|
||||
// artistTracks accumulates each target artist's top recordings for
|
||||
// S2 coverage; merged into recordings before assembly.
|
||||
artistTracks *perArtistTracks
|
||||
@@ -454,10 +473,11 @@ func (imp *dumpImporter) scanCanonicalDump(
|
||||
defer zr.Close()
|
||||
|
||||
scan := &canonicalScan{
|
||||
releaseInfos: make(map[uuid16]releaseInfo, len(ks.releases)),
|
||||
releaseToRG: make(map[uuid16]rgTarget, len(ks.releases)),
|
||||
artistNames: make(map[uuid16]string, len(ks.artists)),
|
||||
artistTracks: newPerArtistTracks(),
|
||||
releaseInfos: make(map[uuid16]releaseInfo, len(ks.releases)),
|
||||
releaseToRG: make(map[uuid16]rgTarget, len(ks.releases)),
|
||||
artistNames: make(map[uuid16]string, len(ks.artists)),
|
||||
releaseTracks: make(map[uuid16]uint16, len(ks.releases)),
|
||||
artistTracks: newPerArtistTracks(),
|
||||
}
|
||||
|
||||
sawData, sawRedirect := false, false
|
||||
@@ -614,9 +634,17 @@ func (imp *dumpImporter) scanCanonicalData(
|
||||
}
|
||||
}
|
||||
|
||||
// Release display info for release-group titling.
|
||||
// Release display info for release-group titling, and the
|
||||
// release's track count. Both are for kept releases only, and
|
||||
// the count is taken here rather than below because every row
|
||||
// of this dump is one track of its release regardless of how
|
||||
// often anyone played it.
|
||||
if relOK {
|
||||
if _, kept := ks.releases[relMBID]; kept {
|
||||
if n := scan.releaseTracks[relMBID]; n < maxReleaseTracks {
|
||||
scan.releaseTracks[relMBID] = n + 1
|
||||
}
|
||||
|
||||
if _, seen := scan.releaseInfos[relMBID]; !seen {
|
||||
firstArtist := ""
|
||||
if len(artistMBIDs) > 0 {
|
||||
@@ -1004,6 +1032,7 @@ func (imp *dumpImporter) assembleIndex(
|
||||
ArtistName: info.artistName,
|
||||
ArtistMBID: info.artistMBID,
|
||||
Popularity: int(agg.listens),
|
||||
TotalTracks: int(scan.releaseTracks[agg.bestRel]),
|
||||
CAAReleaseMBID: formatUUID(agg.canonical[:]),
|
||||
})
|
||||
|
||||
|
||||
@@ -277,6 +277,12 @@ func canonicalDataCSV(t *testing.T) []byte {
|
||||
"3", "11", "{" + artA + "," + artB + "}", "Solo Star feat. Guest",
|
||||
relB, "Duet Album", recC, "Duet Song", "x", "1",
|
||||
},
|
||||
// Unplayed, so it survives no popularity floor and is not
|
||||
// indexed -- but it is still a track on Big Album.
|
||||
{
|
||||
"4", "10", "{" + artA + "}", "Solo Star",
|
||||
relA, "Big Album", recD, "Album Filler", "x", "1",
|
||||
},
|
||||
}
|
||||
|
||||
return csvBytes(t, rows)
|
||||
@@ -594,7 +600,7 @@ func TestDumpImportEndToEnd(t *testing.T) {
|
||||
|
||||
rows, err := db.QueryContext(
|
||||
"SELECT title, popularity FROM explore_index WHERE mbid = ? AND entity_type = ?",
|
||||
mbid, entityType,
|
||||
dbMBID(mbid), dbEntityType(entityType),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query: %v", err)
|
||||
@@ -627,11 +633,51 @@ func TestDumpImportEndToEnd(t *testing.T) {
|
||||
assertRow(rgB, "release_group", "Duet Album", 12)
|
||||
assertRow(artA, "artist", "Solo Star", 35)
|
||||
|
||||
// The per-release-group denominator: how many tracks the release
|
||||
// has, counted from the canonical dump *before* the popularity
|
||||
// filter. Big Album has three, one of which nobody has played and
|
||||
// which is therefore not indexed as a recording at all -- a
|
||||
// denominator built from the kept recordings would say two, and
|
||||
// "you have 2 of 2" about a three-track album is worse than saying
|
||||
// nothing.
|
||||
assertTotalTracks := func(mbid string, want int) {
|
||||
t.Helper()
|
||||
|
||||
var got int
|
||||
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT total_tracks FROM explore_index WHERE mbid = ? AND entity_type = 2",
|
||||
dbMBID(mbid),
|
||||
).Scan(&got); err != nil {
|
||||
t.Fatalf("read total_tracks for %s: %v", mbid, err)
|
||||
}
|
||||
|
||||
if got != want {
|
||||
t.Errorf("total_tracks for %s = %d, want %d", mbid, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
assertTotalTracks(rgA, 3)
|
||||
assertTotalTracks(rgB, 1)
|
||||
|
||||
// And the unplayed track is still not an indexed recording.
|
||||
var fillerRows int
|
||||
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", dbMBID(recD),
|
||||
).Scan(&fillerRows); err != nil {
|
||||
t.Fatalf("count recD rows: %v", err)
|
||||
}
|
||||
|
||||
if fillerRows != 0 {
|
||||
t.Errorf("unplayed track was indexed as a recording (%d rows)", fillerRows)
|
||||
}
|
||||
|
||||
// artB only ever appears in a multi-artist credit: no name is
|
||||
// derivable from the dump, so it must be queued for the API
|
||||
// metadata patch instead of being written nameless.
|
||||
rows, err := db.QueryContext(
|
||||
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", artB,
|
||||
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", dbMBID(artB),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query artB: %v", err)
|
||||
@@ -1006,7 +1052,7 @@ func TestListenerCountUpdateDoesNotTouchPopularity(t *testing.T) {
|
||||
}
|
||||
|
||||
rows, err := db.QueryContext(
|
||||
"SELECT popularity, listener_count FROM explore_index WHERE mbid = ?", recA,
|
||||
"SELECT popularity, listener_count FROM explore_index WHERE mbid = ?", dbMBID(recA),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query: %v", err)
|
||||
|
||||
@@ -277,7 +277,10 @@ func (si *SearchIndex) commitListenDeltas(
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
if _, err := tx.Exec(
|
||||
"CREATE TEMP TABLE IF NOT EXISTS incr_delta (mbid TEXT, kind TEXT, delta INTEGER)",
|
||||
// mbid and kind are stored the way explore_index stores them,
|
||||
// so the join below is a plain equality rather than a
|
||||
// conversion per row.
|
||||
"CREATE TEMP TABLE IF NOT EXISTS incr_delta (mbid BLOB, kind INTEGER, delta INTEGER)",
|
||||
); err != nil {
|
||||
return fmt.Errorf("incremental temp table: %w", err)
|
||||
}
|
||||
@@ -357,8 +360,10 @@ func insertDeltas(tx *sql.Tx, kind string, deltas map[string]uint32) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
code := entityCode(kind)
|
||||
|
||||
for mbid, d := range deltas {
|
||||
rowArgs = append(rowArgs, mbid, kind, int64(d))
|
||||
rowArgs = append(rowArgs, mbidBytes(mbid), code, int64(d))
|
||||
pending++
|
||||
|
||||
if pending >= deltaInsertBatch {
|
||||
|
||||
@@ -36,7 +36,7 @@ func popularityOf(t *testing.T, db *database.DB, mbid string) (int, bool) {
|
||||
t.Helper()
|
||||
|
||||
rows, err := db.QueryContext(
|
||||
"SELECT popularity FROM explore_index WHERE mbid = ?", mbid,
|
||||
"SELECT popularity FROM explore_index WHERE mbid = ?", dbMBID(mbid),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("query popularity: %v", err)
|
||||
|
||||
@@ -54,7 +54,7 @@ func (imp *dumpImporter) runPatchPasses(ctx context.Context) {
|
||||
func (imp *dumpImporter) patchArtistMetadata(ctx context.Context) {
|
||||
rows, err := imp.si.db.QueryContext(`
|
||||
SELECT mbid FROM explore_index
|
||||
WHERE entity_type = 'artist'
|
||||
WHERE entity_type = 1 /* artist */
|
||||
AND (artist_type = '' OR country = '' OR title = '' OR title = mbid)
|
||||
`)
|
||||
if err != nil {
|
||||
@@ -152,7 +152,7 @@ func (imp *dumpImporter) patchSimilarArtists(ctx context.Context) {
|
||||
for _, s := range similar {
|
||||
_, _ = imp.si.db.ExecContext(
|
||||
"UPDATE explore_index SET is_similar = 1 WHERE artist_mbid = ?",
|
||||
s.ArtistMBID,
|
||||
dbMBID(s.ArtistMBID),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -219,7 +219,7 @@ func (imp *dumpImporter) topMBIDs(entityType string, limit int) []string {
|
||||
WHERE entity_type = ? AND listener_count = 0
|
||||
ORDER BY popularity DESC
|
||||
LIMIT ?
|
||||
`, entityType, limit)
|
||||
`, dbEntityType(entityType), limit)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
@@ -229,9 +229,9 @@ func (imp *dumpImporter) topMBIDs(entityType string, limit int) []string {
|
||||
var mbids []string
|
||||
|
||||
for rows.Next() {
|
||||
var m string
|
||||
var m dbMBID
|
||||
if err := rows.Scan(&m); err == nil {
|
||||
mbids = append(mbids, m)
|
||||
mbids = append(mbids, string(m))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -264,7 +264,7 @@ func (si *SearchIndex) updateListenerCounts(updates map[string]PopularityData) i
|
||||
`UPDATE explore_index
|
||||
SET listener_count = ?
|
||||
WHERE mbid = ? AND listener_count < ?`,
|
||||
data.ListenerCount, strings.ToLower(mbid), data.ListenerCount,
|
||||
data.ListenerCount, dbMBID(strings.ToLower(mbid)), data.ListenerCount,
|
||||
)
|
||||
if err != nil {
|
||||
continue
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
)
|
||||
|
||||
// TestStoredEncodingRoundTrips walks every read path in the package
|
||||
// against a row written by the real upsert.
|
||||
//
|
||||
// It exists because of how this storage change fails when it fails.
|
||||
// MBIDs are stored as 16 raw bytes and entity types as codes - which
|
||||
// took the catalog and its indexes from 780 MB to 405 MB, measured on a
|
||||
// real 2,052,200-row catalog - and SQLite does not coerce between TEXT
|
||||
// and BLOB. A query that still compares against a 36-character string
|
||||
// therefore returns *no rows* rather than an error, and a scan into a
|
||||
// plain string yields sixteen bytes of mojibake. Neither shows up as a
|
||||
// failure anywhere except in a result that is quietly empty.
|
||||
//
|
||||
// So this is not a unit test of the encoding (that is TestMBIDRoundTrip)
|
||||
// but a sweep: every query that reads the catalog, asserted to return
|
||||
// something and to hand back canonical dashed ids.
|
||||
func TestStoredEncodingRoundTrips(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, testLogger())
|
||||
|
||||
const (
|
||||
artist = "c0b2500e-0cef-4130-9b13-1b9d9a2f2c07"
|
||||
album = "11111111-2222-3333-4444-555555555555"
|
||||
)
|
||||
|
||||
si.upsertBatch([]SearchIndexResult{
|
||||
{
|
||||
EntityType: EntityArtist,
|
||||
MBID: artist,
|
||||
Title: "Radiohead",
|
||||
ArtistName: "Radiohead",
|
||||
ArtistMBID: artist,
|
||||
Popularity: 90000,
|
||||
},
|
||||
{
|
||||
EntityType: EntityReleaseGroup,
|
||||
MBID: album,
|
||||
Title: "Kid A",
|
||||
ArtistName: "Radiohead",
|
||||
ArtistMBID: artist,
|
||||
Popularity: 80000,
|
||||
CAAReleaseMBID: album,
|
||||
},
|
||||
})
|
||||
|
||||
// The stored form is the compact one, not the strings above.
|
||||
var typeMBID, typeEntity string
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT typeof(mbid), typeof(entity_type) FROM explore_index LIMIT 1",
|
||||
).Scan(&typeMBID, &typeEntity); err != nil {
|
||||
t.Fatalf("typeof: %v", err)
|
||||
}
|
||||
|
||||
if typeMBID != "blob" || typeEntity != "integer" {
|
||||
t.Errorf("stored as mbid=%s entity_type=%s, want blob/integer", typeMBID, typeEntity)
|
||||
}
|
||||
|
||||
si.MarkReadyIfPopulated()
|
||||
|
||||
// Read paths.
|
||||
if got := si.LookupArtistByMBID(artist); got == nil {
|
||||
t.Error("LookupArtistByMBID found nothing")
|
||||
} else if got.MBID != artist || got.ArtistMBID != artist || got.EntityType != EntityArtist {
|
||||
t.Errorf("lookup artist = %+v, want dashed ids and the artist type", got)
|
||||
}
|
||||
|
||||
if got := si.LookupReleaseGroupByMBID(album); got == nil {
|
||||
t.Error("LookupReleaseGroupByMBID found nothing")
|
||||
} else if got.MBID != album || got.EntityType != EntityReleaseGroup {
|
||||
t.Errorf("lookup album = %+v, want the dashed id and the release-group type", got)
|
||||
}
|
||||
|
||||
if rgs := si.TopReleaseGroupsByArtist(artist, 5); len(rgs) == 0 {
|
||||
t.Error("TopReleaseGroupsByArtist found nothing")
|
||||
} else if rgs[0].MBID != album || rgs[0].ArtistMBID != artist {
|
||||
t.Errorf("top release groups[0] = %+v, want %s by %s", rgs[0], album, artist)
|
||||
}
|
||||
|
||||
// The exact-match tier reads the two partial LOWER() indexes, whose
|
||||
// predicate has to agree with its WHERE clause or the seek silently
|
||||
// becomes a scan.
|
||||
if m := si.ExactMatches("radiohead", 3); len(m) == 0 {
|
||||
t.Error("ExactMatches found nothing")
|
||||
} else if m[0].MBID != artist || m[0].EntityType != EntityArtist {
|
||||
t.Errorf("exact match[0] = %+v, want the artist", m[0])
|
||||
}
|
||||
|
||||
if hits := si.Search(t.Context(), "radiohead", 5); len(hits) == 0 {
|
||||
t.Error("Search found nothing")
|
||||
} else if hits[0].MBID != artist || hits[0].EntityType != EntityArtist {
|
||||
t.Errorf("search hit[0] = %+v, want the artist", hits[0])
|
||||
}
|
||||
|
||||
if b := si.GetPopularityBatch([]string{artist, album}); b == nil || len(b.Popularity) != 2 {
|
||||
t.Errorf("GetPopularityBatch = %+v, want two entries", b)
|
||||
}
|
||||
|
||||
if m := si.ReleaseGroupMBIDsForCAAReleaseMBIDs([]string{album}); len(m) != 1 {
|
||||
t.Errorf("CAA lookup = %v, want one entry", m)
|
||||
}
|
||||
}
|
||||
@@ -22,8 +22,8 @@ func seedIndexRow(
|
||||
_, err := db.ExecContext(`
|
||||
INSERT INTO explore_index
|
||||
(entity_type, mbid, title, artist_name, artist_mbid, popularity, listener_count)
|
||||
VALUES (?, ?, ?, ?, '', ?, ?)
|
||||
`, entityType, mbid, title, artist, popularity, popularity/10)
|
||||
VALUES (?, ?, ?, ?, x'', ?, ?)
|
||||
`, dbEntityType(entityType), dbMBID(testMBID(mbid)), title, artist, popularity, popularity/10)
|
||||
if err != nil {
|
||||
t.Fatalf("seed %s/%s: %v", entityType, mbid, err)
|
||||
}
|
||||
@@ -73,12 +73,12 @@ func TestEvalHarnessIndexRanking(t *testing.T) {
|
||||
{
|
||||
Query: "radiohead",
|
||||
Note: "popular exact artist match",
|
||||
Expect: []eval.Expected{{Type: "artist", MBID: "rh"}},
|
||||
Expect: []eval.Expected{{Type: "artist", MBID: testMBID("rh")}},
|
||||
},
|
||||
{
|
||||
Query: "the teenagers",
|
||||
Note: "low-popularity exact match must beat high-popularity article match",
|
||||
Expect: []eval.Expected{{Type: "artist", MBID: "teenagers"}},
|
||||
Expect: []eval.Expected{{Type: "artist", MBID: testMBID("teenagers")}},
|
||||
},
|
||||
}
|
||||
|
||||
@@ -123,8 +123,13 @@ func TestExploreFTSDiacriticFolding(t *testing.T) {
|
||||
t.Fatalf("query %q returned no hits", tc.query)
|
||||
}
|
||||
|
||||
if hits[0].MBID != tc.wantMBID {
|
||||
t.Errorf("query %q: top hit = %q, want %q", tc.query, hits[0].MBID, tc.wantMBID)
|
||||
if hits[0].MBID != testMBID(tc.wantMBID) {
|
||||
t.Errorf(
|
||||
"query %q: top hit = %q, want %q",
|
||||
tc.query,
|
||||
hits[0].MBID,
|
||||
testMBID(tc.wantMBID),
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -148,3 +153,30 @@ func TestEvalFixtureFileParses(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// seedIndexResult writes one explore_index row through the same
|
||||
// upsertBatch every other writer uses.
|
||||
//
|
||||
// The three seeders that used to bind upsertIndexSQL's parameters by
|
||||
// hand said, in a comment, that this was on purpose: a schema change
|
||||
// should break the tests where it breaks the app. It did not - it
|
||||
// broke them at "missing argument with index 25", one file at a time,
|
||||
// for a column none of them cares about. Going through the one writer
|
||||
// keeps the property they wanted (a field written to the wrong column
|
||||
// still fails here) without three copies of a 24-argument list.
|
||||
func seedIndexResult(t *testing.T, db *database.DB, r SearchIndexResult) {
|
||||
t.Helper()
|
||||
|
||||
NewSearchIndex(db, nil, nil, slog.Default()).
|
||||
upsertBatch([]SearchIndexResult{r})
|
||||
|
||||
// upsertBatch logs and swallows, which is right for a background
|
||||
// merge and useless for a fixture, so the row is checked for.
|
||||
var n int
|
||||
|
||||
if err := db.QueryRowWriter(
|
||||
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", dbMBID(r.MBID),
|
||||
).Scan(&n); err != nil || n == 0 {
|
||||
t.Fatalf("seed explore_index row %q: not written (%v)", r.MBID, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -130,12 +130,16 @@ func NewExploreService(logger *slog.Logger, db *database.DB) *Service {
|
||||
|
||||
// MusicBrainz returns the shared cached MB client so other services
|
||||
// (e.g. autotag) can reuse it without spinning up a second limiter.
|
||||
//
|
||||
//wails:ignore // internal wiring, not part of the app's IPC surface.
|
||||
func (e *Service) MusicBrainz() *MusicBrainzClient {
|
||||
return e.mb
|
||||
}
|
||||
|
||||
// CAALimiter returns the shared Cover Art Archive rate limiter.
|
||||
// Consumers must respect it for any fresh CAA HTTP GETs.
|
||||
//
|
||||
//wails:ignore // internal wiring, not part of the app's IPC surface.
|
||||
func (e *Service) CAALimiter() *RateLimiter {
|
||||
return e.caaLimiter
|
||||
}
|
||||
@@ -185,6 +189,8 @@ func (e *Service) CoreCatalogImported() bool {
|
||||
|
||||
// SetJobRegistry wires the background job registry into the search
|
||||
// index so its build reports progress and controls to the frontend.
|
||||
//
|
||||
//wails:ignore // internal wiring, not part of the app's IPC surface.
|
||||
func (e *Service) SetJobRegistry(reg *jobs.Registry) {
|
||||
e.index.SetJobRegistry(reg)
|
||||
}
|
||||
@@ -561,6 +567,7 @@ func (e *Service) LookupReleaseGroup(mbid string) (*MBReleaseGroup, error) {
|
||||
FirstReleaseDate: indexed.ReleaseDate,
|
||||
InLibrary: indexed.InLibrary || indexed.LocalReleaseGroupID > 0,
|
||||
LocalID: indexed.LocalReleaseGroupID,
|
||||
TotalTracks: indexed.TotalTracks,
|
||||
}
|
||||
|
||||
// Background: fetch full MB data once per RG to fill in fields
|
||||
|
||||
@@ -18,9 +18,22 @@ func NewLibraryMBIDIndex(db *database.DB) *LibraryMBIDIndex {
|
||||
return &LibraryMBIDIndex{db: db}
|
||||
}
|
||||
|
||||
// CheckMBIDs returns which of the given MBIDs exist in the local
|
||||
// library. The returned map has MBID → entity type ("artist",
|
||||
// CheckMBIDs returns which of the given MBIDs the library actually
|
||||
// has a file for. The returned map is MBID -> entity type ("artist",
|
||||
// "release_group", or "recording").
|
||||
//
|
||||
// The "has a file" part is the whole point and is what this used to get
|
||||
// wrong. It was three `SELECT mbid FROM <metadata table>` queries, and
|
||||
// a metadata row could outlive the file that created it - retagging a
|
||||
// file abandoned its old recording row, which kept the old MBID
|
||||
// forever. Measured on a real library: 812 orphaned recordings, of
|
||||
// which 218 carried MBIDs, and 129 catalog rows that this function
|
||||
// therefore reported as owned. Every one of them rendered as a track
|
||||
// you have, with a play button that could not work, because playback
|
||||
// resolves files and this resolved metadata.
|
||||
//
|
||||
// Each branch now joins audio_files. An entity is in your library if
|
||||
// and only if a file says so.
|
||||
func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
|
||||
if len(mbids) == 0 {
|
||||
return nil
|
||||
@@ -28,16 +41,26 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
|
||||
|
||||
result := make(map[string]string, len(mbids))
|
||||
|
||||
// Batch check all MBIDs against each table with a single IN query.
|
||||
type tableEntity struct {
|
||||
table string
|
||||
type entityQuery struct {
|
||||
entityType string
|
||||
query string
|
||||
}
|
||||
|
||||
tables := []tableEntity{
|
||||
{"artists", "artist"},
|
||||
{"release_groups", "release_group"},
|
||||
{"recordings", "recording"},
|
||||
queries := []entityQuery{
|
||||
{"recording", `SELECT DISTINCT recording_mbid FROM audio_files
|
||||
WHERE recording_mbid IN (%s)`},
|
||||
{"release_group", `SELECT DISTINCT al.mbid FROM albums al
|
||||
JOIN audio_files af ON af.album_id = al.id
|
||||
WHERE al.mbid IN (%s)`},
|
||||
{"artist", `SELECT DISTINCT a.mbid FROM artists a
|
||||
WHERE a.mbid IN (%s) AND (
|
||||
EXISTS (SELECT 1 FROM audio_files af WHERE af.artist_id = a.id)
|
||||
OR EXISTS (
|
||||
SELECT 1 FROM albums al
|
||||
JOIN audio_files af2 ON af2.album_id = al.id
|
||||
WHERE al.artist_id = a.id
|
||||
)
|
||||
)`},
|
||||
}
|
||||
|
||||
// Build a set of MBIDs still unresolved.
|
||||
@@ -48,12 +71,11 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
|
||||
}
|
||||
}
|
||||
|
||||
for _, te := range tables {
|
||||
for _, eq := range queries {
|
||||
if len(remaining) == 0 {
|
||||
break
|
||||
}
|
||||
|
||||
// Build IN clause from remaining MBIDs.
|
||||
placeholders := make([]string, 0, len(remaining))
|
||||
args := make([]any, 0, len(remaining))
|
||||
|
||||
@@ -62,9 +84,10 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
|
||||
args = append(args, m)
|
||||
}
|
||||
|
||||
//nolint:gosec // table name is hardcoded from the tables slice above
|
||||
query := "SELECT mbid FROM " + te.table + " WHERE mbid IN (" +
|
||||
strings.Join(placeholders, ",") + ")"
|
||||
//nolint:gosec // the query text is a constant from the slice above
|
||||
query := strings.Replace(
|
||||
eq.query, "%s", strings.Join(placeholders, ","), 1,
|
||||
)
|
||||
|
||||
rows, err := idx.db.QueryContext(query, args...)
|
||||
if err != nil {
|
||||
@@ -74,7 +97,7 @@ func (idx *LibraryMBIDIndex) CheckMBIDs(mbids []string) map[string]string {
|
||||
for rows.Next() {
|
||||
var mbid string
|
||||
if err := rows.Scan(&mbid); err == nil {
|
||||
result[mbid] = te.entityType
|
||||
result[mbid] = eq.entityType
|
||||
delete(remaining, mbid)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,6 +33,12 @@ type ListenBrainzClient struct {
|
||||
limiter *RateLimiter
|
||||
cache *Cache
|
||||
logger *slog.Logger
|
||||
|
||||
// baseURL is listenBrainzBaseURL unless a test redirects it. It is
|
||||
// per-client rather than a package variable — the MB client's
|
||||
// SetBaseURL shape — so a test that points one client at an
|
||||
// httptest server does not stop being parallel-safe.
|
||||
baseURL string
|
||||
}
|
||||
|
||||
// NewListenBrainzClient creates a ListenBrainz API client.
|
||||
@@ -46,9 +52,15 @@ func NewListenBrainzClient(
|
||||
limiter: limiter,
|
||||
cache: cache,
|
||||
logger: logger,
|
||||
baseURL: listenBrainzBaseURL,
|
||||
}
|
||||
}
|
||||
|
||||
// SetBaseURL redirects this client at another host. Tests only.
|
||||
func (c *ListenBrainzClient) SetBaseURL(url string) {
|
||||
c.baseURL = strings.TrimSuffix(url, "/")
|
||||
}
|
||||
|
||||
// TopRecordingsForArtist returns the most-listened recordings for
|
||||
// the artist identified by artistMBID.
|
||||
func (c *ListenBrainzClient) TopRecordingsForArtist(
|
||||
@@ -56,7 +68,7 @@ func (c *ListenBrainzClient) TopRecordingsForArtist(
|
||||
) ([]LBTopRecording, error) {
|
||||
url := fmt.Sprintf(
|
||||
"%s/1/popularity/top-recordings-for-artist/%s",
|
||||
listenBrainzBaseURL,
|
||||
c.baseURL,
|
||||
artistMBID,
|
||||
)
|
||||
cacheKey := "lb:top-recordings:" + artistMBID
|
||||
@@ -104,7 +116,7 @@ func (c *ListenBrainzClient) TopReleaseGroupsForArtist(
|
||||
) ([]LBTopReleaseGroup, error) {
|
||||
url := fmt.Sprintf(
|
||||
"%s/1/popularity/top-release-groups-for-artist/%s",
|
||||
listenBrainzBaseURL,
|
||||
c.baseURL,
|
||||
artistMBID,
|
||||
)
|
||||
cacheKey := "lb:top-release-groups:" + artistMBID
|
||||
@@ -234,7 +246,7 @@ func (c *ListenBrainzClient) ArtistPopularity(
|
||||
return nil, nil //nolint:nilnil
|
||||
}
|
||||
|
||||
url := listenBrainzBaseURL + "/1/popularity/artist"
|
||||
url := c.baseURL + "/1/popularity/artist"
|
||||
cacheKey := "lb:pop:artist:" + hashMBIDs(mbids)
|
||||
|
||||
if data, ok := c.cache.Get(cacheKey); ok {
|
||||
@@ -265,7 +277,7 @@ func (c *ListenBrainzClient) RecordingPopularity(
|
||||
return nil, nil //nolint:nilnil
|
||||
}
|
||||
|
||||
url := listenBrainzBaseURL + "/1/popularity/recording"
|
||||
url := c.baseURL + "/1/popularity/recording"
|
||||
cacheKey := "lb:pop:recording:" + hashMBIDs(mbids)
|
||||
|
||||
if data, ok := c.cache.Get(cacheKey); ok {
|
||||
@@ -296,7 +308,7 @@ func (c *ListenBrainzClient) ReleaseGroupPopularity(
|
||||
return nil, nil //nolint:nilnil
|
||||
}
|
||||
|
||||
url := listenBrainzBaseURL + "/1/popularity/release-group"
|
||||
url := c.baseURL + "/1/popularity/release-group"
|
||||
cacheKey := "lb:pop:release-group:" + hashMBIDs(mbids)
|
||||
|
||||
if data, ok := c.cache.Get(cacheKey); ok {
|
||||
@@ -342,7 +354,7 @@ func (c *ListenBrainzClient) BatchArtistMetadata(
|
||||
return nil, nil //nolint:nilnil
|
||||
}
|
||||
|
||||
url := listenBrainzBaseURL + "/1/metadata/artist/?artist_mbids=" + strings.Join(mbids, ",")
|
||||
url := c.baseURL + "/1/metadata/artist/?artist_mbids=" + strings.Join(mbids, ",")
|
||||
cacheKey := "lb:meta:artist:" + hashMBIDs(mbids)
|
||||
|
||||
if data, ok := c.cache.Get(cacheKey); ok {
|
||||
|
||||
@@ -10,7 +10,7 @@ import (
|
||||
// LyricsResult is a single lyric-search hit, mapped from the DB layer
|
||||
// into the camelCase shape the frontend consumes.
|
||||
type LyricsResult struct {
|
||||
RecordingID int64 `json:"recordingId"`
|
||||
AudioFileID int64 `json:"audioFileId"`
|
||||
FilePath string `json:"filePath"`
|
||||
LengthMs int64 `json:"lengthMs"`
|
||||
Title string `json:"title"`
|
||||
@@ -55,7 +55,7 @@ func (e *Service) SearchLyrics(query string) []LyricsResult {
|
||||
out := make([]LyricsResult, 0, len(hits))
|
||||
for _, h := range hits {
|
||||
out = append(out, LyricsResult{
|
||||
RecordingID: h.RecordingID,
|
||||
AudioFileID: h.AudioFileID,
|
||||
FilePath: h.FilePath,
|
||||
LengthMs: h.LengthMilliseconds,
|
||||
Title: h.Title,
|
||||
@@ -67,18 +67,18 @@ func (e *Service) SearchLyrics(query string) []LyricsResult {
|
||||
return out
|
||||
}
|
||||
|
||||
// GetTrackLyrics returns lyrics for a recording. If the library
|
||||
// GetTrackLyrics returns lyrics for a file. If the library
|
||||
// already has them (from embedded tags) they're returned as-is;
|
||||
// otherwise it fetches from LRCLIB, persists them (updating the FTS
|
||||
// index), and returns them. Never returns an error to the frontend —
|
||||
// a miss just yields an empty result.
|
||||
func (e *Service) GetTrackLyrics(recordingID int64) TrackLyrics {
|
||||
stored, err := e.db.GetRecordingLyrics(recordingID)
|
||||
func (e *Service) GetTrackLyrics(audioFileID int64) TrackLyrics {
|
||||
stored, err := e.db.GetLyrics(audioFileID)
|
||||
if err == nil && stored != "" {
|
||||
return TrackLyrics{Plain: stored, Source: "embedded"}
|
||||
}
|
||||
|
||||
lookup, err := e.db.RecordingLyricLookup(recordingID)
|
||||
lookup, err := e.db.FileLyricLookup(audioFileID)
|
||||
if err != nil || lookup == nil {
|
||||
return TrackLyrics{}
|
||||
}
|
||||
@@ -150,7 +150,7 @@ func (e *Service) backfillLibraryLyrics(ctx context.Context) {
|
||||
return
|
||||
}
|
||||
|
||||
candidates, err := e.db.RecordingsMissingLyrics(lyricsBackfillBatch)
|
||||
candidates, err := e.db.FilesMissingLyrics(lyricsBackfillBatch)
|
||||
if err != nil {
|
||||
e.logger.Warn("lyrics backfill: query failed", "err", err)
|
||||
|
||||
@@ -223,8 +223,12 @@ func (e *Service) fetchAndStoreLyrics(
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := e.db.SetRecordingLyrics(c.RecordingID, lyrics.Plain); err != nil {
|
||||
e.logger.Warn("lyrics store failed", "recordingId", c.RecordingID, "err", err)
|
||||
// Marked `lrclib` rather than `tag`: these came off the network and
|
||||
// a rebuild that discards them pays for them again.
|
||||
if err := e.db.SetLyrics(
|
||||
c.AudioFileID, lyrics.Plain, "lrclib", c.RecordingMBID,
|
||||
); err != nil {
|
||||
e.logger.Warn("lyrics store failed", "audioFileId", c.AudioFileID, "err", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"database/sql/driver"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The catalog stores a MusicBrainz id as its 16 raw bytes and an entity
|
||||
// type as a small integer, rather than as the 36-character text and the
|
||||
// words the rest of the app uses.
|
||||
//
|
||||
// This is a size decision and it is a large one. Measured on a real
|
||||
// 2,052,200-row catalog: the three MBID columns and `entity_type` are
|
||||
// 220 MB of a 383 MB table, and they are carried again in every index
|
||||
// that keys on them. Converting the table and its four indexes took
|
||||
// **677 MB to 389 MB** with the same row count.
|
||||
//
|
||||
// Everything above this file still speaks strings: `SearchIndexResult`
|
||||
// carries `"artist"` and a dashed MBID, the frontend receives them, and
|
||||
// the conversion happens only where a value crosses into SQL. The
|
||||
// alternative - blobs and codes reaching the rest of the app - would
|
||||
// trade 288 MB for a type that nothing else wants.
|
||||
//
|
||||
// Two things make a mistake here loud rather than silent, which matters
|
||||
// because SQLite does not coerce between TEXT and BLOB: a query
|
||||
// comparing a blob column against a string returns *no rows* rather
|
||||
// than an error.
|
||||
//
|
||||
// - Writes are guarded by a CHECK on the column (16 bytes, or empty),
|
||||
// so a stringly write fails at the insert rather than sitting in
|
||||
// the table looking fine.
|
||||
// - Reads scan into `dbMBID`, whose Scan rejects anything that is not
|
||||
// 16 bytes or empty. A column that somehow holds text produces an
|
||||
// error instead of a garbled id.
|
||||
type dbMBID string
|
||||
|
||||
// mbidLen is the byte length of a raw MusicBrainz id.
|
||||
const mbidLen = 16
|
||||
|
||||
// Value encodes the id for storage: 16 raw bytes, or empty for "none".
|
||||
func (m dbMBID) Value() (driver.Value, error) {
|
||||
return mbidBytes(string(m)), nil
|
||||
}
|
||||
|
||||
// Scan decodes a stored id back to its canonical dashed form.
|
||||
func (m *dbMBID) Scan(src any) error {
|
||||
switch v := src.(type) {
|
||||
case nil:
|
||||
*m = ""
|
||||
|
||||
return nil
|
||||
case []byte:
|
||||
s, err := mbidFromBytes(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
*m = dbMBID(s)
|
||||
|
||||
return nil
|
||||
case string:
|
||||
// Tolerated for the one caller that reads through a view or a
|
||||
// literal: a canonical id is already the right answer.
|
||||
*m = dbMBID(v)
|
||||
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("%w: %T", errMBIDType, src)
|
||||
}
|
||||
}
|
||||
|
||||
// mbidBytes encodes a dashed MusicBrainz id as its 16 raw bytes. An
|
||||
// id that is not one - including the empty string, which is how "no
|
||||
// MBID" is spelled throughout - encodes as empty.
|
||||
func mbidBytes(s string) []byte {
|
||||
if s == "" {
|
||||
return []byte{}
|
||||
}
|
||||
|
||||
raw, err := hex.DecodeString(strings.ReplaceAll(s, "-", ""))
|
||||
if err != nil || len(raw) != mbidLen {
|
||||
return []byte{}
|
||||
}
|
||||
|
||||
return raw
|
||||
}
|
||||
|
||||
// mbidFromBytes decodes stored bytes back to the canonical dashed form.
|
||||
func mbidFromBytes(b []byte) (string, error) {
|
||||
if len(b) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
if len(b) != mbidLen {
|
||||
return "", fmt.Errorf("%w: %d bytes", errMBIDLength, len(b))
|
||||
}
|
||||
|
||||
h := hex.EncodeToString(b)
|
||||
|
||||
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:], nil
|
||||
}
|
||||
|
||||
// Entity types are stored as codes. The names are the app's, the codes
|
||||
// are the table's, and nothing outside this file should see a code.
|
||||
const (
|
||||
entityCodeArtist = 1
|
||||
entityCodeReleaseGroup = 2
|
||||
entityCodeRecording = 3
|
||||
)
|
||||
|
||||
// Entity type names, as everything above the SQL boundary spells them.
|
||||
const (
|
||||
EntityArtist = "artist"
|
||||
EntityReleaseGroup = "release_group"
|
||||
EntityRecording = "recording"
|
||||
)
|
||||
|
||||
// entityCode maps a name to its stored code. An unknown name yields 0,
|
||||
// which matches no row - the same answer the old string comparison gave
|
||||
// and the reason this is not an error.
|
||||
func entityCode(name string) int {
|
||||
switch name {
|
||||
case EntityArtist:
|
||||
return entityCodeArtist
|
||||
case EntityReleaseGroup:
|
||||
return entityCodeReleaseGroup
|
||||
case EntityRecording:
|
||||
return entityCodeRecording
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// entityName maps a stored code back to its name.
|
||||
func entityName(code int) string {
|
||||
switch code {
|
||||
case entityCodeArtist:
|
||||
return EntityArtist
|
||||
case entityCodeReleaseGroup:
|
||||
return EntityReleaseGroup
|
||||
case entityCodeRecording:
|
||||
return EntityRecording
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// dbEntityType scans a stored entity-type code as its name.
|
||||
//
|
||||
// The db prefix keeps it out of the way of the many `mbid string` and
|
||||
// `entityType string` locals in this package: these two types exist
|
||||
// only at the SQL boundary, and a name that shadowed one of those
|
||||
// would turn a conversion into a confusing compile error rather than
|
||||
// an obvious one.
|
||||
type dbEntityType string
|
||||
|
||||
// Value encodes the name for storage.
|
||||
func (e dbEntityType) Value() (driver.Value, error) {
|
||||
return int64(entityCode(string(e))), nil
|
||||
}
|
||||
|
||||
// Scan decodes a stored code back to its name.
|
||||
func (e *dbEntityType) Scan(src any) error {
|
||||
switch v := src.(type) {
|
||||
case nil:
|
||||
*e = ""
|
||||
|
||||
return nil
|
||||
case int64:
|
||||
*e = dbEntityType(entityName(int(v)))
|
||||
|
||||
return nil
|
||||
case string:
|
||||
*e = dbEntityType(v)
|
||||
|
||||
return nil
|
||||
case []byte:
|
||||
*e = dbEntityType(v)
|
||||
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("%w: %T", errEntityTypeType, src)
|
||||
}
|
||||
}
|
||||
|
||||
// Errors from the encoding boundary. They exist so a type confusion
|
||||
// here is reported rather than silently producing an id that matches
|
||||
// nothing.
|
||||
var (
|
||||
errMBIDType = errors.New("cannot scan MusicBrainz id from")
|
||||
errMBIDLength = errors.New("stored MusicBrainz id has the wrong length")
|
||||
errEntityTypeType = errors.New("cannot scan entity type from")
|
||||
)
|
||||
|
||||
// A query that names an entity type inline writes the code with the
|
||||
// name beside it - `entity_type = 1 /* artist */`. Splicing a Go
|
||||
// constant into the SQL would keep them in step automatically but makes
|
||||
// every such query a concatenation; the codes are pinned by
|
||||
// TestEntityCodesAreStable instead, because they are a storage format
|
||||
// and changing one is not a refactor.
|
||||
@@ -0,0 +1,105 @@
|
||||
package explore
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// testMBID turns a short fixture label into a well-formed MusicBrainz
|
||||
// id, and passes a real one through unchanged.
|
||||
//
|
||||
// The catalog stores an id as its 16 raw bytes and the column says so
|
||||
// (`CHECK(length(mbid) = 16)`), so a fixture can no longer call itself
|
||||
// "rh". Deriving one from the label keeps the fixtures readable — the
|
||||
// same label is the same id in a seed and in the assertion that reads
|
||||
// it back — without letting a test write something the app could not.
|
||||
func testMBID(label string) string {
|
||||
if len(mbidBytes(label)) == mbidLen {
|
||||
return label
|
||||
}
|
||||
|
||||
sum := sha256.Sum256([]byte(label))
|
||||
h := hex.EncodeToString(sum[:mbidLen])
|
||||
|
||||
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:]
|
||||
}
|
||||
|
||||
// TestEntityCodesAreStable pins the stored entity-type codes.
|
||||
//
|
||||
// They are a storage format, not an enum: the queries that name a type
|
||||
// inline write the number with the name beside it
|
||||
// (`entity_type = 1 /* artist */`), so changing one here without
|
||||
// changing those would leave the catalog answering the wrong questions
|
||||
// silently.
|
||||
func TestEntityCodesAreStable(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for name, want := range map[string]int{
|
||||
EntityArtist: 1,
|
||||
EntityReleaseGroup: 2,
|
||||
EntityRecording: 3,
|
||||
} {
|
||||
if got := entityCode(name); got != want {
|
||||
t.Errorf("entityCode(%q) = %d, want %d", name, got, want)
|
||||
}
|
||||
|
||||
if back := entityName(want); back != name {
|
||||
t.Errorf("entityName(%d) = %q, want %q", want, back, name)
|
||||
}
|
||||
}
|
||||
|
||||
if got := entityCode("nonsense"); got != 0 {
|
||||
t.Errorf("entityCode of an unknown name = %d, want 0 (matches nothing)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMBIDRoundTrip pins the encoding both ways, including the two
|
||||
// values that are not ids: empty, which is how "no MBID" is spelled
|
||||
// everywhere, and rubbish, which must not become a valid-looking id.
|
||||
func TestMBIDRoundTrip(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const canonical = "c0b2500e-0cef-4130-9b13-1b9d9a2f2c07"
|
||||
|
||||
encoded := mbidBytes(canonical)
|
||||
if len(encoded) != mbidLen {
|
||||
t.Fatalf("encoded length = %d, want %d", len(encoded), mbidLen)
|
||||
}
|
||||
|
||||
back, err := mbidFromBytes(encoded)
|
||||
if err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
|
||||
if back != canonical {
|
||||
t.Errorf("round trip = %q, want %q", back, canonical)
|
||||
}
|
||||
|
||||
if got := mbidBytes(""); len(got) != 0 {
|
||||
t.Errorf("empty encoded to %d bytes, want 0", len(got))
|
||||
}
|
||||
|
||||
if got := mbidBytes("not-an-mbid"); len(got) != 0 {
|
||||
t.Errorf("rubbish encoded to %d bytes, want 0", len(got))
|
||||
}
|
||||
|
||||
// A stored value of the wrong length is an error, not a guess.
|
||||
if _, err := mbidFromBytes([]byte{1, 2, 3}); err == nil {
|
||||
t.Error("decoding three bytes should fail")
|
||||
}
|
||||
}
|
||||
|
||||
// TestMBIDScanRejectsText is the guard for the failure this encoding
|
||||
// could otherwise hide: SQLite does not coerce TEXT to BLOB, so a
|
||||
// column holding the old 36-character form would silently compare equal
|
||||
// to nothing. Scanning it must say so.
|
||||
func TestMBIDScanRejectsText(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var m dbMBID
|
||||
|
||||
if err := m.Scan([]byte("c0b2500e-0cef-4130-9b13-1b9d9a2f2c07")); err == nil {
|
||||
t.Error("scanning 36 bytes as an MBID should fail")
|
||||
}
|
||||
}
|
||||
+72
-11
@@ -107,12 +107,11 @@ func (e *Service) mixSeedProfile(
|
||||
artistCounts[artist.ArtistMbid]++
|
||||
artistNames[artist.ArtistMbid] = artist.ArtistName
|
||||
}
|
||||
}
|
||||
|
||||
names, err := e.db.ReadQueries.GetGenreNamesByFilePath(ctx, p)
|
||||
if err == nil {
|
||||
for _, g := range names {
|
||||
genres[g] = true
|
||||
}
|
||||
for _, names := range e.genresByPath(ctx, seedPaths) {
|
||||
for _, g := range names {
|
||||
genres[g] = true
|
||||
}
|
||||
}
|
||||
|
||||
@@ -148,6 +147,12 @@ func (e *Service) mixCandidates(
|
||||
|
||||
candidates := map[string]float64{}
|
||||
|
||||
// Every candidate path's genres, in one query. This used to be a
|
||||
// single-row lookup *per candidate* inside two nested loops -
|
||||
// twenty seed artists by twenty similar artists by thirty paths is
|
||||
// twelve thousand queries to assemble one mix.
|
||||
pathGenres := e.genresByPath(ctx, e.similarArtistPaths(ctx, artistCounts))
|
||||
|
||||
for seedArtistMBID, count := range artistCounts {
|
||||
similar, err := e.SimilarArtists(seedArtistMBID)
|
||||
if err != nil {
|
||||
@@ -178,13 +183,11 @@ func (e *Service) mixCandidates(
|
||||
continue
|
||||
}
|
||||
|
||||
if names, err := e.db.ReadQueries.GetGenreNamesByFilePath(ctx, p); err == nil {
|
||||
for _, g := range names {
|
||||
if seedGenres[g] {
|
||||
weight += mixGenreBoost
|
||||
for _, g := range pathGenres[p] {
|
||||
if seedGenres[g] {
|
||||
weight += mixGenreBoost
|
||||
|
||||
break
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
@@ -196,6 +199,64 @@ func (e *Service) mixCandidates(
|
||||
return candidates
|
||||
}
|
||||
|
||||
// genresByPath returns the genres of many files in one query.
|
||||
func (e *Service) genresByPath(
|
||||
ctx context.Context, paths []string,
|
||||
) map[string][]string {
|
||||
out := make(map[string][]string, len(paths))
|
||||
|
||||
if len(paths) == 0 {
|
||||
return out
|
||||
}
|
||||
|
||||
rows, err := e.db.ReadQueries.GetGenreNamesByFilePaths(ctx, paths)
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
|
||||
for _, row := range rows {
|
||||
out[row.FilePath] = append(out[row.FilePath], row.Name)
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// similarArtistPaths collects every owned file by an artist similar to
|
||||
// one of the seeds, so their genres can be fetched in one go.
|
||||
func (e *Service) similarArtistPaths(
|
||||
ctx context.Context, artistCounts map[string]int,
|
||||
) []string {
|
||||
var paths []string
|
||||
|
||||
for seedArtistMBID := range artistCounts {
|
||||
similar, err := e.SimilarArtists(seedArtistMBID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if len(similar) > mixSimilarArtistsPerSeed {
|
||||
similar = similar[:mixSimilarArtistsPerSeed]
|
||||
}
|
||||
|
||||
for _, sim := range similar {
|
||||
if sim.ArtistMBID == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
p, err := e.db.ReadQueries.GetFilePathsByArtistMBID(
|
||||
ctx, sql.NullString{String: sim.ArtistMBID, Valid: true},
|
||||
)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
paths = append(paths, p...)
|
||||
}
|
||||
}
|
||||
|
||||
return paths
|
||||
}
|
||||
|
||||
// weightedSample picks up to n distinct keys from weights without
|
||||
// replacement, biased toward higher weight (roulette-wheel selection).
|
||||
// A key with zero or negative weight is never picked.
|
||||
|
||||
@@ -23,69 +23,14 @@ func seedMixTrack(
|
||||
|
||||
fp := fmt.Sprintf("/music/%s/track%d.mp3", artistName, id)
|
||||
|
||||
_, err := db.ExecContext(
|
||||
"INSERT INTO artists (id, name, mbid) VALUES (?, ?, ?) "+
|
||||
"ON CONFLICT(name) DO NOTHING",
|
||||
id, artistName, artistMBID,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("insert artist: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(
|
||||
"INSERT INTO artist_credit (id, text) VALUES (?, ?) "+
|
||||
"ON CONFLICT(text) DO NOTHING",
|
||||
id, artistName,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("insert artist_credit: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(
|
||||
"INSERT OR IGNORE INTO artist_credit_artist (artist_id, credit_id) VALUES (?, ?)",
|
||||
id, id,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("insert artist_credit_artist: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(
|
||||
"INSERT INTO recordings (id, name, artist_credit_id) VALUES (?, ?, ?)",
|
||||
id, fmt.Sprintf("Track %d", id), id,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("insert recording: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(
|
||||
"INSERT INTO audio_files (id, file_path, length_milliseconds, file_type_id, recording_id) "+
|
||||
"VALUES (?, ?, 180000, 0, ?)",
|
||||
id, fp, id,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("insert audio_file: %v", err)
|
||||
}
|
||||
|
||||
for _, g := range genreNames {
|
||||
var genreID int64
|
||||
|
||||
row := db.QueryRowWriter(
|
||||
"INSERT INTO genres (name) VALUES (?) "+
|
||||
"ON CONFLICT(name) DO UPDATE SET name = name RETURNING id",
|
||||
g,
|
||||
)
|
||||
if err := row.Scan(&genreID); err != nil {
|
||||
t.Fatalf("upsert genre %q: %v", g, err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(
|
||||
"INSERT OR IGNORE INTO recording_genres (recording_id, genre_id) VALUES (?, ?)",
|
||||
id, genreID,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("insert recording_genre: %v", err)
|
||||
}
|
||||
}
|
||||
database.InsertTestTrack(t, db, database.TestTrack{
|
||||
FilePath: fp,
|
||||
Title: fmt.Sprintf("Track %d", id),
|
||||
Artist: artistName,
|
||||
ArtistMBID: artistMBID,
|
||||
Genres: genreNames,
|
||||
LengthMs: 180000,
|
||||
})
|
||||
|
||||
return fp
|
||||
}
|
||||
|
||||
@@ -13,18 +13,27 @@ import (
|
||||
"go.uploadedlobster.com/musicbrainzws2"
|
||||
)
|
||||
|
||||
// How long a MusicBrainz answer is kept.
|
||||
//
|
||||
// The rule is what the answer is *about*, not how big it is. A search
|
||||
// is a ranking and shifts; an entity is a fact about a record that was
|
||||
// published years ago and does not. Entity data used to expire after a
|
||||
// week, which meant a fully-populated artist page re-fetched itself
|
||||
// every week forever - on a real install, 251 of 2,930 cached rows were
|
||||
// already expired and waiting to be paid for again. The bytes are
|
||||
// already on disk; re-fetching them buys nothing and spends someone
|
||||
// else's rate limit.
|
||||
const (
|
||||
// cacheTTLSearch is the TTL for search results (results may shift).
|
||||
cacheTTLSearch = 24 * time.Hour
|
||||
// cacheTTLEntity is the TTL for lookup/browse results (entity data
|
||||
// changes rarely).
|
||||
cacheTTLEntity = 7 * 24 * time.Hour
|
||||
// cacheTTLEntity is the TTL for lookup/browse results. An artist's
|
||||
// name, country and relations, a release group's title and date:
|
||||
// these change on the order of never, and a wrong one is corrected
|
||||
// by the next catalog artifact rather than by an expiry.
|
||||
cacheTTLEntity = 365 * 24 * time.Hour
|
||||
// cacheTTLReleases is the TTL for a release group's releases +
|
||||
// tracklists. This data is effectively immutable once published, so
|
||||
// it's cached far longer than other entities: it's the local store
|
||||
// that keeps an album page's cold fetch a once-per-quarter event
|
||||
// rather than a weekly one.
|
||||
cacheTTLReleases = 90 * 24 * time.Hour
|
||||
// tracklists. Effectively immutable once published.
|
||||
cacheTTLReleases = 365 * 24 * time.Hour
|
||||
)
|
||||
|
||||
// MusicBrainzClient wraps the musicbrainzws2 library with a local
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"yellowjacket/backend/database"
|
||||
"yellowjacket/backend/database/sql/sqlcgen"
|
||||
)
|
||||
|
||||
// TestPruneStaleLocalCrossReferences verifies that an explore_index row
|
||||
@@ -21,10 +20,17 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, slog.Default())
|
||||
|
||||
// A library artist that still exists.
|
||||
artist, err := db.Queries.UpsertArtist(t.Context(), "Still Owned")
|
||||
// A library artist that still exists - which now means one with a
|
||||
// file behind it. An artist row on its own is not ownership; that
|
||||
// was the whole bug.
|
||||
database.InsertTestTrack(t, db, database.TestTrack{
|
||||
FilePath: "/music/still-owned.mp3",
|
||||
Artist: "Still Owned",
|
||||
})
|
||||
|
||||
artist, err := db.Queries.GetArtistByName(t.Context(), "Still Owned")
|
||||
if err != nil {
|
||||
t.Fatalf("upsert artist: %v", err)
|
||||
t.Fatalf("read seeded artist: %v", err)
|
||||
}
|
||||
|
||||
// Two explore_index artist rows: one pointing at the still-existing
|
||||
@@ -33,19 +39,15 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
|
||||
seedArtist := func(mbid, title string, localID int64) {
|
||||
t.Helper()
|
||||
|
||||
if _, err := db.ExecContext(
|
||||
upsertIndexSQL,
|
||||
"artist", mbid, title, title, mbid, "",
|
||||
0, 0,
|
||||
0, "", "",
|
||||
"", "", "",
|
||||
"", "", "", "",
|
||||
1, 0,
|
||||
localID, 0, 0,
|
||||
0,
|
||||
); err != nil {
|
||||
t.Fatalf("seed explore_index row for %q: %v", mbid, err)
|
||||
}
|
||||
seedIndexResult(t, db, SearchIndexResult{
|
||||
EntityType: EntityArtist,
|
||||
MBID: testMBID(mbid),
|
||||
Title: title,
|
||||
ArtistName: title,
|
||||
ArtistMBID: testMBID(mbid),
|
||||
InLibrary: true,
|
||||
LocalArtistID: localID,
|
||||
})
|
||||
}
|
||||
|
||||
seedArtist("still-owned-mbid", "Still Owned", artist.ID)
|
||||
@@ -53,7 +55,7 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
|
||||
|
||||
si.pruneStaleLocalCrossReferences()
|
||||
|
||||
stillOwned := si.LookupArtistByMBID("still-owned-mbid")
|
||||
stillOwned := si.LookupArtistByMBID(testMBID("still-owned-mbid"))
|
||||
if stillOwned == nil {
|
||||
t.Fatal("expected still-owned artist row to survive pruning")
|
||||
}
|
||||
@@ -67,7 +69,7 @@ func TestPruneStaleLocalCrossReferences(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
removed := si.LookupArtistByMBID("removed-mbid")
|
||||
removed := si.LookupArtistByMBID(testMBID("removed-mbid"))
|
||||
if removed == nil {
|
||||
t.Fatal("expected removed-artist row to still exist (only cross-references cleared)")
|
||||
}
|
||||
@@ -90,51 +92,18 @@ func TestUnenrichedLibraryArtistMBIDs_OrdersByOwnedTrackCount(t *testing.T) {
|
||||
|
||||
db := database.NewTestDB(t)
|
||||
si := NewSearchIndex(db, nil, nil, slog.Default())
|
||||
q := db.Queries
|
||||
ctx := t.Context()
|
||||
|
||||
seedArtistWithTracks := func(name, mbid string, trackCount int) {
|
||||
t.Helper()
|
||||
|
||||
artist, err := q.UpsertArtist(ctx, name)
|
||||
if err != nil {
|
||||
t.Fatalf("upsert artist %q: %v", name, err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext("UPDATE artists SET mbid = ? WHERE id = ?", mbid, artist.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("set mbid for %q: %v", name, err)
|
||||
}
|
||||
|
||||
ac, err := q.UpsertArtistCredit(ctx, name)
|
||||
if err != nil {
|
||||
t.Fatalf("upsert artist credit %q: %v", name, err)
|
||||
}
|
||||
|
||||
if _, err := q.CreateArtistCreditArtist(ctx, sqlcgen.CreateArtistCreditArtistParams{
|
||||
ArtistID: artist.ID,
|
||||
CreditID: ac.ID,
|
||||
}); err != nil {
|
||||
t.Fatalf("link artist credit artist %q: %v", name, err)
|
||||
}
|
||||
|
||||
for i := range trackCount {
|
||||
rec, err := q.CreateRecordingFull(ctx, sqlcgen.CreateRecordingFullParams{
|
||||
Name: name,
|
||||
ArtistCreditID: ac.ID,
|
||||
database.InsertTestTrack(t, db, database.TestTrack{
|
||||
FilePath: name + "/" + string(rune('a'+i)) + ".mp3",
|
||||
Title: name,
|
||||
Artist: name,
|
||||
ArtistMBID: mbid,
|
||||
LengthMs: 180000,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create recording for %q: %v", name, err)
|
||||
}
|
||||
|
||||
if _, err := q.CreateAudioFile(ctx, sqlcgen.CreateAudioFileParams{
|
||||
FilePath: name + "/" + string(rune('a'+i)) + ".mp3",
|
||||
LengthMilliseconds: 180000,
|
||||
RecordingID: rec.ID,
|
||||
Basename: string(rune('a'+i)) + ".mp3",
|
||||
}); err != nil {
|
||||
t.Fatalf("create audio file for %q: %v", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+312
-258
@@ -2,6 +2,7 @@ package explore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -146,6 +147,15 @@ type SearchIndexResult struct {
|
||||
SecondaryTypes string `json:"secondaryTypes"` // comma-separated
|
||||
ReleaseDate string `json:"releaseDate"`
|
||||
|
||||
// TotalTracks is the canonical release's track count, or 0 for
|
||||
// "the catalog does not say". It is the denominator the album
|
||||
// page cannot get from the files when the library holds no tags
|
||||
// for the album, and it is deliberately only a denominator: the
|
||||
// tracklist itself is not shipped, because the per-artist track
|
||||
// budget truncates it and a truncated tracklist is a confident
|
||||
// lie about which tracks exist.
|
||||
TotalTracks int `json:"totalTracks"`
|
||||
|
||||
// Artist-specific fields (from MB lookup).
|
||||
ArtistType string `json:"artistType"`
|
||||
Country string `json:"country"`
|
||||
@@ -357,8 +367,9 @@ func (si *SearchIndex) EnsureArtistDiscography(ctx context.Context, artistMBID s
|
||||
// discography fetched, so EnsureArtistDiscography can skip the network.
|
||||
func (si *SearchIndex) artistDiscogFetched(mbid string) bool {
|
||||
rows, err := si.db.QueryContext(
|
||||
"SELECT 1 FROM explore_index WHERE entity_type = 'artist' AND mbid = ? AND discog_fetched = 1 LIMIT 1",
|
||||
mbid,
|
||||
"SELECT 1 FROM explore_index WHERE entity_type = 1 /* artist */ "+
|
||||
"AND mbid = ? AND discog_fetched = 1 LIMIT 1",
|
||||
dbMBID(mbid),
|
||||
)
|
||||
if err != nil {
|
||||
return false
|
||||
@@ -393,11 +404,9 @@ func (si *SearchIndex) unenrichedLibraryArtistMBIDs(limit int) []string {
|
||||
SELECT a.mbid
|
||||
FROM artists a
|
||||
LEFT JOIN explore_index ei
|
||||
ON ei.entity_type = 'artist' AND ei.mbid = a.mbid
|
||||
ON ei.entity_type = 1 /* artist */ AND ei.mbid = unhex(replace(a.mbid, '-', ''))
|
||||
LEFT JOIN artist_enrichment ae ON ae.artist_mbid = a.mbid
|
||||
LEFT JOIN artist_credit_artist aca ON aca.artist_id = a.id
|
||||
LEFT JOIN recordings r ON r.artist_credit_id = aca.credit_id
|
||||
LEFT JOIN audio_files af ON af.recording_id = r.id
|
||||
LEFT JOIN audio_files af ON af.artist_id = a.id
|
||||
WHERE a.mbid IS NOT NULL AND a.mbid != ''
|
||||
AND (ei.id IS NULL OR ei.discog_fetched = 0
|
||||
OR ae.browsed_at IS NULL)
|
||||
@@ -560,11 +569,17 @@ func (si *SearchIndex) backfillOneArtist(
|
||||
// preferring the index title, then the local library, then the MBID
|
||||
// itself. Used to seed the discography fetch's artist entry.
|
||||
func (si *SearchIndex) artistDisplayName(mbid string) string {
|
||||
for _, q := range []string{
|
||||
"SELECT title FROM explore_index WHERE entity_type = 'artist' AND mbid = ? AND title != '' LIMIT 1",
|
||||
"SELECT name FROM artists WHERE mbid = ? AND name != '' LIMIT 1",
|
||||
// The two tables spell an MBID differently: the catalog stores raw
|
||||
// bytes, the library stores text. Each query brings its own form.
|
||||
for _, q := range []struct {
|
||||
sql string
|
||||
arg any
|
||||
}{
|
||||
{"SELECT title FROM explore_index WHERE entity_type = 1 /* artist */ " +
|
||||
"AND mbid = ? AND title != '' LIMIT 1", dbMBID(mbid)},
|
||||
{"SELECT name FROM artists WHERE mbid = ? AND name != '' LIMIT 1", mbid},
|
||||
} {
|
||||
rows, err := si.db.QueryContext(q, mbid)
|
||||
rows, err := si.db.QueryContext(q.sql, q.arg)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
@@ -721,17 +736,17 @@ func (si *SearchIndex) refreshStatusCounts() {
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
et string
|
||||
et dbEntityType
|
||||
count int
|
||||
)
|
||||
|
||||
if err := rows.Scan(&et, &count); err == nil {
|
||||
switch et {
|
||||
case "artist":
|
||||
switch string(et) {
|
||||
case EntityArtist:
|
||||
artists = count
|
||||
case "recording":
|
||||
case EntityRecording:
|
||||
recordings = count
|
||||
case "release_group":
|
||||
case EntityReleaseGroup:
|
||||
rgs = count
|
||||
}
|
||||
}
|
||||
@@ -878,7 +893,7 @@ func (si *SearchIndex) GetPopularity(mbid string) int {
|
||||
|
||||
rows, err := si.db.QueryContext(
|
||||
"SELECT popularity FROM explore_index WHERE mbid = ? LIMIT 1",
|
||||
mbid,
|
||||
dbMBID(mbid),
|
||||
)
|
||||
if err != nil {
|
||||
return 0
|
||||
@@ -917,7 +932,7 @@ func (si *SearchIndex) GetPopularityBatch(mbids []string) *PopularityBatchResult
|
||||
|
||||
for i, m := range mbids {
|
||||
placeholders[i] = "?"
|
||||
args[i] = m
|
||||
args[i] = dbMBID(m)
|
||||
}
|
||||
|
||||
query := "SELECT mbid, popularity, listener_count, in_library FROM explore_index WHERE mbid IN (" +
|
||||
@@ -942,13 +957,15 @@ func (si *SearchIndex) GetPopularityBatch(mbids []string) *PopularityBatchResult
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
mbid string
|
||||
id dbMBID
|
||||
pop int
|
||||
listeners int
|
||||
inLib int
|
||||
)
|
||||
|
||||
if err := rows.Scan(&mbid, &pop, &listeners, &inLib); err == nil {
|
||||
if err := rows.Scan(&id, &pop, &listeners, &inLib); err == nil {
|
||||
mbid := string(id)
|
||||
|
||||
existing, ok := result.Popularity[mbid]
|
||||
if !ok || pop > existing {
|
||||
result.Popularity[mbid] = pop
|
||||
@@ -979,7 +996,7 @@ func (si *SearchIndex) IsInLibrary(mbid string) bool {
|
||||
|
||||
rows, err := si.db.QueryContext(
|
||||
"SELECT in_library FROM explore_index WHERE mbid = ? AND in_library = 1 LIMIT 1",
|
||||
mbid,
|
||||
dbMBID(mbid),
|
||||
)
|
||||
if err != nil {
|
||||
return false
|
||||
@@ -999,8 +1016,8 @@ func (si *SearchIndex) LookupArtistByMBID(mbid string) *SearchIndexResult {
|
||||
artist_type, country, disambiguation, sort_name, aliases,
|
||||
in_library, is_similar, COALESCE(local_artist_id, 0)
|
||||
FROM explore_index
|
||||
WHERE mbid = ? AND entity_type = 'artist' LIMIT 1`,
|
||||
mbid,
|
||||
WHERE mbid = ? AND entity_type = 1 /* artist */ LIMIT 1`,
|
||||
dbMBID(mbid),
|
||||
)
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -1013,18 +1030,22 @@ func (si *SearchIndex) LookupArtistByMBID(mbid string) *SearchIndexResult {
|
||||
}
|
||||
|
||||
r := SearchIndexResult{
|
||||
EntityType: "artist",
|
||||
EntityType: EntityArtist,
|
||||
MBID: mbid,
|
||||
}
|
||||
|
||||
var artist dbMBID
|
||||
|
||||
if err := rows.Scan(
|
||||
&r.Title, &r.ArtistName, &r.ArtistMBID, &r.Popularity, &r.ListenerCount,
|
||||
&r.Title, &r.ArtistName, &artist, &r.Popularity, &r.ListenerCount,
|
||||
&r.ArtistType, &r.Country, &r.Disambiguation, &r.SortName, &r.Aliases,
|
||||
&r.InLibrary, &r.IsSimilar, &r.LocalArtistID,
|
||||
); err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
r.ArtistMBID = string(artist)
|
||||
|
||||
return &r
|
||||
}
|
||||
|
||||
@@ -1070,13 +1091,13 @@ func (si *SearchIndex) ReleaseGroupMBIDsForCAAReleaseMBIDs(
|
||||
|
||||
for i, m := range filtered {
|
||||
placeholders[i] = "?"
|
||||
args[i] = m
|
||||
args[i] = dbMBID(m)
|
||||
}
|
||||
|
||||
query := `SELECT caa_release_mbid, mbid
|
||||
FROM explore_index
|
||||
WHERE entity_type = 'release_group'
|
||||
AND caa_release_mbid != ''
|
||||
WHERE entity_type = 2 /* release_group */
|
||||
AND caa_release_mbid != x''
|
||||
AND caa_release_mbid IN (` + strings.Join(placeholders, ",") + `)`
|
||||
|
||||
rows, err := si.db.QueryContext(query, args...)
|
||||
@@ -1089,9 +1110,9 @@ func (si *SearchIndex) ReleaseGroupMBIDsForCAAReleaseMBIDs(
|
||||
out := make(map[string]string, len(filtered))
|
||||
|
||||
for rows.Next() {
|
||||
var caaMBID, rgMBID string
|
||||
var caaMBID, rgMBID dbMBID
|
||||
if err := rows.Scan(&caaMBID, &rgMBID); err == nil {
|
||||
out[caaMBID] = rgMBID
|
||||
out[string(caaMBID)] = string(rgMBID)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1102,11 +1123,11 @@ func (si *SearchIndex) ReleaseGroupMBIDsForCAAReleaseMBIDs(
|
||||
func (si *SearchIndex) LookupReleaseGroupByMBID(mbid string) *SearchIndexResult {
|
||||
rows, err := si.db.QueryContext(
|
||||
`SELECT title, artist_name, artist_mbid, popularity, listener_count,
|
||||
primary_type, secondary_types, release_date,
|
||||
primary_type, secondary_types, release_date, total_tracks,
|
||||
in_library, COALESCE(local_release_group_id, 0), discog_fetched
|
||||
FROM explore_index
|
||||
WHERE mbid = ? AND entity_type = 'release_group' LIMIT 1`,
|
||||
mbid,
|
||||
WHERE mbid = ? AND entity_type = 2 /* release_group */ LIMIT 1`,
|
||||
dbMBID(mbid),
|
||||
)
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -1119,13 +1140,15 @@ func (si *SearchIndex) LookupReleaseGroupByMBID(mbid string) *SearchIndexResult
|
||||
}
|
||||
|
||||
r := SearchIndexResult{
|
||||
EntityType: "release_group",
|
||||
EntityType: EntityReleaseGroup,
|
||||
MBID: mbid,
|
||||
}
|
||||
|
||||
var artist dbMBID
|
||||
|
||||
if err := rows.Scan(
|
||||
&r.Title, &r.ArtistName, &r.ArtistMBID, &r.Popularity, &r.ListenerCount,
|
||||
&r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate,
|
||||
&r.Title, &r.ArtistName, &artist, &r.Popularity, &r.ListenerCount,
|
||||
&r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate, &r.TotalTracks,
|
||||
&r.InLibrary, &r.LocalReleaseGroupID, &r.DiscogFetched,
|
||||
); err != nil {
|
||||
return nil
|
||||
@@ -1166,10 +1189,10 @@ func (si *SearchIndex) TopRecordingsByArtist(artistMBID string, limit int) []Sea
|
||||
duration, caa_release_mbid, release_name,
|
||||
in_library, COALESCE(local_recording_id, 0)
|
||||
FROM explore_index
|
||||
WHERE artist_mbid = ? AND entity_type = 'recording'
|
||||
WHERE artist_mbid = ? AND entity_type = 3 /* recording */
|
||||
ORDER BY popularity DESC
|
||||
LIMIT ?`,
|
||||
artistMBID, limit,
|
||||
dbMBID(artistMBID), limit,
|
||||
)
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -1180,13 +1203,19 @@ func (si *SearchIndex) TopRecordingsByArtist(artistMBID string, limit int) []Sea
|
||||
var results []SearchIndexResult
|
||||
|
||||
for rows.Next() {
|
||||
var r SearchIndexResult
|
||||
var (
|
||||
r SearchIndexResult
|
||||
id, caa dbMBID
|
||||
)
|
||||
|
||||
if err := rows.Scan(
|
||||
&r.MBID, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
|
||||
&r.Duration, &r.CAAReleaseMBID, &r.ReleaseName,
|
||||
&id, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
|
||||
&r.Duration, &caa, &r.ReleaseName,
|
||||
&r.InLibrary, &r.LocalRecordingID,
|
||||
); err == nil {
|
||||
r.EntityType = "recording"
|
||||
r.MBID = string(id)
|
||||
r.CAAReleaseMBID = string(caa)
|
||||
r.EntityType = EntityRecording
|
||||
r.ArtistMBID = artistMBID
|
||||
results = append(results, r)
|
||||
}
|
||||
@@ -1205,10 +1234,10 @@ func (si *SearchIndex) TopReleaseGroupsByArtist(artistMBID string, limit int) []
|
||||
primary_type, secondary_types, release_date,
|
||||
in_library, COALESCE(local_release_group_id, 0)
|
||||
FROM explore_index
|
||||
WHERE artist_mbid = ? AND entity_type = 'release_group'
|
||||
WHERE artist_mbid = ? AND entity_type = 2 /* release_group */
|
||||
ORDER BY popularity DESC
|
||||
LIMIT ?`,
|
||||
artistMBID, limit,
|
||||
dbMBID(artistMBID), limit,
|
||||
)
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -1219,13 +1248,18 @@ func (si *SearchIndex) TopReleaseGroupsByArtist(artistMBID string, limit int) []
|
||||
var results []SearchIndexResult
|
||||
|
||||
for rows.Next() {
|
||||
var r SearchIndexResult
|
||||
var (
|
||||
r SearchIndexResult
|
||||
id dbMBID
|
||||
)
|
||||
|
||||
if err := rows.Scan(
|
||||
&r.MBID, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
|
||||
&id, &r.Title, &r.ArtistName, &r.Popularity, &r.ListenerCount,
|
||||
&r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate,
|
||||
&r.InLibrary, &r.LocalReleaseGroupID,
|
||||
); err == nil {
|
||||
r.EntityType = "release_group"
|
||||
r.MBID = string(id)
|
||||
r.EntityType = EntityReleaseGroup
|
||||
r.ArtistMBID = artistMBID
|
||||
results = append(results, r)
|
||||
}
|
||||
@@ -1315,29 +1349,21 @@ func (si *SearchIndex) ExactMatches(query string, perCategory int) []SearchIndex
|
||||
// its partial expression index (idx_explore_title_lower /
|
||||
// idx_explore_artist_lower). Ordering is done in Go below since
|
||||
// an ORDER BY here would also defeat the index seek.
|
||||
//
|
||||
// The popularity clause **must match those indexes' predicate** or
|
||||
// the seek becomes a scan of two million rows. It is the champion
|
||||
// set: what the user owns, plus what is popular enough to be worth
|
||||
// an exact-match boost. Anything below the floor is still found by
|
||||
// the FTS tiers; it just does not jump the queue.
|
||||
rows, err := si.db.QueryContext(`
|
||||
SELECT entity_type, mbid, title, artist_name, artist_mbid,
|
||||
popularity, listener_count, duration, primary_type,
|
||||
secondary_types, release_date, caa_release_mbid,
|
||||
release_name, artist_type, country, disambiguation,
|
||||
sort_name, in_library, is_similar,
|
||||
COALESCE(local_artist_id, 0),
|
||||
COALESCE(local_release_group_id, 0),
|
||||
COALESCE(local_recording_id, 0)
|
||||
SELECT `+indexRowColumns+`
|
||||
FROM explore_index
|
||||
WHERE LOWER(title) = ? AND popularity > 0
|
||||
WHERE LOWER(title) = ? AND (popularity >= ? OR in_library = 1)
|
||||
UNION
|
||||
SELECT entity_type, mbid, title, artist_name, artist_mbid,
|
||||
popularity, listener_count, duration, primary_type,
|
||||
secondary_types, release_date, caa_release_mbid,
|
||||
release_name, artist_type, country, disambiguation,
|
||||
sort_name, in_library, is_similar,
|
||||
COALESCE(local_artist_id, 0),
|
||||
COALESCE(local_release_group_id, 0),
|
||||
COALESCE(local_recording_id, 0)
|
||||
SELECT `+indexRowColumns+`
|
||||
FROM explore_index
|
||||
WHERE LOWER(artist_name) = ? AND popularity > 0
|
||||
`, q, q)
|
||||
WHERE LOWER(artist_name) = ? AND (popularity >= ? OR in_library = 1)
|
||||
`, q, championPopThreshold, q, championPopThreshold)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
@@ -1349,14 +1375,7 @@ func (si *SearchIndex) ExactMatches(query string, perCategory int) []SearchIndex
|
||||
for rows.Next() {
|
||||
var r SearchIndexResult
|
||||
|
||||
if err := rows.Scan(
|
||||
&r.EntityType, &r.MBID, &r.Title, &r.ArtistName, &r.ArtistMBID,
|
||||
&r.Popularity, &r.ListenerCount, &r.Duration, &r.PrimaryType,
|
||||
&r.SecondaryTypes, &r.ReleaseDate, &r.CAAReleaseMBID,
|
||||
&r.ReleaseName, &r.ArtistType, &r.Country, &r.Disambiguation,
|
||||
&r.SortName, &r.InLibrary, &r.IsSimilar,
|
||||
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
|
||||
); err != nil {
|
||||
if err := scanIndexRow(rows, &r); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -1623,14 +1642,7 @@ func (si *SearchIndex) rowsByIDs(ctx context.Context, ids []int64) []SearchIndex
|
||||
}
|
||||
|
||||
query := `
|
||||
SELECT id, entity_type, mbid, title, artist_name, artist_mbid,
|
||||
popularity, listener_count, duration, primary_type,
|
||||
secondary_types, release_date, caa_release_mbid,
|
||||
release_name, artist_type, country, disambiguation,
|
||||
sort_name, in_library, is_similar,
|
||||
COALESCE(local_artist_id, 0),
|
||||
COALESCE(local_release_group_id, 0),
|
||||
COALESCE(local_recording_id, 0)
|
||||
SELECT id, ` + indexRowColumns + `
|
||||
FROM explore_index
|
||||
WHERE id IN (` + strings.Join(placeholders, ",") + `)`
|
||||
|
||||
@@ -1653,14 +1665,7 @@ func (si *SearchIndex) rowsByIDs(ctx context.Context, ids []int64) []SearchIndex
|
||||
r SearchIndexResult
|
||||
)
|
||||
|
||||
if err := rows.Scan(
|
||||
&id, &r.EntityType, &r.MBID, &r.Title, &r.ArtistName, &r.ArtistMBID,
|
||||
&r.Popularity, &r.ListenerCount, &r.Duration, &r.PrimaryType,
|
||||
&r.SecondaryTypes, &r.ReleaseDate, &r.CAAReleaseMBID,
|
||||
&r.ReleaseName, &r.ArtistType, &r.Country, &r.Disambiguation,
|
||||
&r.SortName, &r.InLibrary, &r.IsSimilar,
|
||||
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
|
||||
); err != nil {
|
||||
if err := scanIndexRow(rows, &r, &id); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -1742,15 +1747,7 @@ func (si *SearchIndex) queryFTS(
|
||||
) []SearchIndexResult {
|
||||
// ftsTable is a trusted in-package constant, never user input.
|
||||
sqlText := fmt.Sprintf(`
|
||||
SELECT i.entity_type, i.mbid, i.title, i.artist_name,
|
||||
i.artist_mbid, i.popularity, i.listener_count,
|
||||
i.duration, i.primary_type, i.secondary_types, i.release_date,
|
||||
i.caa_release_mbid, i.release_name,
|
||||
i.artist_type, i.country, i.disambiguation, i.sort_name,
|
||||
i.in_library, i.is_similar,
|
||||
COALESCE(i.local_artist_id, 0),
|
||||
COALESCE(i.local_release_group_id, 0),
|
||||
COALESCE(i.local_recording_id, 0)
|
||||
SELECT `+indexRowColumnsFor("i")+`
|
||||
FROM explore_index i
|
||||
JOIN %[1]s f ON f.rowid = i.id
|
||||
WHERE %[1]s MATCH ?
|
||||
@@ -1783,15 +1780,7 @@ func (si *SearchIndex) queryFTS(
|
||||
for rows.Next() {
|
||||
var r SearchIndexResult
|
||||
|
||||
if err := rows.Scan(
|
||||
&r.EntityType, &r.MBID, &r.Title, &r.ArtistName,
|
||||
&r.ArtistMBID, &r.Popularity, &r.ListenerCount,
|
||||
&r.Duration, &r.PrimaryType, &r.SecondaryTypes, &r.ReleaseDate,
|
||||
&r.CAAReleaseMBID, &r.ReleaseName,
|
||||
&r.ArtistType, &r.Country, &r.Disambiguation, &r.SortName,
|
||||
&r.InLibrary, &r.IsSimilar,
|
||||
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
|
||||
); err != nil {
|
||||
if err := scanIndexRow(rows, &r); err != nil {
|
||||
si.logger.Warn("search index scan error", "error", err)
|
||||
|
||||
continue
|
||||
@@ -2051,9 +2040,11 @@ func (si *SearchIndex) indexOneArtist(
|
||||
// concurrently — they use different rate limiters so they
|
||||
// don't block each other.
|
||||
var (
|
||||
rgs []SearchIndexResult
|
||||
recs []SearchIndexResult
|
||||
wg sync.WaitGroup
|
||||
rgs []SearchIndexResult
|
||||
recs []SearchIndexResult
|
||||
rgErr error
|
||||
recErr error
|
||||
wg sync.WaitGroup
|
||||
)
|
||||
|
||||
// LB pipeline: top release groups + top recordings.
|
||||
@@ -2062,8 +2053,8 @@ func (si *SearchIndex) indexOneArtist(
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
|
||||
rgs = si.fetchTopReleaseGroups(ctx, lb, artist, rgLimit)
|
||||
recs = si.fetchTopRecordings(ctx, lb, artist, recLimit)
|
||||
rgs, rgErr = si.fetchTopReleaseGroups(ctx, lb, artist, rgLimit)
|
||||
recs, recErr = si.fetchTopRecordings(ctx, lb, artist, recLimit)
|
||||
}()
|
||||
|
||||
// MB pipeline: cache the artist lookup, which is what the details
|
||||
@@ -2090,14 +2081,19 @@ func (si *SearchIndex) indexOneArtist(
|
||||
|
||||
// Write the artist entry into the index so indexedArtistMBIDs()
|
||||
// recognises this artist as processed on subsequent builds.
|
||||
// Only mark DiscogFetched=true if at least one of the discography
|
||||
// fetches actually returned data — a transient API failure should
|
||||
// allow a retry on the next build, not permanently claim the
|
||||
// artist as indexed. Also stores aliases and detail fields from
|
||||
// the now-cached MB rels (populated by the image resolution above)
|
||||
// for FTS search.
|
||||
// Also stores aliases and detail fields from the now-cached MB rels
|
||||
// (populated by the image resolution above) for FTS search.
|
||||
//
|
||||
// DiscogFetched records that both LB endpoints were *asked*, not
|
||||
// that they had anything to say. A transient failure still leaves
|
||||
// the artist unmarked so the next run retries it — but an artist LB
|
||||
// has no popularity data for (or none above indexMinPopularity,
|
||||
// which is most of a long-tail library) answers empty every single
|
||||
// time, and keying the mark on emptiness made those artists
|
||||
// permanent candidates: the owned-artist backfill re-ran for them
|
||||
// on every launch, forever, which is the bug this replaces.
|
||||
if si.artistImg != nil {
|
||||
gotData := len(rgs) > 0 || len(recs) > 0
|
||||
gotData := rgErr == nil && recErr == nil
|
||||
artistEntry := SearchIndexResult{
|
||||
EntityType: "artist",
|
||||
MBID: artist.ArtistMBID,
|
||||
@@ -2139,10 +2135,10 @@ func (si *SearchIndex) fetchTopReleaseGroups(
|
||||
lb *ListenBrainzClient,
|
||||
artist lbSitewideArtist,
|
||||
maxCount int,
|
||||
) []SearchIndexResult {
|
||||
) ([]SearchIndexResult, error) {
|
||||
url := fmt.Sprintf(
|
||||
"%s/1/popularity/top-release-groups-for-artist/%s",
|
||||
listenBrainzBaseURL, artist.ArtistMBID,
|
||||
lb.baseURL, artist.ArtistMBID,
|
||||
)
|
||||
|
||||
body, err := lb.doGet(ctx, url)
|
||||
@@ -2152,7 +2148,7 @@ func (si *SearchIndex) fetchTopReleaseGroups(
|
||||
"error", err,
|
||||
)
|
||||
|
||||
return nil
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var raw []struct {
|
||||
@@ -2173,7 +2169,7 @@ func (si *SearchIndex) fetchTopReleaseGroups(
|
||||
}
|
||||
|
||||
if err := json.Unmarshal(body, &raw); err != nil {
|
||||
return nil
|
||||
return nil, err
|
||||
}
|
||||
|
||||
limit := maxCount
|
||||
@@ -2209,7 +2205,7 @@ func (si *SearchIndex) fetchTopReleaseGroups(
|
||||
})
|
||||
}
|
||||
|
||||
return results
|
||||
return results, nil
|
||||
}
|
||||
|
||||
func (si *SearchIndex) fetchTopRecordings(
|
||||
@@ -2217,20 +2213,20 @@ func (si *SearchIndex) fetchTopRecordings(
|
||||
lb *ListenBrainzClient,
|
||||
artist lbSitewideArtist,
|
||||
maxCount int,
|
||||
) []SearchIndexResult {
|
||||
) ([]SearchIndexResult, error) {
|
||||
url := fmt.Sprintf(
|
||||
"%s/1/popularity/top-recordings-for-artist/%s",
|
||||
listenBrainzBaseURL, artist.ArtistMBID,
|
||||
lb.baseURL, artist.ArtistMBID,
|
||||
)
|
||||
|
||||
body, err := lb.doGet(ctx, url)
|
||||
if err != nil {
|
||||
return nil
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var raw []lbTopRecordingWire
|
||||
if err := json.Unmarshal(body, &raw); err != nil {
|
||||
return nil
|
||||
return nil, err
|
||||
}
|
||||
|
||||
limit := maxCount
|
||||
@@ -2258,7 +2254,7 @@ func (si *SearchIndex) fetchTopRecordings(
|
||||
})
|
||||
}
|
||||
|
||||
return results
|
||||
return results, nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -2276,6 +2272,89 @@ func (si *SearchIndex) fetchTopRecordings(
|
||||
// empty values, and numeric fields use "highest wins" for popularity/
|
||||
// listener_count/duration so older richer data survives refreshes.
|
||||
|
||||
// indexRowColumns is the full explore_index projection, and
|
||||
// scanIndexRow is the only thing that reads it.
|
||||
//
|
||||
// There were four copies of this column list and four matching Scan
|
||||
// calls, which is what makes the storage encoding dangerous: a blob
|
||||
// column scanned into a string yields sixteen bytes of garbage rather
|
||||
// than an error, and it would have had to be got right four times.
|
||||
// dbMBID and dbEntityType do the decoding, and they refuse anything
|
||||
// that is not what they expect.
|
||||
var indexRowFields = []string{
|
||||
"entity_type", "mbid", "title", "artist_name", "artist_mbid",
|
||||
"popularity", "listener_count", "duration", "primary_type",
|
||||
"secondary_types", "release_date", "total_tracks", "caa_release_mbid",
|
||||
"release_name", "artist_type", "country", "disambiguation",
|
||||
"sort_name", "in_library", "is_similar",
|
||||
"local_artist_id", "local_release_group_id", "local_recording_id",
|
||||
}
|
||||
|
||||
// nullableIndexRowFields are the ones a caller wants zero rather than
|
||||
// NULL for.
|
||||
var nullableIndexRowFields = map[string]bool{
|
||||
"local_artist_id": true,
|
||||
"local_release_group_id": true,
|
||||
"local_recording_id": true,
|
||||
}
|
||||
|
||||
// indexRowColumns is the projection unqualified; indexRowColumnsFor
|
||||
// qualifies it with a table alias, for the joins where the other side
|
||||
// also has a `title`.
|
||||
var indexRowColumns = indexRowColumnsFor("")
|
||||
|
||||
func indexRowColumnsFor(alias string) string {
|
||||
if alias != "" {
|
||||
alias += "."
|
||||
}
|
||||
|
||||
cols := make([]string, 0, len(indexRowFields))
|
||||
|
||||
for _, f := range indexRowFields {
|
||||
if nullableIndexRowFields[f] {
|
||||
cols = append(cols, "COALESCE("+alias+f+", 0)")
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
cols = append(cols, alias+f)
|
||||
}
|
||||
|
||||
return strings.Join(cols, ", ")
|
||||
}
|
||||
|
||||
// scanIndexRow reads one indexRowColumns row into a result.
|
||||
func scanIndexRow(rows *sql.Rows, r *SearchIndexResult, before ...any) error {
|
||||
var (
|
||||
entity dbEntityType
|
||||
id dbMBID
|
||||
artist dbMBID
|
||||
caa dbMBID
|
||||
)
|
||||
|
||||
dest := make([]any, 0, len(before)+len(indexRowFields))
|
||||
dest = append(dest, before...)
|
||||
dest = append(dest,
|
||||
&entity, &id, &r.Title, &r.ArtistName, &artist,
|
||||
&r.Popularity, &r.ListenerCount, &r.Duration, &r.PrimaryType,
|
||||
&r.SecondaryTypes, &r.ReleaseDate, &r.TotalTracks, &caa,
|
||||
&r.ReleaseName, &r.ArtistType, &r.Country, &r.Disambiguation,
|
||||
&r.SortName, &r.InLibrary, &r.IsSimilar,
|
||||
&r.LocalArtistID, &r.LocalReleaseGroupID, &r.LocalRecordingID,
|
||||
)
|
||||
|
||||
if err := rows.Scan(dest...); err != nil {
|
||||
return fmt.Errorf("scan index row: %w", err)
|
||||
}
|
||||
|
||||
r.EntityType = string(entity)
|
||||
r.MBID = string(id)
|
||||
r.ArtistMBID = string(artist)
|
||||
r.CAAReleaseMBID = string(caa)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// upsertIndexSQL is the single index write statement. It is kept as
|
||||
// a const so assembly can prepare it once per transaction instead of
|
||||
// re-parsing this large upsert for every row.
|
||||
@@ -2284,7 +2363,7 @@ const upsertIndexSQL = `
|
||||
entity_type, mbid, title, artist_name, artist_mbid, aliases,
|
||||
popularity, listener_count,
|
||||
duration, caa_release_mbid, release_name,
|
||||
primary_type, secondary_types, release_date,
|
||||
primary_type, secondary_types, release_date, total_tracks,
|
||||
artist_type, country, disambiguation, sort_name,
|
||||
in_library, is_similar,
|
||||
local_artist_id, local_release_group_id, local_recording_id,
|
||||
@@ -2293,7 +2372,7 @@ const upsertIndexSQL = `
|
||||
?, ?, ?, ?, ?, ?,
|
||||
?, ?,
|
||||
?, ?, ?,
|
||||
?, ?, ?,
|
||||
?, ?, ?, ?,
|
||||
?, ?, ?, ?,
|
||||
?, ?,
|
||||
NULLIF(?, 0), NULLIF(?, 0), NULLIF(?, 0),
|
||||
@@ -2311,7 +2390,7 @@ const upsertIndexConflictSQL = `
|
||||
-- as a name; AddFromCache is the path that used to.
|
||||
title = CASE WHEN excluded.title != '' THEN excluded.title ELSE title END,
|
||||
artist_name = CASE WHEN excluded.artist_name != '' THEN excluded.artist_name ELSE artist_name END,
|
||||
artist_mbid = CASE WHEN excluded.artist_mbid != '' THEN excluded.artist_mbid ELSE artist_mbid END,
|
||||
artist_mbid = CASE WHEN excluded.artist_mbid != x'' THEN excluded.artist_mbid ELSE artist_mbid END,
|
||||
aliases = CASE WHEN excluded.aliases != '' THEN excluded.aliases ELSE aliases END,
|
||||
|
||||
-- Highest wins for popularity + listener_count (refreshes can go up).
|
||||
@@ -2320,11 +2399,12 @@ const upsertIndexConflictSQL = `
|
||||
|
||||
-- Non-empty wins for all other optional fields (never clobber with empty).
|
||||
duration = CASE WHEN excluded.duration > 0 THEN excluded.duration ELSE duration END,
|
||||
caa_release_mbid = CASE WHEN excluded.caa_release_mbid != '' THEN excluded.caa_release_mbid ELSE caa_release_mbid END,
|
||||
caa_release_mbid = CASE WHEN excluded.caa_release_mbid != x'' THEN excluded.caa_release_mbid ELSE caa_release_mbid END,
|
||||
release_name = CASE WHEN excluded.release_name != '' THEN excluded.release_name ELSE release_name END,
|
||||
primary_type = CASE WHEN excluded.primary_type != '' THEN excluded.primary_type ELSE primary_type END,
|
||||
secondary_types = CASE WHEN excluded.secondary_types != '' THEN excluded.secondary_types ELSE secondary_types END,
|
||||
release_date = CASE WHEN excluded.release_date != '' THEN excluded.release_date ELSE release_date END,
|
||||
total_tracks = CASE WHEN excluded.total_tracks > 0 THEN excluded.total_tracks ELSE total_tracks END,
|
||||
artist_type = CASE WHEN excluded.artist_type != '' THEN excluded.artist_type ELSE artist_type END,
|
||||
country = CASE WHEN excluded.country != '' THEN excluded.country ELSE country END,
|
||||
disambiguation = CASE WHEN excluded.disambiguation != '' THEN excluded.disambiguation ELSE disambiguation END,
|
||||
@@ -2387,10 +2467,11 @@ func (si *SearchIndex) upsertBatch(entries []SearchIndexResult) {
|
||||
}
|
||||
|
||||
if _, err := stmt.Exec(
|
||||
e.EntityType, e.MBID, e.Title, e.ArtistName, e.ArtistMBID, e.Aliases,
|
||||
dbEntityType(e.EntityType), dbMBID(e.MBID),
|
||||
e.Title, e.ArtistName, dbMBID(e.ArtistMBID), e.Aliases,
|
||||
e.Popularity, e.ListenerCount,
|
||||
e.Duration, e.CAAReleaseMBID, e.ReleaseName,
|
||||
e.PrimaryType, e.SecondaryTypes, e.ReleaseDate,
|
||||
e.Duration, dbMBID(e.CAAReleaseMBID), e.ReleaseName,
|
||||
e.PrimaryType, e.SecondaryTypes, e.ReleaseDate, e.TotalTracks,
|
||||
e.ArtistType, e.Country, e.Disambiguation, e.SortName,
|
||||
inLib, isSim,
|
||||
e.LocalArtistID, e.LocalReleaseGroupID, e.LocalRecordingID,
|
||||
@@ -2480,42 +2561,48 @@ func (si *SearchIndex) pruneStaleLocalCrossReferences() {
|
||||
type prune struct {
|
||||
entityType string
|
||||
column string
|
||||
table string
|
||||
// exists is the test for "this local id still refers to
|
||||
// something the user owns". It is a file test in every case -
|
||||
// the version that tested the metadata table left 129 rows in
|
||||
// a real catalog claiming to be owned by files that were gone.
|
||||
exists string
|
||||
}
|
||||
|
||||
for _, p := range []prune{
|
||||
{"artist", "local_artist_id", "artists"},
|
||||
{"release_group", "local_release_group_id", "release_groups"},
|
||||
{"recording", "local_recording_id", "recordings"},
|
||||
{"artist", "local_artist_id", `
|
||||
SELECT 1 FROM artists a WHERE a.id = explore_index.local_artist_id
|
||||
AND (
|
||||
EXISTS (SELECT 1 FROM audio_files af WHERE af.artist_id = a.id)
|
||||
OR EXISTS (
|
||||
SELECT 1 FROM albums al
|
||||
JOIN audio_files af2 ON af2.album_id = al.id
|
||||
WHERE al.artist_id = a.id
|
||||
)
|
||||
)`},
|
||||
{"release_group", "local_release_group_id", `
|
||||
SELECT 1 FROM audio_files af
|
||||
WHERE af.album_id = explore_index.local_release_group_id`},
|
||||
{"recording", "local_recording_id", `
|
||||
SELECT 1 FROM audio_files af
|
||||
WHERE af.id = explore_index.local_recording_id`},
|
||||
} {
|
||||
result, err := si.db.ExecContext(
|
||||
`UPDATE explore_index
|
||||
SET in_library = 0, `+p.column+` = NULL
|
||||
WHERE entity_type = ?
|
||||
AND `+p.column+` IS NOT NULL
|
||||
AND `+p.column+` NOT IN (SELECT id FROM `+p.table+`)`,
|
||||
p.entityType,
|
||||
WHERE entity_type = ? AND `+p.column+` IS NOT NULL
|
||||
AND NOT EXISTS (`+p.exists+`)`,
|
||||
dbEntityType(p.entityType),
|
||||
)
|
||||
if err != nil {
|
||||
si.logger.Warn(
|
||||
"library sync: prune stale cross-references failed",
|
||||
"entityType",
|
||||
p.entityType,
|
||||
"error",
|
||||
err,
|
||||
)
|
||||
si.logger.Warn("library sync: prune stale cross-references failed",
|
||||
"entityType", p.entityType, "error", err)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if n, err := result.RowsAffected(); err == nil && n > 0 {
|
||||
si.logger.Info(
|
||||
"library sync: cleared stale cross-references",
|
||||
"entityType",
|
||||
p.entityType,
|
||||
"count",
|
||||
n,
|
||||
)
|
||||
if n, _ := result.RowsAffected(); n > 0 {
|
||||
si.logger.Info("library sync: cleared stale cross-references",
|
||||
"entityType", p.entityType, "count", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2527,118 +2614,85 @@ func (si *SearchIndex) pruneStaleLocalCrossReferences() {
|
||||
func (si *SearchIndex) collectLibraryEntities() []SearchIndexResult {
|
||||
var entries []SearchIndexResult
|
||||
|
||||
// Artists.
|
||||
artistRows, err := si.db.QueryContext(
|
||||
"SELECT id, name, mbid FROM artists WHERE mbid IS NOT NULL AND mbid != ''",
|
||||
)
|
||||
if err == nil {
|
||||
for artistRows.Next() {
|
||||
var (
|
||||
id int64
|
||||
name, mbid string
|
||||
)
|
||||
|
||||
if err := artistRows.Scan(&id, &name, &mbid); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
entries = append(entries, SearchIndexResult{
|
||||
EntityType: "artist",
|
||||
MBID: mbid,
|
||||
Title: name,
|
||||
ArtistName: name,
|
||||
ArtistMBID: mbid,
|
||||
InLibrary: true,
|
||||
LocalArtistID: id,
|
||||
})
|
||||
}
|
||||
|
||||
_ = artistRows.Close()
|
||||
} else {
|
||||
si.logger.Warn("library sync: query artists failed", "error", err)
|
||||
// Every one of these is gated on a file existing. They used to
|
||||
// select straight from the metadata tables, so an artist, album or
|
||||
// recording whose files were gone stayed flagged "in library" in
|
||||
// the catalog until something noticed - and nothing did.
|
||||
type entityQuery struct {
|
||||
kind string
|
||||
query string
|
||||
}
|
||||
|
||||
// Release groups. The correlated subquery picks the primary
|
||||
// credited artist's MBID (if it has one).
|
||||
rgRows, err := si.db.QueryContext(`
|
||||
SELECT rg.id, rg.name, rg.mbid, COALESCE(ac.text, ''),
|
||||
COALESCE((
|
||||
SELECT a.mbid FROM artist_credit_artist aca
|
||||
JOIN artists a ON a.id = aca.artist_id
|
||||
WHERE aca.credit_id = rg.album_artist_credit_id
|
||||
AND a.mbid IS NOT NULL AND a.mbid != ''
|
||||
LIMIT 1
|
||||
), '')
|
||||
FROM release_groups rg
|
||||
LEFT JOIN artist_credit ac ON ac.id = rg.album_artist_credit_id
|
||||
WHERE rg.mbid IS NOT NULL AND rg.mbid != ''
|
||||
`)
|
||||
if err == nil {
|
||||
for rgRows.Next() {
|
||||
queries := []entityQuery{
|
||||
{"artist", `
|
||||
SELECT DISTINCT a.id, a.name, a.mbid, a.name, a.mbid
|
||||
FROM artists a
|
||||
WHERE a.mbid IS NOT NULL AND a.mbid != ''
|
||||
AND (
|
||||
EXISTS (SELECT 1 FROM audio_files af WHERE af.artist_id = a.id)
|
||||
OR EXISTS (
|
||||
SELECT 1 FROM albums al
|
||||
JOIN audio_files af2 ON af2.album_id = al.id
|
||||
WHERE al.artist_id = a.id
|
||||
)
|
||||
)`},
|
||||
{"release_group", `
|
||||
SELECT DISTINCT al.id, al.name, al.mbid, al.artist_credit,
|
||||
COALESCE(ar.mbid, '')
|
||||
FROM albums al
|
||||
JOIN audio_files af ON af.album_id = al.id
|
||||
LEFT JOIN artists ar ON ar.id = al.artist_id
|
||||
WHERE al.mbid IS NOT NULL AND al.mbid != ''`},
|
||||
{"recording", `
|
||||
SELECT af.id, af.title, af.recording_mbid, af.artist_credit,
|
||||
COALESCE(ar.mbid, '')
|
||||
FROM audio_files af
|
||||
LEFT JOIN artists ar ON ar.id = af.artist_id
|
||||
WHERE af.recording_mbid IS NOT NULL AND af.recording_mbid != ''`},
|
||||
}
|
||||
|
||||
for _, eq := range queries {
|
||||
rows, err := si.db.QueryContext(eq.query)
|
||||
if err != nil {
|
||||
si.logger.Warn("library sync: query failed", "kind", eq.kind, "error", err)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
id int64
|
||||
name, mbid, credit, artistMB string
|
||||
)
|
||||
|
||||
if err := rgRows.Scan(&id, &name, &mbid, &credit, &artistMB); err != nil {
|
||||
if err := rows.Scan(&id, &name, &mbid, &credit, &artistMB); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
entries = append(entries, SearchIndexResult{
|
||||
EntityType: "release_group",
|
||||
MBID: mbid,
|
||||
Title: name,
|
||||
ArtistName: credit,
|
||||
ArtistMBID: artistMB,
|
||||
InLibrary: true,
|
||||
LocalReleaseGroupID: id,
|
||||
})
|
||||
}
|
||||
|
||||
_ = rgRows.Close()
|
||||
} else {
|
||||
si.logger.Warn("library sync: query release groups failed", "error", err)
|
||||
}
|
||||
|
||||
// Recordings.
|
||||
recRows, err := si.db.QueryContext(`
|
||||
SELECT r.id, r.name, r.mbid, COALESCE(ac.text, ''),
|
||||
COALESCE((
|
||||
SELECT a.mbid FROM artist_credit_artist aca
|
||||
JOIN artists a ON a.id = aca.artist_id
|
||||
WHERE aca.credit_id = r.artist_credit_id
|
||||
AND a.mbid IS NOT NULL AND a.mbid != ''
|
||||
LIMIT 1
|
||||
), '')
|
||||
FROM recordings r
|
||||
LEFT JOIN artist_credit ac ON ac.id = r.artist_credit_id
|
||||
WHERE r.mbid IS NOT NULL AND r.mbid != ''
|
||||
`)
|
||||
if err == nil {
|
||||
for recRows.Next() {
|
||||
var (
|
||||
id int64
|
||||
name, mbid, credit, artistMB string
|
||||
)
|
||||
|
||||
if err := recRows.Scan(&id, &name, &mbid, &credit, &artistMB); err != nil {
|
||||
continue
|
||||
entry := SearchIndexResult{
|
||||
EntityType: eq.kind,
|
||||
MBID: mbid,
|
||||
Title: name,
|
||||
ArtistName: credit,
|
||||
ArtistMBID: artistMB,
|
||||
InLibrary: true,
|
||||
}
|
||||
|
||||
entries = append(entries, SearchIndexResult{
|
||||
EntityType: "recording",
|
||||
MBID: mbid,
|
||||
Title: name,
|
||||
ArtistName: credit,
|
||||
ArtistMBID: artistMB,
|
||||
InLibrary: true,
|
||||
LocalRecordingID: id,
|
||||
})
|
||||
switch eq.kind {
|
||||
case "artist":
|
||||
entry.LocalArtistID = id
|
||||
case "release_group":
|
||||
entry.LocalReleaseGroupID = id
|
||||
case "recording":
|
||||
// The local id of a "recording" is the file's, which is
|
||||
// what every caller wants: it is the thing that plays.
|
||||
entry.LocalRecordingID = id
|
||||
}
|
||||
|
||||
entries = append(entries, entry)
|
||||
}
|
||||
|
||||
_ = recRows.Close()
|
||||
} else {
|
||||
si.logger.Warn("library sync: query recordings failed", "error", err)
|
||||
_ = rows.Close()
|
||||
}
|
||||
|
||||
return entries
|
||||
@@ -2710,7 +2764,7 @@ func (si *SearchIndex) BackfillPopularity(updates map[string]PopularityData) {
|
||||
WHERE mbid = ?`,
|
||||
data.ListenCount, data.ListenCount,
|
||||
data.ListenerCount, data.ListenerCount,
|
||||
mbid,
|
||||
dbMBID(mbid),
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -332,7 +332,7 @@ func (si *SearchIndex) topByPopularity(
|
||||
// The artist rows *are* the artists, so they partition by their own
|
||||
// mbid; release groups partition by whoever made them.
|
||||
partition := "artist_mbid"
|
||||
if entityType == "artist" {
|
||||
if entityType == EntityArtist {
|
||||
partition = "mbid"
|
||||
}
|
||||
|
||||
@@ -350,14 +350,14 @@ func (si *SearchIndex) topByPopularity(
|
||||
WHERE rank = 1
|
||||
ORDER BY popularity DESC
|
||||
LIMIT ?`,
|
||||
entityType, limit+len(skip),
|
||||
dbEntityType(entityType), limit+len(skip),
|
||||
))
|
||||
|
||||
out := make([]SearchIndexResult, 0, limit)
|
||||
|
||||
for _, row := range rows {
|
||||
key := row.ArtistMBID
|
||||
if entityType == "artist" {
|
||||
if entityType == EntityArtist {
|
||||
key = row.MBID
|
||||
}
|
||||
|
||||
@@ -393,13 +393,13 @@ func (si *SearchIndex) unownedAlbumsBySinglyOwnedArtists(
|
||||
return si.rowsByIDs(ctx, si.shelfIDs(
|
||||
ctx,
|
||||
`SELECT id FROM explore_index
|
||||
WHERE entity_type = 'release_group'
|
||||
WHERE entity_type = 2 /* release_group */
|
||||
AND in_library = 0
|
||||
AND artist_mbid IN (
|
||||
SELECT artist_mbid FROM explore_index
|
||||
WHERE entity_type = 'release_group'
|
||||
WHERE entity_type = 2 /* release_group */
|
||||
AND in_library = 1
|
||||
AND artist_mbid != ''
|
||||
AND artist_mbid != x''
|
||||
GROUP BY artist_mbid
|
||||
HAVING COUNT(*) = 1
|
||||
ORDER BY MAX(popularity) DESC
|
||||
|
||||
@@ -28,24 +28,17 @@ func seedShelfRow(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
owned := 0
|
||||
if inLibrary {
|
||||
owned = 1
|
||||
}
|
||||
|
||||
if _, err := db.ExecContext(
|
||||
upsertIndexSQL,
|
||||
entityType, mbid, title, artistName, artistMBID, "",
|
||||
popularity, popularity,
|
||||
0, "", "",
|
||||
"Album", "", "",
|
||||
"", "", "", "",
|
||||
owned, 0,
|
||||
0, 0, 0,
|
||||
0,
|
||||
); err != nil {
|
||||
t.Fatalf("seed explore_index row %q: %v", mbid, err)
|
||||
}
|
||||
seedIndexResult(t, db, SearchIndexResult{
|
||||
EntityType: entityType,
|
||||
MBID: testMBID(mbid),
|
||||
Title: title,
|
||||
ArtistName: artistName,
|
||||
ArtistMBID: testMBID(artistMBID),
|
||||
Popularity: popularity,
|
||||
ListenerCount: popularity,
|
||||
ReleaseName: "Album",
|
||||
InLibrary: inLibrary,
|
||||
})
|
||||
}
|
||||
|
||||
func newShelfService(t *testing.T) (*Service, *database.DB) {
|
||||
@@ -271,12 +264,12 @@ func TestShelves_TheSecondRowIsNotTheFirstRowsArtists(t *testing.T) {
|
||||
t.Fatal("no popular-artists shelf")
|
||||
}
|
||||
|
||||
if albums.Albums[0].ArtistMBID != "ar-huge" {
|
||||
if albums.Albums[0].ArtistMBID != testMBID("ar-huge") {
|
||||
t.Fatalf("albums shelf leads with %q, want ar-huge", albums.Albums[0].ArtistMBID)
|
||||
}
|
||||
|
||||
for _, artist := range artists.Artists {
|
||||
if artist.MBID == "ar-huge" {
|
||||
if artist.MBID == testMBID("ar-huge") {
|
||||
t.Fatal("artists shelf repeats the artist the albums shelf just showed")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,6 +17,11 @@ const (
|
||||
recA = "11111111-1111-1111-1111-111111111111"
|
||||
recB = "22222222-2222-2222-2222-222222222222"
|
||||
recC = "33333333-3333-3333-3333-333333333333"
|
||||
|
||||
// recD is on relA and nobody has ever played it, which is the point:
|
||||
// it must count toward relA's track total without being indexed as a
|
||||
// recording itself.
|
||||
recD = "44444444-4444-4444-4444-444444444444"
|
||||
relA = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
|
||||
relB = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
|
||||
rgA = "cccccccc-cccc-cccc-cccc-cccccccccccc"
|
||||
|
||||
@@ -77,6 +77,14 @@ type MBReleaseGroup struct {
|
||||
ListenerCount int `json:"listenerCount"`
|
||||
InLibrary bool `json:"inLibrary"` // true if the user owns this album
|
||||
LocalID int64 `json:"localId,omitempty"` // local release_group row ID
|
||||
|
||||
// TotalTracks is the catalog's track count for this release group,
|
||||
// or 0 for "the catalog does not say". It answers "how much of
|
||||
// this album do I have" for an album whose files declared no total
|
||||
// -- the case GetAlbumCompleteness cannot answer -- and it is not
|
||||
// filled by the MusicBrainz path below, which has the real
|
||||
// tracklist and does not need a denominator.
|
||||
TotalTracks int `json:"totalTracks"`
|
||||
}
|
||||
|
||||
// MBRelease is a Wails-friendly projection of a MusicBrainz release.
|
||||
|
||||
Reference in New Issue
Block a user