yonlu/yellowjacket
1
1
Fork 0
Code Issues 52 Pull Requests 4 Actions Packages Projects Releases 12 Wiki Activity

feat(database): shape the library like files, and shrink the catalog
CI / check (push) Successful in 3m7s
Details
CI / e2e (push) Canceled after 1m45s
Details

Browse Source 
Plans 013 and 014, the album page that prompted them, and the smaller
fixes they turned up. Changelog, largest first.

## The local library is shaped like files, not like MusicBrainz

`audio_files` carries its own tags and points at `albums` and
`artists`; `file_genres` is the one real many-to-many. `recordings`,
`release_group_recordings`, `artist_credit`, `artist_credit_artist`,
`recording_genres`, `release_groups` and `release_to_rg` are gone from
the local side, and with them a six-way join in every read, a
`MIN(release_group_id)` subquery in eleven queries and a
first-credited-artist subquery in nine. Measured on a real 25,966-file
library, every many-to-many that model expressed was 1:1 in the data.

- Ownership is a file. `GetFilePathsByRecordingMBIDs`,
  `LibraryMBIDIndex.CheckMBIDs`, `collectLibraryEntities` and
  `pruneStaleLocalCrossReferences` all join `audio_files`, so the 812
  orphaned recordings, 216 release groups and 260 artists that library
  carried are now structurally impossible.
- One projection: every track query selects from the `track_metadata`
  view, one row type, one mapper. Nine hand-rolled copies had drifted
  far enough to report different years on different screens.
- `library_id = 0` means every library, so each list query exists once
  instead of scoped and unscoped with a branch at every call site.
- No migration chain. `sql/schemas/` is the one description of the
  shape; `sql/migrations/`, `applyMigrations` and `schema_migrations`
  are squashed away, along with the drift between them that had sqlc
  generating against a stale schema.
- `database.InsertTestTrack` is the one test seeder; twenty test files
  had been assembling the old FK chain each in its own order.

## The catalog stores its ids as bytes

`explore_index`'s three 36-char MBID columns and its entity-type text
are 16 raw bytes and a small integer. The table and its six indexes go
780 MB to 405 MB on a real 2,052,200-row catalog, which is why a fresh
install is ~0.6 GB rather than ~1.0 GB.

- `backend/explore/mbid.go` is the only place the encoding is known;
  everything above it speaks dashed strings.
- `CHECK(length(mbid) = 16)` makes a stringly write fail at the insert
  rather than silently returning no rows, since SQLite does not coerce
  between TEXT and BLOB.
- The importer asks the artifact what encoding it carries and converts
  on the way in, so the artifact already published keeps working and no
  format bump is needed.
- `indexRowColumns`/`scanIndexRow` replace four copies of a 22-column
  list, and `TestStoredEncodingRoundTrips` sweeps every read path.

## An album page that says how much of the album is yours

- One question, asked once: is there a file. `filePaths` is filled by a
  single batched lookup when the tracklist settles, and the badge, the
  Play count, the dimmed rows and every menu item read it — replacing
  four claims of decreasing confidence that could show a green tick on
  an album whose every action did nothing.
- Play, Play 7 of 12, or no play button at all.
- `total_tracks` on `explore_index` (~2 bytes over 400,677 release
  groups) and on `audio_files` from tags that have always carried it:
  a complete MBID-matched album now makes no catalog call at all, where
  it used to spend the most expensive request the app makes.
- A merged cluster shows the running order the most releases agree on,
  and the version list marks the release you own rather than standing a
  synthetic entry in for it.
- `AlbumReleasesFailed`: a slow fetch is no longer reported as a failed
  one by a 12-second timer.
- Rows not in the library are dimmed in place (with `aria-disabled`)
  instead of the owned ones wearing a green tick and a legend.

## Caches and cover art get ceilings

- Only the three tiers of a cover are stored; the full-resolution copy
  nothing rendered was 1,134 MB of a 1.4 GB covers directory.
- One artist portrait is downloaded and the rest are remembered as
  URLs — 4.1 GB of a 5.3 GB cache was candidates no code path reads.
- `browsedArtBudget` and `httpCacheBudget` bound what an age cannot:
  the same install held art for 5,770 artists in a 1,301-artist
  library.
- `OrphanedArtistImagesJob` joined a bare MBID onto a sharded
  directory, so it deleted the rows that were the only record of the
  files it left behind. `explore.ArtistImageDir` is that layout's one
  definition now.

## The autotag queue asks whether there is work

`tagging_items` was a row per album folder, not a queue, and no query
read the `tag_status` column that held the answer. The four queue
queries ask the files, which matters most where it is least visible:
`startPrefetch` was scoring every album in a tagged library against
MusicBrainz.

## Phantom playlist tracks resolve in place

An M3U8 imported before its files leaves phantom rows; they now match
by path and fall back to position, keep their place in the playlist
when resolved, and pair best-first so two phantoms cannot claim the
same file.

## Playing a track plays the list it is in

Double-click, and Play on a single row's menu, queue the list as
displayed with `startIndex` on that row — the album page and the track
list used to queue one track and discard the album around it. A
multi-row selection still plays exactly itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfVYUVExXsx1nSWrXN8mAh
This commit is contained in:
yonlu
2026-08-16 13:58:15 -04:00
co-authored by Claude Opus 5
parent 1128881e8d
commit e7748f1fd5
208 changed files with 10944 additions and 12104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/database/search_test.go
+76 -232
View File
@@ -3,6 +3,8 @@ package database
import (
"fmt"
"testing"
"yellowjacket/backend/database/sql/sqlcgen"
)
// seedSearchData inserts ~7 tracks with the full FK chain required for
@@ -84,128 +86,44 @@ func seedSearchData(t *testing.T, db *DB) {
},
}
// Build unique sets.
artistMap := map[string]int64{}
albumMap := map[string]int64{}
var artistID, albumID int64
for _, tr := range tracks {
if _, ok := artistMap[tr.artist]; !ok {
artistID++
artistMap[tr.artist] = artistID
}
if _, ok := albumMap[tr.album]; !ok {
albumID++
albumMap[tr.album] = albumID
}
}
// Insert artist_credit rows.
for text, id := range artistMap {
_, err := db.ExecContext(
"INSERT INTO artist_credit (id, text) VALUES (?, ?)",
id, text,
)
if err != nil {
t.Fatalf("insert artist_credit %q: %v", text, err)
}
}
// Insert release_groups.
for name, id := range albumMap {
_, err := db.ExecContext(
"INSERT INTO release_groups (id, name) VALUES (?, ?)",
id, name,
)
if err != nil {
t.Fatalf("insert release_group %q: %v", name, err)
}
}
// Insert genres + recording_genres.
genreMap := map[string]int64{}
var genreID int64
for _, tr := range tracks {
if tr.genre == "" {
continue
}
if _, ok := genreMap[tr.genre]; !ok {
genreID++
genreMap[tr.genre] = genreID
_, err := db.ExecContext(
"INSERT INTO genres (id, name) VALUES (?, ?)",
genreID, tr.genre,
)
if err != nil {
t.Fatalf("insert genre %q: %v", tr.genre, err)
}
}
}
for _, tr := range tracks {
acID := artistMap[tr.artist]
rgID := albumMap[tr.album]
// Insert recording.
_, err := db.ExecContext(
"INSERT INTO recordings (id, name, artist_credit_id, "+
"track_number, disc_number, year, genre, composer) "+
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
tr.id, tr.title, acID, tr.trackNum, tr.discNum,
tr.year, tr.genre, tr.composer,
)
if err != nil {
t.Fatalf("insert recording %d %q: %v", tr.id, tr.title, err)
}
// Insert audio_files.
_, err = db.ExecContext(
"INSERT INTO audio_files (id, file_path, "+
"length_milliseconds, file_type_id, recording_id, "+
"sample_rate, bit_depth, channels, bitrate, file_size) "+
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
tr.id, tr.filePath, tr.lenMs, tr.ftID, tr.id,
tr.sr, tr.bd, tr.ch, tr.br, tr.fsize,
)
if err != nil {
t.Fatalf("insert audio_file %d: %v", tr.id, err)
}
// Link recording to release_group.
_, err = db.ExecContext(
"INSERT INTO release_group_recordings "+
"(release_group_id, recording_id, track_number, disc_number) "+
"VALUES (?, ?, ?, ?)",
rgID, tr.id, tr.trackNum, tr.discNum,
)
if err != nil {
t.Fatalf("insert release_group_recordings %d→%d: %v", rgID, tr.id, err)
}
// Insert search_index entry (rowid must match audio_files.id).
if err := db.InsertSearchIndex(
tr.id, tr.filePath, tr.title, tr.artist, tr.album,
); err != nil {
t.Fatalf("insert search_index for %d: %v", tr.id, err)
}
// Insert recording_genres link.
var genres []string
if tr.genre != "" {
gID := genreMap[tr.genre]
genres = []string{tr.genre}
}
_, err = db.ExecContext(
"INSERT INTO recording_genres (recording_id, genre_id) VALUES (?, ?)",
tr.id, gID,
)
if err != nil {
t.Fatalf("insert recording_genres %d→%d: %v", tr.id, gID, err)
}
var trackNum, discNum int64
if tr.trackNum != nil {
trackNum = *tr.trackNum
}
if tr.discNum != nil {
discNum = *tr.discNum
}
id := InsertTestTrack(t, db, TestTrack{
FilePath: tr.filePath,
Title: tr.title,
Artist: tr.artist,
Album: tr.album,
Genres: genres,
TrackNumber: trackNum,
DiscNumber: discNum,
Year: tr.year,
LengthMs: tr.lenMs,
})
// The fixtures assert on audio properties and the composer,
// which InsertTestTrack does not carry - they are not part of
// what a seeder should have to know about a track.
if _, err := db.ExecContext(
`UPDATE audio_files
SET file_type_id = ?, sample_rate = ?, bit_depth = ?,
channels = ?, bitrate = ?, file_size = ?, composer = ?
WHERE id = ?`,
tr.ftID, tr.sr, tr.bd, tr.ch, tr.br, tr.fsize, tr.composer, id,
); err != nil {
t.Fatalf("set audio properties for %q: %v", tr.filePath, err)
}
}
}
@@ -553,7 +471,7 @@ func TestSearchFTSTracks(t *testing.T) {
db := NewTestDB(t)
seedSearchData(t, db)
results, err := db.SearchFTSTracks("queen", 10)
results, err := db.SearchFTSTracks("queen", 0, 10)
if err != nil {
t.Fatalf("SearchFTSTracks: %v", err)
}
@@ -563,7 +481,7 @@ func TestSearchFTSTracks(t *testing.T) {
}
// Find the Bohemian Rhapsody result and verify all 16 fields.
var br *SearchTrackRow
var br *sqlcgen.TrackMetadatum
for i, r := range results {
if r.Title == "Bohemian Rhapsody" {
@@ -635,26 +553,12 @@ func TestInsertAndDeleteSearchIndex(t *testing.T) {
db := NewTestDB(t)
// Set up minimal FK chain for a single track.
_, err := db.ExecContext(
"INSERT INTO artist_credit (id, text) VALUES (1, 'Test Artist')",
)
if err != nil {
t.Fatalf("insert artist_credit: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO recordings (id, name, artist_credit_id) VALUES (1, 'Test Track', 1)",
)
if err != nil {
t.Fatalf("insert recording: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO audio_files (id, file_path, length_milliseconds, file_type_id, recording_id) VALUES (1, '/test/track.mp3', 180000, 0, 1)",
)
if err != nil {
t.Fatalf("insert audio_file: %v", err)
}
InsertTestTrack(t, db, TestTrack{
FilePath: "/test/track.mp3",
Title: "Test Track",
Artist: "Test Artist",
LengthMs: 180000,
})
// Insert into search index.
if err := db.InsertSearchIndex(
@@ -698,41 +602,15 @@ func TestRebuildSearchIndex(t *testing.T) {
db := NewTestDB(t)
// Seed the full entity graph WITHOUT inserting into search_index.
_, err := db.ExecContext(
"INSERT INTO artist_credit (id, text) VALUES (1, 'Rebuild Artist')",
)
if err != nil {
t.Fatalf("insert artist_credit: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO recordings (id, name, artist_credit_id) VALUES (1, 'Rebuild Track', 1)",
)
if err != nil {
t.Fatalf("insert recording: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO audio_files (id, file_path, length_milliseconds, file_type_id, recording_id) VALUES (1, '/rebuild/track.mp3', 200000, 0, 1)",
)
if err != nil {
t.Fatalf("insert audio_file: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO release_groups (id, name) VALUES (1, 'Rebuild Album')",
)
if err != nil {
t.Fatalf("insert release_group: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO release_group_recordings (release_group_id, recording_id) VALUES (1, 1)",
)
if err != nil {
t.Fatalf("insert release_group_recordings: %v", err)
}
// Seed the file WITHOUT putting it in search_index.
InsertTestTrack(t, db, TestTrack{
FilePath: "/rebuild/track.mp3",
Title: "Rebuild Track",
Artist: "Rebuild Artist",
Album: "Rebuild Album",
LengthMs: 200000,
SkipSearchIndex: true,
})
// Search should return nothing before rebuild.
results, err := db.SearchFTS("Rebuild", 10)
@@ -887,36 +765,22 @@ func TestSearchIndexUpdateCycle(t *testing.T) {
db := NewTestDB(t)
// Set up minimal FK chain for a single track at rowid 100.
_, err := db.ExecContext(
"INSERT INTO artist_credit (id, text) VALUES (100, 'Old Artist')",
)
if err != nil {
t.Fatalf("insert artist_credit: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO recordings (id, name, artist_credit_id) VALUES (100, 'Old Title', 100)",
)
if err != nil {
t.Fatalf("insert recording: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO audio_files (id, file_path, length_milliseconds, file_type_id, recording_id) " +
"VALUES (100, '/test/update_cycle.mp3', 200000, 0, 100)",
)
if err != nil {
t.Fatalf("insert audio_file: %v", err)
}
id := InsertTestTrack(t, db, TestTrack{
FilePath: "/test/update_cycle.mp3",
Title: "Old Title",
Artist: "Old Artist",
LengthMs: 200000,
SkipSearchIndex: true,
})
// 1. Insert with old metadata.
if err := db.InsertSearchIndex(
100, "/test/update_cycle.mp3", "Old Title", "Old Artist", "Old Album",
id, "/test/update_cycle.mp3", "Old Title", "Old Artist", "Old Album",
); err != nil {
t.Fatalf("InsertSearchIndex (old): %v", err)
}
// Verify search for "Old Title" returns rowid 100.
// Verify search for "Old Title" finds it.
results, err := db.SearchFTS("Old Title", 10)
if err != nil {
t.Fatalf("SearchFTS(Old Title): %v", err)
@@ -926,9 +790,9 @@ func TestSearchIndexUpdateCycle(t *testing.T) {
t.Fatal("SearchFTS(Old Title): got 0 results after insert")
}
// 2. Delete rowid 100.
if err := db.DeleteSearchIndex(100); err != nil {
t.Fatalf("DeleteSearchIndex(100): %v", err)
// 2. Delete the row.
if err := db.DeleteSearchIndex(id); err != nil {
t.Fatalf("DeleteSearchIndex(%d): %v", id, err)
}
// Verify "Old Title" no longer found.
@@ -944,25 +808,17 @@ func TestSearchIndexUpdateCycle(t *testing.T) {
)
}
// 3. Update the recording name in the DB to simulate tag edit.
// 3. Update the file's title in the DB to simulate a tag edit.
_, err = db.ExecContext(
"UPDATE recordings SET name = 'New Title' WHERE id = 100",
"UPDATE audio_files SET title = 'New Title' WHERE file_path = '/test/update_cycle.mp3'",
)
if err != nil {
t.Fatalf("update recording: %v", err)
t.Fatalf("update title: %v", err)
}
// Also add a new artist_credit for the new artist.
_, err = db.ExecContext(
"INSERT INTO artist_credit (id, text) VALUES (101, 'New Artist')",
)
if err != nil {
t.Fatalf("insert new artist_credit: %v", err)
}
// 4. Re-insert rowid 100 with new metadata.
// 4. Re-insert the row with new metadata.
if err := db.InsertSearchIndex(
100, "/test/update_cycle.mp3", "New Title", "New Artist", "New Album",
id, "/test/update_cycle.mp3", "New Title", "New Artist", "New Album",
); err != nil {
t.Fatalf("InsertSearchIndex (new): %v", err)
}
@@ -1079,25 +935,13 @@ func TestSearchIndexSchema(t *testing.T) {
t.Fatalf("insert artist: %v", err)
}
// The credit tables this used to assert a UNIQUE constraint on are
// gone; a file names its artist directly, and artists are unique by
// name, which is asserted below.
_, err = db.ExecContext(
"INSERT INTO artist_credit (id, text) VALUES (1, 'Test Credit')",
)
if err != nil {
t.Fatalf("insert artist_credit: %v", err)
}
_, err = db.ExecContext(
"INSERT INTO artist_credit_artist (artist_id, credit_id) VALUES (1, 1)",
)
if err != nil {
t.Fatalf("first insert artist_credit_artist: %v", err)
}
// Duplicate insert should fail with UNIQUE constraint.
_, err = db.ExecContext(
"INSERT INTO artist_credit_artist (artist_id, credit_id) VALUES (1, 1)",
"INSERT INTO artists (id, name) VALUES (2, 'Test')",
)
if err == nil {
t.Error("duplicate artist_credit_artist insert should fail, got nil error")
t.Error("duplicate artist name should fail, got nil error")
}
}