yellowjacket/.planning/milestones/v1.1-phases/10-schema-migration/10-02-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
10-schema-migration	02	execute	2	10-01	backend/database/sql/queries/libraries.sql backend/database/sql/queries/audio_files.sql backend/database/sql/queries/playlists.sql backend/database/sql/sqlcgen/db.go backend/database/sql/sqlcgen/models.go backend/database/sql/sqlcgen/querier.go backend/database/sql/sqlcgen/libraries.sql.go backend/database/sql/sqlcgen/audio_files.sql.go backend/database/sql/sqlcgen/playlists.sql.go backend/database/testhelper.go backend/database/database_test.go	true	LIB-04 LIB-05	truths artifacts key_links sqlc-generated queries exist for library CRUD (create, get, list, delete) Playlist track queries handle nullable audio_file_id and phantom columns Audio file queries accept library_id parameter Migration tests verify upgrade path from v5 to v6 Migration tests verify fresh database creates correct schema Migration tests verify TOML config is read and default library created Test helper NewTestDB creates v6 schema including libraries table path provides contains backend/database/sql/queries/libraries.sql sqlc query definitions for libraries CRUD CreateLibrary path provides contains backend/database/sql/queries/playlists.sql Updated playlist queries with phantom column support phantom_title path provides contains backend/database/sql/sqlcgen/libraries.sql.go Generated Go code for library queries func.*CreateLibrary path provides contains backend/database/database_test.go Migration 6 integration tests TestMigration6 from to via pattern backend/database/sql/queries/libraries.sql backend/database/sql/schemas/_libraries.sql sqlc schema awareness libraries from to via pattern backend/database/database_test.go backend/database/database.go migration6 NewTestDB runs all migrations runMigrations

Add sqlc query definitions for the new schema, regenerate Go code, and write migration integration tests.

Purpose: Plan 01 created the schema and migration. This plan makes the new tables usable via type-safe sqlc queries, updates existing playlist queries for phantom support, and verifies the migration works correctly on both fresh and existing databases.

Output: sqlc queries + generated code for libraries and updated playlists + comprehensive migration tests

<execution_context> @/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md @/home/caleb/.config/opencode/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/10-schema-migration/10-CONTEXT.md @.planning/phases/10-schema-migration/10-01-SUMMARY.md

From backend/database/sql/schemas/_libraries.sql (created by Plan 01):

CREATE TABLE IF NOT EXISTS libraries (
    id         INTEGER PRIMARY KEY,
    name       TEXT NOT NULL,
    path       TEXT NOT NULL UNIQUE,
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

From backend/database/sql/schemas/playlist_tracks.sql (updated by Plan 01):

CREATE TABLE IF NOT EXISTS playlist_tracks (
    id INTEGER PRIMARY KEY,
    playlist_id INTEGER NOT NULL,
    audio_file_id INTEGER,          -- nullable for phantom tracks
    position INTEGER NOT NULL,
    phantom_title TEXT,
    phantom_artist TEXT,
    phantom_album TEXT,
    phantom_duration_ms INTEGER,
    phantom_genre TEXT,
    phantom_cover_art_path TEXT,
    FOREIGN KEY(playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
    FOREIGN KEY(audio_file_id) REFERENCES audio_files(id) ON DELETE SET NULL
);

From backend/database/sql/schemas/audio_files.sql (updated by Plan 01):

-- Now includes: library_id int NOT NULL DEFAULT 0
-- FK: FOREIGN KEY(library_id) REFERENCES libraries(id)
-- Index: idx_audio_files_library_id

From backend/database/database.go (updated by Plan 01):

func runMigrations(ctx context.Context, db *sql.DB, logger *slog.Logger, dbPath string) error
func backupDatabase(dbPath string, logger *slog.Logger) (string, error)
func migration6MultiLibrary(ctx context.Context, db *sql.DB, logger *slog.Logger, dbPath string) error

From backend/database/sqlc.yaml:

version: "2"
sql:
  - name: "yellowjacket"
    engine: "sqlite"
    queries: "./sql/queries"
    schema: "./sql/schemas"
    gen:
      go:
        package: "sqlcgen"
        out: "./sql/sqlcgen"

Existing sqlc query patterns from playlists.sql:

-- name: AddPlaylistTrack :one
INSERT INTO playlist_tracks (playlist_id, audio_file_id, position) VALUES (?, ?, ?)
RETURNING *;

-- name: GetPlaylistTracksWithMetadata :many
SELECT pt.id, pt.playlist_id, pt.audio_file_id, pt.position,
    af.file_path, af.length_milliseconds, ...
FROM playlist_tracks pt
JOIN audio_files af ON pt.audio_file_id = af.id
...

Existing test patterns from testhelper.go:

func NewTestDB(t *testing.T) *DB  // runs all schemas + migrations

Existing test patterns from search_test.go:

func seedSearchData(t *testing.T, db *DB) // creates full entity graph

Task 1: Add sqlc queries for libraries and update playlist queries for phantom support backend/database/sql/queries/libraries.sql backend/database/sql/queries/audio_files.sql backend/database/sql/queries/playlists.sql backend/database/sql/sqlcgen/db.go backend/database/sql/sqlcgen/models.go backend/database/sql/sqlcgen/querier.go backend/database/sql/sqlcgen/libraries.sql.go backend/database/sql/sqlcgen/audio_files.sql.go backend/database/sql/sqlcgen/playlists.sql.go **1. Create `backend/database/sql/queries/libraries.sql` (NEW FILE):**

Define the core CRUD queries for the libraries table. These will be consumed by Phase 12 (Library CRUD API) but the type-safe generated code is needed now for migration tests and any early usage.

-- name: CreateLibrary :one
INSERT INTO libraries (name, path) VALUES (?, ?)
RETURNING *;

-- name: GetLibrary :one
SELECT * FROM libraries WHERE id = ? LIMIT 1;

-- name: GetLibraryByPath :one
SELECT * FROM libraries WHERE path = ? LIMIT 1;

-- name: GetAllLibraries :many
SELECT * FROM libraries ORDER BY name;

-- name: UpdateLibraryName :exec
UPDATE libraries SET name = ? WHERE id = ?;

-- name: DeleteLibrary :exec
DELETE FROM libraries WHERE id = ?;

-- name: CountLibraries :one
SELECT COUNT(*) AS count FROM libraries;

2. Update backend/database/sql/queries/playlists.sql:

The existing queries need updates for the new playlist_tracks schema:

a) AddPlaylistTrack — Add phantom metadata columns to the INSERT. The caller populates phantom data eagerly on every insert (per user decision):

-- name: AddPlaylistTrack :one
INSERT INTO playlist_tracks (
    playlist_id, audio_file_id, position,
    phantom_title, phantom_artist, phantom_album,
    phantom_duration_ms, phantom_genre, phantom_cover_art_path
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
RETURNING *;

b) GetPlaylistTracks — Change JOIN to LEFT JOIN on audio_files (audio_file_id is now nullable). Include phantom columns in output so callers can display either live or phantom data:

-- name: GetPlaylistTracks :many
SELECT pt.id, pt.playlist_id, pt.audio_file_id, pt.position,
    COALESCE(af.file_path, '') AS file_path,
    pt.phantom_title, pt.phantom_artist, pt.phantom_album,
    pt.phantom_duration_ms, pt.phantom_genre, pt.phantom_cover_art_path
FROM playlist_tracks pt
LEFT JOIN audio_files af ON pt.audio_file_id = af.id
WHERE pt.playlist_id = ?
ORDER BY pt.position;

c) GetPlaylistTracksWithMetadata — Same LEFT JOIN change, and include phantom fallback columns. When audio_file_id is NULL (phantom), the live metadata JOINs return NULL and callers use phantom_* columns instead:

-- name: GetPlaylistTracksWithMetadata :many
SELECT
    pt.id,
    pt.playlist_id,
    pt.audio_file_id,
    pt.position,
    COALESCE(af.file_path, '') AS file_path,
    COALESCE(af.length_milliseconds, 0) AS length_milliseconds,
    COALESCE(r.name, pt.phantom_title, '') AS title,
    COALESCE(ac.text, pt.phantom_artist, '') AS artist,
    COALESCE(rg.name, pt.phantom_album, '') AS album,
    COALESCE(ca.file_path, pt.phantom_cover_art_path, '') AS cover_art_path,
    CASE WHEN pt.audio_file_id IS NULL THEN 1 ELSE 0 END AS is_phantom
FROM playlist_tracks pt
LEFT JOIN audio_files af ON pt.audio_file_id = af.id
LEFT JOIN recordings r ON af.recording_id = r.id
LEFT JOIN artist_credit ac ON r.artist_credit_id = ac.id
LEFT JOIN (
    SELECT recording_id, MIN(release_group_id) AS release_group_id
    FROM release_group_recordings
    GROUP BY recording_id
) rgr ON r.id = rgr.recording_id
LEFT JOIN release_groups rg ON rgr.release_group_id = rg.id
LEFT JOIN cover_art ca ON rg.cover_art_id = ca.id
WHERE pt.playlist_id = ?
ORDER BY pt.position;

d) GetAllPlaylistTracksWithMetadata — Same LEFT JOIN and phantom fallback pattern, without WHERE clause.

e) IsTrackInPlaylist — Change JOIN to LEFT JOIN (audio_file_id may be NULL for phantom tracks).

f) RemovePlaylistTrackByPath — Change subquery JOIN to handle nullable audio_file_id.

g) GetPlaylistTrackFilePaths — Change to LEFT JOIN, filter out NULLs:

-- name: GetPlaylistTrackFilePaths :many
SELECT COALESCE(af.file_path, '') AS file_path
FROM playlist_tracks pt
LEFT JOIN audio_files af ON pt.audio_file_id = af.id
WHERE pt.playlist_id = ? AND pt.audio_file_id IS NOT NULL
ORDER BY pt.position;

3. Update backend/database/sql/queries/audio_files.sql:

Add a query to get audio files filtered by library:

-- name: GetAudioFilesByLibrary :many
SELECT * FROM audio_files WHERE library_id = ?;

-- name: CountAudioFilesByLibrary :one
SELECT COUNT(*) AS count FROM audio_files WHERE library_id = ?;

4. Regenerate sqlc code:

Run from backend/database/:

go generate ./...

This regenerates all files in sql/sqlcgen/ from the updated schemas and queries.

5. Fix any compilation errors in the generated code or in callers of the changed query signatures (particularly AddPlaylistTrack which now has 9 parameters instead of 3). Check all callers:

• backend/playlist/playlist.go — calls AddPlaylistTrack. Update to pass phantom metadata.
• Any other callers of changed queries.

For AddPlaylistTrack callers: pass the phantom metadata alongside the audio_file_id. The caller should resolve the metadata at insert time (eager population per user decision). Look at how playlist.go currently calls it and add the phantom fields. For now, populate phantom data from the track metadata that the caller already has available.

IMPORTANT: The playlist package's AddTrack/AddTracks methods need to resolve phantom metadata before inserting. Look at how GetPlaylistTracksWithMetadata resolves metadata — the same JOIN pattern should be used to fetch phantom data before insert. Or simpler: the caller already has the file path → look up metadata from DB → pass as phantom columns.

Create a helper query to resolve phantom metadata for a given audio_file_id:

-- name: GetTrackPhantomMetadata :one
SELECT
    COALESCE(r.name, '') AS title,
    COALESCE(ac.text, '') AS artist,
    COALESCE(rg.name, '') AS album,
    af.length_milliseconds AS duration_ms,
    CAST(COALESCE(
        (SELECT GROUP_CONCAT(g.name, '||')
         FROM recording_genres rg_sub
         JOIN genres g ON rg_sub.genre_id = g.id
         WHERE rg_sub.recording_id = r.id),
        ''
    ) AS TEXT) AS genre,
    COALESCE(ca.file_path, '') AS cover_art_path
FROM audio_files af
LEFT JOIN recordings r ON af.recording_id = r.id
LEFT JOIN artist_credit ac ON r.artist_credit_id = ac.id
LEFT JOIN (
    SELECT recording_id, MIN(release_group_id) AS release_group_id
    FROM release_group_recordings
    GROUP BY recording_id
) rgr ON r.id = rgr.recording_id
LEFT JOIN release_groups rg ON rgr.release_group_id = rg.id
LEFT JOIN cover_art ca ON rg.cover_art_id = ca.id
WHERE af.id = ?;

Add this to playlists.sql.

After regenerating, verify compilation:

cd backend && go build ./...

Fix any broken callers of AddPlaylistTrack — the signature change from 3 args to 9 args will cause compile errors in the playlist package. Update each caller to:

1. Look up phantom metadata via GetTrackPhantomMetadata query
2. Pass all 9 params to AddPlaylistTrack cd backend/database && go generate ./... && cd ../.. && go build ./... && go vet ./...
   • libraries.sql query file exists with 7 CRUD queries
   • playlists.sql updated with phantom column support in all track queries
   • audio_files.sql has library-filtered query
   • sqlc regenerated successfully (all files in sql/sqlcgen/ updated)
   • AddPlaylistTrack callers updated for new 9-param signature
   • GetTrackPhantomMetadata helper query exists for eager phantom population
   • go build ./... passes from project root

Task 2: Migration integration tests and NewTestDB update backend/database/testhelper.go backend/database/database_test.go Write integration tests that verify migration 6 works correctly on both fresh and existing databases. Also update `NewTestDB` for the new schema.

1. Update testhelper.go:

The NewTestDB helper runs all schemas + migrations. Since migration 6 reads a TOML config file, and the test helper uses :memory: database with no file path, the migration will skip the TOML reading (existingDir = ""). The test helper needs to handle the updated runMigrations signature that now takes dbPath:

// Pass empty string for dbPath — in-memory DBs don't need backup.
if err := runMigrations(ctx, db, slog.Default(), ""); err != nil {
    t.Fatalf("could not run migrations: %v", err)
}

The backup function should no-op when dbPath is empty. Verify this is handled in the migration 6 code (Plan 01 should have handled it — if not, add a guard).

Also add a NewTestDBWithLibrary helper that creates a test DB with a pre-populated library, useful for tests in other packages:

// NewTestDBWithLibrary returns a test DB with a library row pre-inserted.
// Returns the DB and the library ID.
func NewTestDBWithLibrary(t *testing.T, name, path string) (*DB, int64) {
    t.Helper()
    db := NewTestDB(t)
    lib, err := db.Queries.CreateLibrary(db.Ctx, sqlcgen.CreateLibraryParams{
        Name: name,
        Path: path,
    })
    if err != nil {
        t.Fatalf("could not create test library: %v", err)
    }
    return db, lib.ID
}

2. Create/update database_test.go:

Write these test cases:

a) TestMigration6FreshDB — Verify that a fresh database (no prior data) creates all expected tables including libraries, and that the schema matches expectations:

func TestMigration6FreshDB(t *testing.T) {
    db := NewTestDB(t)

    // Verify libraries table exists
    var tableCount int
    err := db.QueryRow("SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='libraries'").Scan(&tableCount)
    // assert tableCount == 1

    // Verify audio_files has library_id column
    // Query PRAGMA table_info(audio_files), check for library_id

    // Verify playlist_tracks has phantom columns and nullable audio_file_id
    // Query PRAGMA table_info(playlist_tracks), check columns

    // Verify track_metadata VIEW includes library_id
    // Query PRAGMA table_info(track_metadata), check for library_id — wait, VIEWs don't work with table_info
    // Instead: SELECT sql FROM sqlite_master WHERE name='track_metadata'
    // Assert contains 'library_id'

    // Verify user_version is current (>= 6)
    var version int
    err = db.QueryRow("PRAGMA user_version").Scan(&version)
    // assert version >= 6

    // Verify libraries table is empty on fresh DB
    count, err := db.Queries.CountLibraries(db.Ctx)
    // assert count == 0
}

b) TestMigration6LibraryQueries — Verify CRUD operations on libraries table work:

func TestMigration6LibraryQueries(t *testing.T) {
    db := NewTestDB(t)

    // Create a library
    lib, err := db.Queries.CreateLibrary(db.Ctx, sqlcgen.CreateLibraryParams{
        Name: "Music",
        Path: "/home/user/Music",
    })
    // assert lib.Name == "Music", lib.Path == "/home/user/Music"
    // assert lib.ID > 0

    // Get by ID
    got, err := db.Queries.GetLibrary(db.Ctx, lib.ID)
    // assert got matches lib

    // Get by path
    gotByPath, err := db.Queries.GetLibraryByPath(db.Ctx, "/home/user/Music")
    // assert gotByPath matches lib

    // Unique path constraint
    _, err = db.Queries.CreateLibrary(db.Ctx, sqlcgen.CreateLibraryParams{
        Name: "Duplicate",
        Path: "/home/user/Music",
    })
    // assert IsUniqueViolation(err)

    // List libraries
    libs, err := db.Queries.GetAllLibraries(db.Ctx)
    // assert len(libs) == 1

    // Update name
    err = db.Queries.UpdateLibraryName(db.Ctx, sqlcgen.UpdateLibraryNameParams{
        Name: "My Music",
        ID:   lib.ID,
    })
    // Verify name changed

    // Delete
    err = db.Queries.DeleteLibrary(db.Ctx, lib.ID)
    count, _ := db.Queries.CountLibraries(db.Ctx)
    // assert count == 0
}

c) TestMigration6PhantomPlaylistTracks — Verify playlist tracks work with phantom columns:

func TestMigration6PhantomPlaylistTracks(t *testing.T) {
    db, libID := NewTestDBWithLibrary(t, "Test", "/test/music")

    // Create prerequisite data: file_type, recording, audio_file
    // (use pattern from existing seedSearchData or seedAudioFiles)

    // Create playlist
    playlist, _ := db.Queries.CreatePlaylist(db.Ctx, "Test Playlist")

    // Add track with phantom metadata (eager population)
    track, err := db.Queries.AddPlaylistTrack(db.Ctx, sqlcgen.AddPlaylistTrackParams{
        PlaylistID:           playlist.ID,
        AudioFileID:          sql.NullInt64{Int64: audioFileID, Valid: true},
        Position:             0,
        PhantomTitle:         sql.NullString{String: "Test Song", Valid: true},
        PhantomArtist:        sql.NullString{String: "Test Artist", Valid: true},
        PhantomAlbum:         sql.NullString{String: "Test Album", Valid: true},
        PhantomDurationMs:    sql.NullInt64{Int64: 180000, Valid: true},
        PhantomGenre:         sql.NullString{String: "Rock", Valid: true},
        PhantomCoverArtPath:  sql.NullString{String: "", Valid: false},
    })
    // assert track created

    // Delete the audio_file — should SET NULL on audio_file_id
    // (not CASCADE delete the playlist_track)
    _, err = db.ExecContext("DELETE FROM audio_files WHERE id = ?", audioFileID)

    // Verify playlist track still exists with NULL audio_file_id
    tracks, _ := db.Queries.GetPlaylistTracksWithMetadata(db.Ctx, playlist.ID)
    // assert len(tracks) == 1
    // assert tracks[0].AudioFileID is NULL/invalid
    // assert tracks[0].Title == "Test Song" (from phantom)
    // assert tracks[0].IsPhantom == 1
}

d) TestMigration6AudioFilesLibraryFK — Verify library_id FK enforcement:

func TestMigration6AudioFilesLibraryFK(t *testing.T) {
    db, libID := NewTestDBWithLibrary(t, "Test", "/test")

    // Insert audio_file with valid library_id — should succeed
    // Insert audio_file with invalid library_id (999) — should fail FK check

    // Count files by library
    count, _ := db.Queries.CountAudioFilesByLibrary(db.Ctx, libID)
    // assert count == 1
}

e) TestMigration6TrackMetadataViewHasLibraryID — Verify the VIEW includes library_id:

func TestMigration6TrackMetadataViewHasLibraryID(t *testing.T) {
    db, libID := NewTestDBWithLibrary(t, "Test", "/test")
    // Insert an audio file with test data
    // Query track_metadata VIEW
    // Verify library_id column is present and has correct value
}

Test patterns to follow:

• Use NewTestDB(t) or NewTestDBWithLibrary(t, ...) for setup
• Use t.Helper() in helpers
• Use t.Context() — NOT context.Background()
• Table-driven subtests where appropriate
• Use database.IsUniqueViolation(err) for constraint checks
• Follow existing test naming convention: Test{Feature}{Behavior} cd backend/database && go test -v -run "TestMigration6" -count=1 ./...
  • NewTestDB updated for new runMigrations signature (passes empty dbPath)
  • NewTestDBWithLibrary helper exists for tests needing a pre-created library
  • TestMigration6FreshDB verifies all tables, columns, and VIEW exist
  • TestMigration6LibraryQueries verifies CRUD and unique constraint
  • TestMigration6PhantomPlaylistTracks verifies SET NULL FK + phantom metadata preservation
  • TestMigration6AudioFilesLibraryFK verifies FK enforcement
  • TestMigration6TrackMetadataViewHasLibraryID verifies VIEW includes library_id
  • All tests pass

- `go generate ./...` succeeds in backend/database - `go build ./...` succeeds from project root - `go test ./backend/database/... -count=1` — all tests pass including new migration tests - `go test ./backend/playlist/... -count=1` — playlist package still compiles and tests pass (updated AddPlaylistTrack callers) - `golangci-lint run ./backend/...` — no new lint errors

<success_criteria>

• All 7 library CRUD queries generated and working
• Playlist queries correctly handle phantom tracks (nullable audio_file_id, phantom columns)
• Audio file queries support library filtering
• Migration tests verify both fresh install and upgrade paths
• SET NULL FK behavior verified: deleting audio_file preserves playlist_track with phantom metadata
• NewTestDBWithLibrary helper available for downstream test usage </success_criteria>

After completion, create `.planning/phases/10-schema-migration/10-02-SUMMARY.md`