yellowjacket/.planning/phases/10-schema-migration/10-02-PLAN.md

---
phase: 10-schema-migration
plan: 02
type: execute
wave: 2
depends_on:
  - 10-01
files_modified:
  - backend/database/sql/queries/libraries.sql
  - backend/database/sql/queries/audio_files.sql
  - backend/database/sql/queries/playlists.sql
  - backend/database/sql/sqlcgen/db.go
  - backend/database/sql/sqlcgen/models.go
  - backend/database/sql/sqlcgen/querier.go
  - backend/database/sql/sqlcgen/libraries.sql.go
  - backend/database/sql/sqlcgen/audio_files.sql.go
  - backend/database/sql/sqlcgen/playlists.sql.go
  - backend/database/testhelper.go
  - backend/database/database_test.go
autonomous: true
requirements:
  - LIB-04
  - LIB-05

must_haves:
  truths:
    - "sqlc-generated queries exist for library CRUD (create, get, list, delete)"
    - "Playlist track queries handle nullable audio_file_id and phantom columns"
    - "Audio file queries accept library_id parameter"
    - "Migration tests verify upgrade path from v5 to v6"
    - "Migration tests verify fresh database creates correct schema"
    - "Migration tests verify TOML config is read and default library created"
    - "Test helper NewTestDB creates v6 schema including libraries table"
  artifacts:
    - path: "backend/database/sql/queries/libraries.sql"
      provides: "sqlc query definitions for libraries CRUD"
      contains: "CreateLibrary"
    - path: "backend/database/sql/queries/playlists.sql"
      provides: "Updated playlist queries with phantom column support"
      contains: "phantom_title"
    - path: "backend/database/sql/sqlcgen/libraries.sql.go"
      provides: "Generated Go code for library queries"
      contains: "func.*CreateLibrary"
    - path: "backend/database/database_test.go"
      provides: "Migration 6 integration tests"
      contains: "TestMigration6"
  key_links:
    - from: "backend/database/sql/queries/libraries.sql"
      to: "backend/database/sql/schemas/_libraries.sql"
      via: "sqlc schema awareness"
      pattern: "libraries"
    - from: "backend/database/database_test.go"
      to: "backend/database/database.go migration6"
      via: "NewTestDB runs all migrations"
      pattern: "runMigrations"
---

<objective>
Add sqlc query definitions for the new schema, regenerate Go code, and write migration integration tests.

Purpose: Plan 01 created the schema and migration. This plan makes the new tables usable via type-safe sqlc queries, updates existing playlist queries for phantom support, and verifies the migration works correctly on both fresh and existing databases.

Output: sqlc queries + generated code for libraries and updated playlists + comprehensive migration tests
</objective>

<execution_context>
@/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md
@/home/caleb/.config/opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/10-schema-migration/10-CONTEXT.md
@.planning/phases/10-schema-migration/10-01-SUMMARY.md

<interfaces>
<!-- Key types and contracts from Plan 01 output. -->

From backend/database/sql/schemas/_libraries.sql (created by Plan 01):
```sql
CREATE TABLE IF NOT EXISTS libraries (
    id         INTEGER PRIMARY KEY,
    name       TEXT NOT NULL,
    path       TEXT NOT NULL UNIQUE,
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
```

From backend/database/sql/schemas/playlist_tracks.sql (updated by Plan 01):
```sql
CREATE TABLE IF NOT EXISTS playlist_tracks (
    id INTEGER PRIMARY KEY,
    playlist_id INTEGER NOT NULL,
    audio_file_id INTEGER,          -- nullable for phantom tracks
    position INTEGER NOT NULL,
    phantom_title TEXT,
    phantom_artist TEXT,
    phantom_album TEXT,
    phantom_duration_ms INTEGER,
    phantom_genre TEXT,
    phantom_cover_art_path TEXT,
    FOREIGN KEY(playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
    FOREIGN KEY(audio_file_id) REFERENCES audio_files(id) ON DELETE SET NULL
);
```

From backend/database/sql/schemas/audio_files.sql (updated by Plan 01):
```sql
-- Now includes: library_id int NOT NULL DEFAULT 0
-- FK: FOREIGN KEY(library_id) REFERENCES libraries(id)
-- Index: idx_audio_files_library_id
```

From backend/database/database.go (updated by Plan 01):
```go
func runMigrations(ctx context.Context, db *sql.DB, logger *slog.Logger, dbPath string) error
func backupDatabase(dbPath string, logger *slog.Logger) (string, error)
func migration6MultiLibrary(ctx context.Context, db *sql.DB, logger *slog.Logger, dbPath string) error
```

From backend/database/sqlc.yaml:
```yaml
version: "2"
sql:
  - name: "yellowjacket"
    engine: "sqlite"
    queries: "./sql/queries"
    schema: "./sql/schemas"
    gen:
      go:
        package: "sqlcgen"
        out: "./sql/sqlcgen"
```

Existing sqlc query patterns from playlists.sql:
```sql
-- name: AddPlaylistTrack :one
INSERT INTO playlist_tracks (playlist_id, audio_file_id, position) VALUES (?, ?, ?)
RETURNING *;

-- name: GetPlaylistTracksWithMetadata :many
SELECT pt.id, pt.playlist_id, pt.audio_file_id, pt.position,
    af.file_path, af.length_milliseconds, ...
FROM playlist_tracks pt
JOIN audio_files af ON pt.audio_file_id = af.id
...
```

Existing test patterns from testhelper.go:
```go
func NewTestDB(t *testing.T) *DB  // runs all schemas + migrations
```

Existing test patterns from search_test.go:
```go
func seedSearchData(t *testing.T, db *DB) // creates full entity graph
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add sqlc queries for libraries and update playlist queries for phantom support</name>
  <files>
    backend/database/sql/queries/libraries.sql
    backend/database/sql/queries/audio_files.sql
    backend/database/sql/queries/playlists.sql
    backend/database/sql/sqlcgen/db.go
    backend/database/sql/sqlcgen/models.go
    backend/database/sql/sqlcgen/querier.go
    backend/database/sql/sqlcgen/libraries.sql.go
    backend/database/sql/sqlcgen/audio_files.sql.go
    backend/database/sql/sqlcgen/playlists.sql.go
  </files>
  <action>
**1. Create `backend/database/sql/queries/libraries.sql` (NEW FILE):**

Define the core CRUD queries for the libraries table. These will be consumed by Phase 12 (Library CRUD API) but the type-safe generated code is needed now for migration tests and any early usage.

```sql
-- name: CreateLibrary :one
INSERT INTO libraries (name, path) VALUES (?, ?)
RETURNING *;

-- name: GetLibrary :one
SELECT * FROM libraries WHERE id = ? LIMIT 1;

-- name: GetLibraryByPath :one
SELECT * FROM libraries WHERE path = ? LIMIT 1;

-- name: GetAllLibraries :many
SELECT * FROM libraries ORDER BY name;

-- name: UpdateLibraryName :exec
UPDATE libraries SET name = ? WHERE id = ?;

-- name: DeleteLibrary :exec
DELETE FROM libraries WHERE id = ?;

-- name: CountLibraries :one
SELECT COUNT(*) AS count FROM libraries;
```

**2. Update `backend/database/sql/queries/playlists.sql`:**

The existing queries need updates for the new playlist_tracks schema:

a) **`AddPlaylistTrack`** — Add phantom metadata columns to the INSERT. The caller populates phantom data eagerly on every insert (per user decision):
```sql
-- name: AddPlaylistTrack :one
INSERT INTO playlist_tracks (
    playlist_id, audio_file_id, position,
    phantom_title, phantom_artist, phantom_album,
    phantom_duration_ms, phantom_genre, phantom_cover_art_path
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
RETURNING *;
```

b) **`GetPlaylistTracks`** — Change JOIN to LEFT JOIN on audio_files (audio_file_id is now nullable). Include phantom columns in output so callers can display either live or phantom data:
```sql
-- name: GetPlaylistTracks :many
SELECT pt.id, pt.playlist_id, pt.audio_file_id, pt.position,
    COALESCE(af.file_path, '') AS file_path,
    pt.phantom_title, pt.phantom_artist, pt.phantom_album,
    pt.phantom_duration_ms, pt.phantom_genre, pt.phantom_cover_art_path
FROM playlist_tracks pt
LEFT JOIN audio_files af ON pt.audio_file_id = af.id
WHERE pt.playlist_id = ?
ORDER BY pt.position;
```

c) **`GetPlaylistTracksWithMetadata`** — Same LEFT JOIN change, and include phantom fallback columns. When audio_file_id is NULL (phantom), the live metadata JOINs return NULL and callers use phantom_* columns instead:
```sql
-- name: GetPlaylistTracksWithMetadata :many
SELECT
    pt.id,
    pt.playlist_id,
    pt.audio_file_id,
    pt.position,
    COALESCE(af.file_path, '') AS file_path,
    COALESCE(af.length_milliseconds, 0) AS length_milliseconds,
    COALESCE(r.name, pt.phantom_title, '') AS title,
    COALESCE(ac.text, pt.phantom_artist, '') AS artist,
    COALESCE(rg.name, pt.phantom_album, '') AS album,
    COALESCE(ca.file_path, pt.phantom_cover_art_path, '') AS cover_art_path,
    CASE WHEN pt.audio_file_id IS NULL THEN 1 ELSE 0 END AS is_phantom
FROM playlist_tracks pt
LEFT JOIN audio_files af ON pt.audio_file_id = af.id
LEFT JOIN recordings r ON af.recording_id = r.id
LEFT JOIN artist_credit ac ON r.artist_credit_id = ac.id
LEFT JOIN (
    SELECT recording_id, MIN(release_group_id) AS release_group_id
    FROM release_group_recordings
    GROUP BY recording_id
) rgr ON r.id = rgr.recording_id
LEFT JOIN release_groups rg ON rgr.release_group_id = rg.id
LEFT JOIN cover_art ca ON rg.cover_art_id = ca.id
WHERE pt.playlist_id = ?
ORDER BY pt.position;
```

d) **`GetAllPlaylistTracksWithMetadata`** — Same LEFT JOIN and phantom fallback pattern, without WHERE clause.

e) **`IsTrackInPlaylist`** — Change JOIN to LEFT JOIN (audio_file_id may be NULL for phantom tracks).

f) **`RemovePlaylistTrackByPath`** — Change subquery JOIN to handle nullable audio_file_id.

g) **`GetPlaylistTrackFilePaths`** — Change to LEFT JOIN, filter out NULLs:
```sql
-- name: GetPlaylistTrackFilePaths :many
SELECT COALESCE(af.file_path, '') AS file_path
FROM playlist_tracks pt
LEFT JOIN audio_files af ON pt.audio_file_id = af.id
WHERE pt.playlist_id = ? AND pt.audio_file_id IS NOT NULL
ORDER BY pt.position;
```

**3. Update `backend/database/sql/queries/audio_files.sql`:**

Add a query to get audio files filtered by library:
```sql
-- name: GetAudioFilesByLibrary :many
SELECT * FROM audio_files WHERE library_id = ?;

-- name: CountAudioFilesByLibrary :one
SELECT COUNT(*) AS count FROM audio_files WHERE library_id = ?;
```

**4. Regenerate sqlc code:**

Run from `backend/database/`:
```bash
go generate ./...
```

This regenerates all files in `sql/sqlcgen/` from the updated schemas and queries.

**5. Fix any compilation errors** in the generated code or in callers of the changed query signatures (particularly `AddPlaylistTrack` which now has 9 parameters instead of 3). Check all callers:
- `backend/playlist/playlist.go` — calls `AddPlaylistTrack`. Update to pass phantom metadata.
- Any other callers of changed queries.

For `AddPlaylistTrack` callers: pass the phantom metadata alongside the audio_file_id. The caller should resolve the metadata at insert time (eager population per user decision). Look at how `playlist.go` currently calls it and add the phantom fields. For now, populate phantom data from the track metadata that the caller already has available.

**IMPORTANT:** The playlist package's `AddTrack`/`AddTracks` methods need to resolve phantom metadata before inserting. Look at how `GetPlaylistTracksWithMetadata` resolves metadata — the same JOIN pattern should be used to fetch phantom data before insert. Or simpler: the caller already has the file path → look up metadata from DB → pass as phantom columns.

Create a helper query to resolve phantom metadata for a given audio_file_id:
```sql
-- name: GetTrackPhantomMetadata :one
SELECT
    COALESCE(r.name, '') AS title,
    COALESCE(ac.text, '') AS artist,
    COALESCE(rg.name, '') AS album,
    af.length_milliseconds AS duration_ms,
    CAST(COALESCE(
        (SELECT GROUP_CONCAT(g.name, '||')
         FROM recording_genres rg_sub
         JOIN genres g ON rg_sub.genre_id = g.id
         WHERE rg_sub.recording_id = r.id),
        ''
    ) AS TEXT) AS genre,
    COALESCE(ca.file_path, '') AS cover_art_path
FROM audio_files af
LEFT JOIN recordings r ON af.recording_id = r.id
LEFT JOIN artist_credit ac ON r.artist_credit_id = ac.id
LEFT JOIN (
    SELECT recording_id, MIN(release_group_id) AS release_group_id
    FROM release_group_recordings
    GROUP BY recording_id
) rgr ON r.id = rgr.recording_id
LEFT JOIN release_groups rg ON rgr.release_group_id = rg.id
LEFT JOIN cover_art ca ON rg.cover_art_id = ca.id
WHERE af.id = ?;
```

Add this to `playlists.sql`.

After regenerating, verify compilation:
```bash
cd backend && go build ./...
```

Fix any broken callers of `AddPlaylistTrack` — the signature change from 3 args to 9 args will cause compile errors in the playlist package. Update each caller to:
1. Look up phantom metadata via `GetTrackPhantomMetadata` query
2. Pass all 9 params to `AddPlaylistTrack`
  </action>
  <verify>
    <automated>cd backend/database && go generate ./... && cd ../.. && go build ./... && go vet ./...</automated>
  </verify>
  <done>
    - `libraries.sql` query file exists with 7 CRUD queries
    - `playlists.sql` updated with phantom column support in all track queries
    - `audio_files.sql` has library-filtered query
    - sqlc regenerated successfully (all files in sql/sqlcgen/ updated)
    - `AddPlaylistTrack` callers updated for new 9-param signature
    - `GetTrackPhantomMetadata` helper query exists for eager phantom population
    - `go build ./...` passes from project root
  </done>
</task>

<task type="auto">
  <name>Task 2: Migration integration tests and NewTestDB update</name>
  <files>
    backend/database/testhelper.go
    backend/database/database_test.go
  </files>
  <action>
Write integration tests that verify migration 6 works correctly on both fresh and existing databases. Also update `NewTestDB` for the new schema.

**1. Update `testhelper.go`:**

The `NewTestDB` helper runs all schemas + migrations. Since migration 6 reads a TOML config file, and the test helper uses `:memory:` database with no file path, the migration will skip the TOML reading (existingDir = ""). The test helper needs to handle the updated `runMigrations` signature that now takes `dbPath`:

```go
// Pass empty string for dbPath — in-memory DBs don't need backup.
if err := runMigrations(ctx, db, slog.Default(), ""); err != nil {
    t.Fatalf("could not run migrations: %v", err)
}
```

The backup function should no-op when dbPath is empty. Verify this is handled in the migration 6 code (Plan 01 should have handled it — if not, add a guard).

Also add a `NewTestDBWithLibrary` helper that creates a test DB with a pre-populated library, useful for tests in other packages:

```go
// NewTestDBWithLibrary returns a test DB with a library row pre-inserted.
// Returns the DB and the library ID.
func NewTestDBWithLibrary(t *testing.T, name, path string) (*DB, int64) {
    t.Helper()
    db := NewTestDB(t)
    lib, err := db.Queries.CreateLibrary(db.Ctx, sqlcgen.CreateLibraryParams{
        Name: name,
        Path: path,
    })
    if err != nil {
        t.Fatalf("could not create test library: %v", err)
    }
    return db, lib.ID
}
```

**2. Create/update `database_test.go`:**

Write these test cases:

a) **TestMigration6FreshDB** — Verify that a fresh database (no prior data) creates all expected tables including libraries, and that the schema matches expectations:
```go
func TestMigration6FreshDB(t *testing.T) {
    db := NewTestDB(t)

    // Verify libraries table exists
    var tableCount int
    err := db.QueryRow("SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='libraries'").Scan(&tableCount)
    // assert tableCount == 1

    // Verify audio_files has library_id column
    // Query PRAGMA table_info(audio_files), check for library_id

    // Verify playlist_tracks has phantom columns and nullable audio_file_id
    // Query PRAGMA table_info(playlist_tracks), check columns

    // Verify track_metadata VIEW includes library_id
    // Query PRAGMA table_info(track_metadata), check for library_id — wait, VIEWs don't work with table_info
    // Instead: SELECT sql FROM sqlite_master WHERE name='track_metadata'
    // Assert contains 'library_id'

    // Verify user_version is current (>= 6)
    var version int
    err = db.QueryRow("PRAGMA user_version").Scan(&version)
    // assert version >= 6

    // Verify libraries table is empty on fresh DB
    count, err := db.Queries.CountLibraries(db.Ctx)
    // assert count == 0
}
```

b) **TestMigration6LibraryQueries** — Verify CRUD operations on libraries table work:
```go
func TestMigration6LibraryQueries(t *testing.T) {
    db := NewTestDB(t)

    // Create a library
    lib, err := db.Queries.CreateLibrary(db.Ctx, sqlcgen.CreateLibraryParams{
        Name: "Music",
        Path: "/home/user/Music",
    })
    // assert lib.Name == "Music", lib.Path == "/home/user/Music"
    // assert lib.ID > 0

    // Get by ID
    got, err := db.Queries.GetLibrary(db.Ctx, lib.ID)
    // assert got matches lib

    // Get by path
    gotByPath, err := db.Queries.GetLibraryByPath(db.Ctx, "/home/user/Music")
    // assert gotByPath matches lib

    // Unique path constraint
    _, err = db.Queries.CreateLibrary(db.Ctx, sqlcgen.CreateLibraryParams{
        Name: "Duplicate",
        Path: "/home/user/Music",
    })
    // assert IsUniqueViolation(err)

    // List libraries
    libs, err := db.Queries.GetAllLibraries(db.Ctx)
    // assert len(libs) == 1

    // Update name
    err = db.Queries.UpdateLibraryName(db.Ctx, sqlcgen.UpdateLibraryNameParams{
        Name: "My Music",
        ID:   lib.ID,
    })
    // Verify name changed

    // Delete
    err = db.Queries.DeleteLibrary(db.Ctx, lib.ID)
    count, _ := db.Queries.CountLibraries(db.Ctx)
    // assert count == 0
}
```

c) **TestMigration6PhantomPlaylistTracks** — Verify playlist tracks work with phantom columns:
```go
func TestMigration6PhantomPlaylistTracks(t *testing.T) {
    db, libID := NewTestDBWithLibrary(t, "Test", "/test/music")

    // Create prerequisite data: file_type, recording, audio_file
    // (use pattern from existing seedSearchData or seedAudioFiles)

    // Create playlist
    playlist, _ := db.Queries.CreatePlaylist(db.Ctx, "Test Playlist")

    // Add track with phantom metadata (eager population)
    track, err := db.Queries.AddPlaylistTrack(db.Ctx, sqlcgen.AddPlaylistTrackParams{
        PlaylistID:           playlist.ID,
        AudioFileID:          sql.NullInt64{Int64: audioFileID, Valid: true},
        Position:             0,
        PhantomTitle:         sql.NullString{String: "Test Song", Valid: true},
        PhantomArtist:        sql.NullString{String: "Test Artist", Valid: true},
        PhantomAlbum:         sql.NullString{String: "Test Album", Valid: true},
        PhantomDurationMs:    sql.NullInt64{Int64: 180000, Valid: true},
        PhantomGenre:         sql.NullString{String: "Rock", Valid: true},
        PhantomCoverArtPath:  sql.NullString{String: "", Valid: false},
    })
    // assert track created

    // Delete the audio_file — should SET NULL on audio_file_id
    // (not CASCADE delete the playlist_track)
    _, err = db.ExecContext("DELETE FROM audio_files WHERE id = ?", audioFileID)

    // Verify playlist track still exists with NULL audio_file_id
    tracks, _ := db.Queries.GetPlaylistTracksWithMetadata(db.Ctx, playlist.ID)
    // assert len(tracks) == 1
    // assert tracks[0].AudioFileID is NULL/invalid
    // assert tracks[0].Title == "Test Song" (from phantom)
    // assert tracks[0].IsPhantom == 1
}
```

d) **TestMigration6AudioFilesLibraryFK** — Verify library_id FK enforcement:
```go
func TestMigration6AudioFilesLibraryFK(t *testing.T) {
    db, libID := NewTestDBWithLibrary(t, "Test", "/test")

    // Insert audio_file with valid library_id — should succeed
    // Insert audio_file with invalid library_id (999) — should fail FK check

    // Count files by library
    count, _ := db.Queries.CountAudioFilesByLibrary(db.Ctx, libID)
    // assert count == 1
}
```

e) **TestMigration6TrackMetadataViewHasLibraryID** — Verify the VIEW includes library_id:
```go
func TestMigration6TrackMetadataViewHasLibraryID(t *testing.T) {
    db, libID := NewTestDBWithLibrary(t, "Test", "/test")
    // Insert an audio file with test data
    // Query track_metadata VIEW
    // Verify library_id column is present and has correct value
}
```

**Test patterns to follow:**
- Use `NewTestDB(t)` or `NewTestDBWithLibrary(t, ...)` for setup
- Use `t.Helper()` in helpers
- Use `t.Context()` — NOT `context.Background()`
- Table-driven subtests where appropriate
- Use `database.IsUniqueViolation(err)` for constraint checks
- Follow existing test naming convention: `Test{Feature}{Behavior}`
  </action>
  <verify>
    <automated>cd backend/database && go test -v -run "TestMigration6" -count=1 ./...</automated>
  </verify>
  <done>
    - `NewTestDB` updated for new runMigrations signature (passes empty dbPath)
    - `NewTestDBWithLibrary` helper exists for tests needing a pre-created library
    - TestMigration6FreshDB verifies all tables, columns, and VIEW exist
    - TestMigration6LibraryQueries verifies CRUD and unique constraint
    - TestMigration6PhantomPlaylistTracks verifies SET NULL FK + phantom metadata preservation
    - TestMigration6AudioFilesLibraryFK verifies FK enforcement
    - TestMigration6TrackMetadataViewHasLibraryID verifies VIEW includes library_id
    - All tests pass
  </done>
</task>

</tasks>

<verification>
- `go generate ./...` succeeds in backend/database
- `go build ./...` succeeds from project root
- `go test ./backend/database/... -count=1` — all tests pass including new migration tests
- `go test ./backend/playlist/... -count=1` — playlist package still compiles and tests pass (updated AddPlaylistTrack callers)
- `golangci-lint run ./backend/...` — no new lint errors
</verification>

<success_criteria>
- All 7 library CRUD queries generated and working
- Playlist queries correctly handle phantom tracks (nullable audio_file_id, phantom columns)
- Audio file queries support library filtering
- Migration tests verify both fresh install and upgrade paths
- SET NULL FK behavior verified: deleting audio_file preserves playlist_track with phantom metadata
- NewTestDBWithLibrary helper available for downstream test usage
</success_criteria>

<output>
After completion, create `.planning/phases/10-schema-migration/10-02-SUMMARY.md`
</output>