yellowjacket/.planning/milestones/v1.0-phases/07-backend-performance/07-01-PLAN.md

---
phase: 07-backend-performance
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - backend/queue/persistence.go
  - backend/queue/queue.go
autonomous: true
requirements:
  - PERF-01
  - PERF-02

must_haves:
  truths:
    - "AddTrack persists a single INSERT + position shift instead of DELETE ALL + batch INSERT"
    - "RemoveTrack persists a single DELETE + position shift instead of DELETE ALL + batch INSERT"
    - "InsertNext/InsertNextTracks/InsertTracksAt persist incremental INSERTs + position shift instead of DELETE ALL + batch INSERT"
    - "SetQueue Phase 2 skips file paths already resolved in Phase 1, avoiding redundant lookupTrackMetaBatch work"
    - "Bulk operations (SetQueue, Clear, MoveQueueTracks) still use the full DELETE ALL + batch INSERT pattern"
    - "All existing queue persistence roundtrip tests pass"
  artifacts:
    - path: "backend/queue/persistence.go"
      provides: "Incremental persist helpers: persistAddTrack, persistAddTracks, persistRemoveTrack, persistRemoveTracks, persistInsertTracks"
      contains: "func (q *Queue) persistAddTrack"
    - path: "backend/queue/queue.go"
      provides: "Updated AddTrack/RemoveTrack/InsertNext/InsertNextTracks/InsertTracksAt using incremental persistence; resolveRemainingTracks with exclusion set"
      contains: "persistAddTrack"
  key_links:
    - from: "backend/queue/queue.go (AddTrack)"
      to: "backend/queue/persistence.go (persistAddTrack)"
      via: "direct method call replacing commitMutation"
      pattern: "q\\.persistAddTrack"
    - from: "backend/queue/queue.go (RemoveTrack)"
      to: "backend/queue/persistence.go (persistRemoveTrack)"
      via: "direct method call replacing commitMutation"
      pattern: "q\\.persistRemoveTrack"
    - from: "backend/queue/queue.go (resolveRemainingTracks)"
      to: "backend/queue/queue.go (lookupTrackMetaBatch)"
      via: "exclusion set filtering"
      pattern: "exclude"
---

<objective>
Optimize queue persistence for single-track and insert-at-position operations, and eliminate redundant database lookups in SetQueue Phase 2.

Purpose: Single-track queue mutations (add, remove) currently rewrite the entire queue_tracks table (DELETE ALL + batch INSERT). This is O(n) where n is the queue length. For a 500-track queue, adding one track rewrites 501 rows. These operations should use incremental INSERT/DELETE with position shifts, making them O(1) for the actual mutation plus O(k) for position shifts (where k is the number of tracks after the mutation point). SetQueue Phase 2 currently re-resolves ALL file paths even though Phase 1 already resolved up to 50 of them — passing the Phase 1 results as an exclusion set eliminates redundant database work.

Output: Modified persistence.go with incremental persist helpers, modified queue.go with updated mutation methods and Phase 2 dedup.
</objective>

<execution_context>
@/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md
@/home/caleb/.config/opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@backend/queue/persistence.go
@backend/queue/queue.go
@backend/queue/emit.go
@backend/database/sql/queries/queue.sql
@backend/database/sql/sqlcgen/queue.sql.go
</context>

<interfaces>
<!-- Key types and contracts the executor needs. Extracted from codebase. -->
<!-- Executor should use these directly — no codebase exploration needed. -->

From backend/queue/queue.go:
```go
type Track struct {
    ID          int64  `json:"id"`
    AudioFileID int64  `json:"audioFileId"`
    FilePath    string `json:"filePath"`
    Position    int64  `json:"position"`
    Title       string `json:"title"`
    Artist      string `json:"artist"`
}

type trackMeta struct {
    AudioFileID int64
    FilePath    string
    Title       string
    Artist      string
}

func (m trackMeta) toTrack(position int64) Track

// commitMutation persists the current queue state after a mutation.
// When reindex is true, track positions are renumbered first.
// The caller must hold q.mu.
func (q *Queue) commitMutation(reindex bool)

// reindexPositions updates the Position field of all tracks to match slice index.
func (q *Queue) reindexPositions()
```

From backend/database/sql/sqlcgen/queue.sql.go (existing sqlc queries available):
```go
func (q *Queries) InsertQueueTrack(ctx context.Context, arg InsertQueueTrackParams) (QueueTrack, error)
func (q *Queries) RemoveQueueTrackByPosition(ctx context.Context, position int64) error
func (q *Queries) ShiftQueuePositionsDown(ctx context.Context, position int64) error  // position = position - 1 WHERE position > ?
func (q *Queries) ShiftQueuePositionsUp(ctx context.Context, position int64) error    // position = position + 1 WHERE position >= ?
func (q *Queries) ClearQueueTracks(ctx context.Context) error
```
</interfaces>

<tasks>

<task type="auto">
  <name>Task 1: Add incremental persistence helpers and wire into mutation methods</name>
  <files>backend/queue/persistence.go, backend/queue/queue.go</files>
  <action>
**In `persistence.go`, add these incremental persistence methods (all assume caller holds q.mu):**

1. `persistAddTrack(track Track)` — Inserts a single track at position `track.Position` using `InsertQueueTrack`. No position shifting needed because AddTrack always appends to the end.

2. `persistAddTracks(tracks []Track)` — Inserts multiple tracks at consecutive positions at the end of the queue. Use the same `InsertQueueTrack` in a loop (these are appends, so no position shifting needed). Wrap in a transaction for atomicity (use `q.db.BeginTx()`, `q.db.Queries.WithTx(tx)`).

3. `persistInsertTracks(tracks []Track, insertPos int)` — For insert-at-position operations. In a transaction: (a) Call `ShiftQueuePositionsUp` with `insertPos` to make room — but note `ShiftQueuePositionsUp` shifts by 1, so for N tracks, we need to shift by N. Since the sqlc query only shifts by 1, use a hand-crafted UPDATE: `UPDATE queue_tracks SET position = position + ? WHERE position >= ?` with args (len(tracks), insertPos). Add a `// SAFETY:` comment explaining why. (b) Insert each track using `InsertQueueTrack` with positions `insertPos`, `insertPos+1`, ..., `insertPos+N-1`.

4. `persistRemoveTrack(position int)` — In a transaction: (a) Call `RemoveQueueTrackByPosition(position)`. (b) Call `ShiftQueuePositionsDown(position)` to close the gap.

5. `persistRemoveTracks(positions []int)` — For multi-track removal. Since multiple position shifts interact, use the full `persistTracks()` rewrite for simplicity (the bulk path is acceptable for multi-remove — the user decision specified bulk operations keep the full rewrite). Just call `persistTracks()` directly.

**In `queue.go`, update these methods to use incremental persistence instead of `commitMutation`:**

1. `AddTrack` — Replace `q.commitMutation(false)` with: `q.persistAddTrack(track)` then `q.persistState()`. No reindex needed (appending at end, position is already correct).

2. `AddTracks` — Replace `q.commitMutation(false)` with: `q.persistAddTracks(newTracks)` then `q.persistState()`. No reindex needed.

3. `InsertNext` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks([]Track{track}, insertPos)` then `q.persistState()`. The reindex ensures in-memory positions are correct for subsequent operations.

4. `InsertNextTracks` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks(newTracks, insertPos)` then `q.persistState()`.

5. `InsertTracksAt` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks(newTracks, index)` then `q.persistState()`.

6. `RemoveTrack` — Replace `q.commitMutation(true)` with: call `q.persistRemoveTrack(position)` then `q.reindexPositions()` then `q.persistState()`.

7. `RemoveTracks` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` then `q.persistTracks()` (full rewrite, per user decision for bulk ops) then `q.persistState()`.

**Keep `commitMutation` for**: `Clear`, `MoveQueueTracks`, `resolveRemainingTracks` — bulk operations that still do full rewrites per user decision.

**For the hand-crafted SQL in `persistInsertTracks`:** Use `tx.ExecContext(q.db.Ctx, "UPDATE queue_tracks SET position = position + ? WHERE position >= ?", count, insertPos)` with a `// SAFETY: Multi-row position shift by variable N unsupported by sqlc (shift queries only shift by 1). Bind variables match args; no string interpolation.` comment.

**Important:** Shuffle order regeneration was handled by `commitMutation`. For all the methods that previously called `commitMutation` with reindex=true, `generateShuffleOrder()` was also called if shuffleMode was active. Continue this behavior: after the incremental persist, check `q.shuffleMode` and call `q.generateShuffleOrder()` if true. For methods that called `commitMutation(false)` (AddTrack, AddTracks), shuffle order regeneration was also done if active — preserve this.

**Verification approach:** Existing persistence roundtrip tests in `persistence_test.go` exercise `SaveState`/`RestoreState` which uses `persistTracks` (full rewrite). The incremental paths are verified by: (1) the existing queue_test.go tests that call AddTrack/RemoveTrack/InsertNext etc. with a real DB, and (2) adding a focused test.
  </action>
  <verify>
    cd backend && go build ./... && go test ./queue/... -race -count=1
  </verify>
  <done>
    - AddTrack/AddTracks use persistAddTrack/persistAddTracks (no full table rewrite)
    - RemoveTrack uses persistRemoveTrack (single DELETE + position shift, no full table rewrite)
    - InsertNext/InsertNextTracks/InsertTracksAt use persistInsertTracks (position shift + INSERT, no full table rewrite)
    - RemoveTracks uses full persistTracks rewrite (acceptable for bulk operations)
    - MoveQueueTracks, Clear, SetQueue still use commitMutation/persistTracks (unchanged bulk behavior)
    - All existing tests pass with -race
  </done>
</task>

<task type="auto">
  <name>Task 2: Eliminate redundant lookups in SetQueue Phase 2</name>
  <files>backend/queue/queue.go</files>
  <action>
**Modify `resolveRemainingTracks` to accept and use Phase 1's already-resolved metadata:**

1. Change `resolveRemainingTracks` signature to accept the Phase 1 result map:
   ```go
   func (q *Queue) resolveRemainingTracks(
       gen int64,
       filePaths []string,
       playingPath string,
       phase1Meta map[string]trackMeta,  // NEW: already-resolved from Phase 1
   )
   ```

2. Inside `resolveRemainingTracks`, build the exclusion set from `phase1Meta` keys. Filter `filePaths` to get only the paths NOT in `phase1Meta` before calling `lookupTrackMetaBatch`:
   ```go
   // Exclude paths already resolved in Phase 1.
   var unresolvedPaths []string
   for _, fp := range filePaths {
       if _, alreadyResolved := phase1Meta[fp]; !alreadyResolved {
           unresolvedPaths = append(unresolvedPaths, fp)
       }
   }

   // Only look up paths that Phase 1 didn't cover.
   remainingMeta := q.lookupTrackMetaBatch(unresolvedPaths)

   // Merge Phase 1 results into the lookup.
   for k, v := range phase1Meta {
       remainingMeta[k] = v
   }
   ```

3. The rest of the method (building tracks from `allMeta`, finding `playingPath`, calling `commitMutation`) uses `remainingMeta` instead of `allMeta`. Rename the variable for clarity.

4. **Update the call site in `SetQueue`:** Pass `batchMeta` (the Phase 1 result) to `resolveRemainingTracks`:
   ```go
   go q.resolveRemainingTracks(gen, filePaths, playingPath, batchMeta)
   ```

**Keep `initialBatchSize` at 50** — no changes to the Phase 1 window size (per user decision).

**Result:** For a 1000-track SetQueue where Phase 1 resolves 50, Phase 2 now queries only 950 paths instead of all 1000. The 50 already-resolved paths are merged from the Phase 1 map.
  </action>
  <verify>
    cd backend && go build ./... && go test ./queue/... -race -count=1
  </verify>
  <done>
    - resolveRemainingTracks accepts phase1Meta parameter
    - Phase 2 filters out already-resolved paths before calling lookupTrackMetaBatch
    - Phase 1 results are merged into Phase 2 results
    - SetQueue call site passes batchMeta to resolveRemainingTracks
    - initialBatchSize remains at 50
    - All existing tests pass with -race
  </done>
</task>

</tasks>

<verification>
```bash
# All queue tests pass with race detector
cd backend && go test ./queue/... -race -count=1 -v

# Build succeeds
cd backend && go build ./...

# Lint passes
make lint
```
</verification>

<success_criteria>
- Single-track add/remove uses incremental INSERT/DELETE (not full table rewrite)
- Insert-at-position uses position shift + INSERT (not full table rewrite)
- SetQueue Phase 2 only queries unreolved paths (not all paths)
- All existing queue tests pass with -race
- No linting errors
</success_criteria>

<output>
After completion, create `.planning/phases/07-backend-performance/07-01-SUMMARY.md`
</output>