yellowjacket/.planning/milestones/v1.0-phases/07-backend-performance/07-01-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
07-backend-performance	01	execute	1		backend/queue/persistence.go backend/queue/queue.go	true	PERF-01 PERF-02	truths artifacts key_links AddTrack persists a single INSERT + position shift instead of DELETE ALL + batch INSERT RemoveTrack persists a single DELETE + position shift instead of DELETE ALL + batch INSERT InsertNext/InsertNextTracks/InsertTracksAt persist incremental INSERTs + position shift instead of DELETE ALL + batch INSERT SetQueue Phase 2 skips file paths already resolved in Phase 1, avoiding redundant lookupTrackMetaBatch work Bulk operations (SetQueue, Clear, MoveQueueTracks) still use the full DELETE ALL + batch INSERT pattern All existing queue persistence roundtrip tests pass path provides contains backend/queue/persistence.go Incremental persist helpers: persistAddTrack, persistAddTracks, persistRemoveTrack, persistRemoveTracks, persistInsertTracks func (q *Queue) persistAddTrack path provides contains backend/queue/queue.go Updated AddTrack/RemoveTrack/InsertNext/InsertNextTracks/InsertTracksAt using incremental persistence; resolveRemainingTracks with exclusion set persistAddTrack from to via pattern backend/queue/queue.go (AddTrack) backend/queue/persistence.go (persistAddTrack) direct method call replacing commitMutation q.persistAddTrack from to via pattern backend/queue/queue.go (RemoveTrack) backend/queue/persistence.go (persistRemoveTrack) direct method call replacing commitMutation q.persistRemoveTrack from to via pattern backend/queue/queue.go (resolveRemainingTracks) backend/queue/queue.go (lookupTrackMetaBatch) exclusion set filtering exclude

Optimize queue persistence for single-track and insert-at-position operations, and eliminate redundant database lookups in SetQueue Phase 2.

Purpose: Single-track queue mutations (add, remove) currently rewrite the entire queue_tracks table (DELETE ALL + batch INSERT). This is O(n) where n is the queue length. For a 500-track queue, adding one track rewrites 501 rows. These operations should use incremental INSERT/DELETE with position shifts, making them O(1) for the actual mutation plus O(k) for position shifts (where k is the number of tracks after the mutation point). SetQueue Phase 2 currently re-resolves ALL file paths even though Phase 1 already resolved up to 50 of them — passing the Phase 1 results as an exclusion set eliminates redundant database work.

Output: Modified persistence.go with incremental persist helpers, modified queue.go with updated mutation methods and Phase 2 dedup.

<execution_context> @/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md @/home/caleb/.config/opencode/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @backend/queue/persistence.go @backend/queue/queue.go @backend/queue/emit.go @backend/database/sql/queries/queue.sql @backend/database/sql/sqlcgen/queue.sql.go

From backend/queue/queue.go:

type Track struct {
    ID          int64  `json:"id"`
    AudioFileID int64  `json:"audioFileId"`
    FilePath    string `json:"filePath"`
    Position    int64  `json:"position"`
    Title       string `json:"title"`
    Artist      string `json:"artist"`
}

type trackMeta struct {
    AudioFileID int64
    FilePath    string
    Title       string
    Artist      string
}

func (m trackMeta) toTrack(position int64) Track

// commitMutation persists the current queue state after a mutation.
// When reindex is true, track positions are renumbered first.
// The caller must hold q.mu.
func (q *Queue) commitMutation(reindex bool)

// reindexPositions updates the Position field of all tracks to match slice index.
func (q *Queue) reindexPositions()

From backend/database/sql/sqlcgen/queue.sql.go (existing sqlc queries available):

func (q *Queries) InsertQueueTrack(ctx context.Context, arg InsertQueueTrackParams) (QueueTrack, error)
func (q *Queries) RemoveQueueTrackByPosition(ctx context.Context, position int64) error
func (q *Queries) ShiftQueuePositionsDown(ctx context.Context, position int64) error  // position = position - 1 WHERE position > ?
func (q *Queries) ShiftQueuePositionsUp(ctx context.Context, position int64) error    // position = position + 1 WHERE position >= ?
func (q *Queries) ClearQueueTracks(ctx context.Context) error

Task 1: Add incremental persistence helpers and wire into mutation methods backend/queue/persistence.go, backend/queue/queue.go **In `persistence.go`, add these incremental persistence methods (all assume caller holds q.mu):**

1. persistAddTrack(track Track) — Inserts a single track at position track.Position using InsertQueueTrack. No position shifting needed because AddTrack always appends to the end.

2. persistAddTracks(tracks []Track) — Inserts multiple tracks at consecutive positions at the end of the queue. Use the same InsertQueueTrack in a loop (these are appends, so no position shifting needed). Wrap in a transaction for atomicity (use q.db.BeginTx(), q.db.Queries.WithTx(tx)).

3. persistInsertTracks(tracks []Track, insertPos int) — For insert-at-position operations. In a transaction: (a) Call ShiftQueuePositionsUp with insertPos to make room — but note ShiftQueuePositionsUp shifts by 1, so for N tracks, we need to shift by N. Since the sqlc query only shifts by 1, use a hand-crafted UPDATE: UPDATE queue_tracks SET position = position + ? WHERE position >= ? with args (len(tracks), insertPos). Add a // SAFETY: comment explaining why. (b) Insert each track using InsertQueueTrack with positions insertPos, insertPos+1, ..., insertPos+N-1.

4. persistRemoveTrack(position int) — In a transaction: (a) Call RemoveQueueTrackByPosition(position). (b) Call ShiftQueuePositionsDown(position) to close the gap.

5. persistRemoveTracks(positions []int) — For multi-track removal. Since multiple position shifts interact, use the full persistTracks() rewrite for simplicity (the bulk path is acceptable for multi-remove — the user decision specified bulk operations keep the full rewrite). Just call persistTracks() directly.

In queue.go, update these methods to use incremental persistence instead of commitMutation:

1. AddTrack — Replace q.commitMutation(false) with: q.persistAddTrack(track) then q.persistState(). No reindex needed (appending at end, position is already correct).

2. AddTracks — Replace q.commitMutation(false) with: q.persistAddTracks(newTracks) then q.persistState(). No reindex needed.

3. InsertNext — Replace q.commitMutation(true) with: call q.reindexPositions() first, then q.persistInsertTracks([]Track{track}, insertPos) then q.persistState(). The reindex ensures in-memory positions are correct for subsequent operations.

4. InsertNextTracks — Replace q.commitMutation(true) with: call q.reindexPositions() first, then q.persistInsertTracks(newTracks, insertPos) then q.persistState().

5. InsertTracksAt — Replace q.commitMutation(true) with: call q.reindexPositions() first, then q.persistInsertTracks(newTracks, index) then q.persistState().

6. RemoveTrack — Replace q.commitMutation(true) with: call q.persistRemoveTrack(position) then q.reindexPositions() then q.persistState().

7. RemoveTracks — Replace q.commitMutation(true) with: call q.reindexPositions() then q.persistTracks() (full rewrite, per user decision for bulk ops) then q.persistState().

Keep commitMutation for: Clear, MoveQueueTracks, resolveRemainingTracks — bulk operations that still do full rewrites per user decision.

For the hand-crafted SQL in persistInsertTracks: Use tx.ExecContext(q.db.Ctx, "UPDATE queue_tracks SET position = position + ? WHERE position >= ?", count, insertPos) with a // SAFETY: Multi-row position shift by variable N unsupported by sqlc (shift queries only shift by 1). Bind variables match args; no string interpolation. comment.

Important: Shuffle order regeneration was handled by commitMutation. For all the methods that previously called commitMutation with reindex=true, generateShuffleOrder() was also called if shuffleMode was active. Continue this behavior: after the incremental persist, check q.shuffleMode and call q.generateShuffleOrder() if true. For methods that called commitMutation(false) (AddTrack, AddTracks), shuffle order regeneration was also done if active — preserve this.

Verification approach: Existing persistence roundtrip tests in persistence_test.go exercise SaveState/RestoreState which uses persistTracks (full rewrite). The incremental paths are verified by: (1) the existing queue_test.go tests that call AddTrack/RemoveTrack/InsertNext etc. with a real DB, and (2) adding a focused test. cd backend && go build ./... && go test ./queue/... -race -count=1 - AddTrack/AddTracks use persistAddTrack/persistAddTracks (no full table rewrite) - RemoveTrack uses persistRemoveTrack (single DELETE + position shift, no full table rewrite) - InsertNext/InsertNextTracks/InsertTracksAt use persistInsertTracks (position shift + INSERT, no full table rewrite) - RemoveTracks uses full persistTracks rewrite (acceptable for bulk operations) - MoveQueueTracks, Clear, SetQueue still use commitMutation/persistTracks (unchanged bulk behavior) - All existing tests pass with -race

Task 2: Eliminate redundant lookups in SetQueue Phase 2 backend/queue/queue.go **Modify `resolveRemainingTracks` to accept and use Phase 1's already-resolved metadata:**

1. Change resolveRemainingTracks signature to accept the Phase 1 result map:

   func (q *Queue) resolveRemainingTracks(
       gen int64,
       filePaths []string,
       playingPath string,
       phase1Meta map[string]trackMeta,  // NEW: already-resolved from Phase 1
   )
   

2. Inside resolveRemainingTracks, build the exclusion set from phase1Meta keys. Filter filePaths to get only the paths NOT in phase1Meta before calling lookupTrackMetaBatch:

   // Exclude paths already resolved in Phase 1.
   var unresolvedPaths []string
   for _, fp := range filePaths {
       if _, alreadyResolved := phase1Meta[fp]; !alreadyResolved {
           unresolvedPaths = append(unresolvedPaths, fp)
       }
   }
   
   // Only look up paths that Phase 1 didn't cover.
   remainingMeta := q.lookupTrackMetaBatch(unresolvedPaths)
   
   // Merge Phase 1 results into the lookup.
   for k, v := range phase1Meta {
       remainingMeta[k] = v
   }
   

3. The rest of the method (building tracks from allMeta, finding playingPath, calling commitMutation) uses remainingMeta instead of allMeta. Rename the variable for clarity.

4. Update the call site in SetQueue: Pass batchMeta (the Phase 1 result) to resolveRemainingTracks:

   go q.resolveRemainingTracks(gen, filePaths, playingPath, batchMeta)
   

Keep initialBatchSize at 50 — no changes to the Phase 1 window size (per user decision).

Result: For a 1000-track SetQueue where Phase 1 resolves 50, Phase 2 now queries only 950 paths instead of all 1000. The 50 already-resolved paths are merged from the Phase 1 map. cd backend && go build ./... && go test ./queue/... -race -count=1 - resolveRemainingTracks accepts phase1Meta parameter - Phase 2 filters out already-resolved paths before calling lookupTrackMetaBatch - Phase 1 results are merged into Phase 2 results - SetQueue call site passes batchMeta to resolveRemainingTracks - initialBatchSize remains at 50 - All existing tests pass with -race

```bash # All queue tests pass with race detector cd backend && go test ./queue/... -race -count=1 -v

Build succeeds

cd backend && go build ./...

Lint passes

make lint

</verification>

<success_criteria>
- Single-track add/remove uses incremental INSERT/DELETE (not full table rewrite)
- Insert-at-position uses position shift + INSERT (not full table rewrite)
- SetQueue Phase 2 only queries unreolved paths (not all paths)
- All existing queue tests pass with -race
- No linting errors
</success_criteria>

<output>
After completion, create `.planning/phases/07-backend-performance/07-01-SUMMARY.md`
</output>