---
phase: 07-backend-performance
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- backend/queue/persistence.go
- backend/queue/queue.go
autonomous: true
requirements:
- PERF-01
- PERF-02
must_haves:
truths:
- "AddTrack persists a single INSERT + position shift instead of DELETE ALL + batch INSERT"
- "RemoveTrack persists a single DELETE + position shift instead of DELETE ALL + batch INSERT"
- "InsertNext/InsertNextTracks/InsertTracksAt persist incremental INSERTs + position shift instead of DELETE ALL + batch INSERT"
- "SetQueue Phase 2 skips file paths already resolved in Phase 1, avoiding redundant lookupTrackMetaBatch work"
- "Bulk operations (SetQueue, Clear, MoveQueueTracks) still use the full DELETE ALL + batch INSERT pattern"
- "All existing queue persistence roundtrip tests pass"
artifacts:
- path: "backend/queue/persistence.go"
provides: "Incremental persist helpers: persistAddTrack, persistAddTracks, persistRemoveTrack, persistRemoveTracks, persistInsertTracks"
contains: "func (q *Queue) persistAddTrack"
- path: "backend/queue/queue.go"
provides: "Updated AddTrack/RemoveTrack/InsertNext/InsertNextTracks/InsertTracksAt using incremental persistence; resolveRemainingTracks with exclusion set"
contains: "persistAddTrack"
key_links:
- from: "backend/queue/queue.go (AddTrack)"
to: "backend/queue/persistence.go (persistAddTrack)"
via: "direct method call replacing commitMutation"
pattern: "q\\.persistAddTrack"
- from: "backend/queue/queue.go (RemoveTrack)"
to: "backend/queue/persistence.go (persistRemoveTrack)"
via: "direct method call replacing commitMutation"
pattern: "q\\.persistRemoveTrack"
- from: "backend/queue/queue.go (resolveRemainingTracks)"
to: "backend/queue/queue.go (lookupTrackMetaBatch)"
via: "exclusion set filtering"
pattern: "exclude"
---
Optimize queue persistence for single-track and insert-at-position operations, and eliminate redundant database lookups in SetQueue Phase 2.
Purpose: Single-track queue mutations (add, remove) currently rewrite the entire queue_tracks table (DELETE ALL + batch INSERT). This is O(n) where n is the queue length. For a 500-track queue, adding one track rewrites 501 rows. These operations should use incremental INSERT/DELETE with position shifts, making them O(1) for the actual mutation plus O(k) for position shifts (where k is the number of tracks after the mutation point). SetQueue Phase 2 currently re-resolves ALL file paths even though Phase 1 already resolved up to 50 of them — passing the Phase 1 results as an exclusion set eliminates redundant database work.
Output: Modified persistence.go with incremental persist helpers, modified queue.go with updated mutation methods and Phase 2 dedup.
@/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md
@/home/caleb/.config/opencode/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@backend/queue/persistence.go
@backend/queue/queue.go
@backend/queue/emit.go
@backend/database/sql/queries/queue.sql
@backend/database/sql/sqlcgen/queue.sql.go
From backend/queue/queue.go:
```go
type Track struct {
ID int64 `json:"id"`
AudioFileID int64 `json:"audioFileId"`
FilePath string `json:"filePath"`
Position int64 `json:"position"`
Title string `json:"title"`
Artist string `json:"artist"`
}
type trackMeta struct {
AudioFileID int64
FilePath string
Title string
Artist string
}
func (m trackMeta) toTrack(position int64) Track
// commitMutation persists the current queue state after a mutation.
// When reindex is true, track positions are renumbered first.
// The caller must hold q.mu.
func (q *Queue) commitMutation(reindex bool)
// reindexPositions updates the Position field of all tracks to match slice index.
func (q *Queue) reindexPositions()
```
From backend/database/sql/sqlcgen/queue.sql.go (existing sqlc queries available):
```go
func (q *Queries) InsertQueueTrack(ctx context.Context, arg InsertQueueTrackParams) (QueueTrack, error)
func (q *Queries) RemoveQueueTrackByPosition(ctx context.Context, position int64) error
func (q *Queries) ShiftQueuePositionsDown(ctx context.Context, position int64) error // position = position - 1 WHERE position > ?
func (q *Queries) ShiftQueuePositionsUp(ctx context.Context, position int64) error // position = position + 1 WHERE position >= ?
func (q *Queries) ClearQueueTracks(ctx context.Context) error
```
Task 1: Add incremental persistence helpers and wire into mutation methods
backend/queue/persistence.go, backend/queue/queue.go
**In `persistence.go`, add these incremental persistence methods (all assume caller holds q.mu):**
1. `persistAddTrack(track Track)` — Inserts a single track at position `track.Position` using `InsertQueueTrack`. No position shifting needed because AddTrack always appends to the end.
2. `persistAddTracks(tracks []Track)` — Inserts multiple tracks at consecutive positions at the end of the queue. Use the same `InsertQueueTrack` in a loop (these are appends, so no position shifting needed). Wrap in a transaction for atomicity (use `q.db.BeginTx()`, `q.db.Queries.WithTx(tx)`).
3. `persistInsertTracks(tracks []Track, insertPos int)` — For insert-at-position operations. In a transaction: (a) Call `ShiftQueuePositionsUp` with `insertPos` to make room — but note `ShiftQueuePositionsUp` shifts by 1, so for N tracks, we need to shift by N. Since the sqlc query only shifts by 1, use a hand-crafted UPDATE: `UPDATE queue_tracks SET position = position + ? WHERE position >= ?` with args (len(tracks), insertPos). Add a `// SAFETY:` comment explaining why. (b) Insert each track using `InsertQueueTrack` with positions `insertPos`, `insertPos+1`, ..., `insertPos+N-1`.
4. `persistRemoveTrack(position int)` — In a transaction: (a) Call `RemoveQueueTrackByPosition(position)`. (b) Call `ShiftQueuePositionsDown(position)` to close the gap.
5. `persistRemoveTracks(positions []int)` — For multi-track removal. Since multiple position shifts interact, use the full `persistTracks()` rewrite for simplicity (the bulk path is acceptable for multi-remove — the user decision specified bulk operations keep the full rewrite). Just call `persistTracks()` directly.
**In `queue.go`, update these methods to use incremental persistence instead of `commitMutation`:**
1. `AddTrack` — Replace `q.commitMutation(false)` with: `q.persistAddTrack(track)` then `q.persistState()`. No reindex needed (appending at end, position is already correct).
2. `AddTracks` — Replace `q.commitMutation(false)` with: `q.persistAddTracks(newTracks)` then `q.persistState()`. No reindex needed.
3. `InsertNext` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks([]Track{track}, insertPos)` then `q.persistState()`. The reindex ensures in-memory positions are correct for subsequent operations.
4. `InsertNextTracks` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks(newTracks, insertPos)` then `q.persistState()`.
5. `InsertTracksAt` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks(newTracks, index)` then `q.persistState()`.
6. `RemoveTrack` — Replace `q.commitMutation(true)` with: call `q.persistRemoveTrack(position)` then `q.reindexPositions()` then `q.persistState()`.
7. `RemoveTracks` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` then `q.persistTracks()` (full rewrite, per user decision for bulk ops) then `q.persistState()`.
**Keep `commitMutation` for**: `Clear`, `MoveQueueTracks`, `resolveRemainingTracks` — bulk operations that still do full rewrites per user decision.
**For the hand-crafted SQL in `persistInsertTracks`:** Use `tx.ExecContext(q.db.Ctx, "UPDATE queue_tracks SET position = position + ? WHERE position >= ?", count, insertPos)` with a `// SAFETY: Multi-row position shift by variable N unsupported by sqlc (shift queries only shift by 1). Bind variables match args; no string interpolation.` comment.
**Important:** Shuffle order regeneration was handled by `commitMutation`. For all the methods that previously called `commitMutation` with reindex=true, `generateShuffleOrder()` was also called if shuffleMode was active. Continue this behavior: after the incremental persist, check `q.shuffleMode` and call `q.generateShuffleOrder()` if true. For methods that called `commitMutation(false)` (AddTrack, AddTracks), shuffle order regeneration was also done if active — preserve this.
**Verification approach:** Existing persistence roundtrip tests in `persistence_test.go` exercise `SaveState`/`RestoreState` which uses `persistTracks` (full rewrite). The incremental paths are verified by: (1) the existing queue_test.go tests that call AddTrack/RemoveTrack/InsertNext etc. with a real DB, and (2) adding a focused test.
cd backend && go build ./... && go test ./queue/... -race -count=1
- AddTrack/AddTracks use persistAddTrack/persistAddTracks (no full table rewrite)
- RemoveTrack uses persistRemoveTrack (single DELETE + position shift, no full table rewrite)
- InsertNext/InsertNextTracks/InsertTracksAt use persistInsertTracks (position shift + INSERT, no full table rewrite)
- RemoveTracks uses full persistTracks rewrite (acceptable for bulk operations)
- MoveQueueTracks, Clear, SetQueue still use commitMutation/persistTracks (unchanged bulk behavior)
- All existing tests pass with -race
Task 2: Eliminate redundant lookups in SetQueue Phase 2
backend/queue/queue.go
**Modify `resolveRemainingTracks` to accept and use Phase 1's already-resolved metadata:**
1. Change `resolveRemainingTracks` signature to accept the Phase 1 result map:
```go
func (q *Queue) resolveRemainingTracks(
gen int64,
filePaths []string,
playingPath string,
phase1Meta map[string]trackMeta, // NEW: already-resolved from Phase 1
)
```
2. Inside `resolveRemainingTracks`, build the exclusion set from `phase1Meta` keys. Filter `filePaths` to get only the paths NOT in `phase1Meta` before calling `lookupTrackMetaBatch`:
```go
// Exclude paths already resolved in Phase 1.
var unresolvedPaths []string
for _, fp := range filePaths {
if _, alreadyResolved := phase1Meta[fp]; !alreadyResolved {
unresolvedPaths = append(unresolvedPaths, fp)
}
}
// Only look up paths that Phase 1 didn't cover.
remainingMeta := q.lookupTrackMetaBatch(unresolvedPaths)
// Merge Phase 1 results into the lookup.
for k, v := range phase1Meta {
remainingMeta[k] = v
}
```
3. The rest of the method (building tracks from `allMeta`, finding `playingPath`, calling `commitMutation`) uses `remainingMeta` instead of `allMeta`. Rename the variable for clarity.
4. **Update the call site in `SetQueue`:** Pass `batchMeta` (the Phase 1 result) to `resolveRemainingTracks`:
```go
go q.resolveRemainingTracks(gen, filePaths, playingPath, batchMeta)
```
**Keep `initialBatchSize` at 50** — no changes to the Phase 1 window size (per user decision).
**Result:** For a 1000-track SetQueue where Phase 1 resolves 50, Phase 2 now queries only 950 paths instead of all 1000. The 50 already-resolved paths are merged from the Phase 1 map.
cd backend && go build ./... && go test ./queue/... -race -count=1
- resolveRemainingTracks accepts phase1Meta parameter
- Phase 2 filters out already-resolved paths before calling lookupTrackMetaBatch
- Phase 1 results are merged into Phase 2 results
- SetQueue call site passes batchMeta to resolveRemainingTracks
- initialBatchSize remains at 50
- All existing tests pass with -race
```bash
# All queue tests pass with race detector
cd backend && go test ./queue/... -race -count=1 -v
# Build succeeds
cd backend && go build ./...
# Lint passes
make lint
```
- Single-track add/remove uses incremental INSERT/DELETE (not full table rewrite)
- Insert-at-position uses position shift + INSERT (not full table rewrite)
- SetQueue Phase 2 only queries unreolved paths (not all paths)
- All existing queue tests pass with -race
- No linting errors