docs(07): create phase plan
This commit is contained in:
@@ -0,0 +1,251 @@
|
||||
---
|
||||
phase: 07-backend-performance
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- backend/queue/persistence.go
|
||||
- backend/queue/queue.go
|
||||
autonomous: true
|
||||
requirements:
|
||||
- PERF-01
|
||||
- PERF-02
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "AddTrack persists a single INSERT + position shift instead of DELETE ALL + batch INSERT"
|
||||
- "RemoveTrack persists a single DELETE + position shift instead of DELETE ALL + batch INSERT"
|
||||
- "InsertNext/InsertNextTracks/InsertTracksAt persist incremental INSERTs + position shift instead of DELETE ALL + batch INSERT"
|
||||
- "SetQueue Phase 2 skips file paths already resolved in Phase 1, avoiding redundant lookupTrackMetaBatch work"
|
||||
- "Bulk operations (SetQueue, Clear, MoveQueueTracks) still use the full DELETE ALL + batch INSERT pattern"
|
||||
- "All existing queue persistence roundtrip tests pass"
|
||||
artifacts:
|
||||
- path: "backend/queue/persistence.go"
|
||||
provides: "Incremental persist helpers: persistAddTrack, persistAddTracks, persistRemoveTrack, persistRemoveTracks, persistInsertTracks"
|
||||
contains: "func (q *Queue) persistAddTrack"
|
||||
- path: "backend/queue/queue.go"
|
||||
provides: "Updated AddTrack/RemoveTrack/InsertNext/InsertNextTracks/InsertTracksAt using incremental persistence; resolveRemainingTracks with exclusion set"
|
||||
contains: "persistAddTrack"
|
||||
key_links:
|
||||
- from: "backend/queue/queue.go (AddTrack)"
|
||||
to: "backend/queue/persistence.go (persistAddTrack)"
|
||||
via: "direct method call replacing commitMutation"
|
||||
pattern: "q\\.persistAddTrack"
|
||||
- from: "backend/queue/queue.go (RemoveTrack)"
|
||||
to: "backend/queue/persistence.go (persistRemoveTrack)"
|
||||
via: "direct method call replacing commitMutation"
|
||||
pattern: "q\\.persistRemoveTrack"
|
||||
- from: "backend/queue/queue.go (resolveRemainingTracks)"
|
||||
to: "backend/queue/queue.go (lookupTrackMetaBatch)"
|
||||
via: "exclusion set filtering"
|
||||
pattern: "exclude"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Optimize queue persistence for single-track and insert-at-position operations, and eliminate redundant database lookups in SetQueue Phase 2.
|
||||
|
||||
Purpose: Single-track queue mutations (add, remove) currently rewrite the entire queue_tracks table (DELETE ALL + batch INSERT). This is O(n) where n is the queue length. For a 500-track queue, adding one track rewrites 501 rows. These operations should use incremental INSERT/DELETE with position shifts, making them O(1) for the actual mutation plus O(k) for position shifts (where k is the number of tracks after the mutation point). SetQueue Phase 2 currently re-resolves ALL file paths even though Phase 1 already resolved up to 50 of them — passing the Phase 1 results as an exclusion set eliminates redundant database work.
|
||||
|
||||
Output: Modified persistence.go with incremental persist helpers, modified queue.go with updated mutation methods and Phase 2 dedup.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md
|
||||
@/home/caleb/.config/opencode/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@backend/queue/persistence.go
|
||||
@backend/queue/queue.go
|
||||
@backend/queue/emit.go
|
||||
@backend/database/sql/queries/queue.sql
|
||||
@backend/database/sql/sqlcgen/queue.sql.go
|
||||
</context>
|
||||
|
||||
<interfaces>
|
||||
<!-- Key types and contracts the executor needs. Extracted from codebase. -->
|
||||
<!-- Executor should use these directly — no codebase exploration needed. -->
|
||||
|
||||
From backend/queue/queue.go:
|
||||
```go
|
||||
type Track struct {
|
||||
ID int64 `json:"id"`
|
||||
AudioFileID int64 `json:"audioFileId"`
|
||||
FilePath string `json:"filePath"`
|
||||
Position int64 `json:"position"`
|
||||
Title string `json:"title"`
|
||||
Artist string `json:"artist"`
|
||||
}
|
||||
|
||||
type trackMeta struct {
|
||||
AudioFileID int64
|
||||
FilePath string
|
||||
Title string
|
||||
Artist string
|
||||
}
|
||||
|
||||
func (m trackMeta) toTrack(position int64) Track
|
||||
|
||||
// commitMutation persists the current queue state after a mutation.
|
||||
// When reindex is true, track positions are renumbered first.
|
||||
// The caller must hold q.mu.
|
||||
func (q *Queue) commitMutation(reindex bool)
|
||||
|
||||
// reindexPositions updates the Position field of all tracks to match slice index.
|
||||
func (q *Queue) reindexPositions()
|
||||
```
|
||||
|
||||
From backend/database/sql/sqlcgen/queue.sql.go (existing sqlc queries available):
|
||||
```go
|
||||
func (q *Queries) InsertQueueTrack(ctx context.Context, arg InsertQueueTrackParams) (QueueTrack, error)
|
||||
func (q *Queries) RemoveQueueTrackByPosition(ctx context.Context, position int64) error
|
||||
func (q *Queries) ShiftQueuePositionsDown(ctx context.Context, position int64) error // position = position - 1 WHERE position > ?
|
||||
func (q *Queries) ShiftQueuePositionsUp(ctx context.Context, position int64) error // position = position + 1 WHERE position >= ?
|
||||
func (q *Queries) ClearQueueTracks(ctx context.Context) error
|
||||
```
|
||||
</interfaces>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Add incremental persistence helpers and wire into mutation methods</name>
|
||||
<files>backend/queue/persistence.go, backend/queue/queue.go</files>
|
||||
<action>
|
||||
**In `persistence.go`, add these incremental persistence methods (all assume caller holds q.mu):**
|
||||
|
||||
1. `persistAddTrack(track Track)` — Inserts a single track at position `track.Position` using `InsertQueueTrack`. No position shifting needed because AddTrack always appends to the end.
|
||||
|
||||
2. `persistAddTracks(tracks []Track)` — Inserts multiple tracks at consecutive positions at the end of the queue. Use the same `InsertQueueTrack` in a loop (these are appends, so no position shifting needed). Wrap in a transaction for atomicity (use `q.db.BeginTx()`, `q.db.Queries.WithTx(tx)`).
|
||||
|
||||
3. `persistInsertTracks(tracks []Track, insertPos int)` — For insert-at-position operations. In a transaction: (a) Call `ShiftQueuePositionsUp` with `insertPos` to make room — but note `ShiftQueuePositionsUp` shifts by 1, so for N tracks, we need to shift by N. Since the sqlc query only shifts by 1, use a hand-crafted UPDATE: `UPDATE queue_tracks SET position = position + ? WHERE position >= ?` with args (len(tracks), insertPos). Add a `// SAFETY:` comment explaining why. (b) Insert each track using `InsertQueueTrack` with positions `insertPos`, `insertPos+1`, ..., `insertPos+N-1`.
|
||||
|
||||
4. `persistRemoveTrack(position int)` — In a transaction: (a) Call `RemoveQueueTrackByPosition(position)`. (b) Call `ShiftQueuePositionsDown(position)` to close the gap.
|
||||
|
||||
5. `persistRemoveTracks(positions []int)` — For multi-track removal. Since multiple position shifts interact, use the full `persistTracks()` rewrite for simplicity (the bulk path is acceptable for multi-remove — the user decision specified bulk operations keep the full rewrite). Just call `persistTracks()` directly.
|
||||
|
||||
**In `queue.go`, update these methods to use incremental persistence instead of `commitMutation`:**
|
||||
|
||||
1. `AddTrack` — Replace `q.commitMutation(false)` with: `q.persistAddTrack(track)` then `q.persistState()`. No reindex needed (appending at end, position is already correct).
|
||||
|
||||
2. `AddTracks` — Replace `q.commitMutation(false)` with: `q.persistAddTracks(newTracks)` then `q.persistState()`. No reindex needed.
|
||||
|
||||
3. `InsertNext` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks([]Track{track}, insertPos)` then `q.persistState()`. The reindex ensures in-memory positions are correct for subsequent operations.
|
||||
|
||||
4. `InsertNextTracks` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks(newTracks, insertPos)` then `q.persistState()`.
|
||||
|
||||
5. `InsertTracksAt` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` first, then `q.persistInsertTracks(newTracks, index)` then `q.persistState()`.
|
||||
|
||||
6. `RemoveTrack` — Replace `q.commitMutation(true)` with: call `q.persistRemoveTrack(position)` then `q.reindexPositions()` then `q.persistState()`.
|
||||
|
||||
7. `RemoveTracks` — Replace `q.commitMutation(true)` with: call `q.reindexPositions()` then `q.persistTracks()` (full rewrite, per user decision for bulk ops) then `q.persistState()`.
|
||||
|
||||
**Keep `commitMutation` for**: `Clear`, `MoveQueueTracks`, `resolveRemainingTracks` — bulk operations that still do full rewrites per user decision.
|
||||
|
||||
**For the hand-crafted SQL in `persistInsertTracks`:** Use `tx.ExecContext(q.db.Ctx, "UPDATE queue_tracks SET position = position + ? WHERE position >= ?", count, insertPos)` with a `// SAFETY: Multi-row position shift by variable N unsupported by sqlc (shift queries only shift by 1). Bind variables match args; no string interpolation.` comment.
|
||||
|
||||
**Important:** Shuffle order regeneration was handled by `commitMutation`. For all the methods that previously called `commitMutation` with reindex=true, `generateShuffleOrder()` was also called if shuffleMode was active. Continue this behavior: after the incremental persist, check `q.shuffleMode` and call `q.generateShuffleOrder()` if true. For methods that called `commitMutation(false)` (AddTrack, AddTracks), shuffle order regeneration was also done if active — preserve this.
|
||||
|
||||
**Verification approach:** Existing persistence roundtrip tests in `persistence_test.go` exercise `SaveState`/`RestoreState` which uses `persistTracks` (full rewrite). The incremental paths are verified by: (1) the existing queue_test.go tests that call AddTrack/RemoveTrack/InsertNext etc. with a real DB, and (2) adding a focused test.
|
||||
</action>
|
||||
<verify>
|
||||
cd backend && go build ./... && go test ./queue/... -race -count=1
|
||||
</verify>
|
||||
<done>
|
||||
- AddTrack/AddTracks use persistAddTrack/persistAddTracks (no full table rewrite)
|
||||
- RemoveTrack uses persistRemoveTrack (single DELETE + position shift, no full table rewrite)
|
||||
- InsertNext/InsertNextTracks/InsertTracksAt use persistInsertTracks (position shift + INSERT, no full table rewrite)
|
||||
- RemoveTracks uses full persistTracks rewrite (acceptable for bulk operations)
|
||||
- MoveQueueTracks, Clear, SetQueue still use commitMutation/persistTracks (unchanged bulk behavior)
|
||||
- All existing tests pass with -race
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Eliminate redundant lookups in SetQueue Phase 2</name>
|
||||
<files>backend/queue/queue.go</files>
|
||||
<action>
|
||||
**Modify `resolveRemainingTracks` to accept and use Phase 1's already-resolved metadata:**
|
||||
|
||||
1. Change `resolveRemainingTracks` signature to accept the Phase 1 result map:
|
||||
```go
|
||||
func (q *Queue) resolveRemainingTracks(
|
||||
gen int64,
|
||||
filePaths []string,
|
||||
playingPath string,
|
||||
phase1Meta map[string]trackMeta, // NEW: already-resolved from Phase 1
|
||||
)
|
||||
```
|
||||
|
||||
2. Inside `resolveRemainingTracks`, build the exclusion set from `phase1Meta` keys. Filter `filePaths` to get only the paths NOT in `phase1Meta` before calling `lookupTrackMetaBatch`:
|
||||
```go
|
||||
// Exclude paths already resolved in Phase 1.
|
||||
var unresolvedPaths []string
|
||||
for _, fp := range filePaths {
|
||||
if _, alreadyResolved := phase1Meta[fp]; !alreadyResolved {
|
||||
unresolvedPaths = append(unresolvedPaths, fp)
|
||||
}
|
||||
}
|
||||
|
||||
// Only look up paths that Phase 1 didn't cover.
|
||||
remainingMeta := q.lookupTrackMetaBatch(unresolvedPaths)
|
||||
|
||||
// Merge Phase 1 results into the lookup.
|
||||
for k, v := range phase1Meta {
|
||||
remainingMeta[k] = v
|
||||
}
|
||||
```
|
||||
|
||||
3. The rest of the method (building tracks from `allMeta`, finding `playingPath`, calling `commitMutation`) uses `remainingMeta` instead of `allMeta`. Rename the variable for clarity.
|
||||
|
||||
4. **Update the call site in `SetQueue`:** Pass `batchMeta` (the Phase 1 result) to `resolveRemainingTracks`:
|
||||
```go
|
||||
go q.resolveRemainingTracks(gen, filePaths, playingPath, batchMeta)
|
||||
```
|
||||
|
||||
**Keep `initialBatchSize` at 50** — no changes to the Phase 1 window size (per user decision).
|
||||
|
||||
**Result:** For a 1000-track SetQueue where Phase 1 resolves 50, Phase 2 now queries only 950 paths instead of all 1000. The 50 already-resolved paths are merged from the Phase 1 map.
|
||||
</action>
|
||||
<verify>
|
||||
cd backend && go build ./... && go test ./queue/... -race -count=1
|
||||
</verify>
|
||||
<done>
|
||||
- resolveRemainingTracks accepts phase1Meta parameter
|
||||
- Phase 2 filters out already-resolved paths before calling lookupTrackMetaBatch
|
||||
- Phase 1 results are merged into Phase 2 results
|
||||
- SetQueue call site passes batchMeta to resolveRemainingTracks
|
||||
- initialBatchSize remains at 50
|
||||
- All existing tests pass with -race
|
||||
</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
```bash
|
||||
# All queue tests pass with race detector
|
||||
cd backend && go test ./queue/... -race -count=1 -v
|
||||
|
||||
# Build succeeds
|
||||
cd backend && go build ./...
|
||||
|
||||
# Lint passes
|
||||
make lint
|
||||
```
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Single-track add/remove uses incremental INSERT/DELETE (not full table rewrite)
|
||||
- Insert-at-position uses position shift + INSERT (not full table rewrite)
|
||||
- SetQueue Phase 2 only queries unreolved paths (not all paths)
|
||||
- All existing queue tests pass with -race
|
||||
- No linting errors
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/07-backend-performance/07-01-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user