---
phase: 11-per-library-scan-pipeline
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- backend/library/scan_queue.go
- backend/library/library.go
- backend/library/scan_control.go
- backend/library/config.go
- backend/library/rescan.go
- backend/library/metrics.go
- backend/events/events.go
- frontend/src/events.ts
- backend/database/sql/queries/audio_files.sql
- backend/database/sql/sqlcgen/audio_files.sql.go
- backend/database/sql/sqlcgen/models.go
autonomous: true
requirements: [LSCAN-01, LSCAN-02, LSCAN-04]
must_haves:
truths:
- "ScanLibrary(id) scans only the directory associated with that library ID"
- "Only one library scans at a time — additional requests are silently queued"
- "Duplicate scan requests for the same library are silently ignored"
- "Cancel/pause/resume work per-library — cancelling one library starts the next queued"
- "Pausing freezes both the current scan AND the queue"
- "ScanAllLibraries queries all libraries and queues them sequentially"
artifacts:
- path: "backend/library/scan_queue.go"
provides: "Scan queue coordinator with sequential execution"
exports: ["ScanLibrary", "ScanAllLibraries", "CancelCurrentScan", "CancelAllScans"]
- path: "backend/library/library.go"
provides: "Updated Scan() accepting library ID and path"
- path: "backend/events/events.go"
provides: "Updated scan events with library identification"
- path: "backend/database/sql/queries/audio_files.sql"
provides: "CreateAudioFile with library_id parameter"
key_links:
- from: "backend/library/scan_queue.go"
to: "backend/library/library.go"
via: "scanQueue calls scanLibrary which calls internal scan pipeline"
pattern: "l\\.scanInternal"
- from: "backend/library/scan_queue.go"
to: "backend/database/sql/sqlcgen/libraries.sql.go"
via: "GetLibrary query to resolve library path from ID"
pattern: "Queries\\.GetLibrary"
- from: "backend/library/library.go"
to: "backend/database/sql/sqlcgen/audio_files.sql.go"
via: "CreateAudioFile now includes library_id"
pattern: "CreateAudioFileParams.*LibraryID"
---
Refactor the scan pipeline from scanning a single hardcoded directory to scanning individual libraries by database ID, with a sequential scan queue coordinator.
Purpose: Enable per-library scanning (LSCAN-01), sequential coordination (LSCAN-02), and per-library cancel/pause scope (LSCAN-04) at the backend level.
Output: `ScanLibrary(id)` and `ScanAllLibraries()` Wails-bound methods, scan queue coordinator, updated events with library identification, `CreateAudioFile` with `library_id`.
@/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md
@/home/caleb/.config/opencode/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/11-per-library-scan-pipeline/11-CONTEXT.md
@.planning/phases/09-scan-cancellation-keyboard-shortcuts/09-01-SUMMARY.md
@.planning/phases/10-schema-migration/10-02-SUMMARY.md
From backend/library/library.go:
```go
type Library struct {
mu sync.Mutex
ctx context.Context
logger *slog.Logger
conf *Config
db *database.DB
rescanHooks RescanHooks
scanActive bool
scanCancel context.CancelFunc
scanPaused bool
scanPauseCh chan struct{}
}
func (l *Library) Scan() (*ScanMetrics, error)
func (l *Library) SetContext(ctx context.Context)
func (l *Library) CancelScan()
func (l *Library) PauseScan()
func (l *Library) ResumeScan()
func (l *Library) IsScanActive() bool
func (l *Library) IsScanPaused() bool
```
From backend/library/config.go:
```go
type Config struct {
DirectoryPath Directory `toml:"DirectoryPath"`
ScanConcurrency ScanConcurrency `toml:"ScanConcurrency"`
}
```
From backend/library/metrics.go:
```go
type ScanProgress struct {
Phase string `json:"phase"`
Total int64 `json:"total"`
Processed int64 `json:"processed"`
Added int64 `json:"added"`
Skipped int64 `json:"skipped"`
Updated int64 `json:"updated"`
}
type ScanMetrics struct { ... Cancelled bool ... }
```
From backend/events/events.go:
```go
const (
LibraryScanStarted = "LibraryScanStarted"
LibraryScanProgress = "LibraryScanProgress"
LibraryScanComplete = "LibraryScanComplete"
LibraryScanCancelled = "LibraryScanCancelled"
LibraryScanPaused = "LibraryScanPaused"
LibraryScanResumed = "LibraryScanResumed"
)
```
From backend/database/sql/sqlcgen/libraries.sql.go:
```go
func (q *Queries) GetLibrary(ctx context.Context, id int64) (Library, error)
func (q *Queries) GetAllLibraries(ctx context.Context) ([]Library, error)
```
From backend/database/sql/sqlcgen/audio_files.sql.go:
```go
type CreateAudioFileParams struct {
FilePath string
LengthMilliseconds int64
FileTypeID int64
RecordingID int64
SampleRate int64
BitDepth int64
Channels int64
Bitrate int64
FileSize int64
Basename string
// NOTE: library_id NOT included — uses DEFAULT 0
}
func (q *Queries) GetAudioFilesByLibrary(ctx context.Context, libraryID int64) ([]AudioFile, error)
func (q *Queries) GetAllAudioFiles(ctx context.Context) ([]AudioFile, error)
```
Task 1: Add library_id to CreateAudioFile + update events and progress types
backend/database/sql/queries/audio_files.sql
backend/database/sql/sqlcgen/audio_files.sql.go
backend/database/sql/sqlcgen/models.go
backend/events/events.go
frontend/src/events.ts
backend/library/metrics.go
1. **Update CreateAudioFile SQL query** in `backend/database/sql/queries/audio_files.sql`:
- Add `library_id` to the INSERT column list and VALUES: `INSERT INTO audio_files (file_path, length_milliseconds, file_type_id, recording_id, sample_rate, bit_depth, channels, bitrate, file_size, basename, library_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
- This adds the `library_id` parameter so scans can associate files with their library.
2. **Run `sqlc generate`** to regenerate Go code:
```bash
sqlc generate
```
This will update `CreateAudioFileParams` to include `LibraryID int64`.
3. **Add new event constants** to `backend/events/events.go` — add a "Scan queue events" group:
```go
// Scan queue events.
const (
LibraryScanQueued = "LibraryScanQueued"
LibraryScanQueueDrained = "LibraryScanQueueDrained"
)
```
4. **Regenerate TypeScript events** via `go generate ./backend/events/...` (uses the genevents tool).
5. **Add library identification fields** to `ScanProgress` and `ScanMetrics` in `backend/library/metrics.go`:
- Add to `ScanProgress`: `LibraryID int64 \`json:"libraryId"\`` and `LibraryName string \`json:"libraryName"\``
- Add to `ScanProgress`: `QueuedCount int \`json:"queuedCount"\`` (number of libraries still queued after this one)
- Add to `ScanMetrics`: `LibraryID int64 \`json:"libraryId"\`` and `LibraryName string \`json:"libraryName"\``
6. **Fix compilation** — update the `CreateAudioFile` call in `library.go` `saveAudioFile()` method to include `LibraryID` field. The library ID will be threaded through as a parameter to `Scan`/`scanInternal` (done in Task 2), so for now add the field but use a placeholder `0` value that Task 2 will replace. Actually — since Task 2 immediately follows and both are in the same plan, add `libraryID int64` as a field on the `Library` struct (or better: pass it through the scan methods). For the compilation fix, add `LibraryID: 0` to the CreateAudioFileParams in saveAudioFile — Task 2 will thread the real value.
Verify the generated code compiles: `go build ./backend/...`
cd /mnt/vault/dev/golang/yellowjacket && sqlc generate && go generate ./backend/events/... && go build ./backend/...
CreateAudioFileParams includes LibraryID field. ScanProgress and ScanMetrics include library identification fields. New scan queue events exist in both Go and TypeScript. Code compiles.
Task 2: Create scan queue coordinator and refactor Library for per-library scanning
backend/library/scan_queue.go
backend/library/library.go
backend/library/scan_control.go
backend/library/config.go
backend/library/rescan.go
**Create `backend/library/scan_queue.go`** — the scan queue coordinator. This is the core of Phase 11.
Design:
- The `Library` struct gains scan queue fields (protected by `mu`):
- `scanQueue []scanQueueEntry` — FIFO queue of library IDs to scan
- `currentScanLibraryID int64` — the library currently being scanned (0 if none)
- `currentScanLibraryName string` — for event payloads
- `scanQueueEntry` struct: `libraryID int64`, `libraryName string`, `libraryPath string`
**Wails-bound methods** (exported, on `*Library`):
1. `ScanLibrary(id int64) error`:
- Query `l.db.Queries.GetLibrary(l.ctx, id)` to get library name and path
- If library not found, return error
- Acquire `l.mu`:
- If this library ID is already `currentScanLibraryID` or already in `scanQueue`, return nil (silent dedup per CONTEXT.md)
- If no scan is active (`!l.scanActive`), set `currentScanLibraryID = id` and start scanning in a goroutine
- If a scan is active, append to `scanQueue` and emit `LibraryScanQueued` event with library name and queue length
- Release `l.mu`
- Return nil
2. `ScanAllLibraries() error`:
- Query `l.db.Queries.GetAllLibraries(l.ctx)` to get all libraries
- For each library, call `ScanLibrary(lib.ID)` (reuses dedup logic)
- Return nil
3. `CancelCurrentScan()` — cancels only the current library's scan (replaces old `CancelScan`):
- Cancel the scan context (existing `l.scanCancel()` call)
- The scan completion handler (`drainQueue`) will automatically start the next queued library
4. `CancelAllScans()` — cancels current and clears queue:
- Acquire `l.mu`, clear `l.scanQueue`, release `l.mu`
- Then cancel the current scan context
5. `GetScanQueueLength() int` — returns length of scan queue (for UI)
**Internal scan orchestration:**
- `startScan(entry scanQueueEntry)` — goroutine entry point:
- Calls `l.scanInternal(entry.libraryID, entry.libraryName, entry.libraryPath)`
- On completion, calls `l.drainQueue()`
- `drainQueue()` — called after each scan completes:
- Acquire `l.mu`
- If `scanQueue` is not empty, pop first entry, set as `currentScanLibraryID`, release lock, call `startScan` in new goroutine
- If `scanQueue` is empty, set `currentScanLibraryID = 0`, `scanActive = false`, emit `LibraryScanQueueDrained`, release lock
**Refactor `Library.Scan()` → `scanInternal()`:**
- Rename current `Scan()` to `scanInternal(libraryID int64, libraryName string, libraryPath string)` (unexported)
- Remove the `l.conf.DirectoryPath` dependency — use the `libraryPath` parameter instead
- Replace `l.db.Queries.GetAllAudioFiles(l.ctx)` with `l.db.Queries.GetAudioFilesByLibrary(l.ctx, libraryID)` in Phase 1 (load existing)
- Pass `libraryID` through to `saveAudioFile` so `CreateAudioFileParams.LibraryID` is set correctly
- Update all `ScanProgress` emissions to include `LibraryID`, `LibraryName`, and `QueuedCount` (read queue length under lock)
- Update `ScanMetrics` to include `LibraryID` and `LibraryName` before emitting `LibraryScanComplete`/`LibraryScanCancelled`
- The `workerCount` should use `resolveScanWorkerCount(ScanConcurrencyAuto, libraryPath)` — no longer from config (each library path may be on different storage)
**Keep backward-compatible `Scan()` method** — public method that scans using the legacy `l.conf.DirectoryPath` for `handleConfigUpdate`. Mark it deprecated. It should:
- Look up or create a library for `l.conf.DirectoryPath` using `GetLibraryByPath`
- Call `ScanLibrary(lib.ID)`
**Update `scan_control.go`:**
- Rename `CancelScan()` to an internal helper `cancelCurrentScan()` (unexported)
- Keep `PauseScan()` and `ResumeScan()` as-is — they operate on the current scan which is correct
- `IsScanActive()` unchanged
- Add `QueuedLibraryNames() []string` — returns names of queued libraries (for UI display)
**Update `config.go`:**
- The `Config` struct keeps `DirectoryPath` and `ScanConcurrency` for backward compatibility, but `DirectoryPath` is now unused for normal scanning (libraries come from DB). `ScanConcurrency` is still useful as a global default.
**Update `rescan.go`:**
- `FullRescan()` needs updating — it should accept a library ID. For now, keep it working with `l.conf.DirectoryPath` (it's used from the config page). Phase 12 will add per-library rescan.
**Thread `libraryID` through the scan pipeline:**
- Add `libraryID int64` field to `scanWork` struct (or pass it via closure)
- In `saveAudioFile`, use `LibraryID: libraryID` in `CreateAudioFileParams`
- In the `commitBatch` → `saveAudioFile` call chain, thread the library ID through. Simplest: add `libraryID int64` as a parameter to `commitBatch` and `saveAudioFile` and `updateAudioFileMetadata`.
**Linting notes:**
- All exported methods need doc comments ending with period (godot)
- No stuttering (revive) — method names don't repeat "Library"
- Sentinel errors as package vars (err113)
- Blank line after early returns (nlreturn)
- Keep lines under 100 chars (golines)
cd /mnt/vault/dev/golang/yellowjacket && go build ./... && go vet ./backend/library/...
- `ScanLibrary(id)` scans a specific library's directory, associating files with that library_id
- `ScanAllLibraries()` queues all libraries for sequential scanning
- Scan queue coordinator ensures only one scan runs at a time, with silent dedup
- Cancel: `CancelCurrentScan()` cancels current and starts next; `CancelAllScans()` cancels current and clears queue
- Pause freezes current scan AND queue (existing behavior — drainQueue is only called on scan completion, which doesn't happen while paused)
- All scan events include library name and queue count
- `go build ./...` passes
```bash
# Build passes
go build ./...
# Vet passes
go vet ./backend/library/...
# Generated code is up to date
sqlc generate && go generate ./backend/events/...
# Existing tests still pass (scan_test.go uses the old Scan() path)
go test ./backend/library/... -count=1 -timeout 60s
# Events synced
diff <(grep -oP '"[A-Z][a-zA-Z]+"' backend/events/events.go | sort) <(grep -oP '"[A-Z][a-zA-Z]+"' frontend/src/events.ts | sort)
```
- ScanLibrary(id) resolves library path from DB and scans only that directory
- CreateAudioFile includes library_id — new files are associated with their library
- Only one scan runs at a time — queue coordinates sequential execution
- Duplicate requests are silently ignored
- CancelCurrentScan stops current library, next queued starts automatically
- CancelAllScans stops current and clears queue
- Pause freezes scan AND queue
- All scan events include library name and queue count
- go build ./... passes, go test ./backend/library/... passes