yellowjacket/.planning/phases/11-per-library-scan-pipeline/11-01-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
11-per-library-scan-pipeline	01	execute	1		backend/library/scan_queue.go backend/library/library.go backend/library/scan_control.go backend/library/config.go backend/library/rescan.go backend/library/metrics.go backend/events/events.go frontend/src/events.ts backend/database/sql/queries/audio_files.sql backend/database/sql/sqlcgen/audio_files.sql.go backend/database/sql/sqlcgen/models.go	true	LSCAN-01 LSCAN-02 LSCAN-04	truths artifacts key_links ScanLibrary(id) scans only the directory associated with that library ID Only one library scans at a time — additional requests are silently queued Duplicate scan requests for the same library are silently ignored Cancel/pause/resume work per-library — cancelling one library starts the next queued Pausing freezes both the current scan AND the queue ScanAllLibraries queries all libraries and queues them sequentially path provides exports backend/library/scan_queue.go Scan queue coordinator with sequential execution ScanLibrary ScanAllLibraries CancelCurrentScan CancelAllScans path provides backend/library/library.go Updated Scan() accepting library ID and path path provides backend/events/events.go Updated scan events with library identification path provides backend/database/sql/queries/audio_files.sql CreateAudioFile with library_id parameter from to via pattern backend/library/scan_queue.go backend/library/library.go scanQueue calls scanLibrary which calls internal scan pipeline l.scanInternal from to via pattern backend/library/scan_queue.go backend/database/sql/sqlcgen/libraries.sql.go GetLibrary query to resolve library path from ID Queries.GetLibrary from to via pattern backend/library/library.go backend/database/sql/sqlcgen/audio_files.sql.go CreateAudioFile now includes library_id CreateAudioFileParams.*LibraryID

Refactor the scan pipeline from scanning a single hardcoded directory to scanning individual libraries by database ID, with a sequential scan queue coordinator.

Purpose: Enable per-library scanning (LSCAN-01), sequential coordination (LSCAN-02), and per-library cancel/pause scope (LSCAN-04) at the backend level. Output: ScanLibrary(id) and ScanAllLibraries() Wails-bound methods, scan queue coordinator, updated events with library identification, CreateAudioFile with library_id.

<execution_context> @/home/caleb/.config/opencode/get-shit-done/workflows/execute-plan.md @/home/caleb/.config/opencode/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/11-per-library-scan-pipeline/11-CONTEXT.md @.planning/phases/09-scan-cancellation-keyboard-shortcuts/09-01-SUMMARY.md @.planning/phases/10-schema-migration/10-02-SUMMARY.md

From backend/library/library.go:

type Library struct {
    mu          sync.Mutex
    ctx         context.Context
    logger      *slog.Logger
    conf        *Config
    db          *database.DB
    rescanHooks RescanHooks
    scanActive  bool
    scanCancel  context.CancelFunc
    scanPaused  bool
    scanPauseCh chan struct{}
}

func (l *Library) Scan() (*ScanMetrics, error)
func (l *Library) SetContext(ctx context.Context)
func (l *Library) CancelScan()
func (l *Library) PauseScan()
func (l *Library) ResumeScan()
func (l *Library) IsScanActive() bool
func (l *Library) IsScanPaused() bool

From backend/library/config.go:

type Config struct {
    DirectoryPath   Directory       `toml:"DirectoryPath"`
    ScanConcurrency ScanConcurrency `toml:"ScanConcurrency"`
}

From backend/library/metrics.go:

type ScanProgress struct {
    Phase     string `json:"phase"`
    Total     int64  `json:"total"`
    Processed int64  `json:"processed"`
    Added     int64  `json:"added"`
    Skipped   int64  `json:"skipped"`
    Updated   int64  `json:"updated"`
}

type ScanMetrics struct { ... Cancelled bool ... }

From backend/events/events.go:

const (
    LibraryScanStarted   = "LibraryScanStarted"
    LibraryScanProgress  = "LibraryScanProgress"
    LibraryScanComplete  = "LibraryScanComplete"
    LibraryScanCancelled = "LibraryScanCancelled"
    LibraryScanPaused    = "LibraryScanPaused"
    LibraryScanResumed   = "LibraryScanResumed"
)

From backend/database/sql/sqlcgen/libraries.sql.go:

func (q *Queries) GetLibrary(ctx context.Context, id int64) (Library, error)
func (q *Queries) GetAllLibraries(ctx context.Context) ([]Library, error)

From backend/database/sql/sqlcgen/audio_files.sql.go:

type CreateAudioFileParams struct {
    FilePath           string
    LengthMilliseconds int64
    FileTypeID         int64
    RecordingID        int64
    SampleRate         int64
    BitDepth           int64
    Channels           int64
    Bitrate            int64
    FileSize           int64
    Basename           string
    // NOTE: library_id NOT included — uses DEFAULT 0
}

func (q *Queries) GetAudioFilesByLibrary(ctx context.Context, libraryID int64) ([]AudioFile, error)
func (q *Queries) GetAllAudioFiles(ctx context.Context) ([]AudioFile, error)

Task 1: Add library_id to CreateAudioFile + update events and progress types backend/database/sql/queries/audio_files.sql backend/database/sql/sqlcgen/audio_files.sql.go backend/database/sql/sqlcgen/models.go backend/events/events.go frontend/src/events.ts backend/library/metrics.go 1. **Update CreateAudioFile SQL query** in `backend/database/sql/queries/audio_files.sql`: - Add `library_id` to the INSERT column list and VALUES: `INSERT INTO audio_files (file_path, length_milliseconds, file_type_id, recording_id, sample_rate, bit_depth, channels, bitrate, file_size, basename, library_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)` - This adds the `library_id` parameter so scans can associate files with their library.

2. Run sqlc generate to regenerate Go code:

   sqlc generate
   

   This will update CreateAudioFileParams to include LibraryID int64.

3. Add new event constants to backend/events/events.go — add a "Scan queue events" group:

   // Scan queue events.
   const (
       LibraryScanQueued       = "LibraryScanQueued"
       LibraryScanQueueDrained = "LibraryScanQueueDrained"
   )
   

4. Regenerate TypeScript events via go generate ./backend/events/... (uses the genevents tool).

5. Add library identification fields to ScanProgress and ScanMetrics in backend/library/metrics.go:

   • Add to ScanProgress: LibraryID int64 \json:"libraryId"`andLibraryName string `json:"libraryName"``
   • Add to ScanProgress: QueuedCount int \json:"queuedCount"`` (number of libraries still queued after this one)
   • Add to ScanMetrics: LibraryID int64 \json:"libraryId"`andLibraryName string `json:"libraryName"``

6. Fix compilation — update the CreateAudioFile call in library.go saveAudioFile() method to include LibraryID field. The library ID will be threaded through as a parameter to Scan/scanInternal (done in Task 2), so for now add the field but use a placeholder 0 value that Task 2 will replace. Actually — since Task 2 immediately follows and both are in the same plan, add libraryID int64 as a field on the Library struct (or better: pass it through the scan methods). For the compilation fix, add LibraryID: 0 to the CreateAudioFileParams in saveAudioFile — Task 2 will thread the real value.

Verify the generated code compiles: go build ./backend/... cd /mnt/vault/dev/golang/yellowjacket && sqlc generate && go generate ./backend/events/... && go build ./backend/... CreateAudioFileParams includes LibraryID field. ScanProgress and ScanMetrics include library identification fields. New scan queue events exist in both Go and TypeScript. Code compiles.

Task 2: Create scan queue coordinator and refactor Library for per-library scanning backend/library/scan_queue.go backend/library/library.go backend/library/scan_control.go backend/library/config.go backend/library/rescan.go **Create `backend/library/scan_queue.go`** — the scan queue coordinator. This is the core of Phase 11.

Design:

• The Library struct gains scan queue fields (protected by mu):
  • scanQueue []scanQueueEntry — FIFO queue of library IDs to scan
  • currentScanLibraryID int64 — the library currently being scanned (0 if none)
  • currentScanLibraryName string — for event payloads
• scanQueueEntry struct: libraryID int64, libraryName string, libraryPath string

Wails-bound methods (exported, on *Library):

1. ScanLibrary(id int64) error:

   • Query l.db.Queries.GetLibrary(l.ctx, id) to get library name and path
   • If library not found, return error
   • Acquire l.mu:
     • If this library ID is already currentScanLibraryID or already in scanQueue, return nil (silent dedup per CONTEXT.md)
     • If no scan is active (!l.scanActive), set currentScanLibraryID = id and start scanning in a goroutine
     • If a scan is active, append to scanQueue and emit LibraryScanQueued event with library name and queue length
   • Release l.mu
   • Return nil

2. ScanAllLibraries() error:

   • Query l.db.Queries.GetAllLibraries(l.ctx) to get all libraries
   • For each library, call ScanLibrary(lib.ID) (reuses dedup logic)
   • Return nil

3. CancelCurrentScan() — cancels only the current library's scan (replaces old CancelScan):

   • Cancel the scan context (existing l.scanCancel() call)
   • The scan completion handler (drainQueue) will automatically start the next queued library

4. CancelAllScans() — cancels current and clears queue:

   • Acquire l.mu, clear l.scanQueue, release l.mu
   • Then cancel the current scan context

5. GetScanQueueLength() int — returns length of scan queue (for UI)

Internal scan orchestration:

• startScan(entry scanQueueEntry) — goroutine entry point:

  • Calls l.scanInternal(entry.libraryID, entry.libraryName, entry.libraryPath)
  • On completion, calls l.drainQueue()

• drainQueue() — called after each scan completes:

  • Acquire l.mu
  • If scanQueue is not empty, pop first entry, set as currentScanLibraryID, release lock, call startScan in new goroutine
  • If scanQueue is empty, set currentScanLibraryID = 0, scanActive = false, emit LibraryScanQueueDrained, release lock

Refactor Library.Scan() → scanInternal():

• Rename current Scan() to scanInternal(libraryID int64, libraryName string, libraryPath string) (unexported)
• Remove the l.conf.DirectoryPath dependency — use the libraryPath parameter instead
• Replace l.db.Queries.GetAllAudioFiles(l.ctx) with l.db.Queries.GetAudioFilesByLibrary(l.ctx, libraryID) in Phase 1 (load existing)
• Pass libraryID through to saveAudioFile so CreateAudioFileParams.LibraryID is set correctly
• Update all ScanProgress emissions to include LibraryID, LibraryName, and QueuedCount (read queue length under lock)
• Update ScanMetrics to include LibraryID and LibraryName before emitting LibraryScanComplete/LibraryScanCancelled
• The workerCount should use resolveScanWorkerCount(ScanConcurrencyAuto, libraryPath) — no longer from config (each library path may be on different storage)

Keep backward-compatible Scan() method — public method that scans using the legacy l.conf.DirectoryPath for handleConfigUpdate. Mark it deprecated. It should:

• Look up or create a library for l.conf.DirectoryPath using GetLibraryByPath
• Call ScanLibrary(lib.ID)

Update scan_control.go:

• Rename CancelScan() to an internal helper cancelCurrentScan() (unexported)
• Keep PauseScan() and ResumeScan() as-is — they operate on the current scan which is correct
• IsScanActive() unchanged
• Add QueuedLibraryNames() []string — returns names of queued libraries (for UI display)

Update config.go:

• The Config struct keeps DirectoryPath and ScanConcurrency for backward compatibility, but DirectoryPath is now unused for normal scanning (libraries come from DB). ScanConcurrency is still useful as a global default.

Update rescan.go:

• FullRescan() needs updating — it should accept a library ID. For now, keep it working with l.conf.DirectoryPath (it's used from the config page). Phase 12 will add per-library rescan.

Thread libraryID through the scan pipeline:

• Add libraryID int64 field to scanWork struct (or pass it via closure)
• In saveAudioFile, use LibraryID: libraryID in CreateAudioFileParams
• In the commitBatch → saveAudioFile call chain, thread the library ID through. Simplest: add libraryID int64 as a parameter to commitBatch and saveAudioFile and updateAudioFileMetadata.

Linting notes:

• All exported methods need doc comments ending with period (godot)
• No stuttering (revive) — method names don't repeat "Library"
• Sentinel errors as package vars (err113)
• Blank line after early returns (nlreturn)
• Keep lines under 100 chars (golines) cd /mnt/vault/dev/golang/yellowjacket && go build ./... && go vet ./backend/library/...
• ScanLibrary(id) scans a specific library's directory, associating files with that library_id
• ScanAllLibraries() queues all libraries for sequential scanning
• Scan queue coordinator ensures only one scan runs at a time, with silent dedup
• Cancel: CancelCurrentScan() cancels current and starts next; CancelAllScans() cancels current and clears queue
• Pause freezes current scan AND queue (existing behavior — drainQueue is only called on scan completion, which doesn't happen while paused)
• All scan events include library name and queue count
• go build ./... passes

```bash # Build passes go build ./...

Vet passes

go vet ./backend/library/...

Generated code is up to date

sqlc generate && go generate ./backend/events/...

Existing tests still pass (scan_test.go uses the old Scan() path)

go test ./backend/library/... -count=1 -timeout 60s

Events synced

diff <(grep -oP '"[A-Z][a-zA-Z]+"' backend/events/events.go | sort) <(grep -oP '"[A-Z][a-zA-Z]+"' frontend/src/events.ts | sort)

</verification>

<success_criteria>
- ScanLibrary(id) resolves library path from DB and scans only that directory
- CreateAudioFile includes library_id — new files are associated with their library
- Only one scan runs at a time — queue coordinates sequential execution
- Duplicate requests are silently ignored
- CancelCurrentScan stops current library, next queued starts automatically
- CancelAllScans stops current and clears queue
- Pause freezes scan AND queue
- All scan events include library name and queue count
- go build ./... passes, go test ./backend/library/... passes
</success_criteria>

<output>
After completion, create `.planning/phases/11-per-library-scan-pipeline/11-01-SUMMARY.md`
</output>