yellowjacket/.planning/RETROSPECTIVE.md

Project Retrospective

A living document updated after each milestone. Lessons feed forward into future planning.

Milestone: v1.0 — Consolidation

Shipped: 2026-03-05 Phases: 8 | Plans: 17 | Tasks: 34 Timeline: 6 days (2026-02-27 → 2026-03-05)

What Was Built

• Race-free concurrency across all 4 SetContext entry points
• Honest error handling: startupErr to struct, config permissions, MPRIS logging, scan warning separation
• 84 unit tests covering queue, config, player, FTS5 search, library scan, entity cache
• SQL consolidation: track_metadata VIEW, sqlc.slice() migration, SAFETY comments on 12 hand-crafted queries
• AST-based Go→TypeScript event codegen with pre-commit enforcement
• Incremental queue persistence (O(1) add/remove) and SetQueue Phase 2 dedup
• Deferred library store loading for instant app shell
• Frontend design token system, classMap directives, queueMicrotask coalescing
• Visual consistency audit across all 15 components

What Worked

• Dependency-ordered phases: Fixing races → building test infra → writing tests → refactoring → performance → UX created a clean progression where each phase built on the last
• Characterization tests before refactoring: Writing tests in Phase 4-5 before SQL consolidation in Phase 6 caught zero regressions — the tests were accurate safety nets
• Small, focused plans: 2-3 tasks per plan kept execution fast and context fresh — most plans completed in under 10 minutes
• Research phase for SQL consolidation: Phase 6 research validated sqlc + VIEW + FTS5 compatibility before planning, avoiding mid-execution discovery
• Internal package tests: Testing queue/library as package-internal (not _test suffix) gave access to unexported fields for thorough state verification

What Was Inefficient

• Phase 8 repeat() regression: Migrating virtualizers to repeat() directive in Plan 02 broke virtualization (repeat as child content bypasses lit-virtualizer's DOM management). Required a hotfix (72ef719) reverting to .renderItem + .keyFunction. Research should have caught this API distinction.
• Task count tracking: STATE.md only tracked tasks-per-plan for later phases (5-8), making total task count harder to derive at milestone completion
• No startup time measurement: TODO to measure startup time before Phase 7 lazy loading was never done — can't quantify the improvement

Patterns Established

• Mutex-protected setter pattern: Lock → write field → release lock → call callbacks (prevents deadlock from callback re-entry)
• ScanWarning + addWarning pattern: Mutex-protected warning collection for non-fatal errors during long-running operations
• applyPRAGMAs shared function: Single source of truth for SQLite PRAGMAs, shared between production NewDB and test NewTestDB
• SAFETY comment convention: Two-part format (why + safety assurance) for hand-crafted SQL that bypasses sqlc
• AST-based codegen over regex: go/ast + go/parser for cross-language constant synchronization
• Design token CSS custom properties: --yj-icon-sm/md/lg, --yj-text-xs/sm/md/lg/xl scoped to :host in Lit components
• queueMicrotask coalescing: Batch multiple synchronous store notifications into single subscriber update

Key Lessons

1. Test the API contract, not the implementation surface: repeat() inside lit-virtualizer looks correct syntactically but violates the component's rendering contract. Always verify how a library expects to be consumed, not just what compiles.
2. Research before planning pays off immediately: Phase 6 research confirmed sqlc + VIEW compatibility, saving mid-execution discovery and potential re-planning.
3. Incremental persistence is O(complexity) not O(code): The incremental queue persistence (Phase 7) was conceptually simple but required careful position-shift SQL for insert/remove operations — more thought than code.
4. Design tokens must precede visual consistency work: Phase 8 correctly defined tokens in Plan 01 before applying them in Plan 04 — reversing this order would have required double work.
5. Contentless FTS5 has deletion limitations: Cannot DELETE from tables with content=''. Document this in tests rather than fighting it — stale entries are harmless for the use case.

Cost Observations

• Model mix: Primarily opus for planning + execution, sonnet for research
• Total commits: 107 across 6 days
• Notable: Plans averaging 2-6 minutes execution time; Phase 8 Plan 04 (visual audit across 15 components) was the longest at 8 minutes
• Efficiency: 17 plans × ~5 min avg = ~85 min total execution time for 34 tasks across 67 source files

---

Milestone: v1.1 — Multi-Library Support

Shipped: 2026-03-16 Phases: 6 | Plans: 18 Timeline: 10 days (2026-03-06 → 2026-03-16)

What Was Built

• Cancellable/pausable library scans with per-scan context cancellation and sequential queue coordination
• Configurable keyboard shortcuts with record-style capture UI, scope-aware dispatch, and conflict detection
• Multi-library database schema (migration 6) with seamless single-directory migration
• Per-library scan pipeline with scan queue, per-library progress UI, and cancel scope
• Full library CRUD API with 17-step atomic removal (orphan cleanup, phantom metadata, FTS5, cover art, queue compaction)
• Library filter dropdown — all views (tracks, albums, artists, genres, search) respect active filter
• Cross-library playlists with phantom track auto-resolution via ScanHooks + M3U8 path matching
• Performance: CSS containment, view caching, event delegation, content-visibility, scroll polish

What Worked

• 4-phase multi-library progression (schema → scan → CRUD → views): Each phase had clear boundaries and verifiable outputs. Schema first meant scan pipeline had stable types; scan pipeline meant CRUD had working add-then-scan; CRUD meant views could demonstrate the full lifecycle.
• Locked decisions from /gsd-discuss-phase: "Backend filtering, not frontend" and "SET NULL for playlist_tracks FK" were decided once and never revisited — eliminated mid-execution design debates.
• Performance phase running in parallel: Phase 14 (performance) was independent of the multi-library phases (10-13), allowing it to execute when multi-library phases were blocked on human verification.
• Checkpoint-driven bugfinding: The human-verify checkpoint in Phase 13 found 3 bugs (virtualizer event delegation race, missing phantom auto-resolution, M3U8-based resolution needed) that wouldn't have been caught by automated verification alone.
• Hook patterns for cross-package communication: ScanHooks, RemovalHooks, and RescanHooks cleanly broke circular dependencies between library, playlist, and queue packages without coupling.

What Was Inefficient

• Phantom resolution required 3 iterations: First attempt (pure SQL with phantom_file_path) missed pre-existing phantoms. Second attempt (backfill) was fragile. Third attempt (M3U8-based ScanHooks) was the right approach from the start. Should have analyzed the M3U8 data flow before designing the resolution.
• Phase 14 virtualizer bug surfaced late: The event delegation race condition from Phase 14-03 wasn't caught until Phase 13's checkpoint. The Phase 14 verification should have included testing with empty-then-loaded data states.
• Quick task 19 (phantom path resolution) overlapped with Phase 13: The fix for multi-root path resolution in playlists was done as a quick task but directly related to Phase 13's phantom track work. Could have been folded into Phase 13 planning.

Patterns Established

• ScanHooks callback pattern: Post-scan processing without circular imports — library calls hook, playlist implements
• ByLibrary query variants: Parallel filtered/unfiltered sqlc queries with conditional dispatch in store layer
• phantom_file_path column: Preserves original file path at removal time for future re-linking
• M3U8 as source of truth for phantom matching: Position-based + path-based dual matching strategy
• View caching with display:none toggle: Keeps DOM alive for instant navigation, bounded cache (6 entries)
• Event delegation via data-index + closest(): Zero per-item closures in virtualizer renderItem functions
• attachDelegation guard pattern: Retry event delegation in updated() for conditionally-rendered elements
• changeGeneration counter: Simple monotonic counter replaces typed subscription system for store change detection

Key Lessons

1. Analyze data flow before designing resolution strategies: The phantom track resolution should have started with "what data do we have?" (M3U8 files have the paths) rather than "where can we store new data?" (phantom_file_path column). The M3U8 approach was simpler and more robust.
2. Human checkpoints catch integration bugs that automated tests miss: The virtualizer race condition and phantom auto-resolution gap were both found during manual testing, not by build/lint/verify. Budget for checkpoint time.
3. Hook patterns scale well for cross-cutting concerns: ScanHooks, RemovalHooks, and RescanHooks all follow the same pattern — define struct with function fields, set via method, call at lifecycle points. This pattern can be reused for future cross-package coordination.
4. Conditional rendering + lifecycle hooks need careful testing: Components that conditionally render children (lit-virtualizer appears only when data loads) must handle the case where firstUpdated fires before the child exists. Test with both fast and slow data loading.
5. Performance optimization and feature work can truly run in parallel: Phase 14 had zero file conflicts with Phases 10-13 and was executed out of order. Independent subsystem identification at planning time enables this parallelism.

Cost Observations

• Model mix: Primarily opus for planning + execution, sonnet for verification
• Total commits: ~85 across 10 days
• Notable: Most plans completed in 2-10 minutes. Phase 12-02 (frontend library management UI) was the longest at 38 minutes due to complexity (19 files, 3 tasks, new components)
• Efficiency: 18 plans across 6 phases with 4 quick tasks interleaved

---

Cross-Milestone Trends

Process Evolution

Milestone	Days	Phases	Plans	Key Change
v1.0	6	8	17	First milestone — established GSD workflow, research-before-plan pattern
v1.1	10	6	18	Locked decisions, parallel phase execution, hook patterns for cross-package coordination

Cumulative Quality

Milestone	Tests Added	Total Tests	Key Quality Win
v1.0	84	84	From 0 backend tests to comprehensive coverage of queue, config, player, database, library
v1.1	~5	~89	Migration tests, multi-root path resolution tests; human checkpoint caught 3 integration bugs

Top Lessons (Verified Across Milestones)

1. Dependency-ordered phases (fix → test → refactor → optimize; schema → scan → CRUD → views) prevent rework and ensure each phase builds on a stable foundation
2. Small plans (2-3 tasks, <10 min) maintain consistent quality — no context degradation
3. Research phases for unfamiliar domains (sqlc + VIEW, lit-virtualizer API) prevent mid-execution surprises
4. Human checkpoints catch integration bugs that automated verification misses — budget time for them
5. Analyze existing data flows before designing new storage — the simplest solution often uses data that already exists