yonlu/yellowjacket
1
1
Fork 0
Code Issues 48 Pull Requests Actions Packages Projects Releases 12 Wiki Activity

feat: autotag scoring overhaul, dump-based explore index, and lyrics search

Browse Source 
Consolidates in-progress work across autotag, explore, and library:

- autotag: beets/Picard-informed scoring engine — ID-first matching, VA
  handling, recommendation tiers, and a merged distance/rank cascade, with
  an eval harness for regression tracking.
- explore: offline MusicBrainz dump import/incremental refresh replaces the
  legacy tier crawl; index-first local search with fuzzy matching and a
  dedicated ranker; disk-free guards for dump downloads.
- library: artist-credit extraction and matching.
- lyrics: owned-library lyric search (FTS) with LRCLIB backfill.

Also: rewrite README to be user-focused, and migrate upstream to
git.ljones.me/yonlu/yellowjacket.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
yonlu
2026-07-24 12:14:20 -04:00
co-authored by Claude Opus 4.8
parent d5140395da
commit 65048401e8
117 changed files with 17033 additions and 4767 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/NOTES.md
-70
View File
@@ -1,70 +0,0 @@
# Notes
Miscellaneous things worth knowing that aren't documented elsewhere. CLAUDE.md owns the architecture overview, the build commands, and the per-package responsibilities — this file is for the gotchas, the deferred-but-tracked items, the "we already considered and rejected" decisions, and the things that bite if you forget them.
## Hard SQLite-driver constraints (forget at your peril)
- `MaxOpenConns(1)` is set on the SQLite connection. **Holding `*sql.Rows` open while calling another function that queries the same `*database.DB` will deadlock.** Close rows explicitly before downstream calls; don't rely on `defer` when the deferred close has to happen *after* a downstream query. This bit smart-playlist S01/T03 and the smart-playlist auto-edit feature; it bites the play-history hook so `OnPlaybackFinished` unlocks the player mutex *before* recording a play.
- No CGo: the driver is `modernc.org/sqlite`. Cannot switch to a CGo-based driver — design around it.
- Connection pooling, dynamic ORDER BY in sqlc, and `FILTER (WHERE ...)` in older modernc versions all require care. (`FILTER` works in current `modernc.org/sqlite`, but if it ever fails, fall back to `SUM(CASE WHEN ... THEN 1 ELSE 0 END)`.)
## Explore + Autotagger API-call-minimization playbook
Every MusicBrainz interaction follows this resolution order — check before sending:
1. **Local DB** — does a matching `release_groups` row already exist? (zero network cost)
2. **Existing partial MBIDs** on the album's tracks — use as Lucene filters (`arid:`, `rgid:`) to narrow search and produce deterministic cache keys.
3. **`http_cache`** (already wrapped transparently by `MusicBrainzClient`) — serves hits for any previously-fetched entity.
4. **Live MB fetch** — last resort.
Plus: share fan-out (one `LookupArtist` per album, not per track); persist decisions to `tagging_items` so reopening the review UI fires zero MB calls; prefetch album N+1 while user reviews album N; cover art is pull-on-apply only; auto-accept never fetches incrementally.
Cache TTLs: 24 h for searches (data updates often), 7 d for entity lookups (artist details, discography are stable).
## Tag-editing pipeline invariants
- **`AtomicWrite`** writes to `<filename>.yj-tmp` in the same directory, then renames. Same-directory rename avoids cross-device issues; deterministic suffix enables orphan cleanup on startup.
- **Upsert-and-relink for entity sync.** Never mutate shared `artists` / `albums` / `genres` rows in place — create new ones or relink to existing rows. Safe under concurrent reads.
- **Currently-playing file gets stopped before its tag write.** `PlayerStopper` interface (implemented by `playerAdapter` in `app.go`) breaks the import cycle.
- **Scan and write are mutually exclusive** via `pipelineMu` in the library package.
- **Batch writes coalesce events** with a `suppressEvents` flag — one `TrackMetadataChanged` per batch instead of N.
## Frontend gotchas
- **Wails TS bindings for smart-playlist methods are manually maintained** in `frontend/wailsjs/go/playlist/Service.{d.ts,js}`. The Wails build does *not* re-generate them in worktrees, and no build-time check catches drift if Go signatures change. Same for any explore method added outside a clean Wails build.
- **`pnpm build` runs from `frontend/`, not project root.** Root `package.json` is empty.
- **Combobox blur-vs-click race** — the `mousedown` + `preventDefault()` + `requestAnimationFrame` fallback in `combobox.ts` is fragile if option rendering moves into a separate shadow DOM. Re-verify click-to-select after any combobox refactor.
- **`go build ./...` fails in git worktrees** because `main.go:28` embeds `frontend/dist`, which doesn't exist in a fresh worktree. Use `go build ./backend/...` for backend-only verification.
- Explore detail components duplicate `CoverArtGroupURL`, `nameToHue`, `extractYear`, `formatDuration` from `explore-view`. Three consumers as of v1.3 planning. If a fourth emerges, extract to `explore-utils.ts`.
## Lint baseline
`make lint` may report a small number of pre-existing warnings (3 wsl_v5 in `database_test.go`, 1 gci + 1 revive in `smartplaylist.go` last time anyone counted). Don't chase these in unrelated PRs. Note them as pre-existing in verification evidence and move on. Anything new must be clean.
## Out-of-scope decisions worth remembering the *why* for
- **Separate databases per library** — defeats unified presentation; rejected.
- **Auto-dedup across libraries** — complex matching logic, not table stakes.
- **Parallel library scanning** — SQLite single-writer; pointless.
- **ORM / query builder** — fights existing sqlc architecture.
- **Connection pooling for SQLite** — meaningless under `MaxOpenConns(1)`.
- **Database health checking / reconnection** — desktop app context, low priority.
- **Cosmetic file splitting** — large files are only a problem if they cause real issues. Extract for reuse or correctness, not aesthetics.
- **Parenthesized boolean logic in smart playlists** — UI complexity not worth the use case; AND-only with multi-value `is_any_of` covers the vast majority.
- **"Is favorited" as a smart-playlist filter** — favoriting is a special-case relationship, not a queryable field.
- **Playing audio remotely from MB** — MB is a metadata catalog, not a streaming service.
- **Fuzzy auto-accept threshold slider** — strict all-match is the trustworthy default; a slider is a power-user foot-gun.
- **Manual MB search UI in autotag** — Paste-URL covers the escape hatch; full search is surface area we don't need yet.
- **Configurable field whitelist for autotag writes** — hardcoded list in v1; add config only if users actually ask.
- **Folder-level cover art (`folder.jpg`/`cover.png`)** — separate feature area from embedded art.
- **Per-library autotagger on/off** — single global setting; per-library adds UI for no clear benefit.
## Known gap from milestone 007
R032 (offline visual indicator for cached Explore data) was scoped into M004 but not implemented. The cache layer works correctly — entries are served when offline until TTL expires — but `Cache.Get()` doesn't propagate a "from cache" flag, and no frontend component renders a "Cached" badge. If picked up: backend modifies `Cache.Get()` to return a `fromCache` bool, frontend adds a subtle badge. ~30 min of work.
## Open architecture questions
- **VA compilation detection threshold for the autotagger.** Per-track artist credits differing from album-artist is the easy heuristic. What's the threshold on number of differing tracks before we relax the artist-match rule? Will surface during scoring tuning in plan 009.
- **Cover Art Archive minimum-dimension check.** REVIEW-05 (plan 010) sets 500 px on the shortest side. Coarse but cheap. Expect tuning once we see real CAA quality variance across genres.
- **Rate-limit priority queue design.** CFG-02 (plan 012) wants user-initiated MB calls to jump ahead of background auto-accept. Implementation strategy is open — priority channel? Two limiters with a yield mechanism? Solve when 012 starts.
.planning/ROADMAP.md
-61
View File
@@ -1,61 +0,0 @@
# YellowJacket — Roadmap
A cross-platform desktop music player. Plays local MP3 / FLAC / OGG / WAV; manages multiple library directories via SQLite; queue, playlists, smart playlists, play history, full-text search, cover art, MPRIS controls on Linux, full single + batch tag editing across all four formats, and a read-only MusicBrainz / ListenBrainz catalog browser.
The guiding rule for everything below is "the music player works reliably and feels solid" — every interaction should be correct, responsive, and trustworthy. New surface area only goes in once the foundation under it is stable.
## Capability set
- **Playback** — play / pause / seek / volume on MP3, FLAC, OGG, WAV. Queue with shuffle (Fisher-Yates), repeat (off / one / all), auto-advance, persistent across restarts.
- **Library** — multiple library directories, concurrent metadata extraction, adaptive scan concurrency by disk type, cancellable / pausable scans, cross-library playlists with phantom-track preservation when a library is removed.
- **Search & browse** — FTS5 across tracks/artists/albums/paths; browse by albums / artists / genres; library filter respected everywhere; virtual scrolling.
- **Playlists** — CRUD, M3U8 import/export, favorites, smart playlists (rule-based saved queries with combobox editor + live preview), default playlist.
- **Tag editing** — single + batch edit of 8 fields across all four formats; cover art embed/replace/remove; crash-safe atomic writes; instant DB + FTS5 sync, no rescan needed.
- **Play history** — natural-finish play counting, `last_played` timestamp, play-history log, integration into smart playlist rule fields.
- **Explore** — read-only MusicBrainz / ListenBrainz browser: search → artist page → album detail with release-version selection, with rate-limited APIs and SQLite-cached responses.
- **Configuration & system** — TOML config with live reload, configurable keyboard shortcuts (record-style capture, scope-aware dispatch), theme tokens (accent / background shade), MPRIS2 on Linux.
## Milestone sequence
| # | Milestone | Status | File |
|---|-----------|--------|------|
| 001 | v1.0 Consolidation | shipped 2026-03-05 | [completed/001-v1.0-consolidation.md](plans/completed/001-v1.0-consolidation.md) |
| 002 | v1.1 Multi-Library Support | shipped 2026-03-16 | [completed/002-v1.1-multi-library.md](plans/completed/002-v1.1-multi-library.md) |
| 003 | v1.2 Tag Editing (MP3 + FLAC) | shipped 2026-03-18 | [completed/003-v1.2-tag-editing.md](plans/completed/003-v1.2-tag-editing.md) |
| 004 | v1.2.1 Format Parity (OGG + WAV) | shipped 2026-03-21 | [completed/004-v1.2.1-format-parity.md](plans/completed/004-v1.2.1-format-parity.md) |
| 005 | Smart Playlists | shipped 2026-03-22 | [completed/005-smart-playlists.md](plans/completed/005-smart-playlists.md) |
| 006 | Play History & Play Count | shipped 2026-03-22 | [completed/006-play-history.md](plans/completed/006-play-history.md) |
| 007 | MusicBrainz/ListenBrainz Explore Browser | shipped 2026-03-24 | [completed/007-explore-browser.md](plans/completed/007-explore-browser.md) |
| 008 | Autotag — Schema & Grouping Foundation | shipped 2026-04-20 | [completed/008-autotag-schema-grouping.md](plans/completed/008-autotag-schema-grouping.md) |
| 009 | Autotag — Scoring Engine & MB Orchestration | shipped 2026-04-21 | [completed/009-autotag-scoring-engine.md](plans/completed/009-autotag-scoring-engine.md) |
| 010 | Autotag — Review UI & Apply Pipeline | **active** | [active/010-autotag-review-ui.md](plans/active/010-autotag-review-ui.md) |
| 011 | Autotag — Auto-Accept & Entry Points | pending | [pending/011-autotag-auto-accept.md](plans/pending/011-autotag-auto-accept.md) |
| 012 | Autotag — Settings & Polish | pending | [pending/012-autotag-settings-polish.md](plans/pending/012-autotag-settings-polish.md) |
The active phase is the **MusicBrainz autotagger** (collectively v1.3). It builds on the explore-browser API client + cache foundation from milestone 007. Plans 008-012 are sequential — each depends on the prior one. See `NOTES.md` for the API-call-minimization playbook and the design principles that constrain every Autotag plan.
## Beyond v1.3 (not yet planned)
Captured here so they don't get lost; not yet promoted to plan files.
- **ListenBrainz scrobbling.** Submit play data when a track crosses the scrobble threshold (`min(duration / 2, 4 minutes)`). Three submission types — `playing_now`, `single`, `import`. Hook slots in next to the existing play-history pipeline; MBIDs already flow through the stack thanks to v1.3.
- **Gapless playback + crossfade.** Seamless transitions with optional crossfade.
- **Layout customization system.** Section-based UI customization; components declare size constraints, users configure per-section.
- **Plugin system.** Full-access API for UI components and backend hooks.
- **AcoustID fingerprinting** via fpcalc — slots in behind the v1.3 `Identifier` interface seam.
## Out of scope
Things deliberately **not** going in. See `NOTES.md` for the reasoning behind each.
- Separate databases per library, auto-dedup across libraries, user access control per library.
- Parallel library scanning (SQLite single-writer).
- Cross-platform media controls beyond MPRIS on Linux.
- Database health checking / reconnection.
- ORM or query builder, connection pooling for SQLite.
- Parenthesized boolean logic in smart playlists; "is favorited" as a filter; queue that re-evaluates rules during playback.
- Playing audio from MusicBrainz remotely (it's a metadata catalog, not a streaming service).
- Integrating any service beyond MusicBrainz / ListenBrainz / Cover Art Archive without explicit user approval.
- Fuzzy auto-accept threshold sliders (strict all-match is the trustworthy default).
- Manual MB search UI (Paste-URL covers the escape-hatch case).
- Cover art replacement during auto-accept (highest-regret op — never automatic in v1).
.planning/autotag.md
+162
View File
@@ -0,0 +1,162 @@
# Autotag (v1.3) — MusicBrainz Autotagger
The MusicBrainz autotagger, collectively **v1.3**. Builds on the explore-browser API client + cache foundation. Five sequential phases (008–012), each depending on the prior one.
| Phase | Title | Status |
|-------|-------|--------|
| 008 | Schema & Grouping Foundation | shipped 2026-04-20 |
| 009 | Scoring Engine & MB Orchestration | shipped 2026-04-21 |
| 010 | Review UI & Apply Pipeline | **active** |
| 011 | Auto-Accept & Entry Points | pending |
| 012 | Settings & Polish | pending |
---
## 008 — Schema & Grouping Foundation · shipped 2026-04-20
> First v1.3 phase. Lays down the schema + bookkeeping — no scoring, no UI, no MB calls yet.
### What landed
- **008.1 — Migration 31: `audio_files.tag_status`.** Text column with inline `CHECK(tag_status IN (...))` constraint (SQLite `ALTER TABLE ADD COLUMN` supports column constraints, so both fresh and upgraded DBs enforce it). Partial index `idx_audio_files_tag_status_untagged ON audio_files(library_id) WHERE tag_status = 'untagged'` powers the pending badge. Backfill sets `user_confirmed` where the joined recording already has an MBID; everything else defaults to `untagged`.
- **008.2 — Migration 32: `tagging_items` + `group_key`.** New `tagging_items` table (PK `group_key`, `status CHECK IN ('pending','matched','confirmed','skipped')`, two indexes including the partial `WHERE status = 'pending'` for the badge). `audio_files.group_key TEXT NOT NULL DEFAULT ''` with partial index `WHERE group_key != ''`. Column-referencing indexes live in the migration, not the schema file, because `CREATE TABLE IF NOT EXISTS` is a no-op on existing tables and the partial-index predicate would reference a column that hasn't been added yet.
- **008.3 — `backend/autotag.GroupKey` + scan-path integration.** Lower-case hex SHA-1 over `libraryID || 0 || parentDirLower || 0 || albumTrimmed || 0 || discNumber` — intentionally shallow, deeper normalization is scoring territory (009). `saveAudioFile` switched to `CreateAudioFileWithGroupKey` + `UpsertTaggingItemOnTrackAdd`. `updateAudioFileMetadata` rebinds on key change: decrement old group → delete if empty → upsert new group → write new key onto the audio_files row. Migration 32 streams existing rows in batches of 500 and aggregates `tagging_items` at the end, defaulting status to `confirmed` when every track in the group is `user_confirmed`.
- **008.4 — Pending-queue sqlc queries.** `CountPendingTaggingItems`, three `ListPendingTaggingItems...` variants (alphabetical, by-score nulls-last, by-recent) — sqlc has no dynamic ORDER BY so each sort has its own query. `CAST(@param AS INTEGER|TEXT)` hints give typed params (otherwise sqlc emits `interface{}`). `GetTaggingItem` and `ListAudioFilesInTaggingGroup` round out the queue API. `EXPLAIN QUERY PLAN` test asserts the badge query uses `idx_tagging_items_status_pending` so a future schema change that breaks the partial index fails loudly.
### Key decisions retained
- **Hash algorithm in Go, not SQL.** `autotag.GroupKey` is the single source of truth for the key format so we can evolve it without coupling to SQLite functions.
- **SHA-1 over alternatives.** Matches the codebase's existing non-crypto deterministic-key convention. Collision risk at album-group cardinality (millions) is irrelevant.
- **Null-byte separators between hash inputs.** Prevents `a|b` vs `ab|` ambiguity.
- **Per-track integration inside `commitBatch`'s transaction**, not a post-scan callback — keeps `tagging_items` coherent after partial scans.
- **`best_match_release_mbid`, `score`, `last_checked_at` shapes fixed now** even though they stay NULL until 009-010. Avoids schema churn later.
---
## 009 — Scoring Engine & MB Orchestration · shipped 2026-04-21
> Second v1.3 phase. Given an album-group, produce a ranked list of candidate releases with per-track alignment data, using as few MusicBrainz calls as the API-minimization playbook allows. No UI yet — 010 surfaces this to the user.
### What landed
- **`backend/autotag/` domain types** — `Candidate`, `TrackAlignment`, `GroupScore`, `LocalTrack`, `CandidateSource` (`local` / `musicbrainz`), `AlignmentStatus` (`matched` / `missing` / `extra` / `mismatched`).
- **`Normalize(s)`** — Unicode NFC → qualifier-suffix strip (`(Remastered 2009)`, `[Bonus Track]`, `(feat. X)`, etc.) → case fold → punctuation drop → whitespace collapse. Comparison-only, not human-readable.
- **Per-track distance** — weighted 60% title similarity (1.0 − Levenshtein/max-len), 30% length delta (linear: ≤1 s = 1.0, ≥30 s = 0.0, neutral 0.5 when either side unknown), 10% track-number match.
- **Greedy alignment** — `AlignTracks` picks the highest-scoring (local, cand) pair iteratively; not Hungarian-optimal but fine at album cardinalities. Emits `matched` / `missing` / `extra` / `mismatched` rows so the review UI can render the diff.
- **Local-first resolver** — `ListLocalReleaseGroupCandidates` sqlc query pre-filters on `rg.mbid != '' AND r.mbid != '' AND rg.name = ? COLLATE NOCASE`, then Go applies full normalization. Candidate track lists only include recordings that themselves have MBIDs (avoids untagged dupes polluting the "canonical" view when the same `(name, artist)` release group is shared across libraries).
- **MB orchestration** — `MBClient` interface (`SearchReleaseGroups`, `BrowseReleases`, `LookupArtist`) hides `explore.MusicBrainzClient`; `backend/explore/autotagclient.go` adapts one to the other. `buildMBQuery` assembles `release:"X" AND arid:<mbid> AND tracks:N` — `arid:` makes the search cache key deterministic, `tracks:N` filters out box-set-style releases. One search per album + one `BrowseReleases` per candidate RG.
- **Release-level ranker** — aggregate track score (70%) + track-count match (15%, zero at ≥50% delta) + meta (15%, avg of year/Official/country bonuses). `Scorer.ScoreGroup` hits local first, runs MB only when no local candidate scores ≥ 0.90.
- **Persistence** — `SetTaggingItemBestMatch` writes `best_match_release_mbid`, `score`, `last_checked_at = CURRENT_TIMESTAMP`, `status = 'matched'`.
### Key decisions retained
- **SHA-1-grade normalization vs. full MB-equivalent.** Qualifier regex handles the common cases (remaster, deluxe, explicit, feat., bonus, etc.) without dragging in a full MB title-parsing library. Edges that bite in real libraries will show up in 010 review UX and can be patched then.
- **`*sqlcgen.Queries` as the DB boundary for autotag**, not `*database.DB`. The `database` package already depends on `autotag.GroupKey` (from 008.3), so reversing the dependency via `database` would create a cycle. Using `sqlcgen` directly is acyclic and keeps `autotag` swappable.
- **Scorer constructor takes `MBClient` as interface, not `*explore.MusicBrainzClient`.** Lets tests inject a stub without spinning up the HTTP + cache layer. The concrete adapter lives in `explore/autotagclient.go`.
- **`localSufficient = 0.90` threshold for skipping MB.** Empirical guess — will get retuned in 011 auto-accept phase when we observe real corpus scores.
- **Weights `60/30/10` for title/length/track-number.** Cribbed from beets' broad intuition; tuned to emphasize title-matching since length data can be unreliable from Vorbis Comments. Tests document the expected floors (e.g. "exact match should score ≥ 0.99") so nudging weights won't silently regress.
- **`release_groups` and `recordings` must both carry MBIDs** for a local candidate. Otherwise untagged dupes of the same album (across libraries) falsely expand the "canonical" track list.
- **UTF-8 em-dashes in SQL comments broke sqlc's string-literal emitter**, truncating generated query strings mid-word. All autotag SQL comments use ASCII punctuation.
### Known follow-ups into 010+
- **`guessArtistMBID` is a stub** returning `""` because `LocalTrack` doesn't currently carry artist MBIDs. 010 should thread artist MBIDs through `ListAudioFilesInTaggingGroup` so the MB resolver can use `arid:` filters.
- **`yearBonus` uses `time.Now().Year()`** as a placeholder target. Should become the earliest release-date hint from the group's tracks once 010 provides it.
- **VA compilation detection threshold** is still open. The scorer doesn't special-case per-track artist credits differing from album-artist.
---
## 010 — Review UI & Apply Pipeline · ACTIVE
> Third v1.3 phase. User reviews one album at a time, sees the diff clearly, and applies or skips — file tags get written, DB gets synced, cover art follows the never-replace-existing rule.
**Requirements:** REVIEW-01..07 · **Depends on:** 009 (needs candidates + scores)
### Success criteria
1. `/autotag` shows the next pending album with its top candidate as a field-by-field diff. Missing-from-local and extra-in-local tracks are shown explicitly.
2. Keyboard shortcuts work without the mouse: `A` apply, `M` more candidates, `S` skip, `L` leave-as-is, `U` paste URL, `→`/`←` navigate.
3. Apply writes tags to every track in the group via the existing format-specific writers + atomic write + DB sync + FTS5 sync. Whitelisted fields only: title, artist, album, album-artist, year, track#, track-total, disc#, disc-total, all MBIDs.
4. Cover art rule: embed only when the file has no existing art **and** CAA returns ≥500 px on the shortest side. Never replace existing embedded art (auto or manual).
5. First-ever apply per library shows an irreversibility warning. "Don't show again" sets a flag on the `libraries` row; never shows again for that library.
6. While the user reviews album N, candidates for album N+1 are prefetched into `http_cache` so advancing feels instant.
7. "Paste MB URL" dialog accepts a release URL, extracts the MBID, runs one `LookupRelease`, renders the diff against the current album.
### Sub-plans
- Wails bindings — `StartAutotagQueue`, `GetCurrentCandidate`, `GetCandidates(groupKey)`, `Apply(groupKey, releaseMBID)`, `Skip`, `LeaveAsIs(groupKey)`, `RetagGroup(groupKey)`.
- `/autotag` page layout — focused album header, diff table, candidate sidebar, missing/extra panel.
- Keyboard shortcut wiring through the existing scope-aware dispatch.
- Apply pipeline integration with existing tag writers + DB sync.
- Cover art apply rule + CAA fetch + 500 px minimum check.
- File-write warning dialog with per-library persistence.
- Prefetch-next-album goroutine, rate-limiter aware.
- Paste-MB-URL escape hatch.
### Risk callout
Every apply rewrites a file. The `AtomicWrite` pipeline mitigates corruption risk; the per-library warning mitigates surprise. Dry-run mode (from 009) lets developers validate scoring changes without file writes.
---
## 011 — Auto-Accept & Entry Points · pending
> Fourth v1.3 phase. The strict all-match auto-accept path runs as a background job; the tool is reachable from every place a user expects.
**Requirements:** AUTO-01..06 · **Depends on:** 010 (needs the apply pipeline)
### Success criteria
1. An album-group qualifies for auto-accept iff: exact track-count match, every track's normalized title matches, every track's length within ±2s, no cover-art replacement required, no existing-MBID conflicts. Decision uses already-cached candidate data — **no additional MB calls**.
2. Auto-accept job processes all qualifying groups in the queue, emits progress events, is cancellable at any point, honors the shared rate limiter.
3. Right-click on a track / album / artist exposes "Autotag this album" (queues + jumps to review) and "Retag" (flips status to `untagged` and requeues).
4. After a library scan finishes with N new untagged albums, a non-blocking toast appears linking to `/autotag`.
5. Sidebar has an "Autotag" nav entry with a pending-count badge, updates reactively.
6. Pasting a MB release URL into the Paste-URL dialog renders a full diff against the current album with one `LookupRelease` call.
### Sub-plans
- Strict all-match rule + unit tests.
- Auto-accept background job — progress events, cancellation, queue traversal.
- Context menu integrations on track/album/artist views.
- Post-scan toast wiring via the existing scan-complete event.
- Sidebar nav entry + pending-count badge store integration.
### Risk callout
Release selection can pick the wrong edition. The exact-track-count gate prevents most silent misbehavior, but bonus-track editions and remaster reissues with matching track counts are genuine ambiguity. Manual review handles the edge cases — that's why auto-accept is strict by design, and the slider for fuzzy auto-accept is explicitly out of scope.
---
## 012 — Settings & Polish · pending
> Fifth and final v1.3 phase. Configuration surfaces in the existing settings system; the known sharp edges (rate-limit contention, VA compilations, singleton files) get sanded; the fingerprinting seam is in place for the future.
**Requirements:** CFG-01..06 · **Depends on:** 011
### Success criteria
1. Autotag settings panel accessible via the existing templ/HTMX settings UI. Exposes: enable/disable auto-accept, per-library file-write warning reset, default review filter, default sort order.
2. Shared rate limiter distinguishes interactive from background requests. User-initiated MB calls (paste-URL, opening a review, explore browsing) are never blocked behind a running auto-accept job.
3. VA compilation albums (per-track artist credits differ from album-artist) are detected. The auto-accept artist-match rule relaxes for them; the ranker prefers MB releases credited to "Various Artists".
4. Singleton files (`track_count = 1`, no sibling context) use a recording-level match path (`SearchRecordings` with title + artist + length filters). Lower confidence ceiling — never eligible for auto-accept regardless of confidence.
5. `type Identifier interface { Identify(path) ([]Candidate, error) }` exists with `MetadataIdentifier` as the v1 implementation. No fpcalc integration, but the seam is in place for a future `AcoustIDIdentifier`.
6. User-facing quickstart docs exist; CLAUDE.md gets a `backend/autotag/` package description; scoring-function dev notes are committed.
### Sub-plans
- Autotag settings panel (templ + HTMX).
- Rate-limiter priority support.
- VA compilation detection and scoring adjustments.
- Singleton-file match path.
- `Identifier` interface seam with `MetadataIdentifier`.
- Docs — user quickstart + dev notes + CLAUDE.md update.
---
## Ship criteria for v1.3 overall
- All 29 SCHEMA/MATCH/REVIEW/AUTO/CFG requirements complete.
- All five phases' success criteria verified end-to-end on a real library (10k+ tracks, mixed match quality).
- Auto-accept run against a well-tagged subset produces zero incorrect matches.
- Manual review workflow can process 100 albums in under 30 minutes without mouse use.
.planning/plans/active/010-autotag-review-ui.md
-30
View File
@@ -1,30 +0,0 @@
# 010 — Autotag: Review UI & Apply Pipeline
> Third v1.3 phase. User reviews one album at a time, sees the diff clearly, and applies or skips — file tags get written, DB gets synced, cover art follows the never-replace-existing rule.
**Status:** pending · **Requirements:** REVIEW-01..07 · **Depends on:** 009 (needs candidates + scores)
## Success criteria
1. `/autotag` shows the next pending album with its top candidate as a field-by-field diff. Missing-from-local and extra-in-local tracks are shown explicitly.
2. Keyboard shortcuts work without the mouse: `A` apply, `M` more candidates, `S` skip, `L` leave-as-is, `U` paste URL, `→`/`←` navigate.
3. Apply writes tags to every track in the group via the existing format-specific writers + atomic write + DB sync + FTS5 sync. Whitelisted fields only: title, artist, album, album-artist, year, track#, track-total, disc#, disc-total, all MBIDs.
4. Cover art rule: embed only when the file has no existing art **and** CAA returns ≥500 px on the shortest side. Never replace existing embedded art (auto or manual).
5. First-ever apply per library shows an irreversibility warning. "Don't show again" sets a flag on the `libraries` row; never shows again for that library.
6. While the user reviews album N, candidates for album N+1 are prefetched into `http_cache` so advancing feels instant.
7. "Paste MB URL" dialog accepts a release URL, extracts the MBID, runs one `LookupRelease`, renders the diff against the current album.
## Sub-plans
- Wails bindings — `StartAutotagQueue`, `GetCurrentCandidate`, `GetCandidates(groupKey)`, `Apply(groupKey, releaseMBID)`, `Skip`, `LeaveAsIs(groupKey)`, `RetagGroup(groupKey)`.
- `/autotag` page layout — focused album header, diff table, candidate sidebar, missing/extra panel.
- Keyboard shortcut wiring through the existing scope-aware dispatch.
- Apply pipeline integration with existing tag writers + DB sync.
- Cover art apply rule + CAA fetch + 500 px minimum check.
- File-write warning dialog with per-library persistence.
- Prefetch-next-album goroutine, rate-limiter aware.
- Paste-MB-URL escape hatch.
## Risk callout
Every apply rewrites a file. The `AtomicWrite` pipeline mitigates corruption risk; the per-library warning mitigates surprise. Dry-run mode (from 009) lets developers validate scoring changes without file writes.
.planning/plans/completed/001-v1.0-consolidation.md
-22
View File
@@ -1,22 +0,0 @@
# 001 — v1.0 Consolidation
> Foundation milestone: fixed all SetContext data races, established the test infrastructure, consolidated SQL, and tuned backend + frontend performance so feature work could safely follow.
**Shipped:** 2026-03-05 · **Phases:** 1-8
## What landed
- **Phase 1 — Concurrency race fixes.** Mutex-protected `SetContext` across queue, library, playlist, player; collapsed Player double-lock. App runs `-race`-clean.
- **Phase 2 — Backend correctness.** `startupErr` moved to struct field; config files written 0o644; MPRIS callback errors logged; `Library.Scan()` separates warnings (`ScanMetrics`) from fatal errors (return value).
- **Phase 3 — Test infrastructure.** `database.NewTestDB(t)` returns an in-memory SQLite with full migrations + production PRAGMAs (`synchronous=NORMAL`, `cache_size=-8000`, `mmap_size=67108864`).
- **Phase 4 — Queue/Config/Player tests.** ~30 tests covering SetQueue, navigation, shuffle, repeat modes, persistence, config load/save roundtrip, volume conversion, format detection.
- **Phase 5 — Database/Library tests.** ~25 tests covering FTS5 search, search index rebuild, migrations, scan logic, entity cache.
- **Phase 6 — SQL consolidation.** 5-table FTS5 JOIN consolidated into the `track_metadata` VIEW; AST-based event codegen (Go → TypeScript) with pre-commit hook; `sqlc.slice()` for queue batch lookups; every hand-crafted SQL statement carries a `// SAFETY:` comment explaining why.
- **Phase 7 — Backend performance.** Incremental queue persistence (O(1) add/remove), SetQueue Phase-2 dedup, library store deferred loading.
- **Phase 8 — Frontend performance.** Lit `repeat()` with stable keys; `queueMicrotask` notification batching; design token system; visual consistency audit across 15 components.
## Key decisions retained
- **No cosmetic file splitting** — extract only when reuse or correctness demands it.
- **Tests as a means to safe refactoring**, not a coverage target.
- **Fix races → tests → refactoring** order is mandatory: can't run `-race`-clean tests with active races; can't safely refactor without tests.
.planning/plans/completed/002-v1.1-multi-library.md
-23
View File
@@ -1,23 +0,0 @@
# 002 — v1.1 Multi-Library Support
> Lets users register and manage multiple music library directories with per-library scanning, filtered views, cross-library playlists, and phantom-track preservation when a library is removed.
**Shipped:** 2026-03-16 · **Phases:** 9-14
## What landed
- **Phase 9 — Scan cancellation & keyboard shortcuts.** Cancellable/pausable library scans via per-scan context; configurable shortcuts with record-style capture, scope-aware dispatch, conflict detection.
- **Phase 10 — Schema & migration.** New `libraries` table, `library_id` FK on `audio_files`, phantom columns on `playlist_tracks`, seamless single-directory → named-library migration on first launch.
- **Phase 11 — Per-library scan pipeline.** Sequential scan-queue coordinator (single writer is the SQLite reality); per-library progress UI with library name; cancel scope respects the active library.
- **Phase 12 — Library CRUD & data integrity.** Add/rename/remove library through UI; atomic orphan cleanup (shared artists/albums/genres survive); FTS5 entries removed; queue tracks from a removed library cascade-deleted with playback continuing.
- **Phase 13 — Library views & phantom tracks.** Library filter dropdown across all views; tracks/albums/artists/genres/search respect the active filter; cross-library playlists work naturally; removed-library tracks become greyed-out phantoms with preserved metadata, auto-resolved on rescan via `ScanHooks` + M3U8 matching.
- **Phase 14 — Performance optimization.** CSS containment + GPU promotion on scroll containers; view caching (display toggle, no DOM destruction); event delegation on virtualizer; `queueMicrotask` notification batching; pprof profiling guide.
## Key decisions retained
- **Hybrid model — `library_id` only on `audio_files`.** Physical files belong to a library; logical entities (artist/album/genre) are shared.
- **Libraries live in DB, not TOML.** UI-driven CRUD shouldn't require TOML manipulation.
- **`SET NULL` on `playlist_tracks` FK.** Phantom rows preserve playlist structure when a library is removed.
- **Backend filters, frontend doesn't.** `ByLibrary` SQL variants keep the UI responsive at 150k tracks.
- **Sequential scanning.** SQLite single-writer makes parallel scans pointless.
- **`ScanHooks` callback pattern.** Breaks the library → playlist circular dep needed for phantom resolution.
.planning/plans/completed/003-v1.2-tag-editing.md
-24
View File
@@ -1,24 +0,0 @@
# 003 — v1.2 Tag Editing (MP3 + FLAC)
> First write path into user files: edit any track's metadata or cover art (single or batch) with crash-safe writes, then synchronize the database, FTS5 index, and entity graph in the same operation. MP3 and FLAC only — OGG and WAV ship in v1.2.1.
**Shipped:** 2026-03-18 · **Phases:** 15-18
## What landed
- **Phase 15 — Schema migration & write safety.** FTS5 migrated to `contentless_delete=1` so per-row deletes/updates work without ghost search hits. New `backend/fileutil` package provides `AtomicWrite` (write to `.yj-tmp` sibling, then rename) with crash-safe semantics and orphan cleanup on startup.
- **Phase 16 — Tag writers + DB sync.**
- `backend/tagwriter/` writes ID3v2 (MP3) and Vorbis Comments (FLAC), including embedded cover art.
- `WriteTrackTags` pipeline: file write → transactional DB sync (entity upsert-and-relink + FTS5 update + orphan cleanup) → event emission.
- `PlayerStopper` interface (implemented by `playerAdapter` in `app.go`) lets tagwriter request playback stop without an import cycle.
- Scan/write mutual exclusion via `pipelineMu`.
- **Phase 17 — Single-track editor.** Right-click any track anywhere; 8 editable fields (title/artist/album/genre/year/track#/disc#/composer); cover art pick/replace/remove; diff-only saves; all visible views refresh without rescan.
- **Phase 18 — Batch editor.** Multi-select; three-state field model (keep / set / clear) driven implicitly by `editValues` presence; confirmation guard; progress bar with cancellation; partial-failure reporting; batch cover art set/clear.
## Key decisions retained
- **Upsert-and-relink for entity sync.** Never mutate shared entities — create new rows or relink to existing ones. Safe under concurrent reads.
- **`AtomicWrite` with deterministic `.yj-tmp` suffix.** Same-directory rename avoids cross-device issues; deterministic naming enables orphan cleanup.
- **`suppressEvents` flag for batch coalescing.** One `TrackMetadataChanged` event per batch, not N individual ones.
- **Three-state field model via dirty tracking.** `editValues` presence is the signal — no explicit state enum needed.
- **go-flac ecosystem.** Small library (44★) but the only pure-Go FLAC writer. dhowden/tag reads back what go-flac writes; 7 round-trip tests validate.
.planning/plans/completed/004-v1.2.1-format-parity.md
-18
View File
@@ -1,18 +0,0 @@
# 004 — v1.2.1 Format Parity (OGG + WAV)
> Closes the tag-writing format gap: every file YellowJacket plays can also be edited. OGG Vorbis and WAV writers, plus a wsl_v5 lint cleanup. (This was tracked as M001 in the GSD system; S01–S18 in that milestone were just retroactive imports of v1.0–v1.2.)
**Shipped:** 2026-03-21 · **Phases:** 19-21
## What landed
- **Phase 19 — WAV tag writer.** Custom RIFF parser with lenient-read / strict-write behavior. ID3v2 chunk placed at end of file after preserved chunks. Case-insensitive `id3`/`ID3` chunk detection. Existing ID3v2 tags merged so unknown frames written by other tools survive. Read-back uses `bogem/id3v2.ParseReader` (handles cleared tags correctly; dhowden/tag does not).
- **Phase 20 — OGG Vorbis tag writer.** Custom OGG page parser/writer with MSB-first CRC32 (Go's `hash/crc32` uses incompatible reflected bit ordering, so a precomputed 256-entry lookup table lives in-package). Vorbis Comment serializer with `METADATA_BLOCK_PICTURE` cover art. Raw byte preservation for non-edited entries — avoids corrupting non-UTF-8 tags. Tests use a programmatic fixture builder (no embedded binaries). CRC32 validated against independently computed known vectors (`OggS` → `0x5fb0a94f`, `{1..8}` → `0x7d0f3681`).
- **Phase 21 — Cleanup.** wsl → wsl_v5 upgrade, all 19 pre-existing warnings fixed.
## Test coverage
- 7 FLAC round-trip tests (pre-existing).
- WAV round-trip tests covering all 8 text fields + cover art.
- OGG round-trip tests with CRC32 validation against known vectors.
- All pass with `-race`.
.planning/plans/completed/005-smart-playlists.md
-25
View File
@@ -1,25 +0,0 @@
# 005 — Smart Playlists
> Rule-based dynamic playlists: saved queries that evaluate against the library on demand, with a typeable combobox rule editor, live preview, and queue-snapshot-on-play semantics. AND-only logic with multi-value "is any of" — no nested boolean groups.
**Shipped:** 2026-03-22 · **Slices:** S01-S04 (M002 in GSD)
## What landed
- **S01 — Schema & rule engine.** Migration 9 adds `is_smart` + `smart_rules` (JSON) columns to the existing `playlists` table — extends, doesn't duplicate. `backend/smartplaylist/` builds parameterized WHERE clauses via a hardcoded `fieldMap` whitelist (16 columns from `track_metadata`). 7 text operators (`is`, `is_not`, `contains`, `does_not_contain`, `starts_with`, `ends_with`, `is_any_of`) and 5 numeric (`is`, `is_not`, `greater_than`, `less_than`, `between`). 49 unit tests cover all operators, the genre dual-path, and SQL-injection rejection. Optional result limit + sort (whitelisted fields, `ORDER BY RANDOM()` allowed).
- **S02 — CRUD & sidebar.** sqlc regenerated for the new columns; `IsSmart bool` propagated through `playlist.Summary` to TypeScript models. Smart playlists render with a filter icon and "Smart" badge. New `smart-playlist-details` Lit element follows the `genre-details` pattern; refresh/play/shuffle wired. Drag-drop onto smart playlists blocked.
- **S03 — Rule editor UI.** Reusable `yj-combobox` LitElement (typeable input, ARIA, keyboard nav). `smart-playlist-editor` builds rules row-by-row with field combobox, operator switch (text/numeric auto), value input. Live preview at 300 ms debounce calling `PreviewSmartPlaylist`.
- **S04 — Integration.** Create → navigate → `auto-edit` flow — detail view awaits initial data load (avoids `MaxOpenConns(1)` deadlock) before opening the editor.
## Key decisions retained
- **One playlists table** — `is_smart` flag, not a parallel table. One code path for listing.
- **Saved queries, not saved track lists** — no `playlist_tracks` rows for smart playlists; results evaluated on demand.
- **Snapshot to queue on play.** Queue must be stable during playback, not dynamically re-evaluating.
- **Genre dual-path.** Exact ops (`is`, `is_not`, `is_any_of`) use a subquery against `recording_genres JOIN genres` because the `GROUP_CONCAT(g.name, '||')` column in `track_metadata` can't match individual names. LIKE-based ops use the concatenated column.
- **Wails TS bindings for the 5 smart-playlist methods are manually maintained** (`Service.d.ts` / `Service.js`). No build-time check catches drift if Go signatures change.
## Fragility notes
- **Combobox blur-vs-click race.** The `mousedown` + `preventDefault()` + `requestAnimationFrame` fallback in `combobox.ts` is battle-tested but brittle if anyone restructures option rendering into a separate shadow DOM. Re-verify click-to-select in any combobox refactor.
- **`MaxOpenConns(1)` deadlock.** Holding `*sql.Rows` open while calling functions that query the same `*database.DB` will deadlock. Close rows explicitly before downstream calls; never rely on `defer` here.
.planning/plans/completed/006-play-history.md
-19
View File
@@ -1,19 +0,0 @@
# 006 — Play History & Play Count
> Tracks every natural track completion. Powers smart-playlist rules like "most played", "never played", "not played in 30 days", and exposes play count as an optional track-list column.
**Shipped:** 2026-03-22 · **Slices:** S01-S03 (M003 in GSD)
## What landed
- **S01 — Schema, migration, recording.** Migration 10 adds `play_history` table and denormalized `play_count` + `last_played` columns on `audio_files`; `track_metadata` VIEW recreated to include them. `recordPlay()` pipeline: insert history row → increment count → update timestamp → emit event. `OnPlaybackFinished` hook unlocks the player mutex *before* the DB write (avoids SQLite deadlock).
- **S02 — Smart playlist integration.** `play_count` (numeric) and `days_since_played` (computed) added to the rule-engine field whitelist. Explicit NULL handling for never-played tracks in `days_since_played` conditions.
- **S03 — UI.** Optional play-count column in track lists; `PlayCount` / `LastPlayed` added to TypeScript models; editor dropdown entries for the new fields.
## Key decisions retained
- **Natural finish only** — no percentage threshold, no skip counting. (If you scrub past the end, that doesn't count.)
- **Denormalized `play_count`/`last_played` on `audio_files`** — the alternative was per-query aggregation against `play_history`; the cost wasn't worth it.
- **`days_since_played` is a computed field**, not raw timestamp comparison in the rule. Lets the rule engine speak the user's vocabulary.
- **"Recently played" is a smart playlist**, not a dedicated view. Reuses the existing rule engine instead of adding a parallel query path.
- **Mutex unlock before `recordPlay()`** is mandatory — locking around the DB write under `MaxOpenConns(1)` deadlocks.
.planning/plans/completed/007-explore-browser.md
-24
View File
@@ -1,24 +0,0 @@
# 007 — MusicBrainz/ListenBrainz Explore Browser
> Read-only remote catalog browser: search MB/LB, browse artist pages with ListenBrainz-ranked top tracks and similar artists, view album detail with release-version selection. Rate-limited (1 req/sec), SQLite-cached, with cover art from CAA. Lays the API client + cache foundation that v1.3 Autotagger builds on.
**Shipped:** 2026-03-24 · **Slices:** S01-S04 (M004 in GSD)
## What landed
- **S01 — API clients, cache, explore shell.** `backend/explore/` — MusicBrainz client (wraps `musicbrainzws2`, which provides its own internal rate limiter), ListenBrainz client (custom `RateLimiter` via `golang.org/x/time/rate`, 1 req/sec, burst=1), Cover Art Archive client (URL builders only). All clients follow cache-first pattern: check cache → call API → wrap → cache → return. Migration 11 adds `explore_cache` table (URL key, JSON body, datetime expiry, nullable `mbid` + `entity_type` columns for future autotagger correlation). `idx_explore_cache_mbid` index. TTLs: 24 h for search, 7 d for entity lookups. Sidebar "Explore" entry with globe icon.
- **S02 — Smart search.** Unified `Search()` runs concurrent sub-searches with `WaitGroup` + `Mutex`; frontend renders Top Results / Artists / Albums / Tracks with 300 ms debounce and stale-response discarding via version counter. CAA thumbnails for all result types.
- **S03 — Remote artist page.** `explore-artist-details` (904 LOC). Four independent sections via `Promise.allSettled`: header, top tracks (ListenBrainz `TopRecordingsForArtist` with formatted listen counts), discography (grouped Album/EP/Single/Compilation/Other, newest-first), similar artists (recursive navigation). ListenBrainz Labs `SimilarArtists` failures degrade silently to console.error (it's an experimental endpoint).
- **S04 — Album detail & release versions.** `explore-album-details` (828 LOC). Release fingerprinting (sort tracks by `(discNumber, position)`, join MBIDs with `|`) collapses identical editions into one cluster; track-count min/max labelling shows differences across editions; default selection is the earliest-dated release. Disc separators when any track has `discNumber > 1`.
## Known gap (carried into v1.3 follow-up scope)
- **R032 — offline visual indicator** was in the milestone scope but never implemented. Cache infrastructure works (cached entries served when offline until TTL expires), but `Cache.Get()` doesn't propagate a "from cache" flag to the frontend, and no component renders a "Cached" badge. Functional offline mode works; the UI cue is missing.
## Key decisions retained
- **Wrapper types in `backend/explore/types.go` for Wails bindings**, not raw `musicbrainzws2` types. Avoids ugly TS namespaces and serialization issues with unexported fields.
- **Dual rate limiters** — accept it. The `musicbrainzws2` library has its own that can't easily be disabled or shared; our limiter governs LB + CAA. Both enforce 1 req/sec independently.
- **MBID + entity_type columns in `explore_cache`** — scaffolding for the v1.3 autotagger. No autotagging logic in M004, just storage shape.
- **`CoverArtGroupURL` is a synchronous inline string builder.** Pure deterministic template; making it async via Wails would add per-render round-trips for nothing.
- **Detail components share design tokens via `tokens.css.ts`** but are independent Lit components — no inheritance, no shared base. Data sources differ entirely (local SQLite vs remote API).
.planning/plans/completed/008-autotag-schema-grouping.md
-20
View File
@@ -1,20 +0,0 @@
# 008 — Autotag: Schema & Grouping Foundation
> First v1.3 phase. Lays down the schema + bookkeeping for the MusicBrainz autotagger — no scoring, no UI, no MB calls yet. Subsequent autotag phases (009-012) build on this.
**Shipped:** 2026-04-20 · **Sub-plans:** 008.1-008.4
## What landed
- **008.1 — Migration 31: `audio_files.tag_status`.** Text column with inline `CHECK(tag_status IN (...))` constraint (SQLite `ALTER TABLE ADD COLUMN` supports column constraints, so both fresh and upgraded DBs enforce it). Partial index `idx_audio_files_tag_status_untagged ON audio_files(library_id) WHERE tag_status = 'untagged'` powers the pending badge. Backfill sets `user_confirmed` where the joined recording already has an MBID; everything else defaults to `untagged`.
- **008.2 — Migration 32: `tagging_items` + `group_key`.** New `tagging_items` table (PK `group_key`, `status CHECK IN ('pending','matched','confirmed','skipped')`, two indexes including the partial `WHERE status = 'pending'` for the badge). `audio_files.group_key TEXT NOT NULL DEFAULT ''` with partial index `WHERE group_key != ''`. Column-referencing indexes live in the migration, not the schema file, because `CREATE TABLE IF NOT EXISTS` is a no-op on existing tables and the partial-index predicate would reference a column that hasn't been added yet.
- **008.3 — `backend/autotag.GroupKey` + scan-path integration.** Lower-case hex SHA-1 over `libraryID || 0 || parentDirLower || 0 || albumTrimmed || 0 || discNumber` — intentionally shallow, deeper normalization is scoring territory (009). `saveAudioFile` switched to `CreateAudioFileWithGroupKey` + `UpsertTaggingItemOnTrackAdd`. `updateAudioFileMetadata` rebinds on key change: decrement old group → delete if empty → upsert new group → write new key onto the audio_files row. Migration 32 streams existing rows in batches of 500 and aggregates `tagging_items` at the end, defaulting status to `confirmed` when every track in the group is `user_confirmed`.
- **008.4 — Pending-queue sqlc queries.** `CountPendingTaggingItems`, three `ListPendingTaggingItems...` variants (alphabetical, by-score nulls-last, by-recent) — sqlc has no dynamic ORDER BY so each sort has its own query. `CAST(@param AS INTEGER|TEXT)` hints give typed params (otherwise sqlc emits `interface{}`). `GetTaggingItem` and `ListAudioFilesInTaggingGroup` round out the queue API. `EXPLAIN QUERY PLAN` test asserts the badge query uses `idx_tagging_items_status_pending` so a future schema change that breaks the partial index fails loudly.
## Key decisions retained
- **Hash algorithm in Go, not SQL.** `autotag.GroupKey` is the single source of truth for the key format so we can evolve it without coupling to SQLite functions.
- **SHA-1 over alternatives.** Matches the codebase's existing non-crypto deterministic-key convention. Collision risk at album-group cardinality (millions) is irrelevant.
- **Null-byte separators between hash inputs.** Prevents `a|b` vs `ab|` ambiguity.
- **Per-track integration inside `commitBatch`'s transaction**, not a post-scan callback — keeps `tagging_items` coherent after partial scans.
- **`best_match_release_mbid`, `score`, `last_checked_at` shapes fixed now** even though they stay NULL until 009-010. Avoids schema churn later.
.planning/plans/completed/009-autotag-scoring-engine.md
-32
View File
@@ -1,32 +0,0 @@
# 009 — Autotag: Scoring Engine & MB Orchestration
> Second v1.3 phase. Given an album-group, produce a ranked list of candidate releases with per-track alignment data, using as few MusicBrainz calls as the API-minimization playbook allows. No UI yet — 010 surfaces this to the user.
**Shipped:** 2026-04-21 · **Sub-plans:** types + normalization + distance, local resolver, MB orchestration, ranker + persistence, test corpus
## What landed
- **`backend/autotag/` domain types** — `Candidate`, `TrackAlignment`, `GroupScore`, `LocalTrack`, `CandidateSource` (`local` / `musicbrainz`), `AlignmentStatus` (`matched` / `missing` / `extra` / `mismatched`).
- **`Normalize(s)`** — Unicode NFC → qualifier-suffix strip (`(Remastered 2009)`, `[Bonus Track]`, `(feat. X)`, etc.) → case fold → punctuation drop → whitespace collapse. Comparison-only, not human-readable.
- **Per-track distance** — weighted 60% title similarity (1.0 − Levenshtein/max-len), 30% length delta (linear: ≤1 s = 1.0, ≥30 s = 0.0, neutral 0.5 when either side unknown), 10% track-number match.
- **Greedy alignment** — `AlignTracks` picks the highest-scoring (local, cand) pair iteratively; not Hungarian-optimal but fine at album cardinalities. Emits `matched` / `missing` / `extra` / `mismatched` rows so the review UI can render the diff.
- **Local-first resolver** — `ListLocalReleaseGroupCandidates` sqlc query pre-filters on `rg.mbid != '' AND r.mbid != '' AND rg.name = ? COLLATE NOCASE`, then Go applies full normalization. Candidate track lists only include recordings that themselves have MBIDs (avoids untagged dupes polluting the "canonical" view when the same `(name, artist)` release group is shared across libraries).
- **MB orchestration** — `MBClient` interface (`SearchReleaseGroups`, `BrowseReleases`, `LookupArtist`) hides `explore.MusicBrainzClient`; `backend/explore/autotagclient.go` adapts one to the other. `buildMBQuery` assembles `release:"X" AND arid:<mbid> AND tracks:N` — `arid:` makes the search cache key deterministic, `tracks:N` filters out box-set-style releases. One search per album + one `BrowseReleases` per candidate RG.
- **Release-level ranker** — aggregate track score (70%) + track-count match (15%, zero at ≥50% delta) + meta (15%, avg of year/Official/country bonuses). `Scorer.ScoreGroup` hits local first, runs MB only when no local candidate scores ≥ 0.90.
- **Persistence** — `SetTaggingItemBestMatch` writes `best_match_release_mbid`, `score`, `last_checked_at = CURRENT_TIMESTAMP`, `status = 'matched'`.
## Key decisions retained
- **SHA-1-grade normalization vs. full MB-equivalent.** Qualifier regex handles the common cases (remaster, deluxe, explicit, feat., bonus, etc.) without dragging in a full MB title-parsing library. Edges that bite in real libraries will show up in 010 review UX and can be patched then.
- **`*sqlcgen.Queries` as the DB boundary for autotag**, not `*database.DB`. The `database` package already depends on `autotag.GroupKey` (from 008.3), so reversing the dependency via `database` would create a cycle. Using `sqlcgen` directly is acyclic and keeps `autotag` swappable.
- **Scorer constructor takes `MBClient` as interface, not `*explore.MusicBrainzClient`.** Lets tests inject a stub without spinning up the HTTP + cache layer. The concrete adapter lives in `explore/autotagclient.go`.
- **`localSufficient = 0.90` threshold for skipping MB.** Empirical guess — will get retuned in 011 auto-accept phase when we observe real corpus scores.
- **Weights `60/30/10` for title/length/track-number.** Cribbed from beets' broad intuition; tuned to emphasize title-matching since length data can be unreliable from Vorbis Comments. Tests document the expected floors (e.g. "exact match should score ≥ 0.99") so nudging weights won't silently regress.
- **`release_groups` and `recordings` must both carry MBIDs** for a local candidate. Otherwise untagged dupes of the same album (across libraries) falsely expand the "canonical" track list.
- **UTF-8 em-dashes in SQL comments broke sqlc's string-literal emitter**, truncating generated query strings mid-word. All autotag SQL comments use ASCII punctuation.
## Known follow-ups into 010+
- **`guessArtistMBID` is a stub** returning `""` because `LocalTrack` doesn't currently carry artist MBIDs. 010 should thread artist MBIDs through `ListAudioFilesInTaggingGroup` so the MB resolver can use `arid:` filters.
- **`yearBonus` uses `time.Now().Year()`** as a placeholder target. Should become the earliest release-date hint from the group's tracks once 010 provides it.
- **VA compilation detection threshold** is still open (noted in `.planning/NOTES.md`). The scorer doesn't special-case per-track artist credits differing from album-artist.
.planning/plans/pending/011-autotag-auto-accept.md
-26
View File
@@ -1,26 +0,0 @@
# 011 — Autotag: Auto-Accept & Entry Points
> Fourth v1.3 phase. The strict all-match auto-accept path runs as a background job; the tool is reachable from every place a user expects.
**Status:** pending · **Requirements:** AUTO-01..06 · **Depends on:** 010 (needs the apply pipeline)
## Success criteria
1. An album-group qualifies for auto-accept iff: exact track-count match, every track's normalized title matches, every track's length within ±2s, no cover-art replacement required, no existing-MBID conflicts. Decision uses already-cached candidate data — **no additional MB calls**.
2. Auto-accept job processes all qualifying groups in the queue, emits progress events, is cancellable at any point, honors the shared rate limiter.
3. Right-click on a track / album / artist exposes "Autotag this album" (queues + jumps to review) and "Retag" (flips status to `untagged` and requeues).
4. After a library scan finishes with N new untagged albums, a non-blocking toast appears linking to `/autotag`.
5. Sidebar has an "Autotag" nav entry with a pending-count badge, updates reactively.
6. Pasting a MB release URL into the Paste-URL dialog renders a full diff against the current album with one `LookupRelease` call.
## Sub-plans
- Strict all-match rule + unit tests.
- Auto-accept background job — progress events, cancellation, queue traversal.
- Context menu integrations on track/album/artist views.
- Post-scan toast wiring via the existing scan-complete event.
- Sidebar nav entry + pending-count badge store integration.
## Risk callout
Release selection can pick the wrong edition. The exact-track-count gate prevents most silent misbehavior, but bonus-track editions and remaster reissues with matching track counts are genuine ambiguity. Manual review handles the edge cases — that's why auto-accept is strict by design, and the slider for fuzzy auto-accept is explicitly out of scope.
.planning/plans/pending/012-autotag-settings-polish.md
-31
View File
@@ -1,31 +0,0 @@
# 012 — Autotag: Settings & Polish
> Fifth and final v1.3 phase. Configuration surfaces in the existing settings system; the known sharp edges (rate-limit contention, VA compilations, singleton files) get sanded; the fingerprinting seam is in place for the future.
**Status:** pending · **Requirements:** CFG-01..06 · **Depends on:** 011
## Success criteria
1. Autotag settings panel accessible via the existing templ/HTMX settings UI. Exposes: enable/disable auto-accept, per-library file-write warning reset, default review filter, default sort order.
2. Shared rate limiter distinguishes interactive from background requests. User-initiated MB calls (paste-URL, opening a review, explore browsing) are never blocked behind a running auto-accept job.
3. VA compilation albums (per-track artist credits differ from album-artist) are detected. The auto-accept artist-match rule relaxes for them; the ranker prefers MB releases credited to "Various Artists".
4. Singleton files (`track_count = 1`, no sibling context) use a recording-level match path (`SearchRecordings` with title + artist + length filters). Lower confidence ceiling — never eligible for auto-accept regardless of confidence.
5. `type Identifier interface { Identify(path) ([]Candidate, error) }` exists with `MetadataIdentifier` as the v1 implementation. No fpcalc integration, but the seam is in place for a future `AcoustIDIdentifier`.
6. User-facing quickstart docs exist; CLAUDE.md gets a `backend/autotag/` package description; scoring-function dev notes are committed.
## Sub-plans
- Autotag settings panel (templ + HTMX).
- Rate-limiter priority support.
- VA compilation detection and scoring adjustments.
- Singleton-file match path.
- `Identifier` interface seam with `MetadataIdentifier`.
- Docs — user quickstart + dev notes + CLAUDE.md update.
## Ship criteria for v1.3 overall
- All 29 SCHEMA/MATCH/REVIEW/AUTO/CFG requirements complete.
- All five phases' success criteria verified end-to-end on a real library (10k+ tracks, mixed match quality).
- Auto-accept run against a well-tagged subset produces zero incorrect matches.
- Manual review workflow can process 100 albums in under 30 minutes without mouse use.
- Recap files moved to `.planning/plans/completed/`.
README.md
+65 -93
View File
@@ -1,61 +1,77 @@
# YellowJacket
[![CI](https://github.com/onion-4-dinner/yellowjacket/actions/workflows/ci.yml/badge.svg)](https://github.com/onion-4-dinner/yellowjacket/actions/workflows/ci.yml)
[![Release](https://github.com/onion-4-dinner/yellowjacket/actions/workflows/release.yml/badge.svg)](https://github.com/onion-4-dinner/yellowjacket/actions/workflows/release.yml)
*Music how it was meant to bee.*
Music how it was meant to bee.
YellowJacket is a fast, cross-platform desktop music player for your local
collection. It plays your files, keeps your library tidy, and helps you discover
and organize your music — all in a clean, responsive interface. No accounts, no
streaming, no telemetry: just your music on your machine.
YellowJacket is a cross-platform desktop music player built with Go and web technologies. It focuses on local music library management with a clean, responsive interface.
Runs on **Linux**, **macOS**, and **Windows**.
## Features
### Play your music
- Plays **MP3, FLAC, OGG Vorbis, and WAV**
- Play, pause, seek, and volume control with a mute toggle
- Gapless, glitch-free seeking backed by a read-ahead buffer
- A queue you can add to, reorder, and shuffle, with play-next support
- Shuffle and repeat (off / all / one)
- Picks up right where you left off — remembers your track, position, and volume between sessions
- Media-key and MPRIS support on Linux, so your desktop's playback controls just work
### Keep your library organized
- Point it at your music folders and it scans them automatically
- Reads tags and embedded cover art, and de-duplicates artwork so it isn't stored twice
- Incremental sync — only new or changed files get reprocessed, and deleted files are cleaned up
- Browse by **album**, **artist**, or **genre**, or search across everything
- Mark favorites and see what you've been listening to with play history
- Edit track tags directly when something's off
### Playlists
- Create playlists, drag tracks in, and reorder them
- **Smart playlists** that build themselves from rules (by genre, rating, play count, and more)
- Pin a default playlist and spot duplicate tracks at a glance
### Discover and clean up (powered by MusicBrainz)
- **Explore** — browse artists, releases, and genres from the MusicBrainz catalog, not just what's already in your library
- **Auto-tag** — match your files against MusicBrainz to fill in correct artist, album, and track metadata, with a review step before anything is written
- **Lyrics search** — find a track by a line you remember
## Install
Grab the latest release for your platform:
Download the latest build for your platform from the
[releases page](https://git.ljones.me/yonlu/yellowjacket/releases).
**[Download Latest Release](https://github.com/onion-4-dinner/yellowjacket/releases/latest)**
| Platform | Binary |
|----------|--------|
| Platform | Download |
|----------|----------|
| Linux | `yellowjacket-linux-amd64` |
| macOS | `yellowjacket-darwin-universal.app.zip` (Apple Silicon + Intel) |
| Windows | `yellowjacket-windows-amd64.exe` |
## Features
Prefer to build it yourself? See [Building from source](#building-from-source).
**Playback**
- Play, pause, seek, and volume control with mute toggle
- Support for MP3, FLAC, OGG Vorbis, and WAV
- Queue management with add, remove, reorder, and play-next
- Shuffle mode (Fisher-Yates) and repeat modes (off, all, one)
- Session persistence -- resumes volume, track, and seek position on restart
## Getting started
**Library**
- Concurrent library scanning with automatic metadata extraction
- ID3v2, Vorbis Comments, and other tag format support
- Embedded cover art extraction with content-hash deduplication
- Incremental sync -- only processes new or changed files
- Orphan cleanup for deleted files
1. Launch YellowJacket.
2. Open **Settings** and add the folder(s) where your music lives.
3. Let the initial scan finish — you'll see progress as it works.
4. Browse by album, artist, or genre, queue something up, and press play.
**Interface**
- Album cover grid view and track list view
- Now playing display with cover art
- Resizable sidebar navigation
- Slide-out queue panel
- Context menus for tracks and albums (play, add to queue, play next)
- Settings page for library directory configuration
Your library and settings are stored locally:
## Architecture
| | Linux / macOS | Windows |
|---|---|---|
| Config | `~/.config/yellowjacket/` | `%LOCALAPPDATA%\yellowjacket\config` |
| Library data | `~/.local/share/yellowjacket/` | `%LOCALAPPDATA%\yellowjacket\data` |
YellowJacket uses the [Wails v2](https://wails.io/) framework to bridge a Go backend with a TypeScript/[Lit](https://lit.dev/) frontend running in a native webview.
## Building from source
- **Go backend** -- audio decoding and playback ([beep](https://github.com/gopxl/beep)), library scanning, SQLite database, cover art serving, TOML configuration
- **TypeScript frontend** -- Lit web components, singleton stores with reactive controllers, [Web Awesome](https://www.webawesome.com/) UI components
- **Communication** -- bidirectional event system via Wails runtime; backend is the source of truth
- **Database** -- SQLite (pure-Go driver) with type-safe queries generated by [sqlc](https://sqlc.dev/); MusicBrainz-style data model (artists, artist credits, recordings, release groups)
- **Config page** -- HTMX-based, loads HTML fragments rendered by Go [templ](https://templ.guide/) templates
YellowJacket is built with [Go](https://go.dev/) and a
[Lit](https://lit.dev/)/TypeScript frontend, bridged by the
[Wails](https://wails.io/) framework.
## Development
### Prerequisites
**Prerequisites**
| Tool | Version |
|------|---------|
@@ -64,66 +80,22 @@ YellowJacket uses the [Wails v2](https://wails.io/) framework to bridge a Go bac
| pnpm | 10+ |
| Wails CLI | v2 (`go install github.com/wailsapp/wails/v2/cmd/wails@latest`) |
**Linux system dependencies:**
On Linux, install the system libraries Wails needs:
```bash
sudo apt-get install libasound2-dev libgtk-3-dev libwebkit2gtk-4.1-dev
```
On macOS and Windows, no additional system dependencies are needed. Run `wails doctor` to verify your environment.
macOS and Windows need no extra system packages. Run `wails doctor` to check your
environment.
### Build & Run
**Build**
```bash
make setup # Install git hooks (lefthook)
make dev # Development with hot-reload
make build-dev # Debug build
make build-prod # Production build (obfuscated + UPX compressed)
make generate # Run code generators (sqlc, templ)
make setup # install tooling and git hooks
make dev # run with hot-reload
make build-prod # produce a release binary
```
### Testing
```bash
make test # All tests (race detector, no cache, 2min timeout)
# Run tests manually (build tag required):
go test -tags webkit2_41 ./backend/player/ # Single package
go test -tags webkit2_41 -run TestFunctionName ./backend/player/ # Single test
go test -tags webkit2_41 -v -run TestFunctionName ./backend/player/ # Verbose
```
### Linting
```bash
make lint # golangci-lint (v2 config, strict rules)
```
### Data Locations
| | Linux | macOS | Windows |
|---|---|---|---|
| Config | `~/.config/yellowjacket/` | `~/.config/yellowjacket/` | `%LOCALAPPDATA%\yellowjacket\config` |
| Data/DB | `~/.local/share/yellowjacket/` | `~/.local/share/yellowjacket/` | `%LOCALAPPDATA%\yellowjacket\data` |
### Project Structure
```
backend/ Go backend
player/ Audio playback (beep)
queue/ Queue management, shuffle, repeat
library/ Library scanning, metadata extraction, cover art
metadata/ Audio decoding and tag extraction
database/ SQLite connection, sqlc-generated queries
config/ TOML config, HTTP handler for settings page
events/ Event name constants (mirrored in frontend)
models/ Shared data types (Album, Track, Artist)
system/ OS-specific paths
frontend/src/ TypeScript/Lit frontend
components/ UI components (player, sidebar, track list, cover grid, queue)
store/ Singleton stores and reactive controllers
pages/ Config page (HTMX entry point)
internal/dev/ Build-tag dev/prod detection
test_data/ Audio test fixtures
```
Further development documentation is available in [`docs/dev/`](./docs/dev/overview.md).
More detail for contributors lives in
[`docs/dev/overview.md`](./docs/dev/overview.md) and [`CLAUDE.md`](./CLAUDE.md).
backend/app.go
+54 -10
View File
@@ -247,20 +247,28 @@ func (yj *YellowJacketApp) OnStartup(ctx context.Context) {
RepopulatePlaylists: yj.playlist.RepopulateFromM3U,
ResolvePhantoms: yj.playlist.ResolvePhantomTracksAfterScan,
OnAllScansComplete: func() {
// Index new library artists (blocks until done).
yj.explore.IndexNewArtists()
yj.explore.WaitForIndexIdle()
// Populate local_*_id cross-reference columns on
// explore_index so "is this in my library?" is O(1).
// Fold the local library into the search index: every
// MB-verified owned artist/album/track is upserted and
// flagged in_library, straight from the library tables
// with no API calls. Deep discographies stay lazy.
yj.explore.PopulateLocalCrossReferences()
// Always start the full build — it's incremental and
// will skip tiers that are already fresh. This ensures
// sitewide + similar artist tiers run even if the index
// already has library data.
// Start (or resume) the dump-based index build. Skips
// itself once the one-time import has completed, so this
// is cheap on every startup.
yj.explore.StartIndexBuild()
// Fold in any new incremental listen dumps to keep
// popularity fresh (weekly-gated, background, no API).
// No-op while the full import above is still running.
yj.explore.RefreshListenCounts()
// Refresh the lyric-search FTS index from the just-scanned
// library, then backfill any missing lyrics from LRCLIB in
// the background (bounded, resumable, idempotent).
yj.explore.RebuildLyricsIndex()
yj.explore.BackfillLibraryLyrics()
// Sweep the autotag queue for newly-discovered pending
// items so the user sees match scores ready when they
// next open the review page. The worker is idempotent
@@ -276,6 +284,10 @@ func (yj *YellowJacketApp) OnStartup(ctx context.Context) {
yj.library.SetRemovalHooks(library.RemovalHooks{
StopPlayback: func() { yj.player.UnloadTrack() },
CompactQueue: yj.queue.CompactAfterLibraryRemoval,
// Removal deletes owned content outside a scan, so force the
// gated library-sync steps to re-run on the next launch and
// clear stale in_library flags / orphaned lyric-index rows.
PostRemove: yj.explore.InvalidateLibrarySync,
})
// Register playback finished handler to drive queue auto-advance.
@@ -333,6 +345,21 @@ func (yj *YellowJacketApp) OnStartup(ctx context.Context) {
func (yj *YellowJacketApp) OnBeforeClose(ctx context.Context) bool {
w, h := wailsruntime.WindowGetSize(ctx)
// Guard against a bogus size clobbering a good saved one. During
// teardown / hot-reload the runtime can report a zero or below-
// minimum size; persisting that would shrink the window to the
// minimum on next launch. Keep the previously-saved size instead.
if w < config.MinWidth || h < config.MinHeight {
yj.logger.Warn("OnBeforeClose: ignoring bogus window size",
"width", w,
"height", h,
"kept_width", yj.appConfig.Window.Width,
"kept_height", yj.appConfig.Window.Height,
)
return false
}
yj.logger.Info("OnBeforeClose: saving window state",
"width", w,
"height", h,
@@ -393,7 +420,24 @@ func (yj *YellowJacketApp) OnDomReady(ctx context.Context) {
// index build directly. If scans WERE queued, the
// OnAllScansComplete hook starts it after they finish.
if yj.library.GetScanQueueLength() == 0 && !yj.library.IsScanActive() {
// Keep the search index's in_library flags and owned-entity
// rows in sync. Gated: the library is unchanged here, so
// this only does work on the first launch after an upgrade
// or index wipe — steady-state launches skip the write burst.
yj.explore.PopulateLocalCrossReferencesIfNeeded()
yj.explore.StartIndexBuild()
// Weekly-gated incremental popularity refresh (background,
// no API). No-op while the full import is running.
yj.explore.RefreshListenCounts()
// Same for the lyric-search index. The backfill keeps it in
// sync incrementally, so on an unchanged library the full
// rebuild is redundant and gated out; the backfill still runs
// to fill any remaining gaps.
yj.explore.RebuildLyricsIndexIfNeeded()
yj.explore.BackfillLibraryLyrics()
}
// Kick off the autotag prefetch worker so any unscored
backend/autotag/align.go
+114 -62
View File
@@ -1,5 +1,7 @@
package autotag
import "slices"
// alignTitleFloor is the minimum title similarity required to
// pair a local file with a candidate track at all. Below this,
// even the best available pair is treated as no pair: the local
@@ -10,14 +12,28 @@ package autotag
// happens to share a track number stays in its own group.
const alignTitleFloor = 0.30
// AlignTracks pairs local tracks with candidate tracks using a
// greedy best-score algorithm: repeatedly pick the (local, cand)
// pair with the highest trackDistance that hasn't already been
// claimed. Not optimal (Hungarian would be), but good enough for
// the small cardinalities we see (album tracks, ~10-50) and much
// simpler.
// alignPair carries the per-combination scores AlignTracks computes
// once per (local, candidate) pair.
type alignPair struct {
li int
ci int
score float64
title float64
length float64
}
// AlignTracks pairs local tracks with candidate tracks in two
// passes:
//
// Pairs whose title similarity is below alignTitleFloor are
// 1. Recording-MBID locks: a local track whose RecordingMBID equals
// a candidate track's MBID is the same recording by definition —
// it pairs unconditionally, regardless of how the titles compare.
// 2. Greedy best-score: repeatedly pick the remaining (local, cand)
// pair with the highest track score. Not optimal (Hungarian
// would be), but good enough for the small cardinalities we see
// (album tracks, ~10-50) and much simpler.
//
// Greedy pairs whose title similarity is below alignTitleFloor are
// rejected: the local stays "unmatched" and the candidate slot
// surfaces as "missing". This lets a wrong-track-number-but-
// matching-title file pair correctly while keeping a totally
@@ -25,44 +41,72 @@ const alignTitleFloor = 0.30
//
// Returns one TrackAlignment per local track (status = matched,
// mismatched, or unmatched) plus additional missing alignments
// for candidate tracks with no local file. The caller sums Score
// fields to get a release-level score.
// for candidate tracks with no local file.
func AlignTracks(locals []LocalTrack, cands []CandidateTrack) []TrackAlignment {
type pair struct {
li int
ci int
score float64
title float64
localUsed := make([]bool, len(locals))
candUsed := make([]bool, len(cands))
alignments := make([]TrackAlignment, len(locals))
localMatched := 0
// Pass 1: recording-MBID locks.
for li, local := range locals {
if local.RecordingMBID == "" {
continue
}
for ci, cand := range cands {
if candUsed[ci] || cand.MBID == "" || cand.MBID != local.RecordingMBID {
continue
}
localUsed[li] = true
candUsed[ci] = true
localMatched++
alignments[li] = mkAlignment(li, local, cand, alignPair{
title: titleSimilarity(local.Title, cand.Title),
length: lengthScore(local.LengthMillis, cand.LengthMillis),
}, true)
break
}
}
// Score every (local, cand) combination.
pairs := make([]pair, 0, len(locals)*len(cands))
// Pass 2: greedy best-score over the remaining combinations.
pairs := make([]alignPair, 0, len(locals)*len(cands))
for li, local := range locals {
if localUsed[li] {
continue
}
for ci, cand := range cands {
pairs = append(pairs, pair{
li: li,
ci: ci,
score: trackDistance(local, cand),
title: titleSimilarity(local.Title, cand.Title),
if candUsed[ci] {
continue
}
title := titleSimilarity(local.Title, cand.Title)
length := lengthScore(local.LengthMillis, cand.LengthMillis)
pairs = append(pairs, alignPair{
li: li,
ci: ci,
score: combineTrackScore(title, length, trackNumberOK(local, cand)),
title: title,
length: length,
})
}
}
// Sort descending by score — pick best pairs first. Insertion
// sort keeps the dependency surface zero; N^2 here is fine for
// album-sized inputs.
for i := 1; i < len(pairs); i++ {
for j := i; j > 0 && pairs[j].score > pairs[j-1].score; j-- {
pairs[j], pairs[j-1] = pairs[j-1], pairs[j]
// Sort descending by score — pick best pairs first.
slices.SortStableFunc(pairs, func(a, b alignPair) int {
switch {
case a.score > b.score:
return -1
case a.score < b.score:
return 1
default:
return 0
}
}
localUsed := make([]bool, len(locals))
candUsed := make([]bool, len(cands))
alignments := make([]TrackAlignment, len(locals))
localMatched := 0
})
for _, p := range pairs {
if localMatched == len(locals) {
@@ -84,34 +128,7 @@ func AlignTracks(locals []LocalTrack, cands []CandidateTrack) []TrackAlignment {
localUsed[p.li] = true
candUsed[p.ci] = true
localMatched++
l := locals[p.li]
c := cands[p.ci]
status := AlignmentMatched
if p.title < titleReject {
status = AlignmentMismatched
}
delta := l.LengthMillis - c.LengthMillis
if delta < 0 {
delta = -delta
}
alignments[p.li] = TrackAlignment{
LocalIndex: p.li,
LocalTitle: l.Title,
LocalLengthMillis: l.LengthMillis,
CandidatePosition: c.Position,
CandidateDiscNumber: c.DiscNumber,
CandidateTitle: c.Title,
CandidateMBID: c.MBID,
CandidateLength: c.LengthMillis,
TitleScore: p.title,
LengthDeltaMs: delta,
TrackNumberOK: l.TrackNumber > 0 && l.TrackNumber == c.Position,
Status: status,
}
alignments[p.li] = mkAlignment(p.li, locals[p.li], cands[p.ci], p, false)
}
// Local tracks left unclaimed → folder has them, candidate
@@ -151,3 +168,38 @@ func AlignTracks(locals []LocalTrack, cands []CandidateTrack) []TrackAlignment {
return alignments
}
// mkAlignment builds the matched/mismatched alignment for one
// claimed (local, candidate) pair. idMatch pairs are always
// "matched" — same recording MBID means same recording, however
// the titles are spelled.
func mkAlignment(
li int, l LocalTrack, c CandidateTrack, p alignPair, idMatch bool,
) TrackAlignment {
status := AlignmentMatched
if !idMatch && p.title < titleReject {
status = AlignmentMismatched
}
delta := l.LengthMillis - c.LengthMillis
if delta < 0 {
delta = -delta
}
return TrackAlignment{
LocalIndex: li,
LocalTitle: l.Title,
LocalLengthMillis: l.LengthMillis,
CandidatePosition: c.Position,
CandidateDiscNumber: c.DiscNumber,
CandidateTitle: c.Title,
CandidateMBID: c.MBID,
CandidateLength: c.LengthMillis,
TitleScore: p.title,
LengthScore: p.length,
LengthDeltaMs: delta,
TrackNumberOK: trackNumberOK(l, c),
IDMatch: idMatch,
Status: status,
}
}
backend/autotag/distance.go
+164 -27
View File
@@ -1,5 +1,11 @@
package autotag
import (
"regexp"
"strings"
"unicode"
)
// levenshtein returns the Levenshtein edit distance between a and
// b, operating on runes so multi-byte characters count as one edit.
// Allocates a single O(min(len)) scratch slice.
@@ -63,29 +69,142 @@ func min3(a, b, c int) int {
return c
}
// titleSimilarity returns a score in [0, 1] from normalized edit
// distance. 1.0 means identical after normalization, 0.0 means
// fully dissimilar. Both sides are normalized inside.
func titleSimilarity(a, b string) float64 {
na := Normalize(a)
nb := Normalize(b)
// sdPattern couples a regexp with the weight its removal carries in
// stringDist: when deleting the matched portion from both strings
// shrinks their distance, the recovered distance is re-added at
// `weight` instead of counting fully. Weight 0 makes a difference
// in that portion free (known-cosmetic); higher weights make it
// cheap but not free. Modeled on beets' SD_PATTERNS.
type sdPattern struct {
re *regexp.Regexp
weight float64
}
if na == "" && nb == "" {
return 1.0
}
// sdPatterns are applied in order; earlier patterns claim their
// portion of the string first. Known qualifiers (whitelist) are
// free; generic parenthetical / bracketed content, featured-artist
// credits, leading articles, and part suffixes are de-weighted.
var sdPatterns = []sdPattern{
{qualifierPattern, 0.0},
{dashQualifierPattern, 0.0},
{regexp.MustCompile(`^the `), 0.1},
{regexp.MustCompile(`\b(featuring|feat\.?|ft\.?)[ :].*$`), 0.1},
{regexp.MustCompile(`\(.*?\)`), 0.3},
{regexp.MustCompile(`\[.*?\]`), 0.3},
{regexp.MustCompile(`(, )?\b(pt\.|part) .+$`), 0.2},
}
longest := len(na)
if len(nb) > longest {
longest = len(nb)
}
// sdEndWords are articles that user tags sometimes rotate to the
// end with a comma: "Beatles, The" ≡ "The Beatles".
var sdEndWords = []string{"the", "a", "an"}
if longest == 0 {
// stringDistBasic is the normalized edit distance between the two
// strings reduced to lowercase alphanumerics, in [0, 1]. Inputs
// are assumed to be lowercased and ASCII-folded already.
func stringDistBasic(a, b string) float64 {
a = alnumOnly(a)
b = alnumOnly(b)
if a == "" && b == "" {
return 0.0
}
dist := levenshtein(na, nb)
longest := max(len(a), len(b))
return 1.0 - float64(dist)/float64(longest)
return float64(levenshtein(a, b)) / float64(longest)
}
// alnumOnly strips everything but letters and digits.
func alnumOnly(s string) string {
var b strings.Builder
b.Grow(len(s))
for _, r := range s {
if unicode.IsLetter(r) || unicode.IsDigit(r) {
b.WriteRune(r)
}
}
return b.String()
}
// stringDist returns an "intuitive" distance between two titles or
// artist credits, in [0, 1]. It is a normalized edit distance with
// tweaks reflecting how music metadata actually differs (ported
// from beets' string_dist):
//
// - accents transliterated, case ignored
// - "X, The" rotated back to "The X" (same for "A"/"An")
// - "&" ≡ "and"
// - known qualifier suffixes ("(Remastered 2009)", "- Radio Edit")
// are free; unknown parenthesized/bracketed content, featured-
// artist credits, leading articles, and "Part N" suffixes are
// de-weighted rather than counting as full edits
func stringDist(a, b string) float64 {
a = strings.ToLower(asciiFold(a))
b = strings.ToLower(asciiFold(b))
a = rotateEndWord(a)
b = rotateEndWord(b)
a = strings.ReplaceAll(a, "&", " and ")
b = strings.ReplaceAll(b, "&", " and ")
base := stringDistBasic(a, b)
penalty := 0.0
for _, p := range sdPatterns {
ca := p.re.ReplaceAllString(a, "")
cb := p.re.ReplaceAllString(b, "")
if ca == a && cb == b {
continue
}
// The pattern was present: measure how much of the distance
// it accounted for and re-add that share at reduced weight.
caseDist := stringDistBasic(ca, cb)
delta := base - caseDist
if delta <= 0 {
continue
}
a, b = ca, cb
base = caseDist
penalty += p.weight * delta
}
return base + penalty
}
// rotateEndWord undoes sort-style article rotation: "beatles, the"
// → "the beatles". Input must be lowercased.
func rotateEndWord(s string) string {
for _, w := range sdEndWords {
suffix := ", " + w
if strings.HasSuffix(s, suffix) {
return w + " " + s[:len(s)-len(suffix)]
}
}
return s
}
// titleSimilarity returns a score in [0, 1] from stringDist. 1.0
// means identical after normalization, 0.0 means fully dissimilar.
func titleSimilarity(a, b string) float64 {
if strings.TrimSpace(a) == "" && strings.TrimSpace(b) == "" {
return 1.0
}
sim := 1.0 - stringDist(a, b)
if sim < 0 {
return 0.0
}
return sim
}
// Scoring weights for the per-track distance function. Local reads
@@ -98,13 +217,17 @@ const (
// Length deltas at or below lengthExactMs score 1.0 — matches
// the frontend's "subtle drift" threshold so anything the UI
// hides also doesn't count against the score. Past the
// threshold the penalty scales with delta / candidateMs (i.e.
// percentage of candidate-track length): a 5 s delta on a
// hides also doesn't count against the score. MB recording
// lengths routinely differ from file durations by a few seconds
// (encoder padding, different masters), so the grace band is
// deliberately wider than perceptual accuracy; Picard tolerates
// up to 30 s linearly and beets grants a flat 10 s grace. Past
// the threshold the penalty scales with delta / candidateMs
// (i.e. percentage of candidate-track length): a 5 s delta on a
// 4 min track is small, the same delta on a 30 s interlude is
// huge. At lengthFullyWrongPct of candidate length the score
// hits zero; beyond that it stays clamped to zero.
lengthExactMs int64 = 2000
lengthExactMs int64 = 5000
lengthFullyWrongPct float64 = 0.20
// A title below titleReject has too little signal for this
@@ -139,17 +262,31 @@ func lengthScore(localMs, candidateMs int64) float64 {
return 1.0 - pct/lengthFullyWrongPct
}
// trackDistance scores how well one local track aligns with one
// candidate track. Higher is better. Caller decides what to do
// with the result — this function has no threshold.
func trackDistance(local LocalTrack, cand CandidateTrack) float64 {
title := titleSimilarity(local.Title, cand.Title)
length := lengthScore(local.LengthMillis, cand.LengthMillis)
// trackNumberOK reports whether the local track number agrees with
// the candidate position (0 = unknown, never a match).
func trackNumberOK(local LocalTrack, cand CandidateTrack) bool {
return local.TrackNumber > 0 && local.TrackNumber == cand.Position
}
// combineTrackScore folds the per-track components into one score.
// Split out so AlignTracks can compute the components once per pair
// and still share the exact formula with trackDistance.
func combineTrackScore(title, length float64, numberOK bool) float64 {
var trackOK float64
if local.TrackNumber > 0 && local.TrackNumber == cand.Position {
if numberOK {
trackOK = 1.0
}
return title*weightTitle + length*weightLength + trackOK*weightTrackNumber
}
// trackDistance scores how well one local track aligns with one
// candidate track. Higher is better. Caller decides what to do
// with the result — this function has no threshold.
func trackDistance(local LocalTrack, cand CandidateTrack) float64 {
return combineTrackScore(
titleSimilarity(local.Title, cand.Title),
lengthScore(local.LengthMillis, cand.LengthMillis),
trackNumberOK(local, cand),
)
}
backend/autotag/distance_test.go
+115 -6
View File
@@ -33,10 +33,18 @@ func TestTitleSimilarity(t *testing.T) {
minScore float64
}{
{"Hey Jude", "Hey Jude", 1.00},
{"Hey Jude", "Hey Jude (Remastered 2009)", 1.00}, // qualifier stripped
{"Hey Jude", "Hey Jude (Remastered 2009)", 1.00}, // whitelisted qualifier: free
{"Hey Jude", "Hey Jude - 2015 Remaster", 1.00}, // dash-suffix qualifier: free
{"Hey Jude", "HEY JUDE!", 1.00}, // case + punct
{"Hey Jude", "Hay Jude", 0.85}, // one char off
{"Hey Jude", "Let It Be", 0.00}, // different
{"Beyoncé", "Beyonce", 1.00}, // transliteration
{"Simon & Garfunkel", "Simon and Garfunkel", 1.00},
{"Beatles, The", "The Beatles", 1.00}, // article rotation
{"Hey Jude", "Hay Jude", 0.85}, // one char off
// Unknown parenthetical content is de-weighted, not free —
// still similar, but detectably not identical.
{"Song Title (Special Whatever)", "Song Title", 0.80},
{"Yellow (feat. Somebody)", "Yellow", 0.90}, // feat credit nearly free
{"Hey Jude", "Let It Be", 0.00}, // different
}
for _, tc := range cases {
@@ -66,9 +74,9 @@ func TestLengthScore(t *testing.T) {
want float64
}{
{200000, 200000, 1.0}, // exact
{200000, 200500, 1.0}, // 0.5s — under 2s threshold
{200000, 201900, 1.0}, // 1.9s — under 2s threshold
{200000, 202000, 1.0}, // exactly 2s — still full credit
{200000, 200500, 1.0}, // 0.5s — under the grace band
{200000, 204900, 1.0}, // 4.9s — under the grace band
{200000, 205000, 1.0}, // exactly 5s — still full credit
{0, 200000, 0.5}, // unknown local → neutral
{200000, 0, 0.5}, // unknown candidate → neutral
@@ -130,3 +138,104 @@ func TestTrackDistance(t *testing.T) {
t.Errorf("wrong match = %.2f, want < 0.2", got)
}
}
func TestDominantArtist(t *testing.T) {
t.Parallel()
cases := []struct {
name string
local []LocalTrack
want string
}{
{"empty", nil, ""},
{"all blank", []LocalTrack{{Artist: ""}, {Artist: ""}}, ""},
{
"unanimous",
[]LocalTrack{{Artist: "Radiohead"}, {Artist: "Radiohead"}},
"Radiohead",
},
{
"majority wins over a stray",
[]LocalTrack{{Artist: "Radiohead"}, {Artist: "Radiohead"}, {Artist: "Guest"}},
"Radiohead",
},
{
"blanks ignored, one real value wins",
[]LocalTrack{{Artist: ""}, {Artist: "Bjork"}, {Artist: ""}},
"Bjork",
},
}
for _, tc := range cases {
if got := dominantArtist(tc.local); got != tc.want {
t.Errorf("%s: dominantArtist = %q, want %q", tc.name, got, tc.want)
}
}
}
func TestArtistCreditFit(t *testing.T) {
t.Parallel()
cases := []struct {
name string
local, cnd string
wantMin float64 // when > 0, require >= ; when 0, require <= 0.5 (mismatch)
}{
{"unknown local is neutral", "", "Whoever", 1.0},
{"unknown candidate is neutral", "Whoever", "", 1.0},
{"exact", "The Beatles", "The Beatles", 1.0},
{"case/punct only", "The Beatles", "THE BEATLES!", 1.0},
{"almost-right stays high", "Beyonce", "Beyoncé", 0.85},
{"completely different artist", "The Beatles", "Metallica", 0.0},
}
for _, tc := range cases {
got := artistCreditFit(tc.local, tc.cnd)
if tc.wantMin == 0 {
if got > 0.5 { //nolint:mnd
t.Errorf(
"%s: artistCreditFit(%q,%q) = %.2f, want low",
tc.name,
tc.local,
tc.cnd,
got,
)
}
continue
}
if got < tc.wantMin {
t.Errorf(
"%s: artistCreditFit(%q,%q) = %.2f, want >= %.2f",
tc.name,
tc.local,
tc.cnd,
got,
tc.wantMin,
)
}
}
}
func TestEvidenceFactor(t *testing.T) {
t.Parallel()
cases := []struct {
tracks int
want float64
}{
{0, evidenceFloor}, // no tracks — treated as minimum evidence
{1, evidenceFloor}, // singleton — the harshest case
{2, 0.925}, // halfway between floor and full
{3, 1.0}, // full evidence
{10, 1.0}, // large album — unscaled
}
for _, tc := range cases {
got := evidenceFactor(tc.tracks)
if diff := got - tc.want; diff < -0.001 || diff > 0.001 {
t.Errorf("evidenceFactor(%d) = %.4f, want %.4f", tc.tracks, got, tc.want)
}
}
}
backend/autotag/eval/eval.go
+108
View File
@@ -0,0 +1,108 @@
// Package eval is the autotag candidate-scoring evaluation harness.
// It turns "this match feels wrong" into a number that goes up or
// down, so a scoring change can be validated against a frozen set of
// labelled cases instead of tuned by anecdote.
//
// A case describes a local album-group (the files on disk) plus a set
// of candidate releases, and pins expectations: which candidate must
// rank first, and per-candidate score floors/ceilings. The harness
// is decoupled from the scorer: a caller adapts whatever ranking
// function it wants to measure to the Ranker interface (the autotag
// package wires autotag.RankCandidates to it in eval_harness_test.go).
//
// The point of the ceiling assertions is negative testing: a known
// wrong candidate (same title, different artist) must stay BELOW a
// confidence bar, which is exactly the false-positive class the
// artist term + evidence scaling exist to suppress.
package eval
import (
"encoding/json"
"errors"
"fmt"
"io"
"os"
)
// ErrNoCases is returned when a fixture file contains zero cases.
var ErrNoCases = errors.New("eval: fixture set is empty")
// LocalTrackFixture is one on-disk track, in the shape a case author
// hand-writes. Millisecond durations match the scorer's units.
type LocalTrackFixture struct {
Title string `json:"title"`
Artist string `json:"artist,omitempty"`
Track int `json:"track,omitempty"`
Disc int `json:"disc,omitempty"`
LengthMs int64 `json:"lengthMs,omitempty"`
}
// CandidateTrackFixture is one track inside a candidate release.
type CandidateTrackFixture struct {
Pos int `json:"pos"`
Disc int `json:"disc,omitempty"`
Title string `json:"title"`
LengthMs int64 `json:"lengthMs,omitempty"`
}
// CandidateFixture is one candidate release the scorer must rank.
// Source is "local" or "musicbrainz" (default) — it drives
// evidence scaling, so it matters for singleton cases.
type CandidateFixture struct {
MBID string `json:"mbid"`
Title string `json:"title,omitempty"`
ArtistCredit string `json:"artistCredit,omitempty"`
Status string `json:"status,omitempty"`
Country string `json:"country,omitempty"`
PrimaryType string `json:"primaryType,omitempty"`
Source string `json:"source,omitempty"`
Tracks []CandidateTrackFixture `json:"tracks"`
}
// Case is a single labelled scoring scenario. Every real-world
// mismatch worth guarding against belongs here so it can never
// silently regress.
//
// - AlbumName / AlbumArtist mirror the tagging item's fields —
// leave them empty to keep the album-title / artist terms
// neutral for the case.
// - ExpectTop, when set, is the MBID that must rank first.
// - MaxScore pins per-MBID ceilings (candidate must score <= value).
// - MinScore pins per-MBID floors (candidate must score >= value).
type Case struct {
Note string `json:"note,omitempty"`
AlbumName string `json:"albumName,omitempty"`
AlbumArtist string `json:"albumArtist,omitempty"`
Local []LocalTrackFixture `json:"local"`
Candidates []CandidateFixture `json:"candidates"`
ExpectTop string `json:"expectTop,omitempty"`
MaxScore map[string]float64 `json:"maxScore,omitempty"`
MinScore map[string]float64 `json:"minScore,omitempty"`
}
// LoadCases reads a JSON fixture file from disk.
func LoadCases(path string) ([]Case, error) {
f, err := os.Open(path) //nolint:gosec // path is a test fixture, not user input
if err != nil {
return nil, fmt.Errorf("eval: open cases: %w", err)
}
defer func() { _ = f.Close() }()
return ParseCases(f)
}
// ParseCases decodes a JSON case set from a reader.
func ParseCases(r io.Reader) ([]Case, error) {
var cases []Case
if err := json.NewDecoder(r).Decode(&cases); err != nil {
return nil, fmt.Errorf("eval: decode cases: %w", err)
}
if len(cases) == 0 {
return nil, ErrNoCases
}
return cases, nil
}
backend/autotag/eval/evaluate.go
+123
View File
@@ -0,0 +1,123 @@
package eval
import "fmt"
// ScoredCandidate is one ranked candidate reduced to what the harness
// checks: its MBID and final score. The slice a Ranker returns must
// be ordered best-first.
type ScoredCandidate struct {
MBID string
Score float64
}
// Ranker scores and orders the candidates of a single case. Best
// candidate first. Implemented by adapting the real scorer (see
// eval_harness_test.go), which is why the eval package itself never
// imports autotag.
type Ranker interface {
Rank(c Case) []ScoredCandidate
}
// RankerFunc adapts a plain function to the Ranker interface.
type RankerFunc func(c Case) []ScoredCandidate
// Rank calls the underlying function.
func (f RankerFunc) Rank(c Case) []ScoredCandidate {
return f(c)
}
// CaseResult records how one case fared. Failures is empty when the
// case passed every pinned expectation.
type CaseResult struct {
Case Case
Failures []string
}
// Passed reports whether the case met every expectation.
func (r CaseResult) Passed() bool {
return len(r.Failures) == 0
}
// Report aggregates results across a case set.
type Report struct {
Results []CaseResult
}
// Passed counts cases that met every expectation.
func (r Report) Passed() int {
n := 0
for _, c := range r.Results {
if c.Passed() {
n++
}
}
return n
}
// Accuracy is the fraction of cases that passed, in [0, 1].
func (r Report) Accuracy() float64 {
if len(r.Results) == 0 {
return 0
}
return float64(r.Passed()) / float64(len(r.Results))
}
// Evaluate runs every case through the ranker and checks its pinned
// expectations (ExpectTop, MaxScore ceilings, MinScore floors),
// returning a Report the caller can assert on and print.
func Evaluate(cases []Case, ranker Ranker) Report {
rep := Report{Results: make([]CaseResult, 0, len(cases))}
for _, c := range cases {
rep.Results = append(rep.Results, evaluateCase(c, ranker))
}
return rep
}
func evaluateCase(c Case, ranker Ranker) CaseResult {
ranked := ranker.Rank(c)
res := CaseResult{Case: c}
byMBID := make(map[string]float64, len(ranked))
for _, r := range ranked {
byMBID[r.MBID] = r.Score
}
if c.ExpectTop != "" {
switch {
case len(ranked) == 0:
res.Failures = append(
res.Failures,
"expected top "+c.ExpectTop+" but ranking was empty",
)
case ranked[0].MBID != c.ExpectTop:
res.Failures = append(res.Failures, fmt.Sprintf(
"top = %q (%.3f), want %q (%.3f)",
ranked[0].MBID, ranked[0].Score, c.ExpectTop, byMBID[c.ExpectTop],
))
}
}
for mbid, ceil := range c.MaxScore {
if got, ok := byMBID[mbid]; ok && got > ceil {
res.Failures = append(res.Failures, fmt.Sprintf(
"%s scored %.3f, want <= %.3f", mbid, got, ceil,
))
}
}
for mbid, floor := range c.MinScore {
if got, ok := byMBID[mbid]; ok && got < floor {
res.Failures = append(res.Failures, fmt.Sprintf(
"%s scored %.3f, want >= %.3f", mbid, got, floor,
))
}
}
return res
}
backend/autotag/eval/testdata/scoring_cases.json
Vendored
+225
View File
@@ -0,0 +1,225 @@
[
{
"note": "seed: single with a generic title must reject a same-title, different-artist, wrong-length MB hit and prefer the correct artist. This is the reported 92%-false-positive class.",
"local": [
{ "title": "Intro", "artist": "Real Artist", "track": 1, "lengthMs": 90000 }
],
"candidates": [
{
"mbid": "right",
"artistCredit": "Real Artist",
"status": "Official",
"source": "musicbrainz",
"tracks": [{ "pos": 1, "title": "Intro", "lengthMs": 90000 }]
},
{
"mbid": "wrong-artist",
"artistCredit": "Some Other Band",
"status": "Official",
"source": "musicbrainz",
"tracks": [{ "pos": 1, "title": "Intro", "lengthMs": 240000 }]
}
],
"expectTop": "right",
"maxScore": { "wrong-artist": 0.75 }
},
{
"note": "seed: full-album exact match should still read as a confident, near-perfect match (evidence scaling must not punish real albums).",
"local": [
{ "title": "Song A", "artist": "The Band", "track": 1, "lengthMs": 200000 },
{ "title": "Song B", "artist": "The Band", "track": 2, "lengthMs": 210000 },
{ "title": "Song C", "artist": "The Band", "track": 3, "lengthMs": 195000 },
{ "title": "Song D", "artist": "The Band", "track": 4, "lengthMs": 220000 }
],
"candidates": [
{
"mbid": "album",
"artistCredit": "The Band",
"status": "Official",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "title": "Song A", "lengthMs": 200000 },
{ "pos": 2, "title": "Song B", "lengthMs": 210000 },
{ "pos": 3, "title": "Song C", "lengthMs": 195000 },
{ "pos": 4, "title": "Song D", "lengthMs": 220000 }
]
}
],
"expectTop": "album",
"minScore": { "album": 0.9 }
},
{
"note": "seed: near-right artist (accent/spelling) must stay a strong match; soft artist term, not a gate.",
"local": [
{ "title": "Jolene", "artist": "Beyonce", "track": 1, "lengthMs": 200000 },
{ "title": "Halo", "artist": "Beyonce", "track": 2, "lengthMs": 220000 }
],
"candidates": [
{
"mbid": "accented",
"artistCredit": "Beyoncé",
"status": "Official",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "title": "Jolene", "lengthMs": 200000 },
{ "pos": 2, "title": "Halo", "lengthMs": 220000 }
]
}
],
"expectTop": "accented",
"minScore": { "accented": 0.85 }
},
{
"note": "VA compilation: per-track artists all differ, album-artist says Various Artists. The VA candidate must not be penalised for its artist credit and must read as a strong match.",
"albumName": "Now That's What I Call Music! 60",
"albumArtist": "Various Artists",
"local": [
{ "title": "Song One", "artist": "Artist A", "track": 1, "lengthMs": 200000 },
{ "title": "Song Two", "artist": "Artist B", "track": 2, "lengthMs": 210000 },
{ "title": "Song Three", "artist": "Artist C", "track": 3, "lengthMs": 195000 },
{ "title": "Song Four", "artist": "Artist D", "track": 4, "lengthMs": 205000 }
],
"candidates": [
{
"mbid": "va-comp",
"title": "Now That's What I Call Music! 60",
"artistCredit": "Various Artists",
"status": "Official",
"primaryType": "Compilation",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "title": "Song One", "lengthMs": 200000 },
{ "pos": 2, "title": "Song Two", "lengthMs": 210000 },
{ "pos": 3, "title": "Song Three", "lengthMs": 195000 },
{ "pos": 4, "title": "Song Four", "lengthMs": 205000 }
]
}
],
"expectTop": "va-comp",
"minScore": { "va-comp": 0.85 }
},
{
"note": "same recordings, two release groups: folder album name must pull the studio album above the greatest-hits comp with an identical tracklist.",
"albumName": "The Studio Album",
"albumArtist": "The Band",
"local": [
{ "title": "Song A", "artist": "The Band", "track": 1, "lengthMs": 200000 },
{ "title": "Song B", "artist": "The Band", "track": 2, "lengthMs": 210000 },
{ "title": "Song C", "artist": "The Band", "track": 3, "lengthMs": 195000 }
],
"candidates": [
{
"mbid": "studio",
"title": "The Studio Album",
"artistCredit": "The Band",
"status": "Official",
"primaryType": "Album",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "title": "Song A", "lengthMs": 200000 },
{ "pos": 2, "title": "Song B", "lengthMs": 210000 },
{ "pos": 3, "title": "Song C", "lengthMs": 195000 }
]
},
{
"mbid": "hits-comp",
"title": "Greatest Hits",
"artistCredit": "The Band",
"status": "Official",
"primaryType": "Compilation",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "title": "Song A", "lengthMs": 200000 },
{ "pos": 2, "title": "Song B", "lengthMs": 210000 },
{ "pos": 3, "title": "Song C", "lengthMs": 195000 }
]
}
],
"expectTop": "studio",
"minScore": { "studio": 0.9 }
},
{
"note": "disc 1 of a 2-disc release: per-disc group must not be punished for 'missing' disc 2, and must not lose to a random single-disc release with fewer matching tracks.",
"albumName": "The Double Album",
"albumArtist": "The Band",
"local": [
{ "title": "D1 Track 1", "artist": "The Band", "track": 1, "disc": 1, "lengthMs": 200000 },
{ "title": "D1 Track 2", "artist": "The Band", "track": 2, "disc": 1, "lengthMs": 210000 },
{ "title": "D1 Track 3", "artist": "The Band", "track": 3, "disc": 1, "lengthMs": 195000 }
],
"candidates": [
{
"mbid": "double",
"title": "The Double Album",
"artistCredit": "The Band",
"status": "Official",
"primaryType": "Album",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "disc": 1, "title": "D1 Track 1", "lengthMs": 200000 },
{ "pos": 2, "disc": 1, "title": "D1 Track 2", "lengthMs": 210000 },
{ "pos": 3, "disc": 1, "title": "D1 Track 3", "lengthMs": 195000 },
{ "pos": 1, "disc": 2, "title": "D2 Track 1", "lengthMs": 220000 },
{ "pos": 2, "disc": 2, "title": "D2 Track 2", "lengthMs": 230000 },
{ "pos": 3, "disc": 2, "title": "D2 Track 3", "lengthMs": 240000 }
]
}
],
"expectTop": "double",
"minScore": { "double": 0.9 }
},
{
"note": "streaming-style dash qualifiers: '- 2011 Remaster' suffixes on every local title must not cost title score against the clean MB tracklist.",
"albumName": "Classic Album",
"albumArtist": "The Band",
"local": [
{ "title": "Opener - 2011 Remaster", "artist": "The Band", "track": 1, "lengthMs": 200000 },
{ "title": "Middle Cut - 2011 Remaster", "artist": "The Band", "track": 2, "lengthMs": 210000 },
{ "title": "Closer - 2011 Remaster", "artist": "The Band", "track": 3, "lengthMs": 195000 }
],
"candidates": [
{
"mbid": "clean-titles",
"title": "Classic Album",
"artistCredit": "The Band",
"status": "Official",
"primaryType": "Album",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "title": "Opener", "lengthMs": 200000 },
{ "pos": 2, "title": "Middle Cut", "lengthMs": 210000 },
{ "pos": 3, "title": "Closer", "lengthMs": 195000 }
]
}
],
"expectTop": "clean-titles",
"minScore": { "clean-titles": 0.9 }
},
{
"note": "sorted-artist tag: 'Beatles, The' must match 'The Beatles' releases at full strength (article rotation).",
"albumName": "Abbey Road",
"albumArtist": "Beatles, The",
"local": [
{ "title": "Come Together", "artist": "Beatles, The", "track": 1, "lengthMs": 259000 },
{ "title": "Something", "artist": "Beatles, The", "track": 2, "lengthMs": 182000 },
{ "title": "Octopus's Garden", "artist": "Beatles, The", "track": 3, "lengthMs": 170000 }
],
"candidates": [
{
"mbid": "abbey",
"title": "Abbey Road",
"artistCredit": "The Beatles",
"status": "Official",
"primaryType": "Album",
"source": "musicbrainz",
"tracks": [
{ "pos": 1, "title": "Come Together", "lengthMs": 259000 },
{ "pos": 2, "title": "Something", "lengthMs": 182000 },
{ "pos": 3, "title": "Octopus's Garden", "lengthMs": 170000 }
]
}
],
"expectTop": "abbey",
"minScore": { "abbey": 0.9 }
}
]
backend/autotag/eval_harness_test.go
+102
View File
@@ -0,0 +1,102 @@
package autotag_test
import (
"testing"
"yellowjacket/backend/autotag"
"yellowjacket/backend/autotag/eval"
)
// adaptRanker converts an eval.Case (hand-written fixture shapes) into
// the autotag domain types, runs the real RankCandidates, and maps
// the result back to []eval.ScoredCandidate. This is the one place
// the eval package's decoupled fixtures meet the concrete scorer.
func adaptRanker(c eval.Case) []eval.ScoredCandidate {
locals := make([]autotag.LocalTrack, len(c.Local))
for i, l := range c.Local {
locals[i] = autotag.LocalTrack{
Title: l.Title,
Artist: l.Artist,
TrackNumber: l.Track,
DiscNumber: l.Disc,
LengthMillis: l.LengthMs,
}
}
cands := make([]autotag.Candidate, len(c.Candidates))
for i, cf := range c.Candidates {
tracks := make([]autotag.CandidateTrack, len(cf.Tracks))
for j, tf := range cf.Tracks {
tracks[j] = autotag.CandidateTrack{
Position: tf.Pos,
DiscNumber: tf.Disc,
Title: tf.Title,
LengthMillis: tf.LengthMs,
}
}
cands[i] = autotag.Candidate{
ReleaseMBID: cf.MBID,
Title: cf.Title,
ArtistCredit: cf.ArtistCredit,
Status: cf.Status,
Country: cf.Country,
PrimaryType: cf.PrimaryType,
Source: candidateSource(cf.Source),
Tracks: tracks,
}
}
ranked := autotag.RankCandidates(autotag.Group{
AlbumName: c.AlbumName,
AlbumArtist: c.AlbumArtist,
Tracks: locals,
}, cands)
out := make([]eval.ScoredCandidate, len(ranked))
for i, r := range ranked {
out[i] = eval.ScoredCandidate{MBID: r.ReleaseMBID, Score: r.Score}
}
return out
}
// candidateSource maps the fixture string to the domain type,
// defaulting to MusicBrainz (the interesting, evidence-scaled path).
func candidateSource(s string) autotag.CandidateSource {
if s == string(autotag.SourceLocal) {
return autotag.SourceLocal
}
return autotag.SourceMusicBrainz
}
// TestScoringCorpus runs the frozen labelled corpus through the real
// ranker. Add real-world mismatches to testdata/scoring_cases.json —
// every case that fails here is a scoring regression, and the harness
// prints exactly which expectation broke.
func TestScoringCorpus(t *testing.T) {
t.Parallel()
cases, err := eval.LoadCases("eval/testdata/scoring_cases.json")
if err != nil {
t.Fatalf("load cases: %v", err)
}
report := eval.Evaluate(cases, eval.RankerFunc(adaptRanker))
for _, r := range report.Results {
if r.Passed() {
continue
}
for _, f := range r.Failures {
t.Errorf("case %q: %s", r.Case.Note, f)
}
}
t.Logf(
"scoring corpus: %d/%d cases passed (%.0f%% accuracy)",
report.Passed(), len(report.Results), report.Accuracy()*100,
)
}
backend/autotag/mb.go
+327 -51
View File
@@ -32,6 +32,25 @@ type MBRelease struct {
Tracks []CandidateTrack
}
// MBRecordingHit is the minimal projection of a MusicBrainz recording
// search result — used by the in-app search path (singletons).
type MBRecordingHit struct {
MBID string
Title string
ArtistCredit string
LengthMillis int64
}
// MBReleaseRef is a slim reference to one release a recording appears
// on. The resolver ranks these to pick a representative release and
// then resolves it in full via ResolveOneReleaseMBID.
type MBReleaseRef struct {
MBID string
Title string
Status string
Date string
}
// MBClient is the subset of the explore.MusicBrainzClient surface
// the autotagger depends on. Implementations must be cache-first
// — repeated calls with the same inputs must not repeat network
@@ -42,16 +61,42 @@ type MBClient interface {
query string,
limit int,
) ([]MBReleaseGroupHit, int, error)
SearchRecordings(
ctx context.Context,
query string,
limit int,
) ([]MBRecordingHit, int, error)
LookupRecordingReleases(ctx context.Context, recordingMBID string) ([]MBReleaseRef, error)
BrowseReleases(ctx context.Context, releaseGroupMBID string) ([]MBRelease, error)
LookupRelease(ctx context.Context, releaseMBID string) (MBRelease, error)
LookupReleaseGroup(ctx context.Context, releaseGroupMBID string) (MBReleaseGroupHit, error)
LookupArtist(ctx context.Context, mbid string) (string, error) // returns sort name or name
}
// mbidVariousArtists is the MusicBrainz artist MBID for the special
// "Various Artists" entity — used as an arid filter when a group
// looks like a compilation.
const mbidVariousArtists = "89ad4ac3-39f7-470e-963a-56509c546377"
// cascadeSufficient is the merged-best score at which the cascade
// stops issuing looser queries. "First step with any hits" is the
// wrong stop condition — a strict query can return plausible-but-
// wrong release groups and starve the looser steps of the chance to
// surface the right one. Scoring is free; searches and browses are
// rate-limited network calls, so the cascade pays for another step
// only while the best candidate so far is still mediocre.
const cascadeSufficient = 0.70
// hitBrowseFloor is the minimum title-or-artist similarity a search
// hit needs before the resolver pays a rate-limited BrowseReleases
// call for it. Hits failing both checks are junk from MB's fuzzy
// tokenizer.
const hitBrowseFloor = 0.30
// MBResolver orchestrates MusicBrainz lookups for a tagging group.
// Strategy: normalize the user-provided album/artist first, then
// issue a cascade of progressively looser Lucene queries, stopping
// at the first one that yields enough candidates.
// Strategy: use recording MBIDs already present in local tags when
// possible (exact, cheap); otherwise issue a cascade of
// progressively looser Lucene queries, merging results until a
// candidate scores well enough to stop.
type MBResolver struct {
client MBClient
logger *slog.Logger
@@ -73,27 +118,21 @@ type mbQueryStep struct {
}
// ResolveMB returns MB-sourced candidates for a tagging group.
// Runs a cascade of Lucene queries, returning at the first step
// that produces results. Each search hit fans out to
// BrowseReleases (one per release-group) for track-level data.
func (r *MBResolver) ResolveMB(
ctx context.Context,
albumName, albumArtist string,
trackCount int,
knownArtistMBID string,
) ([]Candidate, error) {
if albumName == "" {
return nil, nil
}
nAlbum := Normalize(albumName)
nArtist := Normalize(albumArtist)
// Runs the Lucene query cascade, accumulating deduplicated
// candidates across steps and stopping once the best merged
// candidate scores at least cascadeSufficient against the group.
func (r *MBResolver) ResolveMB(ctx context.Context, g Group) ([]Candidate, error) {
nAlbum := Normalize(g.AlbumName)
if nAlbum == "" {
return nil, nil
}
steps := buildMBQueryCascade(nAlbum, nArtist, trackCount, knownArtistMBID)
nArtist := Normalize(groupArtist(g))
steps := buildMBQueryCascade(nAlbum, nArtist, len(g.Tracks), vaLikely(g))
seen := make(map[string]bool)
var merged []Candidate
for _, step := range steps {
hits, _, err := r.client.SearchReleaseGroups(ctx, step.query, r.limit)
@@ -106,35 +145,65 @@ func (r *MBResolver) ResolveMB(
continue
}
if len(hits) == 0 {
added := r.fanOutBrowse(ctx, g, hits, step.label, seen, &merged)
r.logger.Debug(
"MB search step done",
"step", step.label, "hits", len(hits), "new_candidates", added,
)
if added == 0 {
continue
}
// Score what we have so far; good enough means the looser
// (noisier, costlier) steps aren't needed.
ranked := RankCandidates(g, merged)
if len(ranked) > 0 && ranked[0].Score >= cascadeSufficient {
r.logger.Info(
"MB cascade stopped — sufficient candidate",
"step", step.label, "score", ranked[0].Score,
)
break
}
}
return merged, nil
}
// fanOutBrowse iterates search hits, fetches each plausible
// release-group's releases, and appends previously-unseen ones to
// merged as Candidates. Returns how many candidates were added.
// Errors on individual browses are logged and skipped.
func (r *MBResolver) fanOutBrowse(
ctx context.Context,
g Group,
hits []MBReleaseGroupHit,
step string,
seen map[string]bool,
merged *[]Candidate,
) int {
added := 0
for _, h := range hits {
if seen["rg:"+h.MBID] {
continue
}
seen["rg:"+h.MBID] = true
// Don't pay a rate-limited browse for a hit that resembles
// neither the folder's album name nor its artist.
if !hitPlausible(g, h) {
r.logger.Debug(
"MB search step empty — trying next",
"step", step.label, "query", step.query,
"skipping implausible search hit",
"title", h.Title, "artist", h.ArtistCredit,
)
continue
}
r.logger.Info(
"MB search step succeeded",
"step", step.label, "hits", len(hits),
)
return r.fanOutBrowse(ctx, hits, step.label), nil
}
return nil, nil
}
// fanOutBrowse iterates search hits, fetches each release-group's
// releases, and returns them as Candidates. Errors on individual
// browses are logged and skipped.
func (r *MBResolver) fanOutBrowse(
ctx context.Context, hits []MBReleaseGroupHit, step string,
) []Candidate {
var out []Candidate
for _, h := range hits {
releases, err := r.client.BrowseReleases(ctx, h.MBID)
if err != nil {
r.logger.Warn(
@@ -146,11 +215,105 @@ func (r *MBResolver) fanOutBrowse(
}
for _, rel := range releases {
out = append(out, mkCandidate(h, rel, step))
if rel.MBID != "" && seen[rel.MBID] {
continue
}
seen[rel.MBID] = true
*merged = append(*merged, mkCandidate(h, rel, step))
added++
}
}
return out
return added
}
// hitPlausible reports whether a release-group search hit is worth
// a BrowseReleases round-trip: its title or artist must bear at
// least a loose resemblance to the group's. Unknown local fields
// never disqualify a hit.
func hitPlausible(g Group, h MBReleaseGroupHit) bool {
if g.AlbumName != "" && h.Title != "" &&
titleSimilarity(g.AlbumName, h.Title) >= hitBrowseFloor {
return true
}
artist := groupArtist(g)
if artist != "" && h.ArtistCredit != "" &&
titleSimilarity(artist, h.ArtistCredit) >= hitBrowseFloor {
return true
}
// Nothing to compare against (or a VA credit): stay permissive.
return g.AlbumName == "" || h.Title == "" || isVAName(h.ArtistCredit)
}
// ResolveByRecordingMBIDs resolves candidates from recording MBIDs
// already present in the local tags — the highest-precision signal
// available, and the reason previously-tagged files should never
// need a fuzzy search. Each recording is looked up, releases are
// counted as votes, and the best-voted release (Official and
// earliest among ties) is resolved in full with provenance "id".
// Returns nil when no recording resolves to any release.
func (r *MBResolver) ResolveByRecordingMBIDs(
ctx context.Context, recordingMBIDs []string,
) ([]Candidate, error) {
votes := make(map[string]int)
refs := make(map[string]MBReleaseRef)
for _, id := range recordingMBIDs {
rels, err := r.client.LookupRecordingReleases(ctx, id)
if err != nil {
r.logger.Warn(
"recording lookup failed — skipping",
"recording_mbid", id, "err", err,
)
continue
}
counted := make(map[string]bool, len(rels))
for _, ref := range rels {
if ref.MBID == "" || counted[ref.MBID] {
continue
}
counted[ref.MBID] = true
votes[ref.MBID]++
refs[ref.MBID] = ref
}
}
if len(votes) == 0 {
return nil, nil
}
// Highest vote count wins; betterRelease breaks ties so the
// pick is deterministic and favours Official + earliest.
var (
bestMBID string
bestVotes int
)
for mbid, n := range votes {
switch {
case n > bestVotes:
bestMBID, bestVotes = mbid, n
case n == bestVotes && betterRelease(refs[mbid], refs[bestMBID]):
bestMBID = mbid
}
}
cand, err := r.ResolveOneReleaseMBID(ctx, bestMBID)
if err != nil {
return nil, fmt.Errorf("resolve voted release %s: %w", bestMBID, err)
}
cand.Provenance = "id"
return []Candidate{cand}, nil
}
// ResolveOneReleaseMBID fetches a single release by MBID and
@@ -208,6 +371,7 @@ func mkCandidate(h MBReleaseGroupHit, rel MBRelease, step string) Candidate {
OriginalDate: h.FirstDate,
Country: rel.Country,
Status: rel.Status,
PrimaryType: h.PrimaryType,
TrackCount: len(rel.Tracks),
Tracks: rel.Tracks,
Source: SourceMusicBrainz,
@@ -215,19 +379,131 @@ func mkCandidate(h MBReleaseGroupHit, rel MBRelease, step string) Candidate {
}
}
// SearchReleaseGroupHits runs a single release-group search from a
// user-supplied album + artist (the in-app "suggest a candidate"
// path). Both fields are normalized and phrase-quoted; artist is
// dropped from the query when empty.
func (r *MBResolver) SearchReleaseGroupHits(
ctx context.Context, album, artist string,
) ([]MBReleaseGroupHit, error) {
query := "release:" + luceneQuote(Normalize(album))
if a := Normalize(artist); a != "" {
query += " AND artist:" + luceneQuote(a)
}
hits, _, err := r.client.SearchReleaseGroups(ctx, query, r.limit)
if err != nil {
return nil, fmt.Errorf("search release groups: %w", err)
}
return hits, nil
}
// SearchRecordingHits runs a single recording search from a user-
// supplied title + artist — the singleton path, where the folder has
// one track and release-group search is too coarse.
func (r *MBResolver) SearchRecordingHits(
ctx context.Context, title, artist string,
) ([]MBRecordingHit, error) {
query := "recording:" + luceneQuote(Normalize(title))
if a := Normalize(artist); a != "" {
query += " AND artist:" + luceneQuote(a)
}
hits, _, err := r.client.SearchRecordings(ctx, query, r.limit)
if err != nil {
return nil, fmt.Errorf("search recordings: %w", err)
}
return hits, nil
}
// ResolveOneRecordingMBID turns a picked recording into a fully-scored
// Candidate by resolving it to a representative release (so the
// existing release-based diff + Apply pipeline works unchanged).
// Picks the release the same way a human would default: prefer an
// Official status, then the earliest date. Provenance is
// "search-recording" so the UI can label where it came from.
func (r *MBResolver) ResolveOneRecordingMBID(
ctx context.Context, recordingMBID string,
) (Candidate, error) {
refs, err := r.client.LookupRecordingReleases(ctx, recordingMBID)
if err != nil {
return Candidate{}, fmt.Errorf("lookup recording releases: %w", err)
}
best := pickRepresentativeRelease(refs)
if best.MBID == "" {
return Candidate{}, fmt.Errorf("%w: %s", errNoReleasesForRecording, recordingMBID)
}
cand, err := r.ResolveOneReleaseMBID(ctx, best.MBID)
if err != nil {
return Candidate{}, err
}
cand.Provenance = "search-recording"
return cand, nil
}
// pickRepresentativeRelease chooses the release most likely to be the
// one the user means: an Official release beats a non-Official one,
// and among equals the earliest date wins (favouring the original
// over later reissues). Returns the zero value for an empty slice.
func pickRepresentativeRelease(refs []MBReleaseRef) MBReleaseRef {
var best MBReleaseRef
for _, ref := range refs {
if best.MBID == "" || betterRelease(ref, best) {
best = ref
}
}
return best
}
// betterRelease reports whether a should be preferred over b.
func betterRelease(a, b MBReleaseRef) bool {
aOfficial := strings.EqualFold(a.Status, "Official")
bOfficial := strings.EqualFold(b.Status, "Official")
if aOfficial != bOfficial {
return aOfficial
}
// Same official-ness: earlier date wins. Empty dates sort last
// so a dated release beats an undated one.
switch {
case a.Date == "":
return false
case b.Date == "":
return true
default:
return a.Date < b.Date
}
}
// errNoReleasesForRecording signals a recording that resolved to zero
// releases — nothing to diff or apply against.
var errNoReleasesForRecording = errors.New("autotag: recording has no releases")
// buildMBQueryCascade returns the Lucene queries to try in order.
// Cascade:
//
// 1. Full: release + arid/artist + tracks:N
// 1. Full: release + artist (or VA arid) + tracks:N
// 2. Drop tracks:N (bonus tracks, live editions, etc.)
// 3. Drop artist entirely (wrong artist tag is common)
// 4. Fuzzy title (unquoted; Lucene does token/prefix match)
//
// Each step is only added when it would differ from the previous.
// VA-likely groups filter on the Various Artists arid instead of an
// artist name — per-track artists on a compilation say nothing
// about the release's artist credit. Each step is only added when
// it would differ from the previous.
func buildMBQueryCascade(
normAlbum, normArtist string,
trackCount int,
artistMBID string,
va bool,
) []mbQueryStep {
var steps []mbQueryStep
@@ -235,8 +511,8 @@ func buildMBQueryCascade(
artistClause := ""
switch {
case artistMBID != "":
artistClause = "arid:" + artistMBID
case va:
artistClause = "arid:" + mbidVariousArtists
case normArtist != "":
artistClause = "artist:" + luceneQuote(normArtist)
}
backend/autotag/mb_test.go
+308 -18
View File
@@ -15,11 +15,14 @@ var errFakeNotFound = errors.New("fake: not found")
// cascade tests orchestrate `searchByStep` so step N returns hits
// only when the resolver has already tried steps < N.
type fakeMBClient struct {
queries []string
searchByStep map[int][]MBReleaseGroupHit
browseByMBID map[string][]MBRelease
lookupRels map[string]MBRelease
lookupRGs map[string]MBReleaseGroupHit
queries []string
searchByStep map[int][]MBReleaseGroupHit
browseByMBID map[string][]MBRelease
browseCalls map[string]int
lookupRels map[string]MBRelease
lookupRGs map[string]MBReleaseGroupHit
searchRecs []MBRecordingHit
recRelsByMBID map[string][]MBReleaseRef
}
func (f *fakeMBClient) SearchReleaseGroups(
@@ -36,6 +39,12 @@ func (f *fakeMBClient) SearchReleaseGroups(
func (f *fakeMBClient) BrowseReleases(
_ context.Context, mbid string,
) ([]MBRelease, error) {
if f.browseCalls == nil {
f.browseCalls = make(map[string]int)
}
f.browseCalls[mbid]++
return f.browseByMBID[mbid], nil
}
@@ -61,8 +70,18 @@ func (f *fakeMBClient) LookupReleaseGroup(
return rg, nil
}
func (f *fakeMBClient) LookupArtist(_ context.Context, _ string) (string, error) {
return "", nil
func (f *fakeMBClient) SearchRecordings(
_ context.Context, query string, _ int,
) ([]MBRecordingHit, int, error) {
f.queries = append(f.queries, query)
return f.searchRecs, len(f.searchRecs), nil
}
func (f *fakeMBClient) LookupRecordingReleases(
_ context.Context, mbid string,
) ([]MBReleaseRef, error) {
return f.recRelsByMBID[mbid], nil
}
func TestBuildMBQueryCascade_StepsOrder(t *testing.T) {
@@ -70,7 +89,7 @@ func TestBuildMBQueryCascade_StepsOrder(t *testing.T) {
// Normalize() is a no-op for these inputs — ASCII, no
// qualifier suffix, no punctuation — so cascade builds directly.
steps := buildMBQueryCascade("abbey road", "the beatles", 17, "")
steps := buildMBQueryCascade("abbey road", "the beatles", 17, false)
if len(steps) < 3 {
t.Fatalf("expected ≥3 cascade steps, got %d", len(steps))
@@ -94,7 +113,7 @@ func TestBuildMBQueryCascade_NormalizesInputs(t *testing.T) {
// Qualifier suffix "(Remastered 2009)" must be stripped by
// the *caller*; verify the query emitter doesn't reintroduce it.
steps := buildMBQueryCascade("abbey road", "", 0, "")
steps := buildMBQueryCascade("abbey road", "", 0, false)
for _, step := range steps {
if strings.Contains(strings.ToLower(step.query), "remastered") {
t.Errorf("step %q should not contain qualifier: %q", step.label, step.query)
@@ -102,26 +121,56 @@ func TestBuildMBQueryCascade_NormalizesInputs(t *testing.T) {
}
}
func TestMBResolver_CascadeStopsOnFirstHit(t *testing.T) {
func TestBuildMBQueryCascade_VariousArtists(t *testing.T) {
t.Parallel()
// VA-likely groups must filter on the Various Artists arid, not
// on whatever plurality artist the compilation's tracks have.
steps := buildMBQueryCascade("now that's music", "artist one", 12, true)
if !strings.Contains(steps[0].query, "arid:"+mbidVariousArtists) {
t.Errorf("VA step 1 should carry the VA arid, got %q", steps[0].query)
}
if strings.Contains(steps[0].query, "artist:") {
t.Errorf("VA step 1 must not carry an artist: clause, got %q", steps[0].query)
}
}
// abbeyRoadGroup is a group whose single track matches the rg1
// fixture release well enough to clear cascadeSufficient.
func abbeyRoadGroup() Group {
return Group{
AlbumName: "Abbey Road",
AlbumArtist: "The Beatles",
Tracks: []LocalTrack{{
Title: "Come Together", TrackNumber: 1, LengthMillis: 259000,
}},
}
}
func TestMBResolver_CascadeStopsWhenSufficient(t *testing.T) {
t.Parallel()
fake := &fakeMBClient{
searchByStep: map[int][]MBReleaseGroupHit{
// step 0 (strict) returns nothing; step 1 (no-track-count) hits.
// step 0 (strict) returns nothing; step 1 (no-track-count)
// hits with a release that scores well against the group.
1: {{MBID: "rg1", Title: "Abbey Road"}},
},
browseByMBID: map[string][]MBRelease{
"rg1": {{MBID: "rel1", Title: "Abbey Road", Tracks: []CandidateTrack{
{Position: 1, Title: "Come Together"},
}}},
"rg1": {{
MBID: "rel1", Title: "Abbey Road", Status: "Official",
Tracks: []CandidateTrack{
{Position: 1, Title: "Come Together", LengthMillis: 259000},
},
}},
},
}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
cands, err := r.ResolveMB(
context.Background(), "Abbey Road", "The Beatles", 17, "",
)
cands, err := r.ResolveMB(context.Background(), abbeyRoadGroup())
if err != nil {
t.Fatalf("ResolveMB: %v", err)
}
@@ -139,13 +188,123 @@ func TestMBResolver_CascadeStopsOnFirstHit(t *testing.T) {
}
}
func TestMBResolver_CascadeContinuesPastMediocreHits(t *testing.T) {
t.Parallel()
// Step 0 returns a same-title release whose track list doesn't
// match the folder at all — plausible enough to browse, but it
// must NOT stop the cascade ("first non-empty step wins" was the
// old, wrong behavior). Step 1 surfaces the real album; both
// candidates come back merged.
fake := &fakeMBClient{
searchByStep: map[int][]MBReleaseGroupHit{
0: {{MBID: "rg-decoy", Title: "Abbey Road"}},
1: {{MBID: "rg-real", Title: "Abbey Road"}},
},
browseByMBID: map[string][]MBRelease{
"rg-decoy": {{
MBID: "rel-decoy", Title: "Abbey Road",
Tracks: []CandidateTrack{
{Position: 1, Title: "Something Else Entirely", LengthMillis: 111000},
{Position: 2, Title: "Not It Either", LengthMillis: 122000},
},
}},
"rg-real": {{
MBID: "rel-real", Title: "Abbey Road", Status: "Official",
Tracks: []CandidateTrack{
{Position: 1, Title: "Come Together", LengthMillis: 259000},
},
}},
},
}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
cands, err := r.ResolveMB(context.Background(), abbeyRoadGroup())
if err != nil {
t.Fatalf("ResolveMB: %v", err)
}
if len(cands) != 2 { //nolint:mnd
t.Fatalf("expected merged candidates from both steps, got %d", len(cands))
}
if len(fake.queries) < 2 { //nolint:mnd
t.Errorf(
"cascade should have continued past the decoy step, got %d queries",
len(fake.queries),
)
}
}
func TestMBResolver_CascadeBrowsesEachReleaseGroupOnce(t *testing.T) {
t.Parallel()
// The same release group surfacing at multiple cascade steps must
// only be browsed (and returned) once.
fake := &fakeMBClient{
searchByStep: map[int][]MBReleaseGroupHit{
0: {{MBID: "rg-dup", Title: "Abbey Road"}},
1: {{MBID: "rg-dup", Title: "Abbey Road"}},
2: {{MBID: "rg-dup", Title: "Abbey Road"}},
3: {{MBID: "rg-dup", Title: "Abbey Road"}},
},
browseByMBID: map[string][]MBRelease{
// Poor track match so the cascade keeps going.
"rg-dup": {{
MBID: "rel-dup", Title: "Abbey Road",
Tracks: []CandidateTrack{
{Position: 1, Title: "Unrelated", LengthMillis: 100000},
},
}},
},
}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
cands, err := r.ResolveMB(context.Background(), abbeyRoadGroup())
if err != nil {
t.Fatalf("ResolveMB: %v", err)
}
if len(cands) != 1 {
t.Fatalf("expected 1 deduplicated candidate, got %d", len(cands))
}
if fake.browseCalls["rg-dup"] != 1 {
t.Errorf("browse calls for rg-dup = %d, want 1", fake.browseCalls["rg-dup"])
}
}
func TestMBResolver_SkipsImplausibleHits(t *testing.T) {
t.Parallel()
// A hit resembling neither the album name nor the artist must
// not cost a browse round-trip.
fake := &fakeMBClient{
searchByStep: map[int][]MBReleaseGroupHit{
0: {{MBID: "rg-junk", Title: "Polka Party Hits", ArtistCredit: "Zzyzx Ensemble"}},
},
}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
if _, err := r.ResolveMB(context.Background(), abbeyRoadGroup()); err != nil {
t.Fatalf("ResolveMB: %v", err)
}
if fake.browseCalls["rg-junk"] != 0 {
t.Errorf("junk hit was browsed %d times, want 0", fake.browseCalls["rg-junk"])
}
}
func TestMBResolver_AbortsOnEmptyAlbumName(t *testing.T) {
t.Parallel()
fake := &fakeMBClient{}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
cands, err := r.ResolveMB(context.Background(), "", "", 0, "")
cands, err := r.ResolveMB(context.Background(), Group{})
if err != nil {
t.Fatalf("ResolveMB: %v", err)
}
@@ -191,3 +350,134 @@ func TestMBResolver_ResolveOneReleaseMBID(t *testing.T) {
t.Errorf("expected 1 track, got %d", len(cand.Tracks))
}
}
func TestResolveByRecordingMBIDs_VotesAcrossRecordings(t *testing.T) {
t.Parallel()
// rec-1 and rec-2 both appear on rel-shared; rec-1 also appears
// on rel-solo. The shared release gets 2 votes and wins.
fake := &fakeMBClient{
recRelsByMBID: map[string][]MBReleaseRef{
"rec-1": {
{MBID: "rel-shared", Status: "Official", Date: "1969"},
{MBID: "rel-solo", Status: "Official", Date: "1968"},
},
"rec-2": {
{MBID: "rel-shared", Status: "Official", Date: "1969"},
},
},
lookupRels: map[string]MBRelease{
"rel-shared": {
MBID: "rel-shared", Title: "Abbey Road",
Tracks: []CandidateTrack{
{Position: 1, Title: "Come Together", MBID: "rec-1"},
{Position: 2, Title: "Something", MBID: "rec-2"},
},
},
},
}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
cands, err := r.ResolveByRecordingMBIDs(
context.Background(), []string{"rec-1", "rec-2"},
)
if err != nil {
t.Fatalf("ResolveByRecordingMBIDs: %v", err)
}
if len(cands) != 1 {
t.Fatalf("expected 1 candidate, got %d", len(cands))
}
if cands[0].ReleaseMBID != "rel-shared" {
t.Errorf("release = %q, want 'rel-shared' (2 votes beats 1)", cands[0].ReleaseMBID)
}
if cands[0].Provenance != "id" {
t.Errorf("provenance = %q, want 'id'", cands[0].Provenance)
}
}
func TestResolveByRecordingMBIDs_NoResults(t *testing.T) {
t.Parallel()
fake := &fakeMBClient{recRelsByMBID: map[string][]MBReleaseRef{}}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
cands, err := r.ResolveByRecordingMBIDs(context.Background(), []string{"rec-x"})
if err != nil {
t.Fatalf("ResolveByRecordingMBIDs: %v", err)
}
if cands != nil {
t.Errorf("expected nil candidates for unknown recordings, got %d", len(cands))
}
}
func TestPickRepresentativeRelease(t *testing.T) {
t.Parallel()
refs := []MBReleaseRef{
{MBID: "promo-1970", Status: "Promotion", Date: "1970"},
{MBID: "official-1975", Status: "Official", Date: "1975"},
{MBID: "official-1969", Status: "Official", Date: "1969"},
{MBID: "undated-official", Status: "Official", Date: ""},
}
// Official beats Promotion; among Official the earliest date wins.
best := pickRepresentativeRelease(refs)
if best.MBID != "official-1969" {
t.Errorf("best = %q, want 'official-1969'", best.MBID)
}
if got := pickRepresentativeRelease(nil); got.MBID != "" {
t.Errorf("empty input = %+v, want zero", got)
}
}
func TestResolveOneRecordingMBID_ResolvesToRepresentativeRelease(t *testing.T) {
t.Parallel()
fake := &fakeMBClient{
recRelsByMBID: map[string][]MBReleaseRef{
"rec-1": {
{MBID: "rel-reissue", Status: "Official", Date: "2011"},
{MBID: "rel-original", Status: "Official", Date: "1979"},
},
},
lookupRels: map[string]MBRelease{
"rel-original": {
MBID: "rel-original", Title: "The Wall",
Tracks: []CandidateTrack{{Position: 1, Title: "Hey You"}},
},
},
}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
cand, err := r.ResolveOneRecordingMBID(context.Background(), "rec-1")
if err != nil {
t.Fatalf("ResolveOneRecordingMBID: %v", err)
}
// Earliest official release chosen and resolved in full.
if cand.ReleaseMBID != "rel-original" {
t.Errorf("release = %q, want 'rel-original'", cand.ReleaseMBID)
}
if cand.Provenance != "search-recording" {
t.Errorf("provenance = %q, want 'search-recording'", cand.Provenance)
}
}
func TestResolveOneRecordingMBID_NoReleases(t *testing.T) {
t.Parallel()
fake := &fakeMBClient{recRelsByMBID: map[string][]MBReleaseRef{}}
r := NewMBResolver(fake, slog.New(slog.DiscardHandler))
if _, err := r.ResolveOneRecordingMBID(context.Background(), "rec-x"); err == nil {
t.Fatal("expected error for recording with no releases")
}
}
backend/autotag/normalize.go
+98 -30
View File
@@ -5,60 +5,128 @@ import (
"strings"
"unicode"
"golang.org/x/text/runes"
"golang.org/x/text/transform"
"golang.org/x/text/unicode/norm"
)
// qualifierAlternation is the regex alternation of parenthesized /
// dash-suffixed qualifiers that MB sometimes adds to titles but user
// tags often omit — e.g. `Remastered 2009`, `Bonus Track`, `feat. X`.
// Shared between the parenthesized and dash-suffix qualifier patterns.
const qualifierAlternation = `remaster(ed)?(\s+\d{4})?|` +
`re-?master(ed)?(\s+\d{4})?|` +
`\d{4}\s+remaster(ed)?|` +
`deluxe(\s+(edition|version))?|` +
`expanded(\s+(edition|version))?|` +
`anniversary(\s+(edition|version))?|` +
`explicit|` +
`clean|` +
`bonus\s+track|` +
`live(\s+at\s+[^\)\]]*)?|` +
`acoustic|` +
`radio\s+edit|` +
`single\s+version|` +
`album\s+version|` +
`original\s+mix|` +
`instrumental|` +
`demo|` +
`mono|` +
`stereo|` +
`feat\.?\s+[^\)\]]*|` +
`featuring\s+[^\)\]]*|` +
`ft\.?\s+[^\)\]]*`
// qualifierPattern matches common parenthesized / bracketed
// qualifiers that MB sometimes adds to titles but user tags often
// omit — e.g. `(Remastered 2009)`, `[Bonus Track]`, `(feat. X)`.
// Case-insensitive. Only strips when the qualifier is at a
// word-boundary to avoid mangling titles like "Untitled (1)".
// qualifiers. Case-insensitive. Only strips when the qualifier is
// at a word-boundary to avoid mangling titles like "Untitled (1)".
var qualifierPattern = regexp.MustCompile(
`(?i)\s*[\(\[]\s*(` +
`remaster(ed)?(\s+\d{4})?|` +
`re-?master(ed)?(\s+\d{4})?|` +
`\d{4}\s+remaster(ed)?|` +
`deluxe(\s+edition)?|` +
`expanded(\s+edition)?|` +
`explicit|` +
`clean|` +
`bonus\s+track|` +
`live(\s+at\s+[^\)\]]*)?|` +
`acoustic|` +
`radio\s+edit|` +
`single\s+version|` +
`album\s+version|` +
`instrumental|` +
`demo|` +
`mono|` +
`stereo|` +
`feat\.?\s+[^\)\]]*|` +
`featuring\s+[^\)\]]*|` +
`ft\.?\s+[^\)\]]*` +
`)\s*[\)\]]`,
`(?i)\s*[\(\[]\s*(` + qualifierAlternation + `)\s*[\)\]]`,
)
// dashQualifierPattern matches the dash-suffix form of the same
// qualifiers — `Song - 2009 Remaster`, `Song – Radio Edit` — which
// streaming-service-derived tags use instead of parentheses.
var dashQualifierPattern = regexp.MustCompile(
`(?i)\s+[-–—]\s+(` + qualifierAlternation + `)\s*$`,
)
// whitespaceCollapse replaces runs of whitespace with a single space.
var whitespaceCollapse = regexp.MustCompile(`\s+`)
// asciiSpecials maps letters that unicode decomposition alone can't
// reduce to ASCII (they aren't combining-mark compositions).
var asciiSpecials = strings.NewReplacer(
"ß", "ss", "ẞ", "SS",
"æ", "ae", "Æ", "AE",
"œ", "oe", "Œ", "OE",
"ø", "o", "Ø", "O",
"đ", "d", "Đ", "D",
"ð", "d", "Ð", "D",
"þ", "th", "Þ", "Th",
"ł", "l", "Ł", "L",
"ı", "i",
)
// asciiFold transliterates accented characters to their closest
// ASCII equivalent ("Beyoncé" → "Beyonce", "Björk" → "Bjork") so
// diacritic differences between user tags and MB data don't count
// as edits. Non-Latin scripts pass through unchanged.
func asciiFold(s string) string {
// Fast path: nothing to fold in pure-ASCII strings.
ascii := true
for i := range len(s) {
if s[i] >= 0x80 {
ascii = false
break
}
}
if ascii {
return s
}
s = asciiSpecials.Replace(s)
// NFKD splits accented letters into base + combining marks;
// dropping the marks (unicode.Mn) leaves the base letter. The
// chain is stateful, so build it per call — it's cheap and this
// keeps concurrent scorers safe.
t := transform.Chain(norm.NFKD, runes.Remove(runes.In(unicode.Mn)), norm.NFC)
folded, _, err := transform.String(t, s)
if err != nil {
return s
}
return folded
}
// Normalize returns a comparison-friendly form of a title or
// artist-credit string:
//
// - NFC unicode composition
// - ASCII transliteration (accents folded)
// - qualifier suffixes stripped (see qualifierPattern)
// - all punctuation dropped
// - "&" replaced with "and"
// - all remaining punctuation dropped
// - case folded (lowercased)
// - whitespace collapsed and trimmed
//
// The result is not intended to be human-readable — only used for
// equality and edit-distance comparisons inside the scorer.
// equality comparisons (local candidate matching) and search query
// building. Fuzzy comparisons go through titleSimilarity, which
// keeps more structure.
func Normalize(s string) string {
if s == "" {
return ""
}
s = norm.NFC.String(s)
s = asciiFold(s)
s = qualifierPattern.ReplaceAllString(s, "")
s = dashQualifierPattern.ReplaceAllString(s, "")
s = strings.ReplaceAll(s, "&", " and ")
var b strings.Builder
backend/autotag/normalize_test.go
+5 -1
View File
@@ -20,12 +20,16 @@ func TestNormalize(t *testing.T) {
"Sgt. Pepper's Lonely Hearts Club Band!",
"sgt peppers lonely hearts club band",
},
"NFC unicode": {"Beyoncé", "beyoncé"},
"accents fold to ascii": {"Beyoncé", "beyonce"},
"eszett folds": {"Motörhead & Björk", "motorhead and bjork"},
"ampersand becomes and": {"Simon & Garfunkel", "simon and garfunkel"},
"remastered qualifier": {"Abbey Road (Remastered 2009)", "abbey road"},
"remaster no year": {"Abbey Road (Remaster)", "abbey road"},
"explicit qualifier": {"Lemonade [Explicit]", "lemonade"},
"feat qualifier": {"Yellow (feat. Coldplay)", "yellow"},
"bonus track qualifier": {"Hey Jude [Bonus Track]", "hey jude"},
"dash suffix qualifier": {"Hey Jude - 2015 Remaster", "hey jude"},
"dash radio edit": {"One More Time - Radio Edit", "one more time"},
"collapse whitespace": {" Abbey Road ", "abbey road"},
"non-ascii digits kept": {"Track 7", "track 7"},
"numbered title not mangled": {"Untitled (1)", "untitled 1"},
backend/autotag/rank.go
+315 -37
View File
@@ -8,23 +8,90 @@ import (
// Release-level scoring weights. Aggregate track score is the
// dominant signal — the release-level signals are tie-breakers
// when the track alignment is roughly comparable.
// when the track alignment is roughly comparable. Weights sum to
// 1.0 so a perfect candidate scores 1.0 before the evidence scale.
const (
weightTrackAggregate = 0.70
weightTrackCountMatch = 0.15
weightReleaseMeta = 0.15 // Official + country averaged
weightTrackAggregate = 0.55
weightArtist = 0.12 // album-artist vs candidate artist-credit
weightAlbumTitle = 0.10 // folder album name vs candidate release title
weightTrackCountMatch = 0.13
weightReleaseMeta = 0.10 // official + country + RG type, averaged
// Country preference: a very mild nudge toward releases from
// the user's locale. Will become a config option in 012.
preferredCountry = "US"
// Evidence scaling: a folder with very few tracks offers little
// corroborating signal, so even a perfect title+length match on
// a single track is inherently less trustworthy than the same
// match across a full album. The final release score is scaled
// by evidenceFactor(localTrackCount): folders at or above
// evidenceFullTracks are unscaled; smaller folders are pulled
// toward evidenceFloor. This is the "harsher on singletons"
// lever — a single can never present as a near-certain match on
// its own, which is also why 012 keeps singletons out of
// auto-accept entirely.
evidenceFloor = 0.85
evidenceFullTracks = 3
)
// vaNames are artist strings that signal "various artists" — used
// both to detect VA-likely folders and to recognize VA candidate
// credits. Mirrors beets' VA_ARTISTS.
var vaNames = map[string]bool{
"various artists": true,
"various": true,
"va": true,
"v a": true, // "V.A." after Normalize
"unknown": true,
}
// isVAName reports whether an artist string reads as "various
// artists". Empty strings are NOT VA — they're unknown, which the
// artist term already treats as neutral.
func isVAName(s string) bool {
return vaNames[Normalize(s)]
}
// vaLikely reports whether a group is probably a various-artists
// compilation: the album-artist tag says so outright, or the
// per-track artists have no consensus (≥2 distinct values, or none
// at all). Mirrors beets' va_likely heuristic.
func vaLikely(g Group) bool {
if isVAName(g.AlbumArtist) {
return true
}
if g.AlbumArtist != "" {
return false
}
distinct := make(map[string]bool, 2) //nolint:mnd
for _, t := range g.Tracks {
if t.Artist == "" {
continue
}
distinct[Normalize(t.Artist)] = true
}
return len(distinct) != 1
}
// ScoreCandidate fills in c.Alignments, c.Score, c.Breakdown, and
// c.TrackCount for a single candidate against the given local
// tracks. The returned Candidate is safe to copy — no shared
// state with the caller's slice.
func ScoreCandidate(local []LocalTrack, c Candidate, localTrackCount int) Candidate {
c.Alignments = AlignTracks(local, c.Tracks)
// c.TrackCount for a single candidate against the given group.
// The returned Candidate is safe to copy — no shared state with
// the caller's slice.
func ScoreCandidate(g Group, c Candidate) Candidate {
local := g.Tracks
// When the group is one disc of a multi-disc candidate, align
// and count against that disc only — a "disc 1 of 2" folder is
// complete for its disc, not half an album.
targets := alignmentTargets(local, c.Tracks)
c.Alignments = AlignTracks(local, targets)
var (
titleSum float64
@@ -38,10 +105,19 @@ func ScoreCandidate(local []LocalTrack, c Candidate, localTrackCount int) Candid
}
counted++
titleSum += a.TitleScore
l := local[a.LocalIndex]
lengthSum += lengthScore(l.LengthMillis, a.CandidateLength)
// A recording-MBID lock is identity, not similarity: the
// title may be garbled in the local tag, but the track IS
// the candidate's track. Count it as a perfect title so a
// confirmed match isn't dragged down by its own typos (the
// UI still shows the textual diff).
if a.IDMatch {
titleSum++
} else {
titleSum += a.TitleScore
}
lengthSum += a.LengthScore
}
titleAvg, lengthAvg := 0.0, 0.0
@@ -63,31 +139,213 @@ func ScoreCandidate(local []LocalTrack, c Candidate, localTrackCount int) Candid
trackAgg := ((titleAvg*weightTitle + lengthAvg*weightLength) / trackWeightSum) * coverage
trackCountScore := trackCountMatch(len(c.Tracks), localTrackCount)
// Release-meta is just official-status + country preference,
// averaged. We used to mix in a year bonus too, but that
// compared candidate years against time.Now() — penalising
// every album that wasn't from this year, regardless of how
// well it matched the local files. See git history.
const metaTerms = 2.0
trackCountScore := trackCountMatch(len(targets), len(local))
meta := (officialBonus(c.Status) + countryBonus(c.Country)) / metaTerms
// Artist fit: compare the folder's artist against the
// candidate's release artist-credit. This is a SOFT signal, not
// a gate — user artist tags are often slightly wrong (misspelled,
// "&" vs "and", missing "feat."), so an almost-right artist still
// matches well while a completely different artist is penalised.
// Critically, artist is otherwise only a *search* filter (see
// buildMBQueryCascade), and that cascade drops the artist clause
// on its looser steps — so without this term a same-title,
// different-artist release scores as if the artist matched.
artistFit := artistCreditFit(groupArtist(g), c.ArtistCredit)
c.Score = trackAgg*weightTrackAggregate +
// Album-title fit: the same soft-signal contract for the release
// title. Without it, a compilation containing the same
// recordings scores as if it WERE the album ("Greatest Hits" vs
// the studio album with an identical tracklist).
albumFit := albumTitleFit(g.AlbumName, c.Title)
// Release-meta: official-status + country preference + release-
// group type, averaged. All mild tie-breakers. (We used to mix
// in a year bonus too, but that compared candidate years against
// time.Now() — see git history.)
const metaTerms = 3.0
meta := (officialBonus(c.Status) + countryBonus(c.Country) + rgTypeBonus(c.PrimaryType)) /
metaTerms
// Evidence scaling applies only to MusicBrainz candidates. A
// local candidate is the *same* release-group already tagged with
// MBIDs in another library — its confidence comes from that
// confirmed tagging, not from thin per-track heuristics, so a
// small-folder local match stays fully trusted (and keeps
// clearing the localSufficient MB-skip short-circuit). MB
// matches, by contrast, are fuzzy search results where a
// single-track folder genuinely offers little corroboration.
evidence := 1.0
if c.Source == SourceMusicBrainz {
evidence = evidenceFactor(len(local))
}
c.Score = (trackAgg*weightTrackAggregate +
artistFit*weightArtist +
albumFit*weightAlbumTitle +
trackCountScore*weightTrackCountMatch +
meta*weightReleaseMeta
meta*weightReleaseMeta) * evidence
c.Breakdown = ScoreBreakdown{
TitleAvg: titleAvg,
LengthAvg: lengthAvg,
ArtistFit: artistFit,
AlbumFit: albumFit,
TrackCountFit: trackCountScore,
ReleaseMeta: meta,
Evidence: evidence,
}
c.TrackCount = len(c.Tracks)
return c
}
// alignmentTargets returns the candidate tracks the local group
// should be aligned against. When every local track sits on the
// same disc D and the candidate spans multiple discs including D,
// only disc D's tracks are targets — the group key is per-disc, so
// a single-disc folder must not be penalised for "missing" the
// candidate's other discs.
func alignmentTargets(local []LocalTrack, cands []CandidateTrack) []CandidateTrack {
disc := uniformDisc(local)
if disc == 0 {
return cands
}
var (
onDisc int
multiDiscs bool
)
for _, c := range cands {
if c.DiscNumber == disc {
onDisc++
} else if c.DiscNumber > 0 {
multiDiscs = true
}
}
if !multiDiscs || onDisc == 0 {
return cands
}
out := make([]CandidateTrack, 0, onDisc)
for _, c := range cands {
if c.DiscNumber == disc {
out = append(out, c)
}
}
return out
}
// uniformDisc returns the disc number shared by every local track,
// or 0 when discs are mixed or unknown.
func uniformDisc(local []LocalTrack) int {
disc := 0
for _, t := range local {
switch {
case t.DiscNumber <= 0:
return 0
case disc == 0:
disc = t.DiscNumber
case t.DiscNumber != disc:
return 0
}
}
return disc
}
// groupArtist returns the artist string to compare candidates
// against: the tagging item's album-artist when it's a real name,
// otherwise the most common per-track artist. Returns "" when
// nothing is known (neutral, no penalty).
func groupArtist(g Group) string {
if g.AlbumArtist != "" && !isVAName(g.AlbumArtist) {
return g.AlbumArtist
}
return dominantArtist(g.Tracks)
}
// dominantArtist returns the most common non-empty per-track artist
// in a local group. Ties resolve to the first-seen value so the
// result is deterministic. Returns "" when no track has an artist,
// which artistCreditFit treats as "unknown, no penalty".
func dominantArtist(local []LocalTrack) string {
counts := make(map[string]int, len(local))
var (
best string
bestCount int
)
for _, t := range local {
if t.Artist == "" {
continue
}
counts[t.Artist]++
if counts[t.Artist] > bestCount {
best = t.Artist
bestCount = counts[t.Artist]
}
}
return best
}
// artistCreditFit scores how well a folder's artist matches a
// candidate's release artist-credit, in [0, 1]. Returns 1.0 (no
// penalty) when either side is unknown or reads as "various
// artists": absence of artist data must not push a candidate down,
// and VA credits are placeholders, not disagreements. Reuses the
// edit-distance similarity so near-right artists stay high.
func artistCreditFit(localArtist, candidateArtist string) float64 {
if localArtist == "" || candidateArtist == "" {
return 1.0
}
if isVAName(localArtist) || isVAName(candidateArtist) {
return 1.0
}
return titleSimilarity(localArtist, candidateArtist)
}
// albumTitleFit scores how well the folder's album name matches the
// candidate's release title, in [0, 1]. Neutral (1.0) when either
// side is unknown — same soft-signal contract as artistCreditFit.
func albumTitleFit(albumName, candidateTitle string) float64 {
if albumName == "" || candidateTitle == "" {
return 1.0
}
return titleSimilarity(albumName, candidateTitle)
}
// evidenceFactor scales the release score down when a folder has too
// few tracks to corroborate the match. Folders at or above
// evidenceFullTracks are unscaled (1.0); a single-track folder is
// pulled to evidenceFloor; two tracks land halfway. See the
// evidence-scaling note on the weight constants.
func evidenceFactor(localTrackCount int) float64 {
if localTrackCount >= evidenceFullTracks {
return 1.0
}
if localTrackCount <= 1 {
return evidenceFloor
}
span := float64(localTrackCount-1) / float64(evidenceFullTracks-1)
return evidenceFloor + (1.0-evidenceFloor)*span
}
// trackCountMatch returns 1.0 when equal, 0.0 when off by >= 50%,
// linear between.
func trackCountMatch(a, b int) float64 {
@@ -104,10 +362,7 @@ func trackCountMatch(a, b int) float64 {
diff = -diff
}
larger := a
if b > larger {
larger = b
}
larger := max(a, b)
frac := float64(diff) / float64(larger)
@@ -124,14 +379,11 @@ func trackCountMatch(a, b int) float64 {
func officialBonus(status string) float64 {
const partial = 0.5
switch strings.ToLower(status) {
case "official":
if strings.EqualFold(status, "official") {
return 1.0
case "":
return partial
default:
return partial
}
return partial
}
// countryBonus gives a mild nudge toward releases from the
@@ -153,6 +405,32 @@ func countryBonus(country string) float64 {
return neutral
}
// rgTypeBonus nudges toward studio albums over compilations and
// live releases when the track evidence is otherwise comparable —
// Picard weights release type heavily for the same reason. The
// nudge is mild: a genuine single folder still matches its Single
// release because track count and alignment dominate. Unknown
// types (including all local candidates) sit near the top so the
// term only separates candidates we positively know differ.
func rgTypeBonus(primaryType string) float64 {
switch strings.ToLower(primaryType) {
case "album":
return 1.0
case "ep":
return 0.9
case "single":
return 0.85
case "":
return 0.85
case "soundtrack":
return 0.7
case "compilation", "live":
return 0.6
default:
return 0.7
}
}
// parseYear pulls the first 4-digit year out of date strings like
// "2009", "2009-05-18", "".
func parseYear(date string) int {
@@ -168,13 +446,13 @@ func parseYear(date string) int {
return y
}
// RankCandidates scores each candidate against the local tracks
// and returns a new slice sorted descending by score. Input slice
// is not modified.
func RankCandidates(local []LocalTrack, candidates []Candidate) []Candidate {
// RankCandidates scores each candidate against the group and
// returns a new slice sorted descending by score. Input slice is
// not modified.
func RankCandidates(g Group, candidates []Candidate) []Candidate {
scored := make([]Candidate, 0, len(candidates))
for _, c := range candidates {
scored = append(scored, ScoreCandidate(local, c, len(local)))
scored = append(scored, ScoreCandidate(g, c))
}
sort.SliceStable(scored, func(i, j int) bool {
backend/autotag/rank_test.go
+302 -5
View File
@@ -37,7 +37,10 @@ func TestRankCandidates_PrefersExactTrackCountMatch(t *testing.T) {
},
}
ranked := autotag.RankCandidates(local, []autotag.Candidate{longer, matching})
ranked := autotag.RankCandidates(
autotag.Group{Tracks: local},
[]autotag.Candidate{longer, matching},
)
if ranked[0].ReleaseMBID != "exact" {
t.Errorf(
"top = %q (score %.2f vs %.2f), want 'exact'",
@@ -69,7 +72,10 @@ func TestRankCandidates_PrefersOfficial(t *testing.T) {
Tracks: tracks,
}
ranked := autotag.RankCandidates(local, []autotag.Candidate{promo, official})
ranked := autotag.RankCandidates(
autotag.Group{Tracks: local},
[]autotag.Candidate{promo, official},
)
if ranked[0].ReleaseMBID != "official" {
t.Errorf("top = %q, want 'official'", ranked[0].ReleaseMBID)
}
@@ -98,7 +104,7 @@ func TestRankCandidates_MultiDisc(t *testing.T) {
},
}
ranked := autotag.RankCandidates(local, []autotag.Candidate{cand})
ranked := autotag.RankCandidates(autotag.Group{Tracks: local}, []autotag.Candidate{cand})
if ranked[0].Score < 0.75 { //nolint:mnd
t.Errorf("multi-disc exact match = %.2f, want >= 0.75", ranked[0].Score)
}
@@ -127,12 +133,123 @@ func TestRankCandidates_VariousArtists(t *testing.T) {
},
}
ranked := autotag.RankCandidates(local, []autotag.Candidate{cand})
ranked := autotag.RankCandidates(autotag.Group{Tracks: local}, []autotag.Candidate{cand})
if ranked[0].Score < 0.75 { //nolint:mnd
t.Errorf("VA-compilation exact track match = %.2f, want >= 0.75", ranked[0].Score)
}
}
func TestRankCandidates_PenalizesWrongArtistSingleton(t *testing.T) {
t.Parallel()
// The reported false positive: a single with a generic title
// matched a same-titled song by a DIFFERENT artist with a very
// different length, at high confidence. Artist was only a search
// filter, never a scoring signal, so nothing pulled the wrong
// candidate down. Now the artist term + singleton evidence
// scaling must keep it well below a confident match.
local := []autotag.LocalTrack{
{Title: "Intro", Artist: "Real Artist", TrackNumber: 1, LengthMillis: 90000},
}
// Same title, wrong artist, and a wildly different length — an MB
// search hit that happens to share a common title.
wrongArtist := autotag.Candidate{
ReleaseMBID: "wrong",
Title: "Intro",
ArtistCredit: "Some Other Band",
Status: "Official",
Source: autotag.SourceMusicBrainz,
Tracks: []autotag.CandidateTrack{
{Position: 1, Title: "Intro", LengthMillis: 240000},
},
}
// The correct release: same title, right artist, right length.
rightArtist := autotag.Candidate{
ReleaseMBID: "right",
Title: "Intro",
ArtistCredit: "Real Artist",
Status: "Official",
Source: autotag.SourceMusicBrainz,
Tracks: []autotag.CandidateTrack{
{Position: 1, Title: "Intro", LengthMillis: 90000},
},
}
ranked := autotag.RankCandidates(
autotag.Group{Tracks: local},
[]autotag.Candidate{wrongArtist, rightArtist},
)
if ranked[0].ReleaseMBID != "right" {
t.Fatalf(
"top = %q (%.2f vs %.2f), want 'right'",
ranked[0].ReleaseMBID, ranked[0].Score, ranked[1].Score,
)
}
// The wrong-artist candidate must not read as a confident match.
var wrongScore float64
for _, c := range ranked {
if c.ReleaseMBID == "wrong" {
wrongScore = c.Score
}
}
if wrongScore >= 0.75 { //nolint:mnd
t.Errorf("wrong-artist singleton scored %.2f, want < 0.75", wrongScore)
}
}
func TestRankCandidates_EvidenceScalingIsSourceAware(t *testing.T) {
t.Parallel()
// A perfect single-track match: identical title, artist, length.
// From MusicBrainz it should be evidence-scaled (thin corroboration
// on one track); from a local library it should NOT be — a local
// candidate is the same release-group already tagged with MBIDs
// elsewhere, so its confidence is not heuristic.
local := []autotag.LocalTrack{
{Title: "Solo", Artist: "Someone", TrackNumber: 1, LengthMillis: 200000},
}
tracks := []autotag.CandidateTrack{
{Position: 1, Title: "Solo", LengthMillis: 200000},
}
mbCand := autotag.Candidate{
ReleaseMBID: "mb", ArtistCredit: "Someone", Status: "Official",
Source: autotag.SourceMusicBrainz, Tracks: tracks,
}
localCand := autotag.Candidate{
ReleaseMBID: "local", ArtistCredit: "Someone", Status: "Official",
Source: autotag.SourceLocal, Tracks: tracks,
}
scoredMB := autotag.RankCandidates(autotag.Group{Tracks: local}, []autotag.Candidate{mbCand})[0]
scoredLocal := autotag.RankCandidates(autotag.Group{Tracks: local}, []autotag.Candidate{localCand})[0]
if scoredMB.Breakdown.Evidence >= 1.0 {
t.Errorf("MB singleton evidence = %.3f, want < 1.0", scoredMB.Breakdown.Evidence)
}
if scoredLocal.Breakdown.Evidence < 1.0 {
t.Errorf(
"local singleton evidence = %.3f, want 1.0 (not scaled)",
scoredLocal.Breakdown.Evidence,
)
}
if scoredLocal.Score <= scoredMB.Score {
t.Errorf(
"local perfect single (%.3f) should outscore the evidence-scaled MB single (%.3f)",
scoredLocal.Score, scoredMB.Score,
)
}
}
func TestRankCandidates_AmbiguousAlbumNames(t *testing.T) {
t.Parallel()
@@ -164,7 +281,10 @@ func TestRankCandidates_AmbiguousAlbumNames(t *testing.T) {
},
}
ranked := autotag.RankCandidates(local, []autotag.Candidate{queen, eagles})
ranked := autotag.RankCandidates(
autotag.Group{Tracks: local},
[]autotag.Candidate{queen, eagles},
)
if ranked[0].ReleaseMBID != "eagles-gh" {
t.Errorf(
"top = %q (%.2f vs %.2f), want 'eagles-gh'",
@@ -172,3 +292,180 @@ func TestRankCandidates_AmbiguousAlbumNames(t *testing.T) {
)
}
}
func TestRankCandidates_AlbumTitleSeparatesCompilation(t *testing.T) {
t.Parallel()
// Identical tracklists: the studio album and a greatest-hits comp
// that contains the same recordings. Track alignment can't
// separate them — the folder's album name must.
local := []autotag.LocalTrack{
{Title: "Song A", Artist: "The Band", TrackNumber: 1, LengthMillis: 200000},
{Title: "Song B", Artist: "The Band", TrackNumber: 2, LengthMillis: 210000},
{Title: "Song C", Artist: "The Band", TrackNumber: 3, LengthMillis: 195000},
}
tracks := []autotag.CandidateTrack{
{Position: 1, Title: "Song A", LengthMillis: 200000},
{Position: 2, Title: "Song B", LengthMillis: 210000},
{Position: 3, Title: "Song C", LengthMillis: 195000},
}
album := autotag.Candidate{
ReleaseMBID: "studio", Title: "The Studio Album",
ArtistCredit: "The Band", Status: "Official", Tracks: tracks,
}
comp := autotag.Candidate{
ReleaseMBID: "comp", Title: "Greatest Hits",
ArtistCredit: "The Band", Status: "Official", Tracks: tracks,
}
g := autotag.Group{
AlbumName: "The Studio Album", AlbumArtist: "The Band", Tracks: local,
}
ranked := autotag.RankCandidates(g, []autotag.Candidate{comp, album})
if ranked[0].ReleaseMBID != "studio" {
t.Errorf(
"top = %q (%.3f vs %.3f), want 'studio' (album-title term)",
ranked[0].ReleaseMBID, ranked[0].Score, ranked[1].Score,
)
}
}
func TestRankCandidates_ReleaseGroupTypeBreaksTies(t *testing.T) {
t.Parallel()
// Same tracks, same title — one RG is an Album, the other a
// Compilation. Type preference should break the tie toward the
// studio album (Picard weights release type for the same reason).
local := []autotag.LocalTrack{
{Title: "Song A", TrackNumber: 1, LengthMillis: 200000},
{Title: "Song B", TrackNumber: 2, LengthMillis: 210000},
{Title: "Song C", TrackNumber: 3, LengthMillis: 195000},
}
tracks := []autotag.CandidateTrack{
{Position: 1, Title: "Song A", LengthMillis: 200000},
{Position: 2, Title: "Song B", LengthMillis: 210000},
{Position: 3, Title: "Song C", LengthMillis: 195000},
}
album := autotag.Candidate{
ReleaseMBID: "album", Title: "X", Status: "Official",
PrimaryType: "Album", Tracks: tracks,
}
comp := autotag.Candidate{
ReleaseMBID: "comp", Title: "X", Status: "Official",
PrimaryType: "Compilation", Tracks: tracks,
}
ranked := autotag.RankCandidates(
autotag.Group{Tracks: local}, []autotag.Candidate{comp, album},
)
if ranked[0].ReleaseMBID != "album" {
t.Errorf(
"top = %q (%.3f vs %.3f), want 'album' (RG type preference)",
ranked[0].ReleaseMBID, ranked[0].Score, ranked[1].Score,
)
}
}
func TestRankCandidates_SingleDiscGroupAgainstMultiDiscRelease(t *testing.T) {
t.Parallel()
// The group key is per-disc, so "disc 1 of 2" folders score
// against multi-disc releases. Alignment and track count must
// compare against disc 1's tracks only — not get punished for
// "missing" all of disc 2.
local := []autotag.LocalTrack{
{Title: "D1T1", DiscNumber: 1, TrackNumber: 1, LengthMillis: 200000},
{Title: "D1T2", DiscNumber: 1, TrackNumber: 2, LengthMillis: 210000},
{Title: "D1T3", DiscNumber: 1, TrackNumber: 3, LengthMillis: 195000},
}
cand := autotag.Candidate{
ReleaseMBID: "2disc",
Status: "Official",
Tracks: []autotag.CandidateTrack{
{Position: 1, DiscNumber: 1, Title: "D1T1", LengthMillis: 200000},
{Position: 2, DiscNumber: 1, Title: "D1T2", LengthMillis: 210000},
{Position: 3, DiscNumber: 1, Title: "D1T3", LengthMillis: 195000},
{Position: 1, DiscNumber: 2, Title: "D2T1", LengthMillis: 220000},
{Position: 2, DiscNumber: 2, Title: "D2T2", LengthMillis: 230000},
{Position: 3, DiscNumber: 2, Title: "D2T3", LengthMillis: 240000},
},
}
ranked := autotag.RankCandidates(
autotag.Group{Tracks: local}, []autotag.Candidate{cand},
)
top := ranked[0]
if top.Score < 0.85 { //nolint:mnd
t.Errorf("disc-1 folder vs 2-disc release = %.3f, want >= 0.85", top.Score)
}
if top.Breakdown.TrackCountFit != 1.0 {
t.Errorf(
"track count fit = %.2f, want 1.0 (counted against disc 1 only)",
top.Breakdown.TrackCountFit,
)
}
// No "missing" rows for disc 2 — the folder is complete for its
// disc.
for _, a := range top.Alignments {
if a.Status == autotag.AlignmentMissing {
t.Errorf("unexpected missing alignment for %q", a.CandidateTitle)
}
}
}
func TestRankCandidates_RecordingMBIDLocksAlignment(t *testing.T) {
t.Parallel()
// The local title is garbled, but its recording MBID matches a
// candidate track — identity beats similarity: the pair must
// align, count as matched, and not drag the title average down.
local := []autotag.LocalTrack{
{Title: "trck 01", RecordingMBID: "rec-a", TrackNumber: 1, LengthMillis: 200000},
{Title: "Song B", TrackNumber: 2, LengthMillis: 210000},
{Title: "Song C", TrackNumber: 3, LengthMillis: 195000},
}
cand := autotag.Candidate{
ReleaseMBID: "rel",
Status: "Official",
Tracks: []autotag.CandidateTrack{
{Position: 1, Title: "Song A", LengthMillis: 200000, MBID: "rec-a"},
{Position: 2, Title: "Song B", LengthMillis: 210000},
{Position: 3, Title: "Song C", LengthMillis: 195000},
},
}
ranked := autotag.RankCandidates(
autotag.Group{Tracks: local}, []autotag.Candidate{cand},
)
top := ranked[0]
var locked *autotag.TrackAlignment
for i := range top.Alignments {
if top.Alignments[i].LocalIndex == 0 {
locked = &top.Alignments[i]
}
}
if locked == nil || locked.Status != autotag.AlignmentMatched || !locked.IDMatch {
t.Fatalf("garbled-title track should be ID-locked matched, got %+v", locked)
}
if top.Breakdown.TitleAvg < 0.99 {
t.Errorf(
"title avg = %.3f, want ~1.0 (ID-locked pair counts as perfect title)",
top.Breakdown.TitleAvg,
)
}
}
backend/autotag/recommend.go
+133
View File
@@ -0,0 +1,133 @@
package autotag
// Recommendation is a qualitative confidence tier for a group's
// ranked candidates — the piece a raw score can't express on its
// own. Modeled on beets' Recommendation enum: the tier starts from
// the top candidate's absolute score and is then CAPPED by defects
// (ambiguity with a different release group, missing/unmatched
// tracks, thin evidence). Auto-accept (plan 011) should require
// RecommendationStrong; the review UI can badge the rest.
type Recommendation string
// Recommendation tiers, weakest to strongest.
const (
RecommendationNone Recommendation = "none"
RecommendationLow Recommendation = "low"
RecommendationMedium Recommendation = "medium"
RecommendationStrong Recommendation = "strong"
)
const (
// Absolute score tiers.
strongScoreThresh = 0.90
mediumScoreThresh = 0.75
// A runner-up from a DIFFERENT release group within this margin
// of the top score makes the match ambiguous — two genuinely
// different albums both fit, so a human should look. Editions
// of the same release group are expected to score nearly
// identically and never count as ambiguity.
ambiguityMargin = 0.05
)
// Recommend derives the confidence tier for a ranked candidate
// list. candidates must already be sorted best-first (the shape
// RankCandidates returns).
func Recommend(g Group, candidates []Candidate) Recommendation {
if len(candidates) == 0 {
return RecommendationNone
}
top := candidates[0]
var rec Recommendation
switch {
case top.Score >= strongScoreThresh:
rec = RecommendationStrong
case top.Score >= mediumScoreThresh:
rec = RecommendationMedium
default:
return RecommendationLow
}
// Cap: a different release group scoring within the ambiguity
// margin means the score alone can't pick between two albums.
if rivalWithinMargin(top, candidates[1:]) {
rec = minRecommendation(rec, RecommendationMedium)
}
// Cap: missing or unmatched tracks mean the alignment itself is
// incomplete, however good the matched tracks look (beets caps
// these penalties at "medium" the same way).
for _, a := range top.Alignments {
if a.Status == AlignmentMissing || a.Status == AlignmentUnmatched {
rec = minRecommendation(rec, RecommendationMedium)
break
}
}
// Cap: tiny folders can't corroborate a match strongly enough
// to act on without review, whatever the arithmetic says.
if len(g.Tracks) < evidenceFullTracks {
rec = minRecommendation(rec, RecommendationMedium)
}
return rec
}
// rivalWithinMargin reports whether any candidate from a different
// release group scores within ambiguityMargin of the top candidate.
func rivalWithinMargin(top Candidate, rest []Candidate) bool {
for _, c := range rest {
if top.Score-c.Score > ambiguityMargin {
// Sorted descending: everything further is farther away.
return false
}
if !sameReleaseGroup(top, c) {
return true
}
}
return false
}
// sameReleaseGroup reports whether two candidates belong to the
// same release group — by MBID when both carry one, by normalized
// title + artist-credit otherwise (local candidates may lack RG
// MBIDs).
func sameReleaseGroup(a, b Candidate) bool {
if a.ReleaseGroupMBID != "" && b.ReleaseGroupMBID != "" {
return a.ReleaseGroupMBID == b.ReleaseGroupMBID
}
return Normalize(a.Title) == Normalize(b.Title) &&
Normalize(a.ArtistCredit) == Normalize(b.ArtistCredit)
}
// recommendationRank orders tiers for min-comparison.
func recommendationRank(r Recommendation) int {
switch r {
case RecommendationNone:
return 0
case RecommendationLow:
return 1
case RecommendationMedium:
return 2
case RecommendationStrong:
return 3
default:
return 0
}
}
// minRecommendation returns the weaker of two tiers.
func minRecommendation(a, b Recommendation) Recommendation {
if recommendationRank(a) <= recommendationRank(b) {
return a
}
return b
}
backend/autotag/recommend_test.go
+130
View File
@@ -0,0 +1,130 @@
package autotag
import "testing"
// mkScoredCandidate builds a minimal candidate with a preset score
// for Recommend tests — Recommend never re-scores, it only reads.
func mkScoredCandidate(rgMBID string, score float64) Candidate {
return Candidate{
ReleaseMBID: "rel-" + rgMBID,
ReleaseGroupMBID: rgMBID,
Score: score,
}
}
func fullGroup() Group {
return Group{Tracks: []LocalTrack{{Title: "A"}, {Title: "B"}, {Title: "C"}}}
}
func TestRecommend_Tiers(t *testing.T) {
t.Parallel()
g := fullGroup()
cases := []struct {
name string
cands []Candidate
want Recommendation
}{
{"no candidates", nil, RecommendationNone},
{"strong", []Candidate{mkScoredCandidate("rg1", 0.95)}, RecommendationStrong},
{"medium", []Candidate{mkScoredCandidate("rg1", 0.80)}, RecommendationMedium},
{"low", []Candidate{mkScoredCandidate("rg1", 0.50)}, RecommendationLow},
}
for _, tc := range cases {
if got := Recommend(g, tc.cands); got != tc.want {
t.Errorf("%s: Recommend = %q, want %q", tc.name, got, tc.want)
}
}
}
func TestRecommend_AmbiguousRivalCapsAtMedium(t *testing.T) {
t.Parallel()
// A different release group within the margin → ambiguous, even
// though the top score alone reads strong.
cands := []Candidate{
mkScoredCandidate("rg1", 0.95),
mkScoredCandidate("rg2", 0.93),
}
if got := Recommend(fullGroup(), cands); got != RecommendationMedium {
t.Errorf("ambiguous rival: Recommend = %q, want medium", got)
}
}
func TestRecommend_SameRGEditionsAreNotAmbiguous(t *testing.T) {
t.Parallel()
// Multiple editions of the SAME release group score nearly
// identically by construction — that's not ambiguity.
cands := []Candidate{
mkScoredCandidate("rg1", 0.95),
mkScoredCandidate("rg1", 0.94),
mkScoredCandidate("rg1", 0.93),
}
if got := Recommend(fullGroup(), cands); got != RecommendationStrong {
t.Errorf("same-RG editions: Recommend = %q, want strong", got)
}
}
func TestRecommend_DistantRivalDoesNotCap(t *testing.T) {
t.Parallel()
cands := []Candidate{
mkScoredCandidate("rg1", 0.95),
mkScoredCandidate("rg2", 0.60),
}
if got := Recommend(fullGroup(), cands); got != RecommendationStrong {
t.Errorf("distant rival: Recommend = %q, want strong", got)
}
}
func TestRecommend_AlignmentDefectsCapAtMedium(t *testing.T) {
t.Parallel()
top := mkScoredCandidate("rg1", 0.95)
top.Alignments = []TrackAlignment{
{Status: AlignmentMatched},
{Status: AlignmentMissing, LocalIndex: -1},
}
if got := Recommend(fullGroup(), []Candidate{top}); got != RecommendationMedium {
t.Errorf("missing track: Recommend = %q, want medium", got)
}
}
func TestRecommend_ThinEvidenceCapsAtMedium(t *testing.T) {
t.Parallel()
// A 2-track folder can't be auto-accept confident however well
// it matches. (Local candidates skip evidence *scaling* but not
// this cap — acting without review still needs corroboration.)
g := Group{Tracks: []LocalTrack{{Title: "A"}, {Title: "B"}}}
cands := []Candidate{mkScoredCandidate("rg1", 0.96)}
if got := Recommend(g, cands); got != RecommendationMedium {
t.Errorf("thin evidence: Recommend = %q, want medium", got)
}
}
func TestRecommend_LocalCandidatesWithoutRGMBIDCompareByTitle(t *testing.T) {
t.Parallel()
// Local candidates may lack RG MBIDs; same title+artist means
// same release group for ambiguity purposes.
a := Candidate{Title: "Album", ArtistCredit: "Band", Score: 0.95}
b := Candidate{Title: "Album", ArtistCredit: "Band", Score: 0.94}
if got := Recommend(fullGroup(), []Candidate{a, b}); got != RecommendationStrong {
t.Errorf("same title/artist locals: Recommend = %q, want strong", got)
}
c := Candidate{Title: "Different Album", ArtistCredit: "Band", Score: 0.94}
if got := Recommend(fullGroup(), []Candidate{a, c}); got != RecommendationMedium {
t.Errorf("different-title rival: Recommend = %q, want medium", got)
}
}
backend/autotag/scorer.go
+177 -64
View File
@@ -40,13 +40,51 @@ func NewScorer(q *sqlcgen.Queries, mb MBClient, logger *slog.Logger) *Scorer {
}
}
// ScoreGroup produces the full GroupScore for one tagging item.
// Always runs both the local resolver (free) and the MB resolver
// (cache-first, so repeats cost nothing). Candidates from both
// sources are merged and ranked — the UI displays them with
// provenance badges so the user can compare.
// localSufficient is the local-candidate score above which a
// local-first caller skips the MusicBrainz round-trip entirely: a
// local candidate at or above this is a strong match (the same album
// already tagged correctly in another library), so the MB cascade
// wouldn't change the top pick and isn't worth a rate-limited network
// call. Interactive scoring ignores this and always consults MB so
// the review UI can show both sources side by side.
const localSufficient = 0.90
// idSufficient is the score at which an ID-resolved candidate (built
// from recording MBIDs already present in the local tags) makes the
// fuzzy search cascade unnecessary — mirrors beets, where a strong
// mb_albumid match returns immediately without a text search.
const idSufficient = 0.90
// maxIDSampleTracks caps how many local recording MBIDs the ID-first
// path looks up — three spread across the folder corroborate a
// release without paying for a lookup per track.
const maxIDSampleTracks = 3
// ScoreGroup produces the full GroupScore for one tagging item,
// always consulting MusicBrainz (when a client is configured) so the
// review UI can display local + MB candidates side by side with
// provenance badges. Use this on interactive paths where the user is
// looking at the result.
func (s *Scorer) ScoreGroup(
ctx context.Context, groupKey string,
) (*GroupScore, error) {
return s.scoreGroup(ctx, groupKey, false)
}
// ScoreGroupLocalFirst is the cheap variant for background work
// (prefetch): it scores local candidates first and skips the
// MusicBrainz cascade when the best local candidate is already a
// strong match (score >= localSufficient). Falls back to the full
// MB-consulting path otherwise, so albums with no strong local match
// still get a real score for the sidebar pill.
func (s *Scorer) ScoreGroupLocalFirst(
ctx context.Context, groupKey string,
) (*GroupScore, error) {
return s.scoreGroup(ctx, groupKey, true)
}
func (s *Scorer) scoreGroup(
ctx context.Context, groupKey string, localFirst bool,
) (*GroupScore, error) {
item, err := s.q.GetTaggingItem(ctx, groupKey)
if err != nil {
@@ -62,36 +100,153 @@ func (s *Scorer) ScoreGroup(
return nil, err
}
g := Group{
AlbumName: item.AlbumName,
AlbumArtist: item.AlbumArtist,
Tracks: locals,
}
localHits, err := s.local.ResolveLocal(ctx, item.AlbumName)
if err != nil {
return nil, err
}
// Local-first short-circuit: pre-score the free local candidates
// and, if one is already a strong match, skip the MB round-trip.
var localCandidates []Candidate
skipMB := false
if localFirst && s.mb != nil {
localCandidates = RankCandidates(g, localHits)
skipMB = len(localCandidates) > 0 && localCandidates[0].Score >= localSufficient
}
var mbHits []Candidate
if s.mb != nil {
mbHits, err = s.mb.ResolveMB(
ctx,
item.AlbumName,
item.AlbumArtist,
len(locals),
guessArtistMBID(locals),
)
if s.mb != nil && !skipMB {
mbHits = s.resolveMBCandidates(ctx, g, groupKey)
}
// Reuse the pre-ranked local list when we skipped MB; otherwise
// rank the merged set.
candidates := localCandidates
if !skipMB {
candidates = RankCandidates(g, append(localHits, mbHits...))
}
return &GroupScore{
GroupKey: groupKey,
AlbumName: item.AlbumName,
AlbumArtist: item.AlbumArtist,
LocalTracks: locals,
Candidates: candidates,
Recommendation: Recommend(g, candidates),
}, nil
}
// resolveMBCandidates gathers MusicBrainz candidates for a group:
// ID-first (recording MBIDs already in the tags), then the search
// cascade when the ID path didn't produce a strong match. Failures
// on either path degrade to fewer candidates, never to an error —
// local candidates must still surface when MB is unreachable.
func (s *Scorer) resolveMBCandidates(
ctx context.Context, g Group, groupKey string,
) []Candidate {
var out []Candidate
if ids := sampleRecordingMBIDs(g.Tracks); len(ids) > 0 {
idCands, err := s.mb.ResolveByRecordingMBIDs(ctx, ids)
if err != nil {
s.log.Warn(
"MB resolve failed — returning local-only candidates",
"group_key", groupKey,
"err", err,
"MB ID-first resolve failed — falling back to search",
"group_key", groupKey, "err", err,
)
}
if len(idCands) > 0 {
ranked := RankCandidates(g, idCands)
if ranked[0].Score >= idSufficient {
s.log.Info(
"MB ID-first match — skipping search cascade",
"group_key", groupKey, "score", ranked[0].Score,
)
return idCands
}
out = idCands
}
}
candidates := RankCandidates(locals, append(localHits, mbHits...))
searchHits, err := s.mb.ResolveMB(ctx, g)
if err != nil {
s.log.Warn(
"MB resolve failed — returning local-only candidates",
"group_key", groupKey, "err", err,
)
return &GroupScore{
GroupKey: groupKey,
LocalTracks: locals,
Candidates: candidates,
}, nil
return out
}
return append(out, dropDuplicateReleases(out, searchHits)...)
}
// dropDuplicateReleases filters from `extra` any candidate whose
// release MBID already appears in `have`.
func dropDuplicateReleases(have, extra []Candidate) []Candidate {
if len(have) == 0 {
return extra
}
seen := make(map[string]bool, len(have))
for _, c := range have {
if c.ReleaseMBID != "" {
seen[c.ReleaseMBID] = true
}
}
out := make([]Candidate, 0, len(extra))
for _, c := range extra {
if c.ReleaseMBID != "" && seen[c.ReleaseMBID] {
continue
}
out = append(out, c)
}
return out
}
// sampleRecordingMBIDs picks up to maxIDSampleTracks distinct
// recording MBIDs spread across the group (first, middle, last) —
// enough to corroborate a release via voting without a lookup per
// track.
func sampleRecordingMBIDs(tracks []LocalTrack) []string {
distinct := make([]string, 0, len(tracks))
seen := make(map[string]bool, len(tracks))
for _, t := range tracks {
if t.RecordingMBID == "" || seen[t.RecordingMBID] {
continue
}
seen[t.RecordingMBID] = true
distinct = append(distinct, t.RecordingMBID)
}
if len(distinct) <= maxIDSampleTracks {
return distinct
}
return []string{
distinct[0],
distinct[len(distinct)/2],
distinct[len(distinct)-1],
}
}
// LocalTracksForGroup exposes the local resolver so callers that
@@ -104,32 +259,6 @@ func (s *Scorer) LocalTracksForGroup(
return s.local.LocalTracksForGroup(ctx, groupKey)
}
// PersistBest writes the top candidate's release MBID and score
// onto the tagging_items row, bumping status to 'matched' when a
// candidate exists. No-op when candidates is empty (keeps the
// current status — likely 'pending').
func (s *Scorer) PersistBest(
ctx context.Context, score *GroupScore,
) error {
if score == nil || len(score.Candidates) == 0 {
return nil
}
top := score.Candidates[0]
mbid := top.ReleaseMBID
if mbid == "" {
mbid = top.ReleaseGroupMBID
}
return s.q.SetTaggingItemBestMatch(ctx, sqlcgen.SetTaggingItemBestMatchParams{
BestMatchReleaseMbid: sql.NullString{String: mbid, Valid: mbid != ""},
Score: sql.NullFloat64{Float64: top.Score, Valid: true},
Status: "matched",
GroupKey: score.GroupKey,
})
}
// PersistScore writes the top candidate's release MBID and score
// onto the tagging_items row WITHOUT touching its status. Use
// this from paths that want the sidebar pill / sort to reflect a
@@ -156,19 +285,3 @@ func (s *Scorer) PersistScore(
GroupKey: score.GroupKey,
})
}
// guessArtistMBID returns the first non-empty recording MBID-
// derived artist hint we can find. Tracks carry recording MBIDs,
// not artist MBIDs, but existing partial tags are often consistent
// enough that any non-empty MBID signals "this album already has
// some MB lineage". A follow-up in 010 can resolve actual artist
// MBIDs via the artists table.
func guessArtistMBID(tracks []LocalTrack) string {
// Phase 009 doesn't wire up per-track artist MBIDs through
// the LocalTrack struct yet — keeping the hook so the MB
// resolver still compiles with the empty hint. Real artist
// MBIDs flow in once 010 wires them into LocalTrack.
_ = tracks
return ""
}
backend/autotag/scorer_test.go
+191 -7
View File
@@ -166,7 +166,179 @@ func TestScorer_LocalHitSurfacesFirst(t *testing.T) {
}
}
func TestScorer_PersistBestWritesMatched(t *testing.T) {
// TestScorer_LocalFirstSkipsMB asserts the background (prefetch)
// scoring path makes zero MusicBrainz calls when a local candidate is
// already a strong match — the whole point of ScoreGroupLocalFirst.
func TestScorer_LocalFirstSkipsMB(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
// A canonical local album (MBIDs present) that the pending group
// matches track-for-track — the local candidate scores >= 0.90.
seed(t, db, seededAlbum{
groupKey: "g-canonical",
albumName: "Good Album",
releaseMBID: "rg-abcd",
tracks: []seededTrack{
{
filePath: "/lib/a.mp3", title: "Song A",
trackNumber: 1, lengthMillis: 200000, recordingMBID: "rec-a",
},
{
filePath: "/lib/b.mp3", title: "Song B",
trackNumber: 2, lengthMillis: 180000, recordingMBID: "rec-b",
},
},
})
seed(t, db, seededAlbum{
groupKey: "g-pending",
albumName: "Good Album",
tracks: []seededTrack{
{filePath: "/other/a.mp3", title: "Song A", trackNumber: 1, lengthMillis: 200000},
{filePath: "/other/b.mp3", title: "Song B", trackNumber: 2, lengthMillis: 180000},
},
})
mbCalls := 0
fakeMB := &countingMBClient{onSearch: func() { mbCalls++ }}
scorer := autotag.NewScorer(db.Queries, fakeMB, slog.New(slog.DiscardHandler))
result, err := scorer.ScoreGroupLocalFirst(context.Background(), "g-pending")
if err != nil {
t.Fatalf("ScoreGroupLocalFirst: %v", err)
}
if mbCalls != 0 {
t.Errorf("made %d MB calls, want 0 (strong local match should skip MB)", mbCalls)
}
if len(result.Candidates) == 0 || result.Candidates[0].ReleaseGroupMBID != "rg-abcd" {
t.Errorf("top candidate = %+v, want local rg-abcd", result.Candidates)
}
}
// TestScorer_IDFirstSkipsSearch asserts that a group whose tracks
// already carry recording MBIDs resolves through the ID-first path
// — the release the recordings vote for is looked up directly and
// the fuzzy search cascade never runs.
func TestScorer_IDFirstSkipsSearch(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
seed(t, db, seededAlbum{
groupKey: "g-tagged",
albumName: "Good Album",
tracks: []seededTrack{
{
filePath: "/lib/a.mp3", title: "Song A",
trackNumber: 1, lengthMillis: 200000, recordingMBID: "rec-a",
},
{
filePath: "/lib/b.mp3", title: "Song B",
trackNumber: 2, lengthMillis: 180000, recordingMBID: "rec-b",
},
},
})
fake := &idFakeClient{
recRels: map[string][]autotag.MBReleaseRef{
"rec-a": {{MBID: "rel-full", Status: "Official", Date: "1999"}},
"rec-b": {{MBID: "rel-full", Status: "Official", Date: "1999"}},
},
releases: map[string]autotag.MBRelease{
"rel-full": {
MBID: "rel-full", Title: "Good Album",
ArtistCredit: "Test Artist", Status: "Official", Country: "US",
Tracks: []autotag.CandidateTrack{
{Position: 1, Title: "Song A", LengthMillis: 200000, MBID: "rec-a"},
{Position: 2, Title: "Song B", LengthMillis: 180000, MBID: "rec-b"},
},
},
},
}
scorer := autotag.NewScorer(db.Queries, fake, slog.New(slog.DiscardHandler))
result, err := scorer.ScoreGroup(context.Background(), "g-tagged")
if err != nil {
t.Fatalf("ScoreGroup: %v", err)
}
if fake.searches != 0 {
t.Errorf("made %d search calls, want 0 (ID-first should skip the cascade)", fake.searches)
}
if len(result.Candidates) == 0 {
t.Fatal("no candidates")
}
top := result.Candidates[0]
if top.ReleaseMBID != "rel-full" || top.Provenance != "id" {
t.Errorf(
"top = %q via %q (%.3f), want 'rel-full' via 'id'",
top.ReleaseMBID, top.Provenance, top.Score,
)
}
}
// idFakeClient serves canned recording→release lookups and counts
// search calls so the ID-first test can assert the cascade stayed
// cold.
type idFakeClient struct {
searches int
recRels map[string][]autotag.MBReleaseRef
releases map[string]autotag.MBRelease
}
func (c *idFakeClient) SearchReleaseGroups(
_ context.Context, _ string, _ int,
) ([]autotag.MBReleaseGroupHit, int, error) {
c.searches++
return nil, 0, nil
}
func (c *idFakeClient) SearchRecordings(
_ context.Context, _ string, _ int,
) ([]autotag.MBRecordingHit, int, error) {
c.searches++
return nil, 0, nil
}
func (c *idFakeClient) LookupRecordingReleases(
_ context.Context, mbid string,
) ([]autotag.MBReleaseRef, error) {
return c.recRels[mbid], nil
}
func (c *idFakeClient) BrowseReleases(
_ context.Context, _ string,
) ([]autotag.MBRelease, error) {
return nil, nil
}
func (c *idFakeClient) LookupRelease(
_ context.Context, mbid string,
) (autotag.MBRelease, error) {
rel, ok := c.releases[mbid]
if !ok {
return autotag.MBRelease{}, autotag.ErrGroupNotFound
}
return rel, nil
}
func (c *idFakeClient) LookupReleaseGroup(
_ context.Context, _ string,
) (autotag.MBReleaseGroupHit, error) {
return autotag.MBReleaseGroupHit{}, nil
}
func TestScorer_PersistScoreWritesTopMatch(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
@@ -200,8 +372,8 @@ func TestScorer_PersistBestWritesMatched(t *testing.T) {
t.Fatalf("ScoreGroup: %v", err)
}
if err := scorer.PersistBest(context.Background(), result); err != nil {
t.Fatalf("PersistBest: %v", err)
if err := scorer.PersistScore(context.Background(), result); err != nil {
t.Fatalf("PersistScore: %v", err)
}
got, err := db.Queries.GetTaggingItem(context.Background(), "g-pending")
@@ -209,8 +381,10 @@ func TestScorer_PersistBestWritesMatched(t *testing.T) {
t.Fatalf("reload: %v", err)
}
if got.Status != "matched" {
t.Errorf("status = %q, want 'matched'", got.Status)
// PersistScore records the pill score + best match but must leave
// the review status untouched so the folder stays in the queue.
if got.Status != "pending" {
t.Errorf("status = %q, want 'pending' (PersistScore must not flip status)", got.Status)
}
if !got.BestMatchReleaseMbid.Valid || got.BestMatchReleaseMbid.String != "rg-abcd" {
@@ -258,10 +432,20 @@ func (c *countingMBClient) BrowseReleases(
return nil, nil
}
func (c *countingMBClient) LookupArtist(_ context.Context, _ string) (string, error) {
func (c *countingMBClient) SearchRecordings(
_ context.Context, _ string, _ int,
) ([]autotag.MBRecordingHit, int, error) {
c.onSearch()
return "", nil
return nil, 0, nil
}
func (c *countingMBClient) LookupRecordingReleases(
_ context.Context, _ string,
) ([]autotag.MBReleaseRef, error) {
c.onSearch()
return nil, nil
}
func (c *countingMBClient) LookupRelease(_ context.Context, _ string) (autotag.MBRelease, error) {
backend/autotag/types.go
+25 -7
View File
@@ -14,6 +14,15 @@ type LocalTrack struct {
RecordingMBID string
}
// Group is the folder-level context candidates are ranked against:
// the tagging item's album name/artist plus its local tracks. The
// album fields are soft signals — empty values never penalize.
type Group struct {
AlbumName string
AlbumArtist string
Tracks []LocalTrack
}
// CandidateSource distinguishes candidates served from the local
// release_groups cache (zero network cost) from those fetched live.
type CandidateSource string
@@ -42,22 +51,26 @@ type Candidate struct {
OriginalDate string // "YYYY" or "YYYY-MM-DD" — release group's first release
Country string
Status string // "Official", "Promotion", ...
PrimaryType string // release group's primary type: "Album", "Single", ...
TrackCount int
Tracks []CandidateTrack
Alignments []TrackAlignment
Score float64 // 0..1, higher is better
Breakdown ScoreBreakdown
Source CandidateSource
Provenance string // cascade step that produced this ("strict", "fuzzy-title", "paste", "local")
Provenance string // cascade step that produced this ("strict", "fuzzy-title", "id", "paste", "local")
}
// ScoreBreakdown exposes the four inputs that go into Candidate.Score
// so the review UI can explain the ranking to the user.
// ScoreBreakdown exposes the inputs that go into Candidate.Score so
// the review UI can explain the ranking to the user.
type ScoreBreakdown struct {
TitleAvg float64 // average per-track title similarity (0..1)
LengthAvg float64 // average per-track length similarity (0..1)
ArtistFit float64 // album-artist vs candidate artist-credit similarity (0..1)
AlbumFit float64 // folder album name vs candidate release title similarity (0..1)
TrackCountFit float64 // 1.0 when local and candidate track counts match
ReleaseMeta float64 // year + official + country, averaged
ReleaseMeta float64 // official + country + release-group type, averaged
Evidence float64 // confidence scale from corroborating track count (0..1)
}
// CandidateTrack is one track inside a candidate release.
@@ -97,14 +110,19 @@ type TrackAlignment struct {
CandidateLength int64
TitleScore float64 // 0..1 from normalized-title edit distance
LengthScore float64 // 0..1 from lengthScore; 0.5 (neutral) if either side unknown
LengthDeltaMs int64 // abs(local - candidate); 0 if either side missing
TrackNumberOK bool // local track number matches candidate position
IDMatch bool // local recording MBID equals candidate track MBID
Status AlignmentStatus
}
// GroupScore is the scorer's full output for one tagging group.
type GroupScore struct {
GroupKey string
LocalTracks []LocalTrack
Candidates []Candidate // sorted by Score, descending
GroupKey string
AlbumName string
AlbumArtist string
LocalTracks []LocalTrack
Candidates []Candidate // sorted by Score, descending
Recommendation Recommendation
}
backend/autotagservice/service.go
+389 -52
View File
@@ -8,9 +8,12 @@ package autotagservice
import (
"context"
"database/sql"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
@@ -23,6 +26,7 @@ import (
"yellowjacket/backend/database/sql/sqlcgen"
"yellowjacket/backend/events"
"yellowjacket/backend/explore"
"yellowjacket/backend/metadata"
"yellowjacket/backend/tagwriter"
)
@@ -81,6 +85,13 @@ type Service struct {
exp *explore.Service
logger *slog.Logger
ctx context.Context
// ctxReady reports whether ctx is the Wails lifecycle context set
// via SetContext (rather than the context.Background() default). It
// gates event emission: calling wailsruntime.EventsEmit with a
// non-runtime context triggers log.Fatalf (os.Exit) inside Wails, so
// a background worker that fires before OnStartup wires the context
// would otherwise take the whole app down on launch.
ctxReady bool
// Queue cursor — the group_key of the last item returned.
// GetNextPending uses it to advance. Reset by StartAutotagQueue.
@@ -88,12 +99,6 @@ type Service struct {
cursor string
libraryID int64
// Candidate cache: freezes the candidate list the user saw so
// Apply operates on the exact release they selected even if a
// racing rescore would have changed the ranking. Keyed by
// group_key; replaced by GetCandidates / GetCandidatesForPasteURL.
candidateCache map[string][]autotag.Candidate
// In-flight set for ApplyAsync. Protects against the user
// firing two Apply jobs on the same group while the first is
// still running. Keyed by group_key; entries removed when the
@@ -194,7 +199,6 @@ func NewService(
exp: exp,
logger: logger,
ctx: context.Background(),
candidateCache: make(map[string][]autotag.Candidate),
runningApplies: make(map[string]struct{}),
}
}
@@ -202,7 +206,36 @@ func NewService(
// SetContext stores the Wails runtime context (called from
// OnStartup).
func (s *Service) SetContext(ctx context.Context) {
s.mu.Lock()
defer s.mu.Unlock()
s.ctx = ctx
s.ctxReady = ctx != nil
}
// emitEvent emits a Wails runtime event, but only when the stored
// context actually carries the Wails runtime. Wails' EventsEmit calls
// log.Fatalf — which os.Exit()s the process and cannot be recovered —
// whenever the context lacks its internal "events" value (e.g. the
// context.Background() default, or any non-lifecycle context). A
// background worker (the prefetch/apply sweeps) that emits before, or
// independently of, OnStartup wiring the real context would otherwise
// take the whole app down on launch. We replicate Wails' own
// precondition here so a not-yet-ready context degrades to a no-op
// instead of a crash.
func (s *Service) emitEvent(eventName string, data any) {
s.mu.Lock()
ready := s.ctxReady
ctx := s.ctx
s.mu.Unlock()
// hasWailsRuntime mirrors the check in wails/pkg/runtime.getEvents:
// the runtime is present only when ctx.Value("events") is non-nil.
if !ready || ctx == nil || ctx.Value("events") == nil {
return
}
wailsruntime.EventsEmit(ctx, eventName, data)
}
// StartBackgroundPrefetch kicks off (or restarts) the prefetch
@@ -335,7 +368,7 @@ func (s *Service) startPrefetch(libraryID int64) {
// folder, which populates the cache via the foreground
// GetCandidates path. Skip the redundant re-score.
if cached := s.lookupCachedCandidates(key); len(cached) > 0 {
wailsruntime.EventsEmit(s.ctx, events.AutotagPrefetchProgress, map[string]any{
s.emitEvent(events.AutotagPrefetchProgress, map[string]any{
"processed": i + 1,
"total": total,
})
@@ -343,7 +376,10 @@ func (s *Service) startPrefetch(libraryID int64) {
continue
}
score, err := s.scorer.ScoreGroup(ctx, key)
// Local-first: the background sweep skips the MusicBrainz
// cascade when a local candidate already scores well, so a
// library with cross-library duplicates costs no network here.
score, err := s.scorer.ScoreGroupLocalFirst(ctx, key)
if err != nil {
s.logger.Debug(
"prefetch: score failed — skipping",
@@ -360,13 +396,13 @@ func (s *Service) startPrefetch(libraryID int64) {
}
}
wailsruntime.EventsEmit(s.ctx, events.AutotagPrefetchProgress, map[string]any{
s.emitEvent(events.AutotagPrefetchProgress, map[string]any{
"processed": i + 1,
"total": total,
})
}
wailsruntime.EventsEmit(s.ctx, events.AutotagPrefetchFinished, map[string]any{
s.emitEvent(events.AutotagPrefetchFinished, map[string]any{
"processed": total,
"total": total,
})
@@ -548,11 +584,164 @@ func (s *Service) GetCandidateCoverArt(
return s.exp.GetCandidateThumbnail(releaseMBID, releaseGroupMBID)
}
// GetLocalCoverArt returns a data-URI for the artwork associated
// with the group's local files, so the review UI can show "what the
// folder already looks like" beside the fetched candidate art.
// It first checks each file for embedded ID3/Vorbis pictures, then
// falls back to a sidecar cover image (cover.jpg, folder.png, …)
// sitting in the album directory. Untagged folders carry one or the
// other far more often than they have a DB release-group cover, so we
// read the files directly. Returns "" when nothing is found.
func (s *Service) GetLocalCoverArt(groupKey string) string {
locals, err := s.scorer.LocalTracksForGroup(s.ctx, groupKey)
if err != nil {
s.logger.Warn(
"local cover art: list tracks failed",
"group_key", groupKey, "err", err,
)
return ""
}
// 1. Embedded artwork — guaranteed to travel with the track.
for _, t := range locals {
if t.FilePath == "" {
continue
}
meta, err := metadata.ExtractTags(t.FilePath)
if err != nil || meta == nil || meta.Picture == nil {
continue
}
if len(meta.Picture.Data) == 0 {
continue
}
mime := meta.Picture.MIMEType
if mime == "" {
mime = "image/jpeg"
}
return "data:" + mime + ";base64," +
base64.StdEncoding.EncodeToString(meta.Picture.Data)
}
// 2. Sidecar cover image file in the album directory. Scan the
// distinct directories the tracks live in (usually just one).
seen := make(map[string]struct{}, 1)
for _, t := range locals {
if t.FilePath == "" {
continue
}
dir := filepath.Dir(t.FilePath)
if _, ok := seen[dir]; ok {
continue
}
seen[dir] = struct{}{}
if uri := folderCoverDataURI(dir); uri != "" {
return uri
}
}
return ""
}
// coverImageExts maps recognised sidecar cover-image extensions to
// their MIME type for the data-URI.
var coverImageExts = map[string]string{
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".png": "image/png",
".gif": "image/gif",
".webp": "image/webp",
".bmp": "image/bmp",
}
// preferredCoverStems lists the base filenames (lower-cased, minus
// extension) commonly used for album artwork, in priority order.
var preferredCoverStems = []string{
"cover", "folder", "front", "album", "albumart", "albumartsmall", "thumb",
}
// folderCoverDataURI looks for a conventional cover-image file in dir
// and returns it as a base64 data-URI, or "" when none is found.
// Preferred filenames (cover.*, folder.*, …) win; any other image
// file in the directory is used as a last resort. Matching is
// case-insensitive so COVER.JPG and Folder.Png both hit.
func folderCoverDataURI(dir string) string {
entries, err := os.ReadDir(dir)
if err != nil {
return ""
}
byStem := make(map[string]string) // lower stem -> filename
var fallback string
for _, e := range entries {
if e.IsDir() {
continue
}
name := e.Name()
ext := strings.ToLower(filepath.Ext(name))
if _, ok := coverImageExts[ext]; !ok {
continue
}
stem := strings.ToLower(strings.TrimSuffix(name, filepath.Ext(name)))
if _, dup := byStem[stem]; !dup {
byStem[stem] = name
}
if fallback == "" {
fallback = name
}
}
name := fallback
for _, stem := range preferredCoverStems {
if match, ok := byStem[stem]; ok {
name = match
break
}
}
if name == "" {
return ""
}
data, err := os.ReadFile(filepath.Join(dir, name))
if err != nil || len(data) == 0 {
return ""
}
mime := coverImageExts[strings.ToLower(filepath.Ext(name))]
if mime == "" {
mime = "image/jpeg"
}
return "data:" + mime + ";base64," + base64.StdEncoding.EncodeToString(data)
}
// ScoreView is the Wails-friendly projection of autotag.GroupScore.
type ScoreView struct {
GroupKey string `json:"groupKey"`
LocalTracks []LocalTrackView `json:"localTracks"`
Candidates []CandidateView `json:"candidates"`
// Recommendation is the qualitative confidence tier for the
// ranked list: "none", "low", "medium", or "strong". Unlike the
// raw score it accounts for ambiguity (a rival release group
// scoring nearly as high) and alignment defects.
Recommendation string `json:"recommendation"`
}
// LocalTrackView mirrors autotag.LocalTrack.
@@ -583,6 +772,7 @@ type CandidateView struct {
OriginalDate string `json:"originalDate"`
Country string `json:"country"`
Status string `json:"status"`
PrimaryType string `json:"primaryType"`
TrackCount int `json:"trackCount"`
Score float64 `json:"score"`
Breakdown ScoreBreakdownView `json:"breakdown"`
@@ -597,8 +787,11 @@ type CandidateView struct {
type ScoreBreakdownView struct {
TitleAvg float64 `json:"titleAvg"`
LengthAvg float64 `json:"lengthAvg"`
ArtistFit float64 `json:"artistFit"`
AlbumFit float64 `json:"albumFit"`
TrackCountFit float64 `json:"trackCountFit"`
ReleaseMeta float64 `json:"releaseMeta"`
Evidence float64 `json:"evidence"`
}
// AlignmentView mirrors autotag.TrackAlignment. LocalIndex of -1
@@ -676,21 +869,50 @@ func (s *Service) GetCandidates(groupKey string) (*ScoreView, error) {
return scoreToView(score, s.exp), nil
}
// cacheCandidates replaces the candidate list for the given group.
// Thread-safe.
// cacheCandidates durably persists the scored candidate list for a
// group as a JSON blob in tagging_candidates. This freezes the exact
// list the user saw (so Apply operates on the release they selected
// even if a racing rescore would reorder it) AND survives process
// restarts, so a later session reuses the result instead of re-hitting
// MusicBrainz. Errors are logged, not returned — a failed persist
// only costs a recompute, never correctness.
func (s *Service) cacheCandidates(groupKey string, cands []autotag.Candidate) {
s.mu.Lock()
defer s.mu.Unlock()
blob, err := json.Marshal(cands)
if err != nil {
s.logger.Warn("cache candidates: marshal failed", "group_key", groupKey, "err", err)
s.candidateCache[groupKey] = cands
return
}
if err := s.db.Queries.UpsertTaggingCandidates(s.ctx, sqlcgen.UpsertTaggingCandidatesParams{
GroupKey: groupKey,
Candidates: string(blob),
}); err != nil {
s.logger.Warn("cache candidates: persist failed", "group_key", groupKey, "err", err)
}
}
// lookupCachedCandidates returns the cached list, or empty if none.
// lookupCachedCandidates returns the durably-stored candidate list for
// a group, or nil when none has been computed yet (or the blob can't
// be decoded — treated as a miss so the caller recomputes).
func (s *Service) lookupCachedCandidates(groupKey string) []autotag.Candidate {
s.mu.Lock()
defer s.mu.Unlock()
blob, err := s.db.Queries.GetTaggingCandidates(s.ctx, groupKey)
if err != nil {
if !errors.Is(err, sql.ErrNoRows) && !isNoRows(err) {
s.logger.Warn("lookup candidates: query failed", "group_key", groupKey, "err", err)
}
return s.candidateCache[groupKey]
return nil
}
var cands []autotag.Candidate
if err := json.Unmarshal([]byte(blob), &cands); err != nil {
s.logger.Warn("lookup candidates: unmarshal failed", "group_key", groupKey, "err", err)
return nil
}
return cands
}
// ApplyResultView mirrors autotag.ApplyResult for the frontend.
@@ -713,33 +935,9 @@ type FailureView struct {
// always applies the exact release they saw in the UI.
// Empty releaseMBID picks the top-scored candidate.
func (s *Service) Apply(groupKey, releaseMBID string) (*ApplyResultView, error) {
cands := s.lookupCachedCandidates(groupKey)
if len(cands) == 0 {
// Cache miss (user reloaded the page?): re-score and carry
// on — worst case they get the current top-ranked release.
score, err := s.scorer.ScoreGroup(s.ctx, groupKey)
if err != nil {
return nil, fmt.Errorf("score before apply: %w", err)
}
cands = score.Candidates
s.cacheCandidates(groupKey, cands)
}
locals, err := s.scorer.LocalTracksForGroup(s.ctx, groupKey)
plan, err := s.prepareApplyPlan(groupKey, releaseMBID)
if err != nil {
return nil, fmt.Errorf("load locals: %w", err)
}
groupScore := &autotag.GroupScore{
GroupKey: groupKey,
LocalTracks: locals,
Candidates: cands,
}
plan, err := s.applier.BuildPlan(groupScore, releaseMBID)
if err != nil {
return nil, fmt.Errorf("build plan: %w", err)
return nil, err
}
result, err := s.applier.Apply(s.ctx, plan, nil)
@@ -799,7 +997,7 @@ func (s *Service) ApplyAsync(groupKey, releaseMBID string) error {
}
}
wailsruntime.EventsEmit(s.ctx, events.AutotagApplyStarted, map[string]any{
s.emitEvent(events.AutotagApplyStarted, map[string]any{
"groupKey": groupKey,
"total": total,
})
@@ -852,7 +1050,7 @@ func (s *Service) runApply(groupKey string, plan *autotag.ApplyPlan, total int)
defer s.endApply(groupKey)
onProgress := func(current, total, succeeded, failed int) {
wailsruntime.EventsEmit(s.ctx, events.AutotagApplyProgress, map[string]any{
s.emitEvent(events.AutotagApplyProgress, map[string]any{
"groupKey": groupKey,
"current": current,
"total": total,
@@ -879,7 +1077,7 @@ func (s *Service) runApply(groupKey string, plan *autotag.ApplyPlan, total int)
finished["error"] = err.Error()
}
wailsruntime.EventsEmit(s.ctx, events.AutotagApplyFinished, finished)
s.emitEvent(events.AutotagApplyFinished, finished)
}
// tryStartApply records that an Apply for the given group is
@@ -939,8 +1137,14 @@ func (s *Service) LeaveAsIs(groupKey string) error {
}
// RetagGroup flips a group back to 'pending' so the user can
// re-review after an apply or skip.
// re-review after an apply or skip. Drops the durably-cached
// candidate list too, so the next open recomputes against fresh
// MusicBrainz data rather than reusing the stale stored result.
func (s *Service) RetagGroup(groupKey string) error {
if err := s.db.Queries.DeleteTaggingCandidates(s.ctx, groupKey); err != nil {
s.logger.Warn("retag: clear cached candidates failed", "group_key", groupKey, "err", err)
}
return s.db.Queries.SetTaggingItemStatus(
s.ctx,
sqlcgen.SetTaggingItemStatusParams{Status: "pending", GroupKey: groupKey},
@@ -980,7 +1184,11 @@ func (s *Service) GetCandidatesForPasteURL(
// splice it to the top of the list. RankCandidates would
// re-sort by score; we explicitly keep the pasted candidate
// first so the user's manual pick is visually anchored.
scored := autotag.ScoreCandidate(score.LocalTracks, pasted, len(score.LocalTracks))
scored := autotag.ScoreCandidate(autotag.Group{
AlbumName: score.AlbumName,
AlbumArtist: score.AlbumArtist,
Tracks: score.LocalTracks,
}, pasted)
merged := append([]autotag.Candidate{scored}, score.Candidates...)
score.Candidates = merged
@@ -989,6 +1197,122 @@ func (s *Service) GetCandidatesForPasteURL(
return scoreToView(score, s.exp), nil
}
// SearchHitView is one in-app MusicBrainz search result surfaced to
// the review UI. Kind is "releasegroup" or "recording" so the
// frontend knows which resolve path SelectSearchCandidate must take.
type SearchHitView struct {
MBID string `json:"mbid"`
Kind string `json:"kind"`
Title string `json:"title"`
Artist string `json:"artist"`
Detail string `json:"detail"`
}
// SearchCandidates runs an in-app MusicBrainz search — the "suggest a
// new candidate" escape hatch when the automatic cascade misses.
// kind is "recording" (title + artist, for singletons) or anything
// else (album + artist release-group search). Returns lightweight
// hits; SelectSearchCandidate resolves the picked one into a scored
// candidate.
func (s *Service) SearchCandidates(
kind, query, artist string,
) ([]SearchHitView, error) {
if kind == "recording" {
hits, err := s.mbr.SearchRecordingHits(s.ctx, query, artist)
if err != nil {
return nil, fmt.Errorf("search recordings: %w", err)
}
out := make([]SearchHitView, 0, len(hits))
for _, h := range hits {
out = append(out, SearchHitView{
MBID: h.MBID,
Kind: "recording",
Title: h.Title,
Artist: h.ArtistCredit,
Detail: formatMillis(h.LengthMillis),
})
}
return out, nil
}
hits, err := s.mbr.SearchReleaseGroupHits(s.ctx, query, artist)
if err != nil {
return nil, fmt.Errorf("search release groups: %w", err)
}
out := make([]SearchHitView, 0, len(hits))
for _, h := range hits {
detail := h.PrimaryType
if y := h.FirstDate; len(y) >= 4 { //nolint:mnd
detail = strings.TrimSpace(y[:4] + " " + detail)
}
out = append(out, SearchHitView{
MBID: h.MBID,
Kind: "releasegroup",
Title: h.Title,
Artist: h.ArtistCredit,
Detail: detail,
})
}
return out, nil
}
// SelectSearchCandidate resolves a picked search hit into a fully-
// scored candidate and splices it to the top of the group's candidate
// list — the same shape as the paste-URL path, so Apply works
// unchanged. A "recording" hit is resolved to a representative
// release first.
func (s *Service) SelectSearchCandidate(
groupKey, kind, mbid string,
) (*ScoreView, error) {
score, err := s.scorer.ScoreGroup(s.ctx, groupKey)
if err != nil {
return nil, fmt.Errorf("score group: %w", err)
}
var picked autotag.Candidate
if kind == "recording" {
picked, err = s.mbr.ResolveOneRecordingMBID(s.ctx, mbid)
} else {
picked, err = s.mbr.ResolveOneReleaseMBID(s.ctx, mbid)
}
if err != nil {
return nil, fmt.Errorf("resolve %s %s: %w", kind, mbid, err)
}
scored := autotag.ScoreCandidate(autotag.Group{
AlbumName: score.AlbumName,
AlbumArtist: score.AlbumArtist,
Tracks: score.LocalTracks,
}, picked)
merged := append([]autotag.Candidate{scored}, score.Candidates...)
score.Candidates = merged
s.cacheCandidates(groupKey, merged)
return scoreToView(score, s.exp), nil
}
// formatMillis renders a millisecond duration as m:ss for the search
// result detail line; empty for unknown lengths.
func formatMillis(ms int64) string {
if ms <= 0 {
return ""
}
sec := ms / 1000
minutes := sec / 60
seconds := sec % 60
return fmt.Sprintf("%d:%02d", minutes, seconds)
}
var errInvalidMBURL = errors.New("autotag: not a valid MusicBrainz release URL")
// extractReleaseMBID pulls the UUID out of a MusicBrainz release
@@ -1033,7 +1357,16 @@ func extractReleaseMBID(url string) string {
// top-ranked candidate; pass nil to skip cover art entirely (used
// only by paths that don't need art).
func scoreToView(s *autotag.GroupScore, exp *explore.Service) *ScoreView {
out := &ScoreView{GroupKey: s.GroupKey}
rec := s.Recommendation
if rec == "" {
// Paths that rebuild a GroupScore from cached candidates
// don't run the scorer; derive the tier here.
rec = autotag.Recommend(
autotag.Group{Tracks: s.LocalTracks}, s.Candidates,
)
}
out := &ScoreView{GroupKey: s.GroupKey, Recommendation: string(rec)}
for _, l := range s.LocalTracks {
out.LocalTracks = append(out.LocalTracks, LocalTrackView{
@@ -1058,6 +1391,7 @@ func scoreToView(s *autotag.GroupScore, exp *explore.Service) *ScoreView {
OriginalDate: c.OriginalDate,
Country: c.Country,
Status: c.Status,
PrimaryType: c.PrimaryType,
TrackCount: c.TrackCount,
Score: c.Score,
Source: string(c.Source),
@@ -1065,8 +1399,11 @@ func scoreToView(s *autotag.GroupScore, exp *explore.Service) *ScoreView {
Breakdown: ScoreBreakdownView{
TitleAvg: c.Breakdown.TitleAvg,
LengthAvg: c.Breakdown.LengthAvg,
ArtistFit: c.Breakdown.ArtistFit,
AlbumFit: c.Breakdown.AlbumFit,
TrackCountFit: c.Breakdown.TrackCountFit,
ReleaseMeta: c.Breakdown.ReleaseMeta,
Evidence: c.Breakdown.Evidence,
},
}
backend/config/config.go
+39 -2
View File
@@ -174,9 +174,46 @@ func (c *Config) Save() error {
return fmt.Errorf("could not marshal config struct: %w", err)
}
err = os.WriteFile(c.filePath, confFileData, 0o644)
// Write atomically: marshal into a temp file in the same directory,
// then rename it over the target. os.WriteFile truncates the file
// in place before writing, so a crash or kill mid-write (common
// during dev restarts) can leave a truncated — often empty — config.
// An empty TOML file loads "successfully" as all-defaults and then
// gets re-saved as defaults, silently wiping the user's settings.
// A temp-file + rename makes the replacement atomic: a reader always
// sees either the previous file or the complete new one.
tmp, err := os.CreateTemp(path.Dir(c.filePath), "config-*.toml.tmp")
if err != nil {
return fmt.Errorf("could not write config file (%s): %w", c.filePath, err)
return fmt.Errorf("could not create temp config file: %w", err)
}
tmpName := tmp.Name()
// Best-effort cleanup if we bail before the rename succeeds.
defer func() { _ = os.Remove(tmpName) }()
if _, err := tmp.Write(confFileData); err != nil {
_ = tmp.Close()
return fmt.Errorf("could not write temp config file: %w", err)
}
if err := tmp.Sync(); err != nil {
_ = tmp.Close()
return fmt.Errorf("could not sync temp config file: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("could not close temp config file: %w", err)
}
if err := os.Chmod(tmpName, 0o644); err != nil {
return fmt.Errorf("could not set config file permissions: %w", err)
}
if err := os.Rename(tmpName, c.filePath); err != nil {
return fmt.Errorf("could not replace config file (%s): %w", c.filePath, err)
}
c.logger.Debug("saved config to file", "file", c.filePath)
backend/config/window.go
+14 -4
View File
@@ -1,10 +1,20 @@
package config
const (
// DefaultWidth is the default window width in pixels.
DefaultWidth = 512
// DefaultHeight is the default window height in pixels.
DefaultHeight = 384
// DefaultWidth is the default window width in pixels for a fresh
// config. Kept comfortably above the minimum so a first launch
// (or a config with no saved size) opens at a usable size rather
// than the cramped minimum.
DefaultWidth = 1100
// DefaultHeight is the default window height in pixels for a fresh config.
DefaultHeight = 720
// MinWidth is the smallest allowed window width in pixels. Wails
// enforces this at runtime; it is also the floor below which a
// reported size is treated as bogus and not persisted.
MinWidth = 512
// MinHeight is the smallest allowed window height in pixels.
MinHeight = 384
)
// WindowConfig holds window size preferences.
backend/database/database.go
+602 -10
View File
@@ -30,13 +30,45 @@ import (
var schemas embed.FS
// DB wraps the SQLite database connection and queries.
//
// Two handles back a single database file. db is the single-writer
// connection (MaxOpenConns 1) used for every write and every
// transaction. readDB is a small multi-connection, query-only pool
// used for standalone reads. Under WAL, readers run concurrently
// with the writer, so a long background write (index build, dump
// patch) no longer blocks interactive searches — the reason searches
// stalled for seconds was that the file was in rollback-journal mode
// with a single shared connection, so any writer locked out readers.
type DB struct {
db *sql.DB
Ctx context.Context
db *sql.DB
readDB *sql.DB
Ctx context.Context
// Queries runs on the single-writer connection. Use it for every
// write and for any read that must observe an uncommitted write made
// earlier in the same logical operation.
Queries *sqlcgen.Queries
logger *slog.Logger
// ReadQueries runs on the query-only WAL read pool, so standalone
// reads proceed concurrently with a long background write instead of
// queueing behind it on the single writer. It observes only
// committed data. In tests (no read pool) it aliases Queries.
ReadQueries *sqlcgen.Queries
logger *slog.Logger
}
// Data-source names. modernc.org/sqlite only honours PRAGMAs passed
// as `_pragma=name(value)` — the mattn-style `_journal_mode=WAL`
// form is silently ignored, which is why WAL was never actually on.
const (
writeDSNParams = "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)"
readDSNParams = "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)" +
"&_pragma=query_only(true)&_pragma=synchronous(NORMAL)" +
"&_pragma=cache_size(-8000)&_pragma=mmap_size(67108864)"
// readPoolConns bounds concurrent read connections. A handful is
// plenty for interactive search + art/lookup fan-out and keeps WAL
// reader overhead small.
readPoolConns = 4
)
// NewDB opens the database and applies schema migrations.
func NewDB(logger *slog.Logger) (*DB, error) {
defer profiling.TimeOp(logger, "database.NewDB")()
@@ -52,7 +84,7 @@ func NewDB(logger *slog.Logger) (*DB, error) {
logger.Debug("opening sqlite database", "filepath", sqliteDBFilePath)
db, err := sql.Open("sqlite", sqliteDBFilePath+"?_busy_timeout=5000&_journal_mode=WAL")
db, err := sql.Open("sqlite", sqliteDBFilePath+writeDSNParams)
if err != nil {
return nil, fmt.Errorf("could not connect to sqlite database: %w", err)
}
@@ -126,14 +158,37 @@ func NewDB(logger *slog.Logger) (*DB, error) {
// Get generated queries
queries := sqlcgen.New(db)
// Open a separate query-only read pool. The write handle above
// has already converted the file to WAL, so these connections read
// a consistent snapshot concurrently with in-flight writes.
readDB, err := sql.Open("sqlite", sqliteDBFilePath+readDSNParams)
if err != nil {
return nil, fmt.Errorf("could not open read pool: %w", err)
}
readDB.SetMaxOpenConns(readPoolConns)
return &DB{
db: db,
Ctx: dbCtx,
Queries: queries,
logger: logger,
db: db,
readDB: readDB,
Ctx: dbCtx,
Queries: queries,
ReadQueries: sqlcgen.New(readDB),
logger: logger,
}, err
}
// reader returns the handle standalone reads should use: the
// query-only read pool when present, else the write handle (tests
// share one in-memory connection, which cannot be reopened).
func (d *DB) reader() *sql.DB {
if d.readDB != nil {
return d.readDB
}
return d.db
}
// BeginTx starts a new database transaction.
func (d *DB) BeginTx() (*sql.Tx, error) {
return d.db.BeginTx(d.Ctx, nil)
@@ -144,9 +199,20 @@ func (d *DB) ExecContext(query string, args ...any) (sql.Result, error) {
return d.db.ExecContext(d.Ctx, query, args...)
}
// QueryContext executes a query that returns rows.
// QueryContext executes a query that returns rows. Reads run on the
// query-only read pool so they proceed concurrently with writes under
// WAL instead of queueing behind the single writer connection.
func (d *DB) QueryContext(query string, args ...any) (*sql.Rows, error) {
return d.db.QueryContext(d.Ctx, query, args...)
return d.reader().QueryContext(d.Ctx, query, args...)
}
// QueryContextWith executes a query that returns rows using a
// caller-supplied context instead of the DB's lifecycle context.
// This lets an individual query (e.g. a superseded search) be
// cancelled independently. Like QueryContext it runs on the read
// pool.
func (d *DB) QueryContextWith(ctx context.Context, query string, args ...any) (*sql.Rows, error) {
return d.reader().QueryContext(ctx, query, args...)
}
// Logger returns the structured logger bound to this DB. Callers can
@@ -976,6 +1042,468 @@ func runMigrations(
}
}
if version < 37 { //nolint:mnd
if err := migration37ExploreFTSDiacritics(ctx, db, logger); err != nil {
return err
}
}
if version < 38 { //nolint:mnd
if err := migration38TaggingCandidates(ctx, db, logger); err != nil {
return err
}
}
if version < 39 { //nolint:mnd
if err := migration39LyricsIndex(ctx, db, logger); err != nil {
return err
}
}
if version < 40 { //nolint:mnd
if err := migration40ExploreExactMatchIndexes(ctx, db, logger); err != nil {
return err
}
}
if version < 41 { //nolint:mnd
if err := migration41ExploreChampionFTS(ctx, db, logger); err != nil {
return err
}
}
if version < 42 { //nolint:mnd
if err := migration42ReleaseToRG(ctx, db, logger); err != nil {
return err
}
}
if version < 43 { //nolint:mnd
if err := migration43MergeArtistCredits(ctx, db, logger); err != nil {
return err
}
}
if version < 44 { //nolint:mnd
if err := migration44ExploreCAAReleaseIndex(ctx, db, logger); err != nil {
return err
}
}
if version < 45 { //nolint:mnd
if err := migration45Analyze(ctx, db, logger); err != nil {
return err
}
}
if version < 46 { //nolint:mnd
if err := migration46SmartSnapshot(ctx, db, logger); err != nil {
return err
}
}
return nil
}
// migration46SmartSnapshot adds the smart_snapshot_at column to the
// playlists table. Smart playlists now materialize their evaluated
// membership into playlist_tracks and only re-evaluate on demand; the
// timestamp records when that snapshot was last taken (NULL means the
// playlist has never been materialized, so it is backfilled on first
// open).
func migration46SmartSnapshot(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 46: smart playlist snapshot column")
if _, err := db.ExecContext(ctx,
`ALTER TABLE playlists
ADD COLUMN smart_snapshot_at DATETIME`,
); err != nil {
if !isDuplicateColumnErr(err) {
return fmt.Errorf(
"migration 46: could not add smart_snapshot_at column: %w",
err,
)
}
}
if _, err := db.ExecContext(
ctx, "PRAGMA user_version = 46",
); err != nil {
return fmt.Errorf(
"migration 46: set user_version: %w", err,
)
}
logger.Info("migration 46 complete")
return nil
}
// migration45Analyze runs ANALYZE so SQLite's query planner has real
// table/index statistics. Without stats the planner guesses from row
// counts alone and mis-chose indexes on the ~2M-row explore_index — e.g.
// the top-result parent-release lookup scanned all 400k release_group
// rows via idx_explore_index_entity_pop instead of seeking the new
// idx_explore_caa_release, costing seconds per search. ANALYZE populates
// sqlite_stat1 (a one-time ~1.5s scan) and the planner then picks the
// right index for that query and every other query on these large tables.
func migration45Analyze(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 45: ANALYZE for query planner statistics")
if _, err := db.ExecContext(ctx, "ANALYZE"); err != nil {
return fmt.Errorf("migration 45: analyze: %w", err)
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 45"); err != nil {
return fmt.Errorf("migration 45: set user_version: %w", err)
}
logger.Info("migration 45 complete")
return nil
}
// migration44ExploreCAAReleaseIndex adds a partial index on
// caa_release_mbid so the top-result resolver's parent-release-group
// lookup (SearchIndex.ReleaseGroupMBIDsForCAAReleaseMBIDs) seeks the
// index instead of scanning every release_group row in explore_index
// (~150k) on the hot search path. The index is partial, mirroring the
// query's own filter (entity_type = 'release_group' AND
// caa_release_mbid is non-empty), so it stays small and covers exactly the
// rows that lookup can match. Without it, a generic query whose top
// results include recordings with cover art (e.g. "big") spends
// seconds in this scan.
func migration44ExploreCAAReleaseIndex(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 44: explore caa_release_mbid index")
if _, err := db.ExecContext(ctx, `
CREATE INDEX IF NOT EXISTS idx_explore_caa_release
ON explore_index(caa_release_mbid)
WHERE entity_type = 'release_group' AND caa_release_mbid != ''
`); err != nil {
return fmt.Errorf("migration 44: create caa_release index: %w", err)
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 44"); err != nil {
return fmt.Errorf("migration 44: set user_version: %w", err)
}
logger.Info("migration 44 complete")
return nil
}
// migration43MergeArtistCredits repairs artist rows that were created
// from full credit strings. Before the scanner resolved a track's
// primary artist, a credit like "Lana Del Rey ft. Sean Lennon" was
// stored as its own artists row and stamped with the primary artist's
// single MBID — so one MusicBrainz artist fanned out into many rows that
// shared an MBID, and the explore index (last-write-wins per MBID) then
// displayed a featured-credit string as the artist's name.
//
// This collapses every set of artists rows that share an MBID into the
// one "clean" member (a name with no featuring clause), repoints the
// artist_credit_artist links, deletes the redundant rows, and refreshes
// the explore index's artist titles from the survivors. Clusters with
// no clean member (all names carry a marker) are left untouched.
func migration43MergeArtistCredits(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 43: merge credit-string artists")
// Map each redundant artist row to the clean canonical row for its
// MBID. "Clean" = a name carrying no featuring marker; the lowest
// id among those is the canonical survivor.
if _, err := db.ExecContext(ctx, `
CREATE TEMP TABLE artist_merge_map AS
SELECT a.id AS dirty_id, canon.canon_id AS canon_id
FROM artists a
JOIN (
SELECT mbid, MIN(id) AS canon_id
FROM artists
WHERE mbid IS NOT NULL AND mbid != ''
AND lower(name) NOT LIKE '% feat %'
AND lower(name) NOT LIKE '% feat. %'
AND lower(name) NOT LIKE '% featuring %'
AND lower(name) NOT LIKE '% ft %'
AND lower(name) NOT LIKE '% ft. %'
GROUP BY mbid
) canon ON canon.mbid = a.mbid
WHERE a.id != canon.canon_id
`); err != nil {
return fmt.Errorf("migration 43: build merge map: %w", err)
}
// Drop links that would collide with an existing (canonical, credit)
// link after repointing — the unique index would otherwise reject
// the UPDATE.
if _, err := db.ExecContext(ctx, `
DELETE FROM artist_credit_artist
WHERE id IN (
SELECT aca.id
FROM artist_credit_artist aca
JOIN artist_merge_map m ON m.dirty_id = aca.artist_id
WHERE EXISTS (
SELECT 1 FROM artist_credit_artist keep
WHERE keep.artist_id = m.canon_id
AND keep.credit_id = aca.credit_id
)
)
`); err != nil {
return fmt.Errorf("migration 43: prune colliding links: %w", err)
}
// Repoint surviving links to the canonical artist.
if _, err := db.ExecContext(ctx, `
UPDATE artist_credit_artist
SET artist_id = (
SELECT canon_id FROM artist_merge_map
WHERE dirty_id = artist_credit_artist.artist_id
)
WHERE artist_id IN (SELECT dirty_id FROM artist_merge_map)
`); err != nil {
return fmt.Errorf("migration 43: repoint links: %w", err)
}
// Remove the now-orphaned credit-string artist rows.
if _, err := db.ExecContext(ctx, `
DELETE FROM artists WHERE id IN (SELECT dirty_id FROM artist_merge_map)
`); err != nil {
return fmt.Errorf("migration 43: delete merged artists: %w", err)
}
// Refresh explore-index artist rows from the surviving library
// artists so their (previously clobbered) titles show the clean
// name. The AFTER UPDATE trigger keeps explore_index_fts in sync.
// Only rows backed by a library artist are touched; dump-only rows
// are left alone.
if _, err := db.ExecContext(ctx, `
UPDATE explore_index
SET title = (
SELECT name FROM artists
WHERE artists.mbid = explore_index.mbid ORDER BY id LIMIT 1),
artist_name = (
SELECT name FROM artists
WHERE artists.mbid = explore_index.mbid ORDER BY id LIMIT 1),
local_artist_id = (
SELECT id FROM artists
WHERE artists.mbid = explore_index.mbid ORDER BY id LIMIT 1)
WHERE entity_type = 'artist'
AND EXISTS (SELECT 1 FROM artists WHERE artists.mbid = explore_index.mbid)
`); err != nil {
return fmt.Errorf("migration 43: refresh explore titles: %w", err)
}
if _, err := db.ExecContext(ctx, "DROP TABLE IF EXISTS artist_merge_map"); err != nil {
return fmt.Errorf("migration 43: drop temp table: %w", err)
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 43"); err != nil {
return fmt.Errorf("migration 43: set user_version: %w", err)
}
logger.Info("migration 43 complete")
return nil
}
// migration42ReleaseToRG creates the release_to_rg mapping table: for
// every release under an indexed release group, which release-group it
// belongs to. It is populated from the canonical dump during a full
// import (the mapping is otherwise in-memory only and discarded). The
// incremental-dump popularity refresh uses it to roll per-release listen
// deltas up to their release group, so album popularity stays fresh
// without any API call.
func migration42ReleaseToRG(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 42: release_to_rg table")
if _, err := db.ExecContext(ctx, `
CREATE TABLE IF NOT EXISTS release_to_rg (
release_mbid TEXT PRIMARY KEY,
rg_mbid TEXT NOT NULL
) WITHOUT ROWID
`); err != nil {
return fmt.Errorf("migration 42: create release_to_rg: %w", err)
}
if _, err := db.ExecContext(ctx,
"PRAGMA user_version = 42",
); err != nil {
return fmt.Errorf("migration 42: set user_version: %w", err)
}
logger.Info("migration 42 complete")
return nil
}
// migration41ExploreChampionFTS creates the "champion" full-text index:
// a second external-content FTS5 over explore_index that holds only the
// high-popularity / owned rows. Short generic prefixes ("the", "a")
// match hundreds of thousands of rows in the full index, and the
// popularity-blended ORDER BY must score every one of them — seconds of
// work. Routing those queries at the champion index instead scores only
// the ~90k rows that could plausibly win, cutting the query from seconds
// to tens of milliseconds. The table is created empty here; the search
// index populates it at runtime (see SearchIndex.RebuildChampionIndex)
// because the row set derives from popularity, which changes as the
// index is (re)built.
func migration41ExploreChampionFTS(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 41: explore champion FTS")
if _, err := db.ExecContext(ctx, `
CREATE VIRTUAL TABLE IF NOT EXISTS explore_champion_fts USING fts5(
title, artist_name, aliases,
content='explore_index',
content_rowid='id',
tokenize='unicode61 remove_diacritics 2'
)
`); err != nil {
return fmt.Errorf("migration 41: create champion fts: %w", err)
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 41"); err != nil {
return fmt.Errorf("migration 41: set user_version: %w", err)
}
logger.Info("migration 41 complete")
return nil
}
// migration39LyricsIndex creates the contentless FTS5 lyrics_index
// (see lyrics_index.sql) and back-populates it from any recordings
// that already have embedded lyrics, so lyric search works on
// existing libraries without waiting for a rescan.
func migration39LyricsIndex(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 39: lyrics_index FTS")
if _, err := db.ExecContext(ctx, `
CREATE VIRTUAL TABLE IF NOT EXISTS lyrics_index USING fts5(
lyrics,
content='',
contentless_delete=1,
tokenize='unicode61 remove_diacritics 2'
)
`); err != nil {
return fmt.Errorf("migration 39: create lyrics_index: %w", err)
}
// Back-populate from recordings that already carry lyrics. The
// rowid is the recording id so it stays stable across rebuilds.
if _, err := db.ExecContext(ctx, `
INSERT INTO lyrics_index(rowid, lyrics)
SELECT id, lyrics
FROM recordings
WHERE lyrics IS NOT NULL AND lyrics != ''
`); err != nil {
return fmt.Errorf("migration 39: populate lyrics_index: %w", err)
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 39"); err != nil {
return fmt.Errorf("migration 39: set user_version: %w", err)
}
logger.Info("migration 39 complete")
return nil
}
// migration40ExploreExactMatchIndexes adds partial expression indexes
// on LOWER(title) and LOWER(artist_name) so the interactive top-result
// resolver's exact-match lookup (SearchIndex.ExactMatches) seeks the
// index instead of scanning all ~240k explore_index rows on every
// keystroke. The indexes are partial (WHERE popularity > 0) because
// that lookup always filters on popularity, keeping them small; the
// UNION-of-equalities query shape in ExactMatches is what lets SQLite
// use them (an OR across the two columns forces a scan instead).
func migration40ExploreExactMatchIndexes(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 40: explore exact-match indexes")
if _, err := db.ExecContext(ctx, `
CREATE INDEX IF NOT EXISTS idx_explore_title_lower
ON explore_index(LOWER(title))
WHERE popularity > 0
`); err != nil {
return fmt.Errorf("migration 40: create title index: %w", err)
}
if _, err := db.ExecContext(ctx, `
CREATE INDEX IF NOT EXISTS idx_explore_artist_lower
ON explore_index(LOWER(artist_name))
WHERE popularity > 0
`); err != nil {
return fmt.Errorf("migration 40: create artist index: %w", err)
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 40"); err != nil {
return fmt.Errorf("migration 40: set user_version: %w", err)
}
logger.Info("migration 40 complete")
return nil
}
// migration38TaggingCandidates creates the tagging_candidates table —
// a durable per-group store for the scored candidate list so it is
// computed once and reused across restarts instead of re-hitting
// MusicBrainz every session (see tagging_candidates.sql). A plain
// CREATE TABLE IF NOT EXISTS is safe on both fresh and existing DBs.
func migration38TaggingCandidates(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 38: tagging_candidates")
if _, err := db.ExecContext(ctx, `
CREATE TABLE IF NOT EXISTS tagging_candidates (
group_key TEXT PRIMARY KEY,
candidates TEXT NOT NULL,
computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY(group_key) REFERENCES tagging_items(group_key) ON DELETE CASCADE
)
`); err != nil {
return fmt.Errorf("migration 38: create tagging_candidates: %w", err)
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 38"); err != nil {
return fmt.Errorf("migration 38: set user_version: %w", err)
}
logger.Info("migration 38 complete")
return nil
}
@@ -1010,6 +1538,70 @@ func migration36ClearedAt(
return nil
}
// migration37ExploreFTSDiacritics rebuilds explore_index_fts with the
// "unicode61 remove_diacritics 2" tokeniser so accented queries match
// their unaccented forms (e.g. "beyonce" finds "Beyoncé"), matching the
// library search_index tokeniser. The original table (migration 26)
// was created with the default tokeniser, which does not fold
// diacritics.
//
// Because explore_index_fts is an external-content table over
// explore_index, the rebuild repopulates from the existing content
// rows — no data loss and no need to re-run the expensive tiered index
// build.
func migration37ExploreFTSDiacritics(
ctx context.Context,
db *sql.DB,
logger *slog.Logger,
) error {
logger.Info("applying migration 37: explore_index_fts diacritic folding")
// Drop the sync triggers and the FTS table, then recreate both.
// The triggers must go first — they reference the FTS table.
stmts := []string{
`DROP TRIGGER IF EXISTS explore_index_ai`,
`DROP TRIGGER IF EXISTS explore_index_ad`,
`DROP TRIGGER IF EXISTS explore_index_au`,
`DROP TABLE IF EXISTS explore_index_fts`,
`CREATE VIRTUAL TABLE explore_index_fts USING fts5(
title, artist_name, aliases,
content='explore_index',
content_rowid='id',
tokenize='unicode61 remove_diacritics 2'
)`,
`CREATE TRIGGER explore_index_ai AFTER INSERT ON explore_index BEGIN
INSERT INTO explore_index_fts(rowid, title, artist_name, aliases)
VALUES (new.id, new.title, new.artist_name, new.aliases);
END`,
`CREATE TRIGGER explore_index_ad AFTER DELETE ON explore_index BEGIN
INSERT INTO explore_index_fts(explore_index_fts, rowid, title, artist_name, aliases)
VALUES ('delete', old.id, old.title, old.artist_name, old.aliases);
END`,
`CREATE TRIGGER explore_index_au AFTER UPDATE ON explore_index BEGIN
INSERT INTO explore_index_fts(explore_index_fts, rowid, title, artist_name, aliases)
VALUES ('delete', old.id, old.title, old.artist_name, old.aliases);
INSERT INTO explore_index_fts(rowid, title, artist_name, aliases)
VALUES (new.id, new.title, new.artist_name, new.aliases);
END`,
// Repopulate the FTS index from the content table.
`INSERT INTO explore_index_fts(explore_index_fts) VALUES('rebuild')`,
}
for _, stmt := range stmts {
if _, err := db.ExecContext(ctx, stmt); err != nil {
return fmt.Errorf("migration 37: %w", err)
}
}
if _, err := db.ExecContext(ctx, "PRAGMA user_version = 37"); err != nil {
return fmt.Errorf("migration 37: set user_version: %w", err)
}
logger.Info("migration 37 complete")
return nil
}
// backfills it from file_path, creates the basename index, and
// populates the FTS5 search_index table.
func migration2BasenameAndFTS(
backend/database/lyrics_search.go
+295
View File
@@ -0,0 +1,295 @@
package database
import (
"fmt"
"strings"
"unicode"
)
// LyricsHit is a single result from a lyric-fragment search: the
// matched recording plus enough metadata to render and play it.
type LyricsHit struct {
RecordingID int64
FilePath string
LengthMilliseconds int64
Title string
Artist string
Album string
}
// SearchLyrics finds recordings whose lyrics match the given query,
// ranked by FTS5 relevance. The query is treated as a phrase so a
// fragment like "hello darkness my old friend" matches consecutive
// words rather than each word independently. Returns nil for an
// empty query.
func (d *DB) SearchLyrics(query string, limit int) ([]LyricsHit, error) {
query = strings.TrimSpace(query)
if query == "" {
return nil, nil
}
if limit <= 0 {
limit = 25
}
ftsQuery := buildLyricsPhraseQuery(query)
if ftsQuery == "" {
return nil, nil
}
// Map the matched recording (lyrics_index.rowid == recordings.id)
// to a representative playable file via the lowest audio_files id,
// then to the track_metadata VIEW for display fields.
//
// SAFETY: FTS5 MATCH syntax unsupported by sqlc. Query is parameterized; no string interpolation.
rows, err := d.db.QueryContext(d.Ctx, `
SELECT
r.id,
tm.file_path,
tm.length_milliseconds,
tm.title,
tm.artist_name,
tm.album
FROM lyrics_index li
JOIN recordings r ON r.id = li.rowid
JOIN (
SELECT recording_id, MIN(id) AS af_id
FROM audio_files
GROUP BY recording_id
) af ON af.recording_id = r.id
JOIN track_metadata tm ON tm.id = af.af_id
WHERE lyrics_index MATCH ?
ORDER BY rank
LIMIT ?
`, ftsQuery, limit)
if err != nil {
return nil, fmt.Errorf("lyrics search failed: %w", err)
}
defer func() { _ = rows.Close() }()
var results []LyricsHit
for rows.Next() {
var h LyricsHit
if err := rows.Scan(
&h.RecordingID,
&h.FilePath,
&h.LengthMilliseconds,
&h.Title,
&h.Artist,
&h.Album,
); err != nil {
return nil, fmt.Errorf("could not scan lyrics hit: %w", err)
}
results = append(results, h)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("lyrics hit iteration error: %w", err)
}
return results, nil
}
// GetRecordingLyrics returns the stored lyrics for a recording, or
// an empty string if none are stored.
func (d *DB) GetRecordingLyrics(recordingID int64) (string, error) {
var lyrics string
err := d.db.QueryRowContext(d.Ctx,
"SELECT COALESCE(lyrics, '') FROM recordings WHERE id = ?",
recordingID,
).Scan(&lyrics)
if err != nil {
return "", fmt.Errorf("could not read recording lyrics: %w", err)
}
return lyrics, nil
}
// SetRecordingLyrics writes lyrics onto a recording and keeps the FTS
// lyrics_index in sync (delete + reinsert the single row). Used by
// the LRCLIB backfill to persist fetched lyrics. Passing an empty
// string clears both the column and the index entry.
func (d *DB) SetRecordingLyrics(recordingID int64, lyrics string) error {
if _, err := d.db.ExecContext(d.Ctx,
"UPDATE recordings SET lyrics = ? WHERE id = ?",
lyrics, recordingID,
); err != nil {
return fmt.Errorf("could not update recording lyrics: %w", err)
}
return d.upsertLyricsIndex(recordingID, lyrics)
}
// upsertLyricsIndex refreshes a single recording's entry in the
// contentless lyrics_index. contentless_delete=1 makes the DELETE
// valid; an empty lyrics string leaves the row deleted.
func (d *DB) upsertLyricsIndex(recordingID int64, lyrics string) error {
if _, err := d.db.ExecContext(d.Ctx,
"DELETE FROM lyrics_index WHERE rowid = ?", recordingID,
); err != nil {
return fmt.Errorf("could not delete lyrics_index row: %w", err)
}
if strings.TrimSpace(lyrics) == "" {
return nil
}
// SAFETY: FTS5 virtual table INSERT unsupported by sqlc. All values parameterized.
if _, err := d.db.ExecContext(d.Ctx,
"INSERT INTO lyrics_index(rowid, lyrics) VALUES (?, ?)",
recordingID, lyrics,
); err != nil {
return fmt.Errorf("could not insert lyrics_index row: %w", err)
}
return nil
}
// RebuildLyricsIndex repopulates lyrics_index from scratch using the
// current recordings table. Cheap for a personal library and safe to
// run after every scan.
func (d *DB) RebuildLyricsIndex() error {
if _, err := d.db.ExecContext(d.Ctx,
"DELETE FROM lyrics_index",
); err != nil {
return fmt.Errorf("could not clear lyrics_index: %w", err)
}
// SAFETY: FTS5 virtual table INSERT unsupported by sqlc. Values sourced from recordings; no user input.
if _, err := d.db.ExecContext(d.Ctx, `
INSERT INTO lyrics_index(rowid, lyrics)
SELECT id, lyrics
FROM recordings
WHERE lyrics IS NOT NULL AND lyrics != ''
`); err != nil {
return fmt.Errorf("could not rebuild lyrics_index: %w", err)
}
return nil
}
// RecordingsMissingLyrics returns recordings that have no stored
// lyrics but do carry the artist/title/duration needed to look them
// up from an external provider. Used by the LRCLIB backfill. The
// limit bounds each batch so the backfill can be run incrementally.
func (d *DB) RecordingsMissingLyrics(limit int) ([]LyricsCandidate, error) {
if limit <= 0 {
limit = 200
}
rows, err := d.db.QueryContext(d.Ctx, `
SELECT
r.id,
COALESCE(r.name, ''),
COALESCE(ac.text, ''),
COALESCE(rg.name, ''),
MIN(af.length_milliseconds)
FROM recordings r
JOIN audio_files af ON af.recording_id = r.id
LEFT JOIN artist_credit ac ON r.artist_credit_id = ac.id
LEFT JOIN (
SELECT recording_id, MIN(release_group_id) AS release_group_id
FROM release_group_recordings
GROUP BY recording_id
) rgr ON rgr.recording_id = r.id
LEFT JOIN release_groups rg ON rg.id = rgr.release_group_id
WHERE (r.lyrics IS NULL OR r.lyrics = '')
AND r.name IS NOT NULL AND r.name != ''
AND ac.text IS NOT NULL AND ac.text != ''
GROUP BY r.id
LIMIT ?
`, limit)
if err != nil {
return nil, fmt.Errorf("could not query recordings missing lyrics: %w", err)
}
defer func() { _ = rows.Close() }()
var out []LyricsCandidate
for rows.Next() {
var c LyricsCandidate
if err := rows.Scan(
&c.RecordingID, &c.Title, &c.Artist, &c.Album, &c.LengthMilliseconds,
); err != nil {
return nil, fmt.Errorf("could not scan lyrics candidate: %w", err)
}
out = append(out, c)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("lyrics candidate iteration error: %w", err)
}
return out, nil
}
// LyricsCandidate identifies a recording that needs its lyrics fetched
// and carries the fields an external provider matches on.
type LyricsCandidate struct {
RecordingID int64
Title string
Artist string
Album string
LengthMilliseconds int64
}
// RecordingLyricLookup returns the provider-match fields (artist,
// title, album, duration) for a single recording, so lyrics can be
// fetched on demand. Returns nil if the recording has no audio file
// or no artist/title to match on.
func (d *DB) RecordingLyricLookup(recordingID int64) (*LyricsCandidate, error) {
var c LyricsCandidate
err := d.db.QueryRowContext(d.Ctx, `
SELECT
r.id,
COALESCE(r.name, ''),
COALESCE(ac.text, ''),
COALESCE(rg.name, ''),
COALESCE(MIN(af.length_milliseconds), 0)
FROM recordings r
JOIN audio_files af ON af.recording_id = r.id
LEFT JOIN artist_credit ac ON r.artist_credit_id = ac.id
LEFT JOIN (
SELECT recording_id, MIN(release_group_id) AS release_group_id
FROM release_group_recordings
GROUP BY recording_id
) rgr ON rgr.recording_id = r.id
LEFT JOIN release_groups rg ON rg.id = rgr.release_group_id
WHERE r.id = ?
GROUP BY r.id
`, recordingID).Scan(&c.RecordingID, &c.Title, &c.Artist, &c.Album, &c.LengthMilliseconds)
if err != nil {
return nil, fmt.Errorf("could not look up recording for lyrics: %w", err)
}
if c.Title == "" || c.Artist == "" {
return nil, nil
}
return &c, nil
}
// buildLyricsPhraseQuery turns a user's lyric fragment into an FTS5
// phrase query — a single double-quoted string of tokens — so the
// words must appear adjacently ("hello darkness my old friend"),
// which is what a lyric search means. All non-alphanumeric runes are
// treated as separators (matching the unicode61 tokeniser), so no
// user character can break out of the quoted phrase. Returns "" when
// the fragment has no searchable tokens.
func buildLyricsPhraseQuery(query string) string {
fields := strings.FieldsFunc(query, func(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsNumber(r)
})
if len(fields) == 0 {
return ""
}
return `"` + strings.Join(fields, " ") + `"`
}
backend/database/lyrics_search_test.go
+252
View File
@@ -0,0 +1,252 @@
package database
import (
"testing"
)
// seedLyricsTrack inserts the minimal FK chain (artist_credit →
// recording → audio_file → release_group link) for one track with the
// given lyrics, so lyric-search tests have realistic joins.
func seedLyricsTrack(
t *testing.T,
db *DB,
id int64,
title, artist, album, lyrics string,
lenMs int64,
) {
t.Helper()
if _, err := db.ExecContext(
"INSERT OR IGNORE INTO artist_credit (id, text) VALUES (?, ?)", id, artist,
); err != nil {
t.Fatalf("insert artist_credit: %v", err)
}
if _, err := db.ExecContext(
"INSERT OR IGNORE INTO release_groups (id, name) VALUES (?, ?)", id, album,
); err != nil {
t.Fatalf("insert release_group: %v", err)
}
if _, err := db.ExecContext(
"INSERT INTO recordings (id, name, artist_credit_id, lyrics) VALUES (?, ?, ?, ?)",
id, title, id, nullableLyrics(lyrics),
); err != nil {
t.Fatalf("insert recording: %v", err)
}
if _, err := db.ExecContext(
"INSERT INTO audio_files (id, file_path, length_milliseconds, file_type_id, recording_id) "+
"VALUES (?, ?, ?, ?, ?)",
id, "/music/track"+itoa(id)+".mp3", lenMs, 0, id,
); err != nil {
t.Fatalf("insert audio_file: %v", err)
}
if _, err := db.ExecContext(
"INSERT INTO release_group_recordings (release_group_id, recording_id) VALUES (?, ?)",
id, id,
); err != nil {
t.Fatalf("insert release_group_recordings: %v", err)
}
}
func nullableLyrics(l string) any {
if l == "" {
return nil
}
return l
}
func itoa(v int64) string {
if v == 0 {
return "0"
}
var b []byte
for v > 0 {
b = append([]byte{byte('0' + v%10)}, b...)
v /= 10
}
return string(b)
}
func TestSearchLyrics(t *testing.T) {
t.Parallel()
db := NewTestDB(t)
seedLyricsTrack(
t,
db,
1,
"The Sound of Silence",
"Simon & Garfunkel",
"Sounds of Silence",
"Hello darkness my old friend\nI've come to talk with you again",
180000,
)
seedLyricsTrack(t, db, 2, "Bohemian Rhapsody", "Queen", "A Night at the Opera",
"Is this the real life? Is this just fantasy?", 354000)
seedLyricsTrack(t, db, 3, "Instrumental Track", "Some Artist", "Some Album",
"", 200000) // no lyrics — must never appear in results
if err := db.RebuildLyricsIndex(); err != nil {
t.Fatalf("RebuildLyricsIndex: %v", err)
}
t.Run("phrase match returns the right track with metadata", func(t *testing.T) {
t.Parallel()
hits, err := db.SearchLyrics("hello darkness my old friend", 10)
if err != nil {
t.Fatalf("SearchLyrics: %v", err)
}
if len(hits) != 1 {
t.Fatalf("expected 1 hit, got %d: %+v", len(hits), hits)
}
h := hits[0]
if h.RecordingID != 1 {
t.Errorf("RecordingID = %d, want 1", h.RecordingID)
}
if h.Title != "The Sound of Silence" {
t.Errorf("Title = %q, want The Sound of Silence", h.Title)
}
if h.Artist != "Simon & Garfunkel" {
t.Errorf("Artist = %q, want Simon & Garfunkel", h.Artist)
}
if h.Album != "Sounds of Silence" {
t.Errorf("Album = %q, want Sounds of Silence", h.Album)
}
if h.FilePath == "" {
t.Error("FilePath is empty; expected a playable path")
}
})
t.Run("adjacency: scrambled words do not match as a phrase", func(t *testing.T) {
t.Parallel()
hits, err := db.SearchLyrics("friend old darkness", 10)
if err != nil {
t.Fatalf("SearchLyrics: %v", err)
}
if len(hits) != 0 {
t.Errorf("expected 0 phrase hits for scrambled words, got %d", len(hits))
}
})
t.Run("empty query returns nil", func(t *testing.T) {
t.Parallel()
hits, err := db.SearchLyrics(" ", 10)
if err != nil {
t.Fatalf("SearchLyrics: %v", err)
}
if hits != nil {
t.Errorf("expected nil for empty query, got %+v", hits)
}
})
t.Run("no match returns no hits", func(t *testing.T) {
t.Parallel()
hits, err := db.SearchLyrics("this phrase appears in no song", 10)
if err != nil {
t.Fatalf("SearchLyrics: %v", err)
}
if len(hits) != 0 {
t.Errorf("expected 0 hits, got %d", len(hits))
}
})
}
func TestSetRecordingLyricsUpdatesIndex(t *testing.T) {
t.Parallel()
db := NewTestDB(t)
// Track starts with no lyrics.
seedLyricsTrack(t, db, 1, "Yesterday", "The Beatles", "Help!", "", 125000)
if err := db.RebuildLyricsIndex(); err != nil {
t.Fatalf("RebuildLyricsIndex: %v", err)
}
// Nothing indexed yet.
if hits, _ := db.SearchLyrics("yesterday all my troubles", 10); len(hits) != 0 {
t.Fatalf("expected 0 hits before backfill, got %d", len(hits))
}
// Backfill lyrics — should update both the column and the FTS index.
const lyrics = "Yesterday all my troubles seemed so far away"
if err := db.SetRecordingLyrics(1, lyrics); err != nil {
t.Fatalf("SetRecordingLyrics: %v", err)
}
stored, err := db.GetRecordingLyrics(1)
if err != nil {
t.Fatalf("GetRecordingLyrics: %v", err)
}
if stored != lyrics {
t.Errorf("stored lyrics = %q, want %q", stored, lyrics)
}
hits, err := db.SearchLyrics("all my troubles seemed so far away", 10)
if err != nil {
t.Fatalf("SearchLyrics: %v", err)
}
if len(hits) != 1 || hits[0].RecordingID != 1 {
t.Fatalf("expected recording 1 after backfill, got %+v", hits)
}
}
func TestRecordingsMissingLyrics(t *testing.T) {
t.Parallel()
db := NewTestDB(t)
seedLyricsTrack(t, db, 1, "Has Lyrics", "Artist A", "Album A", "some words here", 100000)
seedLyricsTrack(t, db, 2, "No Lyrics", "Artist B", "Album B", "", 200000)
missing, err := db.RecordingsMissingLyrics(50)
if err != nil {
t.Fatalf("RecordingsMissingLyrics: %v", err)
}
if len(missing) != 1 {
t.Fatalf("expected 1 candidate, got %d: %+v", len(missing), missing)
}
c := missing[0]
if c.RecordingID != 2 || c.Title != "No Lyrics" || c.Artist != "Artist B" {
t.Errorf("unexpected candidate: %+v", c)
}
if c.LengthMilliseconds != 200000 {
t.Errorf("LengthMilliseconds = %d, want 200000", c.LengthMilliseconds)
}
// Single-recording lookup mirrors the batch fields.
one, err := db.RecordingLyricLookup(2)
if err != nil {
t.Fatalf("RecordingLyricLookup: %v", err)
}
if one == nil || one.Artist != "Artist B" || one.Album != "Album B" {
t.Errorf("unexpected lookup: %+v", one)
}
}
backend/database/sql/queries/release_groups.sql
+44
View File
@@ -64,6 +64,17 @@ SELECT
COALESCE(rg.year, 0) AS release_year,
rg.mbid,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
LEFT JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
@@ -89,6 +100,17 @@ SELECT
COALESCE(rg.year, 0) AS release_year,
rg.mbid,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
LEFT JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
@@ -116,6 +138,17 @@ SELECT
COALESCE(rg.original_year, rg.year) AS year,
COALESCE(rg.year, 0) AS release_year,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
@@ -141,6 +174,17 @@ SELECT
COALESCE(rg.original_year, rg.year) AS year,
COALESCE(rg.year, 0) AS release_year,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
backend/database/sql/queries/tagging_candidates.sql
+15
View File
@@ -0,0 +1,15 @@
-- name: UpsertTaggingCandidates :exec
INSERT INTO tagging_candidates (group_key, candidates, computed_at)
VALUES (?, ?, CURRENT_TIMESTAMP)
ON CONFLICT(group_key) DO UPDATE SET
candidates = excluded.candidates,
computed_at = excluded.computed_at;
-- name: GetTaggingCandidates :one
SELECT candidates FROM tagging_candidates
WHERE group_key = ?
LIMIT 1;
-- name: DeleteTaggingCandidates :exec
DELETE FROM tagging_candidates
WHERE group_key = ?;
backend/database/sql/queries/tagging_items.sql
+2 -2
View File
@@ -62,7 +62,7 @@ SELECT
ti.library_id,
COALESCE(lb.name, '') AS library_name,
COALESCE(lb.path, '') AS library_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key LIMIT 1), '') AS TEXT) AS sample_file_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key AND af.group_key != '' LIMIT 1), '') AS TEXT) AS sample_file_path,
ti.track_count,
ti.album_name,
ti.album_artist,
@@ -93,7 +93,7 @@ SELECT
ti.library_id,
COALESCE(lb.name, '') AS library_name,
COALESCE(lb.path, '') AS library_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key LIMIT 1), '') AS TEXT) AS sample_file_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key AND af.group_key != '' LIMIT 1), '') AS TEXT) AS sample_file_path,
ti.track_count,
ti.album_name,
ti.album_artist,
backend/database/sql/schemas/lyrics_index.sql
+15
View File
@@ -0,0 +1,15 @@
-- Contentless FTS5 index over recording lyrics, enabling
-- "search by a lyric fragment → find the song". The rowid is
-- recordings.id. Only recordings with non-empty lyrics are indexed.
--
-- content='' means the lyric text itself is NOT stored a second time
-- (it already lives in recordings.lyrics); the index keeps only the
-- tokenised inverted index, so it stays compact even for large
-- libraries. contentless_delete=1 lets us delete/reinsert a single
-- row when a track's lyrics change (scan update or LRCLIB backfill).
CREATE VIRTUAL TABLE IF NOT EXISTS lyrics_index USING fts5(
lyrics,
content='',
contentless_delete=1,
tokenize='unicode61 remove_diacritics 2'
);
backend/database/sql/schemas/playlists.sql
+1
View File
@@ -3,6 +3,7 @@ CREATE TABLE IF NOT EXISTS playlists (
name TEXT NOT NULL,
is_smart INTEGER NOT NULL DEFAULT 0,
smart_rules TEXT,
smart_snapshot_at DATETIME,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
backend/database/sql/schemas/tagging_candidates.sql
+18
View File
@@ -0,0 +1,18 @@
-- tagging_candidates durably stores the scored candidate list for a
-- tagging group so it survives process restarts. Without it, only the
-- top score (a single REAL on tagging_items) is persisted; the full
-- candidate list — releases, tracks, alignments — is recomputed every
-- session, re-hitting MusicBrainz whenever the short-TTL http_cache has
-- expired. The blob is written once when a group is first scored and
-- read back on every subsequent open.
--
-- candidates holds the JSON-encoded []autotag.Candidate. ON DELETE
-- CASCADE ties the blob's lifetime to its tagging_items row: when a
-- group's tracks change, the scan path deletes the old group_key row
-- (and SQLite, with foreign_keys = ON, drops the stale blob with it).
CREATE TABLE IF NOT EXISTS tagging_candidates (
group_key TEXT PRIMARY KEY,
candidates TEXT NOT NULL,
computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY(group_key) REFERENCES tagging_items(group_key) ON DELETE CASCADE
);
backend/database/sql/sqlcgen/models.go
+17 -6
View File
@@ -85,6 +85,10 @@ type Library struct {
AutotagWarningAcked int64
}
type LyricsIndex struct {
Lyrics string
}
type PlayHistory struct {
ID int64
AudioFileID int64
@@ -100,12 +104,13 @@ type PlayerState struct {
}
type Playlist struct {
ID int64
Name string
IsSmart int64
SmartRules sql.NullString
CreatedAt time.Time
UpdatedAt time.Time
ID int64
Name string
IsSmart int64
SmartRules sql.NullString
SmartSnapshotAt sql.NullTime
CreatedAt time.Time
UpdatedAt time.Time
}
type PlaylistTrack struct {
@@ -184,6 +189,12 @@ type SearchIndex struct {
Album string
}
type TaggingCandidate struct {
GroupKey string
Candidates string
ComputedAt time.Time
}
type TaggingItem struct {
GroupKey string
LibraryID int64
backend/database/sql/sqlcgen/playlists.sql.go
+6 -3
View File
@@ -82,7 +82,7 @@ func (q *Queries) CountPlaylistsByName(ctx context.Context, name string) (int64,
const createPlaylist = `-- name: CreatePlaylist :one
INSERT INTO playlists (name) VALUES (?)
RETURNING id, name, is_smart, smart_rules, created_at, updated_at
RETURNING id, name, is_smart, smart_rules, smart_snapshot_at, created_at, updated_at
`
func (q *Queries) CreatePlaylist(ctx context.Context, name string) (Playlist, error) {
@@ -93,6 +93,7 @@ func (q *Queries) CreatePlaylist(ctx context.Context, name string) (Playlist, er
&i.Name,
&i.IsSmart,
&i.SmartRules,
&i.SmartSnapshotAt,
&i.CreatedAt,
&i.UpdatedAt,
)
@@ -205,7 +206,7 @@ func (q *Queries) GetAllPlaylistTracksWithMetadata(ctx context.Context) ([]GetAl
}
const getAllPlaylists = `-- name: GetAllPlaylists :many
SELECT id, name, is_smart, smart_rules, created_at, updated_at FROM playlists ORDER BY updated_at DESC
SELECT id, name, is_smart, smart_rules, smart_snapshot_at, created_at, updated_at FROM playlists ORDER BY updated_at DESC
`
func (q *Queries) GetAllPlaylists(ctx context.Context) ([]Playlist, error) {
@@ -222,6 +223,7 @@ func (q *Queries) GetAllPlaylists(ctx context.Context) ([]Playlist, error) {
&i.Name,
&i.IsSmart,
&i.SmartRules,
&i.SmartSnapshotAt,
&i.CreatedAt,
&i.UpdatedAt,
); err != nil {
@@ -251,7 +253,7 @@ func (q *Queries) GetNextPlaylistTrackPosition(ctx context.Context, playlistID i
}
const getPlaylist = `-- name: GetPlaylist :one
SELECT id, name, is_smart, smart_rules, created_at, updated_at FROM playlists WHERE id = ? LIMIT 1
SELECT id, name, is_smart, smart_rules, smart_snapshot_at, created_at, updated_at FROM playlists WHERE id = ? LIMIT 1
`
func (q *Queries) GetPlaylist(ctx context.Context, id int64) (Playlist, error) {
@@ -262,6 +264,7 @@ func (q *Queries) GetPlaylist(ctx context.Context, id int64) (Playlist, error) {
&i.Name,
&i.IsSmart,
&i.SmartRules,
&i.SmartSnapshotAt,
&i.CreatedAt,
&i.UpdatedAt,
)
backend/database/sql/sqlcgen/release_groups.sql.go
+52
View File
@@ -109,6 +109,17 @@ SELECT
COALESCE(rg.original_year, rg.year) AS year,
COALESCE(rg.year, 0) AS release_year,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
@@ -131,6 +142,7 @@ type GetAlbumsByArtistRow struct {
Year sql.NullInt64
ReleaseYear int64
ArtistName string
ArtistMbid string
CoverArtPath string
}
@@ -149,6 +161,7 @@ func (q *Queries) GetAlbumsByArtist(ctx context.Context, artistID int64) ([]GetA
&i.Year,
&i.ReleaseYear,
&i.ArtistName,
&i.ArtistMbid,
&i.CoverArtPath,
); err != nil {
return nil, err
@@ -171,6 +184,17 @@ SELECT
COALESCE(rg.original_year, rg.year) AS year,
COALESCE(rg.year, 0) AS release_year,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
@@ -205,6 +229,7 @@ type GetAlbumsByArtistByLibraryRow struct {
Year sql.NullInt64
ReleaseYear int64
ArtistName string
ArtistMbid string
CoverArtPath string
}
@@ -223,6 +248,7 @@ func (q *Queries) GetAlbumsByArtistByLibrary(ctx context.Context, arg GetAlbumsB
&i.Year,
&i.ReleaseYear,
&i.ArtistName,
&i.ArtistMbid,
&i.CoverArtPath,
); err != nil {
return nil, err
@@ -250,6 +276,17 @@ SELECT
COALESCE(rg.year, 0) AS release_year,
rg.mbid,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
LEFT JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
@@ -271,6 +308,7 @@ type GetAllAlbumsWithDetailsRow struct {
ReleaseYear int64
Mbid sql.NullString
ArtistName string
ArtistMbid string
CoverArtPath string
}
@@ -290,6 +328,7 @@ func (q *Queries) GetAllAlbumsWithDetails(ctx context.Context) ([]GetAllAlbumsWi
&i.ReleaseYear,
&i.Mbid,
&i.ArtistName,
&i.ArtistMbid,
&i.CoverArtPath,
); err != nil {
return nil, err
@@ -317,6 +356,17 @@ SELECT
COALESCE(rg.year, 0) AS release_year,
rg.mbid,
COALESCE(ac.text, fallback_ac.text, '') as artist_name,
-- primary (first-credited) album artist's MBID, for linking the
-- artist name to its detail page. Empty when the album has no
-- MB-tagged album-artist credit.
CAST(COALESCE((
SELECT a.mbid
FROM artist_credit_artist aca_p
JOIN artists a ON a.id = aca_p.artist_id
WHERE aca_p.credit_id = rg.album_artist_credit_id
ORDER BY aca_p.id
LIMIT 1
), '') AS TEXT) as artist_mbid,
COALESCE(ca.file_path, '') as cover_art_path
FROM release_groups rg
LEFT JOIN artist_credit ac ON rg.album_artist_credit_id = ac.id
@@ -345,6 +395,7 @@ type GetAllAlbumsWithDetailsByLibraryRow struct {
ReleaseYear int64
Mbid sql.NullString
ArtistName string
ArtistMbid string
CoverArtPath string
}
@@ -364,6 +415,7 @@ func (q *Queries) GetAllAlbumsWithDetailsByLibrary(ctx context.Context, libraryI
&i.ReleaseYear,
&i.Mbid,
&i.ArtistName,
&i.ArtistMbid,
&i.CoverArtPath,
); err != nil {
return nil, err
backend/database/sql/sqlcgen/tagging_candidates.sql.go
+51
View File
@@ -0,0 +1,51 @@
// Code generated by sqlc. DO NOT EDIT.
// versions:
// sqlc v1.30.0
// source: tagging_candidates.sql
package sqlcgen
import (
"context"
)
const deleteTaggingCandidates = `-- name: DeleteTaggingCandidates :exec
DELETE FROM tagging_candidates
WHERE group_key = ?
`
func (q *Queries) DeleteTaggingCandidates(ctx context.Context, groupKey string) error {
_, err := q.db.ExecContext(ctx, deleteTaggingCandidates, groupKey)
return err
}
const getTaggingCandidates = `-- name: GetTaggingCandidates :one
SELECT candidates FROM tagging_candidates
WHERE group_key = ?
LIMIT 1
`
func (q *Queries) GetTaggingCandidates(ctx context.Context, groupKey string) (string, error) {
row := q.db.QueryRowContext(ctx, getTaggingCandidates, groupKey)
var candidates string
err := row.Scan(&candidates)
return candidates, err
}
const upsertTaggingCandidates = `-- name: UpsertTaggingCandidates :exec
INSERT INTO tagging_candidates (group_key, candidates, computed_at)
VALUES (?, ?, CURRENT_TIMESTAMP)
ON CONFLICT(group_key) DO UPDATE SET
candidates = excluded.candidates,
computed_at = excluded.computed_at
`
type UpsertTaggingCandidatesParams struct {
GroupKey string
Candidates string
}
func (q *Queries) UpsertTaggingCandidates(ctx context.Context, arg UpsertTaggingCandidatesParams) error {
_, err := q.db.ExecContext(ctx, upsertTaggingCandidates, arg.GroupKey, arg.Candidates)
return err
}
backend/database/sql/sqlcgen/tagging_items.sql.go
+2 -2
View File
@@ -127,7 +127,7 @@ SELECT
ti.library_id,
COALESCE(lb.name, '') AS library_name,
COALESCE(lb.path, '') AS library_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key LIMIT 1), '') AS TEXT) AS sample_file_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key AND af.group_key != '' LIMIT 1), '') AS TEXT) AS sample_file_path,
ti.track_count,
ti.album_name,
ti.album_artist,
@@ -543,7 +543,7 @@ SELECT
ti.library_id,
COALESCE(lb.name, '') AS library_name,
COALESCE(lb.path, '') AS library_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key LIMIT 1), '') AS TEXT) AS sample_file_path,
CAST(COALESCE((SELECT af.file_path FROM audio_files af WHERE af.group_key = ti.group_key AND af.group_key != '' LIMIT 1), '') AS TEXT) AS sample_file_path,
ti.track_count,
ti.album_name,
ti.album_artist,
backend/database/tagging_items_test.go
+50
View File
@@ -283,6 +283,56 @@ func TestCountPendingTaggingItems_UsesPartialIndex(t *testing.T) {
}
}
// TestListPendingFolders_SampleFilePathUsesIndex guards the folder-list
// query's per-row sample_file_path subquery against regressing to a
// full table scan of audio_files. The `AND af.group_key != ”` guard
// is load-bearing: without it SQLite can't prove the partial index
// idx_audio_files_group_key (WHERE group_key != ”) applies, and the
// subquery degrades to O(folders * audio_files) — the difference
// between the review list loading instantly and taking a minute.
func TestListPendingFolders_SampleFilePathUsesIndex(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
rows, err := db.QueryContext(`
EXPLAIN QUERY PLAN
SELECT
ti.group_key,
CAST(COALESCE((SELECT af.file_path FROM audio_files af
WHERE af.group_key = ti.group_key AND af.group_key != '' LIMIT 1), '') AS TEXT)
FROM tagging_items ti
`)
if err != nil {
t.Fatalf("explain: %v", err)
}
defer func() { _ = rows.Close() }()
var plan strings.Builder
for rows.Next() {
var id, parent, notused int
var detail string
if scanErr := rows.Scan(&id, &parent, &notused, &detail); scanErr != nil {
t.Fatalf("scan: %v", scanErr)
}
plan.WriteString(detail)
plan.WriteString("\n")
}
if !strings.Contains(plan.String(), "idx_audio_files_group_key") {
t.Errorf(
"sample_file_path subquery no longer uses idx_audio_files_group_key "+
"(would full-scan audio_files per folder):\n%s",
plan.String(),
)
}
}
func TestGetTaggingItemAndListAudioFilesInGroup(t *testing.T) {
t.Parallel()
backend/database/testhelper.go
+7 -4
View File
@@ -75,10 +75,13 @@ func NewTestDB(t *testing.T) *DB {
t.Cleanup(func() { _ = db.Close() })
return &DB{
db: db,
Ctx: ctx,
Queries: queries,
logger: slog.Default(),
db: db,
Ctx: ctx,
// The in-memory test DB shares one connection, so reads and
// writes use the same handle; ReadQueries aliases Queries.
Queries: queries,
ReadQueries: queries,
logger: slog.Default(),
}
}
backend/events/events.go
+18
View File
@@ -90,4 +90,22 @@ const (
// Explore / search index events.
const (
IndexStatusChanged = "IndexStatusChanged"
// ArtistDiscographyReady fires (payload: artist MBID string) after a
// lazy background discography fetch persists into the index, so the
// artist detail page can re-fetch its top tracks / top releases
// without the initial request having blocked on a live fetch.
ArtistDiscographyReady = "ArtistDiscographyReady"
// ArtistSimilarReady fires (payload: artist MBID string) after a lazy
// background similar-artists fetch persists into similar_artist_map,
// so the artist detail page can re-fetch that section without the
// initial request having blocked on a live LB labs call.
ArtistSimilarReady = "ArtistSimilarReady"
// AlbumReleasesReady fires (payload: release-group MBID string) after a
// lazy background BrowseReleases fetch populates the response cache, so
// the album detail page can re-fetch its versions / tracklist without
// the initial request having blocked on a live MusicBrainz browse.
AlbumReleasesReady = "AlbumReleasesReady"
)
backend/explore/autotagclient.go
+41 -11
View File
@@ -95,22 +95,52 @@ func (c *AutotagClient) LookupReleaseGroup(
}, nil
}
// LookupArtist returns the artist's sort name when available,
// falling back to the display name. Used by the resolver when
// constructing Lucene-style fallback queries.
func (c *AutotagClient) LookupArtist(
ctx context.Context, mbid string,
) (string, error) {
a, err := c.inner.LookupArtist(ctx, mbid)
// SearchRecordings delegates to the wrapped client and projects hits
// into autotag's minimal recording shape. Length is millisecond-
// aligned to match local audio_files.
func (c *AutotagClient) SearchRecordings(
ctx context.Context, query string, limit int,
) ([]autotag.MBRecordingHit, int, error) {
recs, total, err := c.inner.SearchRecordings(ctx, query, limit)
if err != nil {
return "", err
return nil, 0, err
}
if a.SortName != "" {
return a.SortName, nil
out := make([]autotag.MBRecordingHit, 0, len(recs))
for _, rec := range recs {
out = append(out, autotag.MBRecordingHit{
MBID: rec.MBID,
Title: rec.Title,
ArtistCredit: rec.ArtistCredit,
LengthMillis: int64(rec.Length),
})
}
return a.Name, nil
return out, total, nil
}
// LookupRecordingReleases returns the releases a recording appears on,
// as slim references the resolver ranks to pick a representative
// release.
func (c *AutotagClient) LookupRecordingReleases(
ctx context.Context, recordingMBID string,
) ([]autotag.MBReleaseRef, error) {
refs, err := c.inner.LookupRecordingReleases(ctx, recordingMBID)
if err != nil {
return nil, err
}
out := make([]autotag.MBReleaseRef, 0, len(refs))
for _, r := range refs {
out = append(out, autotag.MBReleaseRef{
MBID: r.MBID,
Title: r.Title,
Status: r.Status,
Date: r.Date,
})
}
return out, nil
}
func exploreToAutotagRelease(rel MBRelease) autotag.MBRelease {
backend/explore/diskfree_unix.go
+19
View File
@@ -0,0 +1,19 @@
//go:build unix
package explore
import "golang.org/x/sys/unix"
// diskFreeBytes returns the free bytes available to the current user
// on the filesystem containing path. ok is false when unknown.
func diskFreeBytes(path string) (free uint64, ok bool) {
var st unix.Statfs_t
if err := unix.Statfs(path, &st); err != nil {
return 0, false
}
// Bavail/Bsize integer types vary by platform.
//nolint:unconvert,gosec
return st.Bavail * uint64(st.Bsize), true
}
backend/explore/diskfree_windows.go
+22
View File
@@ -0,0 +1,22 @@
//go:build windows
package explore
import "golang.org/x/sys/windows"
// diskFreeBytes returns the free bytes available to the current user
// on the volume containing path. ok is false when unknown.
func diskFreeBytes(path string) (free uint64, ok bool) {
p, err := windows.UTF16PtrFromString(path)
if err != nil {
return 0, false
}
var freeToCaller, total, totalFree uint64
if err := windows.GetDiskFreeSpaceEx(p, &freeToCaller, &total, &totalFree); err != nil {
return 0, false
}
return freeToCaller, true
}
backend/explore/dumpcatalog.go
+1079
View File
File diff suppressed because it is too large Load Diff
backend/explore/dumpcounts.go
+584
View File
@@ -0,0 +1,584 @@
package explore
import (
"archive/tar"
"bufio"
"bytes"
"context"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"strings"
"sync"
"github.com/parquet-go/parquet-go"
)
// Stage 1 of the dump import: stream the ListenBrainz spark listens
// dump (a plain tar of ~128MB parquet files) and aggregate listen
// counts per recording, release, and artist MBID. Nothing is written
// to disk except counts.bin — each parquet member is buffered in RAM,
// parsed, and discarded. The aggregate map lives in RAM (~40M entities
// ≈ 2GB) and is flushed atomically with the stream byte offset so an
// interrupted import resumes without re-downloading processed data.
const (
// countKindRecording etc. tag entries in the counts map/file.
countKindRecording = byte(1)
countKindRelease = byte(2)
countKindArtist = byte(3)
// countsFlushEveryMembers controls checkpoint frequency. Each
// flush rewrites counts.bin (~1GB by the end), so this trades
// checkpoint I/O against re-download on crash (~150 members ≈
// 19GB of stream progress).
countsFlushEveryMembers = 150
// countsProgressEveryMembers controls progress log frequency.
countsProgressEveryMembers = 50
// parquetParseWorkers is the number of concurrent parquet
// decoders. Bounded to limit RAM: each worker holds one
// ~128MB member buffer.
parquetParseWorkers = 3
// maxParquetMemberSize guards against unexpected dump format
// changes blowing out RAM.
maxParquetMemberSize = 1 << 30
// countsFileMagic identifies + versions the counts file format.
countsFileMagic = "YJCNTS01"
)
// ErrDumpFormat is returned when dump contents don't match the
// expected format.
var ErrDumpFormat = errors.New("unexpected dump format")
// mbidKey is a parsed UUID plus an entity-kind tag, used as the counts
// map key. 17 bytes instead of a 36-byte string keeps the ~40M-entry
// map around 2GB.
type mbidKey [17]byte
func makeMBIDKey(kind byte, mbid string) (mbidKey, bool) {
var k mbidKey
k[0] = kind
if !parseUUID(mbid, k[1:]) {
return k, false
}
return k, true
}
// parseUUID parses a canonical 36-char UUID string into 16 bytes.
// Returns false for anything malformed.
func parseUUID(s string, out []byte) bool {
if len(s) != 36 || s[8] != '-' || s[13] != '-' || s[18] != '-' || s[23] != '-' {
return false
}
j := 0
for i := 0; i < 36; i++ {
if i == 8 || i == 13 || i == 18 || i == 23 {
continue
}
hi := hexNibble(s[i])
i++
lo := hexNibble(s[i])
if hi == 0xFF || lo == 0xFF {
return false
}
out[j] = hi<<4 | lo
j++
}
return true
}
func hexNibble(c byte) byte {
switch {
case c >= '0' && c <= '9':
return c - '0'
case c >= 'a' && c <= 'f':
return c - 'a' + 10
case c >= 'A' && c <= 'F':
return c - 'A' + 10
default:
return 0xFF
}
}
func formatUUID(b []byte) string {
const hexdigits = "0123456789abcdef"
out := make([]byte, 36)
j := 0
for i := range 16 {
if i == 4 || i == 6 || i == 8 || i == 10 {
out[j] = '-'
j++
}
out[j] = hexdigits[b[i]>>4]
out[j+1] = hexdigits[b[i]&0x0F]
j += 2
}
return string(out)
}
// countsState is the checkpointed stage-1 state: the counts map plus
// the stream position it corresponds to.
type countsState struct {
// SparkURL pins the dump being processed so a resume never mixes
// two different dumps.
SparkURL string `json:"sparkUrl"`
// Offset is the byte offset of the next unprocessed tar member
// header (exact — includes the padding of the previous member).
Offset int64 `json:"offset"`
// MemberIdx is the index of the next unprocessed parquet member
// (logging only; Offset is authoritative for resume).
MemberIdx int `json:"memberIdx"`
// Done marks stage 1 complete.
Done bool `json:"done"`
counts map[mbidKey]uint32
}
// sparkListenRow is the projection of the spark listens parquet schema
// that the aggregator reads. All other columns are skipped.
type sparkListenRow struct {
RecordingMBID string `parquet:"recording_mbid,optional"`
ReleaseMBID string `parquet:"release_mbid,optional"`
ArtistMBIDs []string `parquet:"artist_credit_mbids,optional,list"`
}
type countParseJob struct {
idx int
endOffset int64 // exact offset of the next member header
buf []byte
}
type countParseResult struct {
idx int
endOffset int64
deltas map[mbidKey]uint32
err error
}
// aggregateListenCounts runs stage 1 to completion (or ctx cancel),
// checkpointing to the staging counts file as it goes.
func (imp *dumpImporter) aggregateListenCounts(ctx context.Context, st *countsState) error {
if st.counts == nil {
st.counts = make(map[mbidKey]uint32, 1<<20)
}
stream := newResumableReader(ctx, imp.httpClient, st.SparkURL, st.Offset)
defer func() { _ = stream.Close() }()
buffered := bufio.NewReaderSize(stream, 1<<20)
tr := tar.NewReader(buffered)
// consumedOffset is the absolute stream position of everything the
// tar reader has consumed: bytes delivered by HTTP minus bytes
// still sitting in the bufio buffer.
consumedOffset := func() int64 {
return stream.Offset - int64(buffered.Buffered())
}
jobs := make(chan countParseJob)
results := make(chan countParseResult, parquetParseWorkers)
applierDone := make(chan struct{})
bufPool := sync.Pool{New: func() any { return []byte(nil) }}
var workerWG sync.WaitGroup
for range parquetParseWorkers {
workerWG.Add(1)
go func() {
defer workerWG.Done()
for job := range jobs {
deltas, err := parseListenParquet(job.buf)
// Buffer reuse across members is intentional.
bufPool.Put(job.buf[:0]) //nolint:staticcheck
results <- countParseResult{
idx: job.idx,
endOffset: job.endOffset,
deltas: deltas,
err: err,
}
}
}()
}
// The applier merges results into st in member order, so every
// checkpoint is a contiguous prefix of the stream. It owns
// st.counts, st.Offset, and st.MemberIdx until applierDone closes;
// on error it keeps draining results so nothing deadlocks.
var applyErr error
go func() {
defer close(applierDone)
pending := make(map[int]countParseResult)
next := st.MemberIdx
lastFlushed := st.MemberIdx
for res := range results {
if applyErr != nil {
continue
}
pending[res.idx] = res
for {
r, ok := pending[next]
if !ok {
break
}
delete(pending, next)
if r.err != nil {
applyErr = r.err
break
}
for k, v := range r.deltas {
st.counts[k] += v
}
next++
st.MemberIdx = next
st.Offset = r.endOffset
if next-lastFlushed >= countsFlushEveryMembers {
if err := imp.writeCountsFile(st); err != nil {
applyErr = err
break
}
lastFlushed = next
imp.logCountsProgress(next, r.endOffset, stream.Size, len(st.counts))
if err := imp.checkDiskHeadroom(); err != nil {
applyErr = err
break
}
} else if next%countsProgressEveryMembers == 0 {
imp.logCountsProgress(next, r.endOffset, stream.Size, len(st.counts))
}
}
}
}()
memberIdx := st.MemberIdx
readErr := error(nil)
readLoop:
for {
if err := ctx.Err(); err != nil {
readErr = err
break
}
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
readErr = fmt.Errorf("listens tar: %w", err)
break
}
if hdr.Typeflag != tar.TypeReg || !strings.HasSuffix(hdr.Name, ".parquet") {
continue
}
if hdr.Size > maxParquetMemberSize {
readErr = fmt.Errorf("%w: parquet member %s is %d bytes", ErrDumpFormat, hdr.Name, hdr.Size)
break
}
buf, _ := bufPool.Get().([]byte)
if cap(buf) < int(hdr.Size) {
buf = make([]byte, hdr.Size)
}
buf = buf[:hdr.Size]
if _, err := io.ReadFull(tr, buf); err != nil {
readErr = fmt.Errorf("listens tar member read: %w", err)
break
}
// Exact next-header offset: position after the entry data plus
// the entry's block padding. Correct even when the next member
// uses PAX extension headers (those start at its header offset).
endOffset := consumedOffset() + tarPadding(hdr.Size)
select {
case jobs <- countParseJob{idx: memberIdx, endOffset: endOffset, buf: buf}:
case <-ctx.Done():
readErr = ctx.Err()
break readLoop
}
memberIdx++
}
close(jobs)
workerWG.Wait()
close(results)
<-applierDone
if readErr == nil {
readErr = applyErr
}
if readErr != nil {
// Best-effort checkpoint of applied progress before bailing,
// so even a cancelled run resumes where it left off.
_ = imp.writeCountsFile(st)
return readErr
}
st.Done = true
if err := imp.writeCountsFile(st); err != nil {
return err
}
imp.logger.Info("dump import: listen counts complete",
"members", st.MemberIdx,
"entities", len(st.counts),
)
return nil
}
// tarPadding returns the number of zero bytes following a tar entry of
// the given size (entries are padded to 512-byte blocks).
func tarPadding(size int64) int64 {
const block = 512
return (block - size%block) % block
}
// parseListenParquet decodes one parquet member and returns the
// per-entity listen-count deltas.
func parseListenParquet(buf []byte) (map[mbidKey]uint32, error) {
reader := parquet.NewGenericReader[sparkListenRow](bytes.NewReader(buf))
defer func() { _ = reader.Close() }()
deltas := make(map[mbidKey]uint32, 1<<18)
rows := make([]sparkListenRow, 4096)
for {
n, err := reader.Read(rows)
for _, row := range rows[:n] {
key, ok := makeMBIDKey(countKindRecording, row.RecordingMBID)
if !ok {
// Unmapped listen — no usable recording MBID.
continue
}
deltas[key]++
if relKey, relOK := makeMBIDKey(countKindRelease, row.ReleaseMBID); relOK {
deltas[relKey]++
}
for _, artist := range row.ArtistMBIDs {
if artKey, artOK := makeMBIDKey(countKindArtist, artist); artOK {
deltas[artKey]++
}
}
}
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return nil, fmt.Errorf("parquet read: %w", err)
}
if n == 0 {
break
}
}
return deltas, nil
}
// ---------------------------------------------------------------------------
// counts.bin persistence
// ---------------------------------------------------------------------------
// writeCountsFile atomically persists the counts map + stream position
// (write to temp file, fsync, rename).
func (imp *dumpImporter) writeCountsFile(st *countsState) error {
tmp := imp.countsPath() + ".tmp"
f, err := os.Create(tmp)
if err != nil {
return fmt.Errorf("counts file create: %w", err)
}
w := bufio.NewWriterSize(f, 1<<20)
meta, err := json.Marshal(st)
if err != nil {
_ = f.Close()
return fmt.Errorf("counts meta marshal: %w", err)
}
_, _ = w.WriteString(countsFileMagic)
var lenBuf [4]byte
binary.LittleEndian.PutUint32(lenBuf[:], uint32(len(meta)))
_, _ = w.Write(lenBuf[:])
_, _ = w.Write(meta)
var rec [21]byte
for k, v := range st.counts {
copy(rec[:17], k[:])
binary.LittleEndian.PutUint32(rec[17:], v)
if _, err := w.Write(rec[:]); err != nil {
_ = f.Close()
return fmt.Errorf("counts file write: %w", err)
}
}
if err := w.Flush(); err != nil {
_ = f.Close()
return fmt.Errorf("counts file flush: %w", err)
}
if err := f.Sync(); err != nil {
_ = f.Close()
return fmt.Errorf("counts file sync: %w", err)
}
if err := f.Close(); err != nil {
return fmt.Errorf("counts file close: %w", err)
}
if err := os.Rename(tmp, imp.countsPath()); err != nil {
return fmt.Errorf("counts file rename: %w", err)
}
return nil
}
// readCountsFile loads a previously checkpointed counts file. Returns
// (nil, nil) when no checkpoint exists.
func (imp *dumpImporter) readCountsFile() (*countsState, error) {
f, err := os.Open(imp.countsPath())
if errors.Is(err, os.ErrNotExist) {
return nil, nil //nolint:nilnil // no checkpoint is a valid, non-error state
}
if err != nil {
return nil, fmt.Errorf("counts file open: %w", err)
}
defer func() { _ = f.Close() }()
r := bufio.NewReaderSize(f, 1<<20)
magic := make([]byte, len(countsFileMagic))
if _, err := io.ReadFull(r, magic); err != nil || string(magic) != countsFileMagic {
return nil, fmt.Errorf("%w: bad counts file header", ErrDumpFormat)
}
var lenBuf [4]byte
if _, err := io.ReadFull(r, lenBuf[:]); err != nil {
return nil, fmt.Errorf("counts meta length: %w", err)
}
meta := make([]byte, binary.LittleEndian.Uint32(lenBuf[:]))
if _, err := io.ReadFull(r, meta); err != nil {
return nil, fmt.Errorf("counts meta read: %w", err)
}
st := &countsState{}
if err := json.Unmarshal(meta, st); err != nil {
return nil, fmt.Errorf("counts meta unmarshal: %w", err)
}
st.counts = make(map[mbidKey]uint32, 1<<20)
var rec [21]byte
for {
if _, err := io.ReadFull(r, rec[:]); err != nil {
if errors.Is(err, io.EOF) {
break
}
return nil, fmt.Errorf("counts record read: %w", err)
}
var k mbidKey
copy(k[:], rec[:17])
st.counts[k] = binary.LittleEndian.Uint32(rec[17:])
}
return st, nil
}
func (imp *dumpImporter) logCountsProgress(members int, offset, size int64, entities int) {
pct := float64(0)
if size > 0 {
pct = float64(offset) / float64(size) * 100
}
imp.logger.Info("dump import: listen counts progress",
"members", members,
"gb", fmt.Sprintf("%.1f", float64(offset)/(1<<30)),
"pct", fmt.Sprintf("%.1f", pct),
"entities", entities,
)
imp.setStageProgress(dumpStageCounts, int(pct), 100)
}
backend/explore/dumpimport.go
+574
View File
@@ -0,0 +1,574 @@
package explore
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"regexp"
"strconv"
"time"
"yellowjacket/backend/system"
)
// Dump-based index population. Instead of crawling the ListenBrainz
// API artist-by-artist, the index is built from two MetaBrainz dumps:
//
// 1. The spark listens dump (~170GB, streamed, never stored) yields
// listen counts for every recording/release/artist MBID.
// 2. The MusicBrainz canonical dump (~2GB, streamed) yields names and
// MBIDs, filtered to entities above a popularity floor.
//
// A short API patch pass then fills listener counts (top rows only),
// artist metadata, and the similar-artist map. Total temporary disk
// is one ~1GB counts file, deleted on completion. The pipeline
// resumes from checkpoints after interruption.
const (
// dumpImportDoneKey marks a completed import in explore_index_meta.
dumpImportDoneKey = "dump_import_done"
// listensAppliedSeriesKey stores the dump series number whose listen
// counts are folded into popularity (the high-water-mark for the
// incremental refresh). Set to the full dump's series at import, then
// advanced by each applied incremental.
listensAppliedSeriesKey = "listens_applied_series"
// releaseToRGInsertBatch bounds how many rows are written per
// transaction when persisting the release→release-group map.
releaseToRGInsertBatch = 10_000
// dumpMinStartFreeBytes is the free-disk requirement to begin an
// import (counts file + index growth + headroom).
dumpMinStartFreeBytes = 6 << 30
// dumpAbortFreeBytes aborts a running import when free disk
// drops below it.
dumpAbortFreeBytes = 2 << 30
// dumpStageAssembled in state.json means the index rows are
// written and only patch passes remain.
dumpStageAssembled = "assembled"
)
// ErrDiskSpace is returned when free disk falls below the safety floor.
var ErrDiskSpace = errors.New("insufficient free disk space")
// Dump import stages, mapped to status names shown in the UI.
const (
dumpStageCounts = iota
dumpStageCatalog
dumpStagePatch
dumpStageListeners
)
var dumpStageNames = [...]string{
"Listen Counts",
"Catalog Import",
"Metadata Patch",
"Listener Counts",
}
// Dump discovery patterns.
var (
canonicalDirRe = regexp.MustCompile(`^musicbrainz-canonical-dump-\d{8}-\d+$`)
canonicalFileRe = regexp.MustCompile(`^musicbrainz-canonical-dump-.*\.tar\.zst$`)
listensDirRe = regexp.MustCompile(`^listenbrainz-dump-\d+-\d{8}-\d+-full$`)
sparkFileRe = regexp.MustCompile(`^listenbrainz-spark-dump-.*-full\.tar$`)
)
// Production dump locations (overridable for tests).
const (
defaultCanonicalBaseURL = "https://data.metabrainz.org/pub/musicbrainz/canonical_data/"
defaultListensBaseURL = "https://data.metabrainz.org/pub/musicbrainz/listenbrainz/fullexport/"
)
// dumpImportState is the small persistent state file (staging dir).
// The heavyweight stage-1 checkpoint lives in counts.bin.
type dumpImportState struct {
SparkURL string `json:"sparkUrl"`
CanonicalURL string `json:"canonicalUrl"`
Stage string `json:"stage"`
}
// dumpImporter runs the dump import pipeline.
type dumpImporter struct {
si *SearchIndex
lb *ListenBrainzClient
logger *slog.Logger
httpClient *http.Client
stagingDir string
canonicalBaseURL string
listensBaseURL string
// Disk safety floors (fields so tests can relax them).
minStartFreeBytes uint64
abortFreeBytes uint64
// pendingArtists are kept artists whose names weren't derivable
// from the canonical dump; the metadata patch pass resolves them.
pendingArtists []string
}
func newDumpImporter(si *SearchIndex, lb *ListenBrainzClient) (*dumpImporter, error) {
dataDir, err := system.GetUserDataDirPath()
if err != nil {
return nil, fmt.Errorf("dump import: data dir: %w", err)
}
stagingDir := filepath.Join(dataDir, "explore-staging")
if err := os.MkdirAll(stagingDir, 0o755); err != nil {
return nil, fmt.Errorf("dump import: staging dir: %w", err)
}
return &dumpImporter{
si: si,
lb: lb,
logger: si.logger,
// No client-level timeout: the listens stream runs for hours.
// Discovery requests use per-request context timeouts, and
// resumableReader recovers from stalled connections.
httpClient: &http.Client{},
stagingDir: stagingDir,
canonicalBaseURL: defaultCanonicalBaseURL,
listensBaseURL: defaultListensBaseURL,
minStartFreeBytes: dumpMinStartFreeBytes,
abortFreeBytes: dumpAbortFreeBytes,
}, nil
}
func (imp *dumpImporter) countsPath() string {
return filepath.Join(imp.stagingDir, "counts.bin")
}
func (imp *dumpImporter) statePath() string {
return filepath.Join(imp.stagingDir, "state.json")
}
// run executes the pipeline, resuming from any prior checkpoint.
func (imp *dumpImporter) run(ctx context.Context) error {
if err := checkFreeDisk(imp.stagingDir, imp.minStartFreeBytes); err != nil {
return err
}
state, err := imp.readState()
if err != nil {
return err
}
// Fast path: rows already assembled, only patch passes remain.
if state.Stage == dumpStageAssembled {
imp.si.MarkReadyIfPopulated()
imp.runPatchPasses(ctx)
if err := ctx.Err(); err != nil {
return err
}
return imp.finalize()
}
// Stage 1: listen counts from the spark listens dump.
counts, err := imp.readCountsFile()
if err != nil {
imp.logger.Warn("dump import: discarding unreadable counts checkpoint", "error", err)
counts = nil
}
if counts == nil {
sparkURL, err := discoverDumpFile(
ctx, imp.httpClient, imp.listensBaseURL, listensDirRe, sparkFileRe,
)
if err != nil {
return err
}
imp.logger.Info("dump import: starting", "listensDump", sparkURL)
counts = &countsState{SparkURL: sparkURL}
} else if !counts.Done {
imp.logger.Info("dump import: resuming listen counts",
"offset", counts.Offset,
"members", counts.MemberIdx,
"entities", len(counts.counts),
)
}
// Record which dump series this import is baselined on, so the
// incremental refresh knows where to resume applying daily deltas.
// Written early (before assembly) so it survives a crash-and-resume;
// incrementals only apply once dumpImportDoneKey confirms completion.
imp.recordDumpSeries(counts.SparkURL)
imp.setStageProgress(dumpStageCounts, 0, 100)
if !counts.Done {
if err := imp.aggregateListenCounts(ctx, counts); err != nil {
return err
}
}
imp.si.setTierStatus(dumpStageNames[dumpStageCounts], "complete", 100, 100)
// Stage 2: popularity thresholds (in RAM, deterministic).
kept := imp.computeThreshold(counts.counts)
// Free the full counts map; only the kept sets are needed now.
counts.counts = nil
// Stage 3: canonical dump scan + assembly. Restartable: the
// scan is a cheap 2GB stream and assembly is an idempotent
// upsert, so no intra-stage checkpoint is needed.
if state.CanonicalURL == "" {
state.CanonicalURL, err = discoverDumpFile(
ctx, imp.httpClient, imp.canonicalBaseURL, canonicalDirRe, canonicalFileRe,
)
if err != nil {
return err
}
if err := imp.writeState(state); err != nil {
return err
}
}
imp.setStageProgress(dumpStageCatalog, 0, 0)
scan, err := imp.scanCanonicalDump(ctx, state.CanonicalURL, kept)
if err != nil {
return err
}
if err := imp.checkDiskHeadroom(); err != nil {
return err
}
// One-time reset when migrating from the legacy API-crawled
// index: its popularity values are on a different scale (the LB
// API includes MLHD+ history) and would permanently outrank
// dump-derived counts via the highest-wins upsert. Re-imports
// (dump→dump) skip this — listen counts only grow.
if !imp.si.hasMeta(dumpImportDoneKey) {
if _, err := imp.si.db.ExecContext("DELETE FROM explore_index"); err == nil {
imp.logger.Info("dump import: cleared legacy index for consistent popularity scale")
}
}
if err := imp.assembleIndex(ctx, kept, scan); err != nil {
return err
}
// Persist the release→release-group map (otherwise in-memory only)
// so incremental dumps can roll per-release listen deltas up to their
// album without an API call. Kept in lockstep with the index it was
// just built from.
imp.persistReleaseToRG(ctx, scan.releaseToRG)
imp.si.setTierStatus(dumpStageNames[dumpStageCatalog], "complete", 0, 0)
// Artists that need names from the metadata patch pass.
for mbid := range kept.artists {
if _, ok := scan.artistNames[mbid]; !ok {
imp.pendingArtists = append(imp.pendingArtists, formatUUID(mbid[:]))
}
}
state.Stage = dumpStageAssembled
if err := imp.writeState(state); err != nil {
return err
}
imp.si.MarkReadyIfPopulated()
imp.si.refreshStatusCounts()
// Stage 4: API patch passes (idempotent).
imp.runPatchPasses(ctx)
if err := ctx.Err(); err != nil {
return err
}
return imp.finalize()
}
// finalize records completion and removes all staging data.
func (imp *dumpImporter) finalize() error {
imp.si.setMeta(dumpImportDoneKey, time.Now().UTC().Format(time.RFC3339))
// Retire the legacy tier-crawl freshness keys.
_, _ = imp.si.db.ExecContext(
`DELETE FROM explore_index_meta
WHERE key IN ('tier1_built', 'tier2_built', 'tier3_built', 'tier4_built')`,
)
if err := os.RemoveAll(imp.stagingDir); err != nil {
imp.logger.Warn("dump import: staging cleanup failed", "error", err)
}
imp.si.setTierStatus(dumpStageNames[dumpStagePatch], "complete", 0, 0)
imp.si.setTierStatus(dumpStageNames[dumpStageListeners], "complete", 0, 0)
imp.si.refreshStatusCounts()
// The imported catalog changed which rows are popular, so refresh the
// champion tier used for generic short-prefix searches.
imp.si.scheduleChampionRebuild()
imp.logger.Info("dump import: complete")
return nil
}
// dumpSeriesRe extracts the monotonic series number NNNN from a dump
// URL or directory name (e.g. "listenbrainz-spark-dump-2593-…").
var dumpSeriesRe = regexp.MustCompile(`listenbrainz-(?:spark-)?dump-(\d+)-`)
// parseDumpSeries pulls the series number out of a dump URL/name.
func parseDumpSeries(url string) (int, bool) {
m := dumpSeriesRe.FindStringSubmatch(url)
if m == nil {
return 0, false
}
n, err := strconv.Atoi(m[1])
if err != nil {
return 0, false
}
return n, true
}
// recordDumpSeries stores the baseline series number for this import.
func (imp *dumpImporter) recordDumpSeries(sparkURL string) {
series, ok := parseDumpSeries(sparkURL)
if !ok {
imp.logger.Warn("dump import: could not parse dump series", "url", sparkURL)
return
}
imp.si.setMeta(listensAppliedSeriesKey, strconv.Itoa(series))
}
// persistReleaseToRG replaces the release_to_rg table with the mapping
// captured during this import, so it always reflects the just-built
// index. Idempotent: a full rebuild clears and repopulates it.
func (imp *dumpImporter) persistReleaseToRG(ctx context.Context, m map[uuid16]rgTarget) {
if len(m) == 0 {
return
}
if _, err := imp.si.db.ExecContext("DELETE FROM release_to_rg"); err != nil {
imp.logger.Warn("dump import: clear release_to_rg failed", "error", err)
return
}
written := 0
pending := 0
tx, err := imp.si.db.BeginTx()
if err != nil {
imp.logger.Warn("dump import: begin release_to_rg tx failed", "error", err)
return
}
for rel, target := range m {
if ctx.Err() != nil {
_ = tx.Rollback()
return
}
if _, err := tx.Exec(
"INSERT OR REPLACE INTO release_to_rg (release_mbid, rg_mbid) VALUES (?, ?)",
formatUUID(rel[:]), formatUUID(target.rg[:]),
); err != nil {
imp.logger.Warn("dump import: insert release_to_rg failed", "error", err)
continue
}
written++
pending++
if pending >= releaseToRGInsertBatch {
if err := tx.Commit(); err != nil {
imp.logger.Warn("dump import: commit release_to_rg batch failed", "error", err)
return
}
pending = 0
tx, err = imp.si.db.BeginTx()
if err != nil {
imp.logger.Warn("dump import: begin release_to_rg tx failed", "error", err)
return
}
}
}
if err := tx.Commit(); err != nil {
imp.logger.Warn("dump import: commit release_to_rg failed", "error", err)
return
}
imp.logger.Info("dump import: persisted release→release-group map", "rows", written)
}
func (imp *dumpImporter) readState() (*dumpImportState, error) {
state := &dumpImportState{}
data, err := os.ReadFile(imp.statePath())
if errors.Is(err, os.ErrNotExist) {
return state, nil
}
if err != nil {
return nil, fmt.Errorf("dump import state read: %w", err)
}
if err := json.Unmarshal(data, state); err != nil {
// Corrupt state: start over rather than fail permanently.
return &dumpImportState{}, nil
}
return state, nil
}
func (imp *dumpImporter) writeState(state *dumpImportState) error {
data, err := json.Marshal(state)
if err != nil {
return fmt.Errorf("dump import state marshal: %w", err)
}
tmp := imp.statePath() + ".tmp"
if err := os.WriteFile(tmp, data, 0o644); err != nil {
return fmt.Errorf("dump import state write: %w", err)
}
if err := os.Rename(tmp, imp.statePath()); err != nil {
return fmt.Errorf("dump import state rename: %w", err)
}
return nil
}
// setStageProgress reports stage progress to the UI status feed.
func (imp *dumpImporter) setStageProgress(stage, completed, total int) {
imp.si.setTierStatus(dumpStageNames[stage], "running", total, completed)
}
// checkDiskHeadroom aborts the import when free disk is critically low.
func (imp *dumpImporter) checkDiskHeadroom() error {
return checkFreeDisk(imp.stagingDir, imp.abortFreeBytes)
}
// checkFreeDisk returns ErrDiskSpace when the volume holding path has
// less than minBytes free. Unknown free space (unsupported platform)
// passes.
func checkFreeDisk(path string, minBytes uint64) error {
free, ok := diskFreeBytes(path)
if !ok {
return nil
}
if free < minBytes {
return fmt.Errorf("%w: %d MB free, need %d MB",
ErrDiskSpace, free>>20, minBytes>>20)
}
return nil
}
// ---------------------------------------------------------------------------
// SearchIndex integration
// ---------------------------------------------------------------------------
// runDumpBuild is the build entrypoint called from StartBuild's
// goroutine. It replaces the legacy tier crawl.
func (si *SearchIndex) runDumpBuild(ctx context.Context) {
si.MarkReadyIfPopulated()
// The catalog dump is authoritative and only grows; it is imported
// once and never re-crawled on a timer. Popularity freshness comes
// from incremental dumps, and new releases from lazy per-artist
// fetches — not from re-running this multi-GB import.
if si.hasMeta(dumpImportDoneKey) {
si.logger.Info("search index: dump import already complete, skipping")
si.refreshStatusCounts()
return
}
si.mu.Lock()
si.buildStatus = IndexStatus{
Building: true,
Tiers: []TierStatus{
{Name: dumpStageNames[dumpStageCounts], State: "pending"},
{Name: dumpStageNames[dumpStageCatalog], State: "pending"},
{Name: dumpStageNames[dumpStagePatch], State: "pending"},
{Name: dumpStageNames[dumpStageListeners], State: "pending"},
},
}
si.mu.Unlock()
si.refreshStatusCounts()
// Patch passes use a dedicated rate limiter so background API
// calls never compete with interactive search/browse requests.
var indexLB *ListenBrainzClient
if si.lb != nil {
indexLB = NewListenBrainzClient(
NewRateLimiterN(indexerRate), si.lb.cache, si.logger.WithGroup("indexer"),
)
}
imp, err := newDumpImporter(si, indexLB)
if err != nil {
si.logger.Error("search index: dump import init failed", "error", err)
return
}
start := time.Now()
if err := imp.run(ctx); err != nil {
if errors.Is(err, context.Canceled) || ctx.Err() != nil {
si.logger.Info("search index: dump import paused (will resume)",
"elapsed", time.Since(start).Round(time.Second),
)
} else {
si.logger.Error("search index: dump import failed", "error", err)
si.setTierError(dumpStageNames[dumpStageCounts], err.Error())
}
return
}
si.mu.Lock()
si.buildStatus.Building = false
si.mu.Unlock()
// Fold the local library into the freshly-imported catalog: owned
// entities below the dump's popularity floor are inserted, and
// dump-seeded rows that match the library are flagged in_library.
// Deep discographies stay lazy (fetched when an artist page opens).
si.PopulateLocalCrossReferences()
si.refreshStatusCounts()
si.logger.Info("search index: dump import finished",
"elapsed", time.Since(start).Round(time.Second),
)
}
backend/explore/dumpimport_test.go
+925
View File
@@ -0,0 +1,925 @@
package explore
import (
"archive/tar"
"bytes"
"context"
"encoding/csv"
"fmt"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
"github.com/klauspost/compress/zstd"
"github.com/parquet-go/parquet-go"
"yellowjacket/backend/database"
)
// Fixed MBIDs for fixtures.
const (
recA = "11111111-1111-1111-1111-111111111111"
recB = "22222222-2222-2222-2222-222222222222"
recC = "33333333-3333-3333-3333-333333333333"
relA = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
relB = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
rgA = "cccccccc-cccc-cccc-cccc-cccccccccccc"
rgB = "dddddddd-dddd-dddd-dddd-dddddddddddd"
artA = "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee"
artB = "ffffffff-ffff-ffff-ffff-ffffffffffff"
)
func testLogger() *slog.Logger {
return slog.New(slog.DiscardHandler)
}
// ---------------------------------------------------------------------------
// Unit tests: parsing helpers
// ---------------------------------------------------------------------------
func TestParseUUIDRoundTrip(t *testing.T) {
var buf [16]byte
if !parseUUID(recA, buf[:]) {
t.Fatalf("parseUUID rejected valid UUID %s", recA)
}
if got := formatUUID(buf[:]); got != recA {
t.Fatalf("round trip = %q, want %q", got, recA)
}
invalid := []string{
"", "not-a-uuid",
"11111111-1111-1111-1111-11111111111", // too short
"11111111-1111-1111-1111-1111111111111", // too long
"1111111101111-1111-1111-111111111111", // bad dash
"gggggggg-1111-1111-1111-111111111111", // bad hex
}
for _, s := range invalid {
if parseUUID(s, buf[:]) {
t.Errorf("parseUUID accepted invalid input %q", s)
}
}
}
func TestParsePGStringArray(t *testing.T) {
cases := []struct {
in string
want []string
}{
{"{" + artA + "}", []string{artA}},
{"{" + artA + "," + artB + "}", []string{artA, artB}},
{`{"` + artA + `","` + artB + `"}`, []string{artA, artB}},
{"['" + artA + "', '" + artB + "']", []string{artA, artB}},
{artA, []string{artA}},
{"", nil},
{"{}", nil},
}
for _, c := range cases {
got := parsePGStringArray(c.in)
if len(got) != len(c.want) {
t.Errorf("parsePGStringArray(%q) = %v, want %v", c.in, got, c.want)
continue
}
for i := range got {
if got[i] != c.want[i] {
t.Errorf("parsePGStringArray(%q)[%d] = %q, want %q", c.in, i, got[i], c.want[i])
}
}
}
}
func TestFloorForBudget(t *testing.T) {
vals := []uint32{100, 90, 80, 70, 60, 50, 40, 30, 20, 5}
if got := floorForBudget(vals, 3, 10); got != 80 {
t.Errorf("budget 3: floor = %d, want 80", got)
}
// Budget larger than data → min floor.
if got := floorForBudget(vals, 100, 10); got != 10 {
t.Errorf("budget 100: floor = %d, want 10", got)
}
// Floor clamped up to minFloor.
if got := floorForBudget(vals, 10, 10); got != 10 {
t.Errorf("clamp: floor = %d, want 10", got)
}
if got := floorForBudget(nil, 5, 7); got != 7 {
t.Errorf("empty: floor = %d, want 7", got)
}
}
func TestRankFloor(t *testing.T) {
desc := []uint32{100, 90, 80, 70, 60}
cases := []struct {
rank int
want uint32
}{
{1, 100},
{3, 80},
{5, 60},
{99, 60}, // clamped to the last element
}
for _, c := range cases {
if got := rankFloor(desc, c.rank); got != c.want {
t.Errorf("rankFloor(rank=%d) = %d, want %d", c.rank, got, c.want)
}
}
if got := rankFloor(nil, 3); got != 0 {
t.Errorf("rankFloor(empty) = %d, want 0", got)
}
}
func TestTierBudget(t *testing.T) {
const aFloor, bFloor = uint32(1000), uint32(100)
cases := []struct {
listens uint32
wantTrack, wantRG int
}{
{2000, perArtistTierATrack, perArtistTierARG}, // tier A
{1000, perArtistTierATrack, perArtistTierARG}, // exactly on A floor
{500, perArtistTierBTrack, perArtistTierBRG}, // tier B
{100, perArtistTierBTrack, perArtistTierBRG}, // exactly on B floor
{10, perArtistTierCTrack, perArtistTierCRG}, // tier C
}
for _, c := range cases {
gotTrack, gotRG := tierBudget(c.listens, aFloor, bFloor)
if gotTrack != c.wantTrack || gotRG != c.wantRG {
t.Errorf("tierBudget(%d) = (%d, %d), want (%d, %d)",
c.listens, gotTrack, gotRG, c.wantTrack, c.wantRG)
}
}
}
// mkMBID builds a distinct uuid16 from a single byte, for heap tests.
func mkMBID(b byte) uuid16 {
var id uuid16
id[0] = b
return id
}
func TestArtistTopNBoundedAndDeduped(t *testing.T) {
a := &artistTopN{n: 3, inSet: make(map[uuid16]struct{})}
// Add five distinct recordings; only the top 3 by listens survive.
for i, listens := range []uint32{10, 50, 30, 5, 40} {
a.add(keptRecordingRow{mbid: mkMBID(byte(i + 1)), listens: listens})
}
if len(a.rows) != 3 {
t.Fatalf("len = %d, want 3 (bounded)", len(a.rows))
}
got := map[uint32]bool{}
for _, r := range a.rows {
got[r.listens] = true
}
for _, want := range []uint32{50, 40, 30} {
if !got[want] {
t.Errorf("expected top listens %d retained, have %v", want, got)
}
}
if got[10] || got[5] {
t.Errorf("evicted entries survived: %v", got)
}
// Re-adding an existing MBID is a no-op, even with a higher count.
before := len(a.rows)
a.add(keptRecordingRow{mbid: mkMBID(2), listens: 9999})
if len(a.rows) != before {
t.Errorf("duplicate MBID grew the set: %d != %d", len(a.rows), before)
}
if _, dupHigh := got[9999]; dupHigh {
t.Error("duplicate MBID should not have been re-ranked")
}
}
func TestArtistTopRGBoundedAndDeduped(t *testing.T) {
a := &artistTopRG{n: 2, inSet: make(map[uuid16]struct{})}
a.add(mkMBID(1), 100)
a.add(mkMBID(2), 200)
a.add(mkMBID(3), 50) // below both, dropped
a.add(mkMBID(1), 999) // duplicate, ignored
if len(a.rgs) != 2 {
t.Fatalf("len = %d, want 2", len(a.rgs))
}
for _, c := range a.rgs {
if c.listens == 50 {
t.Error("sub-threshold RG was kept")
}
if c.rg == mkMBID(1) && c.listens != 100 {
t.Errorf("duplicate RG re-ranked to %d", c.listens)
}
}
}
// ---------------------------------------------------------------------------
// Fixture builders
// ---------------------------------------------------------------------------
// sparkFixtureRow mimics the real spark listens schema: the aggregator
// must project just recording/release/artist MBIDs out of it.
type sparkFixtureRow struct {
ListenedAt int64 `parquet:"listened_at"`
UserID int64 `parquet:"user_id"`
ArtistName string `parquet:"artist_name,optional"`
RecordingMBID string `parquet:"recording_mbid,optional"`
ReleaseMBID string `parquet:"release_mbid,optional"`
ArtistMBIDs []string `parquet:"artist_credit_mbids,optional,list"`
}
func makeParquet(t *testing.T, rows []sparkFixtureRow) []byte {
t.Helper()
var buf bytes.Buffer
w := parquet.NewGenericWriter[sparkFixtureRow](&buf)
if _, err := w.Write(rows); err != nil {
t.Fatalf("parquet write: %v", err)
}
if err := w.Close(); err != nil {
t.Fatalf("parquet close: %v", err)
}
return buf.Bytes()
}
func makeTar(t *testing.T, members map[string][]byte, order []string) []byte {
t.Helper()
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
for _, name := range order {
data := members[name]
hdr := &tar.Header{
Name: name,
Mode: 0o644,
Size: int64(len(data)),
Typeflag: tar.TypeReg,
}
if err := tw.WriteHeader(hdr); err != nil {
t.Fatalf("tar header: %v", err)
}
if _, err := tw.Write(data); err != nil {
t.Fatalf("tar write: %v", err)
}
}
if err := tw.Close(); err != nil {
t.Fatalf("tar close: %v", err)
}
return buf.Bytes()
}
func zstdCompress(t *testing.T, data []byte) []byte {
t.Helper()
var buf bytes.Buffer
zw, err := zstd.NewWriter(&buf)
if err != nil {
t.Fatalf("zstd writer: %v", err)
}
if _, err := zw.Write(data); err != nil {
t.Fatalf("zstd write: %v", err)
}
if err := zw.Close(); err != nil {
t.Fatalf("zstd close: %v", err)
}
return buf.Bytes()
}
func csvBytes(t *testing.T, rows [][]string) []byte {
t.Helper()
var buf bytes.Buffer
w := csv.NewWriter(&buf)
if err := w.WriteAll(rows); err != nil {
t.Fatalf("csv write: %v", err)
}
return buf.Bytes()
}
// listensOf builds n identical listen rows for a recording.
func listensOf(n int, recording, release string, artists []string) []sparkFixtureRow {
rows := make([]sparkFixtureRow, n)
for i := range rows {
rows[i] = sparkFixtureRow{
ListenedAt: 1700000000 + int64(i),
UserID: int64(i),
ArtistName: "Fixture Artist",
RecordingMBID: recording,
ReleaseMBID: release,
ArtistMBIDs: artists,
}
}
return rows
}
// canonicalDataCSV builds a canonical_musicbrainz_data.csv fixture.
func canonicalDataCSV(t *testing.T) []byte {
t.Helper()
rows := [][]string{
{
"id", "artist_credit_id", "artist_mbids", "artist_credit_name",
"release_mbid", "release_name", "recording_mbid", "recording_name",
"combined_lookup", "score",
},
{"1", "10", "{" + artA + "}", "Solo Star", relA, "Big Album", recA, "Hit Song", "x", "1"},
{"2", "10", "{" + artA + "}", "Solo Star", relA, "Big Album", recB, "Deep Cut", "x", "1"},
{
"3", "11", "{" + artA + "," + artB + "}", "Solo Star feat. Guest",
relB, "Duet Album", recC, "Duet Song", "x", "1",
},
}
return csvBytes(t, rows)
}
// canonicalRedirectCSV builds a canonical_release_redirect.csv fixture.
func canonicalRedirectCSV(t *testing.T) []byte {
t.Helper()
rows := [][]string{
{"release_mbid", "canonical_release_mbid", "release_group_mbid"},
{relA, relA, rgA},
{relB, relB, rgB},
}
return csvBytes(t, rows)
}
// serveDumps returns an httptest server presenting MetaBrainz-style
// listing pages and Range-capable dump files.
func serveDumps(t *testing.T, sparkTar, canonicalTarZst []byte) *httptest.Server {
t.Helper()
const (
listensDir = "listenbrainz-dump-1-20260101-000003-full"
sparkFile = "listenbrainz-spark-dump-1-20260101-000003-full.tar"
canonicalDir = "musicbrainz-canonical-dump-20260101-080003"
canonicalTar = "musicbrainz-canonical-dump-20260101-080003.tar.zst"
)
modTime := time.Now()
mux := http.NewServeMux()
mux.HandleFunc("/listens/", func(w http.ResponseWriter, r *http.Request) {
switch strings.TrimPrefix(r.URL.Path, "/listens/") {
case "":
_, _ = fmt.Fprintf(w, `<a href="%s/">%s/</a>`, listensDir, listensDir)
case listensDir + "/":
_, _ = fmt.Fprintf(w, `<a href="%s">%s</a>`, sparkFile, sparkFile)
case listensDir + "/" + sparkFile:
http.ServeContent(w, r, sparkFile, modTime, bytes.NewReader(sparkTar))
default:
http.NotFound(w, r)
}
})
mux.HandleFunc("/canonical/", func(w http.ResponseWriter, r *http.Request) {
switch strings.TrimPrefix(r.URL.Path, "/canonical/") {
case "":
_, _ = fmt.Fprintf(w, `<a href="%s/">%s/</a>`, canonicalDir, canonicalDir)
case canonicalDir + "/":
_, _ = fmt.Fprintf(w, `<a href="%s">%s</a>`, canonicalTar, canonicalTar)
case canonicalDir + "/" + canonicalTar:
http.ServeContent(w, r, canonicalTar, modTime, bytes.NewReader(canonicalTarZst))
default:
http.NotFound(w, r)
}
})
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv
}
func testImporter(t *testing.T, si *SearchIndex, srv *httptest.Server) *dumpImporter {
t.Helper()
return &dumpImporter{
si: si,
lb: nil, // patch passes skipped in tests
logger: testLogger(),
httpClient: srv.Client(),
stagingDir: t.TempDir(),
canonicalBaseURL: srv.URL + "/canonical/",
listensBaseURL: srv.URL + "/listens/",
}
}
func fixtureSparkTar(t *testing.T) []byte {
t.Helper()
// Member 1: recA is popular (12 listens). Member 2: recB has 11,
// recC has 12 (multi-artist credit). Totals: artA = 35, artB = 12.
member1 := makeParquet(t, listensOf(12, recA, relA, []string{artA}))
member2 := makeParquet(t, append(
listensOf(11, recB, relA, []string{artA}),
listensOf(12, recC, relB, []string{artA, artB})...,
))
prefix := "listenbrainz-spark-dump-1-20260101-000003-full/listens/"
return makeTar(t,
map[string][]byte{
prefix + "1.parquet": member1,
prefix + "2.parquet": member2,
},
[]string{prefix + "1.parquet", prefix + "2.parquet"},
)
}
func fixtureCanonicalTarZst(t *testing.T) []byte {
t.Helper()
prefix := "musicbrainz-canonical-dump-20260101-080003/"
raw := makeTar(t,
map[string][]byte{
prefix + "canonical_musicbrainz_data.csv": canonicalDataCSV(t),
prefix + "canonical_release_redirect.csv": canonicalRedirectCSV(t),
prefix + "canonical_recording_redirect.csv": {},
},
[]string{
prefix + "canonical_release_redirect.csv",
prefix + "canonical_musicbrainz_data.csv",
prefix + "canonical_recording_redirect.csv",
},
)
return zstdCompress(t, raw)
}
// ---------------------------------------------------------------------------
// Stage tests
// ---------------------------------------------------------------------------
func TestAggregateListenCounts(t *testing.T) {
srv := serveDumps(t, fixtureSparkTar(t), nil)
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
imp := testImporter(t, si, srv)
sparkURL, err := discoverDumpFile(
context.Background(), imp.httpClient, imp.listensBaseURL, listensDirRe, sparkFileRe,
)
if err != nil {
t.Fatalf("discover: %v", err)
}
st := &countsState{SparkURL: sparkURL}
if err := imp.aggregateListenCounts(context.Background(), st); err != nil {
t.Fatalf("aggregate: %v", err)
}
assertCount := func(kind byte, mbid string, want uint32) {
t.Helper()
key, ok := makeMBIDKey(kind, mbid)
if !ok {
t.Fatalf("bad fixture mbid %s", mbid)
}
if got := st.counts[key]; got != want {
t.Errorf("count(kind=%d, %s) = %d, want %d", kind, mbid, got, want)
}
}
assertCount(countKindRecording, recA, 12)
assertCount(countKindRecording, recB, 11)
assertCount(countKindRecording, recC, 12)
assertCount(countKindRelease, relA, 23)
assertCount(countKindRelease, relB, 12)
assertCount(countKindArtist, artA, 35)
assertCount(countKindArtist, artB, 12)
if !st.Done {
t.Error("state not marked done")
}
// The checkpoint file round-trips.
loaded, err := imp.readCountsFile()
if err != nil {
t.Fatalf("read counts file: %v", err)
}
if loaded == nil || !loaded.Done || len(loaded.counts) != len(st.counts) {
t.Fatalf("checkpoint mismatch: %+v", loaded)
}
}
func TestAggregateResumeFromOffset(t *testing.T) {
sparkTar := fixtureSparkTar(t)
srv := serveDumps(t, sparkTar, nil)
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
imp := testImporter(t, si, srv)
sparkURL := srv.URL + "/listens/listenbrainz-dump-1-20260101-000003-full/listenbrainz-spark-dump-1-20260101-000003-full.tar"
// Full run for reference.
full := &countsState{SparkURL: sparkURL}
if err := imp.aggregateListenCounts(context.Background(), full); err != nil {
t.Fatalf("full aggregate: %v", err)
}
// Simulate a checkpoint taken after member 1: offset = header
// block + padded member-1 size (fixture names are short, so the
// header is a single 512-byte block).
member1 := makeParquet(t, listensOf(12, recA, relA, []string{artA}))
offset := int64(512) + (int64(len(member1))+511)/512*512
key, _ := makeMBIDKey(countKindRecording, recA)
relKey, _ := makeMBIDKey(countKindRelease, relA)
artKey, _ := makeMBIDKey(countKindArtist, artA)
resumed := &countsState{
SparkURL: sparkURL,
Offset: offset,
MemberIdx: 1,
counts: map[mbidKey]uint32{
key: 12,
relKey: 12,
artKey: 12,
},
}
if err := imp.aggregateListenCounts(context.Background(), resumed); err != nil {
t.Fatalf("resumed aggregate: %v", err)
}
if len(resumed.counts) != len(full.counts) {
t.Fatalf("resumed entities = %d, want %d", len(resumed.counts), len(full.counts))
}
for k, want := range full.counts {
if got := resumed.counts[k]; got != want {
t.Errorf("resumed count %s = %d, want %d (double count?)", formatUUID(k[1:]), got, want)
}
}
}
func TestResumableReaderReconnects(t *testing.T) {
payload := bytes.Repeat([]byte("0123456789abcdef"), 4096) // 64KB
// A flaky server that truncates every response to 10KB, forcing
// the reader to reconnect with Range requests.
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
offset := int64(0)
if rng := r.Header.Get("Range"); rng != "" {
_, _ = fmt.Sscanf(rng, "bytes=%d-", &offset)
}
chunk := payload[offset:min(offset+10240, int64(len(payload)))]
w.Header().Set("Content-Range",
fmt.Sprintf("bytes %d-%d/%d", offset, offset+int64(len(chunk))-1, len(payload)))
w.WriteHeader(http.StatusPartialContent)
_, _ = w.Write(chunk)
}))
t.Cleanup(srv.Close)
r := newResumableReader(context.Background(), srv.Client(), srv.URL, 0)
got, err := io.ReadAll(r)
if err != nil {
t.Fatalf("read: %v", err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("payload mismatch: got %d bytes, want %d", len(got), len(payload))
}
}
// ---------------------------------------------------------------------------
// End-to-end
// ---------------------------------------------------------------------------
func TestDumpImportEndToEnd(t *testing.T) {
srv := serveDumps(t, fixtureSparkTar(t), fixtureCanonicalTarZst(t))
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
imp := testImporter(t, si, srv)
stagingDir := imp.stagingDir
// A legacy API-crawled row with inflated popularity must be
// cleared by the first dump import (scale consistency).
legacyMBID := "99999999-9999-9999-9999-999999999999"
si.upsertBatch([]SearchIndexResult{{
EntityType: "recording",
MBID: legacyMBID,
Title: "Legacy Row",
ArtistName: "Old Crawl",
ArtistMBID: artA,
Popularity: 123_456_789,
}})
if err := imp.run(context.Background()); err != nil {
t.Fatalf("run: %v", err)
}
legacyRows, err := db.QueryContext(
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", legacyMBID,
)
if err != nil {
t.Fatalf("legacy query: %v", err)
}
if legacyRows.Next() {
var n int
_ = legacyRows.Scan(&n)
if n != 0 {
t.Error("legacy API-crawled row survived the first dump import")
}
}
_ = legacyRows.Close()
// Index rows landed with dump-derived popularity.
assertRow := func(mbid, entityType, title string, popularity int) {
t.Helper()
rows, err := db.QueryContext(
"SELECT title, popularity FROM explore_index WHERE mbid = ? AND entity_type = ?",
mbid, entityType,
)
if err != nil {
t.Fatalf("query: %v", err)
}
defer func() { _ = rows.Close() }()
if !rows.Next() {
t.Fatalf("no %s row for %s", entityType, mbid)
}
var gotTitle string
var gotPop int
if err := rows.Scan(&gotTitle, &gotPop); err != nil {
t.Fatalf("scan: %v", err)
}
if gotTitle != title || gotPop != popularity {
t.Errorf("%s %s = (%q, %d), want (%q, %d)",
entityType, mbid, gotTitle, gotPop, title, popularity)
}
}
assertRow(recA, "recording", "Hit Song", 12)
assertRow(recB, "recording", "Deep Cut", 11)
assertRow(recC, "recording", "Duet Song", 12)
assertRow(rgA, "release_group", "Big Album", 23)
assertRow(rgB, "release_group", "Duet Album", 12)
assertRow(artA, "artist", "Solo Star", 35)
// artB only ever appears in a multi-artist credit: no name is
// derivable from the dump, so it must be queued for the API
// metadata patch instead of being written nameless.
rows, err := db.QueryContext(
"SELECT COUNT(*) FROM explore_index WHERE mbid = ?", artB,
)
if err != nil {
t.Fatalf("query artB: %v", err)
}
if rows.Next() {
var n int
_ = rows.Scan(&n)
if n != 0 {
t.Errorf("artB row written without a name source")
}
}
_ = rows.Close()
found := false
for _, mbid := range imp.pendingArtists {
if mbid == artB {
found = true
}
}
if !found {
t.Errorf("artB not queued for metadata patch: %v", imp.pendingArtists)
}
// FTS search works end to end.
si.MarkReadyIfPopulated()
results := si.Search(context.Background(), "hit song", 10)
if len(results) == 0 || results[0].MBID != recA {
t.Fatalf("search for indexed recording failed: %+v", results)
}
// Completion recorded; staging cleaned up.
if !si.hasMeta(dumpImportDoneKey) {
t.Error("dump_import_done not recorded")
}
// The incremental refresh baseline was recorded, and the
// release→release-group map was persisted for future rollups.
if _, ok := si.metaInt(listensAppliedSeriesKey); !ok {
t.Error("listens_applied_series baseline not recorded")
}
var relToRGRows int
rtrRows, err := db.QueryContext("SELECT COUNT(*) FROM release_to_rg")
if err != nil {
t.Fatalf("query release_to_rg: %v", err)
}
if rtrRows.Next() {
_ = rtrRows.Scan(&relToRGRows)
}
_ = rtrRows.Close()
if relToRGRows == 0 {
t.Error("release_to_rg not populated after import")
}
if _, err := os.Stat(stagingDir); !os.IsNotExist(err) {
t.Errorf("staging dir not cleaned up: %v", err)
}
// Re-running is a cheap no-op that doesn't error.
imp2 := testImporter(t, si, srv)
if err := imp2.run(context.Background()); err != nil {
t.Fatalf("second run: %v", err)
}
}
func TestDumpImportResumesAfterCancel(t *testing.T) {
srv := serveDumps(t, fixtureSparkTar(t), fixtureCanonicalTarZst(t))
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
imp := testImporter(t, si, srv)
// Cancelled before it can start streaming: no partial state may
// break the follow-up run.
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := imp.run(ctx); err == nil {
t.Fatal("cancelled run should return an error")
}
if err := imp.run(context.Background()); err != nil {
t.Fatalf("rerun after cancel: %v", err)
}
results := si.Search(context.Background(), "hit song", 10)
if len(results) == 0 {
t.Fatal("index empty after resumed run")
}
}
func TestCheckFreeDisk(t *testing.T) {
dir := t.TempDir()
if err := checkFreeDisk(dir, 1); err != nil {
t.Errorf("1 byte requirement should pass: %v", err)
}
if err := checkFreeDisk(dir, 1<<62); err == nil {
t.Error("absurd requirement should fail")
}
}
func TestDiscoverDumpFilePickNewest(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/":
_, _ = io.WriteString(w, `
<a href="musicbrainz-canonical-dump-20260101-080003/">old</a>
<a href="musicbrainz-canonical-dump-20260615-080003/">new</a>
<a href="unrelated-dir/">x</a>`)
case "/musicbrainz-canonical-dump-20260615-080003/":
_, _ = io.WriteString(w,
`<a href="musicbrainz-canonical-dump-20260615-080003.tar.zst">f</a>`)
default:
http.NotFound(w, r)
}
})
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
url, err := discoverDumpFile(
context.Background(), srv.Client(), srv.URL+"/", canonicalDirRe, canonicalFileRe,
)
if err != nil {
t.Fatalf("discover: %v", err)
}
want := srv.URL + "/musicbrainz-canonical-dump-20260615-080003/musicbrainz-canonical-dump-20260615-080003.tar.zst"
if url != want {
t.Errorf("url = %s, want %s", url, want)
}
}
func TestListenerCountUpdateDoesNotTouchPopularity(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
si.upsertBatch([]SearchIndexResult{{
EntityType: "recording",
MBID: recA,
Title: "Hit Song",
ArtistName: "Solo Star",
ArtistMBID: artA,
Popularity: 12,
}})
updated := si.updateListenerCounts(map[string]PopularityData{
recA: {ListenCount: 999_999, ListenerCount: 42},
})
if updated != 1 {
t.Fatalf("updated = %d, want 1", updated)
}
rows, err := db.QueryContext(
"SELECT popularity, listener_count FROM explore_index WHERE mbid = ?", recA,
)
if err != nil {
t.Fatalf("query: %v", err)
}
defer func() { _ = rows.Close() }()
if !rows.Next() {
t.Fatal("row missing")
}
var pop, listeners int
if err := rows.Scan(&pop, &listeners); err != nil {
t.Fatalf("scan: %v", err)
}
if pop != 12 {
t.Errorf("popularity = %d, want 12 (dump scale must stay authoritative)", pop)
}
if listeners != 42 {
t.Errorf("listener_count = %d, want 42", listeners)
}
}
backend/explore/dumpincremental.go
+522
View File
@@ -0,0 +1,522 @@
package explore
import (
"archive/tar"
"bufio"
"context"
"database/sql"
"errors"
"fmt"
"io"
"net/http"
"regexp"
"sort"
"strconv"
"strings"
"time"
)
// Incremental listen-count refresh. ListenBrainz publishes a small
// (~250MB) incremental spark dump every day containing only the listens
// submitted since the previous dump, in the same parquet-in-tar format
// as the full dump. This folds those daily deltas into the index's
// popularity numbers additively, so recordings, artists, and albums stay
// fresh without re-streaming the ~170GB full dump and without a single
// ListenBrainz API call.
//
// Album (release-group) popularity is derived locally: the incremental
// gives per-release listen counts, which are rolled up to their release
// group via the release_to_rg map captured during the full import.
//
// Correctness: each dump's deltas plus the high-water-mark advance are
// applied in one transaction, so a crash can't half-apply a dump, and
// the high-water-mark guarantees each dump is applied at most once. The
// sum of the full dump plus every incremental is therefore the exact
// cumulative listen count — the additive model does not drift.
const (
// defaultIncrementalBaseURL is where ListenBrainz publishes daily
// incremental listen dumps.
defaultIncrementalBaseURL = "https://data.metabrainz.org/pub/musicbrainz/listenbrainz/incremental/"
// listensCatchupTsKey records when the incremental refresh last ran
// (RFC3339), gating how often it re-checks for new dumps.
listensCatchupTsKey = "listens_last_catchup"
// listensCatchupInterval is the default minimum time between refresh
// checks. Album/track popularity is slow-moving and each dump is a
// ~250MB download, so a weekly cadence keeps the index current while
// bounding background network use.
listensCatchupInterval = 7 * 24 * time.Hour
// deltaInsertBatch bounds rows per multi-row INSERT into the temp
// delta table during a single incremental apply.
deltaInsertBatch = 500
// releaseLookupBatch bounds release MBIDs per release_to_rg lookup.
releaseLookupBatch = 500
)
var (
incrementalDirRe = regexp.MustCompile(`^listenbrainz-dump-\d+-\d{8}-\d+-incremental$`)
incrementalFileRe = regexp.MustCompile(`^listenbrainz-spark-dump-.*-incremental\.tar$`)
)
// incrementalDump identifies one daily incremental dump.
type incrementalDump struct {
series int
url string
}
// RefreshListenCounts folds any incremental dumps newer than the current
// high-water-mark into the index's popularity numbers. It is a no-op
// when there is no completed baseline import, when a full build is
// running, when the last refresh was within minInterval (pass 0 to
// force), or when offline. Runs synchronously — callers wanting the
// background behaviour should invoke it in a goroutine.
func (si *SearchIndex) RefreshListenCounts(ctx context.Context, minInterval time.Duration) {
if !si.hasMeta(dumpImportDoneKey) {
si.logger.Info("incremental refresh: no baseline import yet, skipping")
return
}
si.mu.RLock()
building := si.cancel != nil
si.mu.RUnlock()
if building {
si.logger.Info("incremental refresh: full build running, skipping")
return
}
if minInterval > 0 && si.refreshedWithin(minInterval) {
si.logger.Info("incremental refresh: checked recently, skipping")
return
}
hwm, ok := si.metaInt(listensAppliedSeriesKey)
if !ok {
si.logger.Warn("incremental refresh: no baseline series recorded, skipping")
return
}
client := &http.Client{}
dumps, err := discoverIncrementalDumps(ctx, client, defaultIncrementalBaseURL, hwm)
if err != nil {
si.logger.Warn("incremental refresh: discovery failed", "error", err)
return
}
// Record the check even when there is nothing to apply, so the
// cadence gate holds regardless of outcome.
defer si.setMeta(listensCatchupTsKey, time.Now().UTC().Format(time.RFC3339))
if len(dumps) == 0 {
si.logger.Info("incremental refresh: up to date", "throughSeries", hwm)
return
}
si.logger.Info("incremental refresh: applying dumps",
"count", len(dumps), "fromSeries", hwm+1, "toSeries", dumps[len(dumps)-1].series,
)
applied := 0
through := hwm
for _, d := range dumps {
if ctx.Err() != nil {
break
}
if err := si.applyIncremental(ctx, client, d); err != nil {
// Stop at the first failure; the high-water-mark is only
// advanced on success, so the next run resumes from here.
si.logger.Warn("incremental refresh: apply failed, stopping",
"series", d.series, "error", err)
break
}
applied++
through = d.series
}
if applied > 0 {
// Popularity changed, so refresh the champion tier that backs
// generic short-prefix searches.
si.scheduleChampionRebuild()
}
si.logger.Info("incremental refresh: complete", "applied", applied, "throughSeries", through)
}
// applyIncremental downloads one incremental dump, aggregates its listen
// counts, rolls release counts up to release groups, and applies the
// deltas atomically together with the high-water-mark advance.
func (si *SearchIndex) applyIncremental(
ctx context.Context, client *http.Client, dump incrementalDump,
) error {
start := time.Now()
stream := newResumableReader(ctx, client, dump.url, 0)
defer func() { _ = stream.Close() }()
counts, err := aggregateTarListens(ctx, stream)
if err != nil {
return fmt.Errorf("aggregate incremental %d: %w", dump.series, err)
}
rec, art, rel := splitCountsByKind(counts)
rg := si.rollupReleaseDeltas(rel)
if err := si.commitListenDeltas(dump.series, rec, art, rg); err != nil {
return err
}
si.logger.Info("incremental refresh: applied dump",
"series", dump.series,
"recordings", len(rec),
"artists", len(art),
"releaseGroups", len(rg),
"elapsed", time.Since(start).Round(time.Millisecond),
)
return nil
}
// splitCountsByKind separates an aggregated counts map into per-kind
// maps of canonical MBID string → delta.
func splitCountsByKind(counts map[mbidKey]uint32) (rec, art, rel map[string]uint32) {
rec = make(map[string]uint32)
art = make(map[string]uint32)
rel = make(map[string]uint32)
for k, v := range counts {
mbid := formatUUID(k[1:])
switch k[0] {
case countKindRecording:
rec[mbid] += v
case countKindArtist:
art[mbid] += v
case countKindRelease:
rel[mbid] += v
}
}
return rec, art, rel
}
// rollupReleaseDeltas maps per-release listen deltas to their release
// group via the release_to_rg table and sums per group. Releases not in
// the table (below the index floor, or unknown) contribute nothing.
func (si *SearchIndex) rollupReleaseDeltas(rel map[string]uint32) map[string]uint32 {
rg := make(map[string]uint32)
if len(rel) == 0 {
return rg
}
mbids := make([]string, 0, len(rel))
for m := range rel {
mbids = append(mbids, m)
}
for i := 0; i < len(mbids); i += releaseLookupBatch {
end := min(i+releaseLookupBatch, len(mbids))
batch := mbids[i:end]
placeholders := strings.TrimSuffix(strings.Repeat("?,", len(batch)), ",")
args := make([]any, len(batch))
for j, m := range batch {
args[j] = m
}
rows, err := si.db.QueryContext(
"SELECT release_mbid, rg_mbid FROM release_to_rg WHERE release_mbid IN ("+placeholders+")",
args...,
)
if err != nil {
si.logger.Warn("incremental refresh: release_to_rg lookup failed", "error", err)
continue
}
for rows.Next() {
var relMBID, rgMBID string
if err := rows.Scan(&relMBID, &rgMBID); err == nil {
rg[rgMBID] += rel[relMBID]
}
}
_ = rows.Close()
}
return rg
}
// commitListenDeltas applies recording, artist, and release-group deltas
// to explore_index and advances the high-water-mark to series, all in a
// single transaction so the apply is crash-atomic and exactly-once.
func (si *SearchIndex) commitListenDeltas(
series int, rec, art, rg map[string]uint32,
) error {
tx, err := si.db.BeginTx()
if err != nil {
return fmt.Errorf("incremental tx: %w", err)
}
defer func() { _ = tx.Rollback() }()
if _, err := tx.Exec(
"CREATE TEMP TABLE IF NOT EXISTS incr_delta (mbid TEXT, kind TEXT, delta INTEGER)",
); err != nil {
return fmt.Errorf("incremental temp table: %w", err)
}
if _, err := tx.Exec("DELETE FROM incr_delta"); err != nil {
return fmt.Errorf("incremental temp reset: %w", err)
}
for _, kd := range []struct {
kind string
deltas map[string]uint32
}{
{"recording", rec},
{"artist", art},
{"release_group", rg},
} {
if err := insertDeltas(tx, kd.kind, kd.deltas); err != nil {
return err
}
}
// Additive apply: bump popularity for every index row that has a
// matching delta. Rows with no delta are untouched; deltas with no
// matching row (entity not indexed) are ignored.
if _, err := tx.Exec(`
UPDATE explore_index
SET popularity = popularity + d.delta
FROM incr_delta d
WHERE d.mbid = explore_index.mbid
AND d.kind = explore_index.entity_type
`); err != nil {
return fmt.Errorf("incremental apply: %w", err)
}
if _, err := tx.Exec("DELETE FROM incr_delta"); err != nil {
return fmt.Errorf("incremental temp cleanup: %w", err)
}
if _, err := tx.Exec(
"INSERT OR REPLACE INTO explore_index_meta (key, value) VALUES (?, ?)",
listensAppliedSeriesKey, strconv.Itoa(series),
); err != nil {
return fmt.Errorf("incremental advance high-water-mark: %w", err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("incremental commit: %w", err)
}
return nil
}
// insertDeltas bulk-inserts a kind's deltas into the temp table.
func insertDeltas(tx *sql.Tx, kind string, deltas map[string]uint32) error {
if len(deltas) == 0 {
return nil
}
rowArgs := make([]any, 0, deltaInsertBatch*3)
pending := 0
flush := func() error {
if pending == 0 {
return nil
}
values := strings.TrimSuffix(strings.Repeat("(?,?,?),", pending), ",")
query := "INSERT INTO incr_delta (mbid, kind, delta) VALUES " + values
if _, err := tx.Exec(query, rowArgs...); err != nil {
return fmt.Errorf("incremental insert deltas: %w", err)
}
rowArgs = rowArgs[:0]
pending = 0
return nil
}
for mbid, d := range deltas {
rowArgs = append(rowArgs, mbid, kind, int64(d))
pending++
if pending >= deltaInsertBatch {
if err := flush(); err != nil {
return err
}
}
}
return flush()
}
// aggregateTarListens streams a tar of parquet listen members and sums
// the per-entity listen counts. Unlike the full-dump path, the whole
// (small) incremental is aggregated in RAM with no checkpointing.
func aggregateTarListens(ctx context.Context, r io.Reader) (map[mbidKey]uint32, error) {
buffered := bufio.NewReaderSize(r, 1<<20)
tr := tar.NewReader(buffered)
counts := make(map[mbidKey]uint32, 1<<18)
for {
if err := ctx.Err(); err != nil {
return nil, err
}
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return nil, fmt.Errorf("incremental tar: %w", err)
}
if hdr.Typeflag != tar.TypeReg || !strings.HasSuffix(hdr.Name, ".parquet") {
continue
}
if hdr.Size > maxParquetMemberSize {
return nil, fmt.Errorf(
"%w: parquet member %s is %d bytes", ErrDumpFormat, hdr.Name, hdr.Size,
)
}
buf := make([]byte, hdr.Size)
if _, err := io.ReadFull(tr, buf); err != nil {
return nil, fmt.Errorf("incremental member read: %w", err)
}
deltas, err := parseListenParquet(buf)
if err != nil {
return nil, fmt.Errorf("incremental parse: %w", err)
}
for k, v := range deltas {
counts[k] += v
}
}
return counts, nil
}
// discoverIncrementalDumps lists the incremental directory and returns
// the spark-dump URLs for every dump with a series greater than
// sinceSeries, sorted ascending so they apply in chronological order.
func discoverIncrementalDumps(
ctx context.Context, client *http.Client, baseURL string, sinceSeries int,
) ([]incrementalDump, error) {
hrefs, err := listHrefs(ctx, client, baseURL)
if err != nil {
return nil, err
}
var dumps []incrementalDump
for _, h := range hrefs {
dir := trimTrailingSlash(h)
if !incrementalDirRe.MatchString(dir) {
continue
}
series, ok := parseDumpSeries(dir)
if !ok || series <= sinceSeries {
continue
}
dirURL := baseURL + dir + "/"
files, err := listHrefs(ctx, client, dirURL)
if err != nil {
continue
}
for _, f := range files {
if incrementalFileRe.MatchString(f) {
dumps = append(dumps, incrementalDump{series: series, url: dirURL + f})
break
}
}
}
sort.Slice(dumps, func(i, j int) bool { return dumps[i].series < dumps[j].series })
return dumps, nil
}
// metaInt reads an integer-valued explore_index_meta key.
func (si *SearchIndex) metaInt(key string) (int, bool) {
rows, err := si.db.QueryContext("SELECT value FROM explore_index_meta WHERE key = ?", key)
if err != nil {
return 0, false
}
defer func() { _ = rows.Close() }()
if !rows.Next() {
return 0, false
}
var v string
if err := rows.Scan(&v); err != nil {
return 0, false
}
n, err := strconv.Atoi(v)
if err != nil {
return 0, false
}
return n, true
}
// refreshedWithin reports whether the incremental refresh last ran less
// than d ago.
func (si *SearchIndex) refreshedWithin(d time.Duration) bool {
rows, err := si.db.QueryContext(
"SELECT value FROM explore_index_meta WHERE key = ?", listensCatchupTsKey,
)
if err != nil {
return false
}
defer func() { _ = rows.Close() }()
if !rows.Next() {
return false
}
var v string
if err := rows.Scan(&v); err != nil {
return false
}
t, err := time.Parse(time.RFC3339, v)
if err != nil {
return false
}
return time.Since(t) < d
}
backend/explore/dumpincremental_test.go
+209
View File
@@ -0,0 +1,209 @@
package explore
import (
"bytes"
"context"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"yellowjacket/backend/database"
)
func TestParseDumpSeries(t *testing.T) {
cases := []struct {
in string
want int
ok bool
}{
{"https://x/fullexport/listenbrainz-dump-2593-20260712-000004-full/y.tar", 2593, true},
{"listenbrainz-dump-2594-20260713-000003-incremental", 2594, true},
{"listenbrainz-spark-dump-2603-20260722-000003-incremental.tar", 2603, true},
{"nothing-here", 0, false},
}
for _, c := range cases {
got, ok := parseDumpSeries(c.in)
if ok != c.ok || (ok && got != c.want) {
t.Errorf("parseDumpSeries(%q) = (%d, %v); want (%d, %v)", c.in, got, ok, c.want, c.ok)
}
}
}
func popularityOf(t *testing.T, db *database.DB, mbid string) (int, bool) {
t.Helper()
rows, err := db.QueryContext(
"SELECT popularity FROM explore_index WHERE mbid = ?", mbid,
)
if err != nil {
t.Fatalf("query popularity: %v", err)
}
defer func() { _ = rows.Close() }()
if !rows.Next() {
return 0, false
}
var pop int
if err := rows.Scan(&pop); err != nil {
t.Fatalf("scan popularity: %v", err)
}
return pop, true
}
// applyTar runs the incremental split → rollup → commit path against an
// in-memory tar, mirroring applyIncremental without the HTTP stream.
func applyTar(t *testing.T, si *SearchIndex, series int, tarBytes []byte) {
t.Helper()
counts, err := aggregateTarListens(context.Background(), bytes.NewReader(tarBytes))
if err != nil {
t.Fatalf("aggregate: %v", err)
}
rec, art, rel := splitCountsByKind(counts)
rg := si.rollupReleaseDeltas(rel)
if err := si.commitListenDeltas(series, rec, art, rg); err != nil {
t.Fatalf("commit series %d: %v", series, err)
}
}
func TestIncrementalApplyAdditive(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
si.upsertBatch([]SearchIndexResult{
{EntityType: "recording", MBID: recA, Title: "Rec A", Popularity: 100},
{EntityType: "artist", MBID: artA, Title: "Art A", Popularity: 100},
{EntityType: "release_group", MBID: rgA, Title: "RG A", Popularity: 100},
})
if _, err := db.ExecContext(
"INSERT INTO release_to_rg (release_mbid, rg_mbid) VALUES (?, ?)", relA, rgA,
); err != nil {
t.Fatalf("seed release_to_rg: %v", err)
}
// 5 listens of recA on relA credited to artA.
tarBytes := makeTar(t,
map[string][]byte{
"listens/1.parquet": makeParquet(t, listensOf(5, recA, relA, []string{artA})),
},
[]string{"listens/1.parquet"},
)
applyTar(t, si, 2, tarBytes)
for _, c := range []struct {
mbid string
want int
}{{recA, 105}, {artA, 105}, {rgA, 105}} {
if got, ok := popularityOf(t, db, c.mbid); !ok || got != c.want {
t.Errorf("popularity(%s) = %d (present=%v); want %d", c.mbid, got, ok, c.want)
}
}
if hwm, ok := si.metaInt(listensAppliedSeriesKey); !ok || hwm != 2 {
t.Errorf("high-water-mark = %d (present=%v); want 2", hwm, ok)
}
// A second, later dump accumulates additively.
applyTar(t, si, 3, tarBytes)
if got, _ := popularityOf(t, db, recA); got != 110 {
t.Errorf("popularity(recA) after second dump = %d; want 110", got)
}
if hwm, _ := si.metaInt(listensAppliedSeriesKey); hwm != 3 {
t.Errorf("high-water-mark after second dump = %d; want 3", hwm)
}
}
func TestIncrementalIgnoresUnknownAndUnmapped(t *testing.T) {
db := database.NewTestDB(t)
si := NewSearchIndex(db, nil, nil, testLogger())
// Only recA is indexed; relA has no release_to_rg mapping.
si.upsertBatch([]SearchIndexResult{
{EntityType: "recording", MBID: recA, Title: "Rec A", Popularity: 100},
})
// Listens for recB (not indexed) on relB (unmapped) credited to
// artB (not indexed). Nothing should change and no row created.
tarBytes := makeTar(t,
map[string][]byte{
"listens/1.parquet": makeParquet(t, listensOf(7, recB, relB, []string{artB})),
},
[]string{"listens/1.parquet"},
)
applyTar(t, si, 5, tarBytes)
if _, ok := popularityOf(t, db, recB); ok {
t.Error("recB should not have been inserted into the index")
}
if got, _ := popularityOf(t, db, recA); got != 100 {
t.Errorf("popularity(recA) = %d; want 100 (untouched)", got)
}
}
func TestDiscoverIncrementalDumps(t *testing.T) {
series := []string{"2594", "2595", "2596"}
mux := http.NewServeMux()
mux.HandleFunc("/incremental/", func(w http.ResponseWriter, r *http.Request) {
// Only the base listing (exact path); dir listings are handled
// by their own more-specific patterns below.
if r.URL.Path != "/incremental/" {
http.NotFound(w, r)
return
}
for _, s := range series {
_, _ = fmt.Fprintf(w,
`<a href="listenbrainz-dump-%s-20260713-000003-incremental/">dir</a>`+"\n", s,
)
}
})
for _, s := range series {
dir := fmt.Sprintf("/incremental/listenbrainz-dump-%s-20260713-000003-incremental/", s)
file := fmt.Sprintf("listenbrainz-spark-dump-%s-20260713-000003-incremental.tar", s)
mux.HandleFunc(dir, func(w http.ResponseWriter, _ *http.Request) {
_, _ = fmt.Fprintf(w, `<a href="%s">file</a>`, file)
})
}
srv := httptest.NewServer(mux)
defer srv.Close()
dumps, err := discoverIncrementalDumps(
context.Background(), srv.Client(), srv.URL+"/incremental/", 2594,
)
if err != nil {
t.Fatalf("discover: %v", err)
}
// Only 2595 and 2596 are newer than the 2594 high-water-mark.
if len(dumps) != 2 {
t.Fatalf("got %d dumps; want 2: %+v", len(dumps), dumps)
}
if dumps[0].series != 2595 || dumps[1].series != 2596 {
t.Errorf("series order wrong: %d, %d; want 2595, 2596", dumps[0].series, dumps[1].series)
}
if !strings.Contains(dumps[0].url, "spark-dump-2595") {
t.Errorf("unexpected url: %s", dumps[0].url)
}
}
backend/explore/dumppatch.go
+279
View File
@@ -0,0 +1,279 @@
package explore
import (
"context"
"strings"
)
// Stage 4 of the dump import: small, idempotent API patch passes that
// fill in what the dumps can't provide. All calls go through the
// shared rate-limited ListenBrainz client and its HTTP cache, so
// re-running after an interruption is cheap.
const (
// Listener-count patch budgets: only the most popular rows get
// listener counts (a secondary ranking signal). 1000 MBIDs per
// batched POST call → ~350 API calls total.
listenerPatchArtists = 50_000
listenerPatchRGs = 100_000
listenerPatchRecordings = 200_000
// popularityBatchSize is the number of MBIDs per LB popularity
// or metadata request. LB accepts up to 1000 per call.
popularityBatchSize = 1000
)
// runPatchPasses fills listener counts, artist metadata, and the
// similar-artist map from the ListenBrainz API.
func (imp *dumpImporter) runPatchPasses(ctx context.Context) {
if imp.lb == nil {
return
}
imp.patchArtistMetadata(ctx)
if ctx.Err() != nil {
return
}
imp.patchSimilarArtists(ctx)
if ctx.Err() != nil {
return
}
imp.patchListenerCounts(ctx)
}
// patchArtistMetadata batch-fetches type/country/name for indexed
// artists that are missing them. Also creates rows for kept artists
// whose name wasn't derivable from the canonical dump (multi-artist
// credits only).
func (imp *dumpImporter) patchArtistMetadata(ctx context.Context) {
rows, err := imp.si.db.QueryContext(`
SELECT mbid FROM explore_index
WHERE entity_type = 'artist'
AND (artist_type = '' OR country = '' OR title = '' OR title = mbid)
`)
if err != nil {
return
}
var mbids []string
for rows.Next() {
var m string
if err := rows.Scan(&m); err == nil {
mbids = append(mbids, m)
}
}
_ = rows.Close()
if len(imp.pendingArtists) > 0 {
mbids = append(mbids, imp.pendingArtists...)
}
if len(mbids) == 0 {
return
}
imp.logger.Info("dump import: patching artist metadata", "artists", len(mbids))
batches := chunkStrings(mbids, popularityBatchSize)
patched := 0
for i, batch := range batches {
if ctx.Err() != nil {
return
}
meta, err := imp.lb.BatchArtistMetadata(ctx, batch)
if err != nil || len(meta) == 0 {
continue
}
entries := make([]SearchIndexResult, 0, len(meta))
for mbid, m := range meta {
if m.Name == "" {
continue
}
entries = append(entries, SearchIndexResult{
EntityType: "artist",
MBID: mbid,
Title: m.Name,
ArtistName: m.Name,
ArtistMBID: mbid,
ArtistType: m.Type,
Country: m.Country,
})
// Pre-populate the MB rels cache so on-demand artist
// image resolution skips a MusicBrainz call.
if imp.si.artistImg != nil {
imp.si.artistImg.PreloadArtistRels(mbid, m)
}
}
imp.si.upsertBatch(entries)
patched += len(entries)
imp.setStageProgress(dumpStagePatch, i+1, len(batches))
}
imp.logger.Info("dump import: artist metadata patched", "artists", patched)
}
// patchSimilarArtists refreshes the similar-artist map for library
// artists (one API call per library artist, cached for a week).
func (imp *dumpImporter) patchSimilarArtists(ctx context.Context) {
libraryMBIDs := imp.si.getLibraryArtistMBIDs()
if len(libraryMBIDs) == 0 {
return
}
for i := 0; i < len(libraryMBIDs); i += similarArtistsBatchSize {
if ctx.Err() != nil {
return
}
end := min(i+similarArtistsBatchSize, len(libraryMBIDs))
grouped := imp.si.fetchSimilarArtistsBatch(ctx, imp.lb, libraryMBIDs[i:end])
for seed, similar := range grouped {
imp.si.storeSimilarArtists(seed, similar)
// Flag indexed similar artists for personalized ranking.
for _, s := range similar {
_, _ = imp.si.db.ExecContext(
"UPDATE explore_index SET is_similar = 1 WHERE artist_mbid = ?",
s.ArtistMBID,
)
}
}
}
imp.logger.Info("dump import: similar artists patched", "libraryArtists", len(libraryMBIDs))
}
// patchListenerCounts fills listener_count for the most popular rows
// of each entity type. Popularity (listen count) is NOT overwritten —
// the dump-derived counts stay authoritative so the ranking scale is
// consistent across the whole index.
func (imp *dumpImporter) patchListenerCounts(ctx context.Context) {
kinds := []struct {
entityType string
limit int
fetch func(context.Context, []string) (map[string]PopularityData, error)
}{
{"artist", listenerPatchArtists, imp.lb.ArtistPopularity},
{"release_group", listenerPatchRGs, imp.lb.ReleaseGroupPopularity},
{"recording", listenerPatchRecordings, imp.lb.RecordingPopularity},
}
for _, kind := range kinds {
if ctx.Err() != nil {
return
}
mbids := imp.topMBIDs(kind.entityType, kind.limit)
if len(mbids) == 0 {
continue
}
batches := chunkStrings(mbids, popularityBatchSize)
filled := 0
for i, batch := range batches {
if ctx.Err() != nil {
return
}
pops, err := kind.fetch(ctx, batch)
if err != nil {
continue
}
filled += imp.si.updateListenerCounts(pops)
imp.setStageProgress(dumpStageListeners, i+1, len(batches))
}
imp.logger.Info("dump import: listener counts patched",
"entityType", kind.entityType,
"rows", filled,
)
}
}
// topMBIDs returns the most popular index MBIDs for an entity type
// that don't have listener counts yet.
func (imp *dumpImporter) topMBIDs(entityType string, limit int) []string {
rows, err := imp.si.db.QueryContext(`
SELECT mbid FROM explore_index
WHERE entity_type = ? AND listener_count = 0
ORDER BY popularity DESC
LIMIT ?
`, entityType, limit)
if err != nil {
return nil
}
defer func() { _ = rows.Close() }()
var mbids []string
for rows.Next() {
var m string
if err := rows.Scan(&m); err == nil {
mbids = append(mbids, m)
}
}
return mbids
}
// updateListenerCounts writes listener counts only (never popularity),
// keeping the dump-derived popularity scale consistent. Returns the
// number of rows updated.
func (si *SearchIndex) updateListenerCounts(updates map[string]PopularityData) int {
if len(updates) == 0 {
return 0
}
tx, err := si.db.BeginTx()
if err != nil {
return 0
}
defer func() { _ = tx.Rollback() }()
updated := 0
for mbid, data := range updates {
if data.ListenerCount <= 0 {
continue
}
res, err := tx.Exec(
`UPDATE explore_index
SET listener_count = ?
WHERE mbid = ? AND listener_count < ?`,
data.ListenerCount, strings.ToLower(mbid), data.ListenerCount,
)
if err != nil {
continue
}
if n, err := res.RowsAffected(); err == nil {
updated += int(n)
}
}
_ = tx.Commit()
return updated
}
backend/explore/dumpstream.go
+329
View File
@@ -0,0 +1,329 @@
package explore
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"regexp"
"sort"
"strconv"
"time"
)
// Streaming helpers for the MetaBrainz dump imports. Dumps are never
// written to disk: the HTTP body is decoded (tar / zstd+tar) in flight.
// resumableReader reconnects with HTTP Range requests on transient
// failures, which also lets the listens import resume across app
// restarts from a checkpointed byte offset.
const (
// maxStreamRetries is the number of consecutive failed reconnect
// attempts before a stream read gives up. The counter resets
// whenever bytes are successfully delivered.
maxStreamRetries = 8
// streamRetryBaseDelay is the initial reconnect backoff; it
// doubles per consecutive failure.
streamRetryBaseDelay = 2 * time.Second
// dumpDiscoveryTimeout bounds the small directory-listing
// requests (not the multi-hour stream requests).
dumpDiscoveryTimeout = 30 * time.Second
)
// ErrDumpDiscovery is returned when a dump directory listing does not
// contain the expected entries.
var ErrDumpDiscovery = errors.New("dump discovery failed")
// ErrDumpStream is returned when a dump stream fails permanently.
var ErrDumpStream = errors.New("dump stream failed")
var hrefRe = regexp.MustCompile(`href="([^"?/][^"?]*)"`)
// listHrefs fetches an Apache-style index page and returns the href
// values (directory entries end with a trailing slash).
func listHrefs(ctx context.Context, client *http.Client, url string) ([]string, error) {
reqCtx, cancel := context.WithTimeout(ctx, dumpDiscoveryTimeout)
defer cancel()
req, err := http.NewRequestWithContext(reqCtx, http.MethodGet, url, nil)
if err != nil {
return nil, fmt.Errorf("dump listing request: %w", err)
}
req.Header.Set("User-Agent", lbUserAgent)
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("dump listing fetch: %w", err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf(
"%w: listing %s returned HTTP %d", ErrDumpDiscovery, url, resp.StatusCode,
)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, fmt.Errorf("dump listing read: %w", err)
}
var hrefs []string
for _, m := range hrefRe.FindAllStringSubmatch(string(body), -1) {
hrefs = append(hrefs, m[1])
}
return hrefs, nil
}
// discoverDumpFile walks a dump base directory, finds subdirectories
// matching dirRe (newest first, lexicographically — MetaBrainz dump
// directory names embed sortable timestamps), and returns the full URL
// of the first file inside matching fileRe. Directories that don't
// contain a matching file (e.g. partial uploads) are skipped.
func discoverDumpFile(
ctx context.Context,
client *http.Client,
baseURL string,
dirRe, fileRe *regexp.Regexp,
) (string, error) {
hrefs, err := listHrefs(ctx, client, baseURL)
if err != nil {
return "", err
}
var dirs []string
for _, h := range hrefs {
trimmed := trimTrailingSlash(h)
if dirRe.MatchString(trimmed) {
dirs = append(dirs, trimmed)
}
}
if len(dirs) == 0 {
return "", fmt.Errorf("%w: no dump directories under %s", ErrDumpDiscovery, baseURL)
}
sort.Sort(sort.Reverse(sort.StringSlice(dirs)))
for _, dir := range dirs {
dirURL := baseURL + dir + "/"
files, err := listHrefs(ctx, client, dirURL)
if err != nil {
continue
}
for _, f := range files {
if fileRe.MatchString(f) {
return dirURL + f, nil
}
}
}
return "", fmt.Errorf("%w: no matching dump file under %s", ErrDumpDiscovery, baseURL)
}
func trimTrailingSlash(s string) string {
if len(s) > 0 && s[len(s)-1] == '/' {
return s[:len(s)-1]
}
return s
}
// resumableReader is an io.Reader over an HTTP resource that survives
// connection failures by reconnecting with a Range request at the
// current offset. Offset is the absolute position of the next byte to
// deliver, so callers can checkpoint it and construct a new
// resumableReader later to resume a partially-processed stream.
type resumableReader struct {
ctx context.Context
client *http.Client
url string
// Offset is the absolute byte position of the next read.
Offset int64
// Size is the total resource size, learned from the first
// response. -1 until known.
Size int64
body io.ReadCloser
retries int
}
func newResumableReader(
ctx context.Context, client *http.Client, url string, offset int64,
) *resumableReader {
return &resumableReader{
ctx: ctx,
client: client,
url: url,
Offset: offset,
Size: -1,
}
}
func (r *resumableReader) Read(p []byte) (int, error) {
for {
if err := r.ctx.Err(); err != nil {
return 0, err
}
if r.body == nil {
if err := r.connect(); err != nil {
return 0, err
}
}
n, err := r.body.Read(p)
r.Offset += int64(n)
if n > 0 {
r.retries = 0
}
switch {
case err == nil:
return n, nil
case errors.Is(err, io.EOF):
// A server that closes early looks like EOF; only
// trust it when we've seen the advertised size.
if r.Size >= 0 && r.Offset < r.Size {
r.closeBody()
if retryErr := r.backoff(err); retryErr != nil {
return n, retryErr
}
if n > 0 {
return n, nil
}
continue
}
return n, io.EOF
default:
r.closeBody()
if retryErr := r.backoff(err); retryErr != nil {
return n, retryErr
}
if n > 0 {
return n, nil
}
}
}
}
// backoff sleeps with exponential backoff, or returns a terminal error
// once the retry budget is exhausted.
func (r *resumableReader) backoff(cause error) error {
r.retries++
if r.retries > maxStreamRetries {
return fmt.Errorf(
"%w: %s after %d retries: %w", ErrDumpStream, r.url, maxStreamRetries, cause,
)
}
delay := streamRetryBaseDelay << (r.retries - 1)
select {
case <-r.ctx.Done():
return r.ctx.Err()
case <-time.After(delay):
return nil
}
}
func (r *resumableReader) connect() error {
req, err := http.NewRequestWithContext(r.ctx, http.MethodGet, r.url, nil)
if err != nil {
return fmt.Errorf("dump stream request: %w", err)
}
req.Header.Set("User-Agent", lbUserAgent)
if r.Offset > 0 {
req.Header.Set("Range", "bytes="+strconv.FormatInt(r.Offset, 10)+"-")
}
resp, err := r.client.Do(req)
if err != nil {
return r.backoff(err)
}
switch resp.StatusCode {
case http.StatusPartialContent:
if r.Size < 0 {
r.Size = parseContentRangeTotal(resp.Header.Get("Content-Range"))
}
r.body = resp.Body
return nil
case http.StatusOK:
if r.Size < 0 && resp.ContentLength > 0 {
r.Size = resp.ContentLength
}
// Server ignored the Range header: discard the prefix so
// the caller still reads from the requested offset.
if r.Offset > 0 {
if _, err := io.CopyN(io.Discard, resp.Body, r.Offset); err != nil {
_ = resp.Body.Close()
return r.backoff(err)
}
}
r.body = resp.Body
return nil
default:
_ = resp.Body.Close()
return r.backoff(fmt.Errorf("%w: HTTP %d from %s", ErrDumpStream, resp.StatusCode, r.url))
}
}
func (r *resumableReader) closeBody() {
if r.body != nil {
_ = r.body.Close()
r.body = nil
}
}
// Close releases the underlying HTTP body, if any.
func (r *resumableReader) Close() error {
r.closeBody()
return nil
}
// parseContentRangeTotal extracts the total size from a Content-Range
// header ("bytes 100-199/12345"). Returns -1 if unavailable.
func parseContentRangeTotal(v string) int64 {
for i := len(v) - 1; i >= 0; i-- {
if v[i] == '/' {
total, err := strconv.ParseInt(v[i+1:], 10, 64)
if err != nil {
return -1
}
return total
}
}
return -1
}
backend/explore/eval/eval.go
+115
View File
@@ -0,0 +1,115 @@
// Package eval is the search-ranking evaluation harness. It turns
// "this query feels wrong" into a number that goes up or down, so a
// ranking change can be validated against a frozen set of labelled
// queries instead of tuned by anecdote.
//
// The harness is deliberately decoupled from the explore package: it
// knows nothing about MusicBrainz, ListenBrainz, or the search index.
// A caller adapts whatever ranking function it wants to measure to the
// Ranker interface, loads a fixture set, and runs Evaluate. The
// explore package wires its real index Search to this in an
// integration test (see explore/eval_harness_test.go).
package eval
import (
"encoding/json"
"errors"
"fmt"
"io"
"os"
"strings"
)
// ErrNoFixtures is returned when a fixture file contains zero queries.
var ErrNoFixtures = errors.New("eval: fixture set is empty")
// Result is one ranked search hit, reduced to the only two fields the
// harness needs to decide whether it matches an expectation.
type Result struct {
EntityType string `json:"entityType"`
MBID string `json:"mbid"`
}
// Ranker produces an ordered result list for a query. Best result
// first. Implemented by adapting a real search function.
type Ranker interface {
Rank(query string, limit int) []Result
}
// RankerFunc adapts a plain function to the Ranker interface.
type RankerFunc func(query string, limit int) []Result
// Rank calls the underlying function.
func (f RankerFunc) Rank(query string, limit int) []Result {
return f(query, limit)
}
// Expected is one acceptable result for a fixture query. Grade is the
// graded-relevance weight used by nDCG (higher = more relevant); it
// defaults to 1 when omitted. Type is optional — when set, a ranked
// result must match both MBID and entity type to count as a hit.
type Expected struct {
Type string `json:"type,omitempty"`
MBID string `json:"mbid"`
Grade int `json:"grade,omitempty"`
}
// Fixture is a single labelled query: the input plus the result(s) a
// user should get. Every edge case ever hand-fixed in the ranker
// belongs here so it can never silently regress.
type Fixture struct {
Query string `json:"query"`
Note string `json:"note,omitempty"`
Expect []Expected `json:"expect"`
}
// LoadFixtures reads a JSON fixture file from disk.
func LoadFixtures(path string) ([]Fixture, error) {
f, err := os.Open(path) //nolint:gosec // path is a test fixture, not user input
if err != nil {
return nil, fmt.Errorf("eval: open fixtures: %w", err)
}
defer func() { _ = f.Close() }()
return ParseFixtures(f)
}
// ParseFixtures decodes a JSON fixture set from a reader.
func ParseFixtures(r io.Reader) ([]Fixture, error) {
var fixtures []Fixture
if err := json.NewDecoder(r).Decode(&fixtures); err != nil {
return nil, fmt.Errorf("eval: decode fixtures: %w", err)
}
if len(fixtures) == 0 {
return nil, ErrNoFixtures
}
return fixtures, nil
}
// matches reports whether a ranked result satisfies an expectation.
// MBID match is required; entity type is checked only when the
// expectation pins one.
func (e Expected) matches(r Result) bool {
if !strings.EqualFold(e.MBID, r.MBID) {
return false
}
if e.Type != "" && !strings.EqualFold(e.Type, r.EntityType) {
return false
}
return true
}
// grade returns the graded-relevance weight, defaulting to 1.
func (e Expected) grade() int {
if e.Grade <= 0 {
return 1
}
return e.Grade
}
backend/explore/eval/metrics.go
+228
View File
@@ -0,0 +1,228 @@
package eval
import (
"fmt"
"math"
"sort"
"strconv"
"strings"
)
// QueryScore holds the per-query metrics for one fixture.
type QueryScore struct {
Query string
Note string
// BestRank is the 1-based rank of the highest-placed expected
// result, or 0 if none of the expected results appear in topK.
BestRank int
ReciprocalRank float64
PrecisionAtK float64
NDCGAtK float64
}
// Hit reports whether any expected result landed in topK.
func (q QueryScore) Hit() bool {
return q.BestRank > 0
}
// Report aggregates per-query scores into the numbers you watch across
// a ranking change: mean reciprocal rank, mean precision@k, mean
// nDCG@k, plus the list of queries that missed entirely.
type Report struct {
K int
NumQueries int
MRR float64
MeanPAtK float64
MeanNDCG float64
HitRate float64 // fraction of queries with any expected result in topK
Top1Rate float64 // fraction whose best expected result is rank 1
PerQuery []QueryScore
}
// Evaluate runs every fixture through the ranker and aggregates the
// results into a Report. topK bounds how deep a result can be and
// still count (a result at rank 20 helps no one).
func Evaluate(r Ranker, fixtures []Fixture, topK int) Report {
if topK <= 0 {
topK = 5
}
report := Report{K: topK, NumQueries: len(fixtures)}
for _, fx := range fixtures {
ranked := r.Rank(fx.Query, topK)
report.PerQuery = append(report.PerQuery, scoreQuery(fx, ranked, topK))
}
for _, q := range report.PerQuery {
report.MRR += q.ReciprocalRank
report.MeanPAtK += q.PrecisionAtK
report.MeanNDCG += q.NDCGAtK
if q.Hit() {
report.HitRate++
}
if q.BestRank == 1 {
report.Top1Rate++
}
}
if n := float64(len(fixtures)); n > 0 {
report.MRR /= n
report.MeanPAtK /= n
report.MeanNDCG /= n
report.HitRate /= n
report.Top1Rate /= n
}
return report
}
// scoreQuery computes the metrics for a single fixture against a ranked
// result list.
func scoreQuery(fx Fixture, ranked []Result, topK int) QueryScore {
score := QueryScore{Query: fx.Query, Note: fx.Note}
limit := min(topK, len(ranked))
relevantInK := 0
for i := range limit {
if !anyMatch(fx.Expect, ranked[i]) {
continue
}
relevantInK++
if score.BestRank == 0 {
score.BestRank = i + 1
score.ReciprocalRank = 1.0 / float64(i+1)
}
}
score.PrecisionAtK = float64(relevantInK) / float64(topK)
score.NDCGAtK = ndcg(fx.Expect, ranked, topK)
return score
}
// anyMatch reports whether a result satisfies any expectation.
func anyMatch(expected []Expected, r Result) bool {
for _, e := range expected {
if e.matches(r) {
return true
}
}
return false
}
// ndcg computes normalized discounted cumulative gain at k using graded
// relevance. Returns 0 when there are no expected results.
func ndcg(expected []Expected, ranked []Result, k int) float64 {
ideal := idealDCG(expected, k)
if ideal == 0 {
return 0
}
limit := min(k, len(ranked))
dcg := 0.0
for i := range limit {
g := matchedGrade(expected, ranked[i])
if g == 0 {
continue
}
dcg += gain(g, i)
}
return dcg / ideal
}
// matchedGrade returns the relevance grade for a result, or 0 if it
// matches no expectation.
func matchedGrade(expected []Expected, r Result) int {
for _, e := range expected {
if e.matches(r) {
return e.grade()
}
}
return 0
}
// idealDCG is the DCG of the best possible ordering: every expected
// result, sorted by grade descending, placed at the front.
func idealDCG(expected []Expected, k int) float64 {
grades := make([]int, 0, len(expected))
for _, e := range expected {
grades = append(grades, e.grade())
}
sort.Sort(sort.Reverse(sort.IntSlice(grades)))
limit := min(k, len(grades))
ideal := 0.0
for i := range limit {
ideal += gain(grades[i], i)
}
return ideal
}
// gain is the discounted gain of a grade at 0-based position i.
func gain(grade, i int) float64 {
return (math.Pow(2, float64(grade)) - 1) / math.Log2(float64(i+2))
}
// Format renders a Report as a human-readable table for test output.
func (r Report) Format() string {
var b strings.Builder
fmt.Fprintf(&b, "ranking eval — %d queries @k=%d\n", r.NumQueries, r.K)
fmt.Fprintf(&b, " MRR %.3f\n", r.MRR)
fmt.Fprintf(&b, " P@%d %.3f\n", r.K, r.MeanPAtK)
fmt.Fprintf(&b, " nDCG@%d %.3f\n", r.K, r.MeanNDCG)
fmt.Fprintf(&b, " hit rate %.3f\n", r.HitRate)
fmt.Fprintf(&b, " top-1 rate %.3f\n", r.Top1Rate)
misses := r.Misses()
if len(misses) > 0 {
b.WriteString(" misses:\n")
for _, m := range misses {
fmt.Fprintf(&b, " %-40q rank=%s\n", m.Query, rankLabel(m.BestRank))
}
}
return b.String()
}
// Misses returns the queries whose best expected result was absent
// from topK or buried below rank 1 — the regression watch-list.
func (r Report) Misses() []QueryScore {
var out []QueryScore
for _, q := range r.PerQuery {
if q.BestRank != 1 {
out = append(out, q)
}
}
return out
}
func rankLabel(rank int) string {
if rank == 0 {
return "absent"
}
return strconv.Itoa(rank)
}
backend/explore/eval/metrics_test.go
+193
View File
@@ -0,0 +1,193 @@
package eval
import (
"math"
"strings"
"testing"
)
// rankerFromIDs builds a Ranker that returns a fixed ordering keyed by
// query, for deterministic metric tests.
func rankerFromIDs(table map[string][]Result) Ranker {
return RankerFunc(func(query string, limit int) []Result {
out := table[query]
if limit < len(out) {
out = out[:limit]
}
return out
})
}
func approx(a, b float64) bool {
return math.Abs(a-b) < 1e-9
}
func TestReciprocalRank(t *testing.T) {
tests := []struct {
name string
ranked []Result
expect []Expected
wantRR float64
wantPos int
}{
{
name: "top result",
ranked: []Result{{MBID: "a"}, {MBID: "b"}},
expect: []Expected{{MBID: "a"}},
wantRR: 1.0,
wantPos: 1,
},
{
name: "third result",
ranked: []Result{{MBID: "x"}, {MBID: "y"}, {MBID: "a"}},
expect: []Expected{{MBID: "a"}},
wantRR: 1.0 / 3.0,
wantPos: 3,
},
{
name: "absent",
ranked: []Result{{MBID: "x"}, {MBID: "y"}},
expect: []Expected{{MBID: "a"}},
wantRR: 0,
wantPos: 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
fx := Fixture{Query: "q", Expect: tt.expect}
got := scoreQuery(fx, tt.ranked, 5)
if !approx(got.ReciprocalRank, tt.wantRR) {
t.Errorf("RR = %v, want %v", got.ReciprocalRank, tt.wantRR)
}
if got.BestRank != tt.wantPos {
t.Errorf("BestRank = %d, want %d", got.BestRank, tt.wantPos)
}
})
}
}
func TestPrecisionAtK(t *testing.T) {
fx := Fixture{
Query: "q",
Expect: []Expected{{MBID: "a"}, {MBID: "b"}},
}
ranked := []Result{{MBID: "a"}, {MBID: "x"}, {MBID: "b"}, {MBID: "y"}}
got := scoreQuery(fx, ranked, 4)
// 2 relevant out of k=4.
if !approx(got.PrecisionAtK, 0.5) {
t.Errorf("P@4 = %v, want 0.5", got.PrecisionAtK)
}
}
func TestNDCGRespectsOrdering(t *testing.T) {
expect := []Expected{{MBID: "a", Grade: 3}, {MBID: "b", Grade: 1}}
// Ideal ordering: high-grade result first.
good := scoreQuery(
Fixture{Query: "q", Expect: expect},
[]Result{{MBID: "a"}, {MBID: "b"}, {MBID: "z"}},
5,
)
// Worse ordering: high-grade result buried below an irrelevant one.
bad := scoreQuery(
Fixture{Query: "q", Expect: expect},
[]Result{{MBID: "z"}, {MBID: "b"}, {MBID: "a"}},
5,
)
if !approx(good.NDCGAtK, 1.0) {
t.Errorf("ideal ordering nDCG = %v, want 1.0", good.NDCGAtK)
}
if bad.NDCGAtK >= good.NDCGAtK {
t.Errorf("worse ordering nDCG %v should be < ideal %v", bad.NDCGAtK, good.NDCGAtK)
}
}
func TestTypeMustMatchWhenPinned(t *testing.T) {
fx := Fixture{
Query: "q",
Expect: []Expected{{Type: "artist", MBID: "a"}},
}
// Same MBID but wrong entity type — must not count.
wrongType := scoreQuery(fx, []Result{{EntityType: "recording", MBID: "a"}}, 5)
if wrongType.Hit() {
t.Error("result with wrong entity type counted as a hit")
}
rightType := scoreQuery(fx, []Result{{EntityType: "artist", MBID: "a"}}, 5)
if !rightType.Hit() {
t.Error("result with matching entity type did not count")
}
}
func TestEvaluateAggregates(t *testing.T) {
fixtures := []Fixture{
{Query: "hit-top", Expect: []Expected{{MBID: "a"}}},
{Query: "hit-second", Expect: []Expected{{MBID: "a"}}},
{Query: "miss", Expect: []Expected{{MBID: "a"}}},
}
r := rankerFromIDs(map[string][]Result{
"hit-top": {{MBID: "a"}},
"hit-second": {{MBID: "x"}, {MBID: "a"}},
"miss": {{MBID: "x"}, {MBID: "y"}},
})
report := Evaluate(r, fixtures, 5)
// MRR = (1 + 1/2 + 0) / 3.
wantMRR := (1.0 + 0.5 + 0.0) / 3.0
if !approx(report.MRR, wantMRR) {
t.Errorf("MRR = %v, want %v", report.MRR, wantMRR)
}
// 2 of 3 queries surfaced the result somewhere in topK.
if !approx(report.HitRate, 2.0/3.0) {
t.Errorf("HitRate = %v, want %v", report.HitRate, 2.0/3.0)
}
// Only 1 of 3 had it at rank 1.
if !approx(report.Top1Rate, 1.0/3.0) {
t.Errorf("Top1Rate = %v, want %v", report.Top1Rate, 1.0/3.0)
}
if len(report.Misses()) != 2 {
t.Errorf("Misses = %d, want 2", len(report.Misses()))
}
}
func TestParseFixtures(t *testing.T) {
const doc = `[
{"query": "radiohead", "expect": [{"type": "artist", "mbid": "abc"}]},
{"query": "ok computer", "note": "album not band", "expect": [{"mbid": "def", "grade": 2}]}
]`
fixtures, err := ParseFixtures(strings.NewReader(doc))
if err != nil {
t.Fatalf("ParseFixtures: %v", err)
}
if len(fixtures) != 2 {
t.Fatalf("got %d fixtures, want 2", len(fixtures))
}
if fixtures[0].Expect[0].MBID != "abc" {
t.Errorf("MBID = %q, want abc", fixtures[0].Expect[0].MBID)
}
}
func TestParseFixturesEmpty(t *testing.T) {
_, err := ParseFixtures(strings.NewReader(`[]`))
if err == nil {
t.Fatal("expected ErrNoFixtures, got nil")
}
}
backend/explore/eval/testdata/eval_queries.json
Vendored
+32
View File
@@ -0,0 +1,32 @@
[
{
"query": "radiohead",
"note": "single-word artist name — should resolve to the artist, not an album titled similarly",
"expect": [{ "type": "artist", "mbid": "a74b1b7f-71a5-4011-9441-d0b5e4122711" }]
},
{
"query": "the beatles",
"note": "common-word prefix must not let the article dominate",
"expect": [{ "type": "artist", "mbid": "b10bbbfc-cf9e-42e0-be17-e2c3e1d2600d" }]
},
{
"query": "abbey road",
"note": "album title — should rank the release group above any track of the same name",
"expect": [{ "type": "release_group", "mbid": "" }]
},
{
"query": "calling you blue october",
"note": "composite title+artist query — recording should win even if the artist is unindexed",
"expect": [{ "type": "recording", "mbid": "" }]
},
{
"query": "the teenagers",
"note": "regression: must rank The Teenagers above The Beatles despite far lower popularity",
"expect": [{ "type": "artist", "mbid": "" }]
},
{
"query": "beyonce",
"note": "diacritic folding (migration 37): unaccented query must find the accented artist Beyoncé",
"expect": [{ "type": "artist", "mbid": "859d0860-d480-4efd-970c-c05d5f1776b8" }]
}
]
backend/explore/eval_harness_test.go
+150
View File
@@ -0,0 +1,150 @@
package explore
import (
"context"
"log/slog"
"testing"
"yellowjacket/backend/database"
"yellowjacket/backend/explore/eval"
)
// seedIndexRow inserts one explore_index row. The FTS triggers keep
// explore_index_fts in sync automatically.
func seedIndexRow(
t *testing.T,
db *database.DB,
entityType, mbid, title, artist string,
popularity int,
) {
t.Helper()
_, err := db.ExecContext(`
INSERT INTO explore_index
(entity_type, mbid, title, artist_name, artist_mbid, popularity, listener_count)
VALUES (?, ?, ?, ?, '', ?, ?)
`, entityType, mbid, title, artist, popularity, popularity/10)
if err != nil {
t.Fatalf("seed %s/%s: %v", entityType, mbid, err)
}
}
// newTestIndex builds a SearchIndex over a seeded in-memory DB. lb and
// artistImg are nil because Search touches neither.
func newTestIndex(t *testing.T, db *database.DB) *SearchIndex {
t.Helper()
idx := NewSearchIndex(db, nil, nil, slog.Default())
idx.MarkReadyIfPopulated()
return idx
}
// TestEvalHarnessIndexRanking is the end-to-end wiring of the eval
// harness against the real FTS index Search. It seeds a controlled
// corpus where the correct answer is known, then asserts the harness
// reports a perfect score — proving both the index ranking and the
// harness plumbing on a case we fully control.
func TestEvalHarnessIndexRanking(t *testing.T) {
db := database.NewTestDB(t)
// Popular exact-match artist should beat a more obscure namesake
// and an unrelated album.
seedIndexRow(t, db, "artist", "rh", "Radiohead", "Radiohead", 5_000_000)
seedIndexRow(t, db, "artist", "radio-obscure", "Radio Birdman", "Radio Birdman", 40_000)
seedIndexRow(t, db, "release_group", "okc", "OK Computer", "Radiohead", 2_000_000)
seedIndexRow(t, db, "artist", "beatles", "The Beatles", "The Beatles", 9_000_000)
seedIndexRow(t, db, "artist", "teenagers", "The Teenagers", "The Teenagers", 60_000)
idx := newTestIndex(t, db)
ranker := eval.RankerFunc(func(query string, limit int) []eval.Result {
hits := idx.Search(context.Background(), query, limit)
out := make([]eval.Result, 0, len(hits))
for _, h := range hits {
out = append(out, eval.Result{EntityType: h.EntityType, MBID: h.MBID})
}
return out
})
fixtures := []eval.Fixture{
{
Query: "radiohead",
Note: "popular exact artist match",
Expect: []eval.Expected{{Type: "artist", MBID: "rh"}},
},
{
Query: "the teenagers",
Note: "low-popularity exact match must beat high-popularity article match",
Expect: []eval.Expected{{Type: "artist", MBID: "teenagers"}},
},
}
report := eval.Evaluate(ranker, fixtures, 5)
t.Log("\n" + report.Format())
if report.HitRate < 1.0 {
t.Errorf("expected every query to surface its result, got hit rate %.3f", report.HitRate)
}
if report.Top1Rate < 1.0 {
t.Errorf("expected every result at rank 1, got top-1 rate %.3f:\n%s",
report.Top1Rate, report.Format())
}
}
// TestExploreFTSDiacriticFolding proves migration 37: an unaccented
// query must find an accented title (and vice versa) now that
// explore_index_fts folds diacritics.
func TestExploreFTSDiacriticFolding(t *testing.T) {
db := database.NewTestDB(t)
seedIndexRow(t, db, "artist", "bey", "Beyoncé", "Beyoncé", 8_000_000)
seedIndexRow(t, db, "artist", "bjork", "Björk", "Björk", 3_000_000)
idx := newTestIndex(t, db)
cases := []struct {
query string
wantMBID string
}{
{"beyonce", "bey"}, // unaccented query → accented title
{"beyoncé", "bey"}, // accented query still works
{"bjork", "bjork"}, // ö → o folding
{"björk", "bjork"}, // accented query still works
}
for _, tc := range cases {
t.Run(tc.query, func(t *testing.T) {
hits := idx.Search(context.Background(), tc.query, 5)
if len(hits) == 0 {
t.Fatalf("query %q returned no hits", tc.query)
}
if hits[0].MBID != tc.wantMBID {
t.Errorf("query %q: top hit = %q, want %q", tc.query, hits[0].MBID, tc.wantMBID)
}
})
}
}
// TestEvalFixtureFileParses guards the checked-in fixture file so a
// malformed edit fails fast rather than silently skipping queries.
func TestEvalFixtureFileParses(t *testing.T) {
fixtures, err := eval.LoadFixtures("eval/testdata/eval_queries.json")
if err != nil {
t.Fatalf("load fixtures: %v", err)
}
for i, fx := range fixtures {
if fx.Query == "" {
t.Errorf("fixture %d has empty query", i)
}
if len(fx.Expect) == 0 {
t.Errorf("fixture %q has no expectations", fx.Query)
}
}
}
backend/explore/explore.go
+726 -1416
View File
File diff suppressed because it is too large Load Diff
backend/explore/fuzzy.go
+58
View File
@@ -0,0 +1,58 @@
package explore
import "strings"
// Character-bigram similarity, used by the search index's typo-tolerant
// rescue pass (see fuzzyRescue). A name is represented as the set of its
// sliding 2-rune windows; a single-character typo alters only the one or
// two bigrams that span it, so most of the set survives. Overlap between
// two such sets (Dice coefficient) therefore stays high across a
// misspelling, where prefix matching collapses to nothing.
// fuzzyNormalize lowercases and collapses runs of whitespace to a single
// space so bigram sets are stable across casing and spacing noise.
func fuzzyNormalize(s string) string {
return strings.Join(strings.Fields(strings.ToLower(s)), " ")
}
// fuzzyBigrams returns the set of character bigrams of s after
// normalization. Returns nil for inputs shorter than two runes, which
// have no bigram and can't be scored.
func fuzzyBigrams(s string) map[string]struct{} {
runes := []rune(fuzzyNormalize(s))
if len(runes) < 2 {
return nil
}
set := make(map[string]struct{}, len(runes))
for i := 0; i+1 < len(runes); i++ {
set[string(runes[i:i+2])] = struct{}{}
}
return set
}
// diceCoefficient is 2·|A∩B| / (|A|+|B|), a similarity in [0, 1] where 1
// is an identical bigram set and 0 is disjoint. Iterating the smaller
// set keeps the intersection count cheap.
func diceCoefficient(a, b map[string]struct{}) float64 {
if len(a) == 0 || len(b) == 0 {
return 0
}
small, large := a, b
if len(large) < len(small) {
small, large = large, small
}
intersection := 0
for bg := range small {
if _, ok := large[bg]; ok {
intersection++
}
}
return 2 * float64(intersection) / float64(len(a)+len(b))
}
backend/explore/fuzzy_test.go
+43
View File
@@ -0,0 +1,43 @@
package explore
import "testing"
func TestDiceCoefficientTypoTolerance(t *testing.T) {
tests := []struct {
name string
a, b string
wantMin float64 // score must be at least this
wantMax float64 // ...and at most this
}{
{name: "identical", a: "beatles", b: "beatles", wantMin: 1.0, wantMax: 1.0},
{name: "single typo", a: "beetles", b: "beatles", wantMin: 0.5, wantMax: 0.9},
{name: "missing char", a: "nirvna", b: "nirvana", wantMin: 0.5, wantMax: 0.95},
// Longer-name typo: a single substitution stays comfortably above
// the rescue threshold, which is the realistic case (artist and
// album names are rarely as short as five characters).
{name: "longer name typo", a: "metalica", b: "metallica", wantMin: 0.5, wantMax: 0.95},
// Adjacent transposition in a short word is bigrams' known weak
// spot — it breaks most windows, so it scores below threshold.
// Documented, not a bug: bigram overlap targets substitution,
// insertion, and deletion typos.
{name: "short transposition", a: "raido", b: "radio", wantMin: 0.0, wantMax: 0.34},
{name: "case and space", a: "The Beatles", b: "the beatles", wantMin: 1.0, wantMax: 1.0},
{name: "unrelated", a: "beatles", b: "metallica", wantMin: 0.0, wantMax: 0.34},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := diceCoefficient(fuzzyBigrams(tc.a), fuzzyBigrams(tc.b))
if got < tc.wantMin || got > tc.wantMax {
t.Errorf("diceCoefficient(%q, %q) = %.3f, want in [%.2f, %.2f]",
tc.a, tc.b, got, tc.wantMin, tc.wantMax)
}
})
}
}
func TestFuzzyBigramsShortInput(t *testing.T) {
if got := fuzzyBigrams("a"); got != nil {
t.Errorf("fuzzyBigrams(%q) = %v, want nil", "a", got)
}
}
backend/explore/lrclib.go
+189
View File
@@ -0,0 +1,189 @@
package explore
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/url"
"strconv"
"time"
)
const (
// lrclibBaseURL is the LRCLIB lyrics API. LRCLIB is a free,
// community-maintained lyrics database (plain + synced) with no
// auth required.
lrclibBaseURL = "https://lrclib.net"
// lrclibUserAgent identifies the app per LRCLIB's guidelines.
lrclibUserAgent = "YellowJacket (https://github.com/yellowjacket)"
// lrclibRate is the requests-per-second budget for LRCLIB. The
// service has no hard published limit but asks clients to be
// gentle; this keeps the library backfill polite.
lrclibRate = 3
)
// ErrLyricsNotFound is returned when LRCLIB has no match for the
// requested track.
var ErrLyricsNotFound = errors.New("lyrics not found")
// Lyrics holds the plain and (optional) time-synced lyrics for a
// track, plus whether the track is marked instrumental.
type Lyrics struct {
Plain string `json:"plain"`
Synced string `json:"synced"`
Instrumental bool `json:"instrumental"`
}
// LRCLibClient is a thin, rate-limited, cached HTTP client for the
// LRCLIB lyrics API.
type LRCLibClient struct {
http *http.Client
limiter *RateLimiter
cache *Cache
logger *slog.Logger
baseURL string // overridable in tests
}
// NewLRCLibClient creates an LRCLIB client sharing the given cache.
func NewLRCLibClient(cache *Cache, logger *slog.Logger) *LRCLibClient {
return &LRCLibClient{
http: &http.Client{Timeout: 20 * time.Second},
limiter: NewRateLimiterN(lrclibRate),
cache: cache,
logger: logger,
baseURL: lrclibBaseURL,
}
}
// lrclibResponse is the wire shape of LRCLIB's /api/get response.
type lrclibResponse struct {
ID int64 `json:"id"`
TrackName string `json:"trackName"`
ArtistName string `json:"artistName"`
AlbumName string `json:"albumName"`
Duration float64 `json:"duration"`
Instrumental bool `json:"instrumental"`
PlainLyrics string `json:"plainLyrics"`
SyncedLyrics string `json:"syncedLyrics"`
}
// GetLyrics fetches lyrics for a track by artist, title, album, and
// duration (seconds; pass 0 if unknown). LRCLIB matches on the
// metadata with a small duration tolerance. Returns ErrLyricsNotFound
// when no match exists. Successful and negative results are both
// cached so a repeated backfill doesn't re-hit the network.
func (c *LRCLibClient) GetLyrics(
ctx context.Context, artist, title, album string, durationSec int,
) (*Lyrics, error) {
if artist == "" || title == "" {
return nil, ErrLyricsNotFound
}
q := url.Values{}
q.Set("artist_name", artist)
q.Set("track_name", title)
if album != "" {
q.Set("album_name", album)
}
if durationSec > 0 {
q.Set("duration", strconv.Itoa(durationSec))
}
reqURL := c.baseURL + "/api/get?" + q.Encode()
cacheKey := "lrclib:get:" + q.Encode()
if data, ok := c.cache.Get(cacheKey); ok {
return decodeCachedLyrics(data)
}
body, status, err := c.doGet(ctx, reqURL)
if err != nil {
return nil, fmt.Errorf("lrclib get: %w", err)
}
if status == http.StatusNotFound {
// Cache the miss as a sentinel so we don't re-request it.
c.cache.Set(cacheKey, []byte(lyricsMissSentinel), cacheTTLSearch, "", "lyrics")
return nil, ErrLyricsNotFound
}
if status < 200 || status >= 300 {
return nil, fmt.Errorf("lrclib get: %w: status %d", ErrListenBrainzHTTP, status)
}
var wire lrclibResponse
if err := json.Unmarshal(body, &wire); err != nil {
return nil, fmt.Errorf("lrclib get unmarshal: %w", err)
}
lyrics := &Lyrics{
Plain: wire.PlainLyrics,
Synced: wire.SyncedLyrics,
Instrumental: wire.Instrumental,
}
// Persist the normalized result (not the raw wire body) so the
// cached shape matches what callers expect.
if encoded, err := json.Marshal(lyrics); err == nil {
c.cache.Set(cacheKey, encoded, cacheTTLSearch, "", "lyrics")
}
return lyrics, nil
}
// lyricsMissSentinel marks a cached "no lyrics found" result.
const lyricsMissSentinel = "\x00miss"
// decodeCachedLyrics interprets a cached LRCLIB payload, mapping the
// miss sentinel back to ErrLyricsNotFound.
func decodeCachedLyrics(data []byte) (*Lyrics, error) {
if string(data) == lyricsMissSentinel {
return nil, ErrLyricsNotFound
}
var lyrics Lyrics
if err := json.Unmarshal(data, &lyrics); err != nil {
return nil, fmt.Errorf("lrclib cache decode: %w", err)
}
return &lyrics, nil
}
// doGet performs a rate-limited GET and returns the body and status.
// Unlike the ListenBrainz client, a 404 is a normal "no lyrics"
// outcome, so the status is returned rather than folded into an error.
func (c *LRCLibClient) doGet(ctx context.Context, reqURL string) ([]byte, int, error) {
if err := c.limiter.Wait(ctx); err != nil {
return nil, 0, fmt.Errorf("rate limiter: %w", err)
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil)
if err != nil {
return nil, 0, err
}
req.Header.Set("User-Agent", lrclibUserAgent)
resp, err := c.http.Do(req)
if err != nil {
return nil, 0, err
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, resp.StatusCode, err
}
return body, resp.StatusCode, nil
}
backend/explore/lrclib_test.go
+127
View File
@@ -0,0 +1,127 @@
package explore
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"testing"
"yellowjacket/backend/database"
)
// newTestLRCLib builds an LRCLIB client whose HTTP calls hit the given
// test server, sharing a real (in-memory) cache so caching behaviour is
// exercised.
func newTestLRCLib(t *testing.T, handler http.HandlerFunc) (*LRCLibClient, *httptest.Server) {
t.Helper()
srv := httptest.NewServer(handler)
t.Cleanup(srv.Close)
db := database.NewTestDB(t)
cache := NewCache(db, testLogger())
c := NewLRCLibClient(cache, testLogger())
// Point the client at the test server instead of the real API by
// overriding its transport to rewrite the host.
c.http = srv.Client()
c.baseURL = srv.URL
return c, srv
}
func TestLRCLibGetLyrics(t *testing.T) {
t.Parallel()
var calls int
c, _ := newTestLRCLib(t, func(w http.ResponseWriter, r *http.Request) {
calls++
if r.URL.Path != "/api/get" {
http.Error(w, "not found", http.StatusNotFound)
return
}
if r.URL.Query().Get("track_name") == "Missing" {
http.Error(w, "not found", http.StatusNotFound)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"id": 42,
"trackName": "Yesterday",
"artistName": "The Beatles",
"albumName": "Help!",
"duration": 125,
"instrumental": false,
"plainLyrics": "Yesterday, all my troubles seemed so far away",
"syncedLyrics": "[00:00.00] Yesterday"
}`))
})
t.Run("hit returns plain + synced lyrics", func(t *testing.T) {
lyrics, err := c.GetLyrics(context.Background(), "The Beatles", "Yesterday", "Help!", 125)
if err != nil {
t.Fatalf("GetLyrics: %v", err)
}
if lyrics.Plain == "" || lyrics.Synced == "" {
t.Errorf("expected plain and synced lyrics, got %+v", lyrics)
}
if lyrics.Instrumental {
t.Error("expected non-instrumental")
}
})
t.Run("second identical request is served from cache", func(t *testing.T) {
before := calls
if _, err := c.GetLyrics(
context.Background(),
"The Beatles",
"Yesterday",
"Help!",
125,
); err != nil {
t.Fatalf("GetLyrics: %v", err)
}
if calls != before {
t.Errorf("expected cache hit (no new HTTP call), calls went %d → %d", before, calls)
}
})
t.Run("404 maps to ErrLyricsNotFound", func(t *testing.T) {
_, err := c.GetLyrics(context.Background(), "Nobody", "Missing", "", 0)
if !errors.Is(err, ErrLyricsNotFound) {
t.Errorf("expected ErrLyricsNotFound, got %v", err)
}
})
t.Run("missing artist/title short-circuits without a request", func(t *testing.T) {
before := calls
if _, err := c.GetLyrics(
context.Background(),
"",
"Yesterday",
"",
0,
); !errors.Is(
err,
ErrLyricsNotFound,
) {
t.Errorf("expected ErrLyricsNotFound, got %v", err)
}
if calls != before {
t.Error("expected no HTTP call for empty artist")
}
})
}
backend/explore/lyrics.go
+210
View File
@@ -0,0 +1,210 @@
package explore
import (
"context"
"errors"
"yellowjacket/backend/database"
)
// LyricsResult is a single lyric-search hit, mapped from the DB layer
// into the camelCase shape the frontend consumes.
type LyricsResult struct {
RecordingID int64 `json:"recordingId"`
FilePath string `json:"filePath"`
LengthMs int64 `json:"lengthMs"`
Title string `json:"title"`
Artist string `json:"artist"`
Album string `json:"album"`
}
// TrackLyrics is the stored or freshly-fetched lyrics for one track.
// Source is "embedded" (from the file's tags / library DB), "lrclib"
// (fetched on demand), or "" when none are available.
type TrackLyrics struct {
Plain string `json:"plain"`
Synced string `json:"synced"`
Instrumental bool `json:"instrumental"`
Source string `json:"source"`
}
const (
// lyricsSearchLimit caps lyric-search hits returned to the UI.
lyricsSearchLimit = 30
// lyricsBackfillBatch is how many missing-lyrics recordings the
// background backfill processes per pass.
lyricsBackfillBatch = 200
// lyricsBackfillMaxPasses bounds a single backfill run so it can't
// loop forever on a huge library; the next launch resumes where
// this one left off (candidates with lyrics now filled are skipped).
lyricsBackfillMaxPasses = 25
)
// SearchLyrics finds library tracks whose lyrics contain the given
// fragment, ranked by relevance. Pure local FTS — no network.
func (e *Service) SearchLyrics(query string) []LyricsResult {
hits, err := e.db.SearchLyrics(query, lyricsSearchLimit)
if err != nil {
e.logger.Warn("lyrics search failed", "query", query, "err", err)
return nil
}
out := make([]LyricsResult, 0, len(hits))
for _, h := range hits {
out = append(out, LyricsResult{
RecordingID: h.RecordingID,
FilePath: h.FilePath,
LengthMs: h.LengthMilliseconds,
Title: h.Title,
Artist: h.Artist,
Album: h.Album,
})
}
return out
}
// GetTrackLyrics returns lyrics for a recording. If the library
// already has them (from embedded tags) they're returned as-is;
// otherwise it fetches from LRCLIB, persists them (updating the FTS
// index), and returns them. Never returns an error to the frontend —
// a miss just yields an empty result.
func (e *Service) GetTrackLyrics(recordingID int64) TrackLyrics {
stored, err := e.db.GetRecordingLyrics(recordingID)
if err == nil && stored != "" {
return TrackLyrics{Plain: stored, Source: "embedded"}
}
lookup, err := e.db.RecordingLyricLookup(recordingID)
if err != nil || lookup == nil {
return TrackLyrics{}
}
fetched := e.fetchAndStoreLyrics(e.ctx, *lookup)
if fetched == nil {
return TrackLyrics{}
}
return TrackLyrics{
Plain: fetched.Plain,
Synced: fetched.Synced,
Instrumental: fetched.Instrumental,
Source: "lrclib",
}
}
// RebuildLyricsIndex rebuilds the FTS lyrics index from the current
// library. Cheap; safe to call after every scan.
func (e *Service) RebuildLyricsIndex() {
if err := e.db.RebuildLyricsIndex(); err != nil {
e.logger.Warn("lyrics index rebuild failed", "err", err)
return
}
e.index.setMeta(lyricsIndexReadyKey, "1")
e.logger.Info("lyrics index rebuilt")
}
// RebuildLyricsIndexIfNeeded rebuilds the lyrics FTS only when it has not
// been built since the last library change. The backfill keeps the index
// in sync incrementally thereafter, so on an unchanged library the full
// rebuild is redundant; the scan-completion path calls the unconditional
// form.
func (e *Service) RebuildLyricsIndexIfNeeded() {
if e.index.hasMeta(lyricsIndexReadyKey) {
return
}
e.RebuildLyricsIndex()
}
// BackfillLibraryLyrics fetches lyrics from LRCLIB for library tracks
// that don't have them, in the background. Idempotent and bounded —
// each recording is tried once (a miss is cached), and a run stops
// after a fixed number of passes, resuming on the next launch.
func (e *Service) BackfillLibraryLyrics() {
go e.backfillLibraryLyrics(e.ctx)
}
func (e *Service) backfillLibraryLyrics(ctx context.Context) {
total := 0
for range lyricsBackfillMaxPasses {
if ctx.Err() != nil {
return
}
candidates, err := e.db.RecordingsMissingLyrics(lyricsBackfillBatch)
if err != nil {
e.logger.Warn("lyrics backfill: query failed", "err", err)
return
}
if len(candidates) == 0 {
break
}
filled := 0
for _, c := range candidates {
if ctx.Err() != nil {
return
}
if e.fetchAndStoreLyrics(ctx, c) != nil {
filled++
total++
}
}
// If a whole batch produced no stored lyrics, every remaining
// candidate is a cached miss with no new data — stop early
// rather than spinning through identical misses.
if filled == 0 {
break
}
}
if total > 0 {
e.logger.Info("lyrics backfill complete", "filled", total)
}
}
// fetchAndStoreLyrics looks up a single candidate on LRCLIB and, on a
// non-instrumental hit with plain lyrics, persists it to the recording
// (which also updates the FTS index). Returns the fetched lyrics, or
// nil on any miss/error. Instrumental hits are recorded as an empty
// lyrics string so they still count as "resolved" and aren't retried.
func (e *Service) fetchAndStoreLyrics(
ctx context.Context, c database.LyricsCandidate,
) *Lyrics {
durationSec := int(c.LengthMilliseconds / 1000) //nolint:mnd
lyrics, err := e.lrclib.GetLyrics(ctx, c.Artist, c.Title, c.Album, durationSec)
if err != nil {
if !errors.Is(err, ErrLyricsNotFound) {
e.logger.Debug("lyrics fetch failed",
"artist", c.Artist, "title", c.Title, "err", err,
)
}
return nil
}
if lyrics.Instrumental || lyrics.Plain == "" {
return nil
}
if err := e.db.SetRecordingLyrics(c.RecordingID, lyrics.Plain); err != nil {
e.logger.Warn("lyrics store failed", "recordingId", c.RecordingID, "err", err)
return nil
}
return lyrics
}
backend/explore/mergeindexhits_test.go
+121
View File
@@ -0,0 +1,121 @@
package explore
import "testing"
// TestMergeIndexHitsBlendedScale is the regression test for the
// merge→filter scale mismatch. An exact-match, in-library album that
// exists only in the local index (not returned by MB) with very low
// popularity must survive filterAndCap and outrank a weakly-matching MB
// result — because index hits are now scored on the same blended scale.
//
// Under the previous behaviour the index hit was scored as
// scalePopularity(50)*0.5 ≈ 12, below minBlendedScore (15), so
// filterAndCap dropped it entirely.
func TestMergeIndexHitsBlendedScale(t *testing.T) {
// A weakly-matching MB release group, already reranked: ~0.35
// relevance, no popularity → blended Score ≈ 35.
result := MBSearchResult{
ReleaseGroups: []MBReleaseGroup{
{MBID: "mb-weak", Title: "Live At Wembley", Score: 35, Popularity: 0},
},
}
// An exact-match, in-library album from the local index with only
// 50 listens.
hits := []SearchIndexResult{
{
EntityType: "release_group",
MBID: "idx-exact",
Title: "Abbey Road",
ArtistName: "The Beatles",
Popularity: 50,
InLibrary: true,
},
}
mergeIndexHits("abbey road", &result, hits)
if got := result.ReleaseGroups[0].MBID; got != "idx-exact" {
t.Fatalf("expected exact in-library index hit to rank first, got %q (score %d)",
got, result.ReleaseGroups[0].Score)
}
idxScore := result.ReleaseGroups[0].Score
if idxScore < minBlendedScore {
t.Errorf("index hit score %d below minBlendedScore %d — would be filtered out",
idxScore, minBlendedScore)
}
// It must survive the filter that previously dropped it.
filterAndCap(&result)
if len(result.ReleaseGroups) == 0 || result.ReleaseGroups[0].MBID != "idx-exact" {
t.Fatalf("index hit did not survive filterAndCap: %+v", result.ReleaseGroups)
}
}
// TestMergeIndexHitsSortsRecordings guards the recordings path, which
// previously had no post-merge sort: a high-scoring index recording was
// merged but its position depended on prepend order, not score.
func TestMergeIndexHitsSortsRecordings(t *testing.T) {
result := MBSearchResult{
Recordings: []MBRecording{
{MBID: "mb-a", Title: "Filler", Score: 60, Popularity: 100},
{MBID: "mb-b", Title: "Filler Two", Score: 20, Popularity: 50},
},
}
hits := []SearchIndexResult{
{
EntityType: "recording",
MBID: "idx-exact",
Title: "Calling You",
Popularity: 100_000,
InLibrary: true,
},
}
mergeIndexHits("calling you", &result, hits)
// Exact + in-library + decent popularity should land on top, and
// the whole list must be in descending Score order.
if result.Recordings[0].MBID != "idx-exact" {
t.Errorf("exact in-library recording should rank first, got %q", result.Recordings[0].MBID)
}
for i := 1; i < len(result.Recordings); i++ {
if result.Recordings[i-1].Score < result.Recordings[i].Score {
t.Errorf("recordings not sorted by score descending: %+v", result.Recordings)
break
}
}
}
func TestIndexHitRelevanceTiers(t *testing.T) {
const q = "abbey road"
exact := indexHitRelevance(q, "Abbey Road", "")
prefix := indexHitRelevance(q, "Abbey Road Sessions", "")
word := indexHitRelevance(q, "Live: Abbey Road Medley", "")
none := indexHitRelevance(q, "Something Unrelated", "")
if !(exact > prefix && prefix > word && word >= indexRelevanceFloor) {
t.Errorf("relevance tiers not ordered: exact=%v prefix=%v word=%v", exact, prefix, word)
}
if exact != indexRelExact {
t.Errorf("exact relevance = %v, want %v", exact, indexRelExact)
}
// A non-matching title still gets the floor (FTS matched something).
if none != indexRelevanceFloor {
t.Errorf("non-match relevance = %v, want floor %v", none, indexRelevanceFloor)
}
// Artist-credit match should count when the title doesn't.
artistMatch := indexHitRelevance("the beatles", "Abbey Road", "The Beatles")
if artistMatch != indexRelExact {
t.Errorf("artist-credit exact match = %v, want %v", artistMatch, indexRelExact)
}
}
backend/explore/musicbrainz.go
+129 -8
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"log/slog"
"strings"
"time"
"unicode"
@@ -18,6 +19,12 @@ const (
// cacheTTLEntity is the TTL for lookup/browse results (entity data
// changes rarely).
cacheTTLEntity = 7 * 24 * time.Hour
// cacheTTLReleases is the TTL for a release group's releases +
// tracklists. This data is effectively immutable once published, so
// it's cached far longer than other entities: it's the local store
// that keeps an album page's cold fetch a once-per-quarter event
// rather than a weekly one.
cacheTTLReleases = 90 * 24 * time.Hour
)
// MusicBrainzClient wraps the musicbrainzws2 library with a local
@@ -360,15 +367,27 @@ func (c *MusicBrainzClient) LookupRelease(
return &out, nil
}
// BrowseReleases fetches the releases for a given release group
// MBID, including media/track information. Cached for 7 days.
func (c *MusicBrainzClient) BrowseReleases(
ctx context.Context, releaseGroupMBID string,
) ([]MBRelease, error) {
cacheKey := "mb:browse:releases:" + releaseGroupMBID
// MBRecordingRelease is a slim reference to one release a recording
// appears on — enough for the autotagger to pick a representative
// release and then LookupRelease it in full.
type MBRecordingRelease struct {
MBID string `json:"mbid"`
Title string `json:"title"`
Status string `json:"status"`
Date string `json:"date"`
}
// LookupRecordingReleases fetches the releases a recording appears on
// (id / title / status / date only). Used by the autotag recording-
// search path to resolve a picked recording to a concrete release.
// Cached for 7 days.
func (c *MusicBrainzClient) LookupRecordingReleases(
ctx context.Context, recordingMBID string,
) ([]MBRecordingRelease, error) {
cacheKey := "mb:lookup:recording-releases:" + recordingMBID
if data, ok := c.cache.Get(cacheKey); ok {
var out []MBRelease
var out []MBRecordingRelease
if err := json.Unmarshal(data, &out); err == nil {
return out, nil
}
@@ -378,6 +397,77 @@ func (c *MusicBrainzClient) BrowseReleases(
return nil, err
}
c.logger.Info("musicbrainz lookup recording releases", "mbid", recordingMBID)
rec, err := c.mb.LookupRecording(
ctx,
mbtypes.MBID(recordingMBID),
musicbrainzws2.IncludesFilter{Includes: []string{"releases"}},
)
if err != nil {
return nil, err
}
out := make([]MBRecordingRelease, 0, len(rec.Releases))
for _, rel := range rec.Releases {
out = append(out, MBRecordingRelease{
MBID: string(rel.ID),
Title: rel.Title,
Status: rel.Status,
Date: rel.Date.String(),
})
}
c.cacheJSON(cacheKey, out, cacheTTLEntity, recordingMBID, "recording")
return out, nil
}
// BrowseReleases fetches the releases for a given release group
// MBID, including media/track information. Cached for 7 days.
// browseReleasesCacheKey returns the response-cache key for a release
// group's releases.
func browseReleasesCacheKey(releaseGroupMBID string) string {
return "mb:browse:releases:" + releaseGroupMBID
}
// BrowseReleasesCached returns a release group's releases from the local
// response cache only, never hitting the network. The bool reports
// whether a fresh (unexpired) cache entry was found. Used by the album
// page's local-first path so a cold fetch can be deferred to the
// background instead of blocking the request.
func (c *MusicBrainzClient) BrowseReleasesCached(
releaseGroupMBID string,
) ([]MBRelease, bool) {
data, ok := c.cache.Get(browseReleasesCacheKey(releaseGroupMBID))
if !ok {
return nil, false
}
var out []MBRelease
if err := json.Unmarshal(data, &out); err != nil {
return nil, false
}
return out, true
}
// BrowseReleases fetches all releases (with recordings + media) for a
// release group, serving from the local response cache when warm and
// otherwise hitting MusicBrainz and caching the result.
func (c *MusicBrainzClient) BrowseReleases(
ctx context.Context, releaseGroupMBID string,
) ([]MBRelease, error) {
cacheKey := browseReleasesCacheKey(releaseGroupMBID)
if out, ok := c.BrowseReleasesCached(releaseGroupMBID); ok {
return out, nil
}
if err := c.limiter.Wait(ctx); err != nil {
return nil, err
}
c.logger.Info("musicbrainz browse releases",
"releaseGroupMBID", releaseGroupMBID,
)
@@ -395,7 +485,7 @@ func (c *MusicBrainzClient) BrowseReleases(
out := convertReleases(result.Releases)
c.cacheJSON(cacheKey, out, cacheTTLEntity, releaseGroupMBID, "release-group")
c.cacheJSON(cacheKey, out, cacheTTLReleases, releaseGroupMBID, "release-group")
return out, nil
}
@@ -534,7 +624,19 @@ func convertRelease(r musicbrainzws2.Release) MBRelease {
}
for _, m := range r.Media {
// Skip video media outright — DVD/Blu-ray bonus discs
// inflate track counts and wreck track-count-based scoring
// (beets ignores video/data tracks for the same reason).
if isVideoFormat(m.Format) {
continue
}
for _, t := range m.Tracks {
// Same for individual video recordings on audio media.
if t.Recording.IsVideo {
continue
}
// Use the recording MBID, not the track MBID. Tracks
// and recordings have distinct MBIDs in MusicBrainz:
// a track is the placement of a recording on a specific
@@ -565,6 +667,25 @@ func convertRelease(r musicbrainzws2.Release) MBRelease {
return rel
}
// isVideoFormat reports whether a medium's format string names a
// video carrier. "DVD-Audio" stays audio; bare "DVD", "DVD-Video",
// "Blu-ray", "HD-DVD", "VHS", "VCD"/"SVCD" are video.
func isVideoFormat(format string) bool {
f := strings.ToLower(format)
if strings.Contains(f, "dvd-audio") {
return false
}
for _, v := range []string{"dvd", "blu-ray", "bluray", "hd-dvd", "vhs", "vcd"} {
if strings.Contains(f, v) {
return true
}
}
return false
}
func convertReleases(releases []musicbrainzws2.Release) []MBRelease {
out := make([]MBRelease, len(releases))
for i, r := range releases {
backend/explore/ranker.go
+195
View File
@@ -0,0 +1,195 @@
package explore
import (
"encoding/json"
"fmt"
"io"
"math"
)
// This file sketches a learned ranking model that replaces the
// hand-tuned scoring constants (the fw* feature weights, tierBonus,
// rgTierBonus, the intent-prior multipliers) with weights that can be
// trained from the user's own click history.
//
// Status: NOT yet wired into Search(). The integration point is the
// candidate scorers in the top-results pipeline
// (scoreArtistCandidate / scoreReleaseGroupCandidate /
// scoreRecordingCandidate) and the main rerank. Swap those additive
// constant sums for RankFeatures + LinearModel.Score once the eval
// harness has real fixtures to prove the change is a win.
//
// Why a linear model and not something fancier: it needs no scale, no
// GPU, trains in microseconds on a desktop's worth of clicks, and —
// crucially — personalises to ONE user. That is the lever a
// multi-million-user system gets from aggregate logs; we get the same
// shape of signal from a single user's repeated intent.
//
// Training prerequisite (important): logistic training needs negatives,
// i.e. results that were SHOWN but NOT clicked. search_clicks records
// only clicks today. To train properly, log impressions too (the
// MBIDs shown for a query); clicked rows are positive labels, the rest
// of the shown set are negatives. Until impressions are logged, use
// DefaultModel(), whose weights reproduce the current behaviour.
// RankFeatures is the feature vector for a single candidate. Every
// field is normalised to roughly [0,1] so weights are comparable.
type RankFeatures struct {
// Textual match strength against the query (mutually exclusive
// tiers collapsed to a single 0..1 magnitude: exact=1.0,
// prefix=0.6, whole-word=0.4, substring=0.2, none=0).
NameMatch float64
// Artist-credit match strength, same scale. Lets "abbey road
// beatles" reward the album credited to The Beatles.
ArtistMatch float64
// Log-scaled popularity and listener count, both via normLog so
// they share the fixed reference scale.
LogPopularity float64
LogListeners float64
// Personalisation signals.
InLibrary float64 // 1.0 when owned, else 0
IsSimilar float64 // 0..1 similarity to an owned artist
// Recency-decayed per-query click signal for this candidate.
ClickRate float64
}
// rankFeatureCount is the number of features (excluding bias). Used by
// the gradient step to iterate fields generically.
const rankFeatureCount = 7
// asSlice returns the features in a stable order so Score and Update
// agree on indexing.
func (f RankFeatures) asSlice() [rankFeatureCount]float64 {
return [rankFeatureCount]float64{
f.NameMatch,
f.ArtistMatch,
f.LogPopularity,
f.LogListeners,
f.InLibrary,
f.IsSimilar,
f.ClickRate,
}
}
// LinearModel scores a candidate as bias + Σ wᵢ·featureᵢ. For ranking
// the raw score is what matters; the logistic squashing is used only
// during training to produce a probability for the gradient.
type LinearModel struct {
Bias float64 `json:"bias"`
Weights [rankFeatureCount]float64 `json:"weights"`
}
// DefaultModel returns weights that reproduce the current hand-tuned
// scorer, so swapping the model in with no training leaves behaviour
// unchanged. The values mirror the fw* constants in explore.go.
func DefaultModel() LinearModel {
return LinearModel{
Bias: 0,
Weights: [rankFeatureCount]float64{
1.00, // NameMatch ~ fwExactTitle / fwPrefixTitle blend
0.90, // ArtistMatch ~ fwExactArtist
0.80, // LogPopularity ~ fwListenLog
0.60, // LogListeners ~ fwListenerLog
0.50, // InLibrary ~ fwInLibrary
0.20, // IsSimilar ~ fwSimilar
0.30, // ClickRate ~ click feature cap
},
}
}
// Score returns the un-squashed ranking score for a candidate. Higher
// is better. This is the value to sort by.
func (m LinearModel) Score(f RankFeatures) float64 {
score := m.Bias
fs := f.asSlice()
for i := range fs {
score += m.Weights[i] * fs[i]
}
return score
}
// Probability squashes the score to (0,1) via the logistic function.
// Used during training to compute the gradient.
func (m LinearModel) Probability(f RankFeatures) float64 {
return 1.0 / (1.0 + math.Exp(-m.Score(f)))
}
// Sample is one training example: a candidate's features and whether
// the user clicked it (1.0) or saw-but-skipped it (0.0).
type Sample struct {
Features RankFeatures
Label float64
}
// Update performs one logistic-regression SGD step toward the label.
// learningRate is typically ~0.05. Returns the pre-update prediction
// so callers can track convergence.
func (m *LinearModel) Update(s Sample, learningRate float64) float64 {
pred := m.Probability(s.Features)
err := s.Label - pred
fs := s.Features.asSlice()
m.Bias += learningRate * err
for i := range fs {
m.Weights[i] += learningRate * err * fs[i]
}
return pred
}
// Train runs SGD over the samples for the given number of epochs. A
// few hundred clicks over a handful of epochs converges fine; this is
// cheap enough to run on startup or after a search session.
func Train(model *LinearModel, samples []Sample, epochs int, learningRate float64) {
for range epochs {
for _, s := range samples {
model.Update(s, learningRate)
}
}
}
// matchStrength collapses the mutually-exclusive textual match tiers
// used throughout the current scorers into a single magnitude, so the
// model has one weight to learn instead of four overlapping constants.
func matchStrength(exact, prefix, wholeWord, substring bool) float64 {
switch {
case exact:
return 1.0
case prefix:
return 0.6
case wholeWord:
return 0.4
case substring:
return 0.2
default:
return 0.0
}
}
// SaveModel writes the model as JSON. Callers persist this to disk or
// the index meta table; the model is small (8 floats).
func SaveModel(w io.Writer, m LinearModel) error {
if err := json.NewEncoder(w).Encode(m); err != nil {
return fmt.Errorf("ranker: encode model: %w", err)
}
return nil
}
// LoadModel reads a model previously written by SaveModel.
func LoadModel(r io.Reader) (LinearModel, error) {
var m LinearModel
if err := json.NewDecoder(r).Decode(&m); err != nil {
return LinearModel{}, fmt.Errorf("ranker: decode model: %w", err)
}
return m, nil
}
backend/explore/ranker_test.go
+102
View File
@@ -0,0 +1,102 @@
package explore
import (
"bytes"
"math"
"testing"
)
func TestDefaultModelScoreOrdering(t *testing.T) {
m := DefaultModel()
// A popular, in-library exact match must outscore an obscure
// substring match.
strong := RankFeatures{NameMatch: 1.0, LogPopularity: 0.9, InLibrary: 1.0}
weak := RankFeatures{NameMatch: 0.2, LogPopularity: 0.1}
if m.Score(strong) <= m.Score(weak) {
t.Errorf("strong candidate %.3f should outscore weak %.3f",
m.Score(strong), m.Score(weak))
}
}
func TestUpdateMovesTowardLabel(t *testing.T) {
m := DefaultModel()
// A candidate the user repeatedly clicks should see its predicted
// probability rise after training on positive labels.
f := RankFeatures{NameMatch: 0.4, LogPopularity: 0.2}
before := m.Probability(f)
for range 50 {
m.Update(Sample{Features: f, Label: 1.0}, 0.1)
}
after := m.Probability(f)
if after <= before {
t.Errorf("probability should rise toward positive label: before=%.4f after=%.4f",
before, after)
}
}
func TestUpdateLearnsNegative(t *testing.T) {
m := DefaultModel()
f := RankFeatures{NameMatch: 0.9, LogPopularity: 0.9}
before := m.Probability(f)
// Shown repeatedly, never clicked — probability should fall.
for range 50 {
m.Update(Sample{Features: f, Label: 0.0}, 0.1)
}
after := m.Probability(f)
if after >= before {
t.Errorf("probability should fall toward negative label: before=%.4f after=%.4f",
before, after)
}
}
func TestMatchStrengthTiers(t *testing.T) {
if matchStrength(true, false, false, false) != 1.0 {
t.Error("exact match should be 1.0")
}
if matchStrength(false, false, false, false) != 0.0 {
t.Error("no match should be 0.0")
}
// Tiers must be strictly ordered.
exact := matchStrength(true, false, false, false)
prefix := matchStrength(false, true, false, false)
word := matchStrength(false, false, true, false)
sub := matchStrength(false, false, false, true)
if !(exact > prefix && prefix > word && word > sub) {
t.Errorf("tiers not strictly ordered: %v %v %v %v", exact, prefix, word, sub)
}
}
func TestModelRoundTrip(t *testing.T) {
m := DefaultModel()
m.Bias = 0.123
m.Weights[0] = 0.777
var buf bytes.Buffer
if err := SaveModel(&buf, m); err != nil {
t.Fatalf("SaveModel: %v", err)
}
got, err := LoadModel(&buf)
if err != nil {
t.Fatalf("LoadModel: %v", err)
}
if math.Abs(got.Bias-m.Bias) > 1e-9 || math.Abs(got.Weights[0]-m.Weights[0]) > 1e-9 {
t.Errorf("round trip mismatch: got %+v want %+v", got, m)
}
}
backend/explore/searchindex.go
+927 -1408
View File
File diff suppressed because it is too large Load Diff
backend/explore/types.go
+31 -13
View File
@@ -24,6 +24,7 @@ type TopResult struct {
MBID string `json:"mbid"`
Name string `json:"name"`
ArtistCredit string `json:"artistCredit,omitempty"` // for tracks/albums
ArtistMBID string `json:"artistMbid,omitempty"` // for linking the artist subtitle
IntentScore float64 `json:"intentScore"`
// Artist-specific
ArtistType string `json:"artistType,omitempty"` // "Group", "Person"
@@ -31,8 +32,14 @@ type TopResult struct {
// Album-specific
PrimaryType string `json:"primaryType,omitempty"`
Year string `json:"year,omitempty"`
// Track-specific
Length int `json:"length,omitempty"`
// Track-specific. ReleaseGroupMBID is resolved (from CAAReleaseMBID)
// so a track click can open its album page with the track highlighted,
// matching how tracks behave everywhere else. ReleaseName is the album
// title used for the album page header.
Length int `json:"length,omitempty"`
CAAReleaseMBID string `json:"caaReleaseMbid,omitempty"`
ReleaseGroupMBID string `json:"releaseGroupMbid,omitempty"`
ReleaseName string `json:"releaseName,omitempty"`
// Library status — populated from index cross-reference columns.
InLibrary bool `json:"inLibrary"`
}
@@ -64,8 +71,9 @@ type MBReleaseGroup struct {
SecondaryTypes []string `json:"secondaryTypes,omitempty"`
FirstReleaseDate string `json:"firstReleaseDate"`
ArtistCredit string `json:"artistCredit"`
Score int `json:"-"` // MB search relevance, used for reranking
Popularity int `json:"popularity"` // raw LB listen count (0 if unknown)
ArtistMBID string `json:"artistMbid,omitempty"` // for linking the artist to its detail page
Score int `json:"-"` // MB search relevance, used for reranking
Popularity int `json:"popularity"` // raw LB listen count (0 if unknown)
ListenerCount int `json:"listenerCount"`
InLibrary bool `json:"inLibrary"` // true if the user owns this album
LocalID int64 `json:"localId,omitempty"` // local release_group row ID
@@ -85,15 +93,22 @@ type MBRelease struct {
// MBRecording is a Wails-friendly projection of a MusicBrainz
// recording.
type MBRecording struct {
MBID string `json:"mbid"`
Title string `json:"title"`
Length int `json:"length"`
ArtistCredit string `json:"artistCredit"`
Score int `json:"score"`
Popularity int `json:"popularity"` // raw LB listen count (0 if unknown)
ListenerCount int `json:"listenerCount"`
InLibrary bool `json:"inLibrary"` // true if the user owns this recording
LocalID int64 `json:"localId,omitempty"` // local recording row ID
MBID string `json:"mbid"`
Title string `json:"title"`
Length int `json:"length"`
ArtistCredit string `json:"artistCredit"`
ArtistMBID string `json:"artistMbid,omitempty"` // for linking the artist to its detail page
Score int `json:"score"`
Popularity int `json:"popularity"` // raw LB listen count (0 if unknown)
ListenerCount int `json:"listenerCount"`
CAAReleaseMBID string `json:"caaReleaseMbid,omitempty"` // parent release, for album navigation
// ReleaseGroupMBID is resolved from CAAReleaseMBID so a track can
// link to its album page with the track highlighted, matching how
// tracks behave everywhere else.
ReleaseGroupMBID string `json:"releaseGroupMbid,omitempty"`
ReleaseName string `json:"releaseName,omitempty"` // album title
InLibrary bool `json:"inLibrary"` // true if the user owns this recording
LocalID int64 `json:"localId,omitempty"` // local recording row ID
}
// MBTrack is a Wails-friendly projection of a MusicBrainz track.
@@ -119,6 +134,9 @@ type LBTopRecording struct {
TrackName string `json:"trackName"`
TotalListenCount int `json:"totalListenCount"`
CAAReleaseMBID string `json:"caaReleaseMbid"`
// ReleaseGroupMBID is resolved from CAAReleaseMBID so a top-track
// row can link to its album page with the track highlighted.
ReleaseGroupMBID string `json:"releaseGroupMbid,omitempty"`
ReleaseName string `json:"releaseName"`
Length int `json:"length"` // milliseconds (from LB API)
InLibrary bool `json:"inLibrary"`
backend/library/artistcredit.go
+72
View File
@@ -0,0 +1,72 @@
package library
import (
"strings"
"yellowjacket/backend/metadata"
)
// featuringSeparators are the credit join phrases that introduce a
// featured (non-primary) artist. Only true "featuring" markers are
// listed: separators like "&", "x", "with", and "," are deliberately
// excluded because they routinely appear inside real artist names
// (e.g. "Simon & Garfunkel", "Tyler, the Creator").
var featuringSeparators = []string{
" feat. ", " feat ", " featuring ", " ft. ", " ft ",
}
// stripFeaturing returns the credit up to its first "featuring" marker,
// yielding the primary-artist portion of a credit string. "Lana Del
// Rey ft. Sean Lennon" becomes "Lana Del Rey"; a credit with no marker
// is returned unchanged (trimmed). Matching is case-insensitive.
func stripFeaturing(credit string) string {
lower := strings.ToLower(credit)
cut := -1
for _, sep := range featuringSeparators {
if i := strings.Index(lower, sep); i >= 0 && (cut < 0 || i < cut) {
cut = i
}
}
if cut < 0 {
return strings.TrimSpace(credit)
}
return strings.TrimSpace(credit[:cut])
}
// primaryArtist resolves the single canonical artist a track credit
// should map to, plus that artist's MusicBrainz ID. A file tags its
// ARTIST as a full credit string ("Lana Del Rey ft. Sean Lennon") but
// carries only one MUSICBRAINZ_ARTISTID — the primary artist's. Storing
// the whole credit as an artist entity, and stamping the primary MBID on
// it, is what produced duplicate, mis-titled artists (one MBID fanned
// out across many rows); instead we resolve the primary artist's clean
// name here and keep the full credit only as the artist_credit text.
//
// The clean name comes from the album-artist tag when the track resolves
// to the same MBID as the album artist (the common "Album Artist feat.
// Guest" case, where ALBUMARTIST is the authoritative name). Otherwise
// the featured clause is stripped from the credit string.
func primaryArtist(tags *metadata.TrackMetadata) (name, mbid string) {
mbid = tags.ArtistMBID
if mbid == "" {
mbid = tags.AlbumArtistMBID
}
if tags.ArtistMBID != "" &&
tags.ArtistMBID == tags.AlbumArtistMBID &&
tags.AlbumArtist != "" {
name = tags.AlbumArtist
} else {
name = stripFeaturing(tags.Artist)
}
if name == "" {
name = "Unknown Artist"
}
return name, mbid
}
backend/library/artistcredit_test.go
+126
View File
@@ -0,0 +1,126 @@
package library
import (
"testing"
"yellowjacket/backend/metadata"
)
func TestStripFeaturing(t *testing.T) {
t.Parallel()
tests := []struct {
name string
credit string
want string
}{
{"plain", "Lana Del Rey", "Lana Del Rey"},
{"ft dot", "Lana Del Rey ft. Sean Lennon", "Lana Del Rey"},
{"feat dot", "2Pac feat. Nate Dogg", "2Pac"},
{"featuring", "Beyoncé featuring The Weeknd", "Beyoncé"},
{"ft no dot", "Drake ft Travis Scott", "Drake"},
{"case insensitive", "Kanye West FEAT. PARTYNEXTDOOR", "Kanye West"},
{"ampersand kept", "Simon & Garfunkel", "Simon & Garfunkel"},
{"comma kept", "Tyler, the Creator", "Tyler, the Creator"},
{"first marker wins", "A feat. B ft. C", "A"},
{"trims", " Daft Punk feat. Panda Bear ", "Daft Punk"},
{"empty", "", ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
if got := stripFeaturing(tt.credit); got != tt.want {
t.Errorf("stripFeaturing(%q) = %q, want %q", tt.credit, got, tt.want)
}
})
}
}
func TestPrimaryArtist(t *testing.T) {
t.Parallel()
const lana = "b7539c32-53e7-4908-bda3-81449c367da6"
tests := []struct {
name string
tags metadata.TrackMetadata
wantName string
wantMBID string
}{
{
name: "collab on own album uses clean album artist",
tags: metadata.TrackMetadata{
Artist: "Lana Del Rey ft. Sean Lennon",
AlbumArtist: "Lana Del Rey",
ArtistMBID: lana,
AlbumArtistMBID: lana,
},
wantName: "Lana Del Rey",
wantMBID: lana,
},
{
name: "solo track",
tags: metadata.TrackMetadata{
Artist: "Lana Del Rey",
AlbumArtist: "Lana Del Rey",
ArtistMBID: lana,
AlbumArtistMBID: lana,
},
wantName: "Lana Del Rey",
wantMBID: lana,
},
{
name: "compilation: album artist differs, strip featuring, keep track mbid",
tags: metadata.TrackMetadata{
Artist: "Some Artist feat. Guest",
AlbumArtist: "Various Artists",
ArtistMBID: "aaaa",
AlbumArtistMBID: "va-mbid",
},
wantName: "Some Artist",
wantMBID: "aaaa",
},
{
name: "no album artist, strip featuring",
tags: metadata.TrackMetadata{
Artist: "Some Artist feat. Guest",
ArtistMBID: "aaaa",
},
wantName: "Some Artist",
wantMBID: "aaaa",
},
{
name: "no track mbid falls back to album mbid",
tags: metadata.TrackMetadata{
Artist: "Solo",
AlbumArtist: "Solo",
AlbumArtistMBID: "album-mbid",
},
wantName: "Solo",
wantMBID: "album-mbid",
},
{
name: "empty artist",
tags: metadata.TrackMetadata{},
wantName: "Unknown Artist",
wantMBID: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
gotName, gotMBID := primaryArtist(&tt.tags)
if gotName != tt.wantName {
t.Errorf("primaryArtist name = %q, want %q", gotName, tt.wantName)
}
if gotMBID != tt.wantMBID {
t.Errorf("primaryArtist mbid = %q, want %q", gotMBID, tt.wantMBID)
}
})
}
}
backend/library/crud.go
+9
View File
@@ -37,6 +37,9 @@ type RemovalHooks struct {
StopPlayback func()
// CompactQueue reloads queue state after cascade deletes.
CompactQueue func()
// PostRemove runs after the removal commits, for cross-cutting
// invalidation (e.g. clearing library-sync "ready" markers).
PostRemove func()
}
// SetRemovalHooks provides optional hooks for cross-cutting
@@ -461,6 +464,12 @@ func (l *Library) RemoveLibrary(id int64) (*RemovalSummary, error) {
l.removalHooks.CompactQueue()
}
// 22. Post-commit: invalidate library-sync markers so the gated
// index/lyric re-sync runs on the next launch.
if l.removalHooks.PostRemove != nil {
l.removalHooks.PostRemove()
}
summary := &RemovalSummary{
TracksDeleted: tracksDeleted,
ArtistsRemoved: artistsRemoved,
backend/library/library.go
+17 -14
View File
@@ -1216,14 +1216,19 @@ func (l *Library) processMetadata(
q, cache, metrics, tags, thumbChan,
)
// 2. Get or create artist credit for track artist.
artistName := tags.Artist
if artistName == "" {
artistName = "Unknown Artist"
// 2. Get or create the artist credit for the track. The credit
// text is the full tagged string (e.g. "Lana Del Rey ft. Sean
// Lennon") and is kept only for display; the artist *entity* it
// links to is the primary artist, resolved cleanly by primaryArtist
// so featured-artist credits don't fork into their own bogus artist
// rows (all sharing the primary's single MBID).
creditText := tags.Artist
if creditText == "" {
creditText = "Unknown Artist"
}
artistCredit, err := l.cachedUpsertArtistCredit(
q, cache, artistName,
q, cache, creditText,
)
if err != nil {
return 0, fmt.Errorf(
@@ -1231,7 +1236,9 @@ func (l *Library) processMetadata(
)
}
l.cachedLinkArtist(q, cache, metrics, artistName, artistCredit.ID)
primaryName, primaryMBID := primaryArtist(tags)
l.cachedLinkArtist(q, cache, metrics, primaryName, artistCredit.ID)
// 3. Get or create artist credit for album artist.
albumArtistCreditID := l.resolveAlbumArtistCredit(
@@ -1289,9 +1296,9 @@ func (l *Library) processMetadata(
// 7. Update MusicBrainz IDs (if present in tags).
if releaseGroupID.Valid {
l.updateMBIDs(tx, cache, tags, artistName, releaseGroupID.Int64, recording.ID)
l.updateMBIDs(tx, cache, tags, primaryName, primaryMBID, releaseGroupID.Int64, recording.ID)
} else {
l.updateMBIDs(tx, cache, tags, artistName, 0, recording.ID)
l.updateMBIDs(tx, cache, tags, primaryName, primaryMBID, 0, recording.ID)
}
return recording.ID, nil
@@ -1306,15 +1313,11 @@ func (l *Library) updateMBIDs(
cache *entityCache,
tags *metadata.TrackMetadata,
artistName string,
artistMBID string,
releaseGroupID int64,
recordingID int64,
) {
// Artist MBID — prefer album artist, fall back to track artist.
artistMBID := tags.AlbumArtistMBID
if artistMBID == "" {
artistMBID = tags.ArtistMBID
}
// Artist MBID (the primary artist's, resolved by primaryArtist).
if artistMBID != "" {
if artist, ok := cache.artists[artistName]; ok {
_, _ = tx.ExecContext(l.ctx,
backend/library/query.go
+21 -16
View File
@@ -183,6 +183,7 @@ type Album struct {
ID int64
Name string
ArtistName string
ArtistMBID string
MBID string
CoverArtPath string
CoverArtSmall string
@@ -194,7 +195,7 @@ type Album struct {
// GetAllTracks returns an array of track structs of every file in the library.
func (l *Library) GetAllTracks() ([]Track, error) {
rows, err := l.db.Queries.GetAllTracksWithFullMetadata(
rows, err := l.db.ReadQueries.GetAllTracksWithFullMetadata(
l.ctx,
)
if err != nil {
@@ -303,7 +304,7 @@ func (l *Library) SearchTracks(
// GetAlbumTracks returns all tracks for a given album (release group), ordered by disc and track number.
func (l *Library) GetAlbumTracks(albumID int64) ([]Track, error) {
rows, err := l.db.Queries.GetAudioFilesByReleaseGroup(l.ctx, albumID)
rows, err := l.db.ReadQueries.GetAudioFilesByReleaseGroup(l.ctx, albumID)
if err != nil {
l.logger.Error("could not retrieve album tracks", "albumID", albumID, "error", err)
@@ -347,7 +348,7 @@ func (l *Library) GetAlbumTracks(albumID int64) ([]Track, error) {
// GetAllAlbums returns all albums with cover art and artist info for the cover grid.
func (l *Library) GetAllAlbums() ([]Album, error) {
rows, err := l.db.Queries.GetAllAlbumsWithDetails(l.ctx)
rows, err := l.db.ReadQueries.GetAllAlbumsWithDetails(l.ctx)
if err != nil {
l.logger.Error("could not retrieve albums", "error", err)
@@ -363,6 +364,7 @@ func (l *Library) GetAllAlbums() ([]Album, error) {
ID: row.ID,
Name: row.Name,
ArtistName: row.ArtistName,
ArtistMBID: row.ArtistMbid,
}
if row.Year.Valid {
@@ -392,7 +394,7 @@ func (l *Library) GetAllAlbums() ([]Album, error) {
// GetAllArtists returns artists that are credited as album artists, ordered by name.
func (l *Library) GetAllArtists() ([]Artist, error) {
rows, err := l.db.Queries.GetAlbumArtists(l.ctx)
rows, err := l.db.ReadQueries.GetAlbumArtists(l.ctx)
if err != nil {
l.logger.Error(
"could not retrieve artists",
@@ -489,7 +491,7 @@ func (l *Library) resolveArtistImages(artists []Artist) {
func (l *Library) GetAlbumsByArtist(
artistID int64,
) ([]Album, error) {
rows, err := l.db.Queries.GetAlbumsByArtist(
rows, err := l.db.ReadQueries.GetAlbumsByArtist(
l.ctx,
artistID,
)
@@ -519,6 +521,7 @@ func (l *Library) GetAlbumsByArtist(
ID: row.ID,
Name: row.Name,
ArtistName: row.ArtistName,
ArtistMBID: row.ArtistMbid,
}
if row.Year.Valid {
@@ -552,7 +555,7 @@ type GenreWithCount struct {
func (l *Library) GetTracksByGenre(
genreName string,
) ([]Track, error) {
rows, err := l.db.Queries.GetTracksByGenre(
rows, err := l.db.ReadQueries.GetTracksByGenre(
l.ctx, genreName,
)
if err != nil {
@@ -600,7 +603,7 @@ func (l *Library) GetTracksByGenre(
func (l *Library) GetAllGenresWithCounts() (
[]GenreWithCount, error,
) {
rows, err := l.db.Queries.GetAllGenresWithCounts(
rows, err := l.db.ReadQueries.GetAllGenresWithCounts(
l.ctx,
)
if err != nil {
@@ -630,7 +633,7 @@ func (l *Library) GetAllGenresWithCounts() (
func (l *Library) GetAllTracksByLibrary(
libraryID int64,
) ([]Track, error) {
rows, err := l.db.Queries.GetAllTracksWithFullMetadataByLibrary(
rows, err := l.db.ReadQueries.GetAllTracksWithFullMetadataByLibrary(
l.ctx, libraryID,
)
if err != nil {
@@ -687,7 +690,7 @@ func (l *Library) GetAllTracksByLibrary(
func (l *Library) GetAllAlbumsByLibrary(
libraryID int64,
) ([]Album, error) {
rows, err := l.db.Queries.GetAllAlbumsWithDetailsByLibrary(
rows, err := l.db.ReadQueries.GetAllAlbumsWithDetailsByLibrary(
l.ctx, libraryID,
)
if err != nil {
@@ -715,6 +718,7 @@ func (l *Library) GetAllAlbumsByLibrary(
ID: row.ID,
Name: row.Name,
ArtistName: row.ArtistName,
ArtistMBID: row.ArtistMbid,
}
if row.Year.Valid {
@@ -746,7 +750,7 @@ func (l *Library) GetAllAlbumsByLibrary(
func (l *Library) GetAllArtistsByLibrary(
libraryID int64,
) ([]Artist, error) {
rows, err := l.db.Queries.GetAlbumArtistsByLibrary(
rows, err := l.db.ReadQueries.GetAlbumArtistsByLibrary(
l.ctx, libraryID,
)
if err != nil {
@@ -792,7 +796,7 @@ func (l *Library) GetAllArtistsByLibrary(
func (l *Library) GetAlbumsByArtistByLibrary(
artistID, libraryID int64,
) ([]Album, error) {
rows, err := l.db.Queries.GetAlbumsByArtistByLibrary(
rows, err := l.db.ReadQueries.GetAlbumsByArtistByLibrary(
l.ctx, sqlcgen.GetAlbumsByArtistByLibraryParams{
ArtistID: artistID,
LibraryID: libraryID,
@@ -826,6 +830,7 @@ func (l *Library) GetAlbumsByArtistByLibrary(
ID: row.ID,
Name: row.Name,
ArtistName: row.ArtistName,
ArtistMBID: row.ArtistMbid,
}
if row.Year.Valid {
@@ -853,7 +858,7 @@ func (l *Library) GetAlbumsByArtistByLibrary(
func (l *Library) GetAllGenresWithCountsByLibrary(
libraryID int64,
) ([]GenreWithCount, error) {
rows, err := l.db.Queries.GetAllGenresWithCountsByLibrary(
rows, err := l.db.ReadQueries.GetAllGenresWithCountsByLibrary(
l.ctx, libraryID,
)
if err != nil {
@@ -885,7 +890,7 @@ func (l *Library) GetAllGenresWithCountsByLibrary(
func (l *Library) GetTracksByGenreByLibrary(
genreName string, libraryID int64,
) ([]Track, error) {
rows, err := l.db.Queries.GetTracksByGenreByLibrary(
rows, err := l.db.ReadQueries.GetTracksByGenreByLibrary(
l.ctx, sqlcgen.GetTracksByGenreByLibraryParams{
Name: genreName,
LibraryID: libraryID,
@@ -939,7 +944,7 @@ func (l *Library) GetTracksByGenreByLibrary(
func (l *Library) GetAlbumTracksByLibrary(
albumID, libraryID int64,
) ([]Track, error) {
rows, err := l.db.Queries.GetAudioFilesByReleaseGroupByLibrary(
rows, err := l.db.ReadQueries.GetAudioFilesByReleaseGroupByLibrary(
l.ctx, sqlcgen.GetAudioFilesByReleaseGroupByLibraryParams{
ReleaseGroupID: albumID,
LibraryID: libraryID,
@@ -1052,7 +1057,7 @@ type Info struct {
// GetAllLibrariesWithTrackCounts returns all libraries with their
// audio file counts. Typically 1-5 libraries so the loop is trivial.
func (l *Library) GetAllLibrariesWithTrackCounts() ([]Info, error) {
libs, err := l.db.Queries.GetAllLibraries(l.ctx)
libs, err := l.db.ReadQueries.GetAllLibraries(l.ctx)
if err != nil {
return nil, fmt.Errorf("could not get libraries: %w", err)
}
@@ -1060,7 +1065,7 @@ func (l *Library) GetAllLibrariesWithTrackCounts() ([]Info, error) {
result := make([]Info, 0, len(libs))
for _, lib := range libs {
count, countErr := l.db.Queries.CountAudioFilesByLibrary(l.ctx, lib.ID)
count, countErr := l.db.ReadQueries.CountAudioFilesByLibrary(l.ctx, lib.ID)
if countErr != nil {
l.logger.Error("could not count tracks for library",
"libraryID", lib.ID, "error", countErr)
backend/player/buffered_streamer.go
+25 -2
View File
@@ -19,6 +19,12 @@ import (
// temporarily empty (read-ahead hasn't caught up), Stream returns
// silence rather than blocking or signaling end-of-stream.
type BufferedStreamer struct {
// srcMu serializes all access to the underlying source. The
// read-ahead goroutine holds it while calling source.Stream;
// callers that need to Seek the source must hold it too (via
// LockSource/UnlockSource) so the non-thread-safe decoder is
// never read and seeked concurrently.
srcMu sync.Mutex
mu sync.Mutex
source beep.Streamer
ring [][2]float64
@@ -89,9 +95,12 @@ func (bs *BufferedStreamer) readAhead() {
bs.mu.Unlock()
// Read from source WITHOUT holding the lock so disk I/O
// does not block the speaker goroutine.
// Read from source WITHOUT holding bs.mu so disk I/O does
// not block the speaker goroutine. srcMu is held to keep
// this read from racing a concurrent source Seek.
bs.srcMu.Lock()
n, ok := bs.source.Stream(tmp[:toRead])
bs.srcMu.Unlock()
if n > 0 {
bs.mu.Lock()
@@ -190,6 +199,20 @@ func (bs *BufferedStreamer) Flush() {
bs.count = 0
}
// LockSource blocks the read-ahead goroutine from touching the
// underlying source, giving the caller exclusive access so it can
// safely Seek the non-thread-safe decoder. Every LockSource must
// be paired with an UnlockSource.
func (bs *BufferedStreamer) LockSource() {
bs.srcMu.Lock()
}
// UnlockSource releases the exclusive source access acquired by
// LockSource, allowing the read-ahead goroutine to resume.
func (bs *BufferedStreamer) UnlockSource() {
bs.srcMu.Unlock()
}
// Close signals the read-ahead goroutine to stop. It is safe to
// call multiple times.
func (bs *BufferedStreamer) Close() {
backend/player/player.go
+14 -3
View File
@@ -764,6 +764,17 @@ func (p *Player) seekLocked(targetSeconds int) error {
return fmt.Errorf("cannot get track length: %w", err)
}
// Block the read-ahead goroutine from reading the source while
// we seek it. The decoder (e.g. FLAC's bufseekio.ReadSeeker) is
// not safe for concurrent Read+Seek, and read-ahead runs on its
// own goroutine — without this it can panic with a slice-bounds
// error, especially right after load when the buffer is empty
// and read-ahead is filling at full speed.
if p.buffered != nil {
p.buffered.LockSource()
defer p.buffered.UnlockSource()
}
speaker.Lock()
samples := int(
@@ -863,7 +874,7 @@ func (p *Player) getCurrentTrackInfoLocked() TrackInfo {
// Try to get metadata from database.
if p.db != nil {
meta, err := p.db.Queries.GetTrackMetadataByPath(
meta, err := p.db.ReadQueries.GetTrackMetadataByPath(
p.ctx, info.FilePath,
)
if err == nil {
@@ -1004,7 +1015,7 @@ func (p *Player) buildMediaMetadata(
// full path; ResolveURLs converts it to relative HTTP paths
// for the frontend, but MPRIS needs the actual file path.
if p.db != nil && info.FilePath != "" {
dbMeta, err := p.db.Queries.GetTrackMetadataByPath(
dbMeta, err := p.db.ReadQueries.GetTrackMetadataByPath(
p.ctx, info.FilePath,
)
if err == nil && dbMeta.CoverArtPath != "" {
@@ -1102,7 +1113,7 @@ func (p *Player) restoreStateLocked() {
return
}
state, err := p.db.Queries.GetPlayerState(p.db.Ctx)
state, err := p.db.ReadQueries.GetPlayerState(p.db.Ctx)
if err != nil {
p.logger.Error(
"Failed to load player state", "err", err,
backend/playlist/playlist.go
+152 -13
View File
@@ -132,6 +132,11 @@ type Service struct {
db *database.DB
libraryDir LibraryDirProvider
favoritesConf FavoritesConfigProvider
// dataDirOverride, when non-empty, replaces the OS user data
// directory as the base for the playlists folder. Set by tests to
// keep M3U writes out of the real user data directory.
dataDirOverride string
}
// NewService creates a new playlist service.
@@ -172,7 +177,7 @@ func (s *Service) SetContext(ctx context.Context) {
// GetAllPlaylists returns all playlists ordered by most recently
// updated.
func (s *Service) GetAllPlaylists() ([]Summary, error) {
playlists, err := s.db.Queries.GetAllPlaylists(s.db.Ctx)
playlists, err := s.db.ReadQueries.GetAllPlaylists(s.db.Ctx)
if err != nil {
s.logger.Error(
"Failed to get playlists", "err", err,
@@ -204,7 +209,7 @@ func (s *Service) GetAllPlaylistsWithTracks() (
[]WithTracks,
error,
) {
playlists, err := s.db.Queries.GetAllPlaylists(s.db.Ctx)
playlists, err := s.db.ReadQueries.GetAllPlaylists(s.db.Ctx)
if err != nil {
s.logger.Error(
"Failed to get playlists", "err", err,
@@ -215,7 +220,7 @@ func (s *Service) GetAllPlaylistsWithTracks() (
)
}
rows, err := s.db.Queries.GetAllPlaylistTracksWithMetadata(
rows, err := s.db.ReadQueries.GetAllPlaylistTracksWithMetadata(
s.db.Ctx,
)
if err != nil {
@@ -288,7 +293,7 @@ func (s *Service) GetAllPlaylistsWithTracks() (
func (s *Service) GetPlaylistTracks(
playlistID int64,
) ([]Track, error) {
rows, err := s.db.Queries.GetPlaylistTracksWithMetadata(
rows, err := s.db.ReadQueries.GetPlaylistTracksWithMetadata(
s.db.Ctx,
playlistID,
)
@@ -563,7 +568,7 @@ func (s *Service) FindDuplicateTracksInPlaylist(
playlistID int64,
filePaths []string,
) (DuplicateCheckResult, error) {
rows, err := s.db.Queries.GetPlaylistTracksWithMetadata(
rows, err := s.db.ReadQueries.GetPlaylistTracksWithMetadata(
s.db.Ctx,
playlistID,
)
@@ -1229,11 +1234,17 @@ func (s *Service) addSingleTrack(
// playlistsDir returns the path to the playlists directory,
// creating it if needed.
func (s *Service) playlistsDir() (string, error) {
dataDir, err := system.GetUserDataDirPath()
if err != nil {
return "", fmt.Errorf(
"could not get user data directory: %w", err,
)
dataDir := s.dataDirOverride
if dataDir == "" {
var err error
dataDir, err = system.GetUserDataDirPath()
if err != nil {
return "", fmt.Errorf(
"could not get user data directory: %w", err,
)
}
}
dir := filepath.Join(dataDir, playlistsDirName)
@@ -1376,7 +1387,7 @@ func (s *Service) saveImportedPlaylistFile(
func (s *Service) buildM3UEntries(
playlistID int64,
) []m3uEntry {
rows, err := s.db.Queries.GetPlaylistTracksWithMetadata(
rows, err := s.db.ReadQueries.GetPlaylistTracksWithMetadata(
s.db.Ctx,
playlistID,
)
@@ -1482,7 +1493,7 @@ func (s *Service) migrateExistingPlaylists() {
}
}
playlists, err := s.db.Queries.GetAllPlaylists(s.db.Ctx)
playlists, err := s.db.ReadQueries.GetAllPlaylists(s.db.Ctx)
if err != nil {
s.logger.Warn(
"Could not get playlists for migration",
@@ -1534,7 +1545,7 @@ func (s *Service) RepopulateFromM3U() {
}
// Get all playlists.
playlists, err := s.db.Queries.GetAllPlaylists(s.db.Ctx)
playlists, err := s.db.ReadQueries.GetAllPlaylists(s.db.Ctx)
if err != nil {
s.logger.Warn("could not get playlists for repopulation", "err", err)
@@ -2677,11 +2688,139 @@ func (s *Service) UpdateSmartPlaylistRules(
"playlistId", playlistID,
)
// Re-materialize the persisted snapshot so it reflects the new
// rules. RefreshSmartPlaylist emits PlaylistTracksChanged.
if err := s.RefreshSmartPlaylist(playlistID); err != nil {
return fmt.Errorf(
"failed to refresh smart playlist after rule update: %w",
err,
)
}
return nil
}
// RefreshSmartPlaylist re-evaluates a smart playlist's rules against
// the current library and replaces its persisted membership in
// playlist_tracks with the result. This is the only path that
// re-evaluates a smart playlist — opening one otherwise reads the
// stored snapshot. Triggered on rule save and by the manual Refresh
// button.
func (s *Service) RefreshSmartPlaylist(
playlistID int64,
) error {
// EvaluateSmartPlaylist validates that the playlist exists and is
// smart, and returns the live rule-matched tracks.
tracks, err := s.EvaluateSmartPlaylist(playlistID)
if err != nil {
return err
}
if err := s.db.Queries.ClearPlaylistTracks(
s.db.Ctx, playlistID,
); err != nil {
return fmt.Errorf(
"failed to clear smart playlist tracks: %w", err,
)
}
for i, t := range tracks {
if strings.TrimSpace(t.FilePath) == "" {
continue
}
if err := s.addSingleTrack(
playlistID, t.FilePath, int64(i),
); err != nil {
// A track can vanish between evaluation and insertion
// (e.g. a concurrent rescan). Skip it rather than abort
// the whole refresh.
s.logger.Warn(
"Skipping smart playlist track during refresh",
"playlistId", playlistID,
"filePath", t.FilePath,
"err", err,
)
continue
}
}
// SAFETY: Hand-crafted UPDATE for smart_snapshot_at column not
// yet in sqlc schema. Parameterized by playlist ID.
if _, err := s.db.ExecContext(
`UPDATE playlists
SET smart_snapshot_at = CURRENT_TIMESTAMP
WHERE id = ? AND is_smart = 1`,
playlistID,
); err != nil {
return fmt.Errorf(
"failed to mark smart playlist snapshot: %w", err,
)
}
s.logger.Info(
"Smart playlist refreshed",
"playlistId", playlistID,
"trackCount", len(tracks),
)
s.savePlaylistFileByID(playlistID)
s.emitEvent(events.PlaylistTracksChanged, playlistID)
return nil
}
// GetSmartPlaylistTracks returns the persisted snapshot of a smart
// playlist as regular playlist tracks (with resolved cover art and
// phantom entries), identical to a normal playlist. If the playlist
// has never been materialized — e.g. it predates snapshot support —
// it is evaluated and stored on first access.
func (s *Service) GetSmartPlaylistTracks(
playlistID int64,
) ([]Track, error) {
// SAFETY: Hand-crafted SELECT for smart_snapshot_at column not
// yet in sqlc schema. Parameterized by playlist ID.
rows, err := s.db.QueryContext(
`SELECT smart_snapshot_at FROM playlists
WHERE id = ? AND is_smart = 1`,
playlistID,
)
if err != nil {
return nil, fmt.Errorf(
"failed to load smart playlist: %w", err,
)
}
if !rows.Next() {
_ = rows.Close()
return nil, errNotSmartPlaylist
}
var snapshotAt sql.NullString
if err := rows.Scan(&snapshotAt); err != nil {
_ = rows.Close()
return nil, fmt.Errorf(
"failed to read smart playlist snapshot state: %w", err,
)
}
// Close before RefreshSmartPlaylist / GetPlaylistTracks issue
// their own queries (MaxOpenConns=1 test DBs would deadlock).
_ = rows.Close()
if !snapshotAt.Valid {
if err := s.RefreshSmartPlaylist(playlistID); err != nil {
return nil, err
}
}
return s.GetPlaylistTracks(playlistID)
}
// EvaluateSmartPlaylist loads the rule set for a smart playlist
// from the database and evaluates it against the track library,
// returning the matching tracks.
backend/playlist/smart_test.go
+138 -2
View File
@@ -170,8 +170,9 @@ func newTestService(t *testing.T, db *database.DB) *Service {
t.Helper()
return &Service{
db: db,
logger: slog.Default(),
db: db,
logger: slog.Default(),
dataDirOverride: t.TempDir(),
}
}
@@ -294,6 +295,141 @@ func TestSmartPlaylistUpdateRules(t *testing.T) {
}
}
// TestSmartPlaylistPersistedSnapshot verifies that a smart playlist's
// membership is materialized and read from a stored snapshot: it is
// backfilled on first access, does NOT re-evaluate when the rules
// change out from under it, and only re-materializes on an explicit
// refresh.
func TestSmartPlaylistPersistedSnapshot(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
seedSmartTestTracks(t, db)
svc := newTestService(t, db)
// Band A → tracks 1 and 3.
bandARules := makeRulesJSON(t, smartplaylist.RuleSet{
Rules: []smartplaylist.Rule{
{Field: "artist", Operator: "is", Value: "Band A"},
},
})
summary, err := svc.CreateSmartPlaylist("Snapshot Test", bandARules)
if err != nil {
t.Fatalf("CreateSmartPlaylist failed: %v", err)
}
// First access backfills the snapshot (snapshot_at was NULL).
tracks, err := svc.GetSmartPlaylistTracks(summary.ID)
if err != nil {
t.Fatalf("GetSmartPlaylistTracks failed: %v", err)
}
if len(tracks) != 2 {
t.Fatalf("initial snapshot: got %d tracks, want 2", len(tracks))
}
for _, tr := range tracks {
if tr.Artist != "Band A" {
t.Errorf("snapshot track %q has artist %q, want Band A",
tr.Title, tr.Artist)
}
}
// Change the rules directly in the DB, bypassing
// UpdateSmartPlaylistRules so no refresh is triggered. The stored
// snapshot must be unaffected.
bandBRules := makeRulesJSON(t, smartplaylist.RuleSet{
Rules: []smartplaylist.Rule{
{Field: "artist", Operator: "is", Value: "Band B"},
},
})
if _, err := db.ExecContext(
"UPDATE playlists SET smart_rules = ? WHERE id = ?",
bandBRules, summary.ID,
); err != nil {
t.Fatalf("failed to rewrite rules: %v", err)
}
// Snapshot is served as-is: still Band A's two tracks, not Band B.
tracks, err = svc.GetSmartPlaylistTracks(summary.ID)
if err != nil {
t.Fatalf("GetSmartPlaylistTracks (post rule change) failed: %v", err)
}
if len(tracks) != 2 {
t.Fatalf("snapshot re-read: got %d tracks, want 2 (must not re-evaluate)",
len(tracks))
}
// An explicit refresh re-materializes against the current rules.
if err := svc.RefreshSmartPlaylist(summary.ID); err != nil {
t.Fatalf("RefreshSmartPlaylist failed: %v", err)
}
tracks, err = svc.GetSmartPlaylistTracks(summary.ID)
if err != nil {
t.Fatalf("GetSmartPlaylistTracks (post refresh) failed: %v", err)
}
if len(tracks) != 1 {
t.Fatalf("post-refresh snapshot: got %d tracks, want 1", len(tracks))
}
if tracks[0].Artist != "Band B" {
t.Errorf("post-refresh artist = %q, want Band B", tracks[0].Artist)
}
}
// TestSmartPlaylistSaveRulesRematerializes verifies that saving new
// rules through UpdateSmartPlaylistRules refreshes the stored snapshot.
func TestSmartPlaylistSaveRulesRematerializes(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
seedSmartTestTracks(t, db)
svc := newTestService(t, db)
bandARules := makeRulesJSON(t, smartplaylist.RuleSet{
Rules: []smartplaylist.Rule{
{Field: "artist", Operator: "is", Value: "Band A"},
},
})
summary, err := svc.CreateSmartPlaylist("Save Refresh", bandARules)
if err != nil {
t.Fatalf("CreateSmartPlaylist failed: %v", err)
}
bandBRules := makeRulesJSON(t, smartplaylist.RuleSet{
Rules: []smartplaylist.Rule{
{Field: "artist", Operator: "is", Value: "Band B"},
},
})
if err := svc.UpdateSmartPlaylistRules(summary.ID, bandBRules); err != nil {
t.Fatalf("UpdateSmartPlaylistRules failed: %v", err)
}
// The stored snapshot should already reflect the new rules without
// any manual refresh.
tracks, err := svc.GetSmartPlaylistTracks(summary.ID)
if err != nil {
t.Fatalf("GetSmartPlaylistTracks failed: %v", err)
}
if len(tracks) != 1 {
t.Fatalf("got %d tracks, want 1", len(tracks))
}
if tracks[0].Artist != "Band B" {
t.Errorf("artist = %q, want Band B", tracks[0].Artist)
}
}
func TestSmartPlaylistCreateInvalidJSON(t *testing.T) {
t.Parallel()
backend/queue/persistence.go
+3 -3
View File
@@ -237,7 +237,7 @@ func (q *Queue) lookupChunk(
return
}
rows, err := q.db.Queries.LookupTrackMetaByPaths(q.db.Ctx, paths)
rows, err := q.db.ReadQueries.LookupTrackMetaByPaths(q.db.Ctx, paths)
if err != nil {
q.logger.Error("Batch metadata lookup failed", "err", err)
@@ -415,7 +415,7 @@ func (q *Queue) RestoreState() {
defer q.mu.Unlock()
// Restore queue metadata.
state, err := q.db.Queries.GetQueueState(q.db.Ctx)
state, err := q.db.ReadQueries.GetQueueState(q.db.Ctx)
if err != nil {
q.logger.Error("Failed to load queue state", "err", err)
@@ -444,7 +444,7 @@ func (q *Queue) RestoreState() {
}
// Restore queue tracks.
rows, err := q.db.Queries.GetQueueTracks(q.db.Ctx)
rows, err := q.db.ReadQueries.GetQueueTracks(q.db.Ctx)
if err != nil {
q.logger.Error("Failed to load queue tracks", "err", err)
backend/smartplaylist/smartplaylist.go
+68 -22
View File
@@ -13,6 +13,7 @@ import (
"strings"
"time"
"yellowjacket/backend/coverart"
"yellowjacket/backend/database"
"yellowjacket/backend/library"
)
@@ -54,6 +55,7 @@ var fieldMap = map[string]string{
"album": "album",
"genre": "genre",
"year": "year",
"release_year": "release_year",
"composer": "composer",
"file_type": "file_type",
"duration": "length_milliseconds",
@@ -72,6 +74,7 @@ var fieldMap = map[string]string{
// numericFields identifies fields that accept numeric operators.
var numericFields = map[string]bool{
"year": true,
"release_year": true,
"duration": true,
"sample_rate": true,
"bit_depth": true,
@@ -552,7 +555,11 @@ const leanTrackQuery = `SELECT
af.bitrate,
af.file_size,
af.play_count,
COALESCE(af.last_played, '') AS last_played
COALESCE(af.last_played, '') AS last_played,
af.cover_art_path,
af.artist_mbid,
af.release_group_mbid,
af.recording_mbid
FROM (
SELECT
af.id,
@@ -564,7 +571,17 @@ FROM (
r.track_number,
r.disc_number,
COALESCE(rg.name, '') AS album,
COALESCE(r.year, 0) AS year,
-- Two year fields, matching the canonical track_metadata view:
-- year — the album's original (first-release) year,
-- the default users filter on. A 1977 album owned
-- as a 2010s reissue still filters as 1977.
-- release_year — the year of the specific release in the library
-- (the file/release-group tag), e.g. 2013 for that
-- reissue.
-- Both fall back through rg.year → r.year so a track without full
-- MusicBrainz data still gets a sensible year.
COALESCE(rg.original_year, rg.year, r.year, 0) AS year,
COALESCE(rg.year, r.year, 0) AS release_year,
COALESCE(r.composer, '') AS composer,
COALESCE(ft.extension, '') AS file_type,
af.sample_rate,
@@ -574,7 +591,15 @@ FROM (
af.file_size,
af.library_id,
af.play_count,
af.last_played
af.last_played,
COALESCE(ca.file_path, '') AS cover_art_path,
COALESCE((SELECT a.mbid
FROM artist_credit_artist aca
JOIN artists a ON a.id = aca.artist_id
WHERE aca.credit_id = ac.id
LIMIT 1), '') AS artist_mbid,
COALESCE(rg.mbid, '') AS release_group_mbid,
COALESCE(r.mbid, '') AS recording_mbid
FROM audio_files af
LEFT JOIN recordings r ON af.recording_id = r.id
LEFT JOIN artist_credit ac ON r.artist_credit_id = ac.id
@@ -585,6 +610,7 @@ FROM (
GROUP BY recording_id
) rgr ON r.id = rgr.recording_id
LEFT JOIN release_groups rg ON rgr.release_group_id = rg.id
LEFT JOIN cover_art ca ON rg.cover_art_id = ca.id
LEFT JOIN file_types ft ON af.file_type_id = ft.id
) af`
@@ -752,6 +778,11 @@ func scanTracks(rows *sql.Rows) ([]library.Track, []int64, error) {
fileSize int64
playCount int64
lastPlayed string
coverArtPath string
artistMBID string
releaseGroupMBID string
recordingMBID string
)
if err := rows.Scan(
@@ -761,31 +792,46 @@ func scanTracks(rows *sql.Rows) ([]library.Track, []int64, error) {
&sampleRate, &bitDepth, &channels,
&bitrate, &fileSize,
&playCount, &lastPlayed,
&coverArtPath, &artistMBID,
&releaseGroupMBID, &recordingMBID,
); err != nil {
return nil, nil, fmt.Errorf(
"could not scan smart playlist row: %w", err,
)
}
tracks = append(tracks, library.Track{
TrackName: title,
ArtistName: artistName,
TrackLength: strconv.FormatInt(lengthMs, 10),
FilePath: filePath,
TrackNumber: trackNumber.Int64,
DiscNumber: discNumber.Int64,
Album: album,
Year: year,
Composer: composer,
FileType: fileType,
SampleRate: sampleRate,
BitDepth: bitDepth,
Channels: channels,
Bitrate: bitrate,
FileSize: fileSize,
PlayCount: playCount,
LastPlayed: lastPlayed,
})
track := library.Track{
TrackName: title,
ArtistName: artistName,
TrackLength: strconv.FormatInt(lengthMs, 10),
FilePath: filePath,
TrackNumber: trackNumber.Int64,
DiscNumber: discNumber.Int64,
Album: album,
Year: year,
Composer: composer,
FileType: fileType,
SampleRate: sampleRate,
BitDepth: bitDepth,
Channels: channels,
Bitrate: bitrate,
FileSize: fileSize,
PlayCount: playCount,
LastPlayed: lastPlayed,
ArtistMBID: artistMBID,
ReleaseGroupMBID: releaseGroupMBID,
RecordingMBID: recordingMBID,
}
if coverArtPath != "" {
urls := coverart.ResolveURLs(coverArtPath)
track.CoverArtPath = urls.Original
track.CoverArtSmall = urls.Small
track.CoverArtMedium = urls.Medium
track.CoverArtLarge = urls.Large
}
tracks = append(tracks, track)
recordingIDs = append(recordingIDs, recordingID.Int64)
}
backend/smartplaylist/smartplaylist_test.go
+117
View File
@@ -1618,3 +1618,120 @@ func TestEvaluate_MultiGenreTrackGenreField(t *testing.T) {
)
}
}
// TestEvaluate_YearUsesOriginalReleaseYear is a regression test for a
// bug where the smart-playlist year filter tested recordings.year (the
// file's ID3/reissue tag year) instead of the release group's original
// first-release year, the way the canonical track_metadata view and the
// UI do. That made a 1977 album owned as a 2010s reissue leak into a
// "2010s" year filter even though it displays as 1977.
func TestEvaluate_YearUsesOriginalReleaseYear(t *testing.T) {
t.Parallel()
db := database.NewTestDB(t)
// One track: a 1977 album the user owns as a 2013 reissue. The file
// tag / recording year is 2013, but the release group's original
// (first-release) year is 1977.
exec := func(query string, args ...any) {
t.Helper()
if _, err := db.ExecContext(query, args...); err != nil {
t.Fatalf("exec %q: %v", query, err)
}
}
exec("INSERT INTO artist_credit (id, text) VALUES (1, ?)", "The B-52's")
exec(
"INSERT INTO release_groups (id, name, year, original_year) "+
"VALUES (1, ?, 2013, 1977)",
"Reissue Compilation",
)
exec(
"INSERT INTO recordings (id, name, artist_credit_id, year) "+
"VALUES (1, ?, 1, 2013)",
"Rock Lobster",
)
exec(
"INSERT INTO audio_files (id, file_path, length_milliseconds, "+
"file_type_id, recording_id) VALUES (1, ?, 300000, 1, 1)",
"/music/b52s/rock_lobster.mp3",
)
exec(
"INSERT INTO release_group_recordings " +
"(release_group_id, recording_id) VALUES (1, 1)",
)
// A "2010s" filter must NOT match — the album is originally from 1977.
tracks, err := Evaluate(db, RuleSet{
Rules: []Rule{
{Field: "year", Operator: "between", Value: "2010,2019"},
},
})
if err != nil {
t.Fatalf("Evaluate 2010s: %v", err)
}
if len(tracks) != 0 {
t.Errorf(
"2010s filter matched %d tracks, want 0 "+
"(reissue year leaked in)", len(tracks),
)
}
// A "1970s" filter must match — original_year is 1977.
tracks, err = Evaluate(db, RuleSet{
Rules: []Rule{
{Field: "year", Operator: "between", Value: "1970,1979"},
},
})
if err != nil {
t.Fatalf("Evaluate 1970s: %v", err)
}
if len(tracks) != 1 {
t.Fatalf(
"1970s filter matched %d tracks, want 1", len(tracks),
)
}
if tracks[0].Year != 1977 {
t.Errorf("Year = %d, want 1977", tracks[0].Year)
}
// The release_year field, by contrast, tracks the specific release
// owned (the 2013 reissue), so a 2010s filter on it MUST match.
tracks, err = Evaluate(db, RuleSet{
Rules: []Rule{
{Field: "release_year", Operator: "between", Value: "2010,2019"},
},
})
if err != nil {
t.Fatalf("Evaluate release_year 2010s: %v", err)
}
if len(tracks) != 1 {
t.Fatalf(
"release_year 2010s filter matched %d tracks, want 1",
len(tracks),
)
}
// And a 1970s release_year filter must NOT match — the owned
// release is from 2013.
tracks, err = Evaluate(db, RuleSet{
Rules: []Rule{
{Field: "release_year", Operator: "between", Value: "1970,1979"},
},
})
if err != nil {
t.Fatalf("Evaluate release_year 1970s: %v", err)
}
if len(tracks) != 0 {
t.Errorf(
"release_year 1970s filter matched %d tracks, want 0",
len(tracks),
)
}
}
frontend/pnpm-workspace.yaml
+2
View File
@@ -0,0 +1,2 @@
allowBuilds:
esbuild: true
frontend/src/components/autotag-view/autotag-view.ts
+1290 -370
View File
File diff suppressed because it is too large Load Diff
frontend/src/components/config-page/config-page.ts
+1 -1
View File
@@ -1945,7 +1945,7 @@ export class ConfigPage extends LitElement {
return html`
<config-section
heading="Search Index"
description="The explore search index pre-caches popular artists, albums, and tracks from ListenBrainz for fast offline search."
description="The explore search index is built from the MusicBrainz/ListenBrainz data dumps — popular artists, albums, and tracks with listen counts — for fast offline search."
.open=${true}
>
<div class="index-status">
frontend/src/components/cover-grid/cover-grid.ts
+3 -2
View File
@@ -46,6 +46,7 @@ import type { DragPayload } from '@utils/drag-controller';
import { ContextMenuController } from '@utils/context-menu-controller.js';
import type { ContextMenuHost } from '@utils/context-menu-controller.js';
import { FavoritesController } from '@store/controllers/favorites-controller';
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
import {
createAlbumArtDragImage,
createDragImage,
@@ -264,7 +265,7 @@ export class CoverGrid
private splitEntriesCacheKey: GridEntry[] | null = null;
private splitEntriesCacheIndex = -1;
static override styles = coverGridStyles;
static override styles = [coverGridStyles, exploreLinkStyles];
/* ====================================================================
* Reactive state
@@ -1832,7 +1833,7 @@ export class CoverGrid
class="artist-name"
title="${album.ArtistName}"
>
${album.ArtistName}
${artistLink(album.ArtistName, album.ArtistMBID ?? '')}
</div>
</div>
</div>
frontend/src/components/explore-album-details/explore-album-details.ts
+72 -5
View File
@@ -15,6 +15,9 @@ type MBTrack = explore.MBTrack;
import { exploreCache } from '../../store/explore-cache';
import { exploreSettings } from '../../store/explore-settings';
import { libraryStore } from '../../store/library-store';
import { artistLink, exploreLinkStyles } from '../../utils/explore-link';
import { EventsOn } from '@runtime/runtime';
import { Events } from '../../events';
import '@awesome.me/webawesome/dist/components/icon/icon.js';
import '../library-status-indicator/library-status-indicator.js';
@@ -113,6 +116,7 @@ export class ExploreAlbumDetails extends LitElement {
static override styles = [
designTokens,
exploreLinkStyles,
css`
:host {
display: flex;
@@ -451,6 +455,13 @@ export class ExploreAlbumDetails extends LitElement {
/* ── Lifecycle ── */
private unsubSettings?: () => void;
private unsubReleasesReady?: () => void;
/** Release-group MBIDs whose AlbumReleasesReady event we've handled,
* so a background BrowseReleases fetch re-hydrates versions once. */
private releasesReloaded = new Set<string>();
/** Fallback timer that stops the versions spinner if the background
* BrowseReleases fetch never signals readiness. */
private releasesFallbackTimer?: number;
override connectedCallback() {
super.connectedCallback();
@@ -465,11 +476,45 @@ export class ExploreAlbumDetails extends LitElement {
void this.loadAllData();
}
});
// A background BrowseReleases fetch (cold album, versions +
// tracklist not cached yet) finished — re-fetch the versions once
// per release group so they fill in without the initial request
// having blocked on a live MusicBrainz browse.
this.unsubReleasesReady = EventsOn(
Events.AlbumReleasesReady,
(mbid: string) => {
if (mbid !== this.releaseGroupMBID) return;
if (this.releasesReloaded.has(mbid)) return;
if (this.releasesFallbackTimer) clearTimeout(this.releasesFallbackTimer);
this.releasesReloaded.add(mbid);
void this.fetchReleases(mbid);
},
);
}
override disconnectedCallback() {
super.disconnectedCallback();
this.unsubSettings?.();
this.unsubReleasesReady?.();
if (this.releasesFallbackTimer) clearTimeout(this.releasesFallbackTimer);
}
/**
* Arm a one-shot fallback that stops the versions spinner if
* AlbumReleasesReady never arrives (e.g. the background browse stalled
* or the release group genuinely has no releases).
*/
private armReleasesFallback(mbid: string) {
if (this.releasesFallbackTimer) clearTimeout(this.releasesFallbackTimer);
this.releasesFallbackTimer = window.setTimeout(() => {
if (this.releasesReloaded.has(mbid)) return;
this.releasesReloaded.add(mbid);
if (this.releases.length === 0) this.loadingReleases = false;
}, 12000);
}
/** Whether we've already scrolled to the highlight target. */
@@ -595,7 +640,11 @@ export class ExploreAlbumDetails extends LitElement {
}
// Phase 2: fire API calls independently so each section
// renders as its data arrives.
// renders as its data arrives. Allow the versions section one
// background-fetch re-fetch and arm a fallback so it can't spin
// forever if AlbumReleasesReady never arrives.
this.releasesReloaded.delete(mbid);
this.armReleasesFallback(mbid);
void this.fetchReleaseGroup(mbid);
void this.fetchReleases(mbid);
@@ -828,13 +877,28 @@ export class ExploreAlbumDetails extends LitElement {
private async fetchReleases(mbid: string) {
try {
const releases = await BrowseReleases(mbid);
this.releases = releases ?? [];
this.buildClusters();
if (releases && releases.length > 0) {
// Warm cache hit (or the background re-fetch landed):
// authoritative MB versions replace any local placeholder.
this.releases = releases;
this.buildClusters();
this.loadingReleases = false;
return;
}
// Cold miss: BrowseReleases is cache-first + async and the
// versions/tracklist are still being fetched in the background.
// Don't clobber a tracklist already hydrated from the library —
// keep showing it. Hold the spinner only when there's nothing
// on screen yet; AlbumReleasesReady (or the fallback) resolves it.
if (this.releases.length > 0 || this.releasesReloaded.has(mbid)) {
this.loadingReleases = false;
}
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
this.errorReleases = msg;
console.error(`[explore-album] BrowseReleases error: ${msg}`);
} finally {
this.loadingReleases = false;
}
}
@@ -1437,6 +1501,7 @@ export class ExploreAlbumDetails extends LitElement {
const rg = this.releaseGroup;
const artist = rg.artistCredit || '';
const artistMbid = rg.artistMbid ?? '';
const year = extractYear(rg.firstReleaseDate);
const type = rg.primaryType || '';
@@ -1446,7 +1511,9 @@ export class ExploreAlbumDetails extends LitElement {
return html`
${artist
? html`<div class="album-artist">${artist}</div>`
? html`<div class="album-artist">
${artistLink(artist, artistMbid)}
</div>`
: nothing}
${metaParts.length > 0
? html`

Some files were not shown because too many files have changed in this diff Show More